Mirrors/Dispatcharr
1
0
Fork 1
mirror of https://github.com/Dispatcharr/Dispatcharr.git synced 2026-07-17 16:50:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

Enhancement: Initial fmp4 support and major ts_proxy refactor

Browse source
This commit is contained in:
SergeantPanda 2026-05-08 17:46:47 -05:00
parent 447c0ea25c
commit dc649ffd88
75 changed files with 3071 additions and 383 deletions
Download patch file Download diff file Expand all files Collapse all files

7
CHANGELOG.md
View file

@ -9,6 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **fMP4 streaming support.** Dispatcharr can now serve live channels as fragmented MP4 in addition to MPEG-TS. A dedicated `FMP4RemuxManager` runs a single FFmpeg remux process per active channel, muxes incoming TS data into fMP4 fragments, and stores them in a Redis-backed ring buffer. Multiple clients requesting the same channel in fMP4 all read from the same shared buffer, so only one FFmpeg process runs regardless of viewer count. Clients that request MPEG-TS continue to be served as before. The output format is selected via `?output_format=mpegts|fmp4` on the stream URL, or by the per-user default configured by an admin on the user account, or by the server-wide default in Stream Settings.
- **Output profiles.** Admins can define named transcode profiles under Settings → Output Profiles. Each profile specifies an executable (e.g. `ffmpeg`) and a parameter string that reads raw TS from `pipe:0` and writes the transcoded output to `pipe:1`. When a client requests a profile, one FFmpeg transcode process runs per active (channel, profile) pair and all requesting clients share the resulting output buffer. Common use case: a profile that converts AC3 audio to AAC for browser and mobile clients while the native stream (AC3 intact) continues to serve Plex/Emby/Jellyfin. Profiles are applied via `?output_profile=<id>` on the stream URL, or by per-user and server-wide defaults. Two locked built-in profiles are seeded on first run: **Media Server (AC3 Audio)** (copies video, re-encodes audio to AC3 384k) and **Web Player (AAC Audio)** (copies video, re-encodes audio to AAC 192k).
- **Container format and output profile shown in Stats client rows.** The expanded client row on the Stats page now displays the container format (`mpegts` or `fmp4`) and, when an output profile is active, the profile name.
- **Browser-local web player output profile setting.** A new "Web Player Output Profile" select in Settings → UI Settings lets each browser choose which output profile (if any) is applied when previewing streams in the built-in floating player. The preference is saved to `localStorage` (`dispatcharr-player-prefs`) and picked up by every preview URL builder at click time, so a user whose account default transcodes to AC3 for their media server can still get a browser-compatible audio stream from the preview player without changing their account settings.
- **DVR series rules: EPG channel is now optional.** Series recording rules no longer require a `tvg_id`. Rules with only a title or description filter search across all EPG channels in the 7-day horizon. The evaluator pre-loads one channel per EPG source (lowest channel number) in a single query and resolves the recording channel per-program, so cross-channel searches remain efficient. Saves and the preview endpoint now require at least one of `title`, or `description` instead of requiring `tvg_id`. The preview response includes a `warn: true` flag when more than 50 programs match, and the editor shows an orange alert when this flag is set. The upsert key for rules changed from `tvg_id` alone to `(tvg_id, title)` so multiple rules can target the same channel.
- **DVR series rules: rich title and description matching.** Series recording rules now accept the same boolean / quoted-phrase / regex / whole-word semantics as the EPG Program Search API on both the title and description fields, plus an optional pinned channel. Existing rules continue to work unchanged (default `title_mode=exact`, no description filter, no pinned channel).
- New rule fields: `title_mode` (`exact` / `contains` / `search` / `regex`), `description`, `description_mode` (`contains` / `search` / `regex`), and `channel_id` (optional integer to pin recordings to a specific channel; defaults to the lowest-numbered channel for the EPG as before).
@ -20,6 +24,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- **`ts_proxy` module refactored and renamed to `live_proxy`.** The live-streaming proxy was reorganized into a structured package (`apps/proxy/live_proxy/`) with explicit submodules for each stage of the pipeline: `input/` (HTTP streamer, stream manager, TS ring buffer), `output/ts/` (MPEG-TS client generator), `output/fmp4/` (fMP4 remux manager, Redis buffer, client generator), and `output/profile/` (output profile transcode manager). A dedicated `redis_keys.py` module centralizes all Redis key name construction for the proxy. Existing behavior for MPEG-TS clients is unchanged.
- **XC API `allowed_output_formats` now includes `mp4`.** The `user_info` block returned by `player_api.php` and `get.php` previously advertised only `["ts"]`. It now advertises `["ts", "mp4"]` for all users, enabling XC-compatible clients that support fMP4 to request `.mp4` stream URLs which are proxied as fMP4.
- **Browser preview URLs always force `mpegts` output.** The four in-app preview buttons (Channels table, channel stream list, Streams table, Stats client row) now append `?output_format=mpegts` (and the web player profile, if set) to the preview URL so the built-in `mpegts.js` player always receives a compatible stream, regardless of the user's configured default output format.
- **Series rules modal redesign.** Rules now display a one-line summary with the title-match mode, description filter (when present), and a "Pinned channel" badge when a `channel_id` is configured. The previous list rendering remains for legacy rules.
- **EPG Program Search API** (`GET /api/epg/programs/search/`): a new endpoint for querying EPG program data with rich filtering and query support. - Thanks [@northernpowerhouse](https://github.com/northernpowerhouse)
- **Text search** on title and description with AND/OR boolean operators (case-insensitive), quoted phrase matching (`"Law and Order"` treats _and_ as literal text), parenthetical grouping (`(Newcastle OR NEW) AND (Villa OR AST)`), whole-word mode (`title_whole_words=true`), and regex mode (`title_regex=true`).

10
apps/accounts/serializers.py
View file

@ -116,12 +116,16 @@ class UserSerializer(serializers.ModelSerializer):
channel_profiles = validated_data.pop("channel_profiles", None)
# Merge custom_properties instead of replacing (prevents data loss)
# Strip null values — sending null for a key omits it rather than overwriting with null
# null values are explicit deletions; all other values overwrite existing
custom_properties = validated_data.pop("custom_properties", None)
if custom_properties is not None:
existing = instance.custom_properties or {}
cleaned = {k: v for k, v in custom_properties.items() if v is not None}
merged = {**existing, **cleaned}
merged = dict(existing)
for k, v in custom_properties.items():
if v is None:
merged.pop(k, None)
else:
merged[k] = v
# Scrub stale nav IDs so the DB self-heals on next save
for nav_field in ('navOrder', 'hiddenNav'):
if nav_field in merged and isinstance(merged[nav_field], list):

8
apps/channels/api_views.py
View file

@ -3161,8 +3161,8 @@ def _stop_dvr_clients(channel_uuid, recording_id=None):
Returns the number of DVR clients stopped.
"""
from core.utils import RedisClient
from apps.proxy.ts_proxy.redis_keys import RedisKeys
from apps.proxy.ts_proxy.services.channel_service import ChannelService
from apps.proxy.live_proxy.redis_keys import RedisKeys
from apps.proxy.live_proxy.services.channel_service import ChannelService
r = RedisClient.get_client()
if not r:
@ -3688,8 +3688,8 @@ class RecordingViewSet(viewsets.ModelViewSet):
channel_uuid = str(instance.channel.uuid)
# Lazy imports to avoid module overhead if proxy isn't used
from core.utils import RedisClient
from apps.proxy.ts_proxy.redis_keys import RedisKeys
from apps.proxy.ts_proxy.services.channel_service import ChannelService
from apps.proxy.live_proxy.redis_keys import RedisKeys
from apps.proxy.live_proxy.services.channel_service import ChannelService
r = RedisClient.get_client()
if r:

20
apps/channels/models.py
View file

@ -3,8 +3,8 @@ from django.core.exceptions import ValidationError
from django.conf import settings
from core.models import StreamProfile, CoreSettings
from core.utils import RedisClient
from apps.proxy.ts_proxy.redis_keys import RedisKeys
from apps.proxy.ts_proxy.constants import ChannelMetadataField
from apps.proxy.live_proxy.redis_keys import RedisKeys
from apps.proxy.live_proxy.constants import ChannelMetadataField
import logging
import uuid
from django.utils import timezone
@ -477,7 +477,7 @@ class Channel(models.Model):
candidates = []
# 1) Try to get active channel IDs for this profile from an index set if available
ch_set_key = f"ts_proxy:profile:{profile_id}:channels"
ch_set_key = f"live:profile:{profile_id}:channels"
try:
ch_ids = { (int(x) if not isinstance(x, int) else x) for x in (redis_client.smembers(ch_set_key) or set()) }
except Exception:
@ -489,7 +489,7 @@ class Channel(models.Model):
# 2) Fallback: scan metadata keys and filter by m3u_profile == profile_id
if not ch_ids:
cursor = 0
pattern = "ts_proxy:channel:*:metadata"
pattern = "live:channel:*:metadata"
while True:
cursor, keys = redis_client.scan(cursor=cursor, match=pattern, count=500)
if keys:
@ -505,7 +505,7 @@ class Channel(models.Model):
pid = None
if pid == profile_id:
parts = k.split(":") # ts_proxy:channel:{id}:metadata
parts = k.split(":") # live:channel:{id}:metadata
if len(parts) >= 4:
try:
ch_ids.add(int(parts[2]))
@ -526,7 +526,7 @@ class Channel(models.Model):
continue
# Skip if recently preempted
last_preempt_key = f"ts_proxy:channel:{ch_id}:last_preempt"
last_preempt_key = f"live:channel:{ch_id}:last_preempt"
try:
last_preempt = float(redis_client.get(last_preempt_key) or 0.0)
except Exception:
@ -535,14 +535,14 @@ class Channel(models.Model):
continue
# Clients and their levels
clients_key = f"ts_proxy:channel:{ch_id}:clients"
clients_key = f"live:channel:{ch_id}:clients"
member_ids = list(redis_client.smembers(clients_key) or [])
viewer_count = len(member_ids)
max_viewer_level = 0
if viewer_count:
pipe = redis_client.pipeline()
for cid in member_ids:
pipe.hget(f"ts_proxy:channel:{ch_id}:clients:{cid}", "user_level")
pipe.hget(f"live:channel:{ch_id}:clients:{cid}", "user_level")
levels_raw = pipe.execute()
levels = []
for lv in levels_raw:
@ -557,7 +557,7 @@ class Channel(models.Model):
continue
# Metadata (protected/recording/started_at_ts)
meta_key = f"ts_proxy:channel:{ch_id}:metadata"
meta_key = f"live:channel:{ch_id}:metadata"
try:
protected, recording, started_at_ts = redis_client.hmget(
meta_key, "protected", "recording", "started_at_ts"
@ -593,7 +593,7 @@ class Channel(models.Model):
# Mark preempt timestamp to avoid thrashing
try:
redis_client.set(f"ts_proxy:channel:{victim_id}:last_preempt", str(time.time()), ex=3600)
redis_client.set(f"live:channel:{victim_id}:last_preempt", str(time.time()), ex=3600)
except Exception:
pass

4
apps/channels/signals.py
View file

@ -65,7 +65,7 @@ def stop_proxy_session_before_channel_delete(sender, instance, **kwargs):
"""
When a Channel is deleted, stop any active TS proxy session for it first.
Without this, the proxy's Redis state (ts_proxy:channel:{uuid}:*) survives
Without this, the proxy's Redis state (live:channel:{uuid}:*) survives
the Channel row and the connected clients' "Stop" button hits
`ChannelService.stop_channel(uuid)`, which calls `Channel.objects.get(uuid=...)`
and crashes with DoesNotExist (reported as 'Channel not found' in the UI).
@ -74,7 +74,7 @@ def stop_proxy_session_before_channel_delete(sender, instance, **kwargs):
via the same signal path.
"""
try:
from apps.proxy.ts_proxy.services.channel_service import ChannelService
from apps.proxy.live_proxy.services.channel_service import ChannelService
channel_uuid = str(instance.uuid) if instance.uuid else None
if not channel_uuid:

4
apps/channels/tasks.py
View file

@ -2729,8 +2729,8 @@ def run_recording(recording_id, channel_id, start_time_str, end_time_str):
# Try to get stream stats from TS proxy Redis metadata
try:
from core.utils import RedisClient
from apps.proxy.ts_proxy.redis_keys import RedisKeys
from apps.proxy.ts_proxy.constants import ChannelMetadataField
from apps.proxy.live_proxy.redis_keys import RedisKeys
from apps.proxy.live_proxy.constants import ChannelMetadataField
r = RedisClient.get_client()
if r is not None:

4
apps/channels/tests/test_recording_stop_cancel.py
View file

@ -282,8 +282,8 @@ class StopDvrClientsTests(TestCase):
def setUp(self):
self.channel = Channel.objects.create(channel_number=95, name="DVR Clients Channel")
self._redis = "core.utils.RedisClient"
self._sc = "apps.proxy.ts_proxy.services.channel_service.ChannelService.stop_client"
self._sch = "apps.proxy.ts_proxy.services.channel_service.ChannelService.stop_channel"
self._sc = "apps.proxy.live_proxy.services.channel_service.ChannelService.stop_client"
self._sch = "apps.proxy.live_proxy.services.channel_service.ChannelService.stop_channel"
def _mock_redis(self, client_ids, ua_map):
r = MagicMock()

24
apps/channels/tests/test_ts_proxy_ghost_clients.py
View file

@ -11,9 +11,9 @@ from unittest.mock import MagicMock, patch, PropertyMock
from django.test import TestCase
from apps.proxy.ts_proxy.client_manager import ClientManager
from apps.proxy.ts_proxy.constants import ChannelMetadataField, ChannelState
from apps.proxy.ts_proxy.redis_keys import RedisKeys
from apps.proxy.live_proxy.client_manager import ClientManager
from apps.proxy.live_proxy.constants import ChannelMetadataField, ChannelState
from apps.proxy.live_proxy.redis_keys import RedisKeys
# ---------------------------------------------------------------------------
@ -141,7 +141,7 @@ class RemoveGhostClientsTests(TestCase):
# Detailed stats path: exercises get_detailed_channel_info()
# ---------------------------------------------------------------------------
@patch("apps.proxy.ts_proxy.channel_status.ProxyServer")
@patch("apps.proxy.live_proxy.channel_status.ProxyServer")
class DetailedStatsGhostClientTests(TestCase):
"""get_detailed_channel_info() should remove ghost clients whose metadata
hash has expired from the Redis client SET."""
@ -162,7 +162,7 @@ class DetailedStatsGhostClientTests(TestCase):
def test_ghost_client_removed_from_set(self, mock_proxy_cls):
"""Ghost client should be SREM'd and excluded from result."""
from apps.proxy.ts_proxy.channel_status import ChannelStatus
from apps.proxy.live_proxy.channel_status import ChannelStatus
def hgetall_side_effect(key):
if "clients:" in key:
@ -181,7 +181,7 @@ class DetailedStatsGhostClientTests(TestCase):
def test_live_client_preserved(self, mock_proxy_cls):
"""Client with valid metadata should appear in results."""
from apps.proxy.ts_proxy.channel_status import ChannelStatus
from apps.proxy.live_proxy.channel_status import ChannelStatus
def hgetall_side_effect(key):
if "clients:" in key:
@ -204,7 +204,7 @@ class DetailedStatsGhostClientTests(TestCase):
def test_mixed_ghost_and_live(self, mock_proxy_cls):
"""Only ghost clients should be removed; live ones preserved."""
from apps.proxy.ts_proxy.channel_status import ChannelStatus
from apps.proxy.live_proxy.channel_status import ChannelStatus
def hgetall_side_effect(key):
if "clients:" in key:
@ -231,7 +231,7 @@ class DetailedStatsGhostClientTests(TestCase):
# Basic stats path: exercises get_basic_channel_info()
# ---------------------------------------------------------------------------
@patch("apps.proxy.ts_proxy.channel_status.ProxyServer")
@patch("apps.proxy.live_proxy.channel_status.ProxyServer")
class BasicStatsGhostClientTests(TestCase):
"""get_basic_channel_info() should call remove_ghost_clients(), skip
ghosts from display, and correct client_count."""
@ -260,7 +260,7 @@ class BasicStatsGhostClientTests(TestCase):
def test_ghost_removed_and_count_corrected(self, mock_proxy_cls):
"""Ghost client should be cleaned and client_count decremented."""
from apps.proxy.ts_proxy.channel_status import ChannelStatus
from apps.proxy.live_proxy.channel_status import ChannelStatus
redis = self._setup_redis(
mock_proxy_cls,
@ -276,7 +276,7 @@ class BasicStatsGhostClientTests(TestCase):
def test_live_client_count_preserved(self, mock_proxy_cls):
"""Live clients should be counted correctly."""
from apps.proxy.ts_proxy.channel_status import ChannelStatus
from apps.proxy.live_proxy.channel_status import ChannelStatus
redis = self._setup_redis(
mock_proxy_cls,
@ -295,7 +295,7 @@ class BasicStatsGhostClientTests(TestCase):
# Orphaned channel cleanup: exercises _check_orphaned_metadata()
# ---------------------------------------------------------------------------
@patch("apps.proxy.ts_proxy.channel_status.ProxyServer")
@patch("apps.proxy.live_proxy.channel_status.ProxyServer")
class OrphanedChannelGhostValidationTests(TestCase):
"""_check_orphaned_metadata() should validate client SET entries when
owner is dead and client_count > 0. If all clients are ghosts, it
@ -334,7 +334,7 @@ class OrphanedChannelGhostValidationTests(TestCase):
def test_all_ghosts_triggers_cleanup(self, mock_proxy_cls):
"""When all clients are ghosts, channel should be cleaned up."""
from apps.proxy.ts_proxy.server import ProxyServer
from apps.proxy.live_proxy.server import ProxyServer
channel_id = "00000000-0000-0000-0000-000000000005"
server, redis = self._make_server_for_orphan_check(

8
apps/channels/tests/test_ts_proxy_initializing.py
View file

@ -11,9 +11,9 @@ from unittest.mock import MagicMock, patch
from django.test import TestCase
from apps.proxy.ts_proxy.constants import ChannelMetadataField, ChannelState
from apps.proxy.ts_proxy.redis_keys import RedisKeys
from apps.proxy.ts_proxy.stream_manager import StreamManager
from apps.proxy.live_proxy.constants import ChannelMetadataField, ChannelState
from apps.proxy.live_proxy.redis_keys import RedisKeys
from apps.proxy.live_proxy.input.manager import StreamManager
# ---------------------------------------------------------------------------
@ -84,7 +84,7 @@ def _run_finally_block(sm, owner_value, current_state):
redis.setex.reset_mock()
with patch.object(threading, 'Thread', return_value=MagicMock()):
with patch('apps.proxy.ts_proxy.stream_manager.ConfigHelper') as mock_cfg:
with patch('apps.proxy.live_proxy.input.manager.ConfigHelper') as mock_cfg:
mock_cfg.max_stream_switches.return_value = 0
mock_cfg.max_retries.return_value = sm.max_retries
sm.run()

36
apps/channels/tests/test_ts_proxy_keepalive.py
View file

@ -22,7 +22,7 @@ class OwnerWorkerKeepaliveTests(TestCase):
"""Owner worker has a stream_manager; keepalive logic uses it directly."""
def _make_generator(self, healthy, at_buffer_head, consecutive_empty):
from apps.proxy.ts_proxy.stream_generator import StreamGenerator
from apps.proxy.live_proxy.output.ts.generator import StreamGenerator
gen = StreamGenerator.__new__(StreamGenerator)
gen.channel_id = "00000000-0000-0000-0000-000000000001"
gen.client_id = "test-client"
@ -73,7 +73,7 @@ class NonOwnerWorkerKeepaliveTests(TestCase):
"""Non-owner worker has stream_manager=None; health determined from Redis."""
def _make_generator(self, consecutive_empty=10):
from apps.proxy.ts_proxy.stream_generator import StreamGenerator
from apps.proxy.live_proxy.output.ts.generator import StreamGenerator
gen = StreamGenerator.__new__(StreamGenerator)
gen.channel_id = "00000000-0000-0000-0000-000000000002"
gen.client_id = "test-client-nonowner"
@ -108,7 +108,7 @@ class NonOwnerWorkerKeepaliveTests(TestCase):
fresh_ts = str(time.time() - 2.0).encode()
server = self._mock_proxy_server(fresh_ts)
with patch("apps.proxy.ts_proxy.stream_generator.ProxyServer") as MockPS:
with patch("apps.proxy.live_proxy.output.ts.generator.ProxyServer") as MockPS:
MockPS.get_instance.return_value = server
result = gen._should_send_keepalive(gen.local_index)
@ -120,7 +120,7 @@ class NonOwnerWorkerKeepaliveTests(TestCase):
stale_ts = str(time.time() - 12.0).encode()
server = self._mock_proxy_server(stale_ts)
with patch("apps.proxy.ts_proxy.stream_generator.ProxyServer") as MockPS:
with patch("apps.proxy.live_proxy.output.ts.generator.ProxyServer") as MockPS:
MockPS.get_instance.return_value = server
result = gen._should_send_keepalive(gen.local_index)
@ -132,7 +132,7 @@ class NonOwnerWorkerKeepaliveTests(TestCase):
ts = str(time.time() - 10.0).encode()
server = self._mock_proxy_server(ts)
with patch("apps.proxy.ts_proxy.stream_generator.ProxyServer") as MockPS:
with patch("apps.proxy.live_proxy.output.ts.generator.ProxyServer") as MockPS:
MockPS.get_instance.return_value = server
result = gen._should_send_keepalive(gen.local_index)
@ -143,7 +143,7 @@ class NonOwnerWorkerKeepaliveTests(TestCase):
gen = self._make_generator()
server = self._mock_proxy_server(None)
with patch("apps.proxy.ts_proxy.stream_generator.ProxyServer") as MockPS:
with patch("apps.proxy.live_proxy.output.ts.generator.ProxyServer") as MockPS:
MockPS.get_instance.return_value = server
result = gen._should_send_keepalive(gen.local_index)
@ -155,7 +155,7 @@ class NonOwnerWorkerKeepaliveTests(TestCase):
server = MagicMock()
server.redis_client = None
with patch("apps.proxy.ts_proxy.stream_generator.ProxyServer") as MockPS:
with patch("apps.proxy.live_proxy.output.ts.generator.ProxyServer") as MockPS:
MockPS.get_instance.return_value = server
result = gen._should_send_keepalive(gen.local_index)
@ -165,7 +165,7 @@ class NonOwnerWorkerKeepaliveTests(TestCase):
"""Non-owner, Redis raises an exception -> conservative, no keepalive."""
gen = self._make_generator()
with patch("apps.proxy.ts_proxy.stream_generator.ProxyServer") as MockPS:
with patch("apps.proxy.live_proxy.output.ts.generator.ProxyServer") as MockPS:
MockPS.get_instance.side_effect = Exception("Redis error")
result = gen._should_send_keepalive(gen.local_index)
@ -177,7 +177,7 @@ class NonOwnerWorkerKeepaliveTests(TestCase):
gen.buffer.index = 100 # far ahead of local_index=10
server = self._mock_proxy_server(None)
with patch("apps.proxy.ts_proxy.stream_generator.ProxyServer") as MockPS:
with patch("apps.proxy.live_proxy.output.ts.generator.ProxyServer") as MockPS:
MockPS.get_instance.return_value = server
result = gen._should_send_keepalive(gen.local_index)
@ -189,7 +189,7 @@ class NonOwnerWorkerKeepaliveTests(TestCase):
stale_ts = str(time.time() - 30.0).encode()
server = self._mock_proxy_server(stale_ts)
with patch("apps.proxy.ts_proxy.stream_generator.ProxyServer") as MockPS:
with patch("apps.proxy.live_proxy.output.ts.generator.ProxyServer") as MockPS:
MockPS.get_instance.return_value = server
result = gen._should_send_keepalive(gen.local_index)
@ -204,7 +204,7 @@ class DoStatsUpdateTests(TestCase):
"""_do_stats_update runs the actual Redis scan + WebSocket call."""
def _make_client_manager(self):
from apps.proxy.ts_proxy.client_manager import ClientManager
from apps.proxy.live_proxy.client_manager import ClientManager
cm = ClientManager.__new__(ClientManager)
cm.channel_id = "00000000-0000-0000-0000-000000000004"
cm._heartbeat_running = False
@ -217,7 +217,7 @@ class DoStatsUpdateTests(TestCase):
mock_redis = MagicMock()
mock_redis.scan.return_value = (0, [])
with patch("apps.proxy.ts_proxy.client_manager.send_websocket_update") as mock_ws, \
with patch("apps.proxy.live_proxy.client_manager.send_websocket_update") as mock_ws, \
patch("redis.Redis.from_url", return_value=mock_redis):
cm._do_stats_update()
@ -238,18 +238,18 @@ class DoStatsUpdateTests(TestCase):
self.fail(f"_do_stats_update raised an exception: {e}")
def test_do_stats_update_scans_channel_client_keys(self):
"""Must scan for ts_proxy:channel:*:clients pattern."""
"""Must scan for live:channel:*:clients pattern."""
cm = self._make_client_manager()
mock_redis = MagicMock()
mock_redis.scan.return_value = (0, [])
with patch("apps.proxy.ts_proxy.client_manager.send_websocket_update"), \
with patch("apps.proxy.live_proxy.client_manager.send_websocket_update"), \
patch("redis.Redis.from_url", return_value=mock_redis):
cm._do_stats_update()
scan_call = mock_redis.scan.call_args
self.assertIn("ts_proxy:channel:*:clients", str(scan_call))
self.assertIn("live:channel:*:clients", str(scan_call))
# ---------------------------------------------------------------------------
@ -261,7 +261,7 @@ class ClientRemoveIntegrationTests(TestCase):
def test_remove_client_does_not_block_on_websocket(self):
"""remove_client() must return quickly even if WebSocket is slow."""
from apps.proxy.ts_proxy.client_manager import ClientManager
from apps.proxy.live_proxy.client_manager import ClientManager
cm = ClientManager.__new__(ClientManager)
cm.channel_id = "00000000-0000-0000-0000-000000000005"
@ -269,7 +269,7 @@ class ClientRemoveIntegrationTests(TestCase):
cm.clients = {"test-client-1"}
cm.last_heartbeat_time = {"test-client-1": time.time()}
cm.last_active_time = time.time()
cm.client_set_key = f"ts_proxy:channel:{cm.channel_id}:clients"
cm.client_set_key = f"live:channel:{cm.channel_id}:clients"
cm.client_ttl = 60
cm.worker_id = "worker-1"
cm.proxy_server = MagicMock()
@ -288,7 +288,7 @@ class ClientRemoveIntegrationTests(TestCase):
slow_ws_called.set()
start = time.time()
with patch("apps.proxy.ts_proxy.client_manager.send_websocket_update", side_effect=slow_websocket):
with patch("apps.proxy.live_proxy.client_manager.send_websocket_update", side_effect=slow_websocket):
cm.remove_client("test-client-1")
elapsed = time.time() - start

42
apps/channels/tests/test_ts_proxy_keepalive_duration.py
View file

@ -12,7 +12,7 @@ from django.test import TestCase
def _make_generator(consecutive_empty=10, local_index=10, buffer_index=10):
"""Minimal StreamGenerator stub for testing _stream_data_generator logic."""
from apps.proxy.ts_proxy.stream_generator import StreamGenerator
from apps.proxy.live_proxy.output.ts.generator import StreamGenerator
gen = StreamGenerator.__new__(StreamGenerator)
gen.channel_id = "00000000-0000-0000-0000-000000000099"
@ -65,11 +65,11 @@ class KeepaliveDurationCapTests(TestCase):
patch.object(gen, '_should_send_keepalive', return_value=True), \
patch.object(gen, '_is_ghost_client', return_value=False), \
patch.object(gen, '_is_timeout', return_value=False), \
patch('apps.proxy.ts_proxy.stream_generator.create_ts_packet', return_value=b'\x00' * 188), \
patch('apps.proxy.ts_proxy.stream_generator.ProxyServer') as MockPS, \
patch('apps.proxy.ts_proxy.stream_generator.Config') as MockConfig, \
patch('apps.proxy.ts_proxy.stream_generator.gevent') as mock_gevent, \
patch('apps.proxy.ts_proxy.stream_generator.time') as mock_time:
patch('apps.proxy.live_proxy.output.ts.generator.create_ts_packet', return_value=b'\x00' * 188), \
patch('apps.proxy.live_proxy.output.ts.generator.ProxyServer') as MockPS, \
patch('apps.proxy.live_proxy.output.ts.generator.Config') as MockConfig, \
patch('apps.proxy.live_proxy.output.ts.generator.gevent') as mock_gevent, \
patch('apps.proxy.live_proxy.output.ts.generator.time') as mock_time:
MockPS.get_instance.return_value = None
MockConfig.KEEPALIVE_INTERVAL = 0
@ -105,11 +105,11 @@ class KeepaliveDurationCapTests(TestCase):
patch.object(gen, '_should_send_keepalive', return_value=True), \
patch.object(gen, '_is_ghost_client', return_value=False), \
patch.object(gen, '_is_timeout', return_value=False), \
patch('apps.proxy.ts_proxy.stream_generator.create_ts_packet', return_value=b'\x00' * 188), \
patch('apps.proxy.ts_proxy.stream_generator.ProxyServer') as MockPS, \
patch('apps.proxy.ts_proxy.stream_generator.Config') as MockConfig, \
patch('apps.proxy.ts_proxy.stream_generator.gevent'), \
patch('apps.proxy.ts_proxy.stream_generator.time') as mock_time:
patch('apps.proxy.live_proxy.output.ts.generator.create_ts_packet', return_value=b'\x00' * 188), \
patch('apps.proxy.live_proxy.output.ts.generator.ProxyServer') as MockPS, \
patch('apps.proxy.live_proxy.output.ts.generator.Config') as MockConfig, \
patch('apps.proxy.live_proxy.output.ts.generator.gevent'), \
patch('apps.proxy.live_proxy.output.ts.generator.time') as mock_time:
MockPS.get_instance.return_value = None
MockConfig.KEEPALIVE_INTERVAL = 0
@ -145,11 +145,11 @@ class KeepaliveDurationCapTests(TestCase):
patch.object(gen, '_is_ghost_client', return_value=False), \
patch.object(gen, '_is_timeout', return_value=False), \
patch.object(gen, '_process_chunks', return_value=iter([chunk])), \
patch('apps.proxy.ts_proxy.stream_generator.create_ts_packet', return_value=b'\x00' * 188), \
patch('apps.proxy.ts_proxy.stream_generator.ProxyServer') as MockPS, \
patch('apps.proxy.ts_proxy.stream_generator.Config') as MockConfig, \
patch('apps.proxy.ts_proxy.stream_generator.gevent'), \
patch('apps.proxy.ts_proxy.stream_generator.time') as mock_time:
patch('apps.proxy.live_proxy.output.ts.generator.create_ts_packet', return_value=b'\x00' * 188), \
patch('apps.proxy.live_proxy.output.ts.generator.ProxyServer') as MockPS, \
patch('apps.proxy.live_proxy.output.ts.generator.Config') as MockConfig, \
patch('apps.proxy.live_proxy.output.ts.generator.gevent'), \
patch('apps.proxy.live_proxy.output.ts.generator.time') as mock_time:
MockPS.get_instance.return_value = None
MockConfig.KEEPALIVE_INTERVAL = 0
@ -170,11 +170,11 @@ class KeepaliveDurationCapTests(TestCase):
patch.object(gen, '_should_send_keepalive', return_value=True), \
patch.object(gen, '_is_ghost_client', return_value=False), \
patch.object(gen, '_is_timeout', return_value=False), \
patch('apps.proxy.ts_proxy.stream_generator.create_ts_packet', return_value=b'\x00' * 188), \
patch('apps.proxy.ts_proxy.stream_generator.ProxyServer') as MockPS, \
patch('apps.proxy.ts_proxy.stream_generator.Config') as MockConfig, \
patch('apps.proxy.ts_proxy.stream_generator.gevent'), \
patch('apps.proxy.ts_proxy.stream_generator.time') as mock_time:
patch('apps.proxy.live_proxy.output.ts.generator.create_ts_packet', return_value=b'\x00' * 188), \
patch('apps.proxy.live_proxy.output.ts.generator.ProxyServer') as MockPS, \
patch('apps.proxy.live_proxy.output.ts.generator.Config') as MockConfig, \
patch('apps.proxy.live_proxy.output.ts.generator.gevent'), \
patch('apps.proxy.live_proxy.output.ts.generator.time') as mock_time:
MockPS.get_instance.return_value = None
MockConfig.KEEPALIVE_INTERVAL = 0

2
apps/m3u/api_views.py
View file

@ -278,7 +278,7 @@ class M3UAccountViewSet(viewsets.ModelViewSet):
"""
instance = self.get_object()
from apps.channels.models import Channel
from apps.proxy.ts_proxy.services.channel_service import (
from apps.proxy.live_proxy.services.channel_service import (
ChannelService,
)

2
apps/m3u/tests/test_account_destroy.py
View file

@ -41,7 +41,7 @@ class M3UAccountDestroyTests(TestCase):
# channel before the DB transaction; the real implementation is
# exercised by integration tests, not unit tests.
self._stop_patch = patch(
"apps.proxy.ts_proxy.services.channel_service."
"apps.proxy.live_proxy.services.channel_service."
"ChannelService.stop_channel"
)
self._stop_patch.start()

4
apps/m3u/tests/test_sync_correctness.py
View file

@ -291,7 +291,7 @@ class ChannelDeleteStopsProxyTests(TestCase):
channel_uuid = str(channel.uuid)
with patch(
"apps.proxy.ts_proxy.services.channel_service.ChannelService.stop_channel"
"apps.proxy.live_proxy.services.channel_service.ChannelService.stop_channel"
) as mock_stop:
channel.delete()
@ -312,7 +312,7 @@ class ChannelDeleteStopsProxyTests(TestCase):
)
with patch(
"apps.proxy.ts_proxy.services.channel_service.ChannelService.stop_channel",
"apps.proxy.live_proxy.services.channel_service.ChannelService.stop_channel",
side_effect=Exception("proxy is down"),
):
channel.delete()

27
apps/output/views.py
View file

@ -184,6 +184,12 @@ def generate_m3u(request, profile_name=None, user=None):
# Check if direct stream URLs should be used instead of proxy
use_direct_urls = request.GET.get('direct', 'false').lower() == 'true'
# Output profile ID to append to proxy stream URLs (triggers pre-delivery transcode)
output_profile_id = request.GET.get('output_profile')
# Output format to append to proxy stream URLs (native ?output_format= or XC-style ?output=)
output_format_param = request.GET.get('output_format') or request.GET.get('output')
# Prefetch streams only when direct URLs are requested (avoids N+1 per channel)
if use_direct_urls:
channels = channels.prefetch_related(
@ -289,7 +295,17 @@ def generate_m3u(request, profile_name=None, user=None):
stream_url = build_absolute_uri_with_port(request, f"/proxy/ts/stream/{channel.uuid}")
else:
# Standard behavior - use proxy URL
stream_url = build_absolute_uri_with_port(request, f"/proxy/ts/stream/{channel.uuid}")
base_stream_url = build_absolute_uri_with_port(request, f"/proxy/ts/stream/{channel.uuid}")
qs_parts = {}
if output_profile_id:
qs_parts['output_profile'] = output_profile_id
if output_format_param:
qs_parts['output_format'] = output_format_param
if qs_parts:
from urllib.parse import urlencode
stream_url = f"{base_stream_url}?{urlencode(qs_parts)}"
else:
stream_url = base_stream_url
m3u_content += extinf_line + stream_url + "\n"
@ -1952,6 +1968,11 @@ def xc_get_user(request):
return user
def _xc_allowed_output_formats(user):
"""Return the list of allowed output formats for the XC API user_info response."""
return ['ts', 'mp4']
def xc_get_info(request, full=False):
user = xc_get_user(request)
@ -1982,9 +2003,7 @@ def xc_get_info(request, full=False):
"exp_date": str(int(time.time()) + (90 * 24 * 60 * 60)),
"active_cons": str(active_cons),
"max_connections": str(max_connections),
"allowed_output_formats": [
"ts",
],
"allowed_output_formats": _xc_allowed_output_formats(user),
},
"server_info": {
"url": hostname,

6
apps/proxy/apps.py
View file

@ -10,8 +10,8 @@ class ProxyConfig(AppConfig):
"""Initialize proxy servers when Django starts"""
if 'manage.py' not in sys.argv:
from .hls_proxy.server import ProxyServer as HLSProxyServer
from .ts_proxy.server import ProxyServer as TSProxyServer
from .live_proxy.server import ProxyServer as LiveProxyServer
# Initialize proxy servers (TS uses singleton to prevent duplicate instances)
# Initialize proxy servers (live uses singleton to prevent duplicate instances)
self.hls_proxy = HLSProxyServer()
self.ts_proxy = TSProxyServer.get_instance()
self.live_proxy = LiveProxyServer.get_instance()

0
apps/proxy/ts_proxy/__init__.py → apps/proxy/live_proxy/__init__.py
View file

6
apps/proxy/ts_proxy/apps.py → apps/proxy/live_proxy/apps.py
View file

@ -1,10 +1,10 @@
import sys
from django.apps import AppConfig
class TSProxyConfig(AppConfig):
class LiveProxyConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'apps.proxy.ts_proxy'
verbose_name = "TS Stream Proxies"
name = 'apps.proxy.live_proxy'
verbose_name = "Live Stream Proxy"
def ready(self):
"""Initialize proxy servers when Django starts"""

24
apps/proxy/ts_proxy/channel_status.py → apps/proxy/live_proxy/channel_status.py
View file

@ -1,13 +1,11 @@
import logging
import time
import re
from .server import ProxyServer
from .redis_keys import RedisKeys
from .constants import TS_PACKET_SIZE, ChannelMetadataField
from redis.exceptions import ConnectionError, TimeoutError
from .utils import get_logger
from .client_manager import ClientManager
from django.db import DatabaseError # Add import for error handling
from django.db import DatabaseError
logger = get_logger()
@ -145,8 +143,15 @@ class ChannelStatus:
'worker_id': client_data.get('worker_id', 'unknown'),
'ip_address': client_data.get('ip_address', 'unknown'),
'user_id': client_data.get('user_id', '0'),
'output_format': client_data.get('output_format', 'mpegts'),
}
raw_profile_id = client_data.get('output_profile_id')
if raw_profile_id and raw_profile_id not in ('None', '0', ''):
client_info['output_profile_id'] = int(raw_profile_id)
else:
client_info['output_profile_id'] = None
if 'connected_at' in client_data:
client_info['connected_at'] = float(client_data['connected_at'])
@ -244,7 +249,7 @@ class ChannelStatus:
all_buffer_keys = []
cursor = 0
buffer_key_pattern = f"ts_proxy:channel:{channel_id}:buffer:chunk:*"
buffer_key_pattern = f"live:channel:{channel_id}:input:buffer:chunk:*"
while True:
cursor, keys = proxy_server.redis_client.scan(cursor, match=buffer_key_pattern, count=100)
@ -473,6 +478,15 @@ class ChannelStatus:
if user_id_bytes:
client_info['user_id'] = user_id_bytes
output_format = proxy_server.redis_client.hget(client_key, 'output_format')
client_info['output_format'] = output_format or 'mpegts'
raw_profile_id = proxy_server.redis_client.hget(client_key, 'output_profile_id')
if raw_profile_id and raw_profile_id not in ('None', '0', ''):
client_info['output_profile_id'] = int(raw_profile_id)
else:
client_info['output_profile_id'] = None
clients.append(client_info)
# Add clients to info
@ -518,5 +532,5 @@ class ChannelStatus:
return info
except Exception as e:
logger.error(f"Error getting channel info: {e}", exc_info=True) # Added exc_info for better debugging
logger.error(f"Error getting channel info: {e}", exc_info=True)
return None

49
apps/proxy/ts_proxy/client_manager.py → apps/proxy/live_proxy/client_manager.py
View file

@ -3,7 +3,6 @@
import threading
import time
import json
from typing import Set, Optional
from apps.proxy.config import TSConfig as Config
from redis.exceptions import ConnectionError, TimeoutError
from .constants import EventType, ChannelState, ChannelMetadataField
@ -26,7 +25,6 @@ class ClientManager:
self.worker_id = worker_id # Store worker ID as instance variable
self._heartbeat_running = True # Flag to control heartbeat thread
# STANDARDIZED KEYS: Move client set under channel namespace
self.client_set_key = RedisKeys.clients(channel_id)
self.client_ttl = ConfigHelper.get('CLIENT_RECORD_TTL', 60)
self.heartbeat_interval = ConfigHelper.get('CLIENT_HEARTBEAT_INTERVAL', 10)
@ -51,18 +49,17 @@ class ClientManager:
def _do_stats_update(self):
"""Perform the stats update in the background."""
try:
from apps.proxy.ts_proxy.channel_status import ChannelStatus
import redis
from django.conf import settings
from apps.proxy.live_proxy.channel_status import ChannelStatus
from core.utils import RedisClient
redis_url = getattr(settings, 'REDIS_URL', 'redis://localhost:6379/0')
ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {})
redis_client = redis.Redis.from_url(redis_url, decode_responses=True, **ssl_params)
redis_client = RedisClient.get_client()
if not redis_client:
return
all_channels = []
cursor = 0
while True:
cursor, keys = redis_client.scan(cursor, match="ts_proxy:channel:*:clients", count=100)
cursor, keys = redis_client.scan(cursor, match="live:channel:*:metadata", count=100)
for key in keys:
parts = key.split(':')
if len(parts) >= 4:
@ -116,7 +113,7 @@ class ClientManager:
# First identify clients that should be removed
for client_id in self.clients:
client_key = f"ts_proxy:channel:{self.channel_id}:clients:{client_id}"
client_key = f"live:channel:{self.channel_id}:clients:{client_id}"
# Check if client exists in Redis at all
exists = self.redis_client.exists(client_key)
@ -158,7 +155,7 @@ class ClientManager:
continue
# Only refresh TTL - do NOT update last_active
client_key = f"ts_proxy:channel:{self.channel_id}:clients:{client_id}"
client_key = f"live:channel:{self.channel_id}:clients:{client_id}"
pipe.expire(client_key, self.client_ttl)
# Keep client in the set with TTL
@ -214,21 +211,19 @@ class ClientManager:
try:
worker_id = self.worker_id or "unknown"
# STANDARDIZED KEY: Worker info under channel namespace
worker_key = f"ts_proxy:channel:{self.channel_id}:worker:{worker_id}"
worker_key = f"live:channel:{self.channel_id}:worker:{worker_id}"
self._execute_redis_command(
lambda: self.redis_client.setex(worker_key, self.client_ttl, str(len(self.clients)))
)
# STANDARDIZED KEY: Activity timestamp under channel namespace
activity_key = f"ts_proxy:channel:{self.channel_id}:activity"
activity_key = f"live:channel:{self.channel_id}:activity"
self._execute_redis_command(
lambda: self.redis_client.setex(activity_key, self.client_ttl, str(time.time()))
)
except Exception as e:
logger.error(f"Error notifying owner of client activity: {e}")
def add_client(self, client_id, client_ip, user_agent=None, user=None):
def add_client(self, client_id, client_ip, user_agent=None, user=None, output_format='mpegts', output_profile_id=None):
"""Add a client with duplicate prevention"""
if client_id in self._registered_clients:
logger.debug(f"Client {client_id} already registered, skipping")
@ -237,7 +232,7 @@ class ClientManager:
self._registered_clients.add(client_id)
# Use a function to get the client key
client_key = f"ts_proxy:channel:{self.channel_id}:clients:{client_id}"
client_key = f"live:channel:{self.channel_id}:clients:{client_id}"
# Prepare client data
current_time = str(time.time())
@ -248,7 +243,8 @@ class ClientManager:
"last_active": current_time,
"worker_id": self.worker_id or "unknown",
"user_id": str(user.id) if user is not None else "0",
# "user_level": user.user_level if user is not None else 100, # default to a high value since no user means the non-user specific M3U/HDHR
"output_format": output_format,
"output_profile_id": str(output_profile_id) if output_profile_id is not None else "",
}
try:
@ -258,7 +254,6 @@ class ClientManager:
# Store in Redis
if self.redis_client:
# FIXED: Store client data just once with proper key
self.redis_client.hset(client_key, mapping=client_data)
self.redis_client.expire(client_key, self.client_ttl)
@ -267,7 +262,7 @@ class ClientManager:
self.redis_client.expire(self.client_set_key, self.client_ttl)
# Clear any initialization timer
init_key = f"ts_proxy:channel:{self.channel_id}:init_time"
init_key = f"live:channel:{self.channel_id}:init_time"
self.redis_client.delete(init_key)
self._notify_owner_of_activity()
@ -321,7 +316,7 @@ class ClientManager:
if self.redis_client:
# Get client data before removing the data
client_key = f"ts_proxy:channel:{self.channel_id}:clients:{client_id}"
client_key = f"live:channel:{self.channel_id}:clients:{client_id}"
client_username = self.redis_client.hget(client_key, "username") or "unknown"
if isinstance(client_username, bytes):
client_username = client_username.decode("utf-8")
@ -329,8 +324,7 @@ class ClientManager:
# Remove from channel's client set
self.redis_client.srem(self.client_set_key, client_id)
# STANDARDIZED KEY: Delete individual client keys
client_key = f"ts_proxy:channel:{self.channel_id}:clients:{client_id}"
client_key = f"live:channel:{self.channel_id}:clients:{client_id}"
self.redis_client.delete(client_key)
# Check if this was the last client
@ -350,10 +344,9 @@ class ClientManager:
if am_i_owner:
# We're the owner - handle the disconnect directly
logger.debug(f"Owner handling CLIENT_DISCONNECTED for client {client_id} locally (not publishing)")
if remaining == 0:
# Trigger shutdown check directly via ProxyServer method
logger.debug(f"No clients left - triggering immediate shutdown check")
# Spawn greenlet to avoid blocking
if (remaining == 0
or self.proxy_server.output_managers.get(self.channel_id)
or self.proxy_server.profile_managers.get(self.channel_id)):
import gevent
gevent.spawn(self.proxy_server.handle_client_disconnect, self.channel_id)
else:
@ -404,7 +397,7 @@ class ClientManager:
# Refresh TTL for all clients belonging to this worker
for client_id in self.clients:
# STANDARDIZED: Use channel namespace for client keys
client_key = f"ts_proxy:channel:{self.channel_id}:clients:{client_id}"
client_key = f"live:channel:{self.channel_id}:clients:{client_id}"
self.redis_client.expire(client_key, self.client_ttl)
# Refresh TTL on the set itself

0
apps/proxy/ts_proxy/config_helper.py → apps/proxy/live_proxy/config_helper.py
View file

5
apps/proxy/ts_proxy/constants.py → apps/proxy/live_proxy/constants.py
View file

@ -4,7 +4,7 @@ Centralizing constants makes it easier to maintain and modify them.
"""
# Redis related constants
REDIS_KEY_PREFIX = "ts_proxy"
REDIS_KEY_PREFIX = "live"
REDIS_TTL_DEFAULT = 3600 # 1 hour
REDIS_TTL_SHORT = 60 # 1 minute
REDIS_TTL_MEDIUM = 300 # 5 minutes
@ -33,6 +33,8 @@ class EventType:
CLIENT_CONNECTED = "client_connected"
CLIENT_DISCONNECTED = "client_disconnected"
CLIENT_STOP = "client_stop"
ENSURE_OUTPUT_FORMAT = "ensure_output_format"
ENSURE_OUTPUT_PROFILE = "ensure_output_profile"
# Stream types
class StreamType:
@ -104,6 +106,7 @@ class ChannelMetadataField:
# Client metadata fields
CONNECTED_AT = "connected_at"
LAST_ACTIVE = "last_active"
OUTPUT_FORMAT = "output_format"
BYTES_SENT = "bytes_sent"
AVG_RATE_KBPS = "avg_rate_KBps"
CURRENT_RATE_KBPS = "current_rate_KBps"

1
apps/proxy/live_proxy/input/__init__.py Normal file
View file

@ -0,0 +1 @@

38
apps/proxy/ts_proxy/stream_buffer.py → apps/proxy/live_proxy/input/buffer.py
View file

@ -1,34 +1,30 @@
"""Buffer management for TS streams"""
import threading
import logging
import time
from collections import deque
from typing import Optional, Deque
import random
from apps.proxy.config import TSConfig as Config
from .redis_keys import RedisKeys
from .config_helper import ConfigHelper
from .constants import TS_PACKET_SIZE
from .utils import get_logger
from ..redis_keys import RedisKeys
from ..config_helper import ConfigHelper
from ..constants import TS_PACKET_SIZE
from ..utils import get_logger
import gevent.event
import gevent # Make sure this import is at the top
import gevent
logger = get_logger()
class StreamBuffer:
"""Manages stream data buffering with optimized chunk storage"""
def __init__(self, channel_id=None, redis_client=None):
def __init__(self, channel_id=None, redis_client=None,
buffer_index_key=None, buffer_chunk_prefix=None, chunk_timestamps_key=None):
self.channel_id = channel_id
self.redis_client = redis_client
self.lock = threading.Lock()
self.index = 0
self.TS_PACKET_SIZE = TS_PACKET_SIZE
# STANDARDIZED KEYS: Use RedisKeys class instead of hardcoded patterns
self.buffer_index_key = RedisKeys.buffer_index(channel_id) if channel_id else ""
self.buffer_prefix = RedisKeys.buffer_chunk_prefix(channel_id) if channel_id else ""
self.buffer_index_key = buffer_index_key or (RedisKeys.buffer_index(channel_id) if channel_id else "")
self.buffer_prefix = buffer_chunk_prefix or (RedisKeys.buffer_chunk_prefix(channel_id) if channel_id else "")
self.chunk_ttl = ConfigHelper.redis_chunk_ttl()
@ -46,7 +42,7 @@ class StreamBuffer:
self.target_chunk_size = ConfigHelper.get('BUFFER_CHUNK_SIZE', TS_PACKET_SIZE * 5644) # ~1MB default
# Sorted-set key for chunk receive-timestamps (time-based positioning)
self.chunk_timestamps_key = RedisKeys.chunk_timestamps(channel_id) if channel_id else ""
self.chunk_timestamps_key = chunk_timestamps_key or (RedisKeys.chunk_timestamps(channel_id) if channel_id else "")
# Register Lua scripts once — subsequent calls use EVALSHA (just the
# SHA hash) instead of sending the full script text on every invocation.
@ -108,7 +104,7 @@ class StreamBuffer:
# remaining writes are pipelined into one round trip.
if self.redis_client:
chunk_index = self.redis_client.incr(self.buffer_index_key)
chunk_key = RedisKeys.buffer_chunk(self.channel_id, chunk_index)
chunk_key = f"{self.buffer_prefix}{chunk_index}"
pipe = self.redis_client.pipeline(transaction=False)
pipe.setex(chunk_key, self.chunk_ttl, bytes(chunk_data))
@ -219,7 +215,7 @@ class StreamBuffer:
# Directly fetch from Redis using pipeline for efficiency
pipe = self.redis_client.pipeline()
for idx in range(start_id, end_id):
chunk_key = RedisKeys.buffer_chunk(self.channel_id, idx)
chunk_key = f"{self.buffer_prefix}{idx}"
pipe.get(chunk_key)
results = pipe.execute()
@ -273,7 +269,7 @@ class StreamBuffer:
# Directly fetch from Redis using pipeline
pipe = self.redis_client.pipeline()
for idx in range(start_id, end_id):
chunk_key = RedisKeys.buffer_chunk(self.channel_id, idx)
chunk_key = f"{self.buffer_prefix}{idx}"
pipe.get(chunk_key)
results = pipe.execute()
@ -394,7 +390,7 @@ class StreamBuffer:
additional_size = sum(len(c) for c in more_chunks)
if total_size + additional_size <= MAX_SIZE:
chunks.extend(more_chunks)
chunk_count += len(more_chunks) # Fixed: count actual additional chunks retrieved
chunk_count += len(more_chunks)
return chunks, client_index + chunk_count
@ -404,7 +400,7 @@ class StreamBuffer:
# Binary search finds the boundary in O(log N) EXISTS calls with zero
# round-trips between steps and no TOCTOU races (Lua scripts are atomic).
#
# ARGV[1] = key prefix (e.g. "ts_proxy:channel:<id>:buffer:chunk:")
# ARGV[1] = key prefix (e.g. "live:channel:<id>:input:buffer:chunk:")
# ARGV[2] = low index (client_index + 1, first chunk the client needs)
# ARGV[3] = high index (buffer head, most recent chunk)
#
@ -464,7 +460,7 @@ class StreamBuffer:
# not the full script text, on every call after the first.
result = self._find_oldest_chunk_sha(
args=[
RedisKeys.buffer_chunk_prefix(self.channel_id),
self.buffer_prefix,
low,
high,
],
@ -538,13 +534,11 @@ class StreamBuffer:
logger.error(f"Error in find_chunk_index_by_time for channel {self.channel_id}: {e}")
return None
# Add a new method to safely create timers
def schedule_timer(self, delay, callback, *args, **kwargs):
"""Schedule a timer and track it for proper cleanup"""
if self.stopping:
return None
# Replace threading.Timer with gevent.spawn_later for better compatibility
timer = gevent.spawn_later(delay, callback, *args, **kwargs)
self.fill_timers.append(timer)
return timer

2
apps/proxy/ts_proxy/http_streamer.py → apps/proxy/live_proxy/input/http_streamer.py
View file

@ -7,7 +7,7 @@ import threading
import os
import requests
from requests.adapters import HTTPAdapter
from .utils import get_logger
from ..utils import get_logger
logger = get_logger()

37
apps/proxy/ts_proxy/stream_manager.py → apps/proxy/live_proxy/input/manager.py
View file

@ -1,28 +1,22 @@
"""Stream connection management for TS proxy"""
import threading
import logging
import time
import socket
import requests
import subprocess
import gevent
import re
from typing import Optional, List
from django.db import connection
from django.shortcuts import get_object_or_404
from urllib3.exceptions import ReadTimeoutError
from apps.proxy.config import TSConfig as Config
from apps.channels.models import Channel, Stream
from apps.m3u.models import M3UAccount, M3UAccountProfile
from core.models import UserAgent, CoreSettings
from core.utils import log_system_event
from .stream_buffer import StreamBuffer
from .utils import detect_stream_type, get_logger
from .redis_keys import RedisKeys
from .constants import ChannelState, EventType, StreamType, ChannelMetadataField, TS_PACKET_SIZE
from .config_helper import ConfigHelper
from .url_utils import get_alternate_streams, get_stream_info_for_switch, get_stream_object
from .buffer import StreamBuffer
from ..utils import detect_stream_type, get_logger
from ..redis_keys import RedisKeys
from ..constants import ChannelState, EventType, StreamType, ChannelMetadataField, TS_PACKET_SIZE
from ..config_helper import ConfigHelper
from ..url_utils import get_alternate_streams, get_stream_info_for_switch, get_stream_object
logger = get_logger()
@ -78,7 +72,6 @@ class StreamManager:
self.current_stream_id = stream_id
self.tried_stream_ids = set()
# IMPROVED LOGGING: Better handle and track stream ID
if stream_id:
self.tried_stream_ids.add(stream_id)
logger.info(f"Initialized stream manager for channel {buffer.channel_id} with stream ID {stream_id}")
@ -110,7 +103,6 @@ class StreamManager:
logger.info(f"Initialized stream manager for channel {buffer.channel_id}")
# Add this flag for tracking transcoding process status
self.transcode_process_active = False
# Track stream command for efficient log parser routing
@ -258,7 +250,7 @@ class StreamManager:
url_failed = False
if self.url_switching:
logger.debug(f"Skipping connection attempt during URL switch for channel {self.channel_id}")
gevent.sleep(0.1) # REPLACE time.sleep(0.1)
gevent.sleep(0.1)
continue
# Connection retry loop for current URL
while self.running and self.retry_count < self.max_retries and not url_failed and not self.needs_stream_switch:
@ -336,7 +328,7 @@ class StreamManager:
# Wait with exponential backoff before retrying
timeout = min(.25 * self.retry_count, 3) # Cap at 3 seconds
logger.info(f"Reconnecting in {timeout} seconds... (attempt {self.retry_count}/{self.max_retries}) for channel: {self.channel_id}")
gevent.sleep(timeout) # REPLACE time.sleep(timeout)
gevent.sleep(timeout)
except Exception as e:
logger.error(f"Connection error on channel: {self.channel_id}: {e}", exc_info=True)
@ -363,7 +355,7 @@ class StreamManager:
# Wait with exponential backoff before retrying
timeout = min(.25 * self.retry_count, 3) # Cap at 3 seconds
logger.info(f"Reconnecting in {timeout} seconds after error... (attempt {self.retry_count}/{self.max_retries}) for channel: {self.channel_id}")
gevent.sleep(timeout) # REPLACE time.sleep(timeout)
gevent.sleep(timeout)
# If URL failed and we're still running, try switching to another stream
if url_failed and self.running:
@ -698,8 +690,8 @@ class StreamManager:
self.ffmpeg_input_phase = False
# Route to appropriate parser based on known command type
from .services.log_parsers import LogParserFactory
from .services.channel_service import ChannelService
from ..services.log_parsers import LogParserFactory
from ..services.channel_service import ChannelService
parse_result = None
@ -795,7 +787,7 @@ class StreamManager:
now = time.time()
if now - self._last_bitrate_db_save_time >= self._bitrate_db_save_interval:
from .services.channel_service import ChannelService
from ..services.channel_service import ChannelService
ChannelService._update_stream_stats_in_db(
self.current_stream_id,
ffmpeg_output_bitrate=round(self._smoothed_output_bitrate, 1)
@ -1082,7 +1074,7 @@ class StreamManager:
if self._smoothed_output_bitrate is not None and self.current_stream_id:
final_bitrate = self._smoothed_output_bitrate
try:
from .services.channel_service import ChannelService
from ..services.channel_service import ChannelService
ChannelService._update_stream_stats_in_db(
self.current_stream_id,
ffmpeg_output_bitrate=round(final_bitrate, 1)
@ -1257,7 +1249,7 @@ class StreamManager:
except Exception as e:
logger.error(f"Error in health monitor: {e}")
gevent.sleep(self.health_check_interval) # REPLACE time.sleep(self.health_check_interval)
gevent.sleep(self.health_check_interval)
def _attempt_reconnect(self):
"""Attempt to reconnect to the current stream"""
@ -1671,7 +1663,6 @@ class StreamManager:
logger.warning(f"All {len(alternate_streams)} alternate streams have been tried for channel {self.channel_id}")
return False
# IMPROVED: Try multiple streams until we find one with a different URL
for next_stream in untried_streams:
stream_id = next_stream['stream_id']
profile_id = next_stream['profile_id'] # This is the M3U profile ID we need

1
apps/proxy/live_proxy/output/__init__.py Normal file
View file

@ -0,0 +1 @@

1
apps/proxy/live_proxy/output/fmp4/__init__.py Normal file
View file

@ -0,0 +1 @@

165
apps/proxy/live_proxy/output/fmp4/buffer.py Normal file
View file

@ -0,0 +1,165 @@
"""fMP4 buffer management - mirrors StreamBuffer but without TS packet alignment."""
import threading
import time
import gevent.event
from ...redis_keys import RedisKeys
from ...config_helper import ConfigHelper
from ...utils import get_logger
logger = get_logger()
class FMP4StreamBuffer:
"""
Redis-backed buffer for fMP4 remux output.
Functionally identical to StreamBuffer except:
- Uses the fmp4:buffer:* Redis keyspace
- No 188-byte TS packet alignment (raw byte accumulation)
"""
def __init__(self, channel_id, redis_client=None, fmt='fmp4'):
self.channel_id = channel_id
self.redis_client = redis_client
self.fmt = fmt
self.lock = threading.Lock()
self.index = 0
self.buffer_index_key = RedisKeys.output_buffer_index(channel_id, fmt)
self.buffer_prefix = RedisKeys.output_buffer_chunk_prefix(channel_id, fmt)
self.chunk_timestamps_key = RedisKeys.output_chunk_timestamps(channel_id, fmt)
self.chunk_ttl = ConfigHelper.redis_chunk_ttl()
self.stopping = False
if self.redis_client and channel_id:
try:
current_index = self.redis_client.get(self.buffer_index_key)
if current_index:
self.index = int(current_index)
except Exception as e:
logger.error(f"[fMP4Buffer:{channel_id}] Error initialising from Redis: {e}")
self.chunk_available = gevent.event.Event()
# Lua script for time-based start positioning (same as StreamBuffer)
_LUA = """
local ts_key = KEYS[1]
local target = tonumber(ARGV[1])
local result = redis.call('ZREVRANGEBYSCORE', ts_key, target, '-inf', 'LIMIT', 0, 1)
if #result == 0 then return -1 end
return tonumber(result[1])
"""
if self.redis_client:
try:
self._find_chunk_by_time_sha = self.redis_client.register_script(_LUA)
except Exception:
self._find_chunk_by_time_sha = None
else:
self._find_chunk_by_time_sha = None
def put_fragment(self, data: bytes) -> bool:
"""Store a single complete fMP4 fragment directly to Redis as its own chunk."""
if not data or not self.redis_client:
return False
try:
now = time.time()
with self.lock:
chunk_index = self.redis_client.incr(self.buffer_index_key)
chunk_key = RedisKeys.output_buffer_chunk(self.channel_id, self.fmt, chunk_index)
pipe = self.redis_client.pipeline(transaction=False)
pipe.setex(chunk_key, self.chunk_ttl, data)
pipe.zadd(self.chunk_timestamps_key, {str(chunk_index): now})
pipe.zremrangebyscore(self.chunk_timestamps_key, '-inf', now - self.chunk_ttl)
pipe.expire(self.chunk_timestamps_key, self.chunk_ttl)
pipe.execute()
self.index = chunk_index
self.chunk_available.set()
self.chunk_available.clear()
return True
except Exception as e:
logger.error(f"[fMP4Buffer:{self.channel_id}] Error putting fragment: {e}")
return False
def get_chunks(self, start_index=None):
"""Retrieve chunks from start_index up to current head. Returns (chunks, new_index)."""
try:
if not self.redis_client:
return [], self.index
current_index = self.redis_client.get(self.buffer_index_key)
if not current_index:
return [], self.index
current_index = int(current_index)
if start_index is None:
start_index = max(0, current_index - 5)
if start_index >= current_index:
return [], current_index
chunks = []
pipe = self.redis_client.pipeline(transaction=False)
indices = range(start_index + 1, current_index + 1)
for i in indices:
pipe.get(RedisKeys.output_buffer_chunk(self.channel_id, self.fmt, i))
results = pipe.execute()
for data in results:
if data:
chunks.append(data)
return chunks, current_index
except Exception as e:
logger.error(f"[fMP4Buffer:{self.channel_id}] Error getting chunks: {e}")
return [], self.index
def find_chunk_index_by_time(self, seconds_behind):
"""Return the fragment index that was received ~seconds_behind seconds ago.
Returns an int (last-consumed convention: next read starts at index+1)
or None if no suitable fragment exists.
"""
if not self.redis_client or not self._find_chunk_by_time_sha:
return None
target_time = time.time() - seconds_behind
try:
result = self._find_chunk_by_time_sha(
keys=[self.chunk_timestamps_key],
args=[target_time],
)
if result is None or int(result) == -1:
oldest = self.redis_client.zrange(self.chunk_timestamps_key, 0, 0)
if oldest:
return max(0, int(oldest[0]) - 1)
return None
return max(0, int(result) - 1)
except Exception as e:
logger.error(f"[fMP4Buffer:{self.channel_id}] Error in find_chunk_index_by_time: {e}")
return None
def stop(self):
self.stopping = True
def cleanup_redis(self):
"""Delete all fMP4 buffer keys for this channel from Redis."""
if not self.redis_client:
return
try:
prefix = self.buffer_prefix
cursor = 0
while True:
cursor, keys = self.redis_client.scan(cursor, match=f"{prefix}*", count=200)
if keys:
self.redis_client.delete(*keys)
if cursor == 0:
break
self.redis_client.delete(self.buffer_index_key)
try:
self.redis_client.delete(self.chunk_timestamps_key)
except Exception:
pass
except Exception as e:
logger.error(f"[fMP4Buffer:{self.channel_id}] Error during Redis cleanup: {e}")

376
apps/proxy/live_proxy/output/fmp4/generator.py Normal file
View file

@ -0,0 +1,376 @@
"""
fMP4 Stream Generator
Yields the fMP4 init segment followed by fMP4 buffer chunks to a single client.
Mirrors StreamGenerator's structure: same resource checks, same cleanup path,
same client registration in the main ClientManager so that the existing
zero-clients → stop_channel shutdown chain works for all client types.
"""
import time
import gevent
from apps.channels.models import Channel, Stream
from core.utils import log_system_event
from ...server import ProxyServer
from ...redis_keys import RedisKeys
from ...constants import ChannelMetadataField
from .buffer import FMP4StreamBuffer
from .manager import FMP4_STATE_ACTIVE, INIT_SEGMENT_TIMEOUT
from ...config_helper import ConfigHelper
from ...utils import get_logger
logger = get_logger()
def create_fmp4_stream_generator(channel_id, client_id, client_ip, client_user_agent,
channel_initializing=False, user=None, fmt='fmp4'):
gen = FMP4StreamGenerator(channel_id, client_id, client_ip, client_user_agent,
channel_initializing, user, fmt=fmt)
return gen.generate
class FMP4StreamGenerator:
def __init__(self, channel_id, client_id, client_ip, client_user_agent,
channel_initializing=False, user=None, fmt='fmp4'):
self.channel_id = channel_id
self.client_id = client_id
self.client_ip = client_ip
self.client_user_agent = client_user_agent
self.channel_initializing = channel_initializing
self.user = user
self.fmt = fmt
try:
_name = Channel.objects.filter(uuid=channel_id).values_list('name', flat=True).first()
self.channel_name = _name if _name else str(channel_id)
except Exception:
self.channel_name = str(channel_id)
self.stream_start_time = time.time()
self.bytes_sent = 0
self.chunks_sent = 0
self.local_index = 0
self.last_yield_time = time.time()
self.consecutive_empty = 0
self._last_resource_check_time = 0.0
self._resource_check_interval = 1.0
self.proxy_server = None
self.fmp4_buffer = None
# ------------------------------------------------------------------
# Main generator
# ------------------------------------------------------------------
def generate(self):
self.stream_start_time = time.time()
try:
logger.info(f"[{self.client_id}] fMP4 stream generator started")
# Wait for the main TS channel to be ready (reuses existing init wait logic)
if self.channel_initializing:
if not self._wait_for_channel_ready():
return
# Wait for fMP4 remux to be ready and init segment available
if not self._wait_for_fmp4_ready():
return
# Set up local references
if not self._setup_streaming():
return
try:
log_system_event(
'client_connect',
channel_id=self.channel_id,
channel_name=self.channel_name,
client_ip=self.client_ip,
client_id=self.client_id,
user_agent=self.client_user_agent[:100] if self.client_user_agent else None,
username=self.user.username if self.user else None,
)
except Exception:
pass
# Yield init segment first - every new client needs it
init_segment = self._fetch_init_segment()
if not init_segment:
logger.error(f"[{self.client_id}] fMP4 init segment disappeared, aborting")
return
yield init_segment
self.bytes_sent += len(init_segment)
# Main data loop
for chunk in self._stream_data_generator():
yield chunk
except Exception as e:
logger.error(f"[{self.client_id}] fMP4 stream error: {e}", exc_info=True)
finally:
self._cleanup()
# ------------------------------------------------------------------
# Wait helpers
# ------------------------------------------------------------------
def _wait_for_channel_ready(self) -> bool:
"""Wait for the main TS channel to reach active/waiting_for_clients state."""
proxy_server = ProxyServer.get_instance()
deadline = time.time() + ConfigHelper.client_wait_timeout()
while time.time() < deadline:
if proxy_server.redis_client:
meta = proxy_server.redis_client.hgetall(
RedisKeys.channel_metadata(self.channel_id)
)
state = meta.get('state', '')
if state in ('waiting_for_clients', 'active'):
return True
if state in ('error', 'stopped', 'stopping'):
logger.error(f"[{self.client_id}] Channel in {state} state during fMP4 init wait")
return False
# Check stop key
if proxy_server.redis_client.exists(RedisKeys.channel_stopping(self.channel_id)):
return False
gevent.sleep(0.1)
logger.warning(f"[{self.client_id}] Timed out waiting for TS channel to become ready")
return False
def _wait_for_fmp4_ready(self) -> bool:
"""Wait for FMP4RemuxManager to store the init segment in Redis."""
proxy_server = ProxyServer.get_instance()
if not proxy_server.redis_client:
return False
init_key = RedisKeys.output_init(self.channel_id, self.fmt)
deadline = time.time() + INIT_SEGMENT_TIMEOUT
while time.time() < deadline:
if proxy_server.redis_client.exists(init_key):
logger.info(f"[{self.client_id}] fMP4 init segment ready")
return True
# Bail out if channel is stopping
if proxy_server.redis_client.exists(RedisKeys.channel_stopping(self.channel_id)):
logger.info(f"[{self.client_id}] Channel stopping while waiting for fMP4 init")
return False
gevent.sleep(0.1)
logger.error(
f"[{self.client_id}] Timed out waiting for fMP4 init segment "
f"({INIT_SEGMENT_TIMEOUT}s)"
)
return False
# ------------------------------------------------------------------
# Setup and streaming
# ------------------------------------------------------------------
def _setup_streaming(self) -> bool:
proxy_server = ProxyServer.get_instance()
self.proxy_server = proxy_server
# Build a local FMP4StreamBuffer reader (shares Redis keyspace, no local state)
from core.utils import RedisClient
self.fmp4_buffer = FMP4StreamBuffer(
self.channel_id, redis_client=RedisClient.get_buffer(), fmt=self.fmt
)
# Determine start position
if self.channel_initializing:
# First client on a new channel - start from the very beginning
self.local_index = 0
logger.info(f"[{self.client_id}] fMP4 channel initializer: starting at index 0")
else:
# Joining an existing stream - use time-based positioning (same as TS clients)
behind_seconds = ConfigHelper.new_client_behind_seconds()
# Read live head fresh from Redis (self.fmp4_buffer.index is stale from __init__)
try:
raw = RedisClient.get_buffer().get(self.fmp4_buffer.buffer_index_key)
current = int(raw) if raw else 0
except Exception:
current = self.fmp4_buffer.index
if behind_seconds > 0:
time_index = self.fmp4_buffer.find_chunk_index_by_time(behind_seconds)
if time_index is not None:
self.local_index = max(0, time_index)
logger.info(
f"[{self.client_id}] fMP4 time-based start: "
f"{behind_seconds}s behind -> index {self.local_index} "
f"(buffer head at {current})"
)
else:
# Not enough buffer - start at oldest available fragment
self.local_index = 0
logger.info(
f"[{self.client_id}] fMP4 buffer shorter than {behind_seconds}s, "
f"starting at oldest available fragment "
f"(buffer head at {current})"
)
else:
self.local_index = max(0, current - 1)
logger.info(
f"[{self.client_id}] fMP4 live start (behind_seconds=0), "
f"index {self.local_index} (buffer head at {current})"
)
logger.info(
f"[{self.client_id}] fMP4 streaming setup complete, "
f"starting at index {self.local_index}"
)
return True
def _fetch_init_segment(self) -> bytes:
from core.utils import RedisClient
redis_buf = RedisClient.get_buffer()
if not redis_buf:
return b''
data = redis_buf.get(RedisKeys.output_init(self.channel_id, self.fmt))
return data if data else b''
def _stream_data_generator(self):
proxy_server = self.proxy_server or ProxyServer.get_instance()
stats_write_interval = 1.0
last_stats_write = 0.0
while True:
if not self._check_resources():
break
chunks, new_index = self.fmp4_buffer.get_chunks(start_index=self.local_index)
if chunks:
self.consecutive_empty = 0
for chunk in chunks:
yield chunk
self.bytes_sent += len(chunk)
self.chunks_sent += 1
self.local_index = new_index
self.last_yield_time = time.time()
# Update last_active in Redis so ghost detection doesn't fire
now = time.time()
if proxy_server.redis_client and now - last_stats_write >= stats_write_interval:
last_stats_write = now
try:
client_key = RedisKeys.client_metadata(self.channel_id, self.client_id)
proxy_server.redis_client.hset(client_key, "last_active", str(now))
except Exception:
pass
else:
self.consecutive_empty += 1
gevent.sleep(0.05)
if self._is_timeout():
break
# ------------------------------------------------------------------
# Resource / timeout checks (mirrors StreamGenerator pattern)
# ------------------------------------------------------------------
def _check_resources(self) -> bool:
proxy_server = self.proxy_server or ProxyServer.get_instance()
if self.channel_id not in proxy_server.stream_buffers:
logger.info(f"[{self.client_id}] TS buffer gone, terminating fMP4 stream")
return False
if self.channel_id not in proxy_server.client_managers:
logger.info(f"[{self.client_id}] Client manager gone, terminating fMP4 stream")
return False
client_manager = proxy_server.client_managers[self.channel_id]
if self.client_id not in client_manager.clients:
logger.info(f"[{self.client_id}] Client no longer in client manager")
return False
if not proxy_server.redis_client:
return True
now = time.time()
if now - self._last_resource_check_time < self._resource_check_interval:
return True
self._last_resource_check_time = now
if proxy_server.redis_client.exists(RedisKeys.channel_stopping(self.channel_id)):
logger.info(f"[{self.client_id}] Channel stop signal, terminating fMP4 stream")
return False
meta = proxy_server.redis_client.hgetall(RedisKeys.channel_metadata(self.channel_id))
if meta and meta.get('state') in ('error', 'stopped', 'stopping'):
logger.info(
f"[{self.client_id}] Channel in {meta.get('state')} state, "
f"terminating fMP4 stream"
)
return False
if proxy_server.redis_client.exists(RedisKeys.client_stop(self.channel_id, self.client_id)):
logger.info(f"[{self.client_id}] Client stop signal")
return False
return True
def _is_timeout(self) -> bool:
timeout = ConfigHelper.stream_timeout() + ConfigHelper.failover_grace_period()
if time.time() - self.last_yield_time > timeout:
logger.warning(
f"[{self.client_id}] fMP4 no data for {timeout}s, disconnecting"
)
return True
return False
# ------------------------------------------------------------------
# Cleanup - mirrors StreamGenerator._cleanup() exactly so shutdown
# chain fires identically for fMP4 clients
# ------------------------------------------------------------------
def _cleanup(self):
elapsed = time.time() - self.stream_start_time
proxy_server = ProxyServer.get_instance()
# Release stream allocation if last client (mirrors StreamGenerator)
if proxy_server.redis_client:
try:
meta_key = RedisKeys.channel_metadata(self.channel_id)
stream_id_bytes = proxy_server.redis_client.hget(
meta_key, ChannelMetadataField.STREAM_ID
)
if stream_id_bytes:
if self.channel_id in proxy_server.client_managers:
client_count = proxy_server.client_managers[
self.channel_id
].get_total_client_count()
if client_count <= 1 and proxy_server.am_i_owner(self.channel_id):
try:
try:
obj = Channel.objects.get(uuid=self.channel_id)
except (Channel.DoesNotExist, Exception):
obj = Stream.objects.get(stream_hash=self.channel_id)
obj.release_stream()
except Exception as e:
logger.error(
f"[{self.client_id}] Error releasing stream: {e}"
)
except Exception as e:
logger.error(f"[{self.client_id}] Error in stream release check: {e}")
# Remove from MAIN ClientManager - this is what triggers handle_client_disconnect
# and the zero-clients → stop_channel path, same as TS clients.
local_clients = 0
total_clients = 0
if self.channel_id in proxy_server.client_managers:
client_manager = proxy_server.client_managers[self.channel_id]
local_clients = client_manager.remove_client(self.client_id)
total_clients = client_manager.get_total_client_count()
logger.info(
f"[{self.client_id}] fMP4 client disconnected after {elapsed:.2f}s "
f"({self.bytes_sent / 1024:.1f} KB sent, "
f"local: {local_clients}, total: {total_clients})"
)

455
apps/proxy/live_proxy/output/fmp4/manager.py Normal file
View file

@ -0,0 +1,455 @@
"""
fMP4 Remux Manager
Reads from the shared TS Redis buffer, pipes data through FFmpeg for container
remux to fragmented MP4, parses the init segment out of the first output bytes,
stores it in Redis, and writes subsequent fMP4 fragment chunks to FMP4StreamBuffer.
One instance per channel per cluster - coordinated via Redis fmp4:owner lock.
"""
import select
import subprocess
import threading
import time
import struct
from core.utils import RedisClient
from .buffer import FMP4StreamBuffer
from ...redis_keys import RedisKeys
from ...config_helper import ConfigHelper
from ...utils import get_logger
logger = get_logger()
# fMP4 remux states stored in Redis
FMP4_STATE_INITIALIZING = "initializing"
FMP4_STATE_ACTIVE = "active"
FMP4_STATE_STOPPED = "stopped"
# FFmpeg command for container-only remux: TS in, fMP4 out, no transcode
# The aac_adtstoasc BSF is required when source audio is AAC (converts ADTS framing
# to the raw format MP4 requires). For other codecs (AC3, EAC3, MP3, etc.) it errors;
# _handle_bsf_error detects that and retries with FFMPEG_REMUX_CMD_NO_BSF.
FFMPEG_REMUX_CMD = [
"ffmpeg",
"-loglevel", "error",
"-f", "mpegts",
"-i", "pipe:0",
"-c", "copy",
"-map", "0",
"-bsf:a", "aac_adtstoasc",
"-use_editlist", "0",
"-flush_packets", "1",
"-f", "mp4",
"-movflags", "frag_keyframe+delay_moov+default_base_moof",
"pipe:1",
]
FFMPEG_REMUX_CMD_NO_BSF = [
"ffmpeg",
"-loglevel", "error",
"-f", "mpegts",
"-i", "pipe:0",
"-c", "copy",
"-use_editlist", "0",
"-flush_packets", "1",
"-f", "mp4",
"-movflags", "frag_keyframe+delay_moov+default_base_moof",
"pipe:1",
]
# Timeout waiting for init segment before giving up (seconds)
INIT_SEGMENT_TIMEOUT = 15
# MP4 box type for the first media fragment
MOOF_BOX_TYPE = b"moof"
# Redis TTL for init segment and buffer state keys
FMP4_KEY_TTL = 3600
def _find_moof_offset(data: bytes, start: int = 0) -> int:
"""
Scan `data` for the start of the first 'moof' box at or after `start`.
Returns the byte offset of the box or -1 if not found.
MP4 boxes: [4-byte big-endian length][4-byte type][payload...]
"""
offset = start
while offset + 8 <= len(data):
try:
box_size = struct.unpack_from(">I", data, offset)[0]
box_type = data[offset + 4: offset + 8]
if box_type == MOOF_BOX_TYPE:
return offset
if box_size < 8:
offset += 1
else:
offset += box_size
except struct.error:
break
return -1
class FMP4RemuxManager:
"""
Reads the TS Redis buffer for a channel, remuxes to fMP4 via FFmpeg,
and writes fMP4 chunks to FMP4StreamBuffer.
"""
def __init__(self, channel_id, ts_buffer, worker_id, fmt='fmp4'):
self.channel_id = channel_id
self.ts_buffer = ts_buffer
self.worker_id = worker_id
self.fmt = fmt
self.running = False
self._process = None
self._reader_thread = None
self._writer_thread = None
self._stderr_thread = None
self.fmp4_buffer = FMP4StreamBuffer(
channel_id, redis_client=RedisClient.get_buffer(), fmt=fmt
)
self._redis = RedisClient.get_client()
# ------------------------------------------------------------------
# Public API
# ------------------------------------------------------------------
def start(self):
"""Acquire the fmp4:owner lock and spawn the remux greenlets."""
if not self._acquire_owner_lock():
logger.info(f"[fMP4Remux:{self.channel_id}] Another worker owns fMP4 remux, skipping start")
return False
self.running = True
self._set_state(FMP4_STATE_INITIALIZING)
self._process = subprocess.Popen(
FFMPEG_REMUX_CMD,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
short_id = self.channel_id[:8]
self._reader_thread = threading.Thread(
target=self._reader_loop, daemon=True,
name=f"fmp4-reader-{short_id}"
)
self._writer_thread = threading.Thread(
target=self._writer_loop, daemon=True,
name=f"fmp4-writer-{short_id}"
)
self._stderr_thread = threading.Thread(
target=self._stderr_loop, daemon=True,
name=f"fmp4-stderr-{short_id}"
)
self._reader_thread.start()
self._writer_thread.start()
self._stderr_thread.start()
logger.info(f"[fMP4Remux:{self.channel_id}] Started (pid={self._process.pid})")
return True
def stop(self):
"""Gracefully stop the remux process and clean up all Redis keys."""
if not self.running:
return
self.running = False
logger.info(f"[fMP4Remux:{self.channel_id}] Stopping")
# Close FFmpeg stdin - signals EOF so it flushes and exits cleanly
try:
if self._process and self._process.stdin:
self._process.stdin.close()
except Exception:
pass
for t in (self._writer_thread, self._reader_thread):
if t and t.is_alive():
try:
t.join(timeout=5)
except Exception:
pass
# Kill FFmpeg if still running
try:
if self._process and self._process.poll() is None:
self._process.kill()
self._process.wait(timeout=3)
except Exception:
pass
self._cleanup_redis()
logger.info(f"[fMP4Remux:{self.channel_id}] Stopped")
# ------------------------------------------------------------------
# Internal loops
# ------------------------------------------------------------------
def _write_all(self, data: bytes):
"""Write all bytes to FFmpeg stdin, looping on partial writes."""
view = memoryview(data)
offset = 0
total = len(view)
while offset < total:
if not self.running:
return
n = self._process.stdin.write(view[offset:])
if n is None or n <= 0:
raise OSError("stdin write returned no bytes")
offset += n
def _writer_loop(self):
"""Read TS chunks from Redis and write to FFmpeg stdin."""
# Start behind live so the fMP4 buffer is pre-populated by the time
# the first client connects, matching TS client positioning behavior.
behind_seconds = ConfigHelper.new_client_behind_seconds()
start_index = self.ts_buffer.find_chunk_index_by_time(behind_seconds) if behind_seconds > 0 else None
if start_index is None:
start_index = self.ts_buffer.index
local_index = start_index
logger.debug(f"[fMP4Remux:{self.channel_id}] Writer started at buffer index {local_index} ({behind_seconds}s behind live)")
try:
while self.running:
chunks, new_index = self.ts_buffer.get_optimized_client_data(local_index)
if chunks:
local_index = new_index
logger.debug(
f"[fMP4Remux:{self.channel_id}] Writer: {len(chunks)} chunk(s) "
f"-> stdin (index now {local_index})"
)
for chunk in chunks:
if not self.running:
break
try:
self._write_all(chunk)
self._process.stdin.flush()
except (BrokenPipeError, OSError) as e:
logger.warning(
f"[fMP4Remux:{self.channel_id}] FFmpeg stdin error: {e}"
)
self.running = False
return
else:
if self.ts_buffer.index > local_index + 20:
local_index = self.ts_buffer.index - 5
time.sleep(0.05)
except Exception as e:
logger.error(f"[fMP4Remux:{self.channel_id}] Writer loop error: {e}", exc_info=True)
finally:
try:
if self._process and self._process.stdin:
self._process.stdin.close()
except Exception:
pass
logger.debug(f"[fMP4Remux:{self.channel_id}] Writer loop exited")
def _flush_complete_fragments(self, frag_buf: bytearray) -> None:
"""
Extract complete moof+mdat(+...) fragments from `frag_buf` (modifies in-place)
and store each one as a single Redis chunk via put_fragment.
A fragment ends where the next moof box begins.
"""
while len(frag_buf) >= 8:
if frag_buf[4:8] != b'moof':
# Stream no longer aligned to a moof - drop bytes until we find one
next_moof = _find_moof_offset(bytes(frag_buf), start=1)
if next_moof < 0:
frag_buf.clear()
return
del frag_buf[:next_moof]
continue
try:
moof_size = struct.unpack_from(">I", frag_buf, 0)[0]
except struct.error:
break
if moof_size < 8:
break
# Find where the NEXT moof box starts (= end of this fragment)
next_moof = _find_moof_offset(bytes(frag_buf), start=moof_size)
if next_moof < 0:
break # Current fragment not complete yet
fragment = bytes(frag_buf[:next_moof])
del frag_buf[:next_moof]
self.fmp4_buffer.put_fragment(fragment)
logger.debug(
f"[fMP4Remux:{self.channel_id}] Fragment {self.fmp4_buffer.index}: "
f"{len(fragment)} bytes"
)
def _reader_loop(self):
"""Read FFmpeg stdout, parse init segment, then feed each complete fMP4 fragment to the buffer."""
init_buf = bytearray()
init_stored = False
frag_buf = bytearray()
read_size = 65536
logger.debug(f"[fMP4Remux:{self.channel_id}] Reader started")
try:
while self.running:
ready, _, _ = select.select([self._process.stdout], [], [], 1.0)
if not ready:
if self._process.poll() is not None:
logger.info(
f"[fMP4Remux:{self.channel_id}] FFmpeg exited "
f"(code={self._process.returncode})"
)
break
continue
data = self._process.stdout.read(read_size)
if not data:
logger.info(f"[fMP4Remux:{self.channel_id}] FFmpeg stdout EOF")
break
if not init_stored:
init_buf.extend(data)
moof_offset = _find_moof_offset(bytes(init_buf))
if moof_offset >= 0:
init_segment = bytes(init_buf[:moof_offset])
frag_buf.extend(init_buf[moof_offset:])
init_buf = bytearray()
self._store_init_segment(init_segment)
self._set_state(FMP4_STATE_ACTIVE)
init_stored = True
logger.info(
f"[fMP4Remux:{self.channel_id}] Init segment stored "
f"({len(init_segment)} bytes)"
)
self._flush_complete_fragments(frag_buf)
elif len(init_buf) > 10 * 1024 * 1024:
logger.error(
f"[fMP4Remux:{self.channel_id}] No moof in first 10 MB, aborting"
)
self.running = False
break
else:
frag_buf.extend(data)
self._flush_complete_fragments(frag_buf)
except Exception as e:
logger.error(f"[fMP4Remux:{self.channel_id}] Reader loop error: {e}", exc_info=True)
finally:
if frag_buf and init_stored:
self.fmp4_buffer.put_fragment(bytes(frag_buf))
logger.info(f"[fMP4Remux:{self.channel_id}] Reader loop exited")
def _stderr_loop(self):
"""Log FFmpeg stderr lines. Detect BSF codec mismatch and trigger a no-BSF retry."""
try:
for raw_line in self._process.stderr:
line = raw_line.decode(errors="replace").rstrip()
if line:
logger.warning(f"[fMP4Remux:{self.channel_id}] FFmpeg: {line}")
if "aac_adtstoasc" in line and "is not supported by the bitstream filter" in line:
threading.Thread(
target=self._handle_bsf_error, daemon=True,
name=f"fmp4-bsf-retry-{self.channel_id[:8]}"
).start()
return
except Exception:
pass
def _handle_bsf_error(self):
"""Restart FFmpeg without the aac_adtstoasc BSF for non-AAC audio streams."""
logger.warning(f"[fMP4Remux:{self.channel_id}] Non-AAC audio detected, retrying without BSF")
self.running = False
try:
if self._process and self._process.poll() is None:
self._process.kill()
self._process.wait(timeout=3)
except Exception:
pass
for t in (self._reader_thread, self._writer_thread):
if t and t.is_alive():
try:
t.join(timeout=5)
except Exception:
pass
self._set_state(FMP4_STATE_INITIALIZING)
self._process = subprocess.Popen(
FFMPEG_REMUX_CMD_NO_BSF,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
self.running = True
if not self.running:
# stop() was called while we were restarting - clean up immediately
try:
self._process.kill()
self._process.wait(timeout=3)
except Exception:
pass
self._cleanup_redis()
return
short_id = self.channel_id[:8]
self._reader_thread = threading.Thread(
target=self._reader_loop, daemon=True,
name=f"fmp4-reader-{short_id}"
)
self._writer_thread = threading.Thread(
target=self._writer_loop, daemon=True,
name=f"fmp4-writer-{short_id}"
)
self._stderr_thread = threading.Thread(
target=self._stderr_loop, daemon=True,
name=f"fmp4-stderr-{short_id}"
)
self._reader_thread.start()
self._writer_thread.start()
self._stderr_thread.start()
logger.info(f"[fMP4Remux:{self.channel_id}] Restarted without BSF (pid={self._process.pid})")
# ------------------------------------------------------------------
# Redis helpers
# ------------------------------------------------------------------
def _acquire_owner_lock(self) -> bool:
if not self._redis:
return True
owner_key = RedisKeys.output_owner(self.channel_id, self.fmt)
acquired = self._redis.set(owner_key, self.worker_id, nx=True, ex=FMP4_KEY_TTL)
if acquired:
return True
existing = self._redis.get(owner_key)
return existing == self.worker_id
def _set_state(self, state: str):
if self._redis:
self._redis.setex(RedisKeys.output_state(self.channel_id, self.fmt), FMP4_KEY_TTL, state)
def _store_init_segment(self, data: bytes):
redis_buf = RedisClient.get_buffer()
if redis_buf:
redis_buf.setex(RedisKeys.output_init(self.channel_id, self.fmt), FMP4_KEY_TTL, data)
def _cleanup_redis(self):
"""Delete all output buffer Redis keys for this channel."""
if not self._redis:
return
try:
keys_to_delete = [
RedisKeys.output_init(self.channel_id, self.fmt),
RedisKeys.output_state(self.channel_id, self.fmt),
RedisKeys.output_owner(self.channel_id, self.fmt),
]
self._redis.delete(*keys_to_delete)
self.fmp4_buffer.cleanup_redis()
except Exception as e:
logger.error(f"[fMP4Remux:{self.channel_id}] Error during Redis cleanup: {e}")

0
apps/proxy/live_proxy/output/profile/__init__.py Normal file
View file

345
apps/proxy/live_proxy/output/profile/manager.py Normal file
View file

@ -0,0 +1,345 @@
"""
Output Profile Transcode Manager
Reads from the shared TS Redis buffer, pipes data through a user-defined
transcoding command (pipe:0 stdin → pipe:1 stdout), and writes the output
chunks to a Redis-backed StreamBuffer under:
live:channel:{channel_id}:profile:{profile_id}:buffer:*
One transcode process runs per active (channel, profile) pair across the
entire cluster. The TS-owning worker starts the process; non-owning workers
create a read-only StreamBuffer pointing at the same Redis keys.
"""
import select
import subprocess
import threading
import time
from core.utils import RedisClient
from ...input.buffer import StreamBuffer
from ...redis_keys import RedisKeys
from ...config_helper import ConfigHelper
from ...utils import get_logger
logger = get_logger()
PROFILE_STATE_ACTIVE = "active"
PROFILE_STATE_STOPPED = "stopped"
PROFILE_KEY_TTL = 3600
class OutputProfileManager:
"""
Reads the TS Redis buffer for a channel, transcodes via a user-supplied
command, and writes output chunks into a profile-namespaced StreamBuffer.
"""
def __init__(self, channel_id, profile_id, command, ts_buffer, worker_id):
"""
Args:
channel_id: Channel UUID string.
profile_id: OutputProfile PK (int).
command: List from OutputProfile.build_command().
ts_buffer: Source StreamBuffer (the channel's raw TS input buffer).
worker_id: This worker's ID string for owner-lock coordination.
"""
self.channel_id = channel_id
self.profile_id = profile_id
self.command = command
self.ts_buffer = ts_buffer
self.worker_id = worker_id
self.running = False
self._process = None
self._writer_thread = None
self._reader_thread = None
self._stderr_thread = None
self._redis = RedisClient.get_client()
self.output_buffer = None # assigned in start()
# ------------------------------------------------------------------
# Public API
# ------------------------------------------------------------------
def start(self) -> bool:
"""
Acquire the owner lock, spawn the transcode process and threads.
Returns True if this worker started the process, False if another
worker already owns it (caller should still use output_buffer for reads).
"""
if not self._acquire_owner_lock():
logger.info(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] "
"Another worker owns transcode, using shared buffer"
)
self.output_buffer = self._make_buffer()
return False
self.output_buffer = self._make_buffer()
try:
self._process = subprocess.Popen(
self.command,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
except (FileNotFoundError, OSError) as e:
logger.error(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] "
f"Failed to start transcode process: {e}"
)
self._release_owner_lock()
return False
self.running = True
self._set_state(PROFILE_STATE_ACTIVE)
short = f"{self.channel_id[:8]}:p{self.profile_id}"
self._writer_thread = threading.Thread(
target=self._writer_loop, daemon=True,
name=f"profile-writer-{short}"
)
self._reader_thread = threading.Thread(
target=self._reader_loop, daemon=True,
name=f"profile-reader-{short}"
)
self._stderr_thread = threading.Thread(
target=self._stderr_loop, daemon=True,
name=f"profile-stderr-{short}"
)
self._writer_thread.start()
self._reader_thread.start()
self._stderr_thread.start()
logger.info(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] "
f"Transcode started (pid={self._process.pid})"
)
return True
def stop(self):
"""Stop the transcode process and clean up all Redis keys."""
if not self.running:
return
self.running = False
logger.info(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] Stopping transcode"
)
try:
if self._process and self._process.stdin:
self._process.stdin.close()
except Exception:
pass
for t in (self._writer_thread, self._reader_thread):
if t and t.is_alive():
try:
t.join(timeout=5)
except Exception:
pass
try:
if self._process and self._process.poll() is None:
self._process.kill()
self._process.wait(timeout=3)
except Exception:
pass
self._cleanup_redis()
logger.info(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] Transcode stopped"
)
# ------------------------------------------------------------------
# Internal threads
# ------------------------------------------------------------------
def _write_all(self, data: bytes):
"""Write all bytes to process stdin, looping on partial writes."""
view = memoryview(data)
offset = 0
total = len(view)
while offset < total:
if not self.running:
return
n = self._process.stdin.write(view[offset:])
if n is None or n <= 0:
raise OSError("stdin write returned no bytes")
offset += n
def _writer_loop(self):
"""Read TS chunks from Redis and write to the transcode process stdin."""
behind_seconds = ConfigHelper.new_client_behind_seconds()
start_index = self.ts_buffer.find_chunk_index_by_time(behind_seconds) if behind_seconds > 0 else None
if start_index is None:
start_index = self.ts_buffer.index
local_index = start_index
logger.debug(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] "
f"Writer started at index {local_index}"
)
try:
while self.running:
chunks, new_index = self.ts_buffer.get_optimized_client_data(local_index)
if chunks:
local_index = new_index
for chunk in chunks:
if not self.running:
break
try:
self._write_all(chunk)
self._process.stdin.flush()
except (BrokenPipeError, OSError) as e:
logger.warning(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] "
f"Stdin error: {e}"
)
self.running = False
return
else:
if self.ts_buffer.index > local_index + 20:
local_index = self.ts_buffer.index - 5
time.sleep(0.05)
except Exception as e:
logger.error(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] "
f"Writer loop error: {e}", exc_info=True
)
finally:
try:
if self._process and self._process.stdin:
self._process.stdin.close()
except Exception:
pass
logger.debug(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] Writer loop exited"
)
def _reader_loop(self):
"""Read chunks from process stdout and write to the output StreamBuffer."""
read_size = 65536
logger.debug(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] Reader started"
)
try:
while self.running:
ready, _, _ = select.select([self._process.stdout], [], [], 1.0)
if not ready:
if self._process.poll() is not None:
logger.info(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] "
f"Process exited (code={self._process.returncode})"
)
break
continue
data = self._process.stdout.read(read_size)
if not data:
logger.info(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] stdout EOF"
)
break
self.output_buffer.add_chunk(data)
except Exception as e:
logger.error(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] "
f"Reader loop error: {e}", exc_info=True
)
finally:
if self.output_buffer:
self.output_buffer.stop()
logger.info(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] Reader loop exited"
)
def _stderr_loop(self):
"""Log process stderr at WARNING level."""
try:
for raw_line in self._process.stderr:
line = raw_line.decode(errors="replace").rstrip()
if line:
logger.warning(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] {line}"
)
except Exception:
pass
# ------------------------------------------------------------------
# Redis helpers
# ------------------------------------------------------------------
def _make_buffer(self) -> StreamBuffer:
"""Create a StreamBuffer wired to this profile's Redis key namespace."""
fmt = f"mpegts:p{self.profile_id}"
return StreamBuffer(
channel_id=self.channel_id,
redis_client=RedisClient.get_buffer(),
buffer_index_key=RedisKeys.output_buffer_index(self.channel_id, fmt),
buffer_chunk_prefix=RedisKeys.output_buffer_chunk_prefix(self.channel_id, fmt),
chunk_timestamps_key=RedisKeys.output_chunk_timestamps(self.channel_id, fmt),
)
def _acquire_owner_lock(self) -> bool:
if not self._redis:
return True
owner_key = RedisKeys.output_owner(self.channel_id, f"mpegts:p{self.profile_id}")
acquired = self._redis.set(owner_key, self.worker_id, nx=True, ex=PROFILE_KEY_TTL)
if acquired:
return True
existing = self._redis.get(owner_key)
return existing == self.worker_id
def _release_owner_lock(self):
if self._redis:
try:
self._redis.delete(RedisKeys.output_owner(self.channel_id, f"mpegts:p{self.profile_id}"))
except Exception:
pass
def _set_state(self, state: str):
if self._redis:
self._redis.setex(
RedisKeys.output_state(self.channel_id, f"mpegts:p{self.profile_id}"),
PROFILE_KEY_TTL,
state,
)
def _cleanup_redis(self):
"""Delete all output:mpegts:p{profile_id}:* Redis keys for this (channel, profile) pair."""
if not self._redis:
return
fmt = f"mpegts:p{self.profile_id}"
try:
keys = [
RedisKeys.output_state(self.channel_id, fmt),
RedisKeys.output_owner(self.channel_id, fmt),
RedisKeys.output_buffer_index(self.channel_id, fmt),
RedisKeys.output_chunk_timestamps(self.channel_id, fmt),
]
self._redis.delete(*keys)
# Delete all chunk keys via scan
buf_client = RedisClient.get_buffer()
if buf_client:
prefix = RedisKeys.output_buffer_chunk_prefix(self.channel_id, fmt)
cursor = 0
while True:
cursor, chunk_keys = buf_client.scan(cursor, match=f"{prefix}*", count=200)
if chunk_keys:
buf_client.delete(*chunk_keys)
if cursor == 0:
break
except Exception as e:
logger.error(
f"[Profile:{self.profile_id}:{self.channel_id[:8]}] "
f"Redis cleanup error: {e}"
)

1
apps/proxy/live_proxy/output/ts/__init__.py Normal file
View file

@ -0,0 +1 @@

31
apps/proxy/ts_proxy/stream_generator.py → apps/proxy/live_proxy/output/ts/generator.py
View file

@ -4,18 +4,15 @@ This module handles generating and delivering video streams to clients.
"""
import time
import logging
import threading
import gevent # Add this import at the top of your file
import gevent
from apps.proxy.config import TSConfig as Config
from apps.channels.models import Channel, Stream
from core.utils import log_system_event
from .server import ProxyServer
from .utils import create_ts_packet, get_logger
from .redis_keys import RedisKeys
from .utils import get_logger
from .constants import ChannelMetadataField
from .config_helper import ConfigHelper # Add this import
from ...server import ProxyServer
from ...utils import create_ts_packet, get_logger
from ...redis_keys import RedisKeys
from ...constants import ChannelMetadataField
from ...config_helper import ConfigHelper
logger = get_logger()
@ -25,7 +22,7 @@ class StreamGenerator:
data delivery, and cleanup.
"""
def __init__(self, channel_id, client_id, client_ip, client_user_agent, channel_initializing=False, user=None):
def __init__(self, channel_id, client_id, client_ip, client_user_agent, channel_initializing=False, user=None, buffer=None):
"""
Initialize the stream generator with client and channel details.
@ -36,6 +33,8 @@ class StreamGenerator:
client_user_agent: User agent string from client
channel_initializing: Whether the channel is still initializing
user: Authenticated user making the request
buffer: Source StreamBuffer to read from. Resolved via ProxyServer.get_buffer()
before construction; passed in so the generator is buffer-agnostic.
"""
self.channel_id = channel_id
self.client_id = client_id
@ -43,6 +42,7 @@ class StreamGenerator:
self.client_user_agent = client_user_agent
self.channel_initializing = channel_initializing
self.user = user
self._source_buffer = buffer
# Cache channel name once to avoid repeated DB queries for logging
try:
_name = Channel.objects.filter(uuid=channel_id).values_list('name', flat=True).first()
@ -204,8 +204,7 @@ class StreamGenerator:
"""Setup streaming parameters and check resources."""
proxy_server = ProxyServer.get_instance()
# Get buffer - stream manager may not exist in this worker
buffer = proxy_server.stream_buffers.get(self.channel_id)
buffer = self._source_buffer
stream_manager = proxy_server.stream_managers.get(self.channel_id)
if not buffer:
@ -369,7 +368,7 @@ class StreamGenerator:
def _check_resources(self):
"""Check if required resources still exist."""
proxy_server = self.proxy_server or ProxyServer.get_instance()
if self.channel_id not in proxy_server.stream_buffers:
if self.buffer is None:
logger.info(f"[{self.client_id}] Channel buffer no longer exists, terminating stream")
return False
@ -477,7 +476,7 @@ class StreamGenerator:
# Refresh TTL on client key
proxy_server.redis_client.expire(client_key, Config.CLIENT_RECORD_TTL)
# Also refresh the client set TTL
client_set_key = f"ts_proxy:channel:{self.channel_id}:clients"
client_set_key = f"live:channel:{self.channel_id}:clients"
proxy_server.redis_client.expire(client_set_key, Config.CLIENT_RECORD_TTL)
self.last_ttl_refresh = current_time
logger.debug(f"[{self.client_id}] Refreshed client TTL (active streaming)")
@ -644,10 +643,10 @@ class StreamGenerator:
gevent.spawn(delayed_shutdown)
def create_stream_generator(channel_id, client_id, client_ip, client_user_agent, channel_initializing=False, user=None):
def create_stream_generator(channel_id, client_id, client_ip, client_user_agent, channel_initializing=False, user=None, buffer=None):
"""
Factory function to create a new stream generator.
Returns a function that can be passed to StreamingHttpResponse.
"""
generator = StreamGenerator(channel_id, client_id, client_ip, client_user_agent, channel_initializing, user=user)
generator = StreamGenerator(channel_id, client_id, client_ip, client_user_agent, channel_initializing, user=user, buffer=buffer)
return generator.generate

132
apps/proxy/live_proxy/redis_keys.py Normal file
View file

@ -0,0 +1,132 @@
"""
Defines Redis key patterns used throughout the TS proxy service.
Centralizing these key patterns makes it easier to maintain and change them if needed.
"""
class RedisKeys:
@staticmethod
def channel_metadata(channel_id):
"""Key for channel metadata hash"""
return f"live:channel:{channel_id}:metadata"
@staticmethod
def buffer_index(channel_id):
"""Key for tracking input buffer index"""
return f"live:channel:{channel_id}:input:buffer:index"
@staticmethod
def buffer_chunk(channel_id, chunk_index):
"""Key for specific input buffer chunk"""
return f"live:channel:{channel_id}:input:buffer:chunk:{chunk_index}"
@staticmethod
def buffer_chunk_prefix(channel_id):
"""Prefix for input buffer chunks"""
return f"live:channel:{channel_id}:input:buffer:chunk:"
@staticmethod
def channel_stopping(channel_id):
"""Key indicating channel is stopping"""
return f"live:channel:{channel_id}:stopping"
@staticmethod
def client_stop(channel_id, client_id):
"""Key requesting client stop"""
return f"live:channel:{channel_id}:client:{client_id}:stop"
@staticmethod
def events_channel(channel_id):
"""PubSub channel for events"""
return f"live:events:{channel_id}"
@staticmethod
def switch_request(channel_id):
"""Key for stream switch request"""
return f"live:channel:{channel_id}:switch_request"
@staticmethod
def channel_owner(channel_id):
"""Key for storing channel owner worker ID"""
return f"live:channel:{channel_id}:owner"
@staticmethod
def clients(channel_id):
"""Key for set of client IDs"""
return f"live:channel:{channel_id}:clients"
@staticmethod
def last_client_disconnect(channel_id):
"""Key for last client disconnect timestamp"""
return f"live:channel:{channel_id}:last_client_disconnect_time"
@staticmethod
def connection_attempt(channel_id):
"""Key for connection attempt timestamp"""
return f"live:channel:{channel_id}:connection_attempt_time"
@staticmethod
def last_data(channel_id):
"""Key for last data timestamp"""
return f"live:channel:{channel_id}:last_data"
@staticmethod
def switch_status(channel_id):
"""Key for stream switch status"""
return f"live:channel:{channel_id}:switch_status"
@staticmethod
def worker_heartbeat(worker_id):
"""Key for worker heartbeat"""
return f"live:worker:{worker_id}:heartbeat"
@staticmethod
def chunk_timestamps(channel_id):
"""Sorted set mapping chunk receive-timestamps (score) to chunk indices (member).
Used for time-based client positioning."""
return f"live:channel:{channel_id}:input:buffer:chunk_timestamps"
@staticmethod
def transcode_active(channel_id):
"""Key indicating active transcode process"""
return f"live:channel:{channel_id}:transcode_active"
@staticmethod
def client_metadata(channel_id, client_id):
"""Key for client metadata hash"""
return f"live:channel:{channel_id}:clients:{client_id}"
# Output format buffer keys - parameterized by format name (e.g. 'fmp4').
# Adding a new output format only requires a new manager; the key structure
# is shared so no new key methods are needed.
@staticmethod
def output_buffer_index(channel_id, fmt):
return f"live:channel:{channel_id}:output:{fmt}:buffer:index"
@staticmethod
def output_buffer_chunk(channel_id, fmt, chunk_index):
return f"live:channel:{channel_id}:output:{fmt}:buffer:chunk:{chunk_index}"
@staticmethod
def output_buffer_chunk_prefix(channel_id, fmt):
return f"live:channel:{channel_id}:output:{fmt}:buffer:chunk:"
@staticmethod
def output_init(channel_id, fmt):
"""Binary init segment for formats that require one (e.g. fMP4 ftyp+moov)."""
return f"live:channel:{channel_id}:output:{fmt}:init"
@staticmethod
def output_state(channel_id, fmt):
"""Remux/transcode manager state for this output format."""
return f"live:channel:{channel_id}:output:{fmt}:state"
@staticmethod
def output_owner(channel_id, fmt):
"""Worker ID owning the output format manager."""
return f"live:channel:{channel_id}:output:{fmt}:owner"
@staticmethod
def output_chunk_timestamps(channel_id, fmt):
"""Sorted set mapping fragment receive-timestamps to fragment indices."""
return f"live:channel:{channel_id}:output:{fmt}:buffer:chunk_timestamps"

389
apps/proxy/ts_proxy/server.py → apps/proxy/live_proxy/server.py
View file

@ -8,22 +8,21 @@ Handles live TS stream proxying with support for:
"""
import threading
import logging
import socket
import random
import time
import sys
import os
import json
import gevent # Add gevent import
from typing import Dict, Optional, Set
import gevent
from apps.proxy.config import TSConfig as Config
from apps.channels.models import Channel, Stream
from core.utils import RedisClient, log_system_event
from redis.exceptions import ConnectionError, TimeoutError
from .stream_manager import StreamManager
from .stream_buffer import StreamBuffer
from .input.manager import StreamManager
from .input.buffer import StreamBuffer
from .client_manager import ClientManager
from .output.fmp4.manager import FMP4RemuxManager
from .output.profile.manager import OutputProfileManager, PROFILE_STATE_ACTIVE
from .redis_keys import RedisKeys
from .constants import ChannelState, EventType, StreamType
from .config_helper import ConfigHelper
@ -45,8 +44,8 @@ class ProxyServer:
cls._instance = cls._INITIALIZING
try:
from .server import ProxyServer
from .stream_manager import StreamManager
from .stream_buffer import StreamBuffer
from .input.manager import StreamManager
from .input.buffer import StreamBuffer
from .client_manager import ClientManager
real_instance = ProxyServer()
cls._instance = real_instance
@ -66,11 +65,12 @@ class ProxyServer:
self.stream_managers = {}
self.stream_buffers = {}
self.client_managers = {}
self.output_managers = {} # {channel_id: {fmt: manager}}
self.profile_managers = {} # {channel_id: {profile_id: OutputProfileManager}}
self.profile_buffers = {} # {channel_id: {profile_id: StreamBuffer}}
self._channel_names = {}
# Generate a unique worker ID
import socket
import os
pid = os.getpid()
hostname = socket.gethostname()
self.worker_id = f"{hostname}:{pid}"
@ -188,7 +188,7 @@ class ProxyServer:
# Create a pubsub instance from the client
pubsub = pubsub_client.pubsub()
pubsub.psubscribe("ts_proxy:events:*")
pubsub.psubscribe("live:events:*")
logger.info(f"Started Redis event listener for client activity")
@ -260,7 +260,7 @@ class ProxyServer:
"timestamp": time.time()
}
self.redis_client.publish(
f"ts_proxy:events:{channel_id}",
f"live:events:{channel_id}",
json.dumps(switch_result)
)
@ -287,7 +287,7 @@ class ProxyServer:
"timestamp": time.time()
}
self.redis_client.publish(
f"ts_proxy:events:{channel_id}",
f"live:events:{channel_id}",
json.dumps(switch_result)
)
elif event_type == EventType.CHANNEL_STOP:
@ -316,7 +316,7 @@ class ProxyServer:
"timestamp": time.time()
}
self.redis_client.publish(
f"ts_proxy:events:{channel_id}",
f"live:events:{channel_id}",
json.dumps(stop_response)
)
elif event_type == EventType.CLIENT_STOP:
@ -336,6 +336,19 @@ class ProxyServer:
stop_key = RedisKeys.client_stop(channel_id, client_id)
self.redis_client.setex(stop_key, 30, "true") # 30 second TTL
logger.info(f"Set stop key for client {client_id}")
elif event_type == EventType.ENSURE_OUTPUT_FORMAT:
fmt = data.get("fmt")
if fmt:
logger.info(f"Owner received ENSURE_OUTPUT_FORMAT for channel {channel_id}, fmt={fmt}")
self.ensure_output_format(channel_id, fmt)
elif event_type == EventType.ENSURE_OUTPUT_PROFILE:
profile_id = data.get("profile_id")
command = data.get("command")
if profile_id is not None and command:
logger.info(f"Owner received ENSURE_OUTPUT_PROFILE for channel {channel_id}, profile={profile_id}")
self.ensure_output_profile(channel_id, profile_id, command)
except Exception as e:
logger.error(f"Error processing event message: {e}")
@ -348,12 +361,12 @@ class ProxyServer:
final_delay = delay + jitter
logger.error(f"Error in event listener: {e}. Retrying in {final_delay:.1f}s (attempt {retry_count})")
gevent.sleep(final_delay) # REPLACE: time.sleep(final_delay)
gevent.sleep(final_delay)
except Exception as e:
logger.error(f"Error in event listener: {e}")
# Add a short delay to prevent rapid retries on persistent errors
gevent.sleep(5) # REPLACE: time.sleep(5)
gevent.sleep(5)
finally:
# Always clean up PubSub connections in all error paths
@ -498,7 +511,6 @@ class ProxyServer:
def initialize_channel(self, url, channel_id, user_agent=None, transcode=False, stream_id=None):
"""Initialize a channel without redundant active key"""
try:
# IMPROVED: First check if channel is already being initialized by another process
if self.redis_client:
metadata_key = RedisKeys.channel_metadata(channel_id)
if self.redis_client.exists(metadata_key):
@ -533,7 +545,6 @@ class ProxyServer:
)
self.client_managers[channel_id] = client_manager
# IMPROVED: Set initializing state in Redis BEFORE any other operations
if self.redis_client:
# Set early initialization state to prevent race conditions
metadata_key = RedisKeys.channel_metadata(channel_id)
@ -761,7 +772,7 @@ class ProxyServer:
# If the channel is in a valid state, check if the owner is still active
if state in valid_states:
# Check if owner still exists by checking heartbeat
owner_heartbeat_key = f"ts_proxy:worker:{owner}:heartbeat"
owner_heartbeat_key = f"live:worker:{owner}:heartbeat"
owner_alive = self.redis_client.exists(owner_heartbeat_key)
if owner_alive:
@ -859,42 +870,348 @@ class ProxyServer:
def handle_client_disconnect(self, channel_id):
"""
Handle client disconnect event - check if channel should shut down.
Can be called directly by owner or via PubSub from non-owner workers.
Handle client disconnect event - check if channel should shut down and
whether any output profile managers can be stopped.
"""
if channel_id not in self.client_managers:
return
try:
# VERIFY REDIS CLIENT COUNT DIRECTLY
client_set_key = RedisKeys.clients(channel_id)
total = self.redis_client.scard(client_set_key) or 0
# Check which output formats/profiles still have active clients
if self.output_managers.get(channel_id) or self.profile_managers.get(channel_id):
if total > 0:
remaining_ids = self.redis_client.smembers(client_set_key)
remaining_list = [
cid.decode() if isinstance(cid, bytes) else cid
for cid in remaining_ids
]
pipe = self.redis_client.pipeline(transaction=False)
for cid in remaining_list:
pipe.hget(RedisKeys.client_metadata(channel_id, cid), "output_format")
pipe.hget(RedisKeys.client_metadata(channel_id, cid), "output_profile_id")
results = pipe.execute()
active_formats = set()
active_profiles = set()
active_manager_keys = set()
for i in range(0, len(results), 2):
fmt = results[i]
pid = results[i + 1]
if fmt:
fmt_str = fmt.decode() if isinstance(fmt, bytes) else fmt
active_formats.add(fmt_str)
pid_str = (pid.decode() if isinstance(pid, bytes) else pid) if pid else ''
if pid_str:
try:
active_manager_keys.add(f"{fmt_str}:p{int(pid_str)}")
except ValueError:
pass
else:
active_manager_keys.add(fmt_str)
if pid:
pid_str = pid.decode() if isinstance(pid, bytes) else pid
if pid_str:
try:
active_profiles.add(int(pid_str))
except ValueError:
pass
else:
active_formats = set()
active_profiles = set()
active_manager_keys = set()
for fmt in list(self.output_managers.get(channel_id, {}).keys()):
if fmt not in active_manager_keys:
logger.info(f"[output:{fmt}] No clients remain, stopping manager for channel {channel_id}")
self.stop_output_format(channel_id, fmt)
for pid in list(self.profile_managers.get(channel_id, {}).keys()):
if pid not in active_profiles:
logger.info(f"[Profile:{pid}] No clients remain, stopping transcode for channel {channel_id}")
self.stop_output_profile(channel_id, pid)
if total == 0:
logger.debug(f"No clients left after disconnect event - stopping channel {channel_id}")
# Set the disconnect timer for other workers to see
disconnect_key = RedisKeys.last_client_disconnect(channel_id)
self.redis_client.setex(disconnect_key, 60, str(time.time()))
# Get configured shutdown delay or default
shutdown_delay = ConfigHelper.channel_shutdown_delay()
if shutdown_delay > 0:
logger.info(f"Waiting {shutdown_delay}s before stopping channel...")
gevent.sleep(shutdown_delay)
# Re-check client count before stopping
total = self.redis_client.scard(client_set_key) or 0
if total > 0:
logger.info(f"New clients connected during shutdown delay - aborting shutdown")
self.redis_client.delete(disconnect_key)
return
# Stop the channel directly
self.stop_channel(channel_id)
except Exception as e:
logger.error(f"Error handling client disconnect for channel {channel_id}: {e}")
# ------------------------------------------------------------------
# Output format lifecycle
# ------------------------------------------------------------------
def get_buffer(self, channel_id, profile=None):
"""
Resolve the source buffer for a given channel and optional profile.
With no profile, returns the raw input StreamBuffer.
With a profile_id, returns that profile's transcoded output StreamBuffer.
Raises KeyError if the profile buffer is not yet active.
"""
if profile is not None:
channel_profiles = self.profile_buffers.get(channel_id, {})
if profile not in channel_profiles:
raise KeyError(f"Profile '{profile}' not active for channel {channel_id}")
return channel_profiles[profile]
return self.stream_buffers.get(channel_id)
def ensure_output_format(self, channel_id, fmt, source_buffer=None) -> bool:
"""
Start an output format manager for this channel if not already running.
Only the TS-owning worker starts the manager; non-owners read the shared buffer.
Returns True if a manager is active (locally or on another worker).
"""
if channel_id in self.output_managers and fmt in self.output_managers[channel_id]:
return True
if not self.redis_client:
return False
state = self.redis_client.get(RedisKeys.output_state(channel_id, fmt))
if state == 'active':
owner_val = self.redis_client.get(RedisKeys.output_owner(channel_id, fmt))
if owner_val and owner_val != self.worker_id:
logger.info(f"[output:{fmt}] Channel {channel_id}: manager active on another worker")
return True
# State says active but we have no local manager - stale state from a dead manager.
# Fall through to restart if we can.
logger.warning(
f"[output:{fmt}] Channel {channel_id}: stale active state detected "
f"(owner={owner_val}), restarting manager"
)
if not self.am_i_owner(channel_id):
# Ask the TS-owning worker to start the manager, then poll until active.
logger.info(f"[output:{fmt}] Channel {channel_id}: requesting owner to start manager")
self.redis_client.publish(
f"live:events:{channel_id}",
json.dumps({
"event": EventType.ENSURE_OUTPUT_FORMAT,
"channel_id": channel_id,
"fmt": fmt,
"timestamp": time.time(),
})
)
deadline = time.time() + 5
while time.time() < deadline:
gevent.sleep(0.1)
state = self.redis_client.get(RedisKeys.output_state(channel_id, fmt))
if state == 'active':
logger.info(f"[output:{fmt}] Channel {channel_id}: manager started by owner")
return True
logger.warning(f"[output:{fmt}] Channel {channel_id}: owner did not start manager within 5s")
return False
ts_buffer = source_buffer
if ts_buffer is None:
_, profile_id = self._parse_output_key(fmt)
if profile_id is not None:
ts_buffer = self.profile_buffers.get(channel_id, {}).get(profile_id)
if ts_buffer is None:
ts_buffer = self.stream_buffers.get(channel_id)
if not ts_buffer:
logger.error(f"[output:{fmt}] Channel {channel_id}: no TS buffer, cannot start manager")
return False
_OUTPUT_FORMAT_MANAGERS = {
'fmp4': FMP4RemuxManager,
}
base_fmt, _ = self._parse_output_key(fmt)
manager_cls = _OUTPUT_FORMAT_MANAGERS.get(base_fmt)
if manager_cls is None:
logger.error(f"[output:{fmt}] Unknown output format '{base_fmt}'")
return False
manager = manager_cls(channel_id, ts_buffer, self.worker_id, fmt=fmt)
started = manager.start()
if started:
self.output_managers.setdefault(channel_id, {})[fmt] = manager
logger.info(f"[output:{fmt}] Channel {channel_id}: manager started")
return started
@staticmethod
def _parse_output_key(fmt):
"""
Split a compound output manager key into (base_format, profile_id).
'fmp4' -> ('fmp4', None)
'fmp4:p1' -> ('fmp4', 1)
'hls:p3' -> ('hls', 3)
"""
if ':p' in fmt:
base, _, suffix = fmt.rpartition(':p')
try:
return base, int(suffix)
except ValueError:
pass
return fmt, None
def stop_output_format(self, channel_id, fmt):
"""Stop and remove a specific output format manager for this channel."""
channel_managers = self.output_managers.get(channel_id, {})
manager = channel_managers.pop(fmt, None)
if not channel_managers:
self.output_managers.pop(channel_id, None)
if manager:
try:
manager.stop()
logger.info(f"[output:{fmt}] Channel {channel_id}: manager stopped")
except Exception as e:
logger.error(f"[output:{fmt}] Channel {channel_id}: error stopping manager: {e}")
def stop_all_output_formats(self, channel_id):
"""Stop all output format managers for a channel."""
for fmt in list(self.output_managers.get(channel_id, {}).keys()):
self.stop_output_format(channel_id, fmt)
# ------------------------------------------------------------------
# Output profile lifecycle
# ------------------------------------------------------------------
def ensure_output_profile(self, channel_id, profile_id, command) -> bool:
"""
Start an OutputProfileManager for this (channel, profile) pair if not
already running. Only the TS-owning worker starts the process; all
workers (including non-owners) get a StreamBuffer wired to the same
Redis keys so they can serve clients.
Returns True once the profile buffer is available in self.profile_buffers.
"""
channel_profiles = self.profile_buffers.get(channel_id, {})
if profile_id in channel_profiles:
return True
if not self.redis_client:
return False
# Check if another worker already owns the transcode
state = self.redis_client.get(RedisKeys.output_state(channel_id, f"mpegts:p{profile_id}"))
if state == PROFILE_STATE_ACTIVE:
owner_val = self.redis_client.get(RedisKeys.output_owner(channel_id, f"mpegts:p{profile_id}"))
if owner_val and owner_val != self.worker_id:
# Non-owner: create a reader buffer pointing at the same Redis keys
manager = OutputProfileManager(
channel_id, profile_id, command,
self.stream_buffers.get(channel_id), self.worker_id
)
# start() will fail to acquire lock, but sets manager.output_buffer
manager.start()
self.profile_buffers.setdefault(channel_id, {})[profile_id] = manager.output_buffer
logger.info(
f"[Profile:{profile_id}:{channel_id[:8]}] "
"Using transcode buffer from owning worker"
)
return True
# State is active but we own it (or owner missing) with no local manager - stale.
logger.warning(
f"[Profile:{profile_id}:{channel_id[:8]}] "
f"Stale active state detected (owner={owner_val}), restarting transcode"
)
if not self.am_i_owner(channel_id):
logger.info(
f"[Profile:{profile_id}:{channel_id[:8]}] "
"Not TS owner, requesting owner to start transcode"
)
self.redis_client.publish(
f"live:events:{channel_id}",
json.dumps({
"event": EventType.ENSURE_OUTPUT_PROFILE,
"channel_id": channel_id,
"profile_id": profile_id,
"command": command,
"timestamp": time.time(),
})
)
state_key = RedisKeys.output_state(channel_id, f"mpegts:p{profile_id}")
deadline = time.time() + 5
while time.time() < deadline:
gevent.sleep(0.1)
state = self.redis_client.get(state_key)
if state == PROFILE_STATE_ACTIVE:
# Non-owner: wire up a reader buffer pointing at the same Redis keys.
manager = OutputProfileManager(
channel_id, profile_id, command,
self.stream_buffers.get(channel_id), self.worker_id
)
manager.start()
self.profile_buffers.setdefault(channel_id, {})[profile_id] = manager.output_buffer
logger.info(
f"[Profile:{profile_id}:{channel_id[:8]}] "
"Using transcode buffer from owning worker"
)
return True
logger.warning(
f"[Profile:{profile_id}:{channel_id[:8]}] "
"Owner did not start transcode within 5s"
)
return False
ts_buffer = self.stream_buffers.get(channel_id)
if not ts_buffer:
logger.error(
f"[Profile:{profile_id}:{channel_id[:8]}] "
"No TS buffer available, cannot start transcode"
)
return False
manager = OutputProfileManager(
channel_id, profile_id, command, ts_buffer, self.worker_id
)
started = manager.start()
if started or manager.output_buffer is not None:
self.profile_managers.setdefault(channel_id, {})[profile_id] = manager
self.profile_buffers.setdefault(channel_id, {})[profile_id] = manager.output_buffer
logger.info(
f"[Profile:{profile_id}:{channel_id[:8]}] "
f"Transcode {'started' if started else 'buffer created'}"
)
return True
return False
def stop_output_profile(self, channel_id, profile_id):
"""Stop a profile transcode manager and remove its buffer."""
channel_managers = self.profile_managers.get(channel_id, {})
manager = channel_managers.pop(profile_id, None)
if not channel_managers:
self.profile_managers.pop(channel_id, None)
self.profile_buffers.get(channel_id, {}).pop(profile_id, None)
if not self.profile_buffers.get(channel_id):
self.profile_buffers.pop(channel_id, None)
if manager:
try:
manager.stop()
logger.info(
f"[Profile:{profile_id}:{channel_id[:8]}] Transcode stopped"
)
except Exception as e:
logger.error(
f"[Profile:{profile_id}:{channel_id[:8]}] Error stopping: {e}"
)
def stop_all_output_profiles(self, channel_id):
"""Stop all profile transcode managers for a channel."""
for pid in list(self.profile_managers.get(channel_id, {}).keys()):
self.stop_output_profile(channel_id, pid)
def stop_channel(self, channel_id):
"""Stop a channel with proper ownership handling"""
try:
@ -938,6 +1255,12 @@ class ProxyServer:
except RuntimeError:
logger.debug(f"Could not join stream thread (may be current thread)")
# Stop all output format managers before releasing ownership
self.stop_all_output_formats(channel_id)
# Stop all output profile transcodes before releasing ownership
self.stop_all_output_profiles(channel_id)
# Release ownership
self.release_ownership(channel_id)
logger.info(f"Released ownership of channel {channel_id}")
@ -1051,7 +1374,7 @@ class ProxyServer:
try:
# Send worker heartbeat first
if self.redis_client:
worker_heartbeat_key = f"ts_proxy:worker:{self.worker_id}:heartbeat"
worker_heartbeat_key = f"live:worker:{self.worker_id}:heartbeat"
self._execute_redis_command(
lambda: self.redis_client.setex(worker_heartbeat_key, 30, str(time.time()))
)
@ -1223,7 +1546,7 @@ class ProxyServer:
else:
# There are clients or we're still connecting - clear any disconnect timestamp
if self.redis_client:
self.redis_client.delete(f"ts_proxy:channel:{channel_id}:last_client_disconnect_time")
self.redis_client.delete(f"live:channel:{channel_id}:last_client_disconnect_time")
else:
# === NON-OWNER CHANNEL HANDLING ===
@ -1302,7 +1625,7 @@ class ProxyServer:
else:
self._last_orphan_check = time.time()
gevent.sleep(ConfigHelper.cleanup_check_interval()) # REPLACE: time.sleep(ConfigHelper.cleanup_check_interval())
gevent.sleep(ConfigHelper.cleanup_check_interval())
thread = threading.Thread(target=cleanup_task, daemon=True)
thread.name = "ts-proxy-cleanup"
@ -1316,7 +1639,7 @@ class ProxyServer:
try:
# Get all active channel keys
channel_pattern = "ts_proxy:channel:*:metadata"
channel_pattern = "live:channel:*:metadata"
channel_keys = self.redis_client.keys(channel_pattern)
for key in channel_keys:
@ -1361,7 +1684,7 @@ class ProxyServer:
try:
# Get all channel metadata keys
channel_pattern = "ts_proxy:channel:*:metadata"
channel_pattern = "live:channel:*:metadata"
channel_keys = self.redis_client.keys(channel_pattern)
for key in channel_keys:
@ -1386,7 +1709,7 @@ class ProxyServer:
# Check if owner is still alive
owner_alive = False
if owner:
owner_heartbeat_key = f"ts_proxy:worker:{owner}:heartbeat"
owner_heartbeat_key = f"live:worker:{owner}:heartbeat"
owner_alive = self.redis_client.exists(owner_heartbeat_key)
# Check client count
@ -1458,7 +1781,7 @@ class ProxyServer:
try:
# Define key patterns to scan for
patterns = [
f"ts_proxy:channel:{channel_id}:*", # All channel keys
f"live:channel:{channel_id}:*", # All channel keys
RedisKeys.events_channel(channel_id) # Event channel
]

1
apps/proxy/live_proxy/services/__init__.py Normal file
View file

@ -0,0 +1 @@

9
apps/proxy/ts_proxy/services/channel_service.py → apps/proxy/live_proxy/services/channel_service.py
View file

@ -6,10 +6,7 @@ This separates business logic from HTTP handling in views.
import logging
import time
import json
import re
from django.shortcuts import get_object_or_404
from apps.channels.models import Channel, Stream
from apps.proxy.config import TSConfig as Config
from ..server import ProxyServer
from ..redis_keys import RedisKeys
from ..constants import EventType, ChannelState, ChannelMetadataField
@ -17,7 +14,7 @@ from ..url_utils import get_stream_info_for_switch
from core.utils import log_system_event
from .log_parsers import LogParserFactory
logger = logging.getLogger("ts_proxy")
logger = logging.getLogger("live_proxy")
class ChannelService:
"""Service class for channel operations"""
@ -42,8 +39,6 @@ class ChannelService:
bool: Success status
"""
proxy_server = ProxyServer.get_instance()
# FIXED: First, ensure that Redis metadata including stream_id is set BEFORE channel initialization
# This ensures the stream ID is available when the StreamManager looks it up
if stream_id and proxy_server.redis_client:
metadata_key = RedisKeys.channel_metadata(channel_id)
# Check if metadata already exists
@ -151,7 +146,7 @@ class ChannelService:
if proxy_server.redis_client:
try:
# This is inefficient but used for diagnostics - in production would use more targeted checks
redis_keys = proxy_server.redis_client.keys(f"ts_proxy:*:{channel_id}*")
redis_keys = proxy_server.redis_client.keys(f"live:*:{channel_id}*")
redis_keys = [k for k in redis_keys] if redis_keys else []
except Exception as e:
logger.error(f"Error checking Redis keys: {e}")

0
apps/proxy/ts_proxy/services/log_parsers.py → apps/proxy/live_proxy/services/log_parsers.py
View file

0
apps/proxy/ts_proxy/url_utils.py → apps/proxy/live_proxy/url_utils.py
View file

2
apps/proxy/ts_proxy/urls.py → apps/proxy/live_proxy/urls.py
View file

@ -1,7 +1,7 @@
from django.urls import path
from . import views
app_name = 'ts_proxy'
app_name = 'live_proxy'
urlpatterns = [
path('stream/<str:channel_id>', views.stream_ts, name='stream'),

10
apps/proxy/ts_proxy/utils.py → apps/proxy/live_proxy/utils.py
View file

@ -3,7 +3,7 @@ import re
from urllib.parse import urlparse
import inspect
logger = logging.getLogger("ts_proxy")
logger = logging.getLogger("live_proxy")
def detect_stream_type(url):
"""
@ -90,7 +90,7 @@ def create_ts_packet(packet_type='null', message=None):
def get_logger(component_name=None):
"""
Get a standardized logger with ts_proxy prefix and optional component name.
Get a standardized logger with live_proxy prefix and optional component name.
Args:
component_name (str, optional): Name of the component. If not provided,
@ -100,7 +100,7 @@ def get_logger(component_name=None):
logging.Logger: A configured logger with standardized naming.
"""
if component_name:
logger_name = f"ts_proxy.{component_name}"
logger_name = f"live_proxy.{component_name}"
else:
# Try to get the calling module name if not explicitly specified
frame = inspect.currentframe().f_back
@ -108,9 +108,9 @@ def get_logger(component_name=None):
if module:
# Extract just the filename without extension
module_name = module.__name__.split('.')[-1]
logger_name = f"ts_proxy.{module_name}"
logger_name = f"live_proxy.{module_name}"
else:
# Default if detection fails
logger_name = "ts_proxy"
logger_name = "live_proxy"
return logging.getLogger(logger_name)

114
apps/proxy/ts_proxy/views.py → apps/proxy/live_proxy/views.py
View file

@ -1,5 +1,4 @@
import json
import threading
import time
import random
import re
@ -7,13 +6,12 @@ import pathlib
from django.http import StreamingHttpResponse, JsonResponse, HttpResponseRedirect, HttpResponse
from django.views.decorators.csrf import csrf_exempt
from django.shortcuts import get_object_or_404
from apps.proxy.config import TSConfig as Config
from .server import ProxyServer
from .channel_status import ChannelStatus
from .stream_generator import create_stream_generator
from .output.ts.generator import create_stream_generator
from .output.fmp4.generator import create_fmp4_stream_generator
from .utils import get_client_ip
from .redis_keys import RedisKeys
import logging
from apps.channels.models import Channel, Stream
from apps.m3u.models import M3UAccount, M3UAccountProfile
from apps.accounts.models import User
@ -46,9 +44,51 @@ from apps.proxy.utils import check_user_stream_limits
logger = get_logger()
def _resolve_output_format(user, force=None, request=None):
"""Return the output format string to use for this client."""
_FORMAT_ALIASES = {
'mpegts': 'mpegts',
'ts': 'mpegts',
'fmp4': 'fmp4',
'mp4': 'fmp4',
}
if force:
return force
if request:
# Support both ?output_format= (native) and ?output= (XC-style)
param = request.GET.get('output_format') or request.GET.get('output')
if param in _FORMAT_ALIASES:
return _FORMAT_ALIASES[param]
if user:
custom = getattr(user, 'custom_properties', None) or {}
user_format = custom.get('output_format')
if user_format:
return user_format
return CoreSettings.get_default_output_format()
def _resolve_output_profile(request, user):
from core.models import OutputProfile
param = request.GET.get('output_profile')
if param:
try:
return OutputProfile.objects.get(id=int(param), is_active=True)
except (OutputProfile.DoesNotExist, ValueError, TypeError):
return None
if user:
custom = getattr(user, 'custom_properties', None) or {}
profile_id = custom.get('output_profile')
if profile_id:
try:
return OutputProfile.objects.get(id=int(profile_id), is_active=True)
except (OutputProfile.DoesNotExist, ValueError, TypeError):
return None
return None
@api_view(["GET"])
@permission_classes([AllowAny])
def stream_ts(request, channel_id, user=None):
def stream_ts(request, channel_id, user=None, force_output_format=None):
if not network_access_allowed(request, "STREAMS"):
return JsonResponse({"error": "Forbidden"}, status=403)
@ -135,7 +175,7 @@ def stream_ts(request, channel_id, user=None):
owner_field = ChannelMetadataField.OWNER
if owner_field in metadata:
owner = metadata[owner_field]
owner_heartbeat_key = f"ts_proxy:worker:{owner}:heartbeat"
owner_heartbeat_key = f"live:worker:{owner}:heartbeat"
if proxy_server.redis_client.exists(owner_heartbeat_key):
# Owner is still active with unknown state - don't reinitialize
needs_initialization = False
@ -455,9 +495,7 @@ def stream_ts(request, channel_id, user=None):
{"error": "Failed to connect"}, status=502
)
gevent.sleep(
0.1
) # FIXED: Using gevent.sleep instead of time.sleep
gevent.sleep(0.1)
logger.info(f"[{client_id}] Successfully initialized channel {channel_id}")
channel_initializing = True
@ -528,19 +566,53 @@ def stream_ts(request, channel_id, user=None):
)
# Register client
buffer = proxy_server.stream_buffers[channel_id]
client_manager = proxy_server.client_managers[channel_id]
client_manager.add_client(client_id, client_ip, client_user_agent, user)
logger.info(f"[{client_id}] Client registered with channel {channel_id}")
output_profile = _resolve_output_profile(request, user)
if output_profile:
cmd = output_profile.build_command()
if not proxy_server.ensure_output_profile(channel_id, output_profile.id, cmd):
return JsonResponse(
{"error": "Failed to start output profile transcode"}, status=500
)
# Create a stream generator for this client
generate = create_stream_generator(
channel_id, client_id, client_ip, client_user_agent, channel_initializing, user=user
source_buffer = proxy_server.get_buffer(
channel_id,
profile=output_profile.id if output_profile else None
)
client_manager = proxy_server.client_managers[channel_id]
output_format = _resolve_output_format(user, force_output_format, request)
# When an output profile is active, append :p{id} to the format key so each
# (format, profile) pair gets its own independent remux pipeline in Redis.
resolved_format = f'{output_format}:p{output_profile.id}' if output_profile else output_format
client_manager.add_client(
client_id, client_ip, client_user_agent, user,
output_format=output_format,
output_profile_id=output_profile.id if output_profile else None,
)
logger.info(
f"[{client_id}] Client registered with channel {channel_id} "
f"(output: {resolved_format}, profile: {output_profile.id if output_profile else None})"
)
if output_format == 'fmp4':
proxy_server.ensure_output_format(
channel_id, resolved_format,
source_buffer=source_buffer if output_profile else None,
)
generate = create_fmp4_stream_generator(
channel_id, client_id, client_ip, client_user_agent, channel_initializing, user=user,
fmt=resolved_format,
)
content_type = "video/mp4"
else:
generate = create_stream_generator(
channel_id, client_id, client_ip, client_user_agent, channel_initializing, user=user, buffer=source_buffer
)
content_type = "video/mp2t"
# Return the StreamingHttpResponse from the main function
response = StreamingHttpResponse(
streaming_content=generate(), content_type="video/mp2t"
streaming_content=generate(), content_type=content_type
)
response["Cache-Control"] = "no-cache"
return response
@ -601,8 +673,8 @@ def stream_xc(request, username, password, channel_id):
else:
channel = get_object_or_404(Channel, id=channel_id)
# @TODO: we've got the file 'type' via extension, support this when we support multiple outputs
return stream_ts(request._request, str(channel.uuid), user)
force_format = 'fmp4' if extension.lower() == '.mp4' else 'mpegts'
return stream_ts(request._request, str(channel.uuid), user, force_output_format=force_format)
@csrf_exempt
@ -719,7 +791,7 @@ def channel_status(request, channel_id=None):
)
else:
# Basic info for all channels
channel_pattern = "ts_proxy:channel:*:metadata"
channel_pattern = "live:channel:*:metadata"
all_channels = []
# Extract channel IDs from keys
@ -730,7 +802,7 @@ def channel_status(request, channel_id=None):
)
for key in keys:
channel_id_match = re.search(
r"ts_proxy:channel:(.*):metadata", key
r"live:channel:(.*):metadata", key
)
if channel_id_match:
ch_id = channel_id_match.group(1)

6
apps/proxy/management/commands/proxy.py
View file

@ -42,7 +42,7 @@ class Command(BaseCommand):
if proxy_type in ('hls', 'all'):
proxy_app.hls_proxy.initialize_channel(url, channel or 'default')
if proxy_type in ('ts', 'all'):
proxy_app.ts_proxy.initialize_channel(url, channel or 'default')
proxy_app.live_proxy.initialize_channel(url, channel or 'default')
self.stdout.write(self.style.SUCCESS('Started proxy servers'))
elif action == 'stop':
@ -53,9 +53,9 @@ class Command(BaseCommand):
proxy_app.hls_proxy.shutdown()
if proxy_type in ('ts', 'all'):
if channel:
proxy_app.ts_proxy.stop_channel(channel)
proxy_app.live_proxy.stop_channel(channel)
else:
proxy_app.ts_proxy.shutdown()
proxy_app.live_proxy.shutdown()
self.stdout.write(self.style.SUCCESS('Stopped proxy servers'))
elif action == 'restart':

4
apps/proxy/signals.py
View file

@ -12,8 +12,8 @@ def cleanup_proxy_servers(sender, **kwargs):
proxy_app = apps.get_app_config('proxy')
for channel_id in list(proxy_app.hls_proxy.stream_managers.keys()):
proxy_app.hls_proxy.stop_channel(channel_id)
for channel_id in list(proxy_app.ts_proxy.stream_managers.keys()):
proxy_app.ts_proxy.stop_channel(channel_id)
for channel_id in list(proxy_app.live_proxy.stream_managers.keys()):
proxy_app.live_proxy.stop_channel(channel_id)
logger.info("Proxy servers cleaned up successfully")
except Exception as e:
logger.error(f"Error during proxy server cleanup: {e}")

6
apps/proxy/tasks.py
View file

@ -4,7 +4,7 @@ import logging
import re
import gc
from core.utils import RedisClient
from apps.proxy.ts_proxy.channel_status import ChannelStatus
from apps.proxy.live_proxy.channel_status import ChannelStatus
from core.utils import send_websocket_update
logger = logging.getLogger(__name__)
@ -18,7 +18,7 @@ def fetch_channel_stats():
try:
# Basic info for all channels
channel_pattern = "ts_proxy:channel:*:metadata"
channel_pattern = "live:channel:*:metadata"
all_channels = []
# Extract channel IDs from keys
@ -26,7 +26,7 @@ def fetch_channel_stats():
while True:
cursor, keys = redis_client.scan(cursor, match=channel_pattern)
for key in keys:
channel_id_match = re.search(r"ts_proxy:channel:(.*):metadata", key)
channel_id_match = re.search(r"live:channel:(.*):metadata", key)
if channel_id_match:
ch_id = channel_id_match.group(1)
channel_info = ChannelStatus.get_basic_channel_info(ch_id)

96
apps/proxy/ts_proxy/redis_keys.py
View file

@ -1,96 +0,0 @@
"""
Defines Redis key patterns used throughout the TS proxy service.
Centralizing these key patterns makes it easier to maintain and change them if needed.
"""
class RedisKeys:
@staticmethod
def channel_metadata(channel_id):
"""Key for channel metadata hash"""
return f"ts_proxy:channel:{channel_id}:metadata"
@staticmethod
def buffer_index(channel_id):
"""Key for tracking buffer index"""
return f"ts_proxy:channel:{channel_id}:buffer:index"
@staticmethod
def buffer_chunk(channel_id, chunk_index):
"""Key for specific buffer chunk"""
return f"ts_proxy:channel:{channel_id}:buffer:chunk:{chunk_index}"
@staticmethod
def buffer_chunk_prefix(channel_id):
"""Prefix for buffer chunks"""
return f"ts_proxy:channel:{channel_id}:buffer:chunk:"
@staticmethod
def channel_stopping(channel_id):
"""Key indicating channel is stopping"""
return f"ts_proxy:channel:{channel_id}:stopping"
@staticmethod
def client_stop(channel_id, client_id):
"""Key requesting client stop"""
return f"ts_proxy:channel:{channel_id}:client:{client_id}:stop"
@staticmethod
def events_channel(channel_id):
"""PubSub channel for events"""
return f"ts_proxy:events:{channel_id}"
@staticmethod
def switch_request(channel_id):
"""Key for stream switch request"""
return f"ts_proxy:channel:{channel_id}:switch_request"
@staticmethod
def channel_owner(channel_id):
"""Key for storing channel owner worker ID"""
return f"ts_proxy:channel:{channel_id}:owner"
@staticmethod
def clients(channel_id):
"""Key for set of client IDs"""
return f"ts_proxy:channel:{channel_id}:clients"
@staticmethod
def last_client_disconnect(channel_id):
"""Key for last client disconnect timestamp"""
return f"ts_proxy:channel:{channel_id}:last_client_disconnect_time"
@staticmethod
def connection_attempt(channel_id):
"""Key for connection attempt timestamp"""
return f"ts_proxy:channel:{channel_id}:connection_attempt_time"
@staticmethod
def last_data(channel_id):
"""Key for last data timestamp"""
return f"ts_proxy:channel:{channel_id}:last_data"
@staticmethod
def switch_status(channel_id):
"""Key for stream switch status"""
return f"ts_proxy:channel:{channel_id}:switch_status"
@staticmethod
def worker_heartbeat(worker_id):
"""Key for worker heartbeat"""
return f"ts_proxy:worker:{worker_id}:heartbeat"
@staticmethod
def chunk_timestamps(channel_id):
"""Sorted set mapping chunk receive-timestamps (score) to chunk indices (member).
Used for time-based client positioning."""
return f"ts_proxy:channel:{channel_id}:buffer:chunk_timestamps"
@staticmethod
def transcode_active(channel_id):
"""Key indicating active transcode process"""
return f"ts_proxy:channel:{channel_id}:transcode_active"
@staticmethod
def client_metadata(channel_id, client_id):
"""Key for client metadata hash"""
return f"ts_proxy:channel:{channel_id}:clients:{client_id}"

2
apps/proxy/urls.py
View file

@ -3,7 +3,7 @@ from django.urls import path, include
app_name = 'proxy'
urlpatterns = [
path('ts/', include('apps.proxy.ts_proxy.urls')),
path('ts/', include('apps.proxy.live_proxy.urls')),
path('hls/', include('apps.proxy.hls_proxy.urls')),
path('vod/', include('apps.proxy.vod_proxy.urls')),
]

4
apps/proxy/utils.py
View file

@ -2,7 +2,7 @@ import logging
from core.utils import RedisClient
from apps.proxy.vod_proxy.multi_worker_connection_manager import MultiWorkerVODConnectionManager, get_vod_client_stop_key
from core.models import CoreSettings
from apps.proxy.ts_proxy.services.channel_service import ChannelService
from apps.proxy.live_proxy.services.channel_service import ChannelService
logger = logging.getLogger("proxy")
@ -92,7 +92,7 @@ def get_user_active_connections(user_id):
try:
# Grab live streams
for key in redis_client.scan_iter(match="ts_proxy:channel:*:clients:*", count=1000):
for key in redis_client.scan_iter(match="live:channel:*:clients:*", count=1000):
parts = key.split(':')
if len(parts) >= 5:
channel_id = parts[2]

2
core/api_urls.py
View file

@ -5,6 +5,7 @@ from rest_framework.routers import DefaultRouter
from .api_views import (
UserAgentViewSet,
StreamProfileViewSet,
OutputProfileViewSet,
CoreSettingsViewSet,
SystemNotificationViewSet,
environment,
@ -17,6 +18,7 @@ from .api_views import (
router = DefaultRouter()
router.register(r'useragents', UserAgentViewSet, basename='useragent')
router.register(r'streamprofiles', StreamProfileViewSet, basename='streamprofile')
router.register(r'outputprofiles', OutputProfileViewSet, basename='outputprofile')
router.register(r'settings', CoreSettingsViewSet, basename='coresettings')
router.register(r'notifications', SystemNotificationViewSet, basename='systemnotification')
urlpatterns = [

17
core/api_views.py
View file

@ -16,6 +16,7 @@ from drf_spectacular.types import OpenApiTypes
from .models import (
UserAgent,
StreamProfile,
OutputProfile,
CoreSettings,
STREAM_SETTINGS_KEY,
DVR_SETTINGS_KEY,
@ -25,6 +26,7 @@ from .models import (
from .serializers import (
UserAgentSerializer,
StreamProfileSerializer,
OutputProfileSerializer,
CoreSettingsSerializer,
ProxySettingsSerializer,
)
@ -75,6 +77,21 @@ class StreamProfileViewSet(viewsets.ModelViewSet):
return [Authenticated()]
class OutputProfileViewSet(viewsets.ModelViewSet):
"""
API endpoint that allows output profiles to be viewed, created, edited, or deleted.
"""
queryset = OutputProfile.objects.all()
serializer_class = OutputProfileSerializer
def get_permissions(self):
try:
return [perm() for perm in permission_classes_by_action[self.action]]
except KeyError:
return [Authenticated()]
class CoreSettingsViewSet(viewsets.ModelViewSet):
"""
API endpoint for editing core settings.

78
core/migrations/0024_outputprofile.py Normal file
View file

@ -0,0 +1,78 @@
# Generated by Django 6.0.4 on 2026-05-08 13:51
from django.db import migrations, models
def create_default_output_profiles(apps, schema_editor):
OutputProfile = apps.get_model('core', 'OutputProfile')
OutputProfile.objects.get_or_create(
name='Media Server (AC3 Audio)',
defaults={
'command': 'ffmpeg',
'parameters': (
'-fflags +discardcorrupt+genpts+nobuffer '
'-i pipe:0 '
'-map 0 '
'-c:v copy '
'-c:a ac3 '
'-b:a 384k '
'-max_muxing_queue_size 4096 '
'-mpegts_flags +pat_pmt_at_frames+resend_headers+initial_discontinuity '
'-f mpegts pipe:1'
),
'locked': True,
'is_active': True,
},
)
OutputProfile.objects.get_or_create(
name='Web Player (AAC Audio)',
defaults={
'command': 'ffmpeg',
'parameters': (
'-fflags +discardcorrupt+genpts+nobuffer '
'-i pipe:0 '
'-map 0 '
'-c:v copy '
'-c:a aac '
'-b:a 192k '
'-max_muxing_queue_size 4096 '
'-mpegts_flags +pat_pmt_at_frames+resend_headers+initial_discontinuity '
'-f mpegts pipe:1'
),
'locked': True,
'is_active': True,
},
)
def remove_default_output_profiles(apps, schema_editor):
OutputProfile = apps.get_model('core', 'OutputProfile')
OutputProfile.objects.filter(
name__in=['Media Server (AC3 Audio)', 'Web Player (AAC Audio)'],
locked=True,
).delete()
class Migration(migrations.Migration):
dependencies = [
('core', '0023_alter_systemevent_event_type'),
]
operations = [
migrations.CreateModel(
name='OutputProfile',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(help_text='Display name for this output profile', max_length=255, unique=True)),
('command', models.CharField(help_text="Executable to run (e.g. 'ffmpeg')", max_length=255)),
('parameters', models.TextField(help_text='Command-line parameters. Must read from pipe:0 (stdin) and write to pipe:1 (stdout).')),
('locked', models.BooleanField(default=False, help_text="Protected - can't be deleted or modified")),
('is_active', models.BooleanField(default=True, help_text='Whether this profile is available for use')),
],
),
migrations.RunPython(
create_default_output_profiles,
remove_default_output_profiles,
),
]

43
core/models.py
View file

@ -125,6 +125,7 @@ class StreamProfile(models.Model):
return False
def build_command(self, stream_url, user_agent):
if self.is_proxy():
return []
@ -148,6 +149,43 @@ class StreamProfile(models.Model):
return part
class OutputProfile(models.Model):
"""
Defines a pre-delivery transcode step applied to a channel's TS stream.
The command and parameters must accept raw MPEG-TS via pipe:0 (stdin) and
write the transcoded output to pipe:1 (stdout). One transcode process runs
per active (channel, OutputProfile) pair regardless of how many clients
request it; all clients share the same Redis-backed output buffer.
Example parameters for a 720p transcode:
-i pipe:0 -c:v libx264 -b:v 2000k -vf scale=-2:720 -c:a copy -f mpegts pipe:1
"""
name = models.CharField(max_length=255, unique=True, help_text="Display name for this output profile")
command = models.CharField(
max_length=255,
help_text="Executable to run (e.g. 'ffmpeg')",
)
parameters = models.TextField(
help_text="Command-line parameters. Must read from pipe:0 (stdin) and write to pipe:1 (stdout).",
)
locked = models.BooleanField(
default=False, help_text="Protected - can't be deleted or modified"
)
is_active = models.BooleanField(
default=True, help_text="Whether this profile is available for use"
)
def __str__(self):
return self.name
def build_command(self):
"""Return the full command as a list suitable for subprocess.Popen."""
from shlex import split as shlex_split
return [self.command] + shlex_split(self.parameters)
# Setting group keys
STREAM_SETTINGS_KEY = "stream_settings"
DVR_SETTINGS_KEY = "dvr_settings"
@ -207,6 +245,7 @@ class CoreSettings(models.Model):
"m3u_hash_key": "",
"preferred_region": None,
"auto_import_mapped_files": None,
"default_output_format": "mpegts",
})
@classmethod
@ -217,6 +256,10 @@ class CoreSettings(models.Model):
def get_default_stream_profile_id(cls):
return cls.get_stream_settings().get("default_stream_profile")
@classmethod
def get_default_output_format(cls):
return cls.get_stream_settings().get("default_output_format", "mpegts")
@classmethod
def get_m3u_hash_key(cls):
return cls.get_stream_settings().get("m3u_hash_key", "")

8
core/serializers.py
View file

@ -3,7 +3,7 @@ import json
import ipaddress
from rest_framework import serializers
from .models import CoreSettings, UserAgent, StreamProfile, DVR_SETTINGS_KEY, NETWORK_ACCESS_KEY
from .models import CoreSettings, UserAgent, StreamProfile, OutputProfile, DVR_SETTINGS_KEY, NETWORK_ACCESS_KEY
class UserAgentSerializer(serializers.ModelSerializer):
@ -34,6 +34,12 @@ class StreamProfileSerializer(serializers.ModelSerializer):
]
class OutputProfileSerializer(serializers.ModelSerializer):
class Meta:
model = OutputProfile
fields = ["id", "name", "command", "parameters", "is_active", "locked"]
class CoreSettingsSerializer(serializers.ModelSerializer):
class Meta:
model = CoreSettings

6
core/tasks.py
View file

@ -7,7 +7,7 @@ import re
import time
import os
from core.utils import RedisClient, send_websocket_update, acquire_task_lock, release_task_lock
from apps.proxy.ts_proxy.channel_status import ChannelStatus
from apps.proxy.live_proxy.channel_status import ChannelStatus
from apps.m3u.models import M3UAccount
from apps.epg.models import EPGSource
from apps.m3u.tasks import refresh_single_m3u_account
@ -396,7 +396,7 @@ def fetch_channel_stats():
try:
# Basic info for all channels
channel_pattern = "ts_proxy:channel:*:metadata"
channel_pattern = "live:channel:*:metadata"
all_channels = []
# Extract channel IDs from keys
@ -404,7 +404,7 @@ def fetch_channel_stats():
while True:
cursor, keys = redis_client.scan(cursor, match=channel_pattern)
for key in keys:
channel_id_match = re.search(r"ts_proxy:channel:(.*):metadata", key)
channel_id_match = re.search(r"live:channel:(.*):metadata", key)
if channel_id_match:
ch_id = channel_id_match.group(1)
channel_info = ChannelStatus.get_basic_channel_info(ch_id)

2
dispatcharr/consumers.py
View file

@ -46,7 +46,7 @@ class MyWebSocketConsumer(AsyncWebsocketConsumer):
data = json.loads(text_data)
if data["type"] == "m3u_profile_test":
from apps.proxy.ts_proxy.url_utils import transform_url
from apps.proxy.live_proxy.url_utils import transform_url
def replace_with_mark(match):
# Wrap the match in <mark> tags

2
dispatcharr/settings.py
View file

@ -79,7 +79,7 @@ INSTALLED_APPS = [
"apps.m3u",
"apps.output",
"apps.proxy.apps.ProxyConfig",
"apps.proxy.ts_proxy",
"apps.proxy.live_proxy",
"apps.vod.apps.VODConfig",
"apps.connect.apps.ConnectConfig",
"core",

2
dispatcharr/urls.py
View file

@ -5,7 +5,7 @@ from django.conf.urls.static import static
from django.views.generic import TemplateView, RedirectView
from .routing import websocket_urlpatterns
from apps.output.views import xc_player_api, xc_panel_api, xc_get, xc_xmltv
from apps.proxy.ts_proxy.views import stream_xc
from apps.proxy.live_proxy.views import stream_xc
from apps.proxy.vod_proxy.views import stream_xc_movie, stream_xc_episode
urlpatterns = [

48
frontend/src/api.js
View file

@ -7,6 +7,7 @@ import usePlaylistsStore from './store/playlists';
import useEPGsStore from './store/epgs';
import useStreamsStore from './store/streams';
import useStreamProfilesStore from './store/streamProfiles';
import useOutputProfilesStore from './store/outputProfiles';
import useSettingsStore from './store/settings';
import { notifications } from '@mantine/notifications';
import useChannelsTableStore from './store/channelsTable';
@ -1719,6 +1720,53 @@ export default class API {
}
}
static async getOutputProfiles() {
try {
const response = await request(`${host}/api/core/outputprofiles/`);
return response;
} catch (e) {
errorNotification('Failed to retrieve output profiles', e);
}
}
static async addOutputProfile(values) {
try {
const response = await request(`${host}/api/core/outputprofiles/`, {
method: 'POST',
body: values,
});
useOutputProfilesStore.getState().addOutputProfile(response);
return response;
} catch (e) {
errorNotification('Failed to create output profile', e);
}
}
static async updateOutputProfile(values) {
const { id, ...payload } = values;
try {
const response = await request(`${host}/api/core/outputprofiles/${id}/`, {
method: 'PUT',
body: payload,
});
useOutputProfilesStore.getState().updateOutputProfile(response);
return response;
} catch (e) {
errorNotification(`Failed to update output profile ${id}`, e);
}
}
static async deleteOutputProfile(id) {
try {
await request(`${host}/api/core/outputprofiles/${id}/`, {
method: 'DELETE',
});
useOutputProfilesStore.getState().removeOutputProfiles([id]);
} catch (e) {
errorNotification(`Failed to delete output profile ${id}`, e);
}
}
static async getGrid() {
try {
const response = await request(`${host}/api/epg/grid/`);

32
frontend/src/components/cards/StreamConnectionCard.jsx
View file

@ -3,6 +3,7 @@ import React, { useEffect, useMemo, useState } from 'react';
import usePlaylistsStore from '../../store/playlists.jsx';
import useSettingsStore from '../../store/settings.jsx';
import useUsersStore from '../../store/users.jsx';
import useOutputProfilesStore from '../../store/outputProfiles.jsx';
import {
ActionIcon,
Badge,
@ -54,6 +55,7 @@ import {
switchStream,
} from '../../utils/cards/StreamConnectionCardUtils.js';
import useVideoStore from '../../store/useVideoStore';
import { buildLiveStreamUrl } from '../../utils/components/FloatingVideoUtils.js';
const formatProgramTime = (seconds) => {
const absSeconds = Math.abs(seconds);
@ -454,9 +456,19 @@ const StreamConnectionCard = ({
actions: renderBodyCell,
},
getExpandedRowHeight: (row) => {
return 20 + 28 * row.original.streams.length;
return (
20 +
28 * row.original.streams.length +
(row.original.output_format ? 28 : 0) +
(row.original.output_profile_id ? 28 : 0)
);
},
expandedRowRenderer: ({ row }) => {
const outputProfileId = row.original.output_profile_id;
const outputProfiles = useOutputProfilesStore.getState().profiles;
const outputProfileName = outputProfileId
? (outputProfiles.find((p) => p.id === outputProfileId)?.name ?? null)
: null;
return (
<Box p="xs">
<Group spacing="xs" align="flex-start">
@ -465,6 +477,22 @@ const StreamConnectionCard = ({
</Text>
<Text size="xs">{row.original.user_agent || 'Unknown'}</Text>
</Group>
{row.original.output_format && (
<Group spacing="xs" align="flex-start" mt={4}>
<Text size="xs" fw={500} color="dimmed">
Container:
</Text>
<Text size="xs">{row.original.output_format}</Text>
</Group>
)}
{outputProfileName && (
<Group spacing="xs" align="flex-start" mt={4}>
<Text size="xs" fw={500} color="dimmed">
Output Profile:
</Text>
<Text size="xs">{outputProfileName}</Text>
</Group>
)}
</Box>
);
},
@ -531,7 +559,7 @@ const StreamConnectionCard = ({
const actualChannel = channels[channelDbId];
if (!actualChannel?.uuid) return;
const uri = `/proxy/ts/stream/${actualChannel.uuid}`;
const uri = buildLiveStreamUrl(`/proxy/ts/stream/${actualChannel.uuid}`);
let url = `${window.location.protocol}//${window.location.host}${uri}`;
if (env_mode === 'dev') {
url = `${window.location.protocol}//${window.location.hostname}:5656${uri}`;

173
frontend/src/components/forms/OutputProfile.jsx Normal file
View file

@ -0,0 +1,173 @@
import React, { useEffect, useMemo, useState } from 'react';
import { useForm } from 'react-hook-form';
import { yupResolver } from '@hookform/resolvers/yup';
import * as Yup from 'yup';
import API from '../../api';
import {
Modal,
TextInput,
Textarea,
Select,
Button,
Flex,
Stack,
Checkbox,
} from '@mantine/core';
const BUILT_IN_COMMANDS = [
{ value: 'ffmpeg', label: 'FFmpeg' },
{ value: '__custom__', label: 'Custom…' },
];
const COMMAND_EXAMPLES = {
ffmpeg:
'-i pipe:0 -c:v libx264 -b:v 2000k -vf scale=-2:720 -c:a copy -f mpegts pipe:1',
};
const toCommandSelection = (command) =>
BUILT_IN_COMMANDS.find((o) => o.value === command && o.value !== '__custom__')
? command
: '__custom__';
const schema = Yup.object({
name: Yup.string().required('Name is required'),
command: Yup.string().required('Command is required'),
parameters: Yup.string(),
});
const OutputProfile = ({ profile = null, isOpen, onClose }) => {
const [commandSelection, setCommandSelection] = useState('ffmpeg');
const defaultValues = useMemo(
() => ({
name: profile?.name || '',
command: profile?.command || 'ffmpeg',
parameters: profile?.parameters || '',
is_active: profile?.is_active ?? true,
}),
[profile]
);
const {
register,
handleSubmit,
formState: { errors, isSubmitting },
reset,
setValue,
watch,
} = useForm({
defaultValues,
resolver: yupResolver(schema),
});
useEffect(() => {
reset(defaultValues);
setCommandSelection(toCommandSelection(profile?.command || 'ffmpeg'));
}, [defaultValues, reset, profile]);
const onSubmit = async (values) => {
if (profile?.id) {
await API.updateOutputProfile({ id: profile.id, ...values });
} else {
await API.addOutputProfile(values);
}
reset();
onClose();
};
if (!isOpen) return <></>;
const isLocked = profile ? profile.locked : false;
const isCustom = commandSelection === '__custom__';
const isActiveValue = watch('is_active');
return (
<Modal opened={isOpen} onClose={onClose} title="Output Profile">
<form onSubmit={handleSubmit(onSubmit)}>
<Stack gap="sm">
<TextInput
label="Name"
description="A unique, descriptive label for this output profile"
disabled={isLocked}
{...register('name')}
error={errors.name?.message}
/>
<Select
label="Command"
description="The executable used to transcode the stream. Must accept stdin (pipe:0) and write to stdout (pipe:1)."
data={BUILT_IN_COMMANDS}
disabled={isLocked}
value={commandSelection}
onChange={(val) => {
setCommandSelection(val);
if (val !== '__custom__') {
setValue('command', val, { shouldValidate: true });
} else {
setValue('command', '', { shouldValidate: false });
}
}}
error={isCustom ? undefined : errors.command?.message}
/>
{isCustom && (
<TextInput
label="Custom Command"
description="Enter the executable name or full path"
disabled={isLocked}
{...register('command')}
error={errors.command?.message}
/>
)}
<Textarea
label="Parameters"
description={
<>
Command-line arguments. Input is piped via{' '}
<strong>pipe:0</strong> (stdin); output must be written to{' '}
<strong>pipe:1</strong> (stdout). Output must be in{' '}
<strong>MPEG-TS</strong> format (<code>-f mpegts</code>).
{COMMAND_EXAMPLES[commandSelection] && (
<>
<br />
Example: <em>{COMMAND_EXAMPLES[commandSelection]}</em>
</>
)}
</>
}
autosize
minRows={2}
placeholder={
COMMAND_EXAMPLES[commandSelection] ||
'Enter command-line arguments…'
}
disabled={isLocked}
{...register('parameters')}
error={errors.parameters?.message}
/>
<Checkbox
label="Is Active"
description="Enable or disable this output profile"
checked={isActiveValue}
onChange={(e) => setValue('is_active', e.currentTarget.checked)}
/>
</Stack>
<Flex mih={50} gap="xs" justify="flex-end" align="flex-end">
<Button
type="submit"
variant="filled"
disabled={isSubmitting}
size="sm"
>
Save
</Button>
</Flex>
</form>
</Modal>
);
};
export default OutputProfile;

73
frontend/src/components/forms/User.jsx
View file

@ -21,7 +21,12 @@ import { RotateCcwKey, X } from 'lucide-react';
import { Copy, Key } from 'lucide-react';
import { useForm } from '@mantine/form';
import useChannelsStore from '../../store/channels';
import { USER_LEVELS, USER_LEVEL_LABELS, NETWORK_ACCESS_OPTIONS } from '../../constants';
import useOutputProfilesStore from '../../store/outputProfiles';
import {
USER_LEVELS,
USER_LEVEL_LABELS,
NETWORK_ACCESS_OPTIONS,
} from '../../constants';
import useAuthStore from '../../store/auth';
import { copyToClipboard } from '../../utils';
import { IPV4_CIDR_REGEX, IPV6_CIDR_REGEX } from '../../utils/networkUtils';
@ -36,6 +41,7 @@ const NETWORK_KEYS = Object.keys(NETWORK_ACCESS_OPTIONS);
const User = ({ user = null, isOpen, onClose }) => {
const profiles = useChannelsStore((s) => s.profiles);
const outputProfiles = useOutputProfilesStore((s) => s.profiles);
const authUser = useAuthStore((s) => s.user);
const setUser = useAuthStore((s) => s.setUser);
@ -58,6 +64,8 @@ const User = ({ user = null, isOpen, onClose }) => {
stream_limit: 0,
password: '',
xc_password: '',
output_format: '',
output_profile: '',
channel_profiles: [],
hide_adult_content: false,
epg_days: 0,
@ -80,7 +88,9 @@ const User = ({ user = null, isOpen, onClose }) => {
values.xc_password && !values.xc_password.match(/^[a-z0-9]+$/i)
? 'XC password must be alphanumeric'
: null,
allowed_ips: (values.allowed_ips || []).some((t) => !isValidNetworkEntry(t))
allowed_ips: (values.allowed_ips || []).some(
(t) => !isValidNetworkEntry(t)
)
? 'Invalid IP address or CIDR range'
: null,
}),
@ -107,6 +117,14 @@ const User = ({ user = null, isOpen, onClose }) => {
customProps.xc_password = values.xc_password || '';
delete values.xc_password;
customProps.output_format = values.output_format || null;
delete values.output_format;
customProps.output_profile = values.output_profile
? parseInt(values.output_profile, 10)
: null;
delete values.output_profile;
customProps.hide_adult_content = values.hide_adult_content || false;
delete values.hide_adult_content;
@ -121,7 +139,10 @@ const User = ({ user = null, isOpen, onClose }) => {
const joined = (values.allowed_ips || []).join(',');
delete values.allowed_ips;
const allowed_networks = {};
if (joined) NETWORK_KEYS.forEach((key) => { allowed_networks[key] = joined; });
if (joined)
NETWORK_KEYS.forEach((key) => {
allowed_networks[key] = joined;
});
customProps.allowed_networks = allowed_networks;
if (values.channel_profiles.includes('0')) {
@ -172,14 +193,20 @@ const User = ({ user = null, isOpen, onClose }) => {
? user.channel_profiles.map((id) => `${id}`)
: ['0'],
xc_password: customProps.xc_password || '',
output_format: customProps.output_format || '',
output_profile: customProps.output_profile
? `${customProps.output_profile}`
: '',
hide_adult_content: customProps.hide_adult_content || false,
epg_days: customProps.epg_days || 0,
epg_prev_days: customProps.epg_prev_days || 0,
allowed_ips: [...new Set(
NETWORK_KEYS.flatMap((key) =>
networks[key] ? networks[key].split(',').filter(Boolean) : []
)
)],
allowed_ips: [
...new Set(
NETWORK_KEYS.flatMap((key) =>
networks[key] ? networks[key].split(',').filter(Boolean) : []
)
),
],
});
if (customProps.xc_password) {
@ -404,6 +431,36 @@ const User = ({ user = null, isOpen, onClose }) => {
</ActionIcon>
}
/>
{isAdmin && (
<Select
label="Output Format Override"
description="Override the system default output format for this user. Clear to use system default."
clearable
placeholder="System default"
disabled={!isAdmin}
data={[
{ value: 'mpegts', label: 'MPEG-TS' },
{ value: 'fmp4', label: 'fMP4 (fragmented MP4)' },
]}
{...form.getInputProps('output_format')}
key={form.key('output_format')}
/>
)}
{isAdmin && (
<Select
label="Output Profile Override"
description="Pre-delivery transcode profile applied to streams for this user. Clear to use no transcoding."
clearable
searchable
placeholder="No transcoding"
disabled={!isAdmin}
data={outputProfiles
.filter((p) => p.is_active)
.map((p) => ({ value: `${p.id}`, label: p.name }))}
{...form.getInputProps('output_profile')}
key={form.key('output_profile')}
/>
)}
{isAdmin && (
<TagsInput
label="Allowed IPs"

10
frontend/src/components/forms/settings/StreamSettingsForm.jsx
View file

@ -195,6 +195,16 @@ const StreamSettingsForm = React.memo(({ active }) => {
value: `${r.value}`,
}))}
/>
<Select
{...form.getInputProps('default_output_format')}
id="default_output_format"
name="default_output_format"
label="Default Output Format"
data={[
{ value: 'mpegts', label: 'MPEG-TS' },
{ value: 'fmp4', label: 'fMP4 (fragmented MP4)' },
]}
/>
<Group justify="space-between" pt={5}>
<Text size="sm" fw={500}>

36
frontend/src/components/forms/settings/UiSettingsForm.jsx
View file

@ -1,17 +1,33 @@
import useSettingsStore from '../../../store/settings.jsx';
import useLocalStorage from '../../../hooks/useLocalStorage.jsx';
import useTablePreferences from '../../../hooks/useTablePreferences.jsx';
import useOutputProfilesStore from '../../../store/outputProfiles.jsx';
import {
getPlayerPrefs,
savePlayerPrefs,
} from '../../../utils/components/FloatingVideoUtils.js';
import {
buildTimeZoneOptions,
getDefaultTimeZone,
} from '../../../utils/dateTimeUtils.js';
import React, { useCallback, useEffect, useMemo, useRef } from 'react';
import React, {
useCallback,
useEffect,
useMemo,
useRef,
useState,
} from 'react';
import { showNotification } from '../../../utils/notificationUtils.js';
import { Select, Switch, Stack } from '@mantine/core';
import { saveTimeZoneSetting } from '../../../utils/forms/settings/UiSettingsFormUtils.js';
const UiSettingsForm = React.memo(() => {
const settings = useSettingsStore((s) => s.settings);
const outputProfiles = useOutputProfilesStore((s) => s.profiles);
const [webPlayerProfileId, setWebPlayerProfileId] = useState(
() => getPlayerPrefs().webPlayerOutputProfileId ?? null
);
const [timeFormat, setTimeFormat] = useLocalStorage('time-format', '12h');
const [dateFormat, setDateFormat] = useLocalStorage('date-format', 'mdy');
@ -81,6 +97,12 @@ const UiSettingsForm = React.memo(() => {
case 'header-pinned':
setHeaderPinned(value);
break;
case 'web-player-profile': {
const id = value ? Number(value) : null;
setWebPlayerProfileId(id);
savePlayerPrefs({ webPlayerOutputProfileId: id });
break;
}
}
};
@ -151,6 +173,18 @@ const UiSettingsForm = React.memo(() => {
onChange={(val) => onUISettingsChange('time-zone', val)}
data={timeZoneOptions}
/>
<Select
label="Web Player Output Profile"
description="Output profile applied when previewing streams in the browser player"
clearable
placeholder="None"
value={webPlayerProfileId ? String(webPlayerProfileId) : null}
onChange={(val) => onUISettingsChange('web-player-profile', val)}
data={outputProfiles.map((p) => ({
value: String(p.id),
label: p.name,
}))}
/>
</Stack>
);
});

3
frontend/src/components/tables/ChannelTableStreams.jsx
View file

@ -7,6 +7,7 @@ import React, {
} from 'react';
import API from '../../api';
import { copyToClipboard } from '../../utils';
import { buildLiveStreamUrl } from '../../utils/components/FloatingVideoUtils.js';
import {
GripHorizontal,
SquareMinus,
@ -508,7 +509,7 @@ const ChannelStreams = ({ channel }) => {
const handleWatchStream = useCallback(
(streamHash, streamName, streamId) => {
let vidUrl = `/proxy/ts/stream/${streamHash}`;
let vidUrl = buildLiveStreamUrl(`/proxy/ts/stream/${streamHash}`);
if (env_mode === 'dev') {
vidUrl = `${window.location.protocol}//${window.location.hostname}:5656${vidUrl}`;
}

47
frontend/src/components/tables/ChannelsTable.jsx
View file

@ -42,6 +42,7 @@ import {
Pencil,
} from 'lucide-react';
import { listOverriddenFields } from '../../utils/forms/ChannelUtils.js';
import { buildLiveStreamUrl } from '../../utils/components/FloatingVideoUtils.js';
import {
Box,
TextInput,
@ -76,6 +77,7 @@ import { useChannelLogoSelection } from '../../hooks/useSmartLogos';
import { CustomTable, useTable } from './CustomTable';
import ChannelsTableOnboarding from './ChannelsTable/ChannelsTableOnboarding';
import ChannelTableHeader from './ChannelsTable/ChannelTableHeader';
import useOutputProfilesStore from '../../store/outputProfiles';
import {
EditableTextCell,
EditableNumberCell,
@ -311,6 +313,7 @@ const ChannelsTable = ({ onReady }) => {
// store/settings
const env_mode = useSettingsStore((s) => s.environment.env_mode);
const outputProfiles = useOutputProfilesStore((s) => s.profiles);
const showVideo = useVideoStore((s) => s.showVideo);
// store/warnings
@ -376,6 +379,8 @@ const ChannelsTable = ({ onReady }) => {
cachedlogos: true,
direct: false,
tvg_id_source: 'channel_number',
output_format: '',
output_profile: '',
});
const [epgParams, setEpgParams] = useState({
cachedlogos: true,
@ -684,7 +689,7 @@ const ChannelsTable = ({ onReady }) => {
return '';
}
const uri = `/proxy/ts/stream/${channel.uuid}`;
const uri = buildLiveStreamUrl(`/proxy/ts/stream/${channel.uuid}`);
let channelUrl = `${window.location.protocol}//${window.location.host}${uri}`;
if (env_mode == 'dev') {
channelUrl = `${window.location.protocol}//${window.location.hostname}:5656${uri}`;
@ -753,6 +758,10 @@ const ChannelsTable = ({ onReady }) => {
if (m3uParams.direct) params.append('direct', 'true');
if (m3uParams.tvg_id_source !== 'channel_number')
params.append('tvg_id_source', m3uParams.tvg_id_source);
if (m3uParams.output_format)
params.append('output_format', m3uParams.output_format);
if (m3uParams.output_profile)
params.append('output_profile', m3uParams.output_profile);
const baseUrl = m3uUrl;
return params.toString() ? `${baseUrl}?${params.toString()}` : baseUrl;
@ -1420,6 +1429,42 @@ const ChannelsTable = ({ onReady }) => {
{ value: 'gracenote', label: 'Gracenote Station ID' },
]}
/>
<Select
label="Output Format"
description="Container format for streams embedded in this M3U"
clearable
placeholder="Server default"
value={m3uParams.output_format || null}
onChange={(value) =>
setM3uParams((prev) => ({
...prev,
output_format: value || '',
}))
}
comboboxProps={{ withinPortal: false }}
data={[
{ value: 'mpegts', label: 'MPEG-TS' },
{ value: 'fmp4', label: 'fMP4 (fragmented MP4)' },
]}
/>
<Select
label="Output Profile"
description="Pre-delivery transcode profile applied to all streams in this M3U"
clearable
searchable
placeholder="No transcoding"
value={m3uParams.output_profile || null}
onChange={(value) =>
setM3uParams((prev) => ({
...prev,
output_profile: value || '',
}))
}
comboboxProps={{ withinPortal: false }}
data={outputProfiles
.filter((p) => p.is_active)
.map((p) => ({ value: `${p.id}`, label: p.name }))}
/>
</Stack>
</Popover.Dropdown>
</Popover>

280
frontend/src/components/tables/OutputProfilesTable.jsx Normal file
View file

@ -0,0 +1,280 @@
import { useEffect, useMemo, useRef, useState } from 'react';
import API from '../../api';
import OutputProfileForm from '../forms/OutputProfile';
import useOutputProfilesStore from '../../store/outputProfiles';
import {
Box,
ActionIcon,
Tooltip,
Text,
Paper,
Flex,
Button,
useMantineTheme,
Center,
Switch,
Stack,
} from '@mantine/core';
import { SquareMinus, SquarePen, Eye, EyeOff, SquarePlus } from 'lucide-react';
import { CustomTable, useTable } from './CustomTable';
import useLocalStorage from '../../hooks/useLocalStorage';
const RowActions = ({ row, editOutputProfile, deleteOutputProfile }) => {
return (
<>
<ActionIcon
variant="transparent"
color="yellow.5"
size="sm"
disabled={row.original.locked}
onClick={() => editOutputProfile(row.original)}
>
<SquarePen size="18" />
</ActionIcon>
<ActionIcon
variant="transparent"
size="sm"
color="red.9"
disabled={row.original.locked}
onClick={() => deleteOutputProfile(row.original.id)}
>
<SquareMinus size="18" />
</ActionIcon>
</>
);
};
const OutputProfiles = () => {
const [profile, setProfile] = useState(null);
const [profileModalOpen, setProfileModalOpen] = useState(false);
const [hideInactive, setHideInactive] = useState(false);
const [data, setData] = useState([]);
const outputProfiles = useOutputProfilesStore((state) => state.profiles);
const [tableSize] = useLocalStorage('table-size', 'default');
const theme = useMantineTheme();
const rowVirtualizerInstanceRef = useRef(null);
const [isLoading, setIsLoading] = useState(true);
const [sorting, setSorting] = useState([]);
const columns = useMemo(
() => [
{
header: 'Name',
accessorKey: 'name',
size: 175,
cell: ({ cell }) => (
<div
style={{
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
}}
>
{cell.getValue()}
</div>
),
},
{
header: 'Command',
accessorKey: 'command',
size: 100,
cell: ({ cell }) => (
<div
style={{
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
}}
>
{cell.getValue()}
</div>
),
},
{
header: 'Parameters',
accessorKey: 'parameters',
grow: true,
cell: ({ cell }) => (
<Tooltip label={cell.getValue()}>
<div
style={{
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
}}
>
{cell.getValue()}
</div>
</Tooltip>
),
},
{
header: 'Active',
accessorKey: 'is_active',
size: 60,
cell: ({ row, cell }) => (
<Center>
<Switch
size="xs"
checked={cell.getValue()}
onChange={() => toggleProfileIsActive(row.original)}
disabled={row.original.locked}
/>
</Center>
),
},
{
id: 'actions',
header: 'Actions',
size: tableSize === 'compact' ? 50 : 75,
},
],
[]
);
const editOutputProfile = async (profile = null) => {
setProfile(profile);
setProfileModalOpen(true);
};
const deleteOutputProfile = async (id) => {
await API.deleteOutputProfile(id);
};
const closeOutputProfileForm = () => {
setProfile(null);
setProfileModalOpen(false);
};
const toggleHideInactive = () => setHideInactive((v) => !v);
const toggleProfileIsActive = async (profile) => {
await API.updateOutputProfile({
id: profile.id,
...profile,
is_active: !profile.is_active,
});
};
useEffect(() => {
if (typeof window !== 'undefined') setIsLoading(false);
}, []);
useEffect(() => {
try {
rowVirtualizerInstanceRef.current?.scrollToIndex?.(0);
} catch (error) {
console.error(error);
}
}, [sorting]);
useEffect(() => {
setData(
outputProfiles.filter((p) =>
hideInactive && !p.is_active ? false : true
)
);
}, [outputProfiles, hideInactive]);
const renderHeaderCell = (header) => (
<Text size="sm" name={header.id}>
{header.column.columnDef.header}
</Text>
);
const renderBodyCell = ({ cell, row }) => {
if (cell.column.id === 'actions') {
return (
<RowActions
row={row}
editOutputProfile={editOutputProfile}
deleteOutputProfile={deleteOutputProfile}
/>
);
}
};
const table = useTable({
columns,
data,
allRowIds: data.map((d) => d.id),
bodyCellRenderFns: { actions: renderBodyCell },
headerCellRenderFns: {
name: renderHeaderCell,
command: renderHeaderCell,
parameters: renderHeaderCell,
is_active: renderHeaderCell,
actions: renderHeaderCell,
},
});
return (
<Stack gap={0} style={{ padding: 0 }}>
<Paper
style={{ bgcolor: theme.palette?.background?.paper, borderRadius: 2 }}
>
<Box
style={{ display: 'flex', justifyContent: 'flex-end', padding: 10 }}
>
<Flex gap={6}>
<Tooltip label={hideInactive ? 'Show All' : 'Hide Inactive'}>
<Center>
<ActionIcon
onClick={toggleHideInactive}
variant="filled"
color="gray"
style={{ borderWidth: '1px', borderColor: 'white' }}
>
{hideInactive ? <EyeOff size={18} /> : <Eye size={18} />}
</ActionIcon>
</Center>
</Tooltip>
<Tooltip label="Add Output Profile">
<Button
leftSection={<SquarePlus size={18} />}
variant="light"
size="xs"
onClick={() => editOutputProfile()}
p={5}
color="green"
style={{
borderWidth: '1px',
borderColor: 'green',
color: 'white',
}}
>
Add Output Profile
</Button>
</Tooltip>
</Flex>
</Box>
</Paper>
<Box style={{ display: 'flex', flexDirection: 'column', maxHeight: 300 }}>
<Box
style={{
flex: 1,
overflowY: 'auto',
overflowX: 'auto',
border: 'solid 1px rgb(68,68,68)',
borderRadius: 'var(--mantine-radius-default)',
}}
>
<div style={{ minWidth: 600 }}>
<CustomTable table={table} />
</div>
</Box>
</Box>
<OutputProfileForm
profile={profile}
isOpen={profileModalOpen}
onClose={closeOutputProfileForm}
/>
</Stack>
);
};
export default OutputProfiles;

3
frontend/src/components/tables/StreamsTable.jsx
View file

@ -10,6 +10,7 @@ import StreamForm from '../forms/Stream';
import usePlaylistsStore from '../../store/playlists';
import useChannelsStore from '../../store/channels';
import { copyToClipboard, useDebounce } from '../../utils';
import { buildLiveStreamUrl } from '../../utils/components/FloatingVideoUtils.js';
import {
SquarePlus,
ListPlus,
@ -1048,7 +1049,7 @@ const StreamsTable = ({ onReady }) => {
};
function handleWatchStream(streamHash, streamName) {
let vidUrl = `/proxy/ts/stream/${streamHash}`;
let vidUrl = buildLiveStreamUrl(`/proxy/ts/stream/${streamHash}`);
if (env_mode == 'dev') {
vidUrl = `${window.location.protocol}//${window.location.hostname}:5656${vidUrl}`;
}

16
frontend/src/pages/Settings.jsx
View file

@ -17,6 +17,9 @@ const UserAgentsTable = React.lazy(
const StreamProfilesTable = React.lazy(
() => import('../components/tables/StreamProfilesTable.jsx')
);
const OutputProfilesTable = React.lazy(
() => import('../components/tables/OutputProfilesTable.jsx')
);
const BackupManager = React.lazy(
() => import('../components/backups/BackupManager.jsx')
);
@ -158,6 +161,19 @@ const SettingsPage = () => {
</AccordionPanel>
</AccordionItem>
<AccordionItem value="output-profiles">
<AccordionControl>Output Profiles</AccordionControl>
<AccordionPanel>
<ErrorBoundary>
<Suspense fallback={<Loader />}>
<OutputProfilesTable
active={accordianValue === 'output-profiles'}
/>
</Suspense>
</ErrorBoundary>
</AccordionPanel>
</AccordionItem>
<AccordionItem value="network-access">
<AccordionControl>
<Box>Network Access</Box>

2
frontend/src/store/auth.jsx
View file

@ -4,6 +4,7 @@ import useChannelsStore from './channels';
import usePlaylistsStore from './playlists';
import useEPGsStore from './epgs';
import useStreamProfilesStore from './streamProfiles';
import useOutputProfilesStore from './outputProfiles';
import useUserAgentsStore from './userAgents';
import useUsersStore from './users';
import API from '../api';
@ -121,6 +122,7 @@ const useAuthStore = create((set, get) => ({
useEPGsStore.getState().fetchEPGs(),
useEPGsStore.getState().fetchEPGData(),
useStreamProfilesStore.getState().fetchProfiles(),
useOutputProfilesStore.getState().fetchProfiles(),
useUserAgentsStore.getState().fetchUserAgents(),
useChannelsStore.getState().fetchChannelIds(),
]);

36
frontend/src/store/outputProfiles.jsx Normal file
View file

@ -0,0 +1,36 @@
import { create } from 'zustand';
import api from '../api';
const useOutputProfilesStore = create((set) => ({
profiles: [],
isLoading: false,
error: null,
fetchProfiles: async () => {
set({ isLoading: true, error: null });
try {
const profiles = await api.getOutputProfiles();
set({ profiles: profiles, isLoading: false });
} catch (error) {
console.error('Failed to fetch output profiles:', error);
set({ error: 'Failed to load output profiles.', isLoading: false });
}
},
addOutputProfile: (profile) =>
set((state) => ({
profiles: [...state.profiles, profile],
})),
updateOutputProfile: (profile) =>
set((state) => ({
profiles: state.profiles.map((p) => (p.id === profile.id ? profile : p)),
})),
removeOutputProfiles: (profileIds) =>
set((state) => ({
profiles: state.profiles.filter((p) => !profileIds.includes(p.id)),
})),
}));
export default useOutputProfilesStore;

13
frontend/src/utils/components/FloatingVideoUtils.js
View file

@ -1,5 +1,18 @@
export const PLAYER_PREFS_KEY = 'dispatcharr-player-prefs';
/**
* Build a live-stream preview URL that always forces mpegts output (required
* for mpegts.js) and optionally appends the browser-local web player output
* profile preference.
*/
export const buildLiveStreamUrl = (path) => {
const prefs = getPlayerPrefs();
const params = new URLSearchParams({ output_format: 'mpegts' });
const profileId = prefs.webPlayerOutputProfileId;
if (profileId) params.set('output_profile', String(profileId));
return `${path}?${params.toString()}`;
};
export const getPlayerPrefs = () => {
try {
return JSON.parse(localStorage.getItem(PLAYER_PREFS_KEY) || '{}');

1
frontend/src/utils/forms/settings/StreamSettingsFormUtils.js
View file

@ -7,6 +7,7 @@ export const getStreamSettingsFormInitialValues = () => {
preferred_region: '',
auto_import_mapped_files: true,
m3u_hash_key: [],
default_output_format: 'mpegts',
};
};

1
frontend/src/utils/pages/SettingsUtils.js
View file

@ -33,6 +33,7 @@ export const saveChangedSettings = async (settings, changedSettings) => {
'm3u_hash_key',
'preferred_region',
'auto_import_mapped_files',
'default_output_format',
];
const epgFields = [
'epg_match_mode',