diff --git a/CHANGELOG.md b/CHANGELOG.md index ea23e1dd..a20b9032 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **XC catch-up (timeshift) support**. Adds a native `/timeshift/{user}/{pass}/{stream_id}/{timestamp}/{duration}.ts` endpoint that proxies replay sessions from an Xtream Codes provider to clients such as iPlayTV (Apple TV) and TiviMate. Auth reuses the same `xc_password` custom-property the live `/live/.../{id}.ts` endpoint already uses. Catch-up sessions appear on `/stats` with a violet `TIMESHIFT` badge alongside live sessions and respect per-channel access rules. — Thanks [@cedric-marcoux](https://github.com/cedric-marcoux) (#1242) + - **Catch-up flags** (`tv_archive`, `tv_archive_duration`) denormalized at import time for zero-cost output queries; end-of-refresh SQL rollup updates channels linked to the refreshed account and self-heals stale flags on those channels only (e.g. after catch-up streams are removed or an account is deactivated). + - **Strictly-UTC API surface, automatic per-provider timezone.** `server_info.timezone` is always `UTC` and the XC EPG `start`/`end` strings are emitted in UTC; the proxy converts the requested instant to the serving provider's own reported timezone (`server_info.timezone` captured on account refresh) at request time. No timezone to configure. + - **Multi-provider failover.** Catch-up walks the channel's catch-up streams in order (like live playback): if one provider cannot serve the archive the next is tried, each attempt with its own account context (credentials, reported timezone, user-agent). Accounts that fail with an auth/ban-class status are not retried via their other streams. + - **Provider pool accounting.** Catch-up reserves a provider profile slot (`connection_pool`) before connecting upstream — same contract as live and VOD — and releases it when the session ends. When the default profile is at capacity it walks the account's alternate profiles with their own resolved credentials, and answers `503` when every profile is full. + - **Per-client session pool.** The first request without `session_id` receives a `301` redirect to the same URL with a stable `?session_id=` query parameter; that ID becomes the client identity for stats, stop keys, and pool coordination. Each viewer gets their own Redis-backed pool entry even when watching the same programme; idle sessions can be reclaimed via fingerprint matching (same user and programme, IP + user-agent score). Real scrubs within a session stop the in-flight stream through the standard stop-key mechanism; parallel startup probes (full-file, open-ended, and EOF tail `Range` requests) are deferred with `503` instead of opening extra upstream connections. Stream-limit exemptions for `ignore_same_channel_connections` apply only to sibling requests from the same `session_id`. + - **`xmltv_prev_days_override` under `Settings → EPG`** (default `0` = auto-detect). Verbose timeshift logging follows the standard logger DEBUG level. + - **EPG XMLTV `prev_days` auto-detection** from the provider's largest `tv_archive_duration` (capped at 30 days), overridable via setting or `?prev_days=` URL parameter. + - **Byte path streams directly via `iter_content` + generator yield** (same pattern as the VOD proxy), with an upfront MPEG-TS sync peek that strips any PHP-warning preamble before bytes reach strict demuxers. Throughput stays comfortably above the typical 5 Mbps FHD bitrate so clients don't buffer. + +### Performance + +- **M3U/XC stream refresh is faster on large accounts.** Steady-state refreshes split `bulk_update` into a lightweight touch pass (`last_seen` / `is_stale` only) for unchanged streams and a full column update only when provider metadata or catch-up fields actually change. +- **M3U stream filters compile once per refresh.** Account filters are regex-compiled before batch workers start; accounts with no filters skip per-stream filter checks entirely. +- **M3U refresh releases parse catalogs sooner.** Standard accounts stream-parse the on-disk M3U instead of loading the full file into RAM; `extinf_data` is dropped immediately after batch DB work, before stale cleanup and auto-sync. +- **XC live refresh avoids redundant work during catalog filtering.** `collect_xc_streams` skips disabled categories before building entries and uses a shared URL prefix instead of formatting each stream URL separately. Auto-sync releases per-group logo/EPG caches after each group iteration. +- **Celery workers return RSS after memory-intensive tasks.** `cleanup_memory()` accepts an optional `trim_heap` flag (glibc `malloc_trim`); Celery `task_postrun` enables it after `close_old_connections()` for M3U account/group refresh, EPG, VOD, and channel-matching tasks so worker memory drops back toward baseline instead of ratcheting across successive large jobs. + +### Fixed + +- **M3U group processing retries on poisoned DB connections.** `_db_query_with_retry` now treats psycopg desync errors (`DatabaseError`, e.g. `lost synchronization with server`) as transient; `process_groups` relationship loading uses it so a stale Celery worker connection resets once instead of failing the whole refresh. + ## [0.27.1] - 2026-06-25 ### Security diff --git a/apps/channels/migrations/0038_add_catchup_fields.py b/apps/channels/migrations/0038_add_catchup_fields.py new file mode 100644 index 00000000..8f653a3c --- /dev/null +++ b/apps/channels/migrations/0038_add_catchup_fields.py @@ -0,0 +1,101 @@ +"""Add denormalized catch-up fields to Stream and Channel.""" + +from django.db import migrations, models + + +def backfill_stream_catchup(apps, schema_editor): + """Derive is_catchup/catchup_days from Stream.custom_properties JSON.""" + with schema_editor.connection.cursor() as cursor: + cursor.execute(""" + UPDATE dispatcharr_channels_stream + SET is_catchup = TRUE, + catchup_days = COALESCE( + CASE WHEN (custom_properties->>'tv_archive_duration') ~ '^\\d+$' + THEN (custom_properties->>'tv_archive_duration')::int + ELSE NULL + END, 7 + ) + WHERE custom_properties IS NOT NULL + AND custom_properties != 'null'::jsonb + AND ( + custom_properties->>'tv_archive' = '1' + -- JSON booleans extract as lowercase 'true' via ->>; the + -- 'True' spelling covers Python-str values stored by + -- older import code. + OR custom_properties->>'tv_archive' = 'true' + OR custom_properties->>'tv_archive' = 'True' + ) + """) + + +def backfill_channel_catchup(apps, schema_editor): + """Roll up catch-up fields from streams to channels.""" + with schema_editor.connection.cursor() as cursor: + cursor.execute(""" + UPDATE dispatcharr_channels_channel c SET + is_catchup = EXISTS ( + SELECT 1 FROM dispatcharr_channels_channelstream cs + JOIN dispatcharr_channels_stream s ON s.id = cs.stream_id + WHERE cs.channel_id = c.id AND s.is_catchup = TRUE + ), + catchup_days = COALESCE(( + SELECT MAX(s.catchup_days) FROM dispatcharr_channels_channelstream cs + JOIN dispatcharr_channels_stream s ON s.id = cs.stream_id + WHERE cs.channel_id = c.id AND s.is_catchup = TRUE + ), 0) + """) + + +class Migration(migrations.Migration): + + dependencies = [ + ("dispatcharr_channels", "0037_auto_sync_overhaul"), + ] + + operations = [ + # Stream fields + migrations.AddField( + model_name="stream", + name="is_catchup", + field=models.BooleanField( + default=False, + db_index=True, + help_text="Whether this stream supports catch-up/timeshift (tv_archive=1)", + ), + ), + migrations.AddField( + model_name="stream", + name="catchup_days", + field=models.PositiveIntegerField( + default=0, + help_text="Number of days of catch-up archive available (tv_archive_duration)", + ), + ), + # Channel fields + migrations.AddField( + model_name="channel", + name="is_catchup", + field=models.BooleanField( + default=False, + db_index=True, + help_text="Whether any stream on this channel supports catch-up (tv_archive=1)", + ), + ), + migrations.AddField( + model_name="channel", + name="catchup_days", + field=models.PositiveIntegerField( + default=0, + help_text="Max catch-up archive days across all streams on this channel", + ), + ), + # Backfill existing data + migrations.RunPython( + backfill_stream_catchup, + reverse_code=migrations.RunPython.noop, + ), + migrations.RunPython( + backfill_channel_catchup, + reverse_code=migrations.RunPython.noop, + ), + ] diff --git a/apps/channels/models.py b/apps/channels/models.py index 747d0b89..fb421080 100644 --- a/apps/channels/models.py +++ b/apps/channels/models.py @@ -135,6 +135,17 @@ class Stream(models.Model): db_index=True ) + # Populated at import from tv_archive / tv_archive_duration. + is_catchup = models.BooleanField( + default=False, + db_index=True, + help_text="Whether this stream supports catch-up/timeshift (tv_archive=1)", + ) + catchup_days = models.PositiveIntegerField( + default=0, + help_text="Number of days of catch-up archive available (tv_archive_duration)", + ) + class Meta: # If you use m3u_account, you might do unique_together = ('name','url','m3u_account') verbose_name = "Stream" @@ -364,6 +375,17 @@ class Channel(models.Model): help_text="The M3U account that auto-created this channel" ) + # Populated at import; rolled up via ChannelStream signal / m3u refresh. + is_catchup = models.BooleanField( + default=False, + db_index=True, + help_text="Whether any stream on this channel supports catch-up (tv_archive=1)", + ) + catchup_days = models.PositiveIntegerField( + default=0, + help_text="Max catch-up archive days across all streams on this channel", + ) + # Hidden channels are excluded from HDHR, M3U, EPG, and XC output queries. # Auto-sync still recognizes them so they are not recreated when their # underlying provider stream persists; this is an output-layer concern, not diff --git a/apps/channels/serializers.py b/apps/channels/serializers.py index 61bbef8d..d62b480e 100644 --- a/apps/channels/serializers.py +++ b/apps/channels/serializers.py @@ -448,6 +448,7 @@ class ChannelSerializer(serializers.ModelSerializer): "logo_id", "user_level", "is_adult", + "is_catchup", "hidden_from_output", "auto_created", "auto_created_by", diff --git a/apps/channels/signals.py b/apps/channels/signals.py index 53f1ea33..1c3d00ef 100644 --- a/apps/channels/signals.py +++ b/apps/channels/signals.py @@ -5,7 +5,7 @@ from django.dispatch import receiver from django.utils.timezone import now, is_aware, make_aware from celery.result import AsyncResult from django_celery_beat.models import ClockedSchedule, PeriodicTask -from .models import Channel, Stream, ChannelProfile, ChannelProfileMembership, Recording +from .models import Channel, Stream, ChannelStream, ChannelProfile, ChannelProfileMembership, Recording from apps.m3u.models import M3UAccount from apps.epg.tasks import parse_programs_for_tvg_id import json @@ -369,3 +369,20 @@ def schedule_task_on_save(sender, instance, created, **kwargs): @receiver(post_delete, sender=Recording) def revoke_task_on_delete(sender, instance, **kwargs): revoke_task(instance.task_id) + + +@receiver([post_save, post_delete], sender=ChannelStream) +def update_channel_catchup_fields(sender, instance, **kwargs): + """Roll up catch-up flags from active streams (UI path; import uses SQL rollup).""" + from django.db.models import Max + + channel = instance.channel + catchup_qs = channel.streams.filter( + is_catchup=True, + m3u_account__is_active=True, + ) + max_days = catchup_qs.aggregate(max_days=Max("catchup_days"))["max_days"] + Channel.objects.filter(pk=channel.pk).update( + is_catchup=catchup_qs.exists(), + catchup_days=max_days or 0, + ) diff --git a/apps/channels/tests/test_catchup_utils.py b/apps/channels/tests/test_catchup_utils.py new file mode 100644 index 00000000..d79eec11 --- /dev/null +++ b/apps/channels/tests/test_catchup_utils.py @@ -0,0 +1,100 @@ +from unittest.mock import patch + +from django.test import RequestFactory, SimpleTestCase, TestCase + +from apps.accounts.models import User +from apps.channels.models import Channel, ChannelStream, Stream +from apps.channels.utils import resolve_xc_epg_prev_days +from apps.m3u.models import M3UAccount + + +class ResolveXcEpgPrevDaysTests(SimpleTestCase): + def setUp(self): + self.factory = RequestFactory() + self.user = User(username="xc-prev", custom_properties={}) + + def test_url_prev_days_zero_is_explicit(self): + request = self.factory.get("/xmltv.php?prev_days=0") + self.assertEqual(resolve_xc_epg_prev_days(request, self.user), 0) + + def test_user_epg_prev_days_used_when_url_omitted(self): + self.user.custom_properties = {"epg_prev_days": 5} + request = self.factory.get("/xmltv.php") + self.assertEqual(resolve_xc_epg_prev_days(request, self.user), 5) + + @patch("apps.channels.utils.compute_provider_archive_days_capped", return_value=14) + def test_auto_detect_only_when_no_url_or_user_default(self, mock_compute): + request = self.factory.get("/xmltv.php") + self.assertEqual(resolve_xc_epg_prev_days(request, self.user), 14) + mock_compute.assert_called_once() + + @patch("apps.channels.utils.compute_provider_archive_days_capped", return_value=14) + def test_per_channel_epg_skips_global_auto_detect(self, mock_compute): + request = self.factory.get("/player_api.php") + self.assertEqual( + resolve_xc_epg_prev_days(request, self.user, auto_detect_fallback=False), + 0, + ) + mock_compute.assert_not_called() + + @patch("apps.channels.utils.compute_provider_archive_days_capped", return_value=14) + def test_user_default_prevents_auto_detect(self, mock_compute): + self.user.custom_properties = {"epg_prev_days": 3} + request = self.factory.get("/xmltv.php") + self.assertEqual(resolve_xc_epg_prev_days(request, self.user), 3) + mock_compute.assert_not_called() + + @patch("apps.channels.utils.compute_provider_archive_days_capped", return_value=14) + @patch("core.models.CoreSettings.get_xmltv_prev_days_override", return_value=7) + def test_epg_settings_override_prevents_auto_detect(self, _override, mock_compute): + request = self.factory.get("/xmltv.php") + self.assertEqual(resolve_xc_epg_prev_days(request, self.user), 7) + mock_compute.assert_not_called() + + +class CatchupRollupActiveAccountTests(TestCase): + """Denormalized catch-up flags ignore disabled M3U accounts.""" + + @classmethod + def setUpTestData(cls): + cls.inactive = M3UAccount.objects.create( + name="catchup-inactive", + server_url="http://example.test", + account_type="XC", + is_active=False, + ) + + def test_channelstream_signal_ignores_inactive_catchup_stream(self): + channel = Channel.objects.create(name="inactive-only") + stream = Stream.objects.create( + name="inactive-catchup", + url="http://example.test/inactive", + m3u_account=self.inactive, + is_catchup=True, + catchup_days=9, + ) + ChannelStream.objects.create(channel=channel, stream=stream, order=0) + + channel.refresh_from_db() + self.assertFalse(channel.is_catchup) + self.assertEqual(channel.catchup_days, 0) + + def test_rollup_ignores_inactive_catchup_stream(self): + from apps.m3u.tasks import rollup_channel_catchup_fields + + channel = Channel.objects.create(name="rollup-inactive-only") + stream = Stream.objects.create( + name="rollup-inactive-catchup", + url="http://example.test/rollup-inactive", + m3u_account=self.inactive, + is_catchup=True, + catchup_days=9, + ) + ChannelStream.objects.create(channel=channel, stream=stream, order=0) + Channel.objects.filter(pk=channel.pk).update(is_catchup=True, catchup_days=9) + + rollup_channel_catchup_fields(self.inactive.id) + + channel.refresh_from_db() + self.assertFalse(channel.is_catchup) + self.assertEqual(channel.catchup_days, 0) diff --git a/apps/channels/utils.py b/apps/channels/utils.py index 60a27146..70b9eecd 100644 --- a/apps/channels/utils.py +++ b/apps/channels/utils.py @@ -1,8 +1,13 @@ import logging import threading +from django.core.cache import cache + logger = logging.getLogger(__name__) +PROVIDER_ARCHIVE_CACHE_TTL_SECONDS = 300 +MAX_AUTO_PREV_DAYS = 30 + # Bound memory/DB work per chunk for large libraries (20k+ channels). EPG_LOGO_APPLY_BATCH_SIZE = 500 EPG_LOGO_APPLY_MAX_ERRORS = 100 @@ -23,6 +28,91 @@ def format_channel_number(value, empty=""): return int(value) return value + +def compute_provider_archive_days_capped(): + """Max ``catchup_days`` across active XC catch-up streams (capped, cached). + + Cached briefly so XC XMLTV exports without an explicit ``prev_days`` do not + repeat the aggregate query on every request. + """ + def _scan(): + from django.db.models import Max + + from apps.channels.models import Stream + + result = Stream.objects.filter( + m3u_account__account_type="XC", + m3u_account__is_active=True, + is_catchup=True, + ).aggregate(max_days=Max("catchup_days")) + return min(result["max_days"] or 0, MAX_AUTO_PREV_DAYS) + + return cache.get_or_set( + "channels:provider_archive_days_capped", + _scan, + PROVIDER_ARCHIVE_CACHE_TTL_SECONDS, + ) + + +def resolve_xc_epg_prev_days(request, user, *, auto_detect_fallback=True): + """Resolve ``prev_days`` for XC XMLTV and player_api EPG. + + Args: + request: HTTP request (reads ``?prev_days=``). + user: Authenticated user (reads ``custom_properties.epg_prev_days``). + auto_detect_fallback: When True (XC XMLTV), fall back to the largest + provider archive depth. When False (per-channel EPG), return 0 so + ``xc_get_epg`` can expand to each channel's ``catchup_days``. + + Resolution order: + 1. URL ``?prev_days=`` (explicit; 0 means no past programmes) + 2. ``user.custom_properties.epg_prev_days`` + 3. ``CoreSettings.epg_settings.xmltv_prev_days_override`` when > 0 + 4. Auto-detect (only when *auto_detect_fallback* is True) + """ + user_custom = (user.custom_properties or {}) if user else {} + url_prev = request.GET.get("prev_days") + user_prev = user_custom.get("epg_prev_days") if user_custom else None + + if url_prev is not None: + try: + return max(0, min(int(url_prev), MAX_AUTO_PREV_DAYS)) + except (ValueError, TypeError): + return 0 + if user_prev not in (None, ""): + try: + return max(0, min(int(user_prev), MAX_AUTO_PREV_DAYS)) + except (ValueError, TypeError): + return 0 + + from core.models import CoreSettings + + try: + override = int(CoreSettings.get_xmltv_prev_days_override() or 0) + except (TypeError, ValueError): + override = 0 + if override > 0: + return max(0, min(override, MAX_AUTO_PREV_DAYS)) + if auto_detect_fallback: + return compute_provider_archive_days_capped() + return 0 + + +def get_channel_catchup_streams(channel): + """Active catch-up streams for a channel, in ``channelstream`` order. + + Inactive M3U accounts are excluded, matching live dispatch. + """ + if not getattr(channel, "is_catchup", False): + return [] + + return list( + channel.streams.filter(is_catchup=True, m3u_account__is_active=True) + .order_by("channelstream__order") + .select_related("m3u_account") + ) + + def increment_stream_count(account): with lock: current_usage = active_streams_map.get(account.id, 0) diff --git a/apps/m3u/tasks.py b/apps/m3u/tasks.py index 462f45db..a5894892 100644 --- a/apps/m3u/tasks.py +++ b/apps/m3u/tasks.py @@ -52,9 +52,9 @@ def _db_query_with_retry(fn, *, label="DB query", max_retries=2): Poisoned Celery worker connections often surface as OperationalError or as ``IndexError: list index out of range`` inside Django's row converters. """ - from django.db import InterfaceError, OperationalError + from django.db import DatabaseError, InterfaceError, OperationalError - transient_errors = (OperationalError, InterfaceError, IndexError) + transient_errors = (OperationalError, InterfaceError, IndexError, DatabaseError) for attempt in range(max_retries): try: return fn() @@ -134,11 +134,23 @@ def _ensure_m3u_refresh_terminal_status(account_id): _EXTINF_ATTR_RE = re.compile(r'([^\s=]+)\s*=\s*(["\'])(.*?)\2') +def _open_m3u_text_source(source_path): + """Open an on-disk M3U (or .m3u.gz) file for line-by-line parsing.""" + if source_path.endswith(".gz"): + return gzip.open(source_path, "rt", encoding="utf-8") + return open(source_path, "r", encoding="utf-8") + + def fetch_m3u_lines(account, use_cache=False): + """Fetch M3U source for parsing. + + On success returns ``(source, True)`` where *source* is either a filesystem + path (streamed during parse) or, for ZIP uploads only, an in-memory line + list. Failures return ``(None, False)``. + """ os.makedirs(m3u_dir, exist_ok=True) file_path = os.path.join(m3u_dir, f"{account.id}.m3u") - """Fetch M3U file lines efficiently.""" if account.server_url: if not use_cache or not os.path.exists(file_path): try: @@ -213,7 +225,7 @@ def fetch_m3u_lines(account, use_cache=False): status="error", error=error_msg, ) - return [], False + return None, False # Only call raise_for_status if we have a success code (this should not raise now) response.raise_for_status() @@ -288,7 +300,7 @@ def fetch_m3u_lines(account, use_cache=False): status="error", error=error_msg, ) - return [], False + return None, False # Validate the file by reading only the first portion from # disk — no need to load the entire file into memory just @@ -360,7 +372,7 @@ def fetch_m3u_lines(account, use_cache=False): status="error", error=error_msg, ) - return [], False + return None, False except UnicodeDecodeError: with open(temp_path, "rb") as vf: @@ -378,7 +390,7 @@ def fetch_m3u_lines(account, use_cache=False): status="error", error=error_msg, ) - return [], False + return None, False # Validation passed — promote temp file to final path os.replace(temp_path, file_path) @@ -433,7 +445,7 @@ def fetch_m3u_lines(account, use_cache=False): status="error", error=error_msg, ) - return [], False + return None, False except requests.exceptions.RequestException as e: # Handle other request errors (connection, timeout, etc.) if "timeout" in str(e).lower(): @@ -454,7 +466,7 @@ def fetch_m3u_lines(account, use_cache=False): status="error", error=error_msg, ) - return [], False + return None, False except Exception as e: # Handle any other unexpected errors error_msg = f"Unexpected error while fetching M3U file from URL: {account.server_url} - {str(e)}" @@ -469,7 +481,7 @@ def fetch_m3u_lines(account, use_cache=False): status="error", error=error_msg, ) - return [], False + return None, False # Check if the file exists and is not empty (fallback check - should not happen with new validation) if not os.path.exists(file_path) or os.path.getsize(file_path) == 0: @@ -481,27 +493,14 @@ def fetch_m3u_lines(account, use_cache=False): send_m3u_update( account.id, "downloading", 100, status="error", error=error_msg ) - return [], False # Return empty list and False for success + return None, False - try: - with open(file_path, "r", encoding="utf-8") as f: - return f.readlines(), True - except Exception as e: - error_msg = f"Error reading M3U file: {str(e)}" - logger.error(error_msg) - account.status = M3UAccount.Status.ERROR - account.last_message = error_msg - account.save(update_fields=["status", "last_message"]) - send_m3u_update( - account.id, "downloading", 100, status="error", error=error_msg - ) - return [], False + return file_path, True elif account.file_path: try: if account.file_path.endswith(".gz"): - with gzip.open(account.file_path, "rt", encoding="utf-8") as f: - return f.readlines(), True + return account.file_path, True elif account.file_path.endswith(".zip"): with zipfile.ZipFile(account.file_path, "r") as zip_file: @@ -522,11 +521,10 @@ def fetch_m3u_lines(account, use_cache=False): send_m3u_update( account.id, "downloading", 100, status="error", error=error_msg ) - return [], False + return None, False else: - with open(account.file_path, "r", encoding="utf-8") as f: - return f.readlines(), True + return account.file_path, True except (IOError, OSError, zipfile.BadZipFile, gzip.BadGzipFile) as e: error_msg = f"Error opening file {account.file_path}: {e}" @@ -537,7 +535,7 @@ def fetch_m3u_lines(account, use_cache=False): send_m3u_update( account.id, "downloading", 100, status="error", error=error_msg ) - return [], False + return None, False # Neither server_url nor uploaded_file is available error_msg = "No M3U source available (missing URL and file)" @@ -546,7 +544,7 @@ def fetch_m3u_lines(account, use_cache=False): account.last_message = error_msg account.save(update_fields=["status", "last_message"]) send_m3u_update(account.id, "downloading", 100, status="error", error=error_msg) - return [], False + return None, False def get_case_insensitive_attr(attributes, key, default=""): @@ -746,13 +744,16 @@ def process_groups(account, groups, scan_start_time=None): all_group_objs = existing_group_objs + newly_created_group_objs # Get existing relationships for this account - existing_relationships = { - rel.channel_group.name: rel - for rel in ChannelGroupM3UAccount.objects.filter( - m3u_account=account, - channel_group__name__in=groups.keys() - ).select_related('channel_group') - } + existing_relationships = _db_query_with_retry( + lambda: { + rel.channel_group.name: rel + for rel in ChannelGroupM3UAccount.objects.filter( + m3u_account=account, + channel_group__name__in=groups.keys(), + ).select_related("channel_group") + }, + label=f"process_groups relationships for account {account.id}", + ) relations_to_create = [] relations_to_update = [] @@ -905,6 +906,11 @@ def collect_xc_streams(account_id, enabled_groups): account.get_user_agent(), ) as xc_client: + stream_url_prefix = ( + f"{xc_client.server_url.rstrip('/')}/live/" + f"{xc_client.username}/{xc_client.password}/" + ) + # Fetch ALL live streams in a single API call (much more efficient) logger.info("Fetching ALL live streams from XC provider...") all_xc_streams = xc_client.get_all_live_streams() # Get all streams without category filter @@ -917,46 +923,49 @@ def collect_xc_streams(account_id, enabled_groups): # Filter streams based on enabled categories for stream in all_xc_streams: - # Fall back to a generated name if the provider returns null/empty - stream_name = stream.get("name") or f"{account.name} - {stream.get('stream_id', 'Unknown')}" + category_id = str(stream.get("category_id", "")) + if category_id not in enabled_category_ids: + continue + + group_info = enabled_category_ids[category_id] + stream_name = stream.get("name") or ( + f"{account.name} - {stream.get('stream_id', 'Unknown')}" + ) if not stream.get("name"): logger.warning( - f"XC stream has null/empty name; using generated name '{stream_name}' " - f"(stream_id={stream.get('stream_id', 'unknown')})" + "XC stream has null/empty name; using generated name '%s' " + "(stream_id=%s)", + stream_name, stream.get("stream_id", "unknown"), ) - # Get the category_id for this stream - category_id = str(stream.get("category_id", "")) - - # Only include streams from enabled categories - if category_id in enabled_category_ids: - group_info = enabled_category_ids[category_id] - - # Convert XC stream to our standard format with all properties preserved - stream_data = { - "name": stream_name, - "url": xc_client.get_stream_url(stream["stream_id"]), - "attributes": { - "tvg-id": stream.get("epg_channel_id", ""), - "tvg-logo": stream.get("stream_icon", ""), - "group-title": group_info["name"], - # Preserve all XC stream properties as custom attributes - "stream_id": str(stream.get("stream_id", "")), - "num": stream.get("num"), - "category_id": category_id, - "stream_type": stream.get("stream_type", ""), - "added": stream.get("added", ""), - "is_adult": str(stream.get("is_adult", "0")), - "custom_sid": stream.get("custom_sid", ""), - # Include any other properties that might be present - **{k: str(v) for k, v in stream.items() if k not in [ - "name", "stream_id", "epg_channel_id", "stream_icon", - "category_id", "stream_type", "added", "is_adult", "custom_sid", "num" - ] and v is not None} - } - } - all_streams.append(stream_data) - filtered_count += 1 + stream_id = stream.get("stream_id") + attributes = { + "tvg-id": stream.get("epg_channel_id", ""), + "tvg-logo": stream.get("stream_icon", ""), + "group-title": group_info["name"], + "stream_id": str(stream.get("stream_id", "")), + "num": stream.get("num"), + "category_id": category_id, + "stream_type": stream.get("stream_type", ""), + "added": stream.get("added", ""), + "is_adult": str(stream.get("is_adult", "0")), + "custom_sid": stream.get("custom_sid", ""), + **{ + k: str(v) + for k, v in stream.items() + if k not in [ + "name", "stream_id", "epg_channel_id", "stream_icon", + "category_id", "stream_type", "added", "is_adult", + "custom_sid", "num", + ] and v is not None + }, + } + all_streams.append({ + "name": stream_name, + "url": f"{stream_url_prefix}{stream_id}.ts", + "attributes": attributes, + }) + filtered_count += 1 # Drop the full provider catalog before returning; only filtered rows are needed. del all_xc_streams @@ -971,6 +980,60 @@ def collect_xc_streams(account_id, enabled_groups): ) return all_streams + +def _compile_m3u_stream_filters(filter_queryset): + """Compile account M3UFilter rows once per refresh for batch workers.""" + compiled = [] + for filter_obj in filter_queryset: + flags = ( + re.IGNORECASE + if (filter_obj.custom_properties or {}).get("case_sensitive", True) is False + else 0 + ) + compiled.append((re.compile(filter_obj.regex_pattern, flags), filter_obj)) + return compiled + + +def _stream_passes_m3u_filters(name, url, group_title, compiled_filters): + """Return False when the first matching filter excludes the stream.""" + for pattern, filter_obj in compiled_filters: + logger.trace("Checking filter pattern %s", pattern.pattern) + if filter_obj.filter_type == "url": + target = url + elif filter_obj.filter_type == "group": + target = group_title + else: + target = name + + if pattern.search(target or ""): + logger.debug( + "Stream %s - %s matches filter pattern %s", + name, url, filter_obj.regex_pattern, + ) + return not filter_obj.exclude + return True + + +_STREAM_TOUCH_FIELDS = ("last_seen", "is_stale") +_STREAM_CHANGED_FIELDS = ( + "name", "url", "logo_url", "tvg_id", "custom_properties", "is_adult", + "last_seen", "updated_at", "is_stale", "stream_id", "stream_chno", + "channel_group_id", "is_catchup", "catchup_days", +) + + +def _bulk_update_stream_refresh_batches(changed_streams, touch_streams, *, batch_size): + """Unchanged streams only need last_seen/is_stale; changed rows get the full set.""" + if touch_streams: + Stream.objects.bulk_update( + touch_streams, list(_STREAM_TOUCH_FIELDS), batch_size=batch_size, + ) + if changed_streams: + Stream.objects.bulk_update( + changed_streams, list(_STREAM_CHANGED_FIELDS), batch_size=batch_size, + ) + + def process_xc_category_direct(account_id, batch, groups, hash_keys): from django.db import connections @@ -981,6 +1044,7 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): streams_to_create = [] streams_to_update = [] + streams_to_touch = [] stream_hashes = {} try: @@ -1053,6 +1117,13 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): name, url, tvg_id, hash_keys, m3u_id=account_id, group=group_title, account_type='XC', stream_id=provider_stream_id ) + _tv_archive = str(stream.get("tv_archive", "0")) + _is_catchup = _tv_archive in ("1", "True") + try: + _catchup_days = int(stream.get("tv_archive_duration", 0) or 0) + except (TypeError, ValueError): + _catchup_days = 0 + stream_props = { "name": name, "url": url, @@ -1066,6 +1137,8 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): "is_stale": False, "stream_id": provider_stream_id, "stream_chno": stream_chno, + "is_catchup": _is_catchup, + "catchup_days": _catchup_days, } if stream_hash not in stream_hashes: @@ -1080,7 +1153,7 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): existing_streams = { s.stream_hash: s for s in Stream.objects.filter(stream_hash__in=stream_hashes.keys()).select_related('m3u_account').only( - 'id', 'stream_hash', 'name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'last_seen', 'updated_at', 'm3u_account', 'stream_id', 'stream_chno', 'channel_group_id' + 'id', 'stream_hash', 'name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'last_seen', 'updated_at', 'm3u_account', 'stream_id', 'stream_chno', 'channel_group_id', 'is_catchup', 'catchup_days' ) } @@ -1097,7 +1170,9 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): obj.is_adult != stream_props["is_adult"] or obj.stream_id != stream_props["stream_id"] or obj.stream_chno != stream_props["stream_chno"] or - obj.channel_group_id != stream_props["channel_group_id"] + obj.channel_group_id != stream_props["channel_group_id"] or + obj.is_catchup != stream_props["is_catchup"] or + obj.catchup_days != stream_props["catchup_days"] ) if changed: @@ -1108,11 +1183,9 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): obj.is_stale = False streams_to_update.append(obj) else: - # Always update last_seen, even if nothing else changed obj.last_seen = timezone.now() obj.is_stale = False - # Don't update updated_at for unchanged streams - streams_to_update.append(obj) + streams_to_touch.append(obj) # Remove from existing_streams since we've processed it del existing_streams[stream_hash] @@ -1129,13 +1202,9 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): if streams_to_create: Stream.objects.bulk_create(streams_to_create, ignore_conflicts=True) - if streams_to_update: - # Simplified bulk update for better performance - Stream.objects.bulk_update( - streams_to_update, - ['name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'is_adult', 'last_seen', 'updated_at', 'is_stale', 'stream_id', 'stream_chno', 'channel_group_id'], - batch_size=150 # Smaller batch size for XC processing - ) + _bulk_update_stream_refresh_batches( + streams_to_update, streams_to_touch, batch_size=150, + ) # Update last_seen for any remaining existing streams that weren't processed if len(existing_streams.keys()) > 0: @@ -1143,7 +1212,10 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): except Exception as e: logger.error(f"Bulk operation failed for XC streams: {str(e)}") - retval = f"Batch processed: {len(streams_to_create)} created, {len(streams_to_update)} updated." + retval = ( + f"Batch processed: {len(streams_to_create)} created, " + f"{len(streams_to_update) + len(streams_to_touch)} updated." + ) except Exception as e: logger.error(f"XC category processing error: {str(e)}") @@ -1153,14 +1225,18 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): connections.close_all() # Aggressive garbage collection - del streams_to_create, streams_to_update, stream_hashes, existing_streams + del streams_to_create, streams_to_update, streams_to_touch, stream_hashes, existing_streams gc.collect() return retval -def process_m3u_batch_direct(account_id, batch, groups, hash_keys): - """Processes a batch of M3U streams using bulk operations with thread-safe DB connections.""" +def process_m3u_batch_direct(account_id, batch, groups, hash_keys, compiled_filters=None): + """Processes a batch of M3U streams using bulk operations with thread-safe DB connections. + + ``compiled_filters`` should be pre-built once per account refresh and shared + across batch workers. Pass an empty list when the account has no filters. + """ from django.db import connections # Ensure clean database connections for threading @@ -1168,33 +1244,22 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys): account = M3UAccount.objects.get(id=account_id) - compiled_filters = [ - ( - re.compile( - f.regex_pattern, - ( - re.IGNORECASE - if (f.custom_properties or {}).get( - "case_sensitive", True - ) - == False - else 0 - ), - ), - f, - ) - for f in account.filters.order_by("order") - ] + if compiled_filters is None: + compiled_filters = _compile_m3u_stream_filters(account.filters.order_by("order")) streams_to_create = [] streams_to_update = [] + streams_to_touch = [] stream_hashes = {} name_max_length = Stream._meta.get_field('name').max_length logger.debug(f"Processing batch of {len(batch)} for M3U account {account_id}") if compiled_filters: - logger.debug(f"Using compiled filters: {[f[1].regex_pattern for f in compiled_filters]}") + logger.debug( + "Using compiled filters: %s", + [filter_obj.regex_pattern for _, filter_obj in compiled_filters], + ) for stream_info in batch: try: name, url = stream_info["name"], stream_info["url"] @@ -1215,25 +1280,12 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys): group_title = get_case_insensitive_attr( stream_info["attributes"], "group-title", "Default Group" ) - logger.debug(f"Processing stream: {name} - {url} in group {group_title}") - include = True - for pattern, filter in compiled_filters: - logger.trace(f"Checking filter pattern {pattern}") - target = name - if filter.filter_type == "url": - target = url - elif filter.filter_type == "group": - target = group_title + logger.trace("Processing stream: %s - %s in group %s", name, url, group_title) - if pattern.search(target or ""): - logger.debug( - f"Stream {name} - {url} matches filter pattern {filter.regex_pattern}" - ) - include = not filter.exclude - break - - if not include: - logger.debug(f"Stream excluded by filter, skipping.") + if compiled_filters and not _stream_passes_m3u_filters( + name, url, group_title, compiled_filters, + ): + logger.debug("Stream excluded by filter, skipping.") continue # Filter out disabled groups for this account @@ -1279,6 +1331,14 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys): account_type=account_type_for_hash, stream_id=provider_stream_id ) + _attrs = stream_info["attributes"] + _tv_archive_m3u = str(_attrs.get("tv_archive", "0")) + _is_catchup_m3u = _tv_archive_m3u in ("1", "True") + try: + _catchup_days_m3u = int(_attrs.get("tv_archive_duration", 0) or 0) + except (TypeError, ValueError): + _catchup_days_m3u = 0 + stream_props = { "name": name, "url": url, @@ -1287,11 +1347,13 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys): "m3u_account": account, "channel_group_id": int(groups.get(group_title)), "stream_hash": stream_hash, - "custom_properties": {**stream_info["attributes"], "vlc_opts": stream_info["vlc_opts"]} if "vlc_opts" in stream_info else stream_info["attributes"], - "is_adult": parse_is_adult(stream_info["attributes"].get("is_adult", 0)), + "custom_properties": {**_attrs, "vlc_opts": stream_info["vlc_opts"]} if "vlc_opts" in stream_info else _attrs, + "is_adult": parse_is_adult(_attrs.get("is_adult", 0)), "is_stale": False, "stream_id": provider_stream_id, "stream_chno": channel_num, + "is_catchup": _is_catchup_m3u, + "catchup_days": _catchup_days_m3u, } if stream_hash not in stream_hashes: @@ -1303,7 +1365,7 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys): existing_streams = { s.stream_hash: s for s in Stream.objects.filter(stream_hash__in=stream_hashes.keys()).select_related('m3u_account').only( - 'id', 'stream_hash', 'name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'last_seen', 'updated_at', 'm3u_account', 'stream_id', 'stream_chno', 'channel_group_id' + 'id', 'stream_hash', 'name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'last_seen', 'updated_at', 'm3u_account', 'stream_id', 'stream_chno', 'channel_group_id', 'is_catchup', 'catchup_days' ) } @@ -1320,14 +1382,15 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys): obj.is_adult != stream_props["is_adult"] or obj.stream_id != stream_props["stream_id"] or obj.stream_chno != stream_props["stream_chno"] or - obj.channel_group_id != stream_props["channel_group_id"] + obj.channel_group_id != stream_props["channel_group_id"] or + obj.is_catchup != stream_props["is_catchup"] or + obj.catchup_days != stream_props["catchup_days"] ) - # Always update last_seen obj.last_seen = timezone.now() + obj.is_stale = False if changed: - # Only update fields that changed and set updated_at obj.name = stream_props["name"] obj.url = stream_props["url"] obj.logo_url = stream_props["logo_url"] @@ -1337,12 +1400,12 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys): obj.stream_id = stream_props["stream_id"] obj.stream_chno = stream_props["stream_chno"] obj.channel_group_id = stream_props["channel_group_id"] + obj.is_catchup = stream_props["is_catchup"] + obj.catchup_days = stream_props["catchup_days"] obj.updated_at = timezone.now() - - # Always mark as not stale since we saw it in this refresh - obj.is_stale = False - - streams_to_update.append(obj) + streams_to_update.append(obj) + else: + streams_to_touch.append(obj) else: # New stream stream_props["last_seen"] = timezone.now() @@ -1355,23 +1418,22 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys): if streams_to_create: Stream.objects.bulk_create(streams_to_create, ignore_conflicts=True) - if streams_to_update: - # Update all streams in a single bulk operation - Stream.objects.bulk_update( - streams_to_update, - ['name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'is_adult', 'last_seen', 'updated_at', 'is_stale', 'stream_id', 'stream_chno', 'channel_group_id'], - batch_size=200 - ) + _bulk_update_stream_refresh_batches( + streams_to_update, streams_to_touch, batch_size=200, + ) except Exception as e: logger.error(f"Bulk operation failed: {str(e)}") - retval = f"M3U account: {account_id}, Batch processed: {len(streams_to_create)} created, {len(streams_to_update)} updated." + retval = ( + f"M3U account: {account_id}, Batch processed: {len(streams_to_create)} created, " + f"{len(streams_to_update) + len(streams_to_touch)} updated." + ) # Clean up database connections for threading connections.close_all() # Free batch data structures (reference-counted deallocation) - del streams_to_create, streams_to_update, stream_hashes, existing_streams + del streams_to_create, streams_to_update, streams_to_touch, stream_hashes, existing_streams gc.collect() return retval @@ -1658,28 +1720,49 @@ def refresh_m3u_groups(account_id, use_cache=False, full_refresh=False, scan_sta release_task_lock("refresh_m3u_account_groups", account_id) return error_msg, None else: - lines, success = fetch_m3u_lines(account, use_cache) + source, success = fetch_m3u_lines(account, use_cache) if not success: # If fetch failed, don't continue processing lock_renewer.stop() release_task_lock("refresh_m3u_account_groups", account_id) return f"Failed to fetch M3U data for account_id={account_id}.", None - # Log basic file structure for debugging - logger.debug(f"Processing {len(lines)} lines from M3U file") - valid_stream_count = 0 - for entry in iter_m3u_entries(lines): - valid_stream_count += 1 - group_title_attr = get_case_insensitive_attr(entry["attributes"], "group-title", "") - if group_title_attr and group_title_attr not in groups: - logger.debug(f"Found new group for M3U account {account_id}: '{group_title_attr}'") - groups[group_title_attr] = {} - extinf_data.append(entry) + if isinstance(source, str): + logger.debug(f"Streaming M3U parse from {source}") + with _open_m3u_text_source(source) as m3u_file: + entry_iter = iter_m3u_entries(m3u_file) + for entry in entry_iter: + valid_stream_count += 1 + group_title_attr = get_case_insensitive_attr(entry["attributes"], "group-title", "") + if group_title_attr and group_title_attr not in groups: + logger.debug(f"Found new group for M3U account {account_id}: '{group_title_attr}'") + groups[group_title_attr] = {} + extinf_data.append(entry) - if valid_stream_count % 1000 == 0: - logger.debug(f"Processed {valid_stream_count} valid streams so far for M3U account: {account_id}") + if valid_stream_count % 1000 == 0: + logger.debug( + f"Processed {valid_stream_count} valid streams so far for M3U account: {account_id}" + ) + else: + logger.debug(f"Processing {len(source)} in-memory M3U lines (zip upload)") + try: + for entry in iter_m3u_entries(source): + valid_stream_count += 1 + group_title_attr = get_case_insensitive_attr(entry["attributes"], "group-title", "") + if group_title_attr and group_title_attr not in groups: + logger.debug(f"Found new group for M3U account {account_id}: '{group_title_attr}'") + groups[group_title_attr] = {} + extinf_data.append(entry) + + if valid_stream_count % 1000 == 0: + logger.debug( + f"Processed {valid_stream_count} valid streams so far for M3U account: {account_id}" + ) + finally: + del source + gc.collect() logger.info(f"M3U parsing complete - Valid streams: {valid_stream_count}") @@ -1865,6 +1948,60 @@ def _classify_sync_failure(exc): return "OTHER" +def rollup_channel_catchup_fields(account_id): + """Roll up catch-up flags from streams to channels (active accounts only). + + Both the aggregate update and the self-heal pass are limited to channels + that still have at least one stream from *account_id*. + """ + from django.db import connection + + account_channels = """ + SELECT DISTINCT cs.channel_id + FROM dispatcharr_channels_channelstream cs + JOIN dispatcharr_channels_stream s ON s.id = cs.stream_id + WHERE s.m3u_account_id = %s + """ + + with connection.cursor() as cur: + cur.execute(f""" + WITH agg AS ( + SELECT + cs.channel_id, + bool_or(s.is_catchup AND a.is_active) AS any_catchup, + MAX(s.catchup_days) FILTER (WHERE s.is_catchup AND a.is_active) AS max_days + FROM dispatcharr_channels_channelstream cs + JOIN dispatcharr_channels_stream s ON s.id = cs.stream_id + JOIN m3u_m3uaccount a ON a.id = s.m3u_account_id + WHERE cs.channel_id IN ({account_channels}) + GROUP BY cs.channel_id + ) + UPDATE dispatcharr_channels_channel c + SET + is_catchup = COALESCE(agg.any_catchup, FALSE), + catchup_days = COALESCE(agg.max_days, 0) + FROM agg + WHERE c.id = agg.channel_id + """, [account_id]) + + # Self-heal stale is_catchup flags on account-linked channels only. + cur.execute(f""" + UPDATE dispatcharr_channels_channel c + SET is_catchup = FALSE, catchup_days = 0 + WHERE c.is_catchup = TRUE + AND c.id IN ({account_channels}) + AND NOT EXISTS ( + SELECT 1 + FROM dispatcharr_channels_channelstream cs + JOIN dispatcharr_channels_stream s ON s.id = cs.stream_id + JOIN m3u_m3uaccount a ON a.id = s.m3u_account_id + WHERE cs.channel_id = c.id + AND s.is_catchup = TRUE + AND a.is_active = TRUE + ) + """, [account_id]) + + @shared_task def sync_auto_channels(account_id, scan_start_time=None): """ @@ -2798,6 +2935,17 @@ def sync_auto_channels(account_id, scan_start_time=None): f"{pack_result['failed']} failed" ) + # Release per-group working sets before the next group iteration. + del ( + current_streams, + logo_cache_by_url, + epg_cache_by_tvg_id, + existing_channel_map, + existing_channels_by_id, + existing_channel_streams, + processed_stream_ids, + ) + # Cleanup mode read from account.custom_properties.orphan_channel_cleanup: # "always" (default; key absent) removes every orphan auto channel; # "preserve_customized" keeps those with a ChannelOverride row; @@ -2833,7 +2981,7 @@ def sync_auto_channels(account_id, scan_start_time=None): f"{channels_created} created, {channels_updated} updated, " f"{channels_deleted} deleted, {channels_failed} failed" ) - return { + result = { "status": "ok", "channels_created": channels_created, "channels_updated": channels_updated, @@ -2841,6 +2989,9 @@ def sync_auto_channels(account_id, scan_start_time=None): "channels_failed": channels_failed, "failed_stream_details": failed_stream_details, } + del failed_stream_details + gc.collect() + return result except Exception as e: logger.error(f"Error in auto channel sync for account {account_id}: {str(e)}") @@ -3224,7 +3375,15 @@ def _refresh_single_m3u_account_impl(account_id): ) account = _get_active_m3u_account(account_id) - filters = list(account.filters.all()) + compiled_stream_filters = _compile_m3u_stream_filters( + account.filters.order_by("order") + ) + if compiled_stream_filters: + logger.debug( + "Account %s has %s stream filter(s) for this refresh", + account_id, + len(compiled_stream_filters), + ) # Check if VOD is enabled for this account vod_enabled = ensure_custom_properties_dict(account.custom_properties).get( @@ -3401,7 +3560,14 @@ def _refresh_single_m3u_account_impl(account_id): with ThreadPoolExecutor(max_workers=max_workers) as executor: # Submit batch processing tasks using direct functions (now thread-safe) future_to_batch = { - executor.submit(process_m3u_batch_direct, account_id, batch, existing_groups, hash_keys): i + executor.submit( + process_m3u_batch_direct, + account_id, + batch, + existing_groups, + hash_keys, + compiled_stream_filters, + ): i for i, batch in enumerate(batches) } @@ -3456,6 +3622,10 @@ def _refresh_single_m3u_account_impl(account_id): batches[batch_idx] = None logger.info(f"Thread-based processing completed for account {account_id}") + + # Parsed catalog is no longer needed; drop before stale cleanup / auto-sync. + del extinf_data, batches + gc.collect() else: # For XC accounts, get the groups with their custom properties containing xc_id logger.debug(f"Processing XC account with groups: {existing_groups}") @@ -3493,6 +3663,8 @@ def _refresh_single_m3u_account_impl(account_id): logger.info("Fetching all XC streams from provider and filtering by enabled categories...") all_xc_streams = collect_xc_streams(account_id, filtered_groups) + del channel_group_relationships, filtered_groups + if not all_xc_streams: logger.warning("No streams collected from XC groups") else: @@ -3514,7 +3686,14 @@ def _refresh_single_m3u_account_impl(account_id): with ThreadPoolExecutor(max_workers=max_workers) as executor: # Submit stream batch processing tasks (reuse standard M3U processing) future_to_batch = { - executor.submit(process_m3u_batch_direct, account_id, batch, existing_groups, hash_keys): i + executor.submit( + process_m3u_batch_direct, + account_id, + batch, + existing_groups, + hash_keys, + compiled_stream_filters, + ): i for i, batch in enumerate(batches) } @@ -3570,6 +3749,9 @@ def _refresh_single_m3u_account_impl(account_id): logger.info(f"XC thread-based processing completed for account {account_id}") + del batches + gc.collect() + # Ensure all database transactions are committed before cleanup logger.info( f"All thread processing completed, ensuring DB transactions are committed before cleanup" @@ -3634,6 +3816,12 @@ def _refresh_single_m3u_account_impl(account_id): f"Error running auto channel sync for account {account_id}: {str(e)}" ) + try: + rollup_channel_catchup_fields(account_id) + logger.debug(f"Catch-up field rollup complete for account {account_id}") + except Exception as e: + logger.error(f"Error rolling up catch-up fields for account {account_id}: {str(e)}") + # Calculate elapsed time elapsed_time = time.time() - start_time @@ -3684,6 +3872,9 @@ def _refresh_single_m3u_account_impl(account_id): message=account.last_message, ) + del auto_sync_result + gc.collect() + # Trigger VOD refresh if enabled and account is XtreamCodes type if vod_enabled and account.account_type == M3UAccount.Types.XC: logger.info(f"VOD is enabled for account {account_id}, triggering VOD refresh") @@ -3715,6 +3906,10 @@ def _refresh_single_m3u_account_impl(account_id): del filtered_groups if 'channel_group_relationships' in locals(): del channel_group_relationships + if 'compiled_stream_filters' in locals(): + del compiled_stream_filters + + gc.collect() # Remove cache file after processing (success or failure) cache_path = os.path.join(m3u_dir, f"{account_id}.json") @@ -3723,8 +3918,6 @@ def _refresh_single_m3u_account_impl(account_id): except OSError: pass - _release_task_db_connection() - return f"Dispatched jobs complete." diff --git a/apps/m3u/tests/test_memory_cleanup.py b/apps/m3u/tests/test_memory_cleanup.py index bfc964d0..b29ca947 100644 --- a/apps/m3u/tests/test_memory_cleanup.py +++ b/apps/m3u/tests/test_memory_cleanup.py @@ -30,7 +30,7 @@ class ProcessM3UBatchCleanupTests(SimpleTestCase): mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123") with patch("django.db.connections") as mock_connections: - process_m3u_batch_direct(1, [], {}, ["name", "url"]) + process_m3u_batch_direct(1, [], {}, ["name", "url"], compiled_filters=[]) mock_connections.close_all.assert_called() @patch("apps.m3u.tasks.Stream") @@ -48,9 +48,29 @@ class ProcessM3UBatchCleanupTests(SimpleTestCase): mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123") with patch("gc.collect") as mock_gc, patch("django.db.connections"): - process_m3u_batch_direct(1, [], {}, ["name", "url"]) + process_m3u_batch_direct(1, [], {}, ["name", "url"], compiled_filters=[]) mock_gc.assert_called() + @patch("apps.m3u.tasks.Stream") + @patch("apps.m3u.tasks.M3UAccount") + def test_precompiled_empty_filters_skip_db_lookup( + self, mock_account_cls, mock_stream_cls, + ): + """When filters are precompiled as empty, batch workers must not query filters.""" + from apps.m3u.tasks import process_m3u_batch_direct + + mock_account = MagicMock() + mock_account_cls.objects.get.return_value = mock_account + mock_stream_cls.objects.filter.return_value.select_related.return_value.only.return_value = ( + [] + ) + mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123") + + with patch("django.db.connections"): + process_m3u_batch_direct(1, [], {}, ["name", "url"], compiled_filters=[]) + + mock_account.filters.order_by.assert_not_called() + class LockReleaseTests(SimpleTestCase): """Verify task lock is released on all exit paths.""" diff --git a/apps/m3u/tests/test_stream_filters.py b/apps/m3u/tests/test_stream_filters.py new file mode 100644 index 00000000..aba1a262 --- /dev/null +++ b/apps/m3u/tests/test_stream_filters.py @@ -0,0 +1,156 @@ +"""Tests for M3U stream filter compilation and batch application.""" +from unittest.mock import MagicMock, patch + +from django.test import SimpleTestCase + +from apps.m3u.tasks import ( + _compile_m3u_stream_filters, + _stream_passes_m3u_filters, + process_m3u_batch_direct, +) + + +class CompileM3UStreamFiltersTests(SimpleTestCase): + def test_compiles_case_insensitive_when_configured(self): + filter_obj = MagicMock() + filter_obj.regex_pattern = "news" + filter_obj.custom_properties = {"case_sensitive": False} + + compiled = _compile_m3u_stream_filters([filter_obj]) + + self.assertEqual(len(compiled), 1) + pattern, _ = compiled[0] + self.assertTrue(pattern.search("NEWS")) + + def test_compiles_case_sensitive_by_default(self): + filter_obj = MagicMock() + filter_obj.regex_pattern = "news" + filter_obj.custom_properties = {} + + compiled = _compile_m3u_stream_filters([filter_obj]) + + pattern, _ = compiled[0] + self.assertIsNone(pattern.search("NEWS")) + self.assertTrue(pattern.search("news")) + + +class StreamPassesM3UFiltersTests(SimpleTestCase): + def _compiled(self, *, filter_type="name", exclude=False, pattern="Adult"): + filter_obj = MagicMock() + filter_obj.filter_type = filter_type + filter_obj.exclude = exclude + filter_obj.regex_pattern = pattern + filter_obj.custom_properties = {} + return _compile_m3u_stream_filters([filter_obj]) + + def test_include_filter_passes_matching_stream(self): + compiled = self._compiled(exclude=False) + self.assertTrue( + _stream_passes_m3u_filters("Adult Channel", "http://x", "News", compiled) + ) + + def test_include_filter_passes_non_matching_stream(self): + """Non-matching streams still pass unless a matching exclude filter hits.""" + compiled = self._compiled(exclude=False, pattern="news") + self.assertTrue( + _stream_passes_m3u_filters("Sports", "http://x", "Sports", compiled) + ) + + def test_exclude_filter_rejects_matching_stream(self): + compiled = self._compiled(exclude=True, pattern="Adult") + self.assertFalse( + _stream_passes_m3u_filters("Adult Channel", "http://x", "News", compiled) + ) + + def test_url_filter_type_targets_url(self): + compiled = self._compiled(filter_type="url", exclude=True, pattern="blocked") + self.assertFalse( + _stream_passes_m3u_filters("OK", "http://blocked.example/live", "News", compiled) + ) + self.assertTrue( + _stream_passes_m3u_filters("blocked name", "http://ok.example/live", "News", compiled) + ) + + def test_group_filter_type_targets_group(self): + compiled = self._compiled(filter_type="group", exclude=True, pattern="Hidden") + self.assertFalse( + _stream_passes_m3u_filters("Channel", "http://x", "Hidden Group", compiled) + ) + + +class ProcessM3UBatchFilterTests(SimpleTestCase): + def _mock_stream_meta(self, mock_stream_cls, max_length=255): + mock_field = MagicMock() + mock_field.max_length = max_length + mock_stream_cls._meta.get_field.return_value = mock_field + + @patch("apps.m3u.tasks._bulk_update_stream_refresh_batches") + @patch("apps.m3u.tasks.Stream") + @patch("apps.m3u.tasks.M3UAccount") + def test_exclude_filter_skips_stream_import( + self, mock_account_cls, mock_stream_cls, mock_bulk_update, + ): + self._mock_stream_meta(mock_stream_cls) + mock_account = MagicMock() + mock_account.account_type = "STD" + mock_account_cls.objects.get.return_value = mock_account + mock_stream_cls.objects.filter.return_value.select_related.return_value.only.return_value = ( + [] + ) + mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123") + + filter_obj = MagicMock() + filter_obj.regex_pattern = "skip-me" + filter_obj.filter_type = "name" + filter_obj.exclude = True + filter_obj.custom_properties = {} + compiled = _compile_m3u_stream_filters([filter_obj]) + + batch = [{ + "name": "skip-me channel", + "url": "http://example/live", + "attributes": {"group-title": "News"}, + "vlc_opts": {}, + }] + + with patch("django.db.connections"): + result = process_m3u_batch_direct( + 1, batch, {"News": 1}, ["name", "url"], compiled_filters=compiled, + ) + + self.assertIn("0 created", result) + mock_stream_cls.objects.bulk_create.assert_not_called() + mock_bulk_update.assert_called_once_with([], [], batch_size=200) + + @patch("apps.m3u.tasks._bulk_update_stream_refresh_batches") + @patch("apps.m3u.tasks.Stream") + @patch("apps.m3u.tasks.M3UAccount") + def test_no_filters_imports_matching_stream( + self, mock_account_cls, mock_stream_cls, mock_bulk_update, + ): + self._mock_stream_meta(mock_stream_cls) + mock_account = MagicMock() + mock_account.account_type = "STD" + mock_account_cls.objects.get.return_value = mock_account + mock_stream_cls.objects.filter.return_value.select_related.return_value.only.return_value = ( + [] + ) + mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123") + mock_stream_cls.objects.bulk_create.return_value = [] + + batch = [{ + "name": "News One", + "url": "http://example/live", + "attributes": {"group-title": "News"}, + "vlc_opts": {}, + }] + + with patch("django.db.connections"), patch( + "apps.m3u.tasks.transaction.atomic", + ): + result = process_m3u_batch_direct( + 1, batch, {"News": 1}, ["name", "url"], compiled_filters=[], + ) + + self.assertIn("1 created", result) + mock_stream_cls.objects.bulk_create.assert_called_once() diff --git a/apps/output/epg.py b/apps/output/epg.py index 626b21aa..d34d209e 100644 --- a/apps/output/epg.py +++ b/apps/output/epg.py @@ -1088,7 +1088,7 @@ def generate_dummy_epg( return xml_lines -def generate_epg(request, profile_name=None, user=None): +def generate_epg(request, profile_name=None, user=None, *, xc_catchup_prev_days=False): """ Dynamically generate an XMLTV (EPG) file using a streaming response. Since the EPG data is stored independently of Channels, we group programmes @@ -1100,11 +1100,16 @@ def generate_epg(request, profile_name=None, user=None): num_days = max(0, min(num_days, 365)) except (ValueError, TypeError): num_days = 0 - try: - prev_days = int(request.GET.get('prev_days', user_custom.get('epg_prev_days', 0))) - prev_days = max(0, min(prev_days, 30)) - except (ValueError, TypeError): - prev_days = 0 + if xc_catchup_prev_days: + from apps.channels.utils import resolve_xc_epg_prev_days + + prev_days = resolve_xc_epg_prev_days(request, user) + else: + try: + prev_days = int(request.GET.get('prev_days', user_custom.get('epg_prev_days', 0))) + prev_days = max(0, min(prev_days, 30)) + except (ValueError, TypeError): + prev_days = 0 use_cached_logos = request.GET.get('cachedlogos', 'true').lower() != 'false' tvg_id_source = request.GET.get('tvg_id_source', 'channel_number').lower() cache_params = ( diff --git a/apps/output/tests.py b/apps/output/tests.py index 30b22a18..25a38d18 100644 --- a/apps/output/tests.py +++ b/apps/output/tests.py @@ -896,3 +896,22 @@ class XcVodSeriesRegressionTests(TestCase): stream = xc_get_vod_streams(self.request, self.user)[0] self.assertEqual(stream["container_extension"], first.container_extension) + + +class GenerateEpgPrevDaysTests(SimpleTestCase): + """Profile EPG keeps legacy prev_days=0 unless URL or user setting says otherwise.""" + + def setUp(self): + self.factory = RequestFactory() + + @patch("apps.output.epg.stream_cached_response") + @patch("apps.output.epg.Channel.objects") + def test_non_xc_epg_defaults_prev_days_to_zero(self, _channels, mock_cache): + from apps.output.epg import generate_epg + + mock_cache.side_effect = lambda cache_key, _source, **_kwargs: cache_key + request = self.factory.get("/epg/") + + cache_key = generate_epg(request, profile_name="test", user=None) + + self.assertIn(":p=0:", cache_key) diff --git a/apps/output/views.py b/apps/output/views.py index 6c4e7c13..a7e34c31 100644 --- a/apps/output/views.py +++ b/apps/output/views.py @@ -11,10 +11,9 @@ from apps.accounts.models import User from dispatcharr.utils import network_access_allowed from django.utils import timezone as django_timezone from django.shortcuts import get_object_or_404 -from datetime import datetime, timedelta +from datetime import datetime, timedelta, timezone as dt_timezone import html import time -from tzlocal import get_localzone from urllib.parse import urlencode import base64 import logging @@ -23,6 +22,7 @@ import os from apps.m3u.utils import calculate_tuner_count from apps.proxy.utils import get_user_active_connections import regex +from core.models import CoreSettings from core.utils import log_system_event, build_absolute_uri_with_port import hashlib from apps.output.epg import generate_epg, generate_dummy_programs @@ -368,6 +368,24 @@ def _xc_allowed_output_formats(user): return ['ts', 'mp4'] +def _build_xc_server_info(request, hostname, port): + """Build XC ``server_info``; keep timezone, ``time_now``, and EPG times in UTC. + + XC clients use ``server_info.timezone`` to interpret EPG start/end strings. + Provider-local conversion happens in the timeshift proxy at request time. + """ + # datetime.timezone.utc, not ZoneInfo("UTC"); avoids mis-set Docker /etc/timezone. + return { + "url": hostname, + "server_protocol": request.scheme, + "port": port, + "timezone": "UTC", + "timestamp_now": int(time.time()), + "time_now": datetime.now(dt_timezone.utc).strftime("%Y-%m-%d %H:%M:%S"), + "process": True, + } + + def xc_get_info(request, full=False): user = xc_get_user(request) @@ -400,15 +418,7 @@ def xc_get_info(request, full=False): "max_connections": str(max_connections), "allowed_output_formats": _xc_allowed_output_formats(user), }, - "server_info": { - "url": hostname, - "server_protocol": request.scheme, - "port": port, - "timezone": get_localzone().key, - "timestamp_now": int(time.time()), - "time_now": datetime.now().strftime("%Y-%m-%d %H:%M:%S"), - "process": True, - }, + "server_info": _build_xc_server_info(request, hostname, port), } if full == True: @@ -534,7 +544,7 @@ def xc_xmltv(request): ) return JsonResponse({'error': 'Unauthorized'}, status=401) - return generate_epg(request, None, user) + return generate_epg(request, None, user, xc_catchup_prev_days=True) def xc_get_live_categories(user): @@ -683,6 +693,14 @@ def _xc_channel_entry(channel, channel_num_map, _get_default_group_id, _logo_url effective_logo = channel.effective_logo_obj effective_group = channel.effective_channel_group_obj group_id = effective_group.id if effective_group else _get_default_group_id() + + if channel.is_catchup: + tv_archive = 1 + tv_archive_duration = channel.catchup_days + else: + tv_archive = 0 + tv_archive_duration = 0 + return { "num": channel_num_int, "name": channel.effective_name, @@ -697,10 +715,10 @@ def _xc_channel_entry(channel, channel_num_map, _get_default_group_id, _logo_url "is_adult": int(channel.is_adult), "category_id": str(group_id), "category_ids": [group_id], - "custom_sid": None, - "tv_archive": 0, + "custom_sid": "", + "tv_archive": tv_archive, "direct_source": "", - "tv_archive_duration": 0, + "tv_archive_duration": tv_archive_duration, } @@ -733,10 +751,12 @@ def xc_get_epg(request, user, short=False): if not channel_id: raise Http404() + try: + resolved_channel_id = int(channel_id) + except (TypeError, ValueError): + raise Http404() + channel = None - # Apply effective-value annotation + hidden-exclusion at every channel - # resolution path so a single channel lookup honors the same visibility - # rules as xc_get_live_streams. def _annotate(qs): return with_effective_values(qs, select_related_fks=True).exclude(hidden_from_output=True) @@ -747,7 +767,7 @@ def xc_get_epg(request, user, short=False): if user_profile_count == 0: # No profile filtering - user sees all channels based on user_level filters = { - "id": channel_id, + "id": resolved_channel_id, "user_level__lte": user.user_level } # Hide adult content if user preference is set @@ -757,7 +777,7 @@ def xc_get_epg(request, user, short=False): else: # User has specific limited profiles assigned filters = { - "id": channel_id, + "id": resolved_channel_id, "channelprofilemembership__enabled": True, "user_level__lte": user.user_level, "channelprofilemembership__channel_profile__in": user.channel_profiles.all() @@ -770,7 +790,7 @@ def xc_get_epg(request, user, short=False): if not channel: raise Http404() else: - channel = _annotate(Channel.objects.filter(id=channel_id).select_related('epg_data__epg_source')).first() + channel = _annotate(Channel.objects.filter(id=resolved_channel_id).select_related('epg_data__epg_source')).first() if not channel: raise Http404() @@ -818,6 +838,8 @@ def xc_get_epg(request, user, short=False): int(channel.effective_channel_number) if channel.effective_channel_number is not None else 0, ) + from apps.channels.utils import resolve_xc_epg_prev_days + limit = int(request.GET.get('limit', 4)) user_custom = user.custom_properties or {} try: @@ -825,12 +847,15 @@ def xc_get_epg(request, user, short=False): num_days = max(0, min(num_days, 365)) except (ValueError, TypeError): num_days = 0 - try: - prev_days = int(request.GET.get('prev_days', user_custom.get('epg_prev_days', 0))) - prev_days = max(0, min(prev_days, 30)) - except (ValueError, TypeError): - prev_days = 0 + prev_days = resolve_xc_epg_prev_days(request, user, auto_detect_fallback=False) now = django_timezone.now() + + # XC catch-up clients expect past programmes when prev_days was not set. + _channel_is_catchup = getattr(channel, "is_catchup", False) + _channel_catchup_days = min(getattr(channel, "catchup_days", 0) or 0, 30) + if _channel_is_catchup and prev_days == 0: + prev_days = _channel_catchup_days + lookback_cutoff = now - timedelta(days=prev_days) forward_cutoff = now + timedelta(days=num_days) if num_days > 0 else None effective_epg_data = channel.effective_epg_data_obj @@ -876,6 +901,13 @@ def xc_get_epg(request, user, short=False): output = {"epg_listings": []} + if _channel_is_catchup: + archive_window = timedelta(days=_channel_catchup_days) + else: + archive_window = None + + _epg_utc = dt_timezone.utc + for program in programs: title = program['title'] if isinstance(program, dict) else program.title description = program['description'] if isinstance(program, dict) else program.description @@ -900,8 +932,8 @@ def xc_get_epg(request, user, short=False): "epg_id": epg_id, "title": base64.b64encode((title or "").encode()).decode(), "lang": "", - "start": start.strftime("%Y-%m-%d %H:%M:%S"), - "end": end.strftime("%Y-%m-%d %H:%M:%S"), + "start": start.astimezone(_epg_utc).strftime("%Y-%m-%d %H:%M:%S"), + "end": end.astimezone(_epg_utc).strftime("%Y-%m-%d %H:%M:%S"), "description": base64.b64encode((description or "").encode()).decode(), "channel_id": str(channel_num_int), "start_timestamp": str(int(start.timestamp())), @@ -909,10 +941,14 @@ def xc_get_epg(request, user, short=False): "stream_id": f"{channel_id}", } - if short == False: - program_output["now_playing"] = 1 if start <= django_timezone.now() <= end else 0 + if archive_window is not None and end < now and end > now - archive_window: + program_output["has_archive"] = 1 + else: program_output["has_archive"] = 0 + if short == False: + program_output["now_playing"] = 1 if start <= now <= end else 0 + output['epg_listings'].append(program_output) return output diff --git a/apps/proxy/live_proxy/channel_status.py b/apps/proxy/live_proxy/channel_status.py index 9bd1d284..300f3b0d 100644 --- a/apps/proxy/live_proxy/channel_status.py +++ b/apps/proxy/live_proxy/channel_status.py @@ -407,6 +407,20 @@ class ChannelStatus: if channel_name: info['channel_name'] = channel_name + info['is_timeshift'] = bool(metadata.get(ChannelMetadataField.IS_TIMESHIFT)) + + for key, field in ( + ('logo_id', ChannelMetadataField.LOGO_ID), + ('m3u_profile_id', ChannelMetadataField.M3U_PROFILE), + ): + raw = metadata.get(field) + if not raw: + continue + try: + info[key] = int(raw) + except (TypeError, ValueError): + pass + stream_id_bytes = metadata.get(ChannelMetadataField.STREAM_ID) if stream_id_bytes: try: diff --git a/apps/proxy/live_proxy/constants.py b/apps/proxy/live_proxy/constants.py index b3839508..414e1a2f 100644 --- a/apps/proxy/live_proxy/constants.py +++ b/apps/proxy/live_proxy/constants.py @@ -54,6 +54,8 @@ class ChannelMetadataField: STREAM_ID = "stream_id" CHANNEL_NAME = "channel_name" STREAM_NAME = "stream_name" + IS_TIMESHIFT = "is_timeshift" + LOGO_ID = "logo_id" # Profile fields STREAM_PROFILE = "stream_profile" diff --git a/apps/proxy/live_proxy/server.py b/apps/proxy/live_proxy/server.py index fd54fc24..6d6a97bc 100644 --- a/apps/proxy/live_proxy/server.py +++ b/apps/proxy/live_proxy/server.py @@ -2079,6 +2079,9 @@ class ProxyServer: if not channel_id: continue + if channel_id.startswith("timeshift_"): + continue + # Get metadata first metadata = self.redis_client.hgetall(key) if not metadata: diff --git a/apps/proxy/utils.py b/apps/proxy/utils.py index 69ee15e4..fc157791 100644 --- a/apps/proxy/utils.py +++ b/apps/proxy/utils.py @@ -1,6 +1,7 @@ import logging from core.utils import RedisClient from apps.proxy.vod_proxy.multi_worker_connection_manager import MultiWorkerVODConnectionManager, get_vod_client_stop_key +from apps.proxy.live_proxy.redis_keys import RedisKeys from core.models import CoreSettings from apps.proxy.live_proxy.services.channel_service import ChannelService @@ -61,6 +62,15 @@ def attempt_stream_termination(user_id, requesting_client_id, active_connections result = ChannelService.stop_client(t['media_id'], t['client_id']) if result.get("status") == "error": logger.warning(f"[stream limits][{requesting_client_id}] Failed to stop client {t['client_id']} on channel {t['media_id']}") + elif t['type'] == 'timeshift': + # Same Redis stop key as live; timeshift generator polls it every 5s. + redis_client = RedisClient.get_client() + if not redis_client: + # Deny the new stream if we cannot stop the old one. + return False + stop_key = RedisKeys.client_stop(t['media_id'], t['client_id']) + redis_client.setex(stop_key, 60, "true") + logger.info(f"[stream limits][{requesting_client_id}] Set stop key for timeshift client {t['client_id']}") else: connection_manager = MultiWorkerVODConnectionManager.get_instance() redis_client = connection_manager.redis_client @@ -106,13 +116,14 @@ def get_user_active_connections(user_id): if user_id is None or (client_user_id and int(client_user_id) == user_id): try: - logger.debug(f"[stream limits] Found LIVE connection for user {user_id} on channel {channel_id} with client ID {client_id}") + conn_type = 'timeshift' if channel_id.startswith('timeshift_') else 'live' + logger.debug(f"[stream limits] Found {conn_type.upper()} connection for user {user_id} on channel {channel_id} with client ID {client_id}") connected_at = float(connected_at) if connected_at else 0 connections.append({ 'media_id': channel_id, 'client_id': client_id, 'connected_at': connected_at, - 'type': 'live', + 'type': conn_type, }) except (ValueError, TypeError): pass @@ -172,6 +183,22 @@ def check_user_stream_limits(user, client_id, media_id=None): logger.debug(f"[stream limits][{client_id}] Same-channel reconnect for {media_id} allowed (ignore_same_channel=True)") return True + # Timeshift sibling range/probe requests share one provider slot per + # session_id. Each distinct client/session still consumes its own slot. + if ignore_same_channel and media_id: + media_id_str = str(media_id) + for conn in active_connections: + if conn.get('type') != 'timeshift': + continue + if conn.get('client_id') != client_id: + continue + conn_media_id = str(conn.get('media_id') or '') + if conn_media_id == media_id_str or conn_media_id.startswith(f"{media_id_str}_"): + logger.debug( + f"[stream limits][{client_id}] Same timeshift session probe for {media_id} allowed (ignore_same_channel=True)" + ) + return True + if user_stream_count >= user.stream_limit: if user_limit_settings.get("terminate_on_limit_exceeded", True) == False: return False @@ -186,3 +213,28 @@ def check_user_stream_limits(user, client_id, media_id=None): return False return True + + +_TS_PACKET_SIZE = 188 +_TS_SYNC_BYTE = 0x47 + + +def find_ts_sync(buf): + """Return byte offset of the first valid MPEG-TS sync chain in *buf*, or -1. + + Args: + buf: Raw bytes from an upstream HTTP response (typically the first 1 KB). + + Returns: + Offset of the first 0x47 byte that starts three consecutive 188-byte + packets, or -1. Used to strip PHP/HTML preamble before streaming. + """ + end = len(buf) - 2 * _TS_PACKET_SIZE + for i in range(0, end): + if ( + buf[i] == _TS_SYNC_BYTE + and buf[i + _TS_PACKET_SIZE] == _TS_SYNC_BYTE + and buf[i + 2 * _TS_PACKET_SIZE] == _TS_SYNC_BYTE + ): + return i + return -1 diff --git a/apps/timeshift/__init__.py b/apps/timeshift/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/apps/timeshift/apps.py b/apps/timeshift/apps.py new file mode 100644 index 00000000..e4ecbfba --- /dev/null +++ b/apps/timeshift/apps.py @@ -0,0 +1,7 @@ +from django.apps import AppConfig + + +class TimeshiftConfig(AppConfig): + default_auto_field = "django.db.models.BigAutoField" + name = "apps.timeshift" + verbose_name = "Timeshift" diff --git a/apps/timeshift/helpers.py b/apps/timeshift/helpers.py new file mode 100644 index 00000000..98d10c36 --- /dev/null +++ b/apps/timeshift/helpers.py @@ -0,0 +1,252 @@ +"""URL builders and timestamp helpers for XC catch-up.""" + +import logging +import re +from collections import namedtuple +from datetime import datetime, timezone +from urllib.parse import quote +from zoneinfo import ZoneInfo + +logger = logging.getLogger(__name__) + +# Credentials for the profile whose pool slot was reserved (not raw account fields). +TimeshiftCredentials = namedtuple( + "TimeshiftCredentials", ("server_url", "username", "password") +) + +DEFAULT_DURATION_MINUTES = 120 +DURATION_BUFFER_MINUTES = 5 +MAX_DURATION_MINUTES = 480 + +# Wall-clock shapes seen from XC / iPlayTV / TiviMate clients. Compiled once. +_CATCHUP_WALL_CLOCK_RE = re.compile( + r"^" + r"(?P\d{4}-\d{2}-\d{2})" + r"(?P[:_]| )" + r"(?P\d{2})" + r"(?P[-:])" + r"(?P\d{2})" + r"(?:" + r":" + r"(?P\d{2})" + r")?" + r"$" +) + + +def normalize_catchup_timestamp_input(timestamp_str): + """Map a client catch-up timestamp to an ISO-8601 string for ``fromisoformat``. + + Supported inputs: + - ``YYYY-MM-DD:HH-MM`` (iPlayTV/TiviMate colon-dash) + - ``YYYY-MM-DD_HH-MM`` (XC underscore) + - ``YYYY-MM-DD:HH:MM[:SS]`` (XC colon time in catch-up URLs) + - ``YYYY-MM-DD HH:MM[:SS]`` (EPG / SQL datetime) + - Unix epoch seconds (10 digits) or milliseconds (13 digits) + + Returns: + An ISO-8601 date-time string (``YYYY-MM-DDTHH:MM:SS``), or None if + the value does not match a known catch-up shape. + """ + if timestamp_str is None: + return None + if not isinstance(timestamp_str, str): + timestamp_str = str(timestamp_str) + value = timestamp_str.strip() + if not value: + return None + + if value.isdigit(): + length = len(value) + if length == 10: + dt = datetime.fromtimestamp(int(value), tz=timezone.utc) + return dt.replace(tzinfo=None).isoformat(timespec="seconds") + if length == 13: + dt = datetime.fromtimestamp(int(value) / 1000, tz=timezone.utc) + return dt.replace(tzinfo=None).isoformat(timespec="seconds") + return None + + match = _CATCHUP_WALL_CLOCK_RE.match(value) + if not match: + return None + + parts = match.groupdict() + second = parts["second"] or "00" + return f"{parts['date']}T{parts['hour']}:{parts['minute']}:{second}" + + +def parse_catchup_timestamp(timestamp_str): + """Parse a catch-up timestamp string into a naive UTC wall-clock datetime. + + See ``normalize_catchup_timestamp_input`` for supported input shapes. + + Returns: + A naive datetime on success, or None. + """ + iso_value = normalize_catchup_timestamp_input(timestamp_str) + if iso_value is None: + if timestamp_str is not None and str(timestamp_str).strip(): + logger.debug( + "Timeshift: unrecognised catch-up timestamp: %r", timestamp_str + ) + return None + try: + return datetime.fromisoformat(iso_value) + except ValueError: + logger.debug( + "Timeshift: invalid catch-up timestamp after normalize: %r -> %r", + timestamp_str, + iso_value, + ) + return None + + +def _reshape_timestamp(timestamp, strftime_fmt, label): + dt = parse_catchup_timestamp(timestamp) + if dt is None: + logger.error( + "Timeshift %s reshape failed for %r: unrecognised format", label, timestamp + ) + return timestamp + return dt.strftime(strftime_fmt) + + +def convert_timestamp_to_provider_tz(timestamp_str, provider_tz_name): + """Convert a UTC catch-up timestamp to the provider's local zone. + + Args: + timestamp_str: UTC wall-clock in ``YYYY-MM-DD:HH-MM`` or underscore form. + provider_tz_name: IANA zone from the provider's ``server_info.timezone`` + (e.g. ``Europe/Brussels``). Falsy, ``UTC``, or unknown: no conversion. + + Returns: + ``YYYY-MM-DD:HH-MM`` in the provider zone, or the input unchanged on skip/failure. + """ + if not provider_tz_name or provider_tz_name == "UTC": + return timestamp_str + dt = parse_catchup_timestamp(timestamp_str) + if dt is None: + return timestamp_str + try: + target = ZoneInfo(provider_tz_name) + except Exception: + logger.warning( + "Timeshift: unknown provider timezone %r, no conversion applied", + provider_tz_name, + ) + return timestamp_str + # timezone.utc, not ZoneInfo("UTC"): avoids mis-set Docker /etc/timezone. + local_dt = dt.replace(tzinfo=timezone.utc).astimezone(target) + return local_dt.strftime("%Y-%m-%d:%H-%M") + + +def get_programme_duration(channel, timestamp_str): + """Look up catch-up duration in minutes from EPG. + + Args: + channel: Channel with optional ``epg_data`` relation loaded. + timestamp_str: Programme start in UTC (same shape as the client URL). + + Returns: + Programme length plus a small buffer, capped at ``MAX_DURATION_MINUTES``, + or ``DEFAULT_DURATION_MINUTES`` when EPG lookup fails. + """ + try: + dt = parse_catchup_timestamp(timestamp_str) + if dt is None: + return DEFAULT_DURATION_MINUTES + # EPG times are timezone-aware; parsed value must be too. + dt = dt.replace(tzinfo=timezone.utc) + if not channel.epg_data: + return DEFAULT_DURATION_MINUTES + + programme = channel.epg_data.programs.filter( + start_time__lte=dt, end_time__gt=dt + ).first() + if not programme: + return DEFAULT_DURATION_MINUTES + + duration_seconds = (programme.end_time - programme.start_time).total_seconds() + duration_minutes = int(duration_seconds / 60) + DURATION_BUFFER_MINUTES + return min(duration_minutes, MAX_DURATION_MINUTES) + except Exception: + return DEFAULT_DURATION_MINUTES + + +def build_timeshift_url_format_a(creds, stream_id, timestamp, duration_minutes): + """QUERY layout: ``/streaming/timeshift.php?username=...&start=...``""" + return ( + f"{creds.server_url.rstrip('/')}/streaming/timeshift.php" + f"?username={quote(str(creds.username), safe='')}" + f"&password={quote(str(creds.password), safe='')}" + f"&stream={stream_id}" + f"&start={timestamp}" + f"&duration={duration_minutes}" + ) + + +def build_timeshift_url_format_b(creds, stream_id, timestamp, duration_minutes): + """PATH layout: ``/timeshift/{user}/{pass}/{dur}/{start}/{id}.ts``""" + return ( + f"{creds.server_url.rstrip('/')}/timeshift" + f"/{quote(str(creds.username), safe='')}" + f"/{quote(str(creds.password), safe='')}" + f"/{duration_minutes}" + f"/{timestamp}" + f"/{stream_id}.ts" + ) + + +def build_timeshift_candidate_urls(creds, stream_id, timestamp, duration_minutes): + """Build ordered upstream URL candidates (PATH forms first, QUERY last). + + Args: + creds: ``TimeshiftCredentials`` for the reserved profile. + stream_id: Provider stream id from the catch-up stream's custom properties. + timestamp: Already converted to the serving provider's local zone. + duration_minutes: Archive window length passed to the provider. + + Returns: + List of URL strings to try in order. QUERY forms are last because some + providers return live TV even when ``start`` is set. + """ + dt = parse_catchup_timestamp(timestamp) + if dt is None: + colon_dash_ts = timestamp + underscore_ts = timestamp + colon_seconds_ts = timestamp + sql_ts = timestamp + else: + colon_dash_ts = dt.strftime("%Y-%m-%d:%H-%M") + underscore_ts = dt.strftime("%Y-%m-%d_%H-%M") + colon_seconds_ts = dt.strftime("%Y-%m-%d:%H:%M:%S") + sql_ts = dt.strftime("%Y-%m-%d %H:%M:%S") + return [ + build_timeshift_url_format_b(creds, stream_id, colon_dash_ts, duration_minutes), + build_timeshift_url_format_b(creds, stream_id, underscore_ts, duration_minutes), + build_timeshift_url_format_b(creds, stream_id, colon_seconds_ts, duration_minutes), + build_timeshift_url_format_a(creds, stream_id, underscore_ts, duration_minutes), + build_timeshift_url_format_a(creds, stream_id, sql_ts, duration_minutes), + build_timeshift_url_format_a(creds, stream_id, colon_dash_ts, duration_minutes), + build_timeshift_url_format_a(creds, stream_id, colon_seconds_ts, duration_minutes), + ] + + +def format_timestamp_as_colon_dash(timestamp): + """Reshape to ``YYYY-MM-DD:HH-MM`` without timezone conversion.""" + return _reshape_timestamp(timestamp, "%Y-%m-%d:%H-%M", "colon-dash") + + +def format_timestamp_as_colon_seconds(timestamp): + """Reshape to ``YYYY-MM-DD:HH:MM:SS`` without timezone conversion.""" + return _reshape_timestamp(timestamp, "%Y-%m-%d:%H:%M:%S", "colon-seconds") + + +def format_timestamp_as_underscore(timestamp): + """Reshape to ``YYYY-MM-DD_HH-MM`` without timezone conversion.""" + return _reshape_timestamp(timestamp, "%Y-%m-%d_%H-%M", "underscore") + + +def format_timestamp_as_sql_datetime(timestamp): + """Reshape to ``YYYY-MM-DD HH:MM:SS`` without timezone conversion.""" + return _reshape_timestamp(timestamp, "%Y-%m-%d %H:%M:%S", "SQL") diff --git a/apps/timeshift/tests/__init__.py b/apps/timeshift/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/apps/timeshift/tests/test_helpers.py b/apps/timeshift/tests/test_helpers.py new file mode 100644 index 00000000..4bd7e0e3 --- /dev/null +++ b/apps/timeshift/tests/test_helpers.py @@ -0,0 +1,325 @@ +"""Tests for `apps.timeshift.helpers`: timestamp shape conversion and URL build.""" + +from datetime import datetime, timezone + +from django.test import TestCase + +from apps.timeshift.helpers import ( + TimeshiftCredentials, + build_timeshift_candidate_urls, + build_timeshift_url_format_a, + build_timeshift_url_format_b, + convert_timestamp_to_provider_tz, + format_timestamp_as_colon_dash, + format_timestamp_as_colon_seconds, + format_timestamp_as_sql_datetime, + format_timestamp_as_underscore, + normalize_catchup_timestamp_input, + parse_catchup_timestamp, +) + + +def _make_creds(): + # The builders consume resolved per-profile credentials, never an account + # object — get_transformed_credentials() produces these in the view. + return TimeshiftCredentials("http://example.test", "user", "pass") + + +class TimestampFormatTests(TestCase): + """Timestamp reshape functions change format only; no timezone conversion.""" + + def test_normalize_colon_dash_shape(self): + self.assertEqual( + normalize_catchup_timestamp_input("2026-05-21:12-55"), + "2026-05-21T12:55:00", + ) + + def test_normalize_colon_seconds_xc_format(self): + self.assertEqual( + normalize_catchup_timestamp_input("2026-06-23:04:00:00"), + "2026-06-23T04:00:00", + ) + + def test_normalize_epg_sql_format(self): + self.assertEqual( + normalize_catchup_timestamp_input("2026-06-23 04:00:00"), + "2026-06-23T04:00:00", + ) + + def test_normalize_unix_epoch_seconds(self): + epoch = str(int(datetime(2026, 6, 23, 4, 0, 0, tzinfo=timezone.utc).timestamp())) + self.assertEqual( + normalize_catchup_timestamp_input(epoch), + "2026-06-23T04:00:00", + ) + + def test_normalize_unix_epoch_milliseconds(self): + epoch_ms = str( + int(datetime(2026, 6, 23, 4, 0, 0, tzinfo=timezone.utc).timestamp() * 1000) + ) + self.assertEqual( + normalize_catchup_timestamp_input(epoch_ms), + "2026-06-23T04:00:00", + ) + + def test_normalize_rejects_garbage(self): + self.assertIsNone(normalize_catchup_timestamp_input("garbage")) + self.assertIsNone(normalize_catchup_timestamp_input("")) + self.assertIsNone(normalize_catchup_timestamp_input("12345")) + + def test_parse_rejects_invalid_calendar_date(self): + self.assertIsNone(parse_catchup_timestamp("2026-13-45:04-00")) + + def test_parse_colon_dash_format(self): + dt = parse_catchup_timestamp("2026-05-21:12-55") + self.assertEqual(dt, datetime(2026, 5, 21, 12, 55, 0)) + + def test_parse_underscore_format(self): + dt = parse_catchup_timestamp("2026-05-21_12-55") + self.assertEqual(dt, datetime(2026, 5, 21, 12, 55, 0)) + + def test_parse_colon_minutes_without_seconds(self): + dt = parse_catchup_timestamp("2026-06-23:04:00") + self.assertEqual(dt, datetime(2026, 6, 23, 4, 0, 0)) + + def test_parse_colon_seconds_xc_format(self): + dt = parse_catchup_timestamp("2026-06-23:04:00:00") + self.assertEqual(dt, datetime(2026, 6, 23, 4, 0, 0)) + + def test_parse_epg_sql_format(self): + dt = parse_catchup_timestamp("2026-06-23 04:00:00") + self.assertEqual(dt, datetime(2026, 6, 23, 4, 0, 0)) + + def test_format_colon_dash_from_colon_seconds(self): + self.assertEqual( + format_timestamp_as_colon_dash("2026-06-23:04:00:00"), + "2026-06-23:04-00", + ) + + def test_format_colon_seconds_from_colon_dash(self): + self.assertEqual( + format_timestamp_as_colon_seconds("2026-06-23:04-00"), + "2026-06-23:04:00:00", + ) + + def test_format_colon_seconds_from_unix_epoch(self): + epoch = str(int(datetime(2026, 6, 23, 4, 0, 0, tzinfo=timezone.utc).timestamp())) + self.assertEqual( + format_timestamp_as_colon_dash(epoch), + "2026-06-23:04-00", + ) + + def test_format_sql_reshapes_without_tz_conversion(self): + self.assertEqual( + format_timestamp_as_sql_datetime("2026-05-12:17-00"), + "2026-05-12 17:00:00", + ) + + def test_format_sql_accepts_underscore_input(self): + self.assertEqual( + format_timestamp_as_sql_datetime("2026-05-12_17-00"), + "2026-05-12 17:00:00", + ) + + def test_format_sql_invalid_falls_back(self): + self.assertEqual(format_timestamp_as_sql_datetime("garbage"), "garbage") + + def test_format_underscore_from_colon_dash(self): + self.assertEqual( + format_timestamp_as_underscore("2026-05-21:12-55"), + "2026-05-21_12-55", + ) + + def test_format_underscore_idempotent(self): + # Underscore input → underscore output (no change) + self.assertEqual( + format_timestamp_as_underscore("2026-05-21_12-55"), + "2026-05-21_12-55", + ) + + def test_format_underscore_invalid_falls_back(self): + self.assertEqual(format_timestamp_as_underscore("garbage"), "garbage") + + +class BuildTimeshiftUrlTests(TestCase): + def setUp(self): + self.creds = _make_creds() + + def test_format_a_passes_dash_shape_unchanged(self): + url = build_timeshift_url_format_a( + self.creds, "22372", "2026-05-12:19-00", 40 + ) + self.assertIn("start=2026-05-12:19-00", url) + self.assertIn("stream=22372", url) + self.assertIn("duration=40", url) + + def test_format_a_passes_sql_shape_unchanged(self): + url = build_timeshift_url_format_a( + self.creds, "22372", "2026-05-12 19:00:00", 40 + ) + self.assertIn("start=2026-05-12 19:00:00", url) + + def test_format_b_path_with_dash_shape(self): + url = build_timeshift_url_format_b( + self.creds, "22372", "2026-05-12:19-00", 40 + ) + self.assertIn("/40/2026-05-12:19-00/22372.ts", url) + + +class CandidateOrderingTests(TestCase): + """`build_timeshift_candidate_urls` must try the PATH form (which seeks the + archive) before the QUERY form (which returns LIVE on some providers, + silently ignoring the requested timestamp). Regression guard for the + "catch-up plays the live stream instead of the requested programme" bug.""" + + def setUp(self): + self.creds = _make_creds() + + def _is_path_form(self, url): + return "/timeshift/" in url and url.endswith(".ts") and "timeshift.php" not in url + + def _is_query_form(self, url): + return "timeshift.php?" in url + + def test_every_path_candidate_precedes_every_query_candidate(self): + urls = build_timeshift_candidate_urls(self.creds, "22372", "2026-05-12:19-00", 40) + path_indices = [i for i, u in enumerate(urls) if self._is_path_form(u)] + query_indices = [i for i, u in enumerate(urls) if self._is_query_form(u)] + # Each URL is classified as exactly one form. + self.assertEqual(len(path_indices) + len(query_indices), len(urls)) + self.assertTrue(path_indices and query_indices) + # The last PATH candidate still comes before the first QUERY candidate. + self.assertLess(max(path_indices), min(query_indices)) + + def test_first_candidate_is_path_form_with_canonical_dash_timestamp(self): + urls = build_timeshift_candidate_urls(self.creds, "22372", "2026-05-12:19-00", 40) + self.assertTrue(self._is_path_form(urls[0])) + # Canonical colon-dash timestamp, passed through unchanged. + self.assertIn("/40/2026-05-12:19-00/22372.ts", urls[0]) + + def test_accepts_colon_seconds_input_timestamp(self): + urls = build_timeshift_candidate_urls( + self.creds, "22372", "2026-06-23:04:00:00", 40 + ) + self.assertTrue(self._is_path_form(urls[0])) + self.assertIn("/40/2026-06-23:04-00/22372.ts", urls[0]) + self.assertIn("/40/2026-06-23:04:00:00/22372.ts", urls[2]) + + def test_accepts_underscore_input_timestamp(self): + # Client may send the underscore shape; PATH form still leads. + urls = build_timeshift_candidate_urls(self.creds, "22372", "2026-05-12_19-00", 40) + self.assertTrue(self._is_path_form(urls[0])) + + +class ConvertTimestampToProviderTzTests(TestCase): + """`convert_timestamp_to_provider_tz` shifts a UTC catch-up timestamp into the + serving provider's local zone (XC providers index archives in their own zone), + DST-correct, and is a no-op when the zone is UTC/unknown/missing.""" + + def test_utc_to_brussels_summer_is_plus_two(self): + # June → CEST (+02:00): 17:00 UTC == 19:00 Brussels (the 19h JT case). + self.assertEqual( + convert_timestamp_to_provider_tz("2026-06-08:17-00", "Europe/Brussels"), + "2026-06-08:19-00", + ) + + def test_utc_to_brussels_winter_is_plus_one(self): + # January → CET (+01:00): 17:00 UTC == 18:00 Brussels (DST handled). + self.assertEqual( + convert_timestamp_to_provider_tz("2026-01-08:17-00", "Europe/Brussels"), + "2026-01-08:18-00", + ) + + def test_day_rollover(self): + # 23:30 UTC + 2h (CEST) crosses midnight into the next day. + self.assertEqual( + convert_timestamp_to_provider_tz("2026-06-08:23-30", "Europe/Brussels"), + "2026-06-09:01-30", + ) + + def test_underscore_input_returns_colon_dash(self): + self.assertEqual( + convert_timestamp_to_provider_tz("2026-06-08_17-00", "Europe/Brussels"), + "2026-06-08:19-00", + ) + + def test_utc_zone_is_noop(self): + self.assertEqual( + convert_timestamp_to_provider_tz("2026-06-08:17-00", "UTC"), + "2026-06-08:17-00", + ) + + def test_none_zone_is_noop(self): + self.assertEqual( + convert_timestamp_to_provider_tz("2026-06-08:17-00", None), + "2026-06-08:17-00", + ) + + def test_unknown_zone_is_noop(self): + self.assertEqual( + convert_timestamp_to_provider_tz("2026-06-08:17-00", "Mars/Phobos"), + "2026-06-08:17-00", + ) + + def test_utc_to_brussels_from_unix_epoch(self): + epoch = str(int(datetime(2026, 6, 8, 17, 0, 0, tzinfo=timezone.utc).timestamp())) + self.assertEqual( + convert_timestamp_to_provider_tz(epoch, "Europe/Brussels"), + "2026-06-08:19-00", + ) + + def test_garbage_timestamp_passthrough(self): + self.assertEqual( + convert_timestamp_to_provider_tz("garbage", "Europe/Brussels"), + "garbage", + ) + + +class GetProgrammeDurationTests(TestCase): + """Duration window resolution: programme length + buffer, capped, with a + safe default whenever the EPG lookup cannot resolve.""" + + def _channel_with_programme(self, minutes): + from datetime import datetime, timedelta, timezone as dt_timezone + from unittest.mock import MagicMock + + start = datetime(2026, 6, 8, 17, 0, tzinfo=dt_timezone.utc) + programme = MagicMock( + start_time=start, end_time=start + timedelta(minutes=minutes) + ) + channel = MagicMock() + channel.epg_data.programs.filter.return_value.first.return_value = programme + return channel + + def test_duration_is_programme_length_plus_buffer(self): + from apps.timeshift.helpers import get_programme_duration + # 40-minute programme + 5-minute buffer. + self.assertEqual( + get_programme_duration(self._channel_with_programme(40), "2026-06-08:17-00"), + 45, + ) + + def test_duration_capped_at_max(self): + from apps.timeshift.helpers import get_programme_duration + self.assertEqual( + get_programme_duration(self._channel_with_programme(1000), "2026-06-08:17-00"), + 480, + ) + + def test_no_epg_data_falls_back_to_default(self): + from unittest.mock import MagicMock + from apps.timeshift.helpers import get_programme_duration + channel = MagicMock(epg_data=None) + self.assertEqual(get_programme_duration(channel, "2026-06-08:17-00"), 120) + + def test_no_matching_programme_falls_back_to_default(self): + from unittest.mock import MagicMock + from apps.timeshift.helpers import get_programme_duration + channel = MagicMock() + channel.epg_data.programs.filter.return_value.first.return_value = None + self.assertEqual(get_programme_duration(channel, "2026-06-08:17-00"), 120) + + def test_garbage_timestamp_falls_back_to_default(self): + from unittest.mock import MagicMock + from apps.timeshift.helpers import get_programme_duration + self.assertEqual(get_programme_duration(MagicMock(), "garbage"), 120) diff --git a/apps/timeshift/tests/test_views.py b/apps/timeshift/tests/test_views.py new file mode 100644 index 00000000..1e45f53b --- /dev/null +++ b/apps/timeshift/tests/test_views.py @@ -0,0 +1,2050 @@ +"""Tests for the timeshift proxy view, focused on upstream status mapping.""" + +import fnmatch +import time +from unittest.mock import MagicMock, patch + +from django.test import RequestFactory, TestCase, override_settings + +from apps.timeshift import views +from apps.proxy.utils import check_user_stream_limits as _check_user_stream_limits +from apps.proxy.utils import find_ts_sync as _find_ts_sync + +TEST_SESSION_ID = "timeshift_testsession1" +TEST_MEDIA_ID = "timeshift_8_2026-06-08-17-00" + + +def _proxy_url(session_id=TEST_SESSION_ID): + base = "/timeshift/u/p/8/2026-06-08:17-00/8.ts" + return f"{base}?session_id={session_id}" if session_id else base + + +def _seed_pool_session( + redis, + session_id=TEST_SESSION_ID, + media_id=TEST_MEDIA_ID, + *, + busy="1", + serving_range=None, + user_id=5, + client_ip="1.2.3.4", + client_user_agent="test-agent", +): + views._create_pool_session( + redis, + session_id=session_id, + media_id=media_id, + user_id=user_id, + client_ip=client_ip, + client_user_agent=client_user_agent, + account_id=1, + profile_id=31, + stream_id="111", + provider_timestamp="2026-06-08:19-00", + ) + if serving_range is not None: + redis.hset(f"timeshift_pool:{session_id}", "serving_range", serving_range) + if busy is not None: + redis.hset(f"timeshift_pool:{session_id}", "busy", busy) + + +class FindTsSyncTests(TestCase): + """Locate the first MPEG-TS sync chain so a leading HTML/PHP preamble + can be skipped before the bytes reach the strict demuxer (ExoPlayer).""" + + def test_returns_zero_when_buffer_already_aligned(self): + buf = b"\x47" + b"\x00" * 187 + b"\x47" + b"\x00" * 187 + b"\x47" + b"\x00" * 187 + self.assertEqual(_find_ts_sync(buf), 0) + + def test_returns_offset_of_first_chain_after_preamble(self): + preamble = b"
\nWarning" + aligned = b"\x47" + b"\x00" * 187 + b"\x47" + b"\x00" * 187 + b"\x47" + b"\x00" * 187 + self.assertEqual(_find_ts_sync(preamble + aligned), len(preamble)) + + def test_returns_minus_one_when_no_chain_exists(self): + # Three lone 0x47 bytes that are NOT spaced at 188 — must not be + # mistaken for a sync chain. + self.assertEqual(_find_ts_sync(b"\x47\x00\x00\x47\x00\x00\x47" * 50), -1) + + def test_returns_minus_one_for_short_buffer(self): + self.assertEqual(_find_ts_sync(b"\x47" * 10), -1) + + + +def _make_ts_payload(size=1024): + """Build a minimal valid MPEG-TS byte sequence with 0x47 sync markers.""" + packet = b"\x47" + b"\x00" * 187 + return (packet * ((size // 188) + 1))[:size] + + +def _fake_upstream(status_code, *, content_type="video/mp2t", body=b""): + resp = MagicMock() + resp.status_code = status_code + resp.headers = {"Content-Type": content_type} + resp.iter_content = MagicMock(return_value=iter([body] if body else [])) + resp.close = MagicMock() + # Simulate raw.read() for the TS sync peek in _stream_from_provider. + # For 200 responses, return valid TS bytes so the peek check passes. + if status_code in (200, 206) and not body: + ts_peek = _make_ts_payload() + resp.raw = MagicMock() + resp.raw.read = MagicMock(return_value=ts_peek) + elif status_code in (200, 206): + resp.raw = MagicMock() + resp.raw.read = MagicMock(return_value=body) + return resp + + +class StreamFromProviderStatusMappingTests(TestCase): + """`_stream_from_provider` must translate upstream HTTP status codes into + semantically correct Django responses so downstream IPTV clients react + the right way (notably: stop retrying on 404).""" + + def setUp(self): + self.factory = RequestFactory() + self.kwargs = dict( + candidate_urls=[ + "http://example.test/streaming/timeshift.php?stream=1&start=2026-05-12:17-00", + "http://example.test/streaming/timeshift.php?stream=1&start=2026-05-12 17:00:00", + "http://example.test/timeshift/u/p/60/2026-05-12:17-00/1.ts", + ], + user_agent="test-agent", + range_header=None, + virtual_channel_id="timeshift_1_2026-05-12-17-00_1", + client_id="timeshift_test123", + client_ip="127.0.0.1", + user=None, + channel_display_name="Test", + timestamp_utc="2026-05-12:17-00", + channel_logo_id=None, + m3u_profile_id=None, + debug=False, + ) + + @patch.object(views, "_open_upstream") + def test_all_candidates_404_returns_404(self, mocked_open): + mocked_open.return_value = _fake_upstream(404) + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 404) + # Every candidate is attempted before giving up. + self.assertEqual(mocked_open.call_count, 3) + + @patch.object(views, "_open_upstream") + def test_upstream_403_short_circuits_loop(self, mocked_open): + # 403 is decisive (auth) — no retry of further candidates. + mocked_open.return_value = _fake_upstream(403) + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 403) + self.assertEqual(mocked_open.call_count, 1) + + @patch.object(views, "_open_upstream") + def test_upstream_302_short_circuits_loop(self, mocked_open): + # Any 3xx is decisive: for XC providers a 302 is the first sign of + # an IP ban, so the cascade must STOP hammering immediately instead + # of retrying other URL shapes (which escalates the ban). + mocked_open.return_value = _fake_upstream(302) + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 400) + self.assertEqual(mocked_open.call_count, 1) + + @patch.object(views, "_open_upstream") + def test_upstream_500_continues_to_next_candidate(self, mocked_open): + # A 5xx is format-specific on many XC servers (PHP fatal with + # display_errors off turns an "Undefined array key" warning into a + # hard 500), so the cascade must keep trying — the next timestamp + # shape often succeeds. Regression: providers that 500 on the first + # shape used to fail outright because the loop short-circuited. + mocked_open.side_effect = [ + _fake_upstream(500), + _fake_upstream(200, body=_make_ts_payload()), + ] + with patch.object(views, "RedisClient"), \ + patch.object(views, "_register_stats_client"), \ + patch.object(views, "_unregister_stats_client"): + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 200) + self.assertEqual(mocked_open.call_count, 2) + + @patch.object(views, "_open_upstream") + def test_all_candidates_500_returns_error(self, mocked_open): + # Every shape 500s → all candidates attempted, then a clean error. + mocked_open.return_value = _fake_upstream(500) + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 400) + self.assertEqual(mocked_open.call_count, 3) + + @patch.object(views, "_open_upstream") + def test_first_candidate_succeeds(self, mocked_open): + mocked_open.side_effect = [_fake_upstream(200, body=_make_ts_payload())] + with patch.object(views, "RedisClient"), \ + patch.object(views, "_register_stats_client"), \ + patch.object(views, "_unregister_stats_client"): + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 200) + self.assertEqual(mocked_open.call_count, 1) + + @patch.object(views, "_open_upstream") + def test_second_candidate_succeeds_after_404(self, mocked_open): + # Primary 404 → second candidate 200 → streams successfully. + mocked_open.side_effect = [ + _fake_upstream(404), + _fake_upstream(200, body=_make_ts_payload()), + ] + with patch.object(views, "RedisClient"), \ + patch.object(views, "_register_stats_client"), \ + patch.object(views, "_unregister_stats_client"): + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 200) + self.assertEqual(mocked_open.call_count, 2) + + @patch.object(views, "_open_upstream") + def test_third_candidate_succeeds_after_400_then_404(self, mocked_open): + mocked_open.side_effect = [ + _fake_upstream(400), + _fake_upstream(404), + _fake_upstream(200, body=_make_ts_payload()), + ] + with patch.object(views, "RedisClient"), \ + patch.object(views, "_register_stats_client"), \ + patch.object(views, "_unregister_stats_client"): + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 200) + self.assertEqual(mocked_open.call_count, 3) + + @override_settings(CACHES={ + "default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"} + }) + @patch.object(views, "_open_upstream") + def test_cache_promotes_winning_index_to_first(self, mocked_open): + """Once a candidate succeeds for an account, the next request reorders + the list so the cached winner is tried first — saving cascade + overhead on fast-forward.""" + # locmem cache: isolates this test from the shared Redis-backed django + # cache (which persists across runs and parallel test sessions). + from django.core.cache import cache as django_cache + django_cache.delete(views._FORMAT_CACHE_KEY.format(999)) + + # First request: candidate index 1 wins after index 0 returns 404. + mocked_open.side_effect = [ + _fake_upstream(404), + _fake_upstream(200, body=_make_ts_payload()), + ] + kwargs = dict(self.kwargs, account_id=999) + with patch.object(views, "RedisClient"), \ + patch.object(views, "_register_stats_client"), \ + patch.object(views, "_unregister_stats_client"): + r1 = views._stream_from_provider(**kwargs) + self.assertEqual(r1.status_code, 200) + self.assertEqual(mocked_open.call_count, 2) + + # Second request: cached winner (index 1) is tried first, succeeds + # immediately — no cascade. + mocked_open.reset_mock() + mocked_open.side_effect = [_fake_upstream(200, body=_make_ts_payload())] + with patch.object(views, "RedisClient"), \ + patch.object(views, "_register_stats_client"), \ + patch.object(views, "_unregister_stats_client"): + r2 = views._stream_from_provider(**kwargs) + self.assertEqual(r2.status_code, 200) + self.assertEqual(mocked_open.call_count, 1) + # Confirm the URL used is the SQL-datetime candidate (index 1 in the + # original list set up in setUp), not the dash-only one (index 0). + self.assertIn("17:00:00", mocked_open.call_args_list[0][0][0]) + + @patch.object(views, "_open_upstream") + def test_php_error_200_cascades_to_next_candidate(self, mocked_open): + """When the provider returns HTTP 200 but the body is PHP error text + (no TS sync), the cascade should try the next candidate URL.""" + php_error = b'
\nWarning: Invalid argument supplied for foreach()' + php_resp = _fake_upstream(200, body=php_error) + php_resp.raw = MagicMock() + php_resp.raw.read = MagicMock(return_value=php_error) + + ts_resp = _fake_upstream(200, body=_make_ts_payload()) + + mocked_open.side_effect = [php_resp, ts_resp] + with patch.object(views, "RedisClient"), \ + patch.object(views, "_register_stats_client"), \ + patch.object(views, "_unregister_stats_client"): + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 200) + # PHP response was rejected, second candidate accepted + self.assertEqual(mocked_open.call_count, 2) + + @patch.object(views, "_open_upstream") + def test_416_range_not_satisfiable_passes_through(self, mocked_open): + # A tail/seek probe past EOF must go back to the client verbatim, + # never cascaded to other URL shapes (byte offsets are file-specific, + # so cascading only multiplies upstream connections). + resp = _fake_upstream(416) + resp.headers = {"Content-Type": "video/mp2t", "Content-Range": "bytes */1000"} + mocked_open.return_value = resp + kwargs = dict(self.kwargs, range_header="bytes=999999-") + response = views._stream_from_provider(**kwargs) + self.assertEqual(response.status_code, 416) + self.assertEqual(response["Content-Range"], "bytes */1000") + self.assertTrue(getattr(response, "timeshift_passthrough", False)) + # No cascade: the first (and only) candidate decided the outcome. + self.assertEqual(mocked_open.call_count, 1) + + @patch.object(views, "_open_upstream") + def test_partial_206_to_range_request_accepted_mid_packet(self, mocked_open): + # A 206 answering a Range request legitimately starts mid-TS-packet + # (no 0x47 sync at offset 0). It must be served, not rejected as a + # PHP error and cascaded across every URL shape and provider account. + mid_packet = b"\x00" * 300 + self.assertEqual(_find_ts_sync(mid_packet), -1) + resp = _fake_upstream(206, body=mid_packet) + resp.raw = MagicMock() + resp.raw.read = MagicMock(return_value=mid_packet) + mocked_open.return_value = resp + kwargs = dict(self.kwargs, range_header="bytes=1000-") + with patch.object(views, "RedisClient"), \ + patch.object(views, "_register_stats_client"), \ + patch.object(views, "_unregister_stats_client"): + response = views._stream_from_provider(**kwargs) + self.assertEqual(response.status_code, 206) + self.assertEqual(mocked_open.call_count, 1) + + @patch.object(views, "_open_upstream") + def test_partial_206_html_error_still_rejected(self, mocked_open): + # The mid-packet allowance is gated on content type: a 206 whose body + # is an HTML/PHP error page must still be rejected and cascaded. + html = b"error" + bad = _fake_upstream(206, content_type="text/html", body=html) + bad.raw = MagicMock() + bad.raw.read = MagicMock(return_value=html) + good = _fake_upstream(206, body=_make_ts_payload()) + mocked_open.side_effect = [bad, good] + kwargs = dict(self.kwargs, range_header="bytes=0-") + with patch.object(views, "RedisClient"), \ + patch.object(views, "_register_stats_client"), \ + patch.object(views, "_unregister_stats_client"): + response = views._stream_from_provider(**kwargs) + self.assertEqual(response.status_code, 206) + self.assertEqual(mocked_open.call_count, 2) + + @patch.object(views, "_open_upstream") + def test_206_without_range_header_still_requires_sync(self, mocked_open): + # Without a Range header a 206 is unexpected; it must still pass the + # TS-sync probe (the mid-packet allowance is range-only). + mid_packet = b"\x00" * 300 + bad = _fake_upstream(206, body=mid_packet) + bad.raw = MagicMock() + bad.raw.read = MagicMock(return_value=mid_packet) + good = _fake_upstream(206, body=_make_ts_payload()) + mocked_open.side_effect = [bad, good] + with patch.object(views, "RedisClient"), \ + patch.object(views, "_register_stats_client"), \ + patch.object(views, "_unregister_stats_client"): + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 206) + self.assertEqual(mocked_open.call_count, 2) + + +class RedactUrlTests(TestCase): + """`_redact_url` is the guard that keeps XC credentials out of logs — + both URL forms carry them (query params in format A, path segments in + format B).""" + + def test_redacts_query_credentials(self): + url = "http://example.test/streaming/timeshift.php?username=u&password=p&stream=1" + self.assertEqual(views._redact_url(url), "http://example.test/...") + + def test_redacts_path_credentials(self): + url = "http://example.test/timeshift/user/pass/60/2026-05-12:17-00/1.ts" + self.assertEqual(views._redact_url(url), "http://example.test/...") + + def test_redacts_userinfo_credentials(self): + url = "http://user:pass@example.test/timeshift/1.ts" + self.assertEqual(views._redact_url(url), "http://example.test/...") + + def test_passes_through_non_urls(self): + self.assertEqual(views._redact_url("not a url"), "not a url") + self.assertIsNone(views._redact_url(None)) + + +def _make_catchup_stream(provider_tz="Europe/Brussels", *, account_id=9, + stream_id="22372", account_type="XC", profile_id=31, + extra_profiles=()): + """Build a mocked catch-up Stream with its own provider context. + + The default (tz-bearing) profile leads the active-profile list the view + walks; ``extra_profiles`` appends alternate (non-default) profiles for + capacity-walk tests. + """ + profile = MagicMock() + profile.id = profile_id + profile.is_default = True + profile.custom_properties = {"server_info": {"timezone": provider_tz}} + m3u_account = MagicMock() + m3u_account.account_type = account_type + m3u_account.id = account_id + m3u_account.profiles.filter.return_value = [profile, *extra_profiles] + stream = MagicMock() + stream.m3u_account = m3u_account + stream.custom_properties = {"stream_id": stream_id} if stream_id else {} + return stream + + +def _make_alt_profile(profile_id): + """A non-default active profile for the capacity walk.""" + profile = MagicMock() + profile.id = profile_id + profile.is_default = False + profile.custom_properties = {} + return profile + + +class _FakeRedis: + """Just enough of the redis-py surface for the idle-session pool: setex/get/ + delete plus a transactional pipeline doing GET+DEL, and the hash, set and + lock primitives the pool entries rely on.""" + + def __init__(self): + self.store = {} + + def setex(self, key, ttl, value): + self.store[key] = str(value) + + def set(self, key, value): + self.store[key] = str(value) + + def get(self, key): + return self.store.get(key) + + def delete(self, *keys): + return sum(1 for k in keys if self.store.pop(k, None) is not None) + + def exists(self, key): + return 1 if key in self.store else 0 + + def pipeline(self, transaction=False): + return _FakeRedisPipeline(self) + + # --- hash + lock surface for the session slot --- + def hgetall(self, key): + value = self.store.get(key) + return dict(value) if isinstance(value, dict) else {} + + def hset(self, key, field=None, value=None, mapping=None, **kwargs): + hash_value = self.store.get(key) + if not isinstance(hash_value, dict): + hash_value = {} + self.store[key] = hash_value + if field is not None and value is not None: + hash_value[str(field)] = str(value) + for f, v in (mapping or {}).items(): + hash_value[str(f)] = str(v) + for f, v in kwargs.items(): + hash_value[str(f)] = str(v) + return len(hash_value) + + def hincrby(self, key, field, amount=1): + hash_value = self.store.get(key) + if not isinstance(hash_value, dict): + hash_value = {} + self.store[key] = hash_value + new_value = int(hash_value.get(field, 0)) + amount + hash_value[field] = str(new_value) + return new_value + + def hget(self, key, field): + hash_value = self.store.get(key) + return hash_value.get(field) if isinstance(hash_value, dict) else None + + def expire(self, key, ttl): + return 1 if key in self.store else 0 + + # --- set surface for the idle-session pool --- + def sadd(self, key, *members): + existing = self.store.get(key) + if not isinstance(existing, set): + existing = set() + self.store[key] = existing + before = len(existing) + existing.update(str(m) for m in members) + return len(existing) - before + + def srem(self, key, *members): + existing = self.store.get(key) + if not isinstance(existing, set): + return 0 + removed = 0 + for member in members: + if str(member) in existing: + existing.discard(str(member)) + removed += 1 + return removed + + def smembers(self, key): + existing = self.store.get(key) + return set(existing) if isinstance(existing, set) else set() + + def scard(self, key): + existing = self.store.get(key) + return len(existing) if isinstance(existing, set) else 0 + + def lock(self, name, timeout=None, blocking_timeout=None): + return _FakeRedisLock() + + def scan(self, cursor=0, match=None, count=100): + keys = sorted( + k for k in self.store + if match is None or fnmatch.fnmatch(k, match) + ) + if cursor >= len(keys): + return 0, [] + batch = keys[cursor:cursor + count] + next_cursor = cursor + len(batch) + if next_cursor >= len(keys): + return 0, batch + return next_cursor, batch + + +class _FakeRedisLock: + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + +class _FakeRedisPipeline: + def __init__(self, redis): + self._redis = redis + self._ops = [] + + def get(self, key): + self._ops.append(("get", key)) + + def delete(self, key): + self._ops.append(("delete", key)) + + def execute(self): + results = [] + for op, key in self._ops: + if op == "get": + results.append(self._redis.get(key)) + else: + results.append(self._redis.delete(key)) + self._ops = [] + return results + + +def _fake_creds(acc, prof): + """Distinguishable per-account credentials, mirroring what + get_transformed_credentials returns for the reserved profile.""" + return (f"http://a{acc.id}.test", f"u{acc.id}", "p") + + +class TimeshiftProxyTimestampWiringTests(TestCase): + """`timeshift_proxy` must convert the client's UTC timestamp to the + serving provider's zone for the upstream URL, while keeping the ORIGINAL + UTC timestamp for the EPG duration lookup — the only timezone conversion + in the chain.""" + + def setUp(self): + self.factory = RequestFactory() + + def _call(self, timestamp, provider_tz="Europe/Brussels"): + request = self.factory.get(f"/timeshift/u/p/8/{timestamp}/8.ts?session_id={TEST_SESSION_ID}") + sentinel = MagicMock(status_code=200) + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", + return_value=[_make_catchup_stream(provider_tz)]), \ + patch.object(views, "get_programme_duration", return_value=40) as duration_mock, \ + patch.object(views, "build_timeshift_candidate_urls", + return_value=["http://example.test/x.ts"]) as build_mock, \ + patch.object(views, "check_user_stream_limits", return_value=True), \ + patch.object(views, "RedisClient") as redis_cls, \ + patch.object(views, "reserve_profile_slot", return_value=(True, 1, None)), \ + patch.object(views, "release_profile_slot"), \ + patch.object(views, "get_transformed_credentials", side_effect=_fake_creds), \ + patch.object(views, "get_user_active_connections", return_value=[]), \ + patch.object(views, "_stream_from_provider", return_value=sentinel) as stream_mock: + redis_cls.get_client.return_value = _FakeRedis() + channel_cls.objects.get.return_value = MagicMock(id=8, name="Test", logo_id=None) + response = views.timeshift_proxy(request, "u", "p", "8", timestamp, "8.ts") + return response, sentinel, build_mock, duration_mock, stream_mock + + def test_candidates_get_provider_local_timestamp(self): + # June → CEST: 17:00 UTC must reach the URL builder as 19:00 Brussels. + response, sentinel, build_mock, duration_mock, _ = self._call("2026-06-08:17-00") + self.assertIs(response, sentinel) + self.assertEqual(build_mock.call_args[0][2], "2026-06-08:19-00") + + def test_duration_lookup_keeps_original_utc_timestamp(self): + # The EPG is stored in UTC — the duration lookup must NOT receive the + # provider-converted value. + _, _, _, duration_mock, _ = self._call("2026-06-08:17-00") + self.assertEqual(duration_mock.call_args[0][1], "2026-06-08:17-00") + + def test_utc_provider_passes_timestamp_unchanged(self): + _, _, build_mock, _, _ = self._call("2026-06-08:17-00", provider_tz="UTC") + self.assertEqual(build_mock.call_args[0][2], "2026-06-08:17-00") + + def test_colon_seconds_timestamp_accepted(self): + response, sentinel, build_mock, duration_mock, _ = self._call( + "2026-06-23:04:00:00" + ) + self.assertIs(response, sentinel) + self.assertEqual(duration_mock.call_args[0][1], "2026-06-23:04:00:00") + + def test_invalid_timestamp_rejected_before_upstream(self): + request = self.factory.get("/timeshift/u/p/8/garbage/8.ts") + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams") as catchup_mock, \ + patch.object(views, "_stream_from_provider") as stream_mock: + channel_cls.objects.get.return_value = MagicMock(id=8) + response = views.timeshift_proxy(request, "u", "p", "8", "garbage", "8.ts") + self.assertEqual(response.status_code, 400) + catchup_mock.assert_not_called() + stream_mock.assert_not_called() + + def test_network_access_denied_returns_403(self): + # Same network gate as other XC API endpoints (player_api, xmltv, etc.). + request = self.factory.get(_proxy_url()) + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=False) as gate, \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_stream_from_provider") as stream_mock: + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts" + ) + self.assertEqual(response.status_code, 403) + self.assertEqual(gate.call_args[0][1], "XC_API") + channel_cls.objects.get.assert_not_called() + stream_mock.assert_not_called() + + +class TimeshiftProxyFailoverTests(TestCase): + """When the first catch-up stream's provider cannot serve the archive, + the proxy must fail over to the channel's next catch-up stream — each + attempt with its own provider context.""" + + def setUp(self): + self.factory = RequestFactory() + + def _call(self, streams, provider_responses): + request = self.factory.get(_proxy_url()) + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", return_value=streams), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "build_timeshift_candidate_urls", + return_value=["http://example.test/x.ts"]) as build_mock, \ + patch.object(views, "check_user_stream_limits", return_value=True) as limits_mock, \ + patch.object(views, "RedisClient") as redis_cls, \ + patch.object(views, "reserve_profile_slot", return_value=(True, 1, None)), \ + patch.object(views, "release_profile_slot"), \ + patch.object(views, "get_transformed_credentials", + side_effect=_fake_creds) as creds_mock, \ + patch.object(views, "get_user_active_connections", return_value=[]), \ + patch.object(views, "_stream_from_provider", + side_effect=provider_responses) as stream_mock: + redis_cls.get_client.return_value = _FakeRedis() + channel_cls.objects.get.return_value = MagicMock(id=8, name="Test", logo_id=None) + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts" + ) + self.creds_mock = creds_mock + return response, stream_mock, build_mock, limits_mock + + def test_second_stream_serves_after_first_fails(self): + streams = [ + _make_catchup_stream(account_id=1, stream_id="111"), + _make_catchup_stream(account_id=2, stream_id="222"), + ] + ok = MagicMock(status_code=200) + response, stream_mock, build_mock, _ = self._call( + streams, [MagicMock(status_code=404), ok] + ) + self.assertIs(response, ok) + self.assertEqual(stream_mock.call_count, 2) + # Each attempt used its own provider context: credentials resolved per + # account/profile (via get_transformed_credentials) and its stream id. + self.assertEqual( + [c.args[0] for c in build_mock.call_args_list], + [("http://a1.test", "u1", "p"), ("http://a2.test", "u2", "p")], + ) + self.assertEqual( + [c.args[0] for c in self.creds_mock.call_args_list], + [streams[0].m3u_account, streams[1].m3u_account], + ) + self.assertEqual( + [c.args[1] for c in build_mock.call_args_list], ["111", "222"] + ) + self.assertEqual( + [c.kwargs["account_id"] for c in stream_mock.call_args_list], [1, 2] + ) + + def test_all_streams_fail_returns_last_failure(self): + streams = [ + _make_catchup_stream(account_id=1, stream_id="111"), + _make_catchup_stream(account_id=2, stream_id="222"), + ] + last = MagicMock(status_code=404) + response, stream_mock, _, _ = self._call( + streams, [MagicMock(status_code=400), last] + ) + self.assertIs(response, last) + self.assertEqual(stream_mock.call_count, 2) + + def test_non_xc_and_missing_stream_id_are_skipped(self): + streams = [ + _make_catchup_stream(account_id=1, account_type="M3U"), + _make_catchup_stream(account_id=2, stream_id=None), + _make_catchup_stream(account_id=3, stream_id="333"), + ] + ok = MagicMock(status_code=200) + response, stream_mock, _, _ = self._call(streams, [ok]) + self.assertIs(response, ok) + # Only the eligible third stream produced an upstream attempt. + self.assertEqual(stream_mock.call_count, 1) + self.assertEqual(stream_mock.call_args.kwargs["account_id"], 3) + + def test_stream_limits_checked_once_for_the_request(self): + streams = [ + _make_catchup_stream(account_id=1, stream_id="111"), + _make_catchup_stream(account_id=2, stream_id="222"), + ] + _, _, _, limits_mock = self._call( + streams, [MagicMock(status_code=404), MagicMock(status_code=200)] + ) + self.assertEqual(limits_mock.call_count, 1) + + def test_passthrough_is_not_failed_over_to_other_accounts(self): + # A terminal range answer (e.g. 416 past EOF) must be returned as-is; + # the loop must NOT try the next account, whose byte offsets would not + # match this file and which would just burn another provider slot. + streams = [ + _make_catchup_stream(account_id=1, stream_id="111"), + _make_catchup_stream(account_id=2, stream_id="222"), + ] + passthrough = MagicMock(status_code=416) + passthrough.timeshift_passthrough = True + response, stream_mock, _, _ = self._call(streams, [passthrough]) + self.assertIs(response, passthrough) + self.assertEqual(stream_mock.call_count, 1) + + +class _ProxyLoopTestMixin: + """Shared driver for tests exercising the failover loop end to end — + pool reservation, credential resolution and Redis are all controlled.""" + + def setUp(self): + self.factory = RequestFactory() + + def _call(self, streams, provider_responses, limits=True, reserve_results=None, + build_side_effect=None): + request = self.factory.get(_proxy_url()) + self.fake_redis = _FakeRedis() + reserve_kwargs = ( + {"side_effect": reserve_results} + if reserve_results is not None + else {"return_value": (True, 1, None)} + ) + build_kwargs = ( + {"side_effect": build_side_effect} + if build_side_effect is not None + else {"return_value": ["http://example.test/x.ts"]} + ) + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", return_value=streams), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "build_timeshift_candidate_urls", + **build_kwargs) as build_mock, \ + patch.object(views, "check_user_stream_limits", return_value=limits), \ + patch.object(views, "RedisClient") as redis_cls, \ + patch.object(views, "reserve_profile_slot", **reserve_kwargs) as reserve_mock, \ + patch.object(views, "release_profile_slot") as release_mock, \ + patch.object(views, "get_transformed_credentials", + side_effect=_fake_creds) as creds_mock, \ + patch.object(views, "get_user_active_connections", return_value=[]), \ + patch.object(views, "_stream_from_provider", + side_effect=provider_responses) as stream_mock: + redis_cls.get_client.return_value = self.fake_redis + channel_cls.objects.get.return_value = MagicMock(id=8, name="Test", logo_id=None) + # Exposed before the call so raising tests can still assert on them. + self.reserve_mock = reserve_mock + self.release_mock = release_mock + self.creds_mock = creds_mock + self.stream_mock = stream_mock + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts" + ) + return response, stream_mock, build_mock + + +class TimeshiftProxyFailoverHardeningTests(_ProxyLoopTestMixin, TestCase): + """Ban-safety and per-provider context guarantees of the failover loop.""" + + def test_decisive_failure_skips_same_accounts_other_streams(self): + # Account 1 carries two variants (e.g. FHD + HD). A decisive + # (auth/ban-class) failure on the first must NOT retry account 1's + # second stream — that would hammer a banning provider — but a + # DIFFERENT account stays fair game. + streams = [ + _make_catchup_stream(account_id=1, stream_id="111"), + _make_catchup_stream(account_id=1, stream_id="112"), + _make_catchup_stream(account_id=2, stream_id="222"), + ] + decisive = MagicMock(status_code=403, timeshift_decisive=True) + ok = MagicMock(status_code=200) + response, stream_mock, _ = self._call(streams, [decisive, ok]) + self.assertIs(response, ok) + self.assertEqual(stream_mock.call_count, 2) + self.assertEqual( + [c.kwargs["account_id"] for c in stream_mock.call_args_list], [1, 2] + ) + + def test_soft_failure_still_tries_same_accounts_other_streams(self): + # A soft failure (404: this stream's archive missing) is stream- + # specific — the same account's other variant may still have it. + streams = [ + _make_catchup_stream(account_id=1, stream_id="111"), + _make_catchup_stream(account_id=1, stream_id="112"), + ] + soft = MagicMock(status_code=404, timeshift_decisive=False) + ok = MagicMock(status_code=200) + response, stream_mock, _ = self._call(streams, [soft, ok]) + self.assertIs(response, ok) + self.assertEqual(stream_mock.call_count, 2) + self.assertEqual( + [c.kwargs["account_id"] for c in stream_mock.call_args_list], [1, 1] + ) + + def test_each_stream_uses_its_own_provider_timezone(self): + # June: 17:00 UTC = 19:00 Brussels (CEST) but 13:00 New York (EDT). + # The converted timestamp must be recomputed per attempt. + streams = [ + _make_catchup_stream(account_id=1, stream_id="111", + provider_tz="Europe/Brussels"), + _make_catchup_stream(account_id=2, stream_id="222", + provider_tz="America/New_York"), + ] + response, _, build_mock = self._call( + streams, + [MagicMock(status_code=404, timeshift_decisive=False), + MagicMock(status_code=200)], + ) + self.assertEqual(response.status_code, 200) + self.assertEqual( + [c.args[2] for c in build_mock.call_args_list], + ["2026-06-08:19-00", "2026-06-08:13-00"], + ) + + def test_stream_limit_exceeded_returns_403_before_upstream(self): + streams = [_make_catchup_stream(account_id=1, stream_id="111")] + response, stream_mock, _ = self._call(streams, [], limits=False) + self.assertEqual(response.status_code, 403) + stream_mock.assert_not_called() + + def test_no_catchup_streams_returns_400(self): + response, stream_mock, _ = self._call([], []) + self.assertEqual(response.status_code, 400) + stream_mock.assert_not_called() + + def test_all_streams_ineligible_returns_400(self): + streams = [ + _make_catchup_stream(account_id=1, account_type="M3U"), + _make_catchup_stream(account_id=2, stream_id=None), + ] + response, stream_mock, _ = self._call(streams, []) + self.assertEqual(response.status_code, 400) + stream_mock.assert_not_called() + + +class XcServerInfoUtcTests(TestCase): + """The XC server_info 'timezone triple' guarantee the timeshift chain + relies on: server_info.timezone is always UTC and time_now is UTC + wall-clock. (Tested here because catch-up seek correctness depends on + it: clients build the timeshift URL from this declared zone.)""" + + def test_server_info_is_strictly_utc(self): + from datetime import datetime, timezone as dt_timezone + from apps.output.views import _build_xc_server_info + + request = MagicMock(scheme="http") + info = _build_xc_server_info(request, "example.test", "9191") + self.assertEqual(info["timezone"], "UTC") + reported = datetime.strptime(info["time_now"], "%Y-%m-%d %H:%M:%S") + now_utc = datetime.now(dt_timezone.utc).replace(tzinfo=None) + self.assertLess(abs((now_utc - reported).total_seconds()), 60) + self.assertIsInstance(info["timestamp_now"], int) + + +class StreamFromProviderDecisiveEdgeTests(TestCase): + """Remaining decisive-status and transport-error paths of the cascade.""" + + def setUp(self): + self.kwargs = dict( + candidate_urls=[ + "http://example.test/timeshift/u/p/60/2026-05-12:17-00/1.ts", + "http://example.test/streaming/timeshift.php?stream=1&start=2026-05-12_17-00", + ], + user_agent="test-agent", + range_header=None, + virtual_channel_id="timeshift_1_2026-05-12-17-00_1", + client_id="timeshift_test456", + client_ip="127.0.0.1", + user=None, + channel_display_name="Test", + timestamp_utc="2026-05-12:17-00", + channel_logo_id=None, + m3u_profile_id=None, + debug=False, + ) + + @patch.object(views, "_open_upstream") + def test_406_is_decisive_and_marks_response(self, mocked_open): + # 406 = IP-wide block in the XC ban escalation — single attempt, + # generic 400 to the client, and the failover loop must see the + # decisive marker so it skips this account's other streams. + mocked_open.return_value = _fake_upstream(406) + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 400) + self.assertEqual(mocked_open.call_count, 1) + self.assertTrue(response.timeshift_decisive) + + @patch.object(views, "_open_upstream") + def test_404_failure_is_not_decisive(self, mocked_open): + mocked_open.return_value = _fake_upstream(404) + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 404) + self.assertFalse(response.timeshift_decisive) + + @patch.object(views, "_open_upstream") + def test_connection_error_returns_400_after_single_attempt(self, mocked_open): + import requests as _requests + mocked_open.side_effect = _requests.exceptions.ConnectionError("boom") + response = views._stream_from_provider(**self.kwargs) + self.assertEqual(response.status_code, 400) + self.assertEqual(mocked_open.call_count, 1) + # Transport errors are host-level, not auth/ban-class: the failover + # loop may still try a different account. + self.assertFalse(getattr(response, "timeshift_decisive", False)) + + +class CatchupStreamsDbTests(TestCase): + """get_channel_catchup_streams: the function that defines the failover + order — channelstream order, catch-up streams only, active accounts only.""" + + @classmethod + def setUpTestData(cls): + from apps.channels.models import Channel, ChannelStream, Stream + from apps.m3u.models import M3UAccount + + cls.active = M3UAccount.objects.create( + name="ts-test-active", server_url="http://example.test", + account_type="XC", is_active=True, + ) + cls.inactive = M3UAccount.objects.create( + name="ts-test-inactive", server_url="http://example.test", + account_type="XC", is_active=False, + ) + cls.channel = Channel.objects.create(name="ts-test-channel", is_catchup=True) + + def add(name, account, *, catchup, order): + s = Stream.objects.create( + name=name, url=f"http://example.test/{name}", + m3u_account=account, is_catchup=catchup, + ) + ChannelStream.objects.create(channel=cls.channel, stream=s, order=order) + return s + + cls.s_inactive = add("s-inactive", cls.inactive, catchup=True, order=0) + cls.s_second = add("s-second", cls.active, catchup=True, order=2) + cls.s_first = add("s-first", cls.active, catchup=True, order=1) + cls.s_live_only = add("s-live-only", cls.active, catchup=False, order=3) + + def test_ordered_active_catchup_streams_only(self): + from apps.channels.utils import get_channel_catchup_streams + + result = get_channel_catchup_streams(self.channel) + # Inactive-account and non-catchup streams excluded; channelstream order. + self.assertEqual([s.id for s in result], [self.s_first.id, self.s_second.id]) + + def test_channel_without_catchup_flag_returns_empty(self): + from apps.channels.models import Channel + from apps.channels.utils import get_channel_catchup_streams + + ch = Channel.objects.create(name="ts-test-nocatchup", is_catchup=False) + self.assertEqual(get_channel_catchup_streams(ch), []) + + +class AuthHelpersDbTests(TestCase): + """_authenticate_user (xc_password custom property) and + _user_can_access_channel (user_level gate) — exercised against real models + instead of being mocked away.""" + + @classmethod + def setUpTestData(cls): + from apps.accounts.models import User + from apps.channels.models import Channel + + cls.viewer = User.objects.create( + username="ts-test-viewer", user_level=0, + custom_properties={"xc_password": "right-pass"}, + ) + cls.no_xc = User.objects.create( + username="ts-test-noxc", user_level=10, + custom_properties={}, + ) + cls.basic_channel = Channel.objects.create(name="ts-test-basic", user_level=0) + cls.admin_channel = Channel.objects.create(name="ts-test-adult", user_level=10) + + def test_valid_xc_password_authenticates(self): + user = views._authenticate_user("ts-test-viewer", "right-pass") + self.assertIsNotNone(user) + self.assertEqual(user.id, self.viewer.id) + + def test_wrong_xc_password_rejected(self): + self.assertIsNone(views._authenticate_user("ts-test-viewer", "wrong")) + + def test_user_without_xc_password_rejected(self): + # Accounts with no xc_password set (e.g. admins) must be denied even + # if the caller guesses any string — there is nothing to compare to. + self.assertIsNone(views._authenticate_user("ts-test-noxc", "")) + self.assertIsNone(views._authenticate_user("ts-test-noxc", "anything")) + + def test_unknown_username_rejected(self): + self.assertIsNone(views._authenticate_user("ts-test-ghost", "x")) + + def test_user_level_gate(self): + # Level-0 viewer with no profiles: allowed on level-0, denied on level-10. + self.assertTrue(views._user_can_access_channel(self.viewer, self.basic_channel)) + self.assertFalse(views._user_can_access_channel(self.viewer, self.admin_channel)) + + +class TimeshiftSlotPoolTests(_ProxyLoopTestMixin, TestCase): + """Provider pool participation: a profile slot is reserved before any + upstream attempt and released exactly once afterwards, the same accounting + contract live (Channel.get_stream) and VOD follow. Each active stream + reserves its own slot so concurrent provider connections stay capped by + max_streams.""" + + POOL_KEY = f"timeshift_pool:{TEST_SESSION_ID}" + + def _pool_entry_ids(self): + return [k for k in self.fake_redis.store if k.startswith("timeshift_pool:")] + + def test_reserve_called_with_default_profile_before_upstream(self): + streams = [_make_catchup_stream(account_id=1, stream_id="111", profile_id=31)] + response, stream_mock, _ = self._call(streams, [MagicMock(status_code=200)]) + self.assertEqual(response.status_code, 200) + self.reserve_mock.assert_called_once() + reserved_profile = self.reserve_mock.call_args.args[0] + self.assertEqual(reserved_profile.id, 31) + # The reserved profile's id is what reaches the stats metadata. + self.assertEqual(stream_mock.call_args.kwargs["m3u_profile_id"], 31) + + def test_slot_released_after_failed_attempt(self): + streams = [_make_catchup_stream(account_id=1, stream_id="111", profile_id=31)] + response, _, _ = self._call( + streams, [MagicMock(status_code=404, timeshift_decisive=False)] + ) + self.assertEqual(response.status_code, 404) + # The failed attempt's slot was released and its pool entry removed. + self.release_mock.assert_called_once_with(31, self.fake_redis) + self.assertEqual(self._pool_entry_ids(), []) + + def test_slot_kept_on_success_for_the_streaming_session(self): + streams = [_make_catchup_stream(account_id=1, stream_id="111", profile_id=31)] + response, _, _ = self._call(streams, [MagicMock(status_code=200)]) + self.assertEqual(response.status_code, 200) + # Slot still owned by the (mocked) streaming session: a busy pool entry + # remains for the next request to reuse, nothing released yet. + self.release_mock.assert_not_called() + self.assertEqual(len(self._pool_entry_ids()), 1) + + def test_decisive_failure_releases_slot_and_skips_account(self): + streams = [ + _make_catchup_stream(account_id=1, stream_id="111", profile_id=31), + _make_catchup_stream(account_id=1, stream_id="112", profile_id=31), + ] + response, stream_mock, _ = self._call( + streams, [MagicMock(status_code=403, timeshift_decisive=True)] + ) + self.assertEqual(response.status_code, 403) + self.assertEqual(stream_mock.call_count, 1) + self.release_mock.assert_called_once_with(31, self.fake_redis) + # Decisive skip means the second stream never reserved a slot. + self.assertEqual(self.reserve_mock.call_count, 1) + + def test_profile_full_walks_to_next_profile_same_account(self): + alt = _make_alt_profile(32) + streams = [_make_catchup_stream( + account_id=1, stream_id="111", profile_id=31, extra_profiles=(alt,) + )] + response, stream_mock, _ = self._call( + streams, [MagicMock(status_code=200)], + reserve_results=[(False, 1, "profile_full"), (True, 1, None)], + ) + self.assertEqual(response.status_code, 200) + # Default profile full -> alternate profile reserved and used. + self.assertEqual( + [c.args[0].id for c in self.reserve_mock.call_args_list], [31, 32] + ) + self.assertEqual(stream_mock.call_args.kwargs["m3u_profile_id"], 32) + # Credentials were resolved for the RESERVED (alternate) profile. + self.assertIs(self.creds_mock.call_args.args[1], alt) + + def test_all_profiles_full_returns_503_without_upstream_attempt(self): + streams = [ + _make_catchup_stream(account_id=1, stream_id="111", profile_id=31), + _make_catchup_stream(account_id=2, stream_id="222", profile_id=41), + ] + response, stream_mock, _ = self._call( + streams, [], + reserve_results=[ + (False, 1, "profile_full"), + (False, 1, "credential_full"), + ], + ) + # Pool capacity exhausted everywhere: 503 (VOD's pool-exhausted + # status), and crucially the provider was never contacted. + self.assertEqual(response.status_code, 503) + stream_mock.assert_not_called() + self.release_mock.assert_not_called() + + def test_capacity_failure_is_not_decisive_for_the_account(self): + # profile_full on account 1's first stream must NOT mark account 1 + # decisive — capacity is transient, unlike a ban-class status. + streams = [ + _make_catchup_stream(account_id=1, stream_id="111", profile_id=31), + _make_catchup_stream(account_id=1, stream_id="112", profile_id=31), + ] + response, stream_mock, _ = self._call( + streams, [MagicMock(status_code=200)], + reserve_results=[(False, 1, "profile_full"), (True, 1, None)], + ) + self.assertEqual(response.status_code, 200) + # Second stream of the SAME account still got its reservation attempt. + self.assertEqual(self.reserve_mock.call_count, 2) + self.assertEqual(stream_mock.call_count, 1) + + def test_account_without_active_default_profile_is_skipped(self): + # Mirrors live dispatch: no active default profile -> skip the account + # without reserving anything. + stream = _make_catchup_stream(account_id=1, stream_id="111") + stream.m3u_account.profiles.filter.return_value = [_make_alt_profile(32)] + response, stream_mock, _ = self._call([stream], []) + self.assertEqual(response.status_code, 400) + self.reserve_mock.assert_not_called() + stream_mock.assert_not_called() + + def test_exception_from_provider_releases_slot(self): + # An unexpected exception between reserve and response construction + # must release the slot before propagating — otherwise the counter + # (no TTL) leaks until the next Redis flush. + streams = [_make_catchup_stream(account_id=1, stream_id="111", profile_id=31)] + with self.assertRaises(RuntimeError): + self._call(streams, RuntimeError("boom")) + self.release_mock.assert_called_once_with(31, self.fake_redis) + self.assertEqual(self._pool_entry_ids(), []) + + def test_exception_before_upstream_releases_slot(self): + # Same guarantee for failures BEFORE the upstream call (URL building, + # credential resolution, user-agent lookup) — the guarded window + # starts right after the reservation. + streams = [_make_catchup_stream(account_id=1, stream_id="111", profile_id=31)] + with self.assertRaises(RuntimeError): + self._call(streams, [], build_side_effect=RuntimeError("boom")) + self.stream_mock.assert_not_called() + self.release_mock.assert_called_once_with(31, self.fake_redis) + self.assertEqual(self._pool_entry_ids(), []) + + def test_mixed_capacity_then_upstream_failure_returns_failure(self): + # Mixed outcome: one stream capacity-blocked, another actually tried + # upstream and failed -> the REAL upstream failure wins over 503 + # (capacity was not the sole blocker). + streams = [ + _make_catchup_stream(account_id=1, stream_id="111", profile_id=31), + _make_catchup_stream(account_id=2, stream_id="222", profile_id=41), + ] + response, _, _ = self._call( + streams, + [MagicMock(status_code=404, timeshift_decisive=False)], + reserve_results=[(False, 1, "profile_full"), (True, 1, None)], + ) + self.assertEqual(response.status_code, 404) + + def test_mixed_upstream_failure_then_capacity_returns_failure(self): + # Same in the opposite order. + streams = [ + _make_catchup_stream(account_id=1, stream_id="111", profile_id=31), + _make_catchup_stream(account_id=2, stream_id="222", profile_id=41), + ] + response, _, _ = self._call( + streams, + [MagicMock(status_code=404, timeshift_decisive=False)], + reserve_results=[(True, 1, None), (False, 1, "profile_full")], + ) + self.assertEqual(response.status_code, 404) + + +class TimeshiftPoolReleaseTests(TestCase): + """Pool slot release and response close paths for a pooled session.""" + + def setUp(self): + self.redis = _FakeRedis() + self.session_id = TEST_SESSION_ID + + def _pool_key(self): + return f"timeshift_pool:{self.session_id}" + + def test_release_callback_frees_slot_exactly_once(self): + _seed_pool_session(self.redis, session_id=self.session_id) + release = views._make_release_once(self.redis, self.session_id, 31) + with patch.object(views, "release_profile_slot") as release_mock: + release() + release() + release_mock.assert_called_once_with(31, self.redis) + self.assertEqual(self.redis.hget(self._pool_key(), "busy"), "0") + self.assertTrue(self.redis.exists(self._pool_key())) + + def test_discard_frees_slot_and_removes_entry(self): + _seed_pool_session(self.redis, session_id=self.session_id) + with patch.object(views, "release_profile_slot") as release_mock: + views._discard_pool_session(self.redis, self.session_id, 31) + release_mock.assert_called_once_with(31, self.redis) + self.assertFalse(self.redis.exists(self._pool_key())) + + def test_release_without_redis_is_noop(self): + release = views._make_release_once(None, self.session_id, 31) + with patch.object(views, "release_profile_slot") as release_mock: + release() + release_mock.assert_not_called() + + def test_wrapper_close_releases_even_when_generator_never_started(self): + # The WSGI layer can close the response before the first chunk is + # pulled; closing a never-started generator runs NO body code, so the + # generator's own finally cannot be the only release point. + finally_ran = [] + + def gen(): + try: + yield b"x" + finally: + finally_ran.append(True) + + on_close = MagicMock() + wrapper = views._SlotReleasingStream(gen(), on_close) + wrapper.close() + on_close.assert_called_once() + self.assertEqual(finally_ran, []) # proves the leak this wrapper fixes + + def test_streaming_response_close_invokes_wrapper_close(self): + # Locks the Django contract the wrapper relies on: an iterator with a + # close() method is registered as a resource closer of the response. + from django.http import StreamingHttpResponse + + on_close = MagicMock() + wrapper = views._SlotReleasingStream(iter([b"x"]), on_close) + response = StreamingHttpResponse(wrapper, content_type="video/mp2t") + response.close() + on_close.assert_called_once() + + +class TimeshiftTakeoverTests(TestCase): + """A new request displaces the user's previous catch-up session(s) on the + same channel at a DIFFERENT position (stats unregister + stop key, leaving + the displaced generator to free its own slot), while leaving sibling range + requests of the same playback alone, and never touching other users, + channels, or live.""" + + def setUp(self): + self.redis = _FakeRedis() + self.user = MagicMock(id=5) + + def _conn(self, media_id, client_id, conn_type="timeshift"): + return { + "media_id": media_id, + "client_id": client_id, + "connected_at": 0.0, + "type": conn_type, + } + + def test_displaces_other_positions_on_same_channel(self): + connections = [ + self._conn("timeshift_8_2026-06-08-17-00_111", "timeshift_old1"), + self._conn("timeshift_9_2026-06-08-17-00_222", "timeshift_other"), + self._conn("42", "live_client", conn_type="live"), + ] + with patch.object(views, "get_user_active_connections", + return_value=connections) as conns_mock, \ + patch.object(views, "release_profile_slot") as release_mock, \ + patch.object(views, "_unregister_stats_client") as unregister_mock: + views._terminate_previous_timeshift_sessions( + self.redis, self.user, 8, "timeshift_8_2026-06-09-20-00", "timeshift_current", + ) + conns_mock.assert_called_once_with(5) + # Takeover defers slot release to the displaced generator's stop path; + # it only drops stats and signals the stop key. + release_mock.assert_not_called() + unregister_mock.assert_called_once_with( + self.redis, "timeshift_8_2026-06-08-17-00_111", "timeshift_old1" + ) + from apps.proxy.live_proxy.redis_keys import RedisKeys + stop_key = RedisKeys.client_stop( + "timeshift_8_2026-06-08-17-00_111", "timeshift_old1" + ) + self.assertIn(stop_key, self.redis.store) + # Channel 9's session untouched: no stop key set for it. + other_stop = RedisKeys.client_stop( + "timeshift_9_2026-06-08-17-00_222", "timeshift_other" + ) + self.assertNotIn(other_stop, self.redis.store) + + def test_leaves_sibling_requests_of_current_playback(self): + # Concurrent range/probe requests of the SAME playback must not + # displace one another. + connections = [ + self._conn("timeshift_8_2026-06-08-17-00_111", "timeshift_sibling"), + ] + with patch.object(views, "get_user_active_connections", + return_value=connections), \ + patch.object(views, "release_profile_slot") as release_mock, \ + patch.object(views, "_unregister_stats_client") as unregister_mock: + views._terminate_previous_timeshift_sessions( + self.redis, self.user, 8, "timeshift_8_2026-06-08-17-00", + "timeshift_sibling", + ) + release_mock.assert_not_called() + unregister_mock.assert_not_called() + + def test_channel_id_prefix_cannot_match_other_channels(self): + # Channel 8 must not displace channel 80/81 sessions (prefix ends + # with an underscore). + connections = [ + self._conn("timeshift_80_2026-06-08-17-00_111", "timeshift_c80"), + ] + with patch.object(views, "get_user_active_connections", + return_value=connections), \ + patch.object(views, "release_profile_slot") as release_mock, \ + patch.object(views, "_unregister_stats_client") as unregister_mock: + views._terminate_previous_timeshift_sessions( + self.redis, self.user, 8, "timeshift_8_2026-06-08-17-00", + "timeshift_new", + ) + release_mock.assert_not_called() + unregister_mock.assert_not_called() + + def test_noop_without_redis_or_user(self): + with patch.object(views, "get_user_active_connections") as conns_mock: + views._terminate_previous_timeshift_sessions( + None, self.user, 8, "timeshift_8_ts", "timeshift_s" + ) + views._terminate_previous_timeshift_sessions( + self.redis, None, 8, "timeshift_8_ts", "timeshift_s" + ) + conns_mock.assert_not_called() + + def test_proxy_runs_takeover_before_stream_limit_check(self): + # Order matters: with terminate_on_limit_exceeded=False a seek must + # displace its own predecessor BEFORE the limit check counts it, or + # the user's own seek gets denied. + call_order = [] + request = RequestFactory().get(_proxy_url()) + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", + return_value=[_make_catchup_stream()]), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "RedisClient") as redis_cls, \ + patch.object(views, "_terminate_previous_timeshift_sessions", + side_effect=lambda *a: call_order.append("takeover")) as takeover_mock, \ + patch.object(views, "check_user_stream_limits", + side_effect=lambda *a, **k: call_order.append("limits") or False): + redis_cls.get_client.return_value = self.redis + channel_cls.objects.get.return_value = MagicMock(id=8, name="Test", logo_id=None) + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts" + ) + self.assertEqual(response.status_code, 403) + self.assertEqual(call_order, ["takeover", "limits"]) + self.assertEqual(takeover_mock.call_args.args[2], 8) + + +class TimeshiftSessionReuseTests(TestCase): + """Per-client session pool acquire/reuse paths.""" + + SESSION = TEST_SESSION_ID + + def setUp(self): + self.redis = _FakeRedis() + self.factory = RequestFactory() + self.channel = MagicMock(id=8, name="Test") + self.user = MagicMock(id=5) + + def _pool_key(self): + return f"timeshift_pool:{self.SESSION}" + + def _make_idle_entry(self): + _seed_pool_session(self.redis, session_id=self.SESSION) + with patch.object(views, "release_profile_slot"): + views._release_pool_session(self.redis, self.SESSION, 31) + + def test_wait_returns_none_without_blocking_when_pool_empty(self): + start = time.monotonic() + acquired = views._wait_for_idle_pool_session(self.redis, self.SESSION) + self.assertIsNone(acquired) + self.assertLess(time.monotonic() - start, 0.5) + + def test_acquire_reuses_idle_entry_and_reserves_slot(self): + self._make_idle_entry() + profile = MagicMock(id=31) + with patch.object(views.M3UAccountProfile.objects, "get", + return_value=profile), \ + patch.object(views, "reserve_profile_slot", + return_value=(True, 1, None)) as reserve_mock: + acquired = views._acquire_idle_pool_session( + self.redis, self.SESSION, user_id=5, + ) + self.assertIsNotNone(acquired) + descriptor, got_profile = acquired + self.assertEqual(descriptor["stream_id"], "111") + self.assertIs(got_profile, profile) + reserve_mock.assert_called_once_with(profile, self.redis) + self.assertEqual(self.redis.hget(self._pool_key(), "busy"), "1") + + def test_acquire_skips_busy_entry(self): + _seed_pool_session(self.redis, session_id=self.SESSION, busy="1") + with patch.object(views.M3UAccountProfile.objects, "get") as prof_mock, \ + patch.object(views, "reserve_profile_slot") as reserve_mock: + acquired = views._acquire_idle_pool_session( + self.redis, self.SESSION, user_id=5, + ) + self.assertIsNone(acquired) + prof_mock.assert_not_called() + reserve_mock.assert_not_called() + + def test_acquire_rejects_foreign_user(self): + self._make_idle_entry() + profile = MagicMock(id=31) + with patch.object(views.M3UAccountProfile.objects, "get", + return_value=profile), \ + patch.object(views, "reserve_profile_slot", + return_value=(True, 1, None)) as reserve_mock: + acquired = views._acquire_idle_pool_session( + self.redis, self.SESSION, user_id=99, + ) + self.assertIsNone(acquired) + reserve_mock.assert_not_called() + + def test_foreign_session_id_redirects_instead_of_reusing_pool(self): + victim_session = "timeshift_victim_session" + _seed_pool_session(self.redis, session_id=victim_session, user_id=99) + request = self.factory.get(_proxy_url(victim_session)) + attacker = MagicMock(id=5) + with patch.object(views, "_authenticate_user", return_value=attacker), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", + return_value=[_make_catchup_stream()]), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "parse_catchup_timestamp", return_value=True), \ + patch.object(views, "RedisClient") as redis_cls, \ + patch.object(views, "_acquire_idle_pool_session") as acquire_mock, \ + patch.object(views, "_attempt_timeshift_stream") as attempt_mock: + redis_cls.get_client.return_value = self.redis + channel_cls.objects.get.return_value = MagicMock(id=8) + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts", + ) + self.assertEqual(response.status_code, 301) + self.assertIn("session_id=timeshift_", response["Location"]) + self.assertNotIn(victim_session, response["Location"]) + acquire_mock.assert_not_called() + attempt_mock.assert_not_called() + + def test_find_matching_idle_session_requires_ip_and_user_agent(self): + _seed_pool_session( + self.redis, session_id="timeshift_other", + user_id=5, client_ip="1.2.3.4", client_user_agent="test-agent", + ) + with patch.object(views, "release_profile_slot"): + views._release_pool_session(self.redis, "timeshift_other", 31) + matched = views._find_matching_idle_session( + self.redis, + media_id=TEST_MEDIA_ID, + user_id=5, + client_ip="1.2.3.4", + client_user_agent="test-agent", + ) + self.assertEqual(matched, "timeshift_other") + + def test_find_matching_idle_session_rejects_ip_only_partial_fingerprint(self): + _seed_pool_session( + self.redis, session_id="timeshift_other", + user_id=5, client_ip="1.2.3.4", client_user_agent="other-agent", + ) + with patch.object(views, "release_profile_slot"): + views._release_pool_session(self.redis, "timeshift_other", 31) + matched = views._find_matching_idle_session( + self.redis, + media_id=TEST_MEDIA_ID, + user_id=5, + client_ip="1.2.3.4", + client_user_agent="test-agent", + ) + self.assertIsNone(matched) + + def test_find_matching_idle_session_rejects_different_user(self): + _seed_pool_session( + self.redis, session_id="timeshift_other", + user_id=99, client_ip="1.2.3.4", client_user_agent="test-agent", + ) + with patch.object(views, "release_profile_slot"): + views._release_pool_session(self.redis, "timeshift_other", 31) + matched = views._find_matching_idle_session( + self.redis, + media_id=TEST_MEDIA_ID, + user_id=5, + client_ip="1.2.3.4", + client_user_agent="test-agent", + ) + self.assertIsNone(matched) + + def test_legacy_pool_entry_exists_helper_removed(self): + self.assertFalse(hasattr(views, "_pool_entry_exists")) + + def test_new_session_uses_single_hgetall_before_pool_create(self): + redis = _FakeRedis() + request = self.factory.get(_proxy_url("timeshift_newsession1")) + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", + return_value=[_make_catchup_stream()]), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "check_user_stream_limits", return_value=True), \ + patch.object(views, "_find_matching_idle_session", return_value=None), \ + patch.object(views, "_attempt_timeshift_stream", + return_value=MagicMock(status_code=200)), \ + patch.object(views, "RedisClient") as redis_cls, \ + patch.object(views, "reserve_profile_slot", return_value=(True, 1, None)), \ + patch.object(views, "release_profile_slot"), \ + patch.object(views, "get_transformed_credentials", side_effect=_fake_creds), \ + patch.object(views, "get_user_active_connections", return_value=[]): + redis_cls.get_client.return_value = redis + channel_cls.objects.get.return_value = MagicMock(id=8, name="Test", logo_id=None) + with patch.object(redis, "hgetall", wraps=redis.hgetall) as hgetall_mock: + views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts", + ) + pool_key = "timeshift_pool:timeshift_newsession1" + self.assertEqual( + sum(1 for c in hgetall_mock.call_args_list if c.args == (pool_key,)), + 1, + ) + + +class TimeshiftSessionRedirectTests(TestCase): + """First request must establish a session via 301 redirect (VOD-style).""" + + def setUp(self): + self.factory = RequestFactory() + + def test_missing_session_id_redirects(self): + request = self.factory.get(_proxy_url(session_id=None)) + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=1)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", + return_value=[_make_catchup_stream()]), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "parse_catchup_timestamp", return_value=True), \ + patch.object(views, "RedisClient") as redis_cls: + redis_cls.get_client.return_value = _FakeRedis() + channel_cls.objects.get.return_value = MagicMock(id=8) + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts", + ) + self.assertEqual(response.status_code, 301) + self.assertIn("session_id=timeshift_", response["Location"]) + + def test_redirect_preserves_existing_query_params(self): + request = self.factory.get( + "/timeshift/u/p/8/2026-06-08:17-00/8.ts?foo=bar&baz=1", + ) + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=1)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", + return_value=[_make_catchup_stream()]), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "parse_catchup_timestamp", return_value=True), \ + patch.object(views, "RedisClient") as redis_cls: + redis_cls.get_client.return_value = _FakeRedis() + channel_cls.objects.get.return_value = MagicMock(id=8) + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts", + ) + self.assertEqual(response.status_code, 301) + location = response["Location"] + self.assertIn("session_id=timeshift_", location) + self.assertIn("foo=bar", location) + self.assertIn("baz=1", location) + + +class TimeshiftStreamLimitExemptionTests(TestCase): + """Timeshift stream-limit bypass requires the same client session.""" + + MEDIA = TEST_MEDIA_ID + + def setUp(self): + self.user = MagicMock(id=5, username="viewer", stream_limit=1) + + def _limits_settings(self, ignore_same_channel=True): + return { + "ignore_same_channel_connections": ignore_same_channel, + "terminate_on_limit_exceeded": False, + } + + def test_same_session_probe_allowed_at_limit(self): + connections = [{ + "media_id": f"{self.MEDIA}_111", + "client_id": TEST_SESSION_ID, + "connected_at": 0.0, + "type": "timeshift", + }] + with patch("apps.proxy.utils.get_user_active_connections", + return_value=connections), \ + patch("apps.proxy.utils.CoreSettings.get_user_limits_settings", + return_value=self._limits_settings()): + allowed = _check_user_stream_limits( + self.user, TEST_SESSION_ID, media_id=self.MEDIA, + ) + self.assertTrue(allowed) + + def test_different_session_same_programme_counts_against_limit(self): + connections = [{ + "media_id": f"{self.MEDIA}_111", + "client_id": "timeshift_other_session", + "connected_at": 0.0, + "type": "timeshift", + }] + with patch("apps.proxy.utils.get_user_active_connections", + return_value=connections), \ + patch("apps.proxy.utils.CoreSettings.get_user_limits_settings", + return_value=self._limits_settings()): + allowed = _check_user_stream_limits( + self.user, TEST_SESSION_ID, media_id=self.MEDIA, + ) + self.assertFalse(allowed) + + +class FakeRedisScanTests(TestCase): + """FakeRedis SCAN matches redis-py glob semantics used by the pool scanner.""" + + def setUp(self): + self.redis = _FakeRedis() + self.redis.store["timeshift_pool:timeshift_a"] = {"busy": "0"} + self.redis.store["timeshift_pool:timeshift_b"] = {"busy": "0"} + self.redis.store["timeshift_pool:other_c"] = {"busy": "0"} + self.redis.store["vod_persistent_connection:x"] = {} + + def test_scan_glob_filters_pool_keys(self): + cursor = 0 + seen = [] + while True: + cursor, keys = self.redis.scan( + cursor, match="timeshift_pool:timeshift_*", count=1, + ) + seen.extend(keys) + if cursor == 0: + break + self.assertEqual( + seen, + ["timeshift_pool:timeshift_a", "timeshift_pool:timeshift_b"], + ) + + +class TimeshiftRangeClassificationTests(TestCase): + """Startup probes must not be treated as scrubs.""" + + def test_full_file_request_is_not_displacing(self): + self.assertFalse(views._should_displace_busy_playback(None)) + + def test_bytes_zero_displaces_full_file_probe(self): + self.assertTrue( + views._should_displace_busy_playback("bytes=0-", busy_serving_range="none") + ) + + def test_bytes_zero_does_not_displace_active_start_stream(self): + self.assertFalse( + views._should_displace_busy_playback("bytes=0-", busy_serving_range="start") + ) + + def test_bytes_zero_without_busy_context_is_not_displacing(self): + self.assertFalse(views._should_displace_busy_playback("bytes=0-")) + + def test_near_eof_probe_is_not_displacing(self): + self.assertTrue(views._is_near_eof_probe("bytes=2527702896-")) + self.assertFalse(views._should_displace_busy_playback("bytes=2527702896-")) + + def test_near_eof_probe_uses_cached_content_length(self): + # 5 MB into a 10 MB file is a scrub, not a tail probe. + self.assertFalse( + views._is_near_eof_probe("bytes=5000000-", content_length="10000000") + ) + self.assertTrue( + views._should_displace_busy_playback("bytes=5000000-", content_length="10000000") + ) + # Within 512 KB of EOF is a tail probe once length is known. + self.assertTrue( + views._is_near_eof_probe("bytes=9990000-", content_length="10000000") + ) + + def test_midfile_seek_is_displacing(self): + self.assertTrue(views._should_displace_busy_playback("bytes=5000000-")) + + def test_small_nonzero_range_is_displacing(self): + self.assertTrue(views._should_displace_busy_playback("bytes=1000-")) + + +class TimeshiftScrubPreemptTests(TestCase): + """Scrub/range requests must stop the in-flight stream and reuse the pooled + provider slot instead of opening parallel upstream connections.""" + + def setUp(self): + self.redis = _FakeRedis() + self.user = MagicMock(id=5) + self.factory = RequestFactory() + + def _conn(self, media_id, client_id): + return { + "media_id": media_id, + "client_id": client_id, + "connected_at": 0.0, + "type": "timeshift", + } + + def test_preempt_stops_sibling_clients_of_same_playback(self): + connections = [ + self._conn(f"{TEST_MEDIA_ID}_111", TEST_SESSION_ID), + self._conn("timeshift_9_2026-06-08-17-00_222", "timeshift_other"), + ] + with patch.object(views, "get_user_active_connections", + return_value=connections), \ + patch.object(views, "_unregister_stats_client") as unregister_mock: + views._preempt_playback_streams(self.redis, TEST_SESSION_ID, self.user) + unregister_mock.assert_called_once_with( + self.redis, f"{TEST_MEDIA_ID}_111", TEST_SESSION_ID, + ) + from apps.proxy.live_proxy.redis_keys import RedisKeys + stop_key = RedisKeys.client_stop(f"{TEST_MEDIA_ID}_111", TEST_SESSION_ID) + self.assertIn(stop_key, self.redis.store) + + def test_preempt_leaves_other_playbacks_alone(self): + connections = [ + self._conn("timeshift_8_2026-06-09-20-00_111", "timeshift_other_pos"), + ] + with patch.object(views, "get_user_active_connections", + return_value=connections), \ + patch.object(views, "_unregister_stats_client") as unregister_mock: + views._preempt_playback_streams(self.redis, TEST_SESSION_ID, self.user) + unregister_mock.assert_not_called() + + def test_busy_pool_returns_503_instead_of_second_provider_connection(self): + _seed_pool_session(self.redis, session_id=TEST_SESSION_ID) + request = self.factory.get( + _proxy_url(TEST_SESSION_ID), + HTTP_RANGE="bytes=1000-", + ) + streams = [_make_catchup_stream(account_id=1, stream_id="111", profile_id=31)] + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", return_value=streams), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "check_user_stream_limits", return_value=True), \ + patch.object(views, "RedisClient") as redis_cls, \ + patch.object(views, "reserve_profile_slot", return_value=(True, 1, None)), \ + patch.object(views, "release_profile_slot"), \ + patch.object(views, "get_transformed_credentials", side_effect=_fake_creds), \ + patch.object(views, "get_user_active_connections", return_value=[]), \ + patch.object(views, "_preempt_playback_streams") as preempt_mock, \ + patch.object(views, "_wait_for_idle_pool_session", return_value=None), \ + patch.object(views, "_attempt_timeshift_stream") as attempt_mock: + redis_cls.get_client.return_value = self.redis + channel_cls.objects.get.return_value = MagicMock( + id=8, name="Test", logo_id=None, + ) + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts", + ) + self.assertEqual(response.status_code, 503) + preempt_mock.assert_called_once() + attempt_mock.assert_not_called() + self.assertEqual(len(self._pool_entry_ids()), 1) + + def _pool_entry_ids(self): + return [k for k in self.redis.store if k.startswith("timeshift_pool:")] + + def test_startup_bytes_zero_deferred_without_preempt(self): + _seed_pool_session( + self.redis, session_id=TEST_SESSION_ID, serving_range="start", + ) + request = self.factory.get( + _proxy_url(TEST_SESSION_ID), + HTTP_RANGE="bytes=0-", + ) + streams = [_make_catchup_stream(account_id=1, stream_id="111", profile_id=31)] + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", return_value=streams), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "check_user_stream_limits", return_value=True), \ + patch.object(views, "RedisClient") as redis_cls, \ + patch.object(views, "reserve_profile_slot", return_value=(True, 1, None)), \ + patch.object(views, "release_profile_slot"), \ + patch.object(views, "get_transformed_credentials", side_effect=_fake_creds), \ + patch.object(views, "get_user_active_connections", return_value=[]), \ + patch.object(views, "_preempt_playback_streams") as preempt_mock, \ + patch.object(views, "_attempt_timeshift_stream") as attempt_mock: + redis_cls.get_client.return_value = self.redis + channel_cls.objects.get.return_value = MagicMock( + id=8, name="Test", logo_id=None, + ) + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts", + ) + self.assertEqual(response.status_code, 503) + preempt_mock.assert_not_called() + attempt_mock.assert_not_called() + + def test_eof_probe_deferred_without_preempt(self): + _seed_pool_session(self.redis, session_id=TEST_SESSION_ID) + request = self.factory.get( + _proxy_url(TEST_SESSION_ID), + HTTP_RANGE="bytes=2527702896-", + ) + streams = [_make_catchup_stream(account_id=1, stream_id="111", profile_id=31)] + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", return_value=streams), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "check_user_stream_limits", return_value=True), \ + patch.object(views, "RedisClient") as redis_cls, \ + patch.object(views, "reserve_profile_slot", return_value=(True, 1, None)), \ + patch.object(views, "release_profile_slot"), \ + patch.object(views, "get_transformed_credentials", side_effect=_fake_creds), \ + patch.object(views, "get_user_active_connections", return_value=[]), \ + patch.object(views, "_preempt_playback_streams") as preempt_mock, \ + patch.object(views, "_attempt_timeshift_stream") as attempt_mock: + redis_cls.get_client.return_value = self.redis + channel_cls.objects.get.return_value = MagicMock( + id=8, name="Test", logo_id=None, + ) + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts", + ) + self.assertEqual(response.status_code, 503) + preempt_mock.assert_not_called() + attempt_mock.assert_not_called() + + def test_create_pool_session_rejects_duplicate_entry(self): + first = views._create_pool_session( + self.redis, + session_id=TEST_SESSION_ID, + media_id=TEST_MEDIA_ID, + user_id=5, + client_ip="1.2.3.4", + client_user_agent="test-agent", + account_id=1, + profile_id=31, + stream_id="111", + provider_timestamp="2026", + ) + second = views._create_pool_session( + self.redis, + session_id=TEST_SESSION_ID, + media_id=TEST_MEDIA_ID, + user_id=5, + client_ip="1.2.3.4", + client_user_agent="test-agent", + account_id=2, + profile_id=41, + stream_id="222", + provider_timestamp="2026", + ) + self.assertTrue(first) + self.assertFalse(second) + self.assertTrue(self.redis.exists(f"timeshift_pool:{TEST_SESSION_ID}")) + + def test_scrub_reuses_idle_pool_without_opening_failover(self): + _seed_pool_session(self.redis, session_id=TEST_SESSION_ID) + with patch.object(views, "release_profile_slot"): + views._release_pool_session(self.redis, TEST_SESSION_ID, 31) + + request = self.factory.get( + _proxy_url(TEST_SESSION_ID), + HTTP_RANGE="bytes=5000-", + ) + streams = [_make_catchup_stream(account_id=1, stream_id="111", profile_id=31)] + profile = MagicMock(id=31) + ok = MagicMock(status_code=206) + with patch.object(views, "_authenticate_user", return_value=MagicMock(id=5)), \ + patch.object(views, "network_access_allowed", return_value=True), \ + patch.object(views, "Channel") as channel_cls, \ + patch.object(views, "_user_can_access_channel", return_value=True), \ + patch.object(views, "get_channel_catchup_streams", return_value=streams), \ + patch.object(views, "get_programme_duration", return_value=40), \ + patch.object(views, "check_user_stream_limits", return_value=True), \ + patch.object(views, "RedisClient") as redis_cls, \ + patch.object(views, "reserve_profile_slot", + return_value=(True, 1, None)) as reserve_mock, \ + patch.object(views, "release_profile_slot"), \ + patch.object(views.M3UAccountProfile.objects, "get", + return_value=profile), \ + patch.object(views, "get_transformed_credentials", side_effect=_fake_creds), \ + patch.object(views, "get_user_active_connections", return_value=[]), \ + patch.object(views, "_preempt_playback_streams") as preempt_mock, \ + patch.object(views, "_stream_reused_session", return_value=ok) as reuse_mock, \ + patch.object(views, "_attempt_timeshift_stream") as attempt_mock: + redis_cls.get_client.return_value = self.redis + channel_cls.objects.get.return_value = MagicMock( + id=8, name="Test", logo_id=None, + ) + response = views.timeshift_proxy( + request, "u", "p", "8", "2026-06-08:17-00", "8.ts", + ) + self.assertIs(response, ok) + preempt_mock.assert_not_called() + reuse_mock.assert_called_once() + attempt_mock.assert_not_called() + # Pool acquire re-reserves the idle slot once; failover must not add another. + reserve_mock.assert_called_once_with(profile, self.redis) + + + +class RollupSelfHealDbTests(TestCase): + """Catch-up flag consistency after stream removal. + + The ChannelStream signal handles bulk deletes (locked by a regression test). + The account-scoped rollup self-heals stale flags on channels still linked + to that account. + """ + + @classmethod + def setUpTestData(cls): + from apps.m3u.models import M3UAccount + + cls.account = M3UAccount.objects.create( + name="ts-rollup-account", server_url="http://example.test", + account_type="XC", is_active=True, + ) + + def _make_channel_with_catchup_stream(self, name, days=5): + from apps.channels.models import Channel, ChannelStream, Stream + + channel = Channel.objects.create(name=name) + stream = Stream.objects.create( + name=f"{name}-stream", url=f"http://example.test/{name}", + m3u_account=self.account, is_catchup=True, catchup_days=days, + ) + ChannelStream.objects.create(channel=channel, stream=stream, order=0) + return channel, stream + + def test_bulk_stream_delete_resets_channel_flags_via_signal(self): + # cleanup_streams() removes stale streams with a queryset bulk delete; + # the cascaded ChannelStream rows still fire post_delete (signal + # listeners disable Django's fast-delete path), which must reset the + # channel's denormalized catch-up fields. + from apps.channels.models import Stream + + channel, stream = self._make_channel_with_catchup_stream("ts-rollup-bulk") + channel.refresh_from_db() + self.assertTrue(channel.is_catchup) + self.assertEqual(channel.catchup_days, 5) + + Stream.objects.filter(id=stream.id).delete() + + channel.refresh_from_db() + self.assertFalse(channel.is_catchup) + self.assertEqual(channel.catchup_days, 0) + + def test_rollup_self_heals_stale_channel_with_non_catchup_stream(self): + # Channel still linked to the account but no active catch-up streams + # (e.g. catch-up flag cleared on import). Rollup must reset stale flags. + from apps.channels.models import Channel, ChannelStream, Stream + from apps.m3u.tasks import rollup_channel_catchup_fields + + channel = Channel.objects.create(name="ts-rollup-stale") + stream = Stream.objects.create( + name="ts-rollup-stale-stream", + url="http://example.test/ts-rollup-stale", + m3u_account=self.account, + is_catchup=False, + catchup_days=0, + ) + ChannelStream.objects.create(channel=channel, stream=stream, order=0) + Channel.objects.filter(pk=channel.pk).update(is_catchup=True, catchup_days=9) + + rollup_channel_catchup_fields(self.account.id) + + channel.refresh_from_db() + self.assertFalse(channel.is_catchup) + self.assertEqual(channel.catchup_days, 0) + + def test_rollup_self_heal_skips_channels_not_linked_to_account(self): + from apps.channels.models import Channel + from apps.m3u.models import M3UAccount + from apps.m3u.tasks import rollup_channel_catchup_fields + + other_account = M3UAccount.objects.create( + name="ts-rollup-other", + server_url="http://example.test/other", + account_type="XC", + is_active=True, + ) + channel = Channel.objects.create(name="ts-rollup-unrelated") + Channel.objects.filter(pk=channel.pk).update(is_catchup=True, catchup_days=9) + + rollup_channel_catchup_fields(other_account.id) + + channel.refresh_from_db() + self.assertTrue(channel.is_catchup) + self.assertEqual(channel.catchup_days, 9) + + def test_rollup_keeps_and_corrects_channels_with_catchup_streams(self): + # The self-heal pass must not touch channels that legitimately have + # catch-up streams. The account-scoped pass still corrects their values. + from apps.channels.models import Channel + from apps.m3u.tasks import rollup_channel_catchup_fields + + channel, _ = self._make_channel_with_catchup_stream("ts-rollup-valid", days=7) + # Knock the denormalized values out of sync (bypasses signals). + Channel.objects.filter(pk=channel.pk).update(is_catchup=False, catchup_days=0) + + rollup_channel_catchup_fields(self.account.id) + + channel.refresh_from_db() + self.assertTrue(channel.is_catchup) + self.assertEqual(channel.catchup_days, 7) diff --git a/apps/timeshift/views.py b/apps/timeshift/views.py new file mode 100644 index 00000000..0ba3a5a7 --- /dev/null +++ b/apps/timeshift/views.py @@ -0,0 +1,1355 @@ +"""XC catch-up (timeshift) proxy with multi-provider failover.""" + +import hmac +import itertools +import logging +import secrets +import time +from urllib.parse import urlencode + +import requests +from django.core.cache import cache +from django.http import ( + Http404, + HttpResponse, + HttpResponseBadRequest, + HttpResponseForbidden, + HttpResponseNotFound, + StreamingHttpResponse, +) + +from apps.accounts.models import User +from apps.channels.models import Channel +from apps.channels.utils import get_channel_catchup_streams +from apps.m3u.connection_pool import release_profile_slot, reserve_profile_slot +from apps.m3u.models import M3UAccount, M3UAccountProfile +from apps.m3u.tasks import get_transformed_credentials +from apps.proxy.live_proxy.config_helper import ConfigHelper +from apps.proxy.live_proxy.constants import ChannelMetadataField, ChannelState +from apps.proxy.live_proxy.redis_keys import RedisKeys +from apps.proxy.live_proxy.utils import get_client_ip +from apps.proxy.utils import ( + check_user_stream_limits, + find_ts_sync, + get_user_active_connections, +) +from core.utils import RedisClient +from dispatcharr.utils import network_access_allowed + +from .helpers import ( + TimeshiftCredentials, + build_timeshift_candidate_urls, + convert_timestamp_to_provider_tz, + get_programme_duration, + parse_catchup_timestamp, +) + +logger = logging.getLogger(__name__) + +CLIENT_TTL_SECONDS = 60 +_MATCH_SCORE_THRESHOLD = 8 # client_ip (5) + client_user_agent (3) + + +def timeshift_proxy(request, username, password, stream_id, timestamp, duration): # noqa: ARG001 stream_id + """Proxy an XC catch-up request to the provider with multi-stream failover. + + URL shape (iPlayTV / TiviMate): + ``stream_id``: EPG channel number (ignored here). + ``duration``: Dispatcharr ``Channel.id`` (XC API exposes channel.id as stream_id). + ``timestamp``: UTC programme start (``YYYY-MM-DD:HH-MM`` or XC colon form + ``YYYY-MM-DD:HH:MM:SS``). + """ + raw_id = duration[:-3] if duration.endswith(".ts") else duration + + user = _authenticate_user(username, password) + if user is None: + return HttpResponseForbidden("Invalid credentials") + + if not network_access_allowed(request, "XC_API", user): + return HttpResponseForbidden("Access denied") + + try: + channel = Channel.objects.get(id=int(raw_id)) + except (Channel.DoesNotExist, ValueError, TypeError): + raise Http404("Channel not found") from None + + if not _user_can_access_channel(user, channel): + return HttpResponseForbidden("Access denied") + + # Shape helpers pass through on parse failure; reject bad input before upstream. + if parse_catchup_timestamp(timestamp) is None: + return HttpResponseBadRequest("Invalid timestamp") + + catchup_streams = get_channel_catchup_streams(channel) + if not catchup_streams: + return HttpResponseBadRequest("Timeshift not supported for this channel") + + debug = logger.isEnabledFor(logging.DEBUG) + + # EPG duration lookup stays in UTC; provider TZ conversion is per-attempt below. + duration_minutes = get_programme_duration(channel, timestamp) + + safe_ts = timestamp.replace(":", "-").replace("/", "-") + client_ip = get_client_ip(request) + client_user_agent = request.META.get("HTTP_USER_AGENT", "") or "" + range_header = request.META.get("HTTP_RANGE") + channel_logo_id = getattr(channel, "logo_id", None) + + redis_client = RedisClient.get_client() + + # Content identity (channel + catch-up position). Provider slot sharing is + # scoped per client session; never assume all requests for the same + # programme belong to one viewer. + media_id = f"timeshift_{channel.id}_{safe_ts}" + + session_id = request.GET.get("session_id") + if not session_id: + logger.debug("Timeshift session redirect: %s (new session)", request.path) + return _redirect_with_new_session(request) + + session_entry = _get_pool_entry(redis_client, session_id) + if session_entry and not _pool_entry_owned_by_user(session_entry, user.id): + logger.info( + "Timeshift: rejecting foreign session_id for user %s", user.id, + ) + return _redirect_with_new_session(request) + + # Stable client identity for stats, stop keys, and the provider pool. + effective_session_id = session_id + client_id = session_id + + # Reuse an idle pool owned by this session, or fingerprint-match a prior + # idle session from the same client (VOD-style) before opening upstream. + if not session_entry: + matched = _find_matching_idle_session( + redis_client, + media_id=media_id, + user_id=user.id, + client_ip=client_ip, + client_user_agent=client_user_agent, + ) + if matched: + logger.info( + "Timeshift fingerprint matched idle session %s for %s", + matched, session_id, + ) + effective_session_id = matched + client_id = matched + + if debug: + if effective_session_id != session_id: + logger.debug( + "Timeshift request: channel=%s media=%s session=%s " + "effective=%s user=%s range=%s ip=%s", + channel.name, media_id, session_id, effective_session_id, + user.id, range_header or "(none)", client_ip, + ) + else: + logger.debug( + "Timeshift request: channel=%s media=%s session=%s " + "user=%s range=%s ip=%s", + channel.name, media_id, effective_session_id, user.id, + range_header or "(none)", client_ip, + ) + + # Displace this user's prior catch-up on other positions of this channel. + _terminate_previous_timeshift_sessions( + redis_client, user, channel.id, media_id, effective_session_id, + ) + + if not check_user_stream_limits(user, client_id, media_id=media_id): + return HttpResponseForbidden("Stream limit exceeded") + + if effective_session_id == session_id: + pool = _snapshot_from_entry(session_entry) + else: + pool = _pool_snapshot(redis_client, effective_session_id) + pool_exists = pool is not None + pool_busy = pool["busy"] if pool else False + pool_content_length = pool["content_length"] if pool else None + busy_serving_range = pool["serving_range"] if pool else None + + acquired = None + if pool_exists: + if pool_busy: + if _should_displace_busy_playback( + range_header, pool_content_length, busy_serving_range, + ): + _preempt_playback_streams(redis_client, effective_session_id, user) + acquired = _wait_for_idle_pool_session( + redis_client, + effective_session_id, + user_id=user.id, + wait_seconds=_POOL_PREEMPT_WAIT_SECONDS, + ) + else: + acquired = _acquire_idle_pool_session( + redis_client, effective_session_id, user_id=user.id, + ) + + if acquired is not None: + descriptor, profile = acquired + reuse_response = _stream_reused_session( + redis_client, + session_id=effective_session_id, + descriptor=descriptor, + profile=profile, + channel=channel, + safe_ts=safe_ts, + timestamp=timestamp, + duration_minutes=duration_minutes, + client_id=client_id, + client_ip=client_ip, + range_header=range_header, + channel_logo_id=channel_logo_id, + user=user, + debug=debug, + ) + if reuse_response is not None: + return reuse_response + + if pool_exists and pool_busy and not _should_displace_busy_playback( + range_header, pool_content_length, busy_serving_range, + ): + logger.debug( + "Timeshift: deferring non-displacing request for session %s range=%s", + effective_session_id, range_header or "(none)", + ) + return HttpResponse("Stream slot busy", status=503) + + if pool_exists and pool_busy: + logger.warning( + "Timeshift: session %s did not become idle in time", + effective_session_id, + ) + return HttpResponse("Stream slot busy", status=503) + + last_response = None + decisive_accounts = set() + capacity_blocked = False + for catchup_stream in catchup_streams: + m3u_account = catchup_stream.m3u_account + if m3u_account is None or m3u_account.account_type != "XC": + continue + if m3u_account.id in decisive_accounts: + continue + + stream_id_value = (catchup_stream.custom_properties or {}).get("stream_id") + if stream_id_value is None: + continue + + m3u_profiles = list(m3u_account.profiles.filter(is_active=True)) + default_profile = next((p for p in m3u_profiles if p.is_default), None) + if default_profile is None: + logger.debug( + "Timeshift: account %s has no active default profile, skipping", + m3u_account.id, + ) + continue + profile_walk = [default_profile] + [ + p for p in m3u_profiles if not p.is_default + ] + + # Providers index archives in their own timezone (from server_info on auth). + provider_tz_name = None + _server_info = (default_profile.custom_properties or {}).get("server_info") or {} + if isinstance(_server_info, dict): + provider_tz_name = _server_info.get("timezone") + provider_timestamp = convert_timestamp_to_provider_tz(timestamp, provider_tz_name) + + # Reserve a provider profile slot before connecting (same contract as live/VOD). + reserved_profile = None + for profile in profile_walk: + if redis_client is None: + reserved_profile = profile + break + reserved, _count, reason = reserve_profile_slot(profile, redis_client) + if reserved: + reserved_profile = profile + break + logger.info( + "Timeshift: profile %s %s on account %s, trying next profile", + profile.id, reason or "unavailable", m3u_account.id, + ) + if reserved_profile is None: + capacity_blocked = True + logger.warning( + "Timeshift: all profiles at capacity on account %s for channel %s", + m3u_account.id, channel.name, + ) + continue + + if not _create_pool_session( + redis_client, + session_id=effective_session_id, + media_id=media_id, + user_id=user.id, + client_ip=client_ip, + client_user_agent=client_user_agent, + account_id=m3u_account.id, + profile_id=reserved_profile.id, + stream_id=stream_id_value, + provider_timestamp=provider_timestamp, + ): + try: + release_profile_slot(reserved_profile.id, redis_client) + except Exception as exc: + logger.warning( + "Timeshift slot release failed after pool race on profile %s: %s", + reserved_profile.id, exc, + ) + logger.debug( + "Timeshift: pool entry already exists for session %s, deferring", + effective_session_id, + ) + return HttpResponse("Stream slot busy", status=503) + release_cb = _make_release_once( + redis_client, effective_session_id, reserved_profile.id + ) + + try: + response = _attempt_timeshift_stream( + m3u_account=m3u_account, + profile=reserved_profile, + stream_id_value=stream_id_value, + provider_timestamp=provider_timestamp, + provider_tz_name=provider_tz_name, + duration_minutes=duration_minutes, + channel=channel, + safe_ts=safe_ts, + timestamp=timestamp, + client_id=client_id, + client_ip=client_ip, + range_header=range_header, + channel_logo_id=channel_logo_id, + user=user, + redis_client=redis_client, + debug=debug, + release_cb=release_cb, + pool_session_id=effective_session_id, + ) + except Exception: + _discard_pool_session(redis_client, effective_session_id, reserved_profile.id) + raise + if response.status_code < 400: + # Streaming: the generator's close path frees the slot via release_cb. + return response + + if getattr(response, "timeshift_passthrough", False) is True: + # Terminal range answer (e.g. 416 past EOF): the upstream session is + # healthy, so free the slot but keep the entry idle for the next + # probe, and return verbatim without failing over to other accounts. + release_cb() + return response + + # Real failure: drop this session entirely and fail over. + _discard_pool_session(redis_client, effective_session_id, reserved_profile.id) + last_response = response + if getattr(response, "timeshift_decisive", False): + decisive_accounts.add(m3u_account.id) + logger.warning( + "Timeshift attempt failed (HTTP %d%s) on account %s for channel %s, " + "trying next catch-up stream", + response.status_code, + ", decisive: skipping this account's other streams" + if m3u_account.id in decisive_accounts else "", + m3u_account.id, channel.name, + ) + + if last_response is not None: + return last_response + if capacity_blocked: + return HttpResponse("No available stream slot", status=503) + return HttpResponseBadRequest("Cannot build timeshift URL") + + +def _authenticate_user(username, password): + try: + user = User.objects.get(username=username) + except User.DoesNotExist: + return None + expected = (user.custom_properties or {}).get("xc_password") + if not expected: + return None + if not hmac.compare_digest(str(expected), str(password)): + return None + return user + + +def _user_can_access_channel(user, channel): + if user.user_level < channel.user_level: + return False + if user.user_level >= User.UserLevel.ADMIN: + return True + profile_count = user.channel_profiles.count() + if profile_count == 0: + return True + return ( + type(channel).objects.filter( + id=channel.id, + channelprofilemembership__enabled=True, + channelprofilemembership__channel_profile__in=user.channel_profiles.all(), + ) + .exists() + ) + + +# Per-client session pool (keyed by session_id from the 301 redirect). Each +# viewer gets their own provider slot even when watching the same catch-up +# programme. Idle sessions can be fingerprint-matched (VOD-style) when a client +# returns without its prior session_id. +_POOL_KEY = "timeshift_pool:{session_id}" +_POOL_LOCK_KEY = "timeshift_pool_lock:{session_id}" +_POOL_ENTRY_TTL = 6 * 3600 +_POOL_IDLE_TTL = 30 +_POOL_WAIT_SECONDS = 1.0 +_POOL_PREEMPT_WAIT_SECONDS = 5.0 +_POOL_POLL_INTERVAL = 0.05 +_EOF_PROBE_TAIL_BYTES = 512_000 +_EOF_PROBE_UNKNOWN_LENGTH_MIN = 100_000_000 + + +def _pool_key(session_id): + return _POOL_KEY.format(session_id=session_id) + + +def _parse_range_start(range_header): + """Return the byte offset from a ``Range: bytes=START-`` header, or None.""" + if not range_header or not range_header.startswith("bytes="): + return None + start_part = range_header[6:].split("-", 1)[0] + if not start_part: + return None + try: + return int(start_part) + except (TypeError, ValueError): + return None + + +def _is_near_eof_probe(range_header, content_length=None): + """True for tail/duration probes IPTV clients fire during startup.""" + start = _parse_range_start(range_header) + if start is None: + return False + if content_length is not None: + try: + total = int(content_length) + except (TypeError, ValueError): + total = None + else: + return start >= max(0, total - _EOF_PROBE_TAIL_BYTES) + return start >= _EOF_PROBE_UNKNOWN_LENGTH_MIN + + +def _should_displace_busy_playback( + range_header, content_length=None, busy_serving_range=None, +): + """True when this request should stop the in-flight stream (actual scrub).""" + if not range_header: + return False + start = _parse_range_start(range_header) + if start is None: + return False + if _is_near_eof_probe(range_header, content_length): + return False + if start == 0: + # Only displace a known full-file probe; unknown busy context is not a scrub. + return busy_serving_range == "none" + return True + + +def _score_pool_fingerprint(entry, client_ip, client_user_agent): + """Score IP/UA overlap for fingerprint adoption (user and media pre-filtered).""" + score = 0 + if entry.get("client_ip") and entry.get("client_ip") == client_ip: + score += 5 + if entry.get("client_user_agent") and entry.get("client_user_agent") == client_user_agent: + score += 3 + return score + + +def _mint_timeshift_session_id(): + return f"timeshift_{secrets.token_urlsafe(16)}" + + +def _redirect_with_new_session(request): + session_id = _mint_timeshift_session_id() + query_params = {k: request.GET.getlist(k) for k in request.GET} + query_params["session_id"] = [session_id] + redirect_url = f"{request.path}?{urlencode(query_params, doseq=True)}" + return HttpResponse(status=301, headers={"Location": redirect_url}) + + +def _pool_entry_owned_by_user(entry, user_id): + """True when *entry* is unclaimed or owned by *user_id*.""" + if not entry or not entry.get("profile_id"): + return True + owner = entry.get("user_id") + if owner is None or owner == "": + return False + return str(owner) == str(user_id) + + +def _find_matching_idle_session( + redis_client, *, media_id, user_id, client_ip, client_user_agent, +): + """Find an idle pooled session that likely belongs to the same client.""" + if redis_client is None: + return None + matches = [] + try: + cursor = 0 + while True: + cursor, keys = redis_client.scan( + cursor, match="timeshift_pool:timeshift_*", count=100, + ) + for key in keys: + try: + data = redis_client.hgetall(key) + if not data or data.get("busy") == "1": + continue + if str(data.get("user_id") or "") != str(user_id): + continue + if str(data.get("media_id") or "") != str(media_id): + continue + session_id = key.rsplit(":", 1)[-1] + score = _score_pool_fingerprint( + data, client_ip, client_user_agent, + ) + if score >= _MATCH_SCORE_THRESHOLD: + last_activity = float(data.get("last_activity") or "0") + matches.append((session_id, score, last_activity)) + except Exception as exc: + logger.debug("Timeshift pool scan skip %s: %s", key, exc) + if cursor == 0: + break + except Exception as exc: + logger.warning("Timeshift idle session search failed: %s", exc) + return None + + if not matches: + return None + matches.sort(key=lambda item: (item[1], item[2]), reverse=True) + best = matches[0][0] + logger.debug( + "Timeshift idle match: session=%s score=%s media=%s", + best, matches[0][1], media_id, + ) + return best + + +def _get_pool_entry(redis_client, session_id): + if redis_client is None or not session_id: + return {} + try: + return redis_client.hgetall(_pool_key(session_id)) or {} + except Exception: + return {} + + +def _snapshot_from_entry(entry): + if not entry: + return None + busy = entry.get("busy") == "1" + return { + "entry": entry, + "busy": busy, + "serving_range": (entry.get("serving_range") or "none") if busy else None, + "content_length": entry.get("content_length"), + } + + +def _pool_snapshot(redis_client, session_id): + """Single HGETALL view of pool state for request handling.""" + return _snapshot_from_entry(_get_pool_entry(redis_client, session_id)) + + +def _store_pool_serving_range(redis_client, session_id, range_header): + if redis_client is None or not session_id: + return + start = _parse_range_start(range_header) + if not range_header: + serving_range = "none" + elif start == 0: + serving_range = "start" + else: + serving_range = "range" + try: + redis_client.hset(_pool_key(session_id), "serving_range", serving_range) + except Exception as exc: + logger.debug("Timeshift pool serving_range store failed: %s", exc) + + +def _store_pool_content_length(redis_client, session_id, upstream_response): + if redis_client is None or not session_id or upstream_response is None: + return + content_length = upstream_response.headers.get("Content-Length") + content_range = upstream_response.headers.get("Content-Range", "") + if content_range and "/" in content_range: + total = content_range.rsplit("/", 1)[-1] + if total != "*": + content_length = total + if not content_length: + return + try: + redis_client.hset( + _pool_key(session_id), "content_length", str(content_length), + ) + except Exception as exc: + logger.debug("Timeshift pool content_length store failed: %s", exc) + + +def _pool_lock(redis_client, session_id): + return redis_client.lock( + _POOL_LOCK_KEY.format(session_id=session_id), + timeout=10, + blocking_timeout=5, + ) + + +def _acquire_idle_pool_session(redis_client, session_id, *, user_id=None): + """Re-reserve an idle session's profile slot and mark it busy.""" + if redis_client is None or not session_id: + return None + key = _pool_key(session_id) + try: + with _pool_lock(redis_client, session_id): + data = redis_client.hgetall(key) + if not data or not data.get("profile_id"): + return None + if user_id is not None and not _pool_entry_owned_by_user(data, user_id): + return None + if data.get("busy") == "1": + return None + try: + profile = M3UAccountProfile.objects.get(id=int(data["profile_id"])) + except M3UAccountProfile.DoesNotExist: + redis_client.delete(key) + return None + reserved, _count, _reason = reserve_profile_slot(profile, redis_client) + if not reserved: + return None + redis_client.hset(key, mapping={ + "busy": "1", + "last_activity": str(time.time()), + }) + redis_client.expire(key, _POOL_ENTRY_TTL) + return dict(data), profile + except Exception as exc: + logger.warning("Timeshift pool acquire failed for %s: %s", session_id, exc) + return None + + +def _wait_for_idle_pool_session( + redis_client, session_id, *, user_id=None, wait_seconds=_POOL_WAIT_SECONDS, +): + if redis_client is None or not session_id: + return None + deadline = time.time() + wait_seconds + while True: + acquired = _acquire_idle_pool_session( + redis_client, session_id, user_id=user_id, + ) + if acquired is not None: + return acquired + if not _get_pool_entry(redis_client, session_id): + return None + if time.time() >= deadline: + return None + time.sleep(_POOL_POLL_INTERVAL) + + +def _create_pool_session( + redis_client, + *, + session_id, + media_id, + user_id, + client_ip, + client_user_agent, + account_id, + profile_id, + stream_id, + provider_timestamp, +): + """Register an already-reserved slot for this client session.""" + if redis_client is None or not session_id: + return False + key = _pool_key(session_id) + now = str(time.time()) + try: + with _pool_lock(redis_client, session_id): + if redis_client.exists(key): + return False + redis_client.hset(key, mapping={ + "media_id": str(media_id), + "user_id": str(user_id), + "client_ip": str(client_ip or ""), + "client_user_agent": str(client_user_agent or ""), + "account_id": str(account_id), + "profile_id": str(profile_id), + "stream_id": str(stream_id), + "provider_timestamp": str(provider_timestamp), + "busy": "1", + "last_activity": now, + }) + redis_client.expire(key, _POOL_ENTRY_TTL) + return True + except Exception as exc: + logger.warning("Timeshift pool create failed for %s: %s", session_id, exc) + return False + + +def _release_pool_session(redis_client, session_id, profile_id): + if redis_client is None: + return + if profile_id is not None: + try: + release_profile_slot(int(profile_id), redis_client) + except Exception as exc: + logger.warning( + "Timeshift slot release failed for profile %s: %s", profile_id, exc + ) + if not session_id: + return + key = _pool_key(session_id) + try: + with _pool_lock(redis_client, session_id): + if redis_client.exists(key): + redis_client.hset(key, mapping={ + "busy": "0", + "last_activity": str(time.time()), + }) + redis_client.expire(key, _POOL_IDLE_TTL) + except Exception as exc: + logger.warning("Timeshift pool release failed for %s: %s", session_id, exc) + + +def _discard_pool_session(redis_client, session_id, profile_id): + if redis_client is None: + return + if profile_id is not None: + try: + release_profile_slot(int(profile_id), redis_client) + except Exception as exc: + logger.warning( + "Timeshift slot release failed for profile %s: %s", profile_id, exc + ) + if not session_id: + return + try: + with _pool_lock(redis_client, session_id): + redis_client.delete(_pool_key(session_id)) + except Exception as exc: + logger.warning("Timeshift pool discard failed for %s: %s", session_id, exc) + + +def _make_release_once(redis_client, session_id, profile_id): + state = {"done": False} + + def _release(): + if state["done"]: + return + state["done"] = True + _release_pool_session(redis_client, session_id, profile_id) + + return _release + + +def _preempt_playback_streams(redis_client, session_id, user): + """Stop in-flight streams for this client session only.""" + if redis_client is None or not session_id or user is None: + return + try: + for conn in get_user_active_connections(user.id): + if conn.get("type") != "timeshift": + continue + if conn.get("client_id") != session_id: + continue + conn_media_id = str(conn.get("media_id") or "") + old_client_id = conn.get("client_id") + logger.debug( + "Timeshift preempt: stopping client %s on %s for reuse", + old_client_id, conn_media_id, + ) + _unregister_stats_client(redis_client, conn_media_id, old_client_id) + stop_key = RedisKeys.client_stop(conn_media_id, old_client_id) + redis_client.setex(stop_key, 60, "true") + except Exception as exc: + logger.warning("Timeshift preempt failed: %s", exc) + + +def _terminate_previous_timeshift_sessions( + redis_client, user, channel_id, current_media_id, current_session_id, +): + """Displace this user's other catch-up positions on the same channel.""" + if redis_client is None or user is None: + return + prefix = f"timeshift_{channel_id}_" + try: + for conn in get_user_active_connections(user.id): + if conn.get("type") != "timeshift": + continue + if conn.get("client_id") == current_session_id: + continue + conn_media_id = str(conn.get("media_id") or "") + if not conn_media_id.startswith(prefix): + continue + if conn_media_id.startswith(f"{current_media_id}_") or conn_media_id == current_media_id: + continue + old_client_id = conn.get("client_id") + logger.info( + "Timeshift takeover: displacing session %s on %s", + old_client_id, conn_media_id, + ) + _unregister_stats_client(redis_client, conn_media_id, old_client_id) + stop_key = RedisKeys.client_stop(conn_media_id, old_client_id) + redis_client.setex(stop_key, 60, "true") + except Exception as exc: + logger.warning("Timeshift takeover check failed: %s", exc) + + +def _attempt_timeshift_stream( + *, + m3u_account, + profile, + stream_id_value, + provider_timestamp, + provider_tz_name, + duration_minutes, + channel, + safe_ts, + timestamp, + client_id, + client_ip, + range_header, + channel_logo_id, + user, + redis_client, + debug, + release_cb=None, + pool_session_id=None, +): + """Build the provider URL set for one (account, profile, stream) and stream it.""" + server_url, xc_username, xc_password = get_transformed_credentials( + m3u_account, profile + ) + creds = TimeshiftCredentials(server_url, xc_username, xc_password) + candidate_urls = build_timeshift_candidate_urls( + creds, stream_id_value, provider_timestamp, duration_minutes + ) + + try: + user_agent = m3u_account.get_user_agent().user_agent + except AttributeError: + user_agent = "" + + virtual_channel_id = f"timeshift_{channel.id}_{safe_ts}_{stream_id_value}" + + if debug: + logger.debug( + "Timeshift attempt: channel=%s ts=%s (provider tz=%s -> %s) " + "account=%s profile=%s provider_sid=%s vid=%s client=%s range=%s", + channel.name, timestamp, provider_tz_name, provider_timestamp, + m3u_account.id, profile.id, stream_id_value, + virtual_channel_id, client_id, range_header or "(none)", + ) + + return _stream_from_provider( + candidate_urls=candidate_urls, + user_agent=user_agent, + range_header=range_header, + virtual_channel_id=virtual_channel_id, + client_id=client_id, + client_ip=client_ip, + user=user, + channel_display_name=channel.name, + timestamp_utc=timestamp, + channel_logo_id=channel_logo_id, + m3u_profile_id=profile.id, + debug=debug, + account_id=m3u_account.id, + redis_client=redis_client, + release_cb=release_cb, + pool_session_id=pool_session_id, + ) + + +def _stream_reused_session( + redis_client, + *, + session_id, + descriptor, + profile, + channel, + safe_ts, + timestamp, + duration_minutes, + client_id, + client_ip, + range_header, + channel_logo_id, + user, + debug, +): + """Stream an idle pooled session that was just re-reserved for this request.""" + try: + m3u_account = M3UAccount.objects.get(id=int(descriptor["account_id"])) + except (M3UAccount.DoesNotExist, ValueError, TypeError): + _discard_pool_session(redis_client, session_id, profile.id) + return None + + provider_timestamp = descriptor.get("provider_timestamp") + if not provider_timestamp: + provider_tz_name = None + server_info = (profile.custom_properties or {}).get("server_info") or {} + if isinstance(server_info, dict): + provider_tz_name = server_info.get("timezone") + provider_timestamp = convert_timestamp_to_provider_tz( + timestamp, provider_tz_name + ) + + release_cb = _make_release_once(redis_client, session_id, profile.id) + try: + response = _attempt_timeshift_stream( + m3u_account=m3u_account, + profile=profile, + stream_id_value=descriptor["stream_id"], + provider_timestamp=provider_timestamp, + provider_tz_name=None, + duration_minutes=duration_minutes, + channel=channel, + safe_ts=safe_ts, + timestamp=timestamp, + client_id=client_id, + client_ip=client_ip, + range_header=range_header, + channel_logo_id=channel_logo_id, + user=user, + redis_client=redis_client, + debug=debug, + release_cb=release_cb, + pool_session_id=session_id, + ) + except Exception: + _discard_pool_session(redis_client, session_id, profile.id) + raise + + if response.status_code < 400: + return response + + if getattr(response, "timeshift_passthrough", False) is True: + release_cb() + return response + + _discard_pool_session(redis_client, session_id, profile.id) + return None + + +class _SlotReleasingStream: + """Iterator wrapper that releases the pool slot when WSGI closes the response.""" + + def __init__(self, generator, on_close): + self._generator = generator + self._on_close = on_close + + def __iter__(self): + return self + + def __next__(self): + return next(self._generator) + + def close(self): + try: + self._generator.close() + finally: + self._on_close() + + +def _register_stats_client( + redis_client, + virtual_channel_id, + client_id, + client_ip, + user_agent, + user, + *, + channel_display_name, + timestamp_utc, + primary_url, + channel_logo_id=None, + m3u_profile_id=None, +): + """Write Redis keys so catch-up viewers appear on ``/stats``.""" + if redis_client is None: + return + client_set_key = RedisKeys.clients(virtual_channel_id) + client_key = RedisKeys.client_metadata(virtual_channel_id, client_id) + metadata_key = RedisKeys.channel_metadata(virtual_channel_id) + now = str(time.time()) + client_payload = { + "user_agent": user_agent or "unknown", + "ip_address": client_ip, + "connected_at": now, + "last_active": now, + "user_id": str(user.id) if user is not None else "0", + "username": user.username if user is not None else "unknown", + } + metadata_payload = { + ChannelMetadataField.STATE: ChannelState.ACTIVE, + ChannelMetadataField.INIT_TIME: now, + ChannelMetadataField.OWNER: "timeshift", + ChannelMetadataField.CHANNEL_NAME: channel_display_name or "Timeshift", + ChannelMetadataField.STREAM_NAME: f"Catch-up @ {timestamp_utc} UTC" if timestamp_utc else "Catch-up", + ChannelMetadataField.URL: _redact_url(primary_url) if primary_url else "", + ChannelMetadataField.IS_TIMESHIFT: "1", + } + if channel_logo_id is not None: + metadata_payload[ChannelMetadataField.LOGO_ID] = str(channel_logo_id) + if m3u_profile_id is not None: + metadata_payload[ChannelMetadataField.M3U_PROFILE] = str(m3u_profile_id) + try: + pipe = redis_client.pipeline(transaction=False) + pipe.hset(client_key, mapping=client_payload) + pipe.expire(client_key, CLIENT_TTL_SECONDS) + pipe.sadd(client_set_key, client_id) + pipe.expire(client_set_key, CLIENT_TTL_SECONDS) + pipe.hset(metadata_key, mapping=metadata_payload) + pipe.expire(metadata_key, CLIENT_TTL_SECONDS) + pipe.execute() + except Exception as exc: + logger.warning("Timeshift stats register failed: %s", exc) + + +def _heartbeat_stats_client(redis_client, virtual_channel_id, client_id, bytes_delta=0): + if redis_client is None: + return + client_set_key = RedisKeys.clients(virtual_channel_id) + client_key = RedisKeys.client_metadata(virtual_channel_id, client_id) + metadata_key = RedisKeys.channel_metadata(virtual_channel_id) + try: + pipe = redis_client.pipeline(transaction=False) + pipe.hset(client_key, "last_active", str(time.time())) + pipe.expire(client_key, CLIENT_TTL_SECONDS) + pipe.expire(client_set_key, CLIENT_TTL_SECONDS) + if bytes_delta > 0: + pipe.hincrby(metadata_key, ChannelMetadataField.TOTAL_BYTES, bytes_delta) + pipe.expire(metadata_key, CLIENT_TTL_SECONDS) + pipe.execute() + except Exception as exc: + logger.debug("Timeshift stats heartbeat failed: %s", exc) + + +def _unregister_stats_client(redis_client, virtual_channel_id, client_id): + if redis_client is None: + return + client_set_key = RedisKeys.clients(virtual_channel_id) + client_key = RedisKeys.client_metadata(virtual_channel_id, client_id) + metadata_key = RedisKeys.channel_metadata(virtual_channel_id) + try: + redis_client.srem(client_set_key, client_id) + redis_client.delete(client_key) + if (redis_client.scard(client_set_key) or 0) == 0: + redis_client.delete(client_set_key) + redis_client.delete(metadata_key) + except Exception as exc: + logger.warning("Timeshift stats unregister failed: %s", exc) + + +def _open_upstream(url, user_agent, range_header): + """Open upstream HTTP; redirects are followed (XC load-balancer nodes).""" + # identity: raw peek bytes are not gzip-transparent. + headers = {"Accept-Encoding": "identity"} + if user_agent: + headers["User-Agent"] = user_agent + if range_header: + headers["Range"] = range_header + return requests.get( + url, + headers=headers, + stream=True, + timeout=ConfigHelper.connection_timeout(), + ) + + +_FORMAT_CACHE_KEY = "timeshift:format_idx:{}" +_FORMAT_CACHE_TTL = 3600 # 1 hour + + +def _get_cached_format_index(account_id): + """Index of the URL shape that last worked for this account, or None.""" + if account_id is None: + return None + return cache.get(_FORMAT_CACHE_KEY.format(account_id)) + + +def _set_cached_format_index(account_id, index): + if account_id is None: + return + cache.set(_FORMAT_CACHE_KEY.format(account_id), index, _FORMAT_CACHE_TTL) + + +def _passthrough_response(status, content_range=None): + """A terminal response handed straight to the client (no streaming). + + Marked so the failover loop and reuse path return it verbatim instead of + cascading other URL shapes or failing over to another provider. + """ + response = HttpResponse(status=status) + if content_range: + response["Content-Range"] = content_range + response.timeshift_passthrough = True + return response + + +def _stream_from_provider( + *, + candidate_urls, + user_agent, + range_header, + virtual_channel_id, + client_id, + client_ip, + user, + channel_display_name, + timestamp_utc, + channel_logo_id, + m3u_profile_id, + debug, + account_id=None, + redis_client=None, + release_cb=None, + pool_session_id=None, +): + """Try each upstream URL until one returns streamable MPEG-TS. + + Sets ``timeshift_decisive`` on auth/ban-class failures (401/403/406) so the + failover loop skips the rest of that account's streams. ``release_cb`` frees + the provider slot when the streaming response is closed. + """ + chunk_size = max(ConfigHelper.chunk_size(), 262144) + if release_cb is None: + release_cb = lambda: None # noqa: E731 + + cached_index = _get_cached_format_index(account_id) + if cached_index is not None and 0 <= cached_index < len(candidate_urls): + ordered_urls = [candidate_urls[cached_index]] + [ + u for i, u in enumerate(candidate_urls) if i != cached_index + ] + original_indices = [cached_index] + [ + i for i in range(len(candidate_urls)) if i != cached_index + ] + else: + ordered_urls = list(candidate_urls) + original_indices = list(range(len(candidate_urls))) + + # Peek for MPEG-TS sync; some providers return HTTP 200 with PHP/HTML errors. + upstream = None + last_status = None + last_url = ordered_urls[0] + winning_index = None + decisive_failure = False + for url, orig_idx in zip(ordered_urls, original_indices): + try: + response = _open_upstream(url, user_agent, range_header) + except requests.exceptions.RequestException as exc: + logger.error( + "Timeshift provider unreachable (%s): %s", + _redact_url(url), type(exc).__name__, + ) + return HttpResponseBadRequest("Provider connection error") + last_status = response.status_code + last_url = url + if debug: + logger.debug( + "Timeshift cascade[%d]: status=%d type=%s url=%s", + orig_idx, response.status_code, + response.headers.get("Content-Type", "?"), + _redact_url(url), + ) + if response.status_code == 416: + # Range Not Satisfiable: a seek/tail probe past EOF. Hand it back to + # the client verbatim. Byte offsets are file-specific, so trying + # other URL shapes or failing over to another provider is pointless + # and only multiplies upstream connections. + content_range = response.headers.get("Content-Range") + response.close() + return _passthrough_response(416, content_range) + if response.status_code in (200, 206): + peek = response.raw.read(1024) + sync_offset = find_ts_sync(peek) if peek else -1 + if sync_offset >= 0: + response._peek_data = peek[sync_offset:] + upstream = response + winning_index = orig_idx + break + # A 206 to a Range request legitimately starts mid-packet, so the + # sync byte rarely lands at offset 0. Trust the partial status and + # content type rather than the sync probe; only a full 200 carrying + # a PHP/HTML error page must be rejected here. + content_type = response.headers.get("Content-Type", "") + is_partial = response.status_code == 206 and bool(range_header) + if is_partial and peek and "html" not in content_type and "json" not in content_type: + response._peek_data = peek + upstream = response + winning_index = orig_idx + break + snippet = peek[:200].decode("utf-8", errors="replace") if peek else "(empty)" + logger.warning( + "Timeshift upstream returned %d but no TS sync in first %d " + "bytes (likely PHP error): %s, url=%s", + response.status_code, + len(peek) if peek else 0, + snippet.replace("\n", " ")[:120], + _redact_url(url), + ) + response.close() + last_status = 404 # Treat as soft rejection for cascade + continue + response.close() + # Auth/ban-class statuses stop trying more shapes on this account; 5xx does not. + code = response.status_code + if code in (401, 403, 406) or 300 <= code < 400: + decisive_failure = True + break + + if winning_index is not None: + _set_cached_format_index(account_id, winning_index) + + if upstream is None: + logger.error("Timeshift upstream rejected: status=%s url=%s", + last_status, _redact_url(last_url)) + # Map 404/403 to meaningful client responses; other failures stay 400. + if last_status == 404: + failure = HttpResponseNotFound("Catch-up not available yet") + elif last_status == 403: + failure = HttpResponseForbidden("Provider denied access") + else: + failure = HttpResponseBadRequest("Provider error") + failure.timeshift_decisive = decisive_failure + return failure + + content_type = upstream.headers.get("Content-Type", "video/mp2t") + content_range = upstream.headers.get("Content-Range", "") + status = upstream.status_code + + _store_pool_content_length(redis_client, pool_session_id, upstream) + _store_pool_serving_range(redis_client, pool_session_id, range_header) + + _register_stats_client( + redis_client, + virtual_channel_id, + client_id, + client_ip, + user_agent, + user, + channel_display_name=channel_display_name, + timestamp_utc=timestamp_utc, + primary_url=last_url, + channel_logo_id=channel_logo_id, + m3u_profile_id=m3u_profile_id, + ) + + peek_data = getattr(upstream, "_peek_data", None) + chunks_iter = upstream.iter_content(chunk_size=chunk_size) + if peek_data: + chunks_iter = itertools.chain([peek_data], chunks_iter) + + session_closed = {"done": False} + + def _finish_session(*, close_upstream=False): + if session_closed["done"]: + return + session_closed["done"] = True + if close_upstream: + try: + upstream.close() + except Exception: + pass + _unregister_stats_client(redis_client, virtual_channel_id, client_id) + release_cb() + + def stream_generator(): + last_heartbeat = time.time() + bytes_since_heartbeat = 0 + total_yielded = 0 + loop_start = time.time() + stop_key = RedisKeys.client_stop(virtual_channel_id, client_id) + stream_started_logged = False + try: + for data in chunks_iter: + if not data: + continue + if debug and not stream_started_logged: + stream_started_logged = True + logger.debug( + "Timeshift stream started: client=%s vid=%s range=%s status=%d", + client_id, virtual_channel_id, range_header or "(none)", status, + ) + yield data + bytes_since_heartbeat += len(data) + total_yielded += len(data) + + now = time.time() + if redis_client and redis_client.exists(stop_key): + logger.info("Timeshift client %s received stop signal", client_id) + redis_client.delete(stop_key) + break + # Refresh stats every 5 seconds. + if now - last_heartbeat >= 5: + if debug and total_yielded > 0: + elapsed = now - loop_start + mbps = (total_yielded * 8) / elapsed / 1_000_000 if elapsed > 0 else 0 + logger.debug( + "Timeshift streaming: client=%s range=%s total=%d bytes " + "in %.1fs (%.2f Mbps avg)", + client_id, range_header or "(none)", + total_yielded, elapsed, mbps, + ) + _heartbeat_stats_client( + redis_client, virtual_channel_id, client_id, + bytes_delta=bytes_since_heartbeat, + ) + last_heartbeat = now + bytes_since_heartbeat = 0 + except GeneratorExit: + pass + except Exception: + logger.exception("Timeshift stream loop error") + finally: + elapsed = time.time() - loop_start + if bytes_since_heartbeat > 0: + _heartbeat_stats_client( + redis_client, virtual_channel_id, client_id, + bytes_delta=bytes_since_heartbeat, + ) + if debug and total_yielded > 0: + mbps = (total_yielded * 8) / elapsed / 1_000_000 if elapsed > 0 else 0 + logger.debug( + "Timeshift disconnect: vid=%s client=%s yielded=%d bytes in %.1fs (%.2f Mbps avg)", + virtual_channel_id, client_id, total_yielded, elapsed, mbps, + ) + _finish_session(close_upstream=True) + + stream_iter = _SlotReleasingStream(stream_generator(), _finish_session) + response = StreamingHttpResponse( + stream_iter, + content_type=content_type, + status=status, + ) + response["X-Accel-Buffering"] = "no" # avoid nginx throttling the stream + if content_range: + response["Content-Range"] = content_range + response["Accept-Ranges"] = "bytes" + return response + + +def _redact_url(url): + """Truncate *url* to ``scheme://host/...`` for safe logging (drops credentials).""" + if not url or "://" not in url: + return url + scheme, rest = url.split("://", 1) + if "@" in rest: + rest = rest.split("@", 1)[1] + host = rest.split("/", 1)[0] + return f"{scheme}://{host}/..." diff --git a/core/models.py b/core/models.py index 60be2f0e..58fd37da 100644 --- a/core/models.py +++ b/core/models.py @@ -280,8 +280,18 @@ class CoreSettings(models.Model): "epg_match_ignore_prefixes": [], "epg_match_ignore_suffixes": [], "epg_match_ignore_custom": [], + # XC catch-up: forced XMLTV lookback (0 = auto-detect, capped at 30). + "xmltv_prev_days_override": 0, }) + @classmethod + def get_xmltv_prev_days_override(cls): + """Global XC XMLTV prev_days default (0 = auto-detect from provider archives).""" + try: + return int(cls.get_epg_settings().get("xmltv_prev_days_override", 0) or 0) + except (TypeError, ValueError): + return 0 + @classmethod def _safe_string_list(cls, value): """Return a list of strings, filtering out non-list or non-string values.""" diff --git a/core/utils.py b/core/utils.py index 44504b6a..16d9ed85 100644 --- a/core/utils.py +++ b/core/utils.py @@ -565,13 +565,15 @@ def trim_c_allocator_heap(): return False -def cleanup_memory(log_usage=False, force_collection=True): +def cleanup_memory(log_usage=False, force_collection=True, trim_heap=False): """ Comprehensive memory cleanup function to reduce memory footprint Args: log_usage: Whether to log memory usage before and after cleanup force_collection: Whether to force garbage collection + trim_heap: Return freed C heap pages to the OS. Only use after DB + connections are closed (e.g. Celery task_postrun). """ logger.trace("Starting memory cleanup django memory cleanup") # Skip logging if log level is not set to debug or more verbose (like trace) @@ -606,6 +608,8 @@ def cleanup_memory(log_usage=False, force_collection=True): logger.debug(f"Memory after cleanup: {after_mem:.2f} MB (change: {after_mem-before_mem:.2f} MB)") except (ImportError, Exception): pass + if trim_heap: + trim_c_allocator_heap() logger.trace("Memory cleanup complete for django") @@ -618,8 +622,7 @@ def spawn_memory_trim(close_connections=False): so the pooled DB connection is released first. """ def _run(): - cleanup_memory(force_collection=True) - trim_c_allocator_heap() + cleanup_memory(force_collection=True, trim_heap=True) if close_connections: from django.db import close_old_connections diff --git a/dispatcharr/celery.py b/dispatcharr/celery.py index 18be81eb..16a393dc 100644 --- a/dispatcharr/celery.py +++ b/dispatcharr/celery.py @@ -99,6 +99,7 @@ def cleanup_task_memory(**kwargs): memory_intensive_tasks = [ 'apps.m3u.tasks.refresh_single_m3u_account', 'apps.m3u.tasks.refresh_m3u_accounts', + 'apps.m3u.tasks.refresh_m3u_groups', 'apps.m3u.tasks.process_m3u_batch', 'apps.m3u.tasks.process_xc_category', 'apps.m3u.tasks.sync_auto_channels', @@ -121,7 +122,7 @@ def cleanup_task_memory(**kwargs): from core.utils import cleanup_memory # Use the comprehensive cleanup function - cleanup_memory(log_usage=True, force_collection=True) + cleanup_memory(log_usage=True, force_collection=True, trim_heap=True) # Log memory usage if psutil is installed try: diff --git a/dispatcharr/settings.py b/dispatcharr/settings.py index a22e5f9c..a1e2b798 100644 --- a/dispatcharr/settings.py +++ b/dispatcharr/settings.py @@ -99,6 +99,7 @@ INSTALLED_APPS = [ "django_filters", "django_celery_beat", "apps.plugins", + "apps.timeshift.apps.TimeshiftConfig", ] # EPG Processing optimization settings diff --git a/dispatcharr/urls.py b/dispatcharr/urls.py index 9123f3da..cdedb0c4 100644 --- a/dispatcharr/urls.py +++ b/dispatcharr/urls.py @@ -7,6 +7,7 @@ from .routing import websocket_urlpatterns from apps.output.views import xc_player_api, xc_panel_api, xc_get, xc_xmltv from apps.proxy.live_proxy.views import stream_xc from apps.proxy.vod_proxy.views import stream_xc_movie, stream_xc_episode +from apps.timeshift.views import timeshift_proxy urlpatterns = [ # API Routes @@ -41,6 +42,11 @@ urlpatterns = [ stream_xc, name="xc_stream_endpoint", ), + path( + "timeshift/////", + timeshift_proxy, + name="timeshift_proxy", + ), # XC VOD endpoints path( "movie///.", diff --git a/frontend/src/components/cards/StreamConnectionCard.jsx b/frontend/src/components/cards/StreamConnectionCard.jsx index a1748143..ae85b410 100644 --- a/frontend/src/components/cards/StreamConnectionCard.jsx +++ b/frontend/src/components/cards/StreamConnectionCard.jsx @@ -22,6 +22,7 @@ import { Gauge, HardDriveDownload, HardDriveUpload, + Rewind, SquareX, Timer, Users, @@ -486,7 +487,10 @@ const StreamConnectionCard = ({ }, [channel.name, channel.stream_id]); const channelName = - channel.name || previewedStream?.name || 'Unnamed Channel'; + channel.name || + channel.channel_name || + previewedStream?.name || + 'Unnamed Channel'; const uptime = channel.uptime || 0; const bitrates = channel.bitrates || []; const totalBytes = channel.total_bytes || 0; @@ -661,6 +665,18 @@ const StreamConnectionCard = ({ {/* Add stream information badges */} + {channel.is_timeshift && ( + + } + > + TIMESHIFT + + + )} {channel.resolution && ( diff --git a/frontend/src/components/forms/settings/EpgSettingsForm.jsx b/frontend/src/components/forms/settings/EpgSettingsForm.jsx new file mode 100644 index 00000000..5668dbdd --- /dev/null +++ b/frontend/src/components/forms/settings/EpgSettingsForm.jsx @@ -0,0 +1,81 @@ +import useSettingsStore from '../../../store/settings.jsx'; +import React, { useEffect, useState } from 'react'; +import { useForm } from '@mantine/form'; +import { + Alert, + Button, + Flex, + NumberInput, + Stack, + Text, +} from '@mantine/core'; +import { EPG_SETTINGS_OPTIONS } from '../../../constants.js'; +import { + getChangedSettings, + parseSettings, + saveChangedSettings, +} from '../../../utils/pages/SettingsUtils.js'; +import { getEpgSettingsFormInitialValues } from '../../../utils/forms/settings/EpgSettingsFormUtils.js'; + +const EpgSettingsForm = React.memo(({ active }) => { + const settings = useSettingsStore((s) => s.settings); + const [saved, setSaved] = useState(false); + + const form = useForm({ + mode: 'controlled', + initialValues: getEpgSettingsFormInitialValues(), + }); + + useEffect(() => { + if (!active) setSaved(false); + }, [active]); + + useEffect(() => { + if (settings) { + const parsed = parseSettings(settings); + form.setFieldValue( + 'xmltv_prev_days_override', + parsed.xmltv_prev_days_override ?? 0, + ); + } + }, [settings]); + + const onSubmit = async () => { + setSaved(false); + const changedSettings = getChangedSettings(form.getValues(), settings); + try { + await saveChangedSettings(settings, changedSettings); + setSaved(true); + } catch (error) { + console.error('Error saving EPG settings:', error); + } + }; + + const prevDaysConfig = EPG_SETTINGS_OPTIONS.xmltv_prev_days_override; + + return ( +
+ + {saved && ( + + )} + + + Per-user defaults and URL parameters still override this global value. + EPG channel matching options are configured from the Channels page. + + + + + +
+ ); +}); + +export default EpgSettingsForm; diff --git a/frontend/src/constants.js b/frontend/src/constants.js index b130553e..b4503dc6 100644 --- a/frontend/src/constants.js +++ b/frontend/src/constants.js @@ -63,6 +63,14 @@ export const PROXY_SETTINGS_OPTIONS = { }, }; +export const EPG_SETTINGS_OPTIONS = { + xmltv_prev_days_override: { + label: 'XMLTV prev_days Override (catch-up)', + description: + 'Days of past programmes in the XC EPG output. 0 = auto-detect from the providers’ tv_archive_duration (capped at 30).', + }, +}; + export const USER_LIMITS_OPTIONS = { terminate_on_limit_exceeded: { label: 'Terminate on Limit Exceeded', diff --git a/frontend/src/pages/Settings.jsx b/frontend/src/pages/Settings.jsx index 29d46dd8..1c7964cc 100644 --- a/frontend/src/pages/Settings.jsx +++ b/frontend/src/pages/Settings.jsx @@ -45,6 +45,9 @@ const DvrSettingsForm = React.lazy( const SystemSettingsForm = React.lazy( () => import('../components/forms/settings/SystemSettingsForm.jsx') ); +const EpgSettingsForm = React.lazy( + () => import('../components/forms/settings/EpgSettingsForm.jsx') +); const NavOrderForm = React.lazy( () => import('../components/forms/settings/NavOrderForm.jsx') ); @@ -122,6 +125,19 @@ const SettingsPage = () => { + + EPG + + + }> + + + + + + System Settings diff --git a/frontend/src/utils/forms/settings/EpgSettingsFormUtils.js b/frontend/src/utils/forms/settings/EpgSettingsFormUtils.js new file mode 100644 index 00000000..d2138b6a --- /dev/null +++ b/frontend/src/utils/forms/settings/EpgSettingsFormUtils.js @@ -0,0 +1,3 @@ +export const getEpgSettingsFormInitialValues = () => ({ + xmltv_prev_days_override: 0, +}); diff --git a/frontend/src/utils/pages/SettingsUtils.js b/frontend/src/utils/pages/SettingsUtils.js index b097accd..bd46ac52 100644 --- a/frontend/src/utils/pages/SettingsUtils.js +++ b/frontend/src/utils/pages/SettingsUtils.js @@ -39,6 +39,7 @@ export const saveChangedSettings = async (settings, changedSettings) => { 'epg_match_ignore_prefixes', 'epg_match_ignore_suffixes', 'epg_match_ignore_custom', + 'xmltv_prev_days_override', ]; const dvrFields = [ 'tv_template', @@ -125,6 +126,7 @@ export const saveChangedSettings = async (settings, changedSettings) => { 'retention_count', 'schedule_day_of_week', 'max_system_events', + 'xmltv_prev_days_override', ]; if (numericFields.includes(formKey) && value != null) { value = typeof value === 'number' ? value : parseInt(value, 10); @@ -222,6 +224,20 @@ export const getChangedSettings = (values, settings) => { continue; } + if (settingKey === 'xmltv_prev_days_override') { + const baseline = Number( + settings['epg_settings']?.value?.xmltv_prev_days_override ?? 0, + ); + const nextVal = + typeof actualValue === 'number' + ? actualValue + : parseInt(actualValue, 10) || 0; + if (nextVal !== baseline) { + changedSettings[settingKey] = nextVal; + } + continue; + } + // Convert array values (like m3u_hash_key) to comma-separated strings for comparison if (Array.isArray(actualValue)) { actualValue = actualValue.join(','); @@ -296,6 +312,12 @@ export const parseSettings = (settings) => { epgSettings && Array.isArray(epgSettings.epg_match_ignore_custom) ? epgSettings.epg_match_ignore_custom : []; + parsed.xmltv_prev_days_override = + epgSettings && epgSettings.xmltv_prev_days_override != null + ? typeof epgSettings.xmltv_prev_days_override === 'number' + ? epgSettings.xmltv_prev_days_override + : parseInt(epgSettings.xmltv_prev_days_override, 10) || 0 + : 0; // DVR settings - direct mapping with underscore keys const dvrSettings = settings['dvr_settings']?.value;