diff --git a/CHANGELOG.md b/CHANGELOG.md index 0517493d..0ad1ef3b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Performance + +- **Lighter EPG export: fewer escape calls and a slimmer channel prefetch.** The primary channel id is escaped once per `epg_id` group instead of once per programme (saves ~750k `html.escape` calls on a large guide). The per-channel `streams` prefetch now loads only `id`/`name` (the only fields the export reads) instead of full `Stream` rows, reducing worker RSS during the channel phase. +- **XMLTV EPG export streams without holding the full guide in the worker.** `generate_epg()` streams incrementally; on a cache miss each chunk is pushed to a Redis list as it is yielded (no `''.join()` in the worker). Repeat requests within 300s stream chunks back one at a time from Redis. Post-export `malloc_trim` runs after cold builds. Real programmes use fast `(epg_id, id)` keyset pagination with a per-source `start_time` sort before emit. +- **EPG export no longer fetches the multi-MB `programme_index` per channel.** The channel query `select_related`s `epg_data__epg_source`, which pulled and JSON-parsed each source's byte-offset `programme_index` blob once per channel (~13s of the request on a ~2000-channel guide). The index now lives in its own `EPGSourceIndex` table, so the JOIN never pulls it. +- **`ProgramData` composite index `(epg_id, id)`.** The EPG export scans hundreds of thousands of programmes with keyset pagination on `(epg_id, id)`; without a matching index PostgreSQL re-sorted every chunk. A composite index (created `CONCURRENTLY` on PostgreSQL so it does not lock the table) lets each chunk use an ordered index range scan. +- **EPG grid endpoint releases its payload memory back to the OS.** `/api/epg/grid/` drops the redundant full-list copy when appending dummy programmes and runs `malloc_trim` once the response is sent, so worker RSS no longer ratchets up ~20MB per request. + ### Changed +- **`programme_index` moved off `EPGSource` into a dedicated `EPGSourceIndex` table.** The multi-MB byte-offset index was repeatedly pulled into web and Celery workers by ordinary `EPGSource` queries and `select_related` JOINs (which ignore manager-level `defer()`). It now lives in a one-to-one `EPGSourceIndex` row, read only when explicitly accessed through the `EPGSource.programme_index` property, so no list, detail, or JOIN query can load it by accident. Migration `0026` copies existing indices across. No API or index-build behavior change. + +- **EPG generation extracted into `apps/output/epg.py`.** All XMLTV output logic (`generate_epg`, `generate_dummy_programs`, `generate_custom_dummy_programs`, `generate_dummy_epg`, and supporting helpers) moved from `apps/output/views.py` into a dedicated module. `views.py` retains the thin HTTP endpoint wrappers and auth checks; `epg.py` handles all content generation. No behavior change. + - **Channel list with nested streams loads faster.** `GET /api/channels/channels/?include_streams=true` (Channels UI and single-channel fetch) now builds nested stream payloads from the prefetched `channelstream_set` instead of issuing one extra streams M2M query per channel. ### Performance @@ -20,7 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **M3U refresh recovers DB connections and account status after failures.** Celery workers now call `close_old_connections()` before and after every task; `refresh_single_m3u_account` also resets its DB connection at task start and retries account loads once after a connection reset (avoids opaque `list index out of range` errors from poisoned worker connections). Accounts leave `fetching`/`parsing` for a terminal `error` or `success` state when refresh aborts. Error-path status updates use `QuerySet.update()` on a clean connection instead of `save()` on a connection that may already be INTRANS after `refresh_m3u_groups` or batch ORM failures. Failed group downloads now set `error` instead of returning silently. Refresh start replaces stale `last_message` text. `refresh_account_profiles` releases DB connections after each profile failure. (Fixes #1338) - **EPG refresh recovers DB connections and source status after failures.** `refresh_epg_data` now mirrors the M3U hardening: connection reset at task start/end, retried source load after a poisoned-worker connection reset, `QuerySet.update()` for error status on a clean connection, and a `finally` guard that moves sources stuck in `fetching`/`parsing` to `error`. Programme index builds are skipped when programme parsing fails. `parse_channels_only` now persists the error status when a missing file has no URL to refetch. - **M3U `custom_properties` JSON normalization.** Legacy rows and some API writes could store a JSON-encoded string instead of an object in `custom_properties`, causing refresh, profile sync, and auto-sync to fail with `'str' object has no attribute 'get'`. M3U account, profile, and group-relation models normalize on `save()`; group settings bulk writes and read/merge paths use shared helpers in `core.utils` without re-parsing dict values. -- **`POST /api/epg/import/` no longer loads `programme_index`.** The dummy-source guard uses a narrow existence query instead of `objects.get()`, so import triggers avoid pulling the multi-MB byte-offset index into the uWSGI worker. Request/response shape is unchanged for the UI, plugins, and external importers. +- **`POST /api/epg/import/` no longer loads the full EPG source row.** The dummy-source guard uses a narrow existence query instead of `objects.get()`, keeping the import trigger lightweight. Request/response shape is unchanged for the UI, plugins, and external importers. - **XC live playback URL building now normalizes account server URLs.** On-demand live URLs (introduced for Server Group profile rotation in 0.27.0) rebuild `/live/{user}/{pass}/{stream_id}.ts` from current credentials instead of reusing the synced `stream.url`. That path now runs `normalize_server_url()` so pasted API URLs (e.g. `/player_api.php` with query params) are stripped while sub-paths like `/server1` are preserved. `get_transformed_credentials()` normalizes the base URL at the source; VOD movie and episode relations use the same helper instead of constructing a throwaway `XCClient` (which also avoided per-request HTTP session setup). (Fixes #1363) - **Live proxy now releases geventpool DB connections on more paths.** `stream_ts()` calls `close_old_connections()` before `StreamingHttpResponse` so clients waiting on channel init do not keep a pool slot while the generator polls Redis. `generate_stream_url()`, `get_stream_info_for_switch()`, `get_alternate_streams()`, and `get_connections_left()` release in `finally` blocks after ORM work on stream-manager failover paths that run outside Django's request cycle. TS client disconnect cleanup matches fMP4, and `ChannelService` metadata helpers release after their ORM lookups. diff --git a/apps/epg/api_views.py b/apps/epg/api_views.py index fbc419c5..cdb39a47 100644 --- a/apps/epg/api_views.py +++ b/apps/epg/api_views.py @@ -64,8 +64,6 @@ class EPGSourceViewSet(viewsets.ModelViewSet): from apps.channels.models import Channel return EPGSource.objects.select_related( "refresh_task__crontab", "refresh_task__interval" - ).defer( - 'programme_index' ).annotate( has_channels=Exists( Channel.objects.filter(epg_data__epg_source_id=OuterRef('pk')) @@ -986,7 +984,9 @@ class EPGGridAPIView(APIView): channel_name=name_to_parse, num_days=1, program_length_hours=4, - epg_source=epg_source + epg_source=epg_source, + export_lookback=one_hour_ago, + export_cutoff=twenty_four_hours_later, ) # Custom dummy should always return data (either from patterns or fallback) @@ -1082,13 +1082,19 @@ class EPGGridAPIView(APIView): f"Error creating standard dummy programs for channel {channel.name} (ID: {channel.id}): {str(e)}" ) - # Combine regular and dummy programs - all_programs = list(serialized_programs) + dummy_programs + # Combine regular and dummy programs in place to avoid copying the large list + serialized_programs.extend(dummy_programs) logger.debug( - f"EPGGridAPIView: Returning {len(all_programs)} total programs (including {len(dummy_programs)} dummy programs)." + f"EPGGridAPIView: Returning {len(serialized_programs)} total programs (including {len(dummy_programs)} dummy programs)." ) - return Response({"data": all_programs}, status=status.HTTP_200_OK) + # The grid materializes tens of thousands of program dicts plus the + # rendered JSON; trim once the response is sent so worker RSS does not + # ratchet up per request. + from core.utils import spawn_memory_trim + response = Response({"data": serialized_programs}, status=status.HTTP_200_OK) + response._resource_closers.append(spawn_memory_trim) + return response # ───────────────────────────── @@ -1119,7 +1125,7 @@ class EPGImportAPIView(APIView): epg_id = request.data.get("id", None) force = bool(request.data.get("force", False)) - # Reject dummy sources without loading programme_index (multi-MB JSON). + # Reject dummy sources with a narrow existence query, no full row load. if epg_id is not None: from .models import EPGSource @@ -1236,10 +1242,9 @@ class CurrentProgramsAPIView(APIView): # Limit to 50 IDs per request epg_data_ids = epg_data_ids[:50] - # Defer the multi-MB programme_index the JOIN would pull once per row. The lookup reads it via a targeted refresh_from_db - epg_data_entries = EPGData.objects.select_related('epg_source').defer( - 'epg_source__programme_index' - ).filter(id__in=epg_data_ids) + epg_data_entries = EPGData.objects.select_related('epg_source').filter( + id__in=epg_data_ids + ) # Batch-fetch current programs for all requested EPG entries in one query db_programs = ProgramData.objects.filter( diff --git a/apps/epg/migrations/0025_programdata_epg_id_index.py b/apps/epg/migrations/0025_programdata_epg_id_index.py new file mode 100644 index 00000000..1a350b20 --- /dev/null +++ b/apps/epg/migrations/0025_programdata_epg_id_index.py @@ -0,0 +1,40 @@ +from django.contrib.postgres.operations import AddIndexConcurrently +from django.db import migrations, models + + +class AddIndexConcurrentlyIfPostgres(AddIndexConcurrently): + """Create the index CONCURRENTLY on PostgreSQL (no table lock on large + tables), falling back to a normal blocking AddIndex on other backends + such as the sqlite dev/test fallback.""" + + def database_forwards(self, app_label, schema_editor, from_state, to_state): + if schema_editor.connection.vendor == 'postgresql': + super().database_forwards(app_label, schema_editor, from_state, to_state) + else: + migrations.AddIndex.database_forwards( + self, app_label, schema_editor, from_state, to_state + ) + + def database_backwards(self, app_label, schema_editor, from_state, to_state): + if schema_editor.connection.vendor == 'postgresql': + super().database_backwards(app_label, schema_editor, from_state, to_state) + else: + migrations.AddIndex.database_backwards( + self, app_label, schema_editor, from_state, to_state + ) + + +class Migration(migrations.Migration): + # CREATE INDEX CONCURRENTLY cannot run inside a transaction. + atomic = False + + dependencies = [ + ('epg', '0024_remove_epgsource_api_key_epgsource_password_and_more'), + ] + + operations = [ + AddIndexConcurrentlyIfPostgres( + model_name='programdata', + index=models.Index(fields=['epg', 'id'], name='epg_prog_epg_id_idx'), + ), + ] diff --git a/apps/epg/migrations/0026_epgsourceindex.py b/apps/epg/migrations/0026_epgsourceindex.py new file mode 100644 index 00000000..75d9d4eb --- /dev/null +++ b/apps/epg/migrations/0026_epgsourceindex.py @@ -0,0 +1,52 @@ +import django.db.models.deletion +from django.db import migrations, models + + +def copy_index_forward(apps, schema_editor): + EPGSource = apps.get_model('epg', 'EPGSource') + EPGSourceIndex = apps.get_model('epg', 'EPGSourceIndex') + rows = list( + EPGSource.objects.exclude(programme_index__isnull=True).values_list( + 'id', 'programme_index' + ) + ) + for source_id, data in rows: + EPGSourceIndex.objects.update_or_create( + source_id=source_id, defaults={'data': data} + ) + + +def copy_index_backward(apps, schema_editor): + EPGSource = apps.get_model('epg', 'EPGSource') + EPGSourceIndex = apps.get_model('epg', 'EPGSourceIndex') + for source_id, data in EPGSourceIndex.objects.values_list('source_id', 'data'): + EPGSource.objects.filter(id=source_id).update(programme_index=data) + + +class Migration(migrations.Migration): + + dependencies = [ + ('epg', '0025_programdata_epg_id_index'), + ] + + operations = [ + migrations.CreateModel( + name='EPGSourceIndex', + fields=[ + ('source', models.OneToOneField( + on_delete=django.db.models.deletion.CASCADE, + primary_key=True, + related_name='index_record', + serialize=False, + to='epg.epgsource', + )), + ('data', models.JSONField(blank=True, default=None, null=True)), + ('updated_at', models.DateTimeField(auto_now=True)), + ], + ), + migrations.RunPython(copy_index_forward, copy_index_backward), + migrations.RemoveField( + model_name='epgsource', + name='programme_index', + ), + ] diff --git a/apps/epg/models.py b/apps/epg/models.py index 025945db..b85453af 100644 --- a/apps/epg/models.py +++ b/apps/epg/models.py @@ -62,12 +62,6 @@ class EPGSource(models.Model): blank=True, help_text="Last status message, including success results or error information" ) - programme_index = models.JSONField( - null=True, - blank=True, - default=None, - help_text="Byte-offset index mapping tvg_id to file positions, built after each EPG refresh" - ) created_at = models.DateTimeField( auto_now_add=True, help_text="Time when this source was created" @@ -124,6 +118,37 @@ class EPGSource(models.Model): kwargs['update_fields'].remove('updated_at') super().save(*args, **kwargs) + @property + def programme_index(self): + """Byte-offset index for this source, read on demand from the separate + EPGSourceIndex table so the multi-MB blob is never pulled into EPGSource + queries or select_related JOINs. Returns the stored dict or None.""" + return ( + EPGSourceIndex.objects.filter(source_id=self.pk) + .values_list('data', flat=True) + .first() + ) + + +class EPGSourceIndex(models.Model): + """Byte-offset programme index for an EPGSource, stored in its own table. + + Kept out of EPGSource so the multi-MB JSON blob is only loaded when read + explicitly, never when querying or joining EPGSource rows. + """ + source = models.OneToOneField( + EPGSource, + on_delete=models.CASCADE, + related_name='index_record', + primary_key=True, + ) + data = models.JSONField(null=True, blank=True, default=None) + updated_at = models.DateTimeField(auto_now=True) + + def __str__(self): + return f"Programme index for source {self.source_id}" + + class EPGData(models.Model): tvg_id = models.CharField(max_length=255, null=True, blank=True, db_index=True) name = models.CharField(max_length=512) @@ -153,6 +178,11 @@ class ProgramData(models.Model): program_id = models.CharField(max_length=64, null=True, blank=True, help_text='Schedules Direct programID (e.g. EP123456789). Null for XMLTV sources.') custom_properties = models.JSONField(default=dict, blank=True, null=True) + class Meta: + indexes = [ + models.Index(fields=['epg', 'id'], name='epg_prog_epg_id_idx'), + ] + def __str__(self): return f"{self.title} ({self.start_time} - {self.end_time})" diff --git a/apps/epg/tasks.py b/apps/epg/tasks.py index 9661c5b9..83aaa275 100644 --- a/apps/epg/tasks.py +++ b/apps/epg/tasks.py @@ -26,7 +26,7 @@ from core.models import UserAgent, CoreSettings from asgiref.sync import async_to_sync from channels.layers import get_channel_layer -from .models import EPGSource, EPGData, ProgramData, SDScheduleMD5, SDProgramMD5 +from .models import EPGSource, EPGSourceIndex, EPGData, ProgramData, SDScheduleMD5, SDProgramMD5 from core.utils import ( acquire_task_lock, is_task_lock_held, @@ -653,7 +653,9 @@ def _refresh_epg_data_impl(source_id, force=False): if source.source_type == 'xmltv': # Invalidate the byte-offset index before downloading the new file # so stale offsets are never used during the refresh window. - EPGSource.objects.filter(id=source.id).update(programme_index=None) + EPGSourceIndex.objects.update_or_create( + source_id=source.id, defaults={'data': None} + ) if not fetch_xmltv(source): logger.error(f"Failed to fetch XMLTV for source {source.name}") return @@ -4484,7 +4486,7 @@ def _programme_to_dict(elem, start_time, end_time): def build_programme_index(source_id): """ Scan the XML file with raw binary I/O to build a {tvg_id: [byte_offset, ...]} map. - Persists the result to EPGSource.programme_index. Most XMLTV files group programmes + Persists the result to the EPGSourceIndex table. Most XMLTV files group programmes by channel, but some split a channel across multiple non-contiguous blocks, so we record block starts up to _OFFSET_CAP and mark only channels that exceed the cap as interleaved. @@ -4573,7 +4575,9 @@ def build_programme_index(source_id): 'channels': index, 'interleaved_channels': sorted(interleaved_channels), } - EPGSource.objects.filter(id=source_id).update(programme_index=result) + EPGSourceIndex.objects.update_or_create( + source_id=source_id, defaults={'data': result} + ) @shared_task @@ -4620,9 +4624,8 @@ def find_current_program_for_tvg_id(epg_or_id): return None now = timezone.now() - # Force a fresh read of the DB-backed index to avoid using stale related-object - # state when an EPG refresh invalidates/rebuilds the index concurrently. - source.refresh_from_db(fields=['programme_index']) + # The property reads the EPGSourceIndex table fresh on each access, so a + # concurrent refresh invalidating/rebuilding the index can't serve stale state. index = source.programme_index if index is not None: diff --git a/apps/epg/tests/test_epg_import_api.py b/apps/epg/tests/test_epg_import_api.py index c58f2a45..cc96c214 100644 --- a/apps/epg/tests/test_epg_import_api.py +++ b/apps/epg/tests/test_epg_import_api.py @@ -7,7 +7,7 @@ from django.test import TestCase from rest_framework import status from rest_framework.test import APIClient -from apps.epg.models import EPGSource +from apps.epg.models import EPGSource, EPGSourceIndex User = get_user_model() @@ -47,7 +47,10 @@ class EPGImportAPITests(TestCase): name="Large Index XMLTV", source_type="xmltv", url="http://example.com/epg.xml", - programme_index={ + ) + EPGSourceIndex.objects.create( + source=source, + data={ "channels": {f"ch.{i}": {"offsets": [0, 100]} for i in range(200)}, "interleaved_channels": [], }, diff --git a/apps/epg/tests/test_programme_index.py b/apps/epg/tests/test_programme_index.py index 5ece0ae1..829afc87 100644 --- a/apps/epg/tests/test_programme_index.py +++ b/apps/epg/tests/test_programme_index.py @@ -519,7 +519,6 @@ class FindCurrentProgramTests(TestCase): name="No Index", source_type="xmltv", file_path=FIXTURE_XML, - programme_index=None, ) epg = EPGData.objects.create( tvg_id="channel.current", diff --git a/apps/output/epg.py b/apps/output/epg.py new file mode 100644 index 00000000..626b21aa --- /dev/null +++ b/apps/output/epg.py @@ -0,0 +1,1727 @@ +"""XMLTV (EPG) output generation. + +Consolidates the EPG export logic that backs the `/epg` endpoint and the XC +XMLTV endpoint: real programme streaming, dummy/custom dummy program +generation, and the streaming XMLTV builder. HTTP endpoints live in views.py +and call into this module; Redis chunk caching lives in streaming_chunk_cache.py. +""" + +import html +import logging +from datetime import datetime, timedelta + +import regex + +from django.db.models import Prefetch +from django.http import Http404 +from django.urls import reverse +from django.utils import timezone as django_timezone + +from apps.channels.models import Channel, ChannelProfile, Stream +from apps.channels.utils import format_channel_number +from apps.epg.models import ProgramData +from apps.output.streaming_chunk_cache import stream_cached_response +from core.utils import build_absolute_uri_with_port, log_system_event + +logger = logging.getLogger(__name__) + +_EPG_CHANNEL_XML_BATCH_SIZE = 200 +_EPG_PROGRAM_YIELD_BATCH_SIZE = 1000 +_EPG_PROGRAM_DB_CHUNK_SIZE = 20000 + + +def _programme_overlaps_export_window(start_time, end_time, lookback_cutoff, cutoff_date): + if end_time < lookback_cutoff: + return False + if cutoff_date is not None and start_time >= cutoff_date: + return False + return True + + +def _ceil_to_half_hour(dt): + """Round a datetime up to the next :00 or :30 boundary.""" + original = dt.replace(microsecond=0) + aligned = dt.replace(second=0, microsecond=0) + remainder = aligned.minute % 30 + if remainder != 0: + aligned += timedelta(minutes=30 - remainder) + if aligned < original: + aligned += timedelta(minutes=30) + return aligned + + +def _epg_export_teardown(): + from core.utils import spawn_memory_trim + + spawn_memory_trim(close_connections=True) + + +def _ordered_channel_streams(channel): + """Return a channel's streams ordered by channelstream join order.""" + prefetched = getattr(channel, '_prefetched_objects_cache', {}).get('streams') + if prefetched is not None: + return list(prefetched) + return list(channel.streams.all().order_by('channelstream__order')) + + +def _pattern_match_name_from_custom_props(channel, effective_name, custom_props): + """Name used for custom dummy EPG regex matching (channel or stream title). + + Returns (name, stream_lookup_failed). stream_lookup_failed is True only when + name_source is 'stream' but the configured index is missing or out of range. + """ + if custom_props.get('name_source') != 'stream': + return effective_name, False + stream_index = custom_props.get('stream_index', 1) - 1 + streams = _ordered_channel_streams(channel) + if 0 <= stream_index < len(streams): + return streams[stream_index].name, False + return effective_name, True + + +def generate_fallback_programs(channel_id, channel_name, now, num_days, program_length_hours, fallback_title, fallback_description): + """ + Generate dummy programs using custom fallback templates when patterns don't match. + + Args: + channel_id: Channel ID for the programs + channel_name: Channel name to use as fallback in templates + now: Current datetime (in UTC) + num_days: Number of days to generate programs for + program_length_hours: Length of each program in hours + fallback_title: Custom fallback title template (empty string if not provided) + fallback_description: Custom fallback description template (empty string if not provided) + + Returns: + List of program dictionaries + """ + programs = [] + + # Use custom fallback title or channel name as default + title = fallback_title if fallback_title else channel_name + + # Use custom fallback description or a simple default message + if fallback_description: + description = fallback_description + else: + description = f"EPG information is currently unavailable for {channel_name}" + + # Create programs for each day + for day in range(num_days): + day_start = now + timedelta(days=day) + + # Create programs with specified length throughout the day + for hour_offset in range(0, 24, program_length_hours): + # Calculate program start and end times + start_time = day_start + timedelta(hours=hour_offset) + end_time = start_time + timedelta(hours=program_length_hours) + + programs.append({ + "channel_id": channel_id, + "start_time": start_time, + "end_time": end_time, + "title": title, + "description": description, + }) + + return programs + + +def generate_dummy_programs( + channel_id, + channel_name, + num_days=1, + program_length_hours=4, + epg_source=None, + export_lookback=None, + export_cutoff=None, +): + """ + Generate dummy EPG programs for channels. + + If epg_source is provided and it's a custom dummy EPG with patterns, + use those patterns to generate programs from the channel title. + Otherwise, generate default dummy programs. + + Args: + channel_id: Channel ID for the programs + channel_name: Channel title/name + num_days: Number of days to generate programs for + program_length_hours: Length of each program in hours + epg_source: Optional EPGSource for custom dummy EPG with patterns + + Returns: + List of program dictionaries + """ + # Get current time rounded to hour + now = django_timezone.now() + now = now.replace(minute=0, second=0, microsecond=0) + + # Check if this is a custom dummy EPG with regex patterns + if epg_source and epg_source.source_type == 'dummy' and epg_source.custom_properties: + custom_programs = generate_custom_dummy_programs( + channel_id, channel_name, now, num_days, + epg_source.custom_properties, + export_lookback=export_lookback, + export_cutoff=export_cutoff, + ) + if custom_programs is not None: + return custom_programs + + logger.info(f"Custom pattern didn't match for '{channel_name}', checking for custom fallback templates") + + custom_props = epg_source.custom_properties + fallback_title = custom_props.get('fallback_title_template', '').strip() + fallback_description = custom_props.get('fallback_description_template', '').strip() + + if fallback_title or fallback_description: + logger.info(f"Using custom fallback templates for '{channel_name}'") + return generate_fallback_programs( + channel_id, channel_name, now, num_days, + program_length_hours, fallback_title, fallback_description + ) + logger.info(f"No custom fallback templates found, using default dummy EPG") + + # Default humorous program descriptions based on time of day + time_descriptions = { + (0, 4): [ + f"Late Night with {channel_name} - Where insomniacs unite!", + f"The 'Why Am I Still Awake?' Show on {channel_name}", + f"Counting Sheep - A {channel_name} production for the sleepless", + ], + (4, 8): [ + f"Dawn Patrol - Rise and shine with {channel_name}!", + f"Early Bird Special - Coffee not included", + f"Morning Zombies - Before coffee viewing on {channel_name}", + ], + (8, 12): [ + f"Mid-Morning Meetings - Pretend you're paying attention while watching {channel_name}", + f"The 'I Should Be Working' Hour on {channel_name}", + f"Productivity Killer - {channel_name}'s daytime programming", + ], + (12, 16): [ + f"Lunchtime Laziness with {channel_name}", + f"The Afternoon Slump - Brought to you by {channel_name}", + f"Post-Lunch Food Coma Theater on {channel_name}", + ], + (16, 20): [ + f"Rush Hour - {channel_name}'s alternative to traffic", + f"The 'What's For Dinner?' Debate on {channel_name}", + f"Evening Escapism - {channel_name}'s remedy for reality", + ], + (20, 24): [ + f"Prime Time Placeholder - {channel_name}'s finest not-programming", + f"The 'Netflix Was Too Complicated' Show on {channel_name}", + f"Family Argument Avoider - Courtesy of {channel_name}", + ], + } + + programs = [] + + # Create programs for each day + for day in range(num_days): + day_start = now + timedelta(days=day) + + # Create programs with specified length throughout the day + for hour_offset in range(0, 24, program_length_hours): + # Calculate program start and end times + start_time = day_start + timedelta(hours=hour_offset) + end_time = start_time + timedelta(hours=program_length_hours) + + # Get the hour for selecting a description + hour = start_time.hour + + # Find the appropriate time slot for description + for time_range, descriptions in time_descriptions.items(): + start_range, end_range = time_range + if start_range <= hour < end_range: + # Pick a description using the sum of the hour and day as seed + # This makes it somewhat random but consistent for the same timeslot + description = descriptions[(hour + day) % len(descriptions)] + break + else: + # Fallback description if somehow no range matches + description = f"Placeholder program for {channel_name} - EPG data went on vacation" + + programs.append({ + "channel_id": channel_id, + "start_time": start_time, + "end_time": end_time, + "title": channel_name, + "description": description, + }) + + return programs + + +def generate_custom_dummy_programs( + channel_id, + channel_name, + now, + num_days, + custom_properties, + export_lookback=None, + export_cutoff=None, +): + """ + Generate programs using custom dummy EPG regex patterns. + + Extracts information from channel title using regex patterns and generates + programs based on the extracted data. + + TIMEZONE HANDLING: + ------------------ + The timezone parameter specifies the timezone of the event times in your channel + titles using standard timezone names (e.g., 'US/Eastern', 'US/Pacific', 'Europe/London'). + DST (Daylight Saving Time) is handled automatically by pytz. + + Examples: + - Channel: "NHL 01: Bruins VS Maple Leafs @ 8:00PM ET" + - Set timezone = "US/Eastern" + - In October (DST): 8:00PM EDT → 12:00AM UTC (automatically uses UTC-4) + - In January (no DST): 8:00PM EST → 1:00AM UTC (automatically uses UTC-5) + + Args: + channel_id: Channel ID for the programs + channel_name: Channel title to parse + now: Current datetime (in UTC) + num_days: Number of days to generate programs for + custom_properties: Dict with title_pattern, time_pattern, templates, etc. + - timezone: Timezone name (e.g., 'US/Eastern') + + Returns: + List of program dictionaries with start_time/end_time in UTC + """ + import pytz + + logger.info(f"Generating custom dummy programs for channel: {channel_name}") + + # Extract patterns from custom properties + title_pattern = custom_properties.get('title_pattern', '') + time_pattern = custom_properties.get('time_pattern', '') + date_pattern = custom_properties.get('date_pattern', '') + + # Get timezone name (e.g., 'US/Eastern', 'US/Pacific', 'Europe/London') + timezone_value = custom_properties.get('timezone', 'UTC') + output_timezone_value = custom_properties.get('output_timezone', '') # Optional: display times in different timezone + program_duration = custom_properties.get('program_duration', 180) # Minutes + title_template = custom_properties.get('title_template', '') + subtitle_template = custom_properties.get('subtitle_template', '') + description_template = custom_properties.get('description_template', '') + + # Templates for upcoming/ended programs + upcoming_title_template = custom_properties.get('upcoming_title_template', '') + upcoming_description_template = custom_properties.get('upcoming_description_template', '') + ended_title_template = custom_properties.get('ended_title_template', '') + ended_description_template = custom_properties.get('ended_description_template', '') + + # Image URL templates + channel_logo_url_template = custom_properties.get('channel_logo_url', '') + program_poster_url_template = custom_properties.get('program_poster_url', '') + + # EPG metadata options + category_string = custom_properties.get('category', '') + # Split comma-separated categories and strip whitespace, filter out empty strings + categories = [cat.strip() for cat in category_string.split(',') if cat.strip()] if category_string else [] + include_date = custom_properties.get('include_date', True) + include_live = custom_properties.get('include_live', False) + include_new = custom_properties.get('include_new', False) + + # Parse timezone name + try: + source_tz = pytz.timezone(timezone_value) + logger.debug(f"Using timezone: {timezone_value} (DST will be handled automatically)") + except pytz.exceptions.UnknownTimeZoneError: + logger.warning(f"Unknown timezone: {timezone_value}, defaulting to UTC") + source_tz = pytz.utc + + # Parse output timezone if provided (for display purposes) + output_tz = None + if output_timezone_value: + try: + output_tz = pytz.timezone(output_timezone_value) + logger.debug(f"Using output timezone for display: {output_timezone_value}") + except pytz.exceptions.UnknownTimeZoneError: + logger.warning(f"Unknown output timezone: {output_timezone_value}, will use source timezone") + output_tz = None + + if not title_pattern: + logger.warning(f"No title_pattern in custom_properties, falling back to default") + return None + + logger.debug(f"Title pattern from DB: {repr(title_pattern)}") + + # Convert PCRE/JavaScript named groups (?) to Python format (?P) + # This handles patterns created with JavaScript regex syntax + # Use negative lookahead to avoid matching lookbehind (?<=) and negative lookbehind (?]+)>', r'(?P<\1>', title_pattern) + logger.debug(f"Converted title pattern: {repr(title_pattern)}") + + # Compile regex patterns using the enhanced regex module + # (supports variable-width lookbehinds like JavaScript) + try: + title_regex = regex.compile(title_pattern) + except Exception as e: + logger.error(f"Invalid title regex pattern after conversion: {e}") + logger.error(f"Pattern was: {repr(title_pattern)}") + return None + + time_regex = None + if time_pattern: + # Convert PCRE/JavaScript named groups to Python format + # Use negative lookahead to avoid matching lookbehind (?<=) and negative lookbehind (?]+)>', r'(?P<\1>', time_pattern) + logger.debug(f"Converted time pattern: {repr(time_pattern)}") + try: + time_regex = regex.compile(time_pattern) + except Exception as e: + logger.warning(f"Invalid time regex pattern after conversion: {e}") + logger.warning(f"Pattern was: {repr(time_pattern)}") + + # Compile date regex if provided + date_regex = None + if date_pattern: + # Convert PCRE/JavaScript named groups to Python format + # Use negative lookahead to avoid matching lookbehind (?<=) and negative lookbehind (?]+)>', r'(?P<\1>', date_pattern) + logger.debug(f"Converted date pattern: {repr(date_pattern)}") + try: + date_regex = regex.compile(date_pattern) + except Exception as e: + logger.warning(f"Invalid date regex pattern after conversion: {e}") + logger.warning(f"Pattern was: {repr(date_pattern)}") + + # Try to match the channel name with the title pattern + # Use search() instead of match() to match JavaScript behavior where .match() searches anywhere in the string + title_match = title_regex.search(channel_name) + if not title_match: + logger.debug(f"Channel name '{channel_name}' doesn't match title pattern") + return None + + groups = title_match.groupdict() + logger.debug(f"Title pattern matched. Groups: {groups}") + + # Helper function to format template with matched groups + def format_template(template, groups, url_encode=False): + """Replace {groupname} placeholders with matched group values + + Args: + template: Template string with {groupname} placeholders + groups: Dict of group names to values + url_encode: If True, URL encode the group values for safe use in URLs + """ + if not template: + return '' + result = template + for key, value in groups.items(): + if url_encode and value: + # URL encode the value to handle spaces and special characters + from urllib.parse import quote + encoded_value = quote(str(value), safe='') + result = result.replace(f'{{{key}}}', encoded_value) + else: + result = result.replace(f'{{{key}}}', str(value) if value else '') + return result + + # Extract time from title if time pattern exists + time_info = None + time_groups = {} + if time_regex: + time_match = time_regex.search(channel_name) + if time_match: + time_groups = time_match.groupdict() + try: + hour = int(time_groups.get('hour')) + # Handle optional minute group - could be None if not captured + minute_value = time_groups.get('minute') + minute = int(minute_value) if minute_value is not None else 0 + ampm = time_groups.get('ampm') + ampm = ampm.lower() if ampm else None + + # Determine if this is 12-hour or 24-hour format + if ampm in ('am', 'pm'): + # 12-hour format: convert to 24-hour + if ampm == 'pm' and hour != 12: + hour += 12 + elif ampm == 'am' and hour == 12: + hour = 0 + logger.debug(f"Extracted time (12-hour): {hour}:{minute:02d} {ampm}") + else: + # 24-hour format: hour is already in 24-hour format + # Validate that it's actually a 24-hour time (0-23) + if hour > 23: + logger.warning(f"Invalid 24-hour time: {hour}. Must be 0-23.") + hour = hour % 24 # Wrap around just in case + logger.debug(f"Extracted time (24-hour): {hour}:{minute:02d}") + + time_info = {'hour': hour, 'minute': minute} + except (ValueError, TypeError) as e: + logger.warning(f"Error parsing time: {e}") + + # Extract date from title if date pattern exists + date_info = None + date_groups = {} + if date_regex: + date_match = date_regex.search(channel_name) + if date_match: + date_groups = date_match.groupdict() + try: + # Support various date group names: month, day, year + month_str = date_groups.get('month', '') + day_str = date_groups.get('day', '') + year_str = date_groups.get('year', '') + + # Parse day - default to current day if empty or invalid + day = int(day_str) if day_str else now.day + + # Parse year - default to current year if empty or invalid (matches frontend behavior) + year = int(year_str) if year_str else now.year + + # Parse month - can be numeric (1-12) or text (Jan, January, etc.) + month = None + if month_str: + if month_str.isdigit(): + month = int(month_str) + else: + # Try to parse text month names + import calendar + month_str_lower = month_str.lower() + # Check full month names + for i, month_name in enumerate(calendar.month_name): + if month_name.lower() == month_str_lower: + month = i + break + # Check abbreviated month names if not found + if month is None: + for i, month_abbr in enumerate(calendar.month_abbr): + if month_abbr.lower() == month_str_lower: + month = i + break + + # Default to current month if not extracted or invalid + if month is None: + month = now.month + + if month and 1 <= month <= 12 and 1 <= day <= 31: + date_info = {'year': year, 'month': month, 'day': day} + logger.debug(f"Extracted date: {year}-{month:02d}-{day:02d}") + else: + logger.warning(f"Invalid date values: month={month}, day={day}, year={year}") + except (ValueError, TypeError) as e: + logger.warning(f"Error parsing date: {e}") + + # Merge title groups, time groups, and date groups for template formatting + all_groups = {**groups, **time_groups, **date_groups} + + # Add normalized versions of all groups for cleaner URLs + # These remove all non-alphanumeric characters and convert to lowercase + for key, value in list(all_groups.items()): + if value: + # Remove all non-alphanumeric characters (except spaces temporarily) + # then replace spaces with nothing, and convert to lowercase + normalized = regex.sub(r'[^a-zA-Z0-9\s]', '', str(value)) + normalized = regex.sub(r'\s+', '', normalized).lower() + all_groups[f'{key}_normalize'] = normalized + + # Format channel logo URL if template provided (with URL encoding) + channel_logo_url = None + if channel_logo_url_template: + channel_logo_url = format_template(channel_logo_url_template, all_groups, url_encode=True) + logger.debug(f"Formatted channel logo URL: {channel_logo_url}") + + # Format program poster URL if template provided (with URL encoding) + program_poster_url = None + if program_poster_url_template: + program_poster_url = format_template(program_poster_url_template, all_groups, url_encode=True) + logger.debug(f"Formatted program poster URL: {program_poster_url}") + + # Add formatted time strings for better display (handles minutes intelligently) + if time_info: + hour_24 = time_info['hour'] + minute = time_info['minute'] + + # Determine the base date to use for placeholders + # If date was extracted, use it; otherwise use current date + if date_info: + base_date = datetime(date_info['year'], date_info['month'], date_info['day']) + else: + base_date = datetime.now() + + # If output_timezone is specified, convert the display time to that timezone + if output_tz: + # Create a datetime in the source timezone using the base date + temp_date = source_tz.localize(base_date.replace(hour=hour_24, minute=minute, second=0, microsecond=0)) + # Convert to output timezone + temp_date_output = temp_date.astimezone(output_tz) + # Extract converted hour and minute for display + hour_24 = temp_date_output.hour + minute = temp_date_output.minute + logger.debug(f"Converted display time from {source_tz} to {output_tz}: {hour_24}:{minute:02d}") + + # Add date placeholders based on the OUTPUT timezone + # This ensures {date}, {month}, {day}, {year} reflect the converted timezone + all_groups['date'] = temp_date_output.strftime('%Y-%m-%d') + all_groups['month'] = str(temp_date_output.month) + all_groups['day'] = str(temp_date_output.day) + all_groups['year'] = str(temp_date_output.year) + logger.debug(f"Converted date placeholders to {output_tz}: {all_groups['date']}") + else: + # No output timezone conversion - use source timezone for date + # Create temp date to get proper date in source timezone using the base date + temp_date_source = source_tz.localize(base_date.replace(hour=hour_24, minute=minute, second=0, microsecond=0)) + all_groups['date'] = temp_date_source.strftime('%Y-%m-%d') + all_groups['month'] = str(temp_date_source.month) + all_groups['day'] = str(temp_date_source.day) + all_groups['year'] = str(temp_date_source.year) + + # Format 24-hour start time string - only include minutes if non-zero + if minute > 0: + all_groups['starttime24'] = f"{hour_24}:{minute:02d}" + else: + all_groups['starttime24'] = f"{hour_24:02d}:00" + + # Convert 24-hour to 12-hour format for {starttime} placeholder + # Note: hour_24 is ALWAYS in 24-hour format at this point (converted earlier if needed) + ampm = 'AM' if hour_24 < 12 else 'PM' + hour_12 = hour_24 + if hour_24 == 0: + hour_12 = 12 + elif hour_24 > 12: + hour_12 = hour_24 - 12 + + # Format 12-hour start time string - only include minutes if non-zero + if minute > 0: + all_groups['starttime'] = f"{hour_12}:{minute:02d} {ampm}" + else: + all_groups['starttime'] = f"{hour_12} {ampm}" + + # Format long version that always includes minutes (e.g., "9:00 PM" instead of "9 PM") + all_groups['starttime_long'] = f"{hour_12}:{minute:02d} {ampm}" + + # Calculate end time based on program duration + # Create a datetime for calculations + temp_start = datetime.now(source_tz).replace(hour=hour_24, minute=minute, second=0, microsecond=0) + temp_end = temp_start + timedelta(minutes=program_duration) + + # Extract end time components (already in correct timezone if output_tz was applied above) + end_hour_24 = temp_end.hour + end_minute = temp_end.minute + + # Format 24-hour end time string - only include minutes if non-zero + if end_minute > 0: + all_groups['endtime24'] = f"{end_hour_24}:{end_minute:02d}" + else: + all_groups['endtime24'] = f"{end_hour_24:02d}:00" + + # Convert 24-hour to 12-hour format for {endtime} placeholder + end_ampm = 'AM' if end_hour_24 < 12 else 'PM' + end_hour_12 = end_hour_24 + if end_hour_24 == 0: + end_hour_12 = 12 + elif end_hour_24 > 12: + end_hour_12 = end_hour_24 - 12 + + # Format 12-hour end time string - only include minutes if non-zero + if end_minute > 0: + all_groups['endtime'] = f"{end_hour_12}:{end_minute:02d} {end_ampm}" + else: + all_groups['endtime'] = f"{end_hour_12} {end_ampm}" + + # Format long version that always includes minutes (e.g., "9:00 PM" instead of "9 PM") + all_groups['endtime_long'] = f"{end_hour_12}:{end_minute:02d} {end_ampm}" + + # Generate programs + programs = [] + + # If we have extracted time AND date, the event happens on a SPECIFIC date + # If we have time but NO date, generate for multiple days (existing behavior) + # All other days and times show "Upcoming" before or "Ended" after + event_happened = False + + # Determine how many iterations we need + if date_info and time_info: + # Specific date extracted - only generate for that one date + iterations = 1 + logger.debug(f"Date extracted, generating single event for specific date") + else: + # No specific date - use num_days (existing behavior) + iterations = num_days + + for day in range(iterations): + event_overlaps_window = True + if date_info and time_info: + current_date = datetime( + date_info['year'], + date_info['month'], + date_info['day'], + ).date() + event_start_naive = datetime.combine( + current_date, + datetime.min.time().replace( + hour=time_info['hour'], + minute=time_info['minute'], + ), + ) + try: + event_start_utc = source_tz.localize(event_start_naive).astimezone(pytz.utc) + except Exception as e: + logger.error(f"Error localizing time to {source_tz}: {e}") + event_start_utc = django_timezone.make_aware(event_start_naive, pytz.utc) + event_end_utc = event_start_utc + timedelta(minutes=program_duration) + + lookback = export_lookback if export_lookback is not None else now + event_overlaps_window = _programme_overlaps_export_window( + event_start_utc, event_end_utc, lookback, export_cutoff + ) + if not event_overlaps_window: + logger.debug( + "Custom dummy event outside export window; filling window only: %s", + channel_name, + ) + event_happened = event_end_utc < lookback + day_start = _ceil_to_half_hour(lookback) + if export_cutoff is not None: + day_end = export_cutoff + else: + day_end = now + timedelta(days=num_days if num_days > 0 else 3) + else: + day_start = source_tz.localize( + datetime.combine(current_date, datetime.min.time()) + ).astimezone(pytz.utc) + day_end = day_start + timedelta(days=1) + if export_lookback is not None: + day_start = max(day_start, export_lookback) + if export_cutoff is not None: + day_end = min(day_end, export_cutoff) + else: + day_start = now + timedelta(days=day) + day_end = day_start + timedelta(days=1) + if export_lookback is not None: + day_start = max(day_start, export_lookback) + if export_cutoff is not None: + day_end = min(day_end, export_cutoff) + + if day_start >= day_end: + continue + + if time_info: + if not date_info: + now_in_source_tz = now.astimezone(source_tz) + current_date = (now_in_source_tz + timedelta(days=day)).date() + logger.debug(f"No date extracted, using day offset in {source_tz}: {current_date}") + + event_start_naive = datetime.combine( + current_date, + datetime.min.time().replace( + hour=time_info['hour'], + minute=time_info['minute'], + ), + ) + try: + event_start_local = source_tz.localize(event_start_naive) + event_start_utc = event_start_local.astimezone(pytz.utc) + logger.debug(f"Converted {event_start_local} to UTC: {event_start_utc}") + except Exception as e: + logger.error(f"Error localizing time to {source_tz}: {e}") + event_start_utc = django_timezone.make_aware(event_start_naive, pytz.utc) + + event_end_utc = event_start_utc + timedelta(minutes=program_duration) + + lookback = export_lookback if export_lookback is not None else now + if not _programme_overlaps_export_window( + event_start_utc, event_end_utc, lookback, export_cutoff + ): + continue + else: + logger.debug(f"Using extracted date: {current_date}") + + # Pre-generate the main event title and description for reuse + if title_template: + main_event_title = format_template(title_template, all_groups) + else: + title_parts = [] + if 'league' in all_groups and all_groups['league']: + title_parts.append(all_groups['league']) + if 'team1' in all_groups and 'team2' in all_groups: + title_parts.append(f"{all_groups['team1']} vs {all_groups['team2']}") + elif 'title' in all_groups and all_groups['title']: + title_parts.append(all_groups['title']) + main_event_title = ' - '.join(title_parts) if title_parts else channel_name + + if subtitle_template: + main_event_subtitle = format_template(subtitle_template, all_groups) + else: + main_event_subtitle = None + + if description_template: + main_event_description = format_template(description_template, all_groups) + else: + main_event_description = main_event_title + + + + # Determine if this day is before, during, or after the event + # Event only happens on day 0 (first day) when it falls inside the window + is_event_day = (day == 0) and event_overlaps_window + + if is_event_day and not event_happened: + current_time = day_start + + while current_time < event_start_utc and current_time < day_end: + program_start_utc = current_time + program_end_utc = min(current_time + timedelta(minutes=program_duration), event_start_utc) + + # Use custom upcoming templates if provided, otherwise use defaults + if upcoming_title_template: + upcoming_title = format_template(upcoming_title_template, all_groups) + else: + upcoming_title = main_event_title + + if upcoming_description_template: + upcoming_description = format_template(upcoming_description_template, all_groups) + else: + upcoming_description = f"Upcoming: {main_event_description}" + + # Build custom_properties for upcoming programs (only date, no category/live) + program_custom_properties = {} + + # Add date if requested (YYYY-MM-DD format from start time in event timezone) + if include_date: + # Convert UTC time to event timezone for date calculation + local_time = program_start_utc.astimezone(source_tz) + date_str = local_time.strftime('%Y-%m-%d') + program_custom_properties['date'] = date_str + + # Add program poster URL if provided + if program_poster_url: + program_custom_properties['icon'] = program_poster_url + + programs.append({ + "channel_id": channel_id, + "start_time": program_start_utc, + "end_time": program_end_utc, + "title": upcoming_title, + "sub_title": None, # No subtitle for filler programs + "description": upcoming_description, + "custom_properties": program_custom_properties, + "channel_logo_url": channel_logo_url, # Pass channel logo for EPG generation + }) + + current_time += timedelta(minutes=program_duration) + + # Add the MAIN EVENT at the extracted time + # Build custom_properties for main event (includes category and live) + main_event_custom_properties = {} + + # Add categories if provided + if categories: + main_event_custom_properties['categories'] = categories + + # Add date if requested (YYYY-MM-DD format from start time in event timezone) + if include_date: + # Convert UTC time to event timezone for date calculation + local_time = event_start_utc.astimezone(source_tz) + date_str = local_time.strftime('%Y-%m-%d') + main_event_custom_properties['date'] = date_str + + # Add live flag if requested + if include_live: + main_event_custom_properties['live'] = True + + # Add new flag if requested + if include_new: + main_event_custom_properties['new'] = True + + # Add program poster URL if provided + if program_poster_url: + main_event_custom_properties['icon'] = program_poster_url + + programs.append({ + "channel_id": channel_id, + "start_time": event_start_utc, + "end_time": event_end_utc, + "title": main_event_title, + "sub_title": main_event_subtitle, + "description": main_event_description, + "custom_properties": main_event_custom_properties, + "channel_logo_url": channel_logo_url, # Pass channel logo for EPG generation + }) + + event_happened = True + + # Fill programs AFTER the event until end of export day window + current_time = max(event_end_utc, day_start) + + while current_time < day_end: + program_start_utc = current_time + program_end_utc = min(current_time + timedelta(minutes=program_duration), day_end) + + # Use custom ended templates if provided, otherwise use defaults + if ended_title_template: + ended_title = format_template(ended_title_template, all_groups) + else: + ended_title = main_event_title + + if ended_description_template: + ended_description = format_template(ended_description_template, all_groups) + else: + ended_description = f"Ended: {main_event_description}" + + # Build custom_properties for ended programs (only date, no category/live) + program_custom_properties = {} + + # Add date if requested (YYYY-MM-DD format from start time in event timezone) + if include_date: + # Convert UTC time to event timezone for date calculation + local_time = program_start_utc.astimezone(source_tz) + date_str = local_time.strftime('%Y-%m-%d') + program_custom_properties['date'] = date_str + + # Add program poster URL if provided + if program_poster_url: + program_custom_properties['icon'] = program_poster_url + + programs.append({ + "channel_id": channel_id, + "start_time": program_start_utc, + "end_time": program_end_utc, + "title": ended_title, + "sub_title": None, # No subtitle for filler programs + "description": ended_description, + "custom_properties": program_custom_properties, + "channel_logo_url": channel_logo_url, # Pass channel logo for EPG generation + }) + + current_time += timedelta(minutes=program_duration) + else: + # This day is either before the event (future days) or after the event happened + # Fill entire day with appropriate message + current_time = day_start + + # If event already happened, all programs show "Ended" + # If event hasn't happened yet (shouldn't occur with day 0 logic), show "Upcoming" + is_ended = event_happened + + while current_time < day_end: + program_start_utc = current_time + program_end_utc = min(current_time + timedelta(minutes=program_duration), day_end) + + # Use custom templates based on whether event has ended or is upcoming + if is_ended: + if ended_title_template: + program_title = format_template(ended_title_template, all_groups) + else: + program_title = main_event_title + + if ended_description_template: + program_description = format_template(ended_description_template, all_groups) + else: + program_description = f"Ended: {main_event_description}" + else: + if upcoming_title_template: + program_title = format_template(upcoming_title_template, all_groups) + else: + program_title = main_event_title + + if upcoming_description_template: + program_description = format_template(upcoming_description_template, all_groups) + else: + program_description = f"Upcoming: {main_event_description}" + + # Build custom_properties (only date for upcoming/ended filler programs) + program_custom_properties = {} + + # Add date if requested (YYYY-MM-DD format from start time in event timezone) + if include_date: + # Convert UTC time to event timezone for date calculation + local_time = program_start_utc.astimezone(source_tz) + date_str = local_time.strftime('%Y-%m-%d') + program_custom_properties['date'] = date_str + + # Add program poster URL if provided + if program_poster_url: + program_custom_properties['icon'] = program_poster_url + + programs.append({ + "channel_id": channel_id, + "start_time": program_start_utc, + "end_time": program_end_utc, + "title": program_title, + "sub_title": None, # No subtitle for filler programs + "description": program_description, + "custom_properties": program_custom_properties, + "channel_logo_url": channel_logo_url, + }) + + current_time += timedelta(minutes=program_duration) + else: + # No extracted time - fill entire day with regular intervals + # day_start and day_end are already in UTC, so no conversion needed + programs_per_day = max(1, int(24 / (program_duration / 60))) + + for program_num in range(programs_per_day): + program_start_utc = day_start + timedelta(minutes=program_num * program_duration) + program_end_utc = program_start_utc + timedelta(minutes=program_duration) + + if title_template: + title = format_template(title_template, all_groups) + else: + title_parts = [] + if 'league' in all_groups and all_groups['league']: + title_parts.append(all_groups['league']) + if 'team1' in all_groups and 'team2' in all_groups: + title_parts.append(f"{all_groups['team1']} vs {all_groups['team2']}") + elif 'title' in all_groups and all_groups['title']: + title_parts.append(all_groups['title']) + title = ' - '.join(title_parts) if title_parts else channel_name + + if subtitle_template: + subtitle = format_template(subtitle_template, all_groups) + else: + subtitle = None + + if description_template: + description = format_template(description_template, all_groups) + else: + description = title + + # Build custom_properties for this program + program_custom_properties = {} + + # Add categories if provided + if categories: + program_custom_properties['categories'] = categories + + # Add date if requested (YYYY-MM-DD format from start time in event timezone) + if include_date: + # Convert UTC time to event timezone for date calculation + local_time = program_start_utc.astimezone(source_tz) + date_str = local_time.strftime('%Y-%m-%d') + program_custom_properties['date'] = date_str + + # Add live flag if requested + if include_live: + program_custom_properties['live'] = True + + # Add new flag if requested + if include_new: + program_custom_properties['new'] = True + + # Add program poster URL if provided + if program_poster_url: + program_custom_properties['icon'] = program_poster_url + + programs.append({ + "channel_id": channel_id, + "start_time": program_start_utc, + "end_time": program_end_utc, + "title": title, + "sub_title": subtitle, + "description": description, + "custom_properties": program_custom_properties, + "channel_logo_url": channel_logo_url, # Pass channel logo for EPG generation + }) + + logger.info(f"Generated {len(programs)} custom dummy programs for {channel_name}") + return programs + + +def generate_dummy_epg( + channel_id, channel_name, xml_lines=None, num_days=1, program_length_hours=4 +): + """ + Generate dummy EPG programs for channels without EPG data. + Creates program blocks for a specified number of days. + + Args: + channel_id: The channel ID to use in the program entries + channel_name: The name of the channel to use in program titles + xml_lines: Optional list to append lines to, otherwise returns new list + num_days: Number of days to generate EPG data for (default: 1) + program_length_hours: Length of each program block in hours (default: 4) + + Returns: + List of XML lines for the dummy EPG entries + """ + if xml_lines is None: + xml_lines = [] + + for program in generate_dummy_programs(channel_id, channel_name, num_days=1, program_length_hours=4): + # Format times in XMLTV format + start_str = program['start_time'].strftime("%Y%m%d%H%M%S %z") + stop_str = program['end_time'].strftime("%Y%m%d%H%M%S %z") + + # Create program entry with escaped channel name + xml_lines.append( + f' ' + ) + xml_lines.append(f" {html.escape(program['title'])}") + + # Add subtitle if available + if program.get('sub_title'): + xml_lines.append(f" {html.escape(program['sub_title'])}") + + xml_lines.append(f" {html.escape(program['description'])}") + + # Add custom_properties if present + custom_data = program.get('custom_properties', {}) + + # Categories + if 'categories' in custom_data: + for cat in custom_data['categories']: + xml_lines.append(f" {html.escape(cat)}") + + # Date tag + if 'date' in custom_data: + xml_lines.append(f" {html.escape(custom_data['date'])}") + + # Live tag + if custom_data.get('live', False): + xml_lines.append(f" ") + + # New tag + if custom_data.get('new', False): + xml_lines.append(f" ") + + xml_lines.append(f" ") + + return xml_lines + + +def generate_epg(request, profile_name=None, user=None): + """ + Dynamically generate an XMLTV (EPG) file using a streaming response. + Since the EPG data is stored independently of Channels, we group programmes + by their associated EPGData record. + """ + user_custom = (user.custom_properties or {}) if user else {} + try: + num_days = int(request.GET.get('days', user_custom.get('epg_days', 0))) + num_days = max(0, min(num_days, 365)) + except (ValueError, TypeError): + num_days = 0 + try: + prev_days = int(request.GET.get('prev_days', user_custom.get('epg_prev_days', 0))) + prev_days = max(0, min(prev_days, 30)) + except (ValueError, TypeError): + prev_days = 0 + use_cached_logos = request.GET.get('cachedlogos', 'true').lower() != 'false' + tvg_id_source = request.GET.get('tvg_id_source', 'channel_number').lower() + cache_params = ( + f"{profile_name or 'all'}:{user.username if user else 'anonymous'}" + f":d={num_days}:p={prev_days}:logos={use_cached_logos}:tvgid={tvg_id_source}" + ) + content_cache_key = f"epg_content:{cache_params}" + + def epg_generator(): + """Generator function that yields EPG data with keep-alives during processing.""" + + yield '\n' + yield ( + '\n' + ) + + # Get channels based on user/profile + if user is not None: + if user.user_level < 10: + user_profile_count = user.channel_profiles.count() + + # If user has ALL profiles or NO profiles, give unrestricted access + if user_profile_count == 0: + # No profile filtering - user sees all channels based on user_level + filters = {"user_level__lte": user.user_level} + # Hide adult content if user preference is set + if (user.custom_properties or {}).get('hide_adult_content', False): + filters["is_adult"] = False + base_qs = Channel.objects.filter(**filters).select_related('logo', 'epg_data__epg_source') + else: + # User has specific limited profiles assigned + filters = { + "channelprofilemembership__enabled": True, + "user_level__lte": user.user_level, + "channelprofilemembership__channel_profile__in": user.channel_profiles.all() + } + # Hide adult content if user preference is set + if (user.custom_properties or {}).get('hide_adult_content', False): + filters["is_adult"] = False + base_qs = Channel.objects.filter(**filters).select_related('logo', 'epg_data__epg_source').distinct() + else: + base_qs = Channel.objects.filter(user_level__lte=user.user_level).select_related('logo', 'epg_data__epg_source') + else: + if profile_name is not None: + try: + channel_profile = ChannelProfile.objects.get(name=profile_name) + except ChannelProfile.DoesNotExist: + logger.warning("Requested channel profile (%s) during epg generation does not exist", profile_name) + raise Http404(f"Channel profile '{profile_name}' not found") + base_qs = Channel.objects.filter( + channelprofilemembership__channel_profile=channel_profile, + channelprofilemembership__enabled=True, + ).select_related('logo', 'epg_data__epg_source') + else: + base_qs = Channel.objects.all().select_related('logo', 'epg_data__epg_source') + + # Resolve effective values at SQL level and exclude hidden channels + # so output ordering/display honors user overrides. + from apps.channels.managers import with_effective_values + channels = list( + with_effective_values(base_qs, select_related_fks=True) + .exclude(hidden_from_output=True) + .order_by("effective_channel_number") + .prefetch_related( + Prefetch('streams', queryset=Stream.objects.only('id', 'name').order_by('channelstream__order')) + ) + ) + channel_count = len(channels) + + # For dummy EPG, use either the specified value or default to 3 days + dummy_days = num_days if num_days > 0 else 3 + + # Calculate cutoff dates for EPG data filtering + now = django_timezone.now() + cutoff_date = now + timedelta(days=num_days) if num_days > 0 else None + lookback_cutoff = now - timedelta(days=prev_days) + + # Build collision-free channel number mapping for XC clients (if user is authenticated) + # XC clients require integer channel numbers, so we need to ensure no conflicts + channel_num_map = {} + if user is not None: + # This is an XC client - build collision-free mapping + used_numbers = set() + + # First pass: assign integers for channels that already have integer numbers + for channel in channels: + effective_num = channel.effective_channel_number + if effective_num is not None and effective_num == int(effective_num): + num = int(effective_num) + channel_num_map[channel.id] = num + used_numbers.add(num) + + # Second pass: assign integers for channels with float numbers + for channel in channels: + effective_num = channel.effective_channel_number + if effective_num is not None and effective_num != int(effective_num): + candidate = int(effective_num) + while candidate in used_numbers: + candidate += 1 + channel_num_map[channel.id] = candidate + used_numbers.add(candidate) + + # Host/port/scheme are constant per request; precompute logo URL prefix once. + _base_url = build_absolute_uri_with_port(request, "") + _sample_logo_path = reverse("api:channels:logo-cache", args=[0]) + _logo_prefix_raw, _, _logo_suffix_raw = _sample_logo_path.partition("/0/") + _logo_url_prefix = _base_url + _logo_prefix_raw + "/" + _logo_url_suffix = "/" + _logo_suffix_raw + + dummy_program_list = [] + real_epg_map = {} + channel_xml_batch = [] + + for channel in channels: + effective_name = channel.effective_name + effective_epg_data = channel.effective_epg_data_obj + effective_epg_data_id = channel.effective_epg_data_id + effective_logo = channel.effective_logo_obj + effective_number = channel.effective_channel_number + + # user is set only for XC clients, which require integer channel numbers + if user is not None: + formatted_channel_number = channel_num_map[channel.id] + else: + formatted_channel_number = format_channel_number(effective_number) + + # Determine the channel ID based on the selected source + if tvg_id_source == 'tvg_id' and channel.effective_tvg_id: + channel_id = channel.effective_tvg_id + elif tvg_id_source == 'gracenote' and channel.effective_tvc_guide_stationid: + channel_id = channel.effective_tvc_guide_stationid + else: + channel_id = str(formatted_channel_number) if formatted_channel_number != "" else str(channel.id) + + tvg_logo = "" + + # Check if this is a custom dummy EPG with channel logo URL template + if effective_epg_data and effective_epg_data.epg_source and effective_epg_data.epg_source.source_type == 'dummy': + epg_source = effective_epg_data.epg_source + if epg_source.custom_properties: + custom_props = epg_source.custom_properties + channel_logo_url_template = custom_props.get('channel_logo_url', '') + + if channel_logo_url_template: + pattern_match_name, _ = _pattern_match_name_from_custom_props( + channel, effective_name, custom_props + ) + + # Try to extract groups from the channel/stream name and build the logo URL + title_pattern = custom_props.get('title_pattern', '') + if title_pattern: + try: + # Convert PCRE/JavaScript named groups to Python format + title_pattern = regex.sub(r'\(\?<(?![=!])([^>]+)>', r'(?P<\1>', title_pattern) + title_regex = regex.compile(title_pattern) + title_match = title_regex.search(pattern_match_name) + + if title_match: + groups = title_match.groupdict() + + # Add normalized versions of all groups for cleaner URLs + for key, value in list(groups.items()): + if value: + # Remove all non-alphanumeric characters and convert to lowercase + normalized = regex.sub(r'[^a-zA-Z0-9\s]', '', str(value)) + normalized = regex.sub(r'\s+', '', normalized).lower() + groups[f'{key}_normalize'] = normalized + + # Format the logo URL template with the matched groups (with URL encoding) + from urllib.parse import quote + for key, value in groups.items(): + if value: + encoded_value = quote(str(value), safe='') + channel_logo_url_template = channel_logo_url_template.replace(f'{{{key}}}', encoded_value) + else: + channel_logo_url_template = channel_logo_url_template.replace(f'{{{key}}}', '') + tvg_logo = channel_logo_url_template + logger.debug(f"Built channel logo URL from template: {tvg_logo}") + except Exception as e: + logger.warning(f"Failed to build channel logo URL for {effective_name}: {e}") + + # If no custom dummy logo, use regular logo logic + if not tvg_logo and effective_logo: + if use_cached_logos: + tvg_logo = f"{_logo_url_prefix}{effective_logo.id}{_logo_url_suffix}" + else: + # Use direct URL if available, otherwise fall back to cached version + direct_logo = effective_logo.url if effective_logo.url.startswith(('http://', 'https://')) else None + if direct_logo: + tvg_logo = direct_logo + else: + tvg_logo = f"{_logo_url_prefix}{effective_logo.id}{_logo_url_suffix}" + channel_xml_batch.append(f' ') + channel_xml_batch.append(f' {html.escape(effective_name)}') + channel_xml_batch.append(f' ') + channel_xml_batch.append(" ") + + if len(channel_xml_batch) >= _EPG_CHANNEL_XML_BATCH_SIZE * 4: + yield '\n'.join(channel_xml_batch) + '\n' + channel_xml_batch = [] + + pattern_match_name = effective_name + if effective_epg_data and effective_epg_data.epg_source: + epg_source = effective_epg_data.epg_source + if epg_source.custom_properties: + custom_props = epg_source.custom_properties + pattern_match_name, stream_lookup_failed = _pattern_match_name_from_custom_props( + channel, effective_name, custom_props + ) + if ( + custom_props.get('name_source') == 'stream' + and not stream_lookup_failed + and pattern_match_name != effective_name + ): + stream_index = custom_props.get('stream_index', 1) - 1 + logger.debug( + f"Using stream name for parsing: {pattern_match_name} " + f"(stream index: {stream_index})" + ) + elif stream_lookup_failed: + stream_index = custom_props.get('stream_index', 1) - 1 + logger.warning( + f"Stream index {stream_index} not found for channel " + f"{effective_name}, falling back to channel name" + ) + + if not effective_epg_data: + dummy_program_list.append((channel_id, pattern_match_name, None)) + elif effective_epg_data.epg_source and effective_epg_data.epg_source.source_type == 'dummy': + dummy_program_list.append((channel_id, pattern_match_name, effective_epg_data.epg_source)) + else: + real_epg_map.setdefault(effective_epg_data_id, []).append(channel_id) + + if channel_xml_batch: + yield '\n'.join(channel_xml_batch) + '\n' + + del channels + del channel_num_map + + batch_size = _EPG_PROGRAM_YIELD_BATCH_SIZE + + all_epg_ids = list(real_epg_map.keys()) + if all_epg_ids: + if num_days > 0: + programs_qs = ProgramData.objects.filter( + epg_id__in=all_epg_ids, + end_time__gte=lookback_cutoff, + start_time__lt=cutoff_date, + ) + else: + programs_qs = ProgramData.objects.filter( + epg_id__in=all_epg_ids, + end_time__gte=lookback_cutoff, + ) + + programs_base_qs = programs_qs.order_by('epg_id', 'id').values( + 'id', 'epg_id', 'start_time', 'end_time', 'title', 'sub_title', + 'description', 'custom_properties', + ) + + current_epg_id = None + channel_ids_for_epg = None + escaped_primary_cid = None + pending = [] + program_batch = [] + chunk_size = _EPG_PROGRAM_DB_CHUNK_SIZE + last_epg_id = 0 + last_id = 0 + _poster_url_base = build_absolute_uri_with_port(request, "/api/epg/programs/") + + def flush_pending(): + nonlocal program_batch, pending + if not pending: + return + pending.sort(key=lambda row: (row[0], row[1])) + escaped_primary = ( + escaped_primary_cid if len(channel_ids_for_epg) > 1 else None + ) + for _, _, xml_text in pending: + program_batch.append(xml_text) + if escaped_primary: + for cid in channel_ids_for_epg[1:]: + program_batch.append(xml_text.replace( + f'channel="{escaped_primary}"', + f'channel="{html.escape(cid)}"', + 1, + )) + if len(program_batch) >= batch_size: + yield '\n'.join(program_batch) + '\n' + program_batch = [] + pending.clear() + + while True: + program_chunk = list( + programs_base_qs.filter(epg_id__gte=last_epg_id) + .exclude(epg_id=last_epg_id, id__lte=last_id)[:chunk_size] + ) + if not program_chunk: + break + + last_row = program_chunk[-1] + last_epg_id = last_row['epg_id'] + last_id = last_row['id'] + + for prog in program_chunk: + epg_id = prog['epg_id'] + + if epg_id != current_epg_id: + yield from flush_pending() + current_epg_id = epg_id + channel_ids_for_epg = real_epg_map[epg_id] + escaped_primary_cid = html.escape(channel_ids_for_epg[0]) + + # DB datetimes are UTC (USE_TZ=True, TIME_ZONE=UTC); format + # directly instead of strftime("%Y%m%d%H%M%S %z"), which is + # ~10x slower and dominates XML build over 750k rows. + st = prog['start_time'] + et = prog['end_time'] + start_str = f"{st.year:04d}{st.month:02d}{st.day:02d}{st.hour:02d}{st.minute:02d}{st.second:02d} +0000" + stop_str = f"{et.year:04d}{et.month:02d}{et.day:02d}{et.hour:02d}{et.minute:02d}{et.second:02d} +0000" + + program_xml = [f' '] + program_xml.append(f' {html.escape(prog["title"])}') + + if prog['sub_title']: + program_xml.append(f" {html.escape(prog['sub_title'])}") + + if prog['description']: + program_xml.append(f" {html.escape(prog['description'])}") + + custom_data = prog['custom_properties'] or {} + if custom_data: + + if "categories" in custom_data and custom_data["categories"]: + for category in custom_data["categories"]: + program_xml.append(f" {html.escape(category)}") + + if "keywords" in custom_data and custom_data["keywords"]: + for keyword in custom_data["keywords"]: + program_xml.append(f" {html.escape(keyword)}") + + # onscreen_episode takes priority over episode for the onscreen system + if "onscreen_episode" in custom_data: + program_xml.append(f' {html.escape(custom_data["onscreen_episode"])}') + elif "episode" in custom_data: + program_xml.append(f' E{custom_data["episode"]}') + + # Handle dd_progid format + if 'dd_progid' in custom_data: + program_xml.append(f' {html.escape(custom_data["dd_progid"])}') + + # Handle external database IDs + for system in ['thetvdb.com', 'themoviedb.org', 'imdb.com']: + if f'{system}_id' in custom_data: + program_xml.append(f' {html.escape(custom_data[f"{system}_id"])}') + + # Add season and episode numbers in xmltv_ns format if available + if "season" in custom_data and "episode" in custom_data: + season = ( + int(custom_data["season"]) - 1 + if str(custom_data["season"]).isdigit() + else 0 + ) + episode = ( + int(custom_data["episode"]) - 1 + if str(custom_data["episode"]).isdigit() + else 0 + ) + program_xml.append(f' {season}.{episode}.') + + if "language" in custom_data: + program_xml.append(f' {html.escape(custom_data["language"])}') + + if "original_language" in custom_data: + program_xml.append(f' {html.escape(custom_data["original_language"])}') + + if "length" in custom_data and isinstance(custom_data["length"], dict): + length_value = custom_data["length"].get("value", "") + length_units = custom_data["length"].get("units", "minutes") + program_xml.append(f' {html.escape(str(length_value))}') + + if "video" in custom_data and isinstance(custom_data["video"], dict): + program_xml.append(" ") + + if "audio" in custom_data and isinstance(custom_data["audio"], dict): + program_xml.append(" ") + + if "subtitles" in custom_data and isinstance(custom_data["subtitles"], list): + for subtitle in custom_data["subtitles"]: + if isinstance(subtitle, dict): + subtitle_type = subtitle.get("type", "") + type_attr = f' type="{html.escape(subtitle_type)}"' if subtitle_type else "" + program_xml.append(f" ") + if "language" in subtitle: + program_xml.append(f" {html.escape(subtitle['language'])}") + program_xml.append(" ") + + if "rating" in custom_data: + rating_system = custom_data.get("rating_system", "TV Parental Guidelines") + program_xml.append(f' ') + program_xml.append(f' {html.escape(custom_data["rating"])}') + program_xml.append(f" ") + + if "star_ratings" in custom_data and isinstance(custom_data["star_ratings"], list): + for star_rating in custom_data["star_ratings"]: + if isinstance(star_rating, dict) and "value" in star_rating: + system_attr = f' system="{html.escape(star_rating["system"])}"' if "system" in star_rating else "" + program_xml.append(f" ") + program_xml.append(f" {html.escape(star_rating['value'])}") + program_xml.append(" ") + + if "reviews" in custom_data and isinstance(custom_data["reviews"], list): + for review in custom_data["reviews"]: + if isinstance(review, dict) and "content" in review: + review_type = review.get("type", "text") + attrs = [f'type="{html.escape(review_type)}"'] + if "source" in review: + attrs.append(f'source="{html.escape(review["source"])}"') + if "reviewer" in review: + attrs.append(f'reviewer="{html.escape(review["reviewer"])}"') + attr_str = " ".join(attrs) + program_xml.append(f' {html.escape(review["content"])}') + + if "images" in custom_data and isinstance(custom_data["images"], list): + for image in custom_data["images"]: + if isinstance(image, dict) and "url" in image: + attrs = [] + for attr in ['type', 'size', 'orient', 'system']: + if attr in image: + attrs.append(f'{attr}="{html.escape(image[attr])}"') + attr_str = " " + " ".join(attrs) if attrs else "" + program_xml.append(f' {html.escape(image["url"])}') + + # Add enhanced credits handling + if "credits" in custom_data: + program_xml.append(" ") + credits = custom_data["credits"] + + for role in ['director', 'writer', 'adapter', 'producer', 'composer', 'editor', 'presenter', 'commentator', 'guest']: + if role in credits: + people = credits[role] + if isinstance(people, list): + for person in people: + program_xml.append(f" <{role}>{html.escape(person)}") + else: + program_xml.append(f" <{role}>{html.escape(people)}") + + # Handle actors separately to include role and guest attributes + if "actor" in credits: + actors = credits["actor"] + if isinstance(actors, list): + for actor in actors: + if isinstance(actor, dict): + name = actor.get("name", "") + role_attr = f' role="{html.escape(actor["role"])}"' if "role" in actor else "" + guest_attr = ' guest="yes"' if actor.get("guest") else "" + program_xml.append(f" {html.escape(name)}") + else: + program_xml.append(f" {html.escape(actor)}") + else: + program_xml.append(f" {html.escape(actors)}") + + program_xml.append(" ") + + if "date" in custom_data: + program_xml.append(f' {html.escape(custom_data["date"])}') + + if "country" in custom_data: + program_xml.append(f' {html.escape(custom_data["country"])}') + + if "icon" in custom_data: + program_xml.append(f' ') + elif "sd_icon" in custom_data: + program_xml.append(f' ') + + # Add special flags as proper tags with enhanced handling + if custom_data.get("previously_shown", False): + prev_shown_details = custom_data.get("previously_shown_details", {}) + attrs = [] + if "start" in prev_shown_details: + attrs.append(f'start="{html.escape(prev_shown_details["start"])}"') + if "channel" in prev_shown_details: + attrs.append(f'channel="{html.escape(prev_shown_details["channel"])}"') + attr_str = " " + " ".join(attrs) if attrs else "" + program_xml.append(f" ") + + if custom_data.get("premiere", False): + premiere_text = custom_data.get("premiere_text", "") + if premiere_text: + program_xml.append(f" {html.escape(premiere_text)}") + else: + program_xml.append(" ") + + if custom_data.get("last_chance", False): + last_chance_text = custom_data.get("last_chance_text", "") + if last_chance_text: + program_xml.append(f" {html.escape(last_chance_text)}") + else: + program_xml.append(" ") + + if custom_data.get("new", False): + program_xml.append(" ") + + if custom_data.get('live', False): + program_xml.append(' ') + + program_xml.append(" ") + + xml_text = '\n'.join(program_xml) + pending.append((prog['start_time'], prog['id'], xml_text)) + + del program_chunk + + yield from flush_pending() + + if program_batch: + yield '\n'.join(program_batch) + '\n' + + del real_epg_map + + for channel_id, pattern_match_name, epg_source in dummy_program_list: + program_length_hours = 4 + dummy_programs = generate_dummy_programs( + channel_id, pattern_match_name, + num_days=dummy_days, + program_length_hours=program_length_hours, + epg_source=epg_source, + export_lookback=lookback_cutoff, + export_cutoff=cutoff_date, + ) + if not dummy_programs: + continue + dummy_batch = [] + for program in dummy_programs: + start_str = program['start_time'].strftime("%Y%m%d%H%M%S %z") + stop_str = program['end_time'].strftime("%Y%m%d%H%M%S %z") + lines = [ + f' ', + f" {html.escape(program['title'])}", + ] + if program.get('sub_title'): + lines.append(f" {html.escape(program['sub_title'])}") + lines.append(f" {html.escape(program['description'])}") + custom_data = program.get('custom_properties', {}) + if 'categories' in custom_data: + for cat in custom_data['categories']: + lines.append(f" {html.escape(cat)}") + if 'date' in custom_data: + lines.append(f" {html.escape(custom_data['date'])}") + if custom_data.get('live', False): + lines.append(" ") + if custom_data.get('new', False): + lines.append(" ") + if 'icon' in custom_data: + lines.append(f' ') + lines.append(" ") + dummy_batch.append('\n'.join(lines)) + if len(dummy_batch) >= batch_size: + yield '\n'.join(dummy_batch) + '\n' + dummy_batch = [] + del dummy_programs + if dummy_batch: + yield '\n'.join(dummy_batch) + '\n' + + del dummy_program_list + + yield "\n" + + from apps.output.views import get_client_identifier + + client_id, client_ip, user_agent = get_client_identifier(request) + event_cache_key = f"epg_download:{user.username if user else 'anonymous'}:{profile_name or 'all'}:{client_id}" + + def _log_epg_download(): + from django.core.cache import cache as event_cache + + if not event_cache.get(event_cache_key): + log_system_event( + event_type='epg_download', + profile=profile_name or 'all', + user=user.username if user else 'anonymous', + channels=channel_count, + client_ip=client_ip, + user_agent=user_agent, + ) + event_cache.set(event_cache_key, True, 2) + + try: + from core.utils import _is_gevent_monkey_patched + + if _is_gevent_monkey_patched(): + import gevent + + gevent.spawn(_log_epg_download) + else: + _log_epg_download() + except Exception: + _log_epg_download() + + def build_epg_stream(): + try: + yield from epg_generator() + finally: + _epg_export_teardown() + + return stream_cached_response( + content_cache_key, + build_epg_stream, + content_type="application/xml", + filename="Dispatcharr.xml", + ) diff --git a/apps/output/streaming_chunk_cache.py b/apps/output/streaming_chunk_cache.py new file mode 100644 index 00000000..4373f5ce --- /dev/null +++ b/apps/output/streaming_chunk_cache.py @@ -0,0 +1,239 @@ +"""Single-flight Redis chunk cache for large streaming HTTP responses.""" + +import logging +import time + +from django.http import StreamingHttpResponse + +logger = logging.getLogger(__name__) + +STATUS_BUILDING = "building" +STATUS_READY = "ready" +STATUS_ERROR = "error" + +DEFAULT_CACHE_TTL = 300 +DEFAULT_LOCK_TTL = 120 +DEFAULT_POLL_INTERVAL = 0.05 +DEFAULT_MAX_FOLLOWER_WAIT = 600 + + +def _chunks_key(base_key): + return f"{base_key}:chunks" + + +def _ready_key(base_key): + return f"{base_key}:ready" + + +def _status_key(base_key): + return f"{base_key}:status" + + +def _lock_key(base_key): + return f"{base_key}:lock" + + +def _decode_chunk(chunk): + if chunk is None: + return None + if isinstance(chunk, bytes): + return chunk.decode("utf-8") + return chunk + + +def _encode_chunk(chunk): + if isinstance(chunk, bytes): + return chunk + return chunk.encode("utf-8") + + +def _poll_wait(interval): + try: + from core.utils import _is_gevent_monkey_patched + + if _is_gevent_monkey_patched(): + import gevent + + gevent.sleep(interval) + return + except ImportError: + pass + time.sleep(interval) + + +def _get_redis(): + from django_redis import get_redis_connection + + return get_redis_connection("default") + + +def _get_status(redis, base_key): + raw = redis.get(_status_key(base_key)) + if raw is None: + return None + return _decode_chunk(raw) + + +def _clear_build_keys(redis, base_key): + redis.delete( + _chunks_key(base_key), + _status_key(base_key), + _ready_key(base_key), + _lock_key(base_key), + ) + + +def _try_acquire_lock(redis, base_key, lock_ttl): + return bool(redis.set(_lock_key(base_key), "1", nx=True, ex=lock_ttl)) + + +def _refresh_build_ttl(redis, base_key, lock_ttl): + redis.expire(_lock_key(base_key), lock_ttl) + redis.expire(_status_key(base_key), lock_ttl) + redis.expire(_chunks_key(base_key), lock_ttl) + + +def _stream_ready(redis, base_key): + offset = 0 + chunks_key = _chunks_key(base_key) + while True: + chunk = redis.lindex(chunks_key, offset) + if chunk is None: + break + yield _decode_chunk(chunk) + offset += 1 + + +def _stream_build(redis, base_key, source, cache_ttl, lock_ttl): + """Leader: stream to client and append each chunk to Redis.""" + chunks_key = _chunks_key(base_key) + status_key = _status_key(base_key) + try: + from django.core.cache import cache as django_cache + + django_cache.delete(base_key) # clear any non-chunked entry under this key + redis.delete(chunks_key, _ready_key(base_key)) + redis.set(status_key, STATUS_BUILDING, ex=lock_ttl) + refresh_interval = max(1, lock_ttl // 4) + last_refresh = 0.0 + for chunk in source(): + redis.rpush(chunks_key, _encode_chunk(chunk)) + now = time.monotonic() + if now - last_refresh >= refresh_interval: + _refresh_build_ttl(redis, base_key, lock_ttl) + last_refresh = now + yield chunk + redis.set(status_key, STATUS_READY) + redis.set(_ready_key(base_key), "1") + redis.expire(chunks_key, cache_ttl) + redis.expire(status_key, cache_ttl) + redis.expire(_ready_key(base_key), cache_ttl) + logger.debug("Cached response in %s chunks", redis.llen(chunks_key)) + except Exception: + logger.exception("Chunk cache build failed for %s", base_key) + redis.delete(chunks_key) + redis.set(status_key, STATUS_ERROR, ex=60) + raise + finally: + redis.delete(_lock_key(base_key)) + + +def _stream_follow(redis, base_key, source, cache_ttl, lock_ttl, poll_interval, max_follower_wait): + """Follower: read chunks as the leader writes them.""" + offset = 0 + deadline = time.monotonic() + max_follower_wait + idle_polls = 0 + chunks_key = _chunks_key(base_key) + lock_key = _lock_key(base_key) + + while True: + chunk = redis.lindex(chunks_key, offset) + if chunk is not None: + idle_polls = 0 + yield _decode_chunk(chunk) + offset += 1 + continue + + status = _get_status(redis, base_key) + if status == STATUS_READY: + break + + if status == STATUS_ERROR: + _clear_build_keys(redis, base_key) + if offset == 0 and _try_acquire_lock(redis, base_key, lock_ttl): + yield from _stream_build(redis, base_key, source, cache_ttl, lock_ttl) + return + raise RuntimeError("Chunk cache build failed") + + if time.monotonic() >= deadline: + if offset == 0 and _try_acquire_lock(redis, base_key, lock_ttl): + logger.warning("Chunk cache follower timed out; rebuilding %s", base_key) + yield from _stream_build(redis, base_key, source, cache_ttl, lock_ttl) + return + logger.warning("Chunk cache follower timed out after partial read for %s", base_key) + break + + lock_active = bool(redis.exists(lock_key)) + if status != STATUS_BUILDING and not lock_active: + idle_polls += 1 + if offset == 0 and idle_polls >= max(1, int(1.0 / poll_interval)): + if _try_acquire_lock(redis, base_key, lock_ttl): + logger.warning("Chunk cache leader lost; rebuilding %s", base_key) + yield from _stream_build(redis, base_key, source, cache_ttl, lock_ttl) + return + else: + idle_polls = 0 + + _poll_wait(poll_interval) + + +def stream_cached_response( + cache_key, + source, + *, + content_type="application/xml", + filename=None, + cache_ttl=DEFAULT_CACHE_TTL, + lock_ttl=DEFAULT_LOCK_TTL, + poll_interval=DEFAULT_POLL_INTERVAL, + max_follower_wait=DEFAULT_MAX_FOLLOWER_WAIT, + redis=None, +): + """ + Stream a large response with single-flight Redis chunk caching. + + ``source`` must be a callable returning a chunk iterator. Only the leader + invokes it; concurrent followers replay chunks already written to Redis, so + the expensive ``source`` runs at most once per ``cache_key``. + """ + if redis is None: + redis = _get_redis() + + if redis.get(_ready_key(cache_key)): + logger.debug("Serving response from chunk cache") + stream = _stream_ready(redis, cache_key) + else: + status = _get_status(redis, cache_key) + if status == STATUS_ERROR: + _clear_build_keys(redis, cache_key) + + if _try_acquire_lock(redis, cache_key, lock_ttl): + logger.debug("Building response (cache leader)") + stream = _stream_build(redis, cache_key, source, cache_ttl, lock_ttl) + else: + logger.debug("Following in-flight cache build") + stream = _stream_follow( + redis, + cache_key, + source, + cache_ttl, + lock_ttl, + poll_interval, + max_follower_wait, + ) + + response = StreamingHttpResponse(stream, content_type=content_type) + if filename: + response["Content-Disposition"] = f'attachment; filename="{filename}"' + response["Cache-Control"] = "no-cache" + return response diff --git a/apps/output/test_streaming_chunk_cache.py b/apps/output/test_streaming_chunk_cache.py new file mode 100644 index 00000000..599501b7 --- /dev/null +++ b/apps/output/test_streaming_chunk_cache.py @@ -0,0 +1,187 @@ +import threading +import time +from unittest import TestCase + +from apps.output.streaming_chunk_cache import ( + STATUS_BUILDING, + STATUS_READY, + _chunks_key, + _lock_key, + _ready_key, + _status_key, + stream_cached_response, +) + + +class FakeRedis: + """Minimal Redis stand-in for chunk-cache unit tests.""" + + def __init__(self): + self._strings = {} + self._lists = {} + self._expires_at = {} + + def _purge_expired(self): + now = time.monotonic() + expired = [key for key, deadline in self._expires_at.items() if deadline <= now] + for key in expired: + self._strings.pop(key, None) + self._lists.pop(key, None) + self._expires_at.pop(key, None) + + def get(self, key): + self._purge_expired() + return self._strings.get(key) + + def set(self, key, value, nx=False, ex=None): + self._purge_expired() + if nx and key in self._strings: + return None + self._strings[key] = value + if ex is not None: + self._expires_at[key] = time.monotonic() + ex + return True + + def delete(self, *keys): + for key in keys: + self._strings.pop(key, None) + self._lists.pop(key, None) + self._expires_at.pop(key, None) + + def exists(self, key): + self._purge_expired() + return key in self._strings or key in self._lists + + def expire(self, key, ttl): + if key in self._strings or key in self._lists: + self._expires_at[key] = time.monotonic() + ttl + return True + + def rpush(self, key, value): + self._lists.setdefault(key, []).append(value) + + def lindex(self, key, offset): + items = self._lists.get(key, []) + if offset < len(items): + return items[offset] + return None + + def llen(self, key): + return len(self._lists.get(key, [])) + + +def _consume(response): + return b"".join(response.streaming_content).decode("utf-8") + + +class StreamingChunkCacheTests(TestCase): + def test_leader_caches_chunks_and_sets_ready(self): + redis = FakeRedis() + calls = [] + + def source(): + calls.append(1) + yield "" + yield "" + + body = _consume(stream_cached_response("cache:test", source, redis=redis)) + + self.assertEqual(body, "") + self.assertEqual(calls, [1]) + self.assertEqual(redis.get(_ready_key("cache:test")), "1") + self.assertEqual(redis.get(_status_key("cache:test")), STATUS_READY) + self.assertEqual(redis.llen(_chunks_key("cache:test")), 2) + self.assertFalse(redis.exists(_lock_key("cache:test"))) + + def test_cache_hit_skips_source(self): + redis = FakeRedis() + calls = [] + + def source(): + calls.append(1) + yield "" + yield "" + + _consume(stream_cached_response("cache:test", source, redis=redis)) + calls.clear() + body = _consume(stream_cached_response("cache:test", source, redis=redis)) + + self.assertEqual(body, "") + self.assertEqual(calls, []) + + def test_follower_reads_leader_chunks_without_rebuilding(self): + redis = FakeRedis() + base = "cache:follow" + leader_started = threading.Event() + rebuild_calls = [] + + def slow_source(): + rebuild_calls.append(1) + leader_started.set() + yield "a" + time.sleep(0.05) + yield "b" + + def forbidden_source(): + rebuild_calls.append(2) + yield "SHOULD_NOT_RUN" + + def leader(): + _consume( + stream_cached_response( + base, + slow_source, + redis=redis, + poll_interval=0.01, + ) + ) + + leader_thread = threading.Thread(target=leader) + leader_thread.start() + leader_started.wait(timeout=5) + follower_body = _consume( + stream_cached_response( + base, + forbidden_source, + redis=redis, + poll_interval=0.01, + ) + ) + leader_thread.join(timeout=5) + + self.assertEqual(follower_body, "ab") + self.assertEqual(rebuild_calls, [1]) + + def test_only_one_leader_when_two_clients_start_together(self): + redis = FakeRedis() + build_calls = [] + barrier = threading.Barrier(2) + results = {} + + def source(): + build_calls.append(threading.current_thread().name) + yield "x" + + def worker(): + barrier.wait() + results[threading.current_thread().name] = _consume( + stream_cached_response( + "cache:race", + source, + redis=redis, + poll_interval=0.01, + ) + ) + + threads = [ + threading.Thread(target=worker, name="t1"), + threading.Thread(target=worker, name="t2"), + ] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=10) + + self.assertEqual(results["t1"], "x") + self.assertEqual(results["t2"], "x") + self.assertEqual(len(build_calls), 1) diff --git a/apps/output/tests.py b/apps/output/tests.py index b1c7d589..e713df21 100644 --- a/apps/output/tests.py +++ b/apps/output/tests.py @@ -1,31 +1,117 @@ -from django.test import TestCase, Client +from django.test import TestCase, Client, SimpleTestCase from django.urls import reverse -from apps.channels.models import Channel, ChannelGroup +from unittest.mock import patch +from uuid import uuid4 +from apps.channels.models import Channel, ChannelGroup, ChannelProfile, ChannelProfileMembership from apps.epg.models import EPGData, EPGSource import xml.etree.ElementTree as ET +from datetime import timedelta + + +def _response_text(response): + """Read body from HttpResponse or StreamingHttpResponse.""" + if getattr(response, "streaming", False): + return b"".join(response.streaming_content).decode() + return response.content.decode() + + +def _epg_response_without_redis(cache_key, source, **kwargs): + """Test helper: stream EPG directly without Redis chunk caching.""" + from django.http import StreamingHttpResponse + + response = StreamingHttpResponse(source(), content_type="application/xml") + response["Content-Disposition"] = 'attachment; filename="Dispatcharr.xml"' + response["Cache-Control"] = "no-cache" + return response + + +class OutputEndpointTestMixin: + """Isolate HTTP endpoint tests from network ACL, logging, DB teardown, and Redis.""" -class OutputM3UTest(TestCase): def setUp(self): + super().setUp() + self._network_patch = patch( + "apps.output.views.network_access_allowed", + return_value=True, + ) + self._epg_teardown_patch = patch("apps.output.epg._epg_export_teardown") + self._log_event_patch = patch("apps.output.views.log_system_event") + self._epg_log_event_patch = patch("apps.output.epg.log_system_event") + self._close_db_patch = patch("django.db.close_old_connections") + self._epg_cache_patch = patch( + "apps.output.epg.stream_cached_response", + side_effect=_epg_response_without_redis, + ) + self._network_patch.start() + self._epg_teardown_patch.start() + self._log_event_patch.start() + self._epg_log_event_patch.start() + self._close_db_patch.start() + self._epg_cache_patch.start() + + def tearDown(self): + from django.core.cache import cache + + cache.clear() + self._epg_cache_patch.stop() + self._close_db_patch.stop() + self._epg_log_event_patch.stop() + self._log_event_patch.stop() + self._epg_teardown_patch.stop() + self._network_patch.stop() + super().tearDown() + + def _create_isolated_profile(self, prefix): + """New profiles auto-include every channel via signal; clear that for tests.""" + profile = ChannelProfile.objects.create(name=f"{prefix}-{uuid4().hex[:8]}") + ChannelProfileMembership.objects.filter(channel_profile=profile).delete() + return profile + + def _add_channel_to_profile(self, profile, group, **kwargs): + channel = Channel.objects.create(channel_group=group, **kwargs) + ChannelProfileMembership.objects.create( + channel_profile=profile, + channel=channel, + enabled=True, + ) + return channel + + +class OutputM3UTest(OutputEndpointTestMixin, TestCase): + def setUp(self): + super().setUp() self.client = Client() - + self.group = ChannelGroup.objects.create(name=f"M3U Group {uuid4().hex[:8]}") + self.profile = self._create_isolated_profile("m3u") + self._add_channel_to_profile( + self.profile, + self.group, + channel_number=1.0, + name="Test M3U Channel", + ) + + def _m3u_url(self): + return reverse("output:m3u_endpoint", kwargs={"profile_name": self.profile.name}) + def test_generate_m3u_response(self): """ Test that the M3U endpoint returns a valid M3U file. """ - url = reverse('output:generate_m3u') - response = self.client.get(url) + response = self.client.get(self._m3u_url()) self.assertEqual(response.status_code, 200) - content = response.content.decode() + content = _response_text(response) self.assertIn("#EXTM3U", content) def test_generate_m3u_response_post_empty_body(self): """ Test that a POST request with an empty body returns 200 OK. """ - url = reverse('output:generate_m3u') - - response = self.client.post(url, data=None, content_type='application/x-www-form-urlencoded') - content = response.content.decode() + response = self.client.post( + self._m3u_url(), + data=None, + content_type="application/x-www-form-urlencoded", + ) + content = _response_text(response) self.assertEqual(response.status_code, 200, "POST with empty body should return 200 OK") self.assertIn("#EXTM3U", content) @@ -34,35 +120,40 @@ class OutputM3UTest(TestCase): """ Test that a POST request with a non-empty body returns 403 Forbidden. """ - url = reverse('output:generate_m3u') - - response = self.client.post(url, data={'evilstring': 'muhahaha'}) + response = self.client.post(self._m3u_url(), data={"evilstring": "muhahaha"}) self.assertEqual(response.status_code, 403, "POST with body should return 403 Forbidden") - self.assertIn("POST requests with body are not allowed, body is:", response.content.decode()) + self.assertIn("POST requests with body are not allowed", _response_text(response)) -class OutputEPGXMLEscapingTest(TestCase): +class OutputEPGXMLEscapingTest(OutputEndpointTestMixin, TestCase): """Test XML escaping of channel_id attributes in EPG generation""" def setUp(self): + super().setUp() self.client = Client() - self.group = ChannelGroup.objects.create(name="Test Group") + self.group = ChannelGroup.objects.create(name=f"Test Group {uuid4().hex[:8]}") + self.profile = self._create_isolated_profile("epg-xml") + + def _add_channel(self, **kwargs): + return self._add_channel_to_profile(self.profile, self.group, **kwargs) + + def _epg_url(self, query="tvg_id_source=tvg_id&days=0&prev_days=0"): + base = reverse("output:epg_endpoint", kwargs={"profile_name": self.profile.name}) + return f"{base}?{query}" def test_channel_id_with_ampersand(self): """Test channel ID with ampersand is properly escaped""" - channel = Channel.objects.create( + self._add_channel( channel_number=1.0, name="Test Channel", tvg_id="News & Sports", - channel_group=self.group ) - url = reverse('output:generate_epg') + '?tvg_id_source=tvg_id' - response = self.client.get(url) + response = self.client.get(self._epg_url()) self.assertEqual(response.status_code, 200) - content = response.content.decode() + content = _response_text(response) # Should contain escaped ampersand self.assertIn('id="News & Sports"', content) @@ -76,17 +167,15 @@ class OutputEPGXMLEscapingTest(TestCase): def test_channel_id_with_angle_brackets(self): """Test channel ID with < and > characters""" - channel = Channel.objects.create( + self._add_channel( channel_number=2.0, name="HD Channel", tvg_id="Channel ", - channel_group=self.group ) - url = reverse('output:generate_epg') + '?tvg_id_source=tvg_id' - response = self.client.get(url) + response = self.client.get(self._epg_url()) - content = response.content.decode() + content = _response_text(response) self.assertIn('id="Channel <HD>"', content) try: @@ -96,23 +185,28 @@ class OutputEPGXMLEscapingTest(TestCase): def test_channel_id_with_all_special_chars(self): """Test channel ID with all XML special characters""" - channel = Channel.objects.create( + expected_id = 'Test & "Special" ' + self._add_channel( channel_number=3.0, name="Complex Channel", - tvg_id='Test & "Special" ', - channel_group=self.group + tvg_id=expected_id, ) - url = reverse('output:generate_epg') + '?tvg_id_source=tvg_id' - response = self.client.get(url) + response = self.client.get(self._epg_url()) - content = response.content.decode() + content = _response_text(response) self.assertIn('id="Test & "Special" <Chars>"', content) try: tree = ET.fromstring(content) - # Verify we can find the channel with correct ID in parsed tree - channel_elem = tree.find('.//channel[@id="Test & \\"Special\\" "]') + channel_elem = next( + ( + elem + for elem in tree.findall(".//channel") + if elem.get("id") == expected_id + ), + None, + ) self.assertIsNotNone(channel_elem) except ET.ParseError as e: self.fail(f"Generated EPG with all special chars is not valid XML: {e}") @@ -121,25 +215,203 @@ class OutputEPGXMLEscapingTest(TestCase): """Test that programme elements also have escaped channel attributes""" epg_source = EPGSource.objects.create(name="Test EPG", source_type="dummy") epg_data = EPGData.objects.create(name="Test EPG Data", epg_source=epg_source) - channel = Channel.objects.create( + self._add_channel( channel_number=4.0, name="Program Test", tvg_id="News & Sports", epg_data=epg_data, - channel_group=self.group ) - url = reverse('output:generate_epg') + '?tvg_id_source=tvg_id' - response = self.client.get(url) + response = self.client.get(self._epg_url()) - content = response.content.decode() + content = _response_text(response) # Check programme elements have escaped channel attributes self.assertIn('channel="News & Sports"', content) try: tree = ET.fromstring(content) - programmes = tree.findall('.//programme[@channel="News & Sports"]') + programmes = [ + programme + for programme in tree.findall(".//programme") + if programme.get("channel") == "News & Sports" + ] self.assertGreater(len(programmes), 0) except ET.ParseError as e: self.fail(f"Generated EPG with programme elements is not valid XML: {e}") + + def test_programmes_emitted_in_start_time_order(self): + """Programmes for a channel are emitted in start_time order, not insert order.""" + from django.utils import timezone + from apps.epg.models import ProgramData + + epg_source = EPGSource.objects.create(name="Real EPG", source_type="xmltv") + epg_data = EPGData.objects.create(name="Station", epg_source=epg_source, tvg_id="station1") + self._add_channel( + channel_number=149.0, + name="Food Network", + tvg_id="station1", + epg_data=epg_data, + ) + now = timezone.now() + # Insert out of chronological order so id order != start_time order. + ProgramData.objects.create( + epg=epg_data, + start_time=now + timedelta(days=3), + end_time=now + timedelta(days=3, hours=1), + title="Third", + tvg_id="station1", + ) + ProgramData.objects.create( + epg=epg_data, + start_time=now + timedelta(days=1), + end_time=now + timedelta(days=1, hours=1), + title="First", + tvg_id="station1", + ) + ProgramData.objects.create( + epg=epg_data, + start_time=now + timedelta(days=2), + end_time=now + timedelta(days=2, hours=1), + title="Second", + tvg_id="station1", + ) + + content = _response_text(self.client.get(self._epg_url("tvg_id_source=tvg_id&days=7"))) + + self.assertLess(content.find('First'), content.find('Second')) + self.assertLess(content.find('Second'), content.find('Third')) + + +class OutputEPGCustomDummyTest(TestCase): + """Custom dummy EPG must not fall back to default when pattern matched but event is outside window.""" + + def setUp(self): + self.group = ChannelGroup.objects.create(name="Sports Group") + + def test_custom_dummy_outside_window_fills_with_ended_programmes(self): + from django.utils import timezone + from apps.output.views import generate_dummy_programs + + epg_source = EPGSource.objects.create( + name="NHL Dummy", + source_type="dummy", + custom_properties={ + "title_pattern": r"(?.*)\s\d+:\s(?.*?)(?:\s+vs\s+)(?.*?)\s*@.*", + "time_pattern": r"(?\d{1,2}):(?\d{2})\s*(?AM|PM)", + "date_pattern": r"@ (?[A-Za-z]+)\s+(?\d{1,2})", + "timezone": "US/Eastern", + "program_duration": 180, + }, + ) + channel_name = ( + "NHL 01: Washington Capitals vs Philadelphia Flyers @ April 16 07:30 PM ET" + ) + now = timezone.now() + lookback = now - timedelta(days=7) + + programs = generate_dummy_programs( + channel_id="nhl01", + channel_name=channel_name, + num_days=7, + epg_source=epg_source, + export_lookback=lookback, + export_cutoff=now + timedelta(days=7), + ) + + self.assertGreater(len(programs), 0) + self.assertTrue( + all(p['end_time'] >= lookback for p in programs), + "All programmes should fall inside the export window", + ) + self.assertTrue( + any('Ended' in p['description'] for p in programs), + "Past events outside the window should still show ended filler", + ) + for program in programs: + start = program['start_time'] + self.assertEqual(start.second, 0) + self.assertEqual(start.microsecond, 0) + self.assertIn( + start.minute, (0, 30), + "Filler programmes should start on half-hour boundaries", + ) + self.assertGreaterEqual(programs[0]['start_time'], lookback) + + def test_custom_dummy_future_event_fills_grid_window_with_upcoming(self): + """Grid-style window: future event should show upcoming filler, not empty.""" + from django.utils import timezone + from apps.output.epg import _programme_overlaps_export_window, generate_dummy_programs + + epg_source = EPGSource.objects.create( + name="NHL Dummy Future", + source_type="dummy", + custom_properties={ + "title_pattern": r"(?.*)\s\d+:\s(?.*?)(?:\s+vs\s+)(?.*?)\s*@.*", + "time_pattern": r"(?\d{1,2}):(?\d{2})\s*(?AM|PM)", + "date_pattern": r"@ (?[A-Za-z]+)\s+(?\d{1,2})", + "timezone": "US/Eastern", + "program_duration": 180, + }, + ) + now = timezone.now() + grid_start = now - timedelta(hours=1) + grid_end = now + timedelta(hours=24) + future = now + timedelta(days=3) + channel_name = ( + f"NHL 01: Washington Capitals vs Philadelphia Flyers @ " + f"{future.strftime('%B')} {future.day} 07:30 PM ET" + ) + + programs = generate_dummy_programs( + channel_id="nhl01", + channel_name=channel_name, + num_days=1, + epg_source=epg_source, + export_lookback=grid_start, + export_cutoff=grid_end, + ) + + self.assertGreater(len(programs), 0) + self.assertTrue( + all( + _programme_overlaps_export_window( + p["start_time"], p["end_time"], grid_start, grid_end + ) + for p in programs + ), + "All programmes should overlap the grid query window", + ) + self.assertTrue( + any("Upcoming" in p.get("description", "") for p in programs), + "Future events outside the window should show upcoming filler", + ) + + +class OutputEPGHelperTest(SimpleTestCase): + def test_ceil_to_half_hour_on_boundary(self): + from django.utils import timezone + from apps.output.epg import _ceil_to_half_hour + + dt = timezone.now().replace(minute=30, second=0, microsecond=0) + self.assertEqual(_ceil_to_half_hour(dt), dt) + + def test_ceil_to_half_hour_rounds_up(self): + from django.utils import timezone + from apps.output.epg import _ceil_to_half_hour + + dt = timezone.now().replace(minute=17, second=42, microsecond=123456) + aligned = _ceil_to_half_hour(dt) + self.assertEqual(aligned.minute, 30) + self.assertEqual(aligned.second, 0) + self.assertGreaterEqual(aligned, dt.replace(microsecond=0)) + + def test_ceil_to_half_hour_past_boundary_second(self): + from django.utils import timezone + from apps.output.epg import _ceil_to_half_hour + + dt = timezone.now().replace(minute=0, second=52, microsecond=123456) + aligned = _ceil_to_half_hour(dt) + self.assertEqual(aligned.minute, 30) + self.assertEqual(aligned.second, 0) + self.assertGreaterEqual(aligned, dt.replace(microsecond=0)) diff --git a/apps/output/views.py b/apps/output/views.py index dba35951..4224fb35 100644 --- a/apps/output/views.py +++ b/apps/output/views.py @@ -25,9 +25,11 @@ from apps.proxy.utils import get_user_active_connections import regex from core.utils import log_system_event, build_absolute_uri_with_port import hashlib +from apps.output.epg import generate_epg, generate_dummy_programs logger = logging.getLogger(__name__) + def get_client_identifier(request): """Get client information including IP, user agent, and a unique hash identifier @@ -112,6 +114,10 @@ def generate_m3u(request, profile_name=None, user=None): # Check if this is a POST request and the body is not empty (which we don't want to allow) logger.debug("Generating M3U for profile: %s, user: %s, method: %s", profile_name, user.username if user else "Anonymous", request.method) + if request.method == "POST" and request.body: + if request.body.decode() != '{}': + return HttpResponseForbidden("POST requests with body are not allowed.") + # Check cache for recent identical request (helps with double-GET from browsers) from django.core.cache import cache cache_params = f"{profile_name or 'all'}:{user.username if user else 'anonymous'}:{request.GET.urlencode()}" @@ -123,10 +129,6 @@ def generate_m3u(request, profile_name=None, user=None): response = HttpResponse(cached_content, content_type="audio/x-mpegurl") response["Content-Disposition"] = 'attachment; filename="channels.m3u"' return response - # Check if this is a POST request with data (which we don't want to allow) - if request.method == "POST" and request.body: - if request.body.decode() != '{}': - return HttpResponseForbidden("POST requests with body are not allowed.") if user is not None: if user.user_level < 10: @@ -338,1663 +340,6 @@ def generate_m3u(request, profile_name=None, user=None): return response -def _ordered_channel_streams(channel): - """Return a channel's streams ordered by channelstream join order.""" - prefetched = getattr(channel, '_prefetched_objects_cache', {}).get('streams') - if prefetched is not None: - return list(prefetched) - return list(channel.streams.all().order_by('channelstream__order')) - - -def _pattern_match_name_from_custom_props(channel, effective_name, custom_props): - """Name used for custom dummy EPG regex matching (channel or stream title). - - Returns (name, stream_lookup_failed). stream_lookup_failed is True only when - name_source is 'stream' but the configured index is missing or out of range. - """ - if custom_props.get('name_source') != 'stream': - return effective_name, False - stream_index = custom_props.get('stream_index', 1) - 1 - streams = _ordered_channel_streams(channel) - if 0 <= stream_index < len(streams): - return streams[stream_index].name, False - return effective_name, True - - -def generate_fallback_programs(channel_id, channel_name, now, num_days, program_length_hours, fallback_title, fallback_description): - """ - Generate dummy programs using custom fallback templates when patterns don't match. - - Args: - channel_id: Channel ID for the programs - channel_name: Channel name to use as fallback in templates - now: Current datetime (in UTC) - num_days: Number of days to generate programs for - program_length_hours: Length of each program in hours - fallback_title: Custom fallback title template (empty string if not provided) - fallback_description: Custom fallback description template (empty string if not provided) - - Returns: - List of program dictionaries - """ - programs = [] - - # Use custom fallback title or channel name as default - title = fallback_title if fallback_title else channel_name - - # Use custom fallback description or a simple default message - if fallback_description: - description = fallback_description - else: - description = f"EPG information is currently unavailable for {channel_name}" - - # Create programs for each day - for day in range(num_days): - day_start = now + timedelta(days=day) - - # Create programs with specified length throughout the day - for hour_offset in range(0, 24, program_length_hours): - # Calculate program start and end times - start_time = day_start + timedelta(hours=hour_offset) - end_time = start_time + timedelta(hours=program_length_hours) - - programs.append({ - "channel_id": channel_id, - "start_time": start_time, - "end_time": end_time, - "title": title, - "description": description, - }) - - return programs - - -def generate_dummy_programs(channel_id, channel_name, num_days=1, program_length_hours=4, epg_source=None): - """ - Generate dummy EPG programs for channels. - - If epg_source is provided and it's a custom dummy EPG with patterns, - use those patterns to generate programs from the channel title. - Otherwise, generate default dummy programs. - - Args: - channel_id: Channel ID for the programs - channel_name: Channel title/name - num_days: Number of days to generate programs for - program_length_hours: Length of each program in hours - epg_source: Optional EPGSource for custom dummy EPG with patterns - - Returns: - List of program dictionaries - """ - # Get current time rounded to hour - now = django_timezone.now() - now = now.replace(minute=0, second=0, microsecond=0) - - # Check if this is a custom dummy EPG with regex patterns - if epg_source and epg_source.source_type == 'dummy' and epg_source.custom_properties: - custom_programs = generate_custom_dummy_programs( - channel_id, channel_name, now, num_days, - epg_source.custom_properties - ) - # If custom generation succeeded, return those programs - # If it returned empty (pattern didn't match), check for custom fallback templates - if custom_programs: - return custom_programs - else: - logger.info(f"Custom pattern didn't match for '{channel_name}', checking for custom fallback templates") - - # Check if custom fallback templates are provided - custom_props = epg_source.custom_properties - fallback_title = custom_props.get('fallback_title_template', '').strip() - fallback_description = custom_props.get('fallback_description_template', '').strip() - - # If custom fallback templates exist, use them instead of default - if fallback_title or fallback_description: - logger.info(f"Using custom fallback templates for '{channel_name}'") - return generate_fallback_programs( - channel_id, channel_name, now, num_days, - program_length_hours, fallback_title, fallback_description - ) - else: - logger.info(f"No custom fallback templates found, using default dummy EPG") - - # Default humorous program descriptions based on time of day - time_descriptions = { - (0, 4): [ - f"Late Night with {channel_name} - Where insomniacs unite!", - f"The 'Why Am I Still Awake?' Show on {channel_name}", - f"Counting Sheep - A {channel_name} production for the sleepless", - ], - (4, 8): [ - f"Dawn Patrol - Rise and shine with {channel_name}!", - f"Early Bird Special - Coffee not included", - f"Morning Zombies - Before coffee viewing on {channel_name}", - ], - (8, 12): [ - f"Mid-Morning Meetings - Pretend you're paying attention while watching {channel_name}", - f"The 'I Should Be Working' Hour on {channel_name}", - f"Productivity Killer - {channel_name}'s daytime programming", - ], - (12, 16): [ - f"Lunchtime Laziness with {channel_name}", - f"The Afternoon Slump - Brought to you by {channel_name}", - f"Post-Lunch Food Coma Theater on {channel_name}", - ], - (16, 20): [ - f"Rush Hour - {channel_name}'s alternative to traffic", - f"The 'What's For Dinner?' Debate on {channel_name}", - f"Evening Escapism - {channel_name}'s remedy for reality", - ], - (20, 24): [ - f"Prime Time Placeholder - {channel_name}'s finest not-programming", - f"The 'Netflix Was Too Complicated' Show on {channel_name}", - f"Family Argument Avoider - Courtesy of {channel_name}", - ], - } - - programs = [] - - # Create programs for each day - for day in range(num_days): - day_start = now + timedelta(days=day) - - # Create programs with specified length throughout the day - for hour_offset in range(0, 24, program_length_hours): - # Calculate program start and end times - start_time = day_start + timedelta(hours=hour_offset) - end_time = start_time + timedelta(hours=program_length_hours) - - # Get the hour for selecting a description - hour = start_time.hour - - # Find the appropriate time slot for description - for time_range, descriptions in time_descriptions.items(): - start_range, end_range = time_range - if start_range <= hour < end_range: - # Pick a description using the sum of the hour and day as seed - # This makes it somewhat random but consistent for the same timeslot - description = descriptions[(hour + day) % len(descriptions)] - break - else: - # Fallback description if somehow no range matches - description = f"Placeholder program for {channel_name} - EPG data went on vacation" - - programs.append({ - "channel_id": channel_id, - "start_time": start_time, - "end_time": end_time, - "title": channel_name, - "description": description, - }) - - return programs - - -def generate_custom_dummy_programs(channel_id, channel_name, now, num_days, custom_properties): - """ - Generate programs using custom dummy EPG regex patterns. - - Extracts information from channel title using regex patterns and generates - programs based on the extracted data. - - TIMEZONE HANDLING: - ------------------ - The timezone parameter specifies the timezone of the event times in your channel - titles using standard timezone names (e.g., 'US/Eastern', 'US/Pacific', 'Europe/London'). - DST (Daylight Saving Time) is handled automatically by pytz. - - Examples: - - Channel: "NHL 01: Bruins VS Maple Leafs @ 8:00PM ET" - - Set timezone = "US/Eastern" - - In October (DST): 8:00PM EDT → 12:00AM UTC (automatically uses UTC-4) - - In January (no DST): 8:00PM EST → 1:00AM UTC (automatically uses UTC-5) - - Args: - channel_id: Channel ID for the programs - channel_name: Channel title to parse - now: Current datetime (in UTC) - num_days: Number of days to generate programs for - custom_properties: Dict with title_pattern, time_pattern, templates, etc. - - timezone: Timezone name (e.g., 'US/Eastern') - - Returns: - List of program dictionaries with start_time/end_time in UTC - """ - import pytz - - logger.info(f"Generating custom dummy programs for channel: {channel_name}") - - # Extract patterns from custom properties - title_pattern = custom_properties.get('title_pattern', '') - time_pattern = custom_properties.get('time_pattern', '') - date_pattern = custom_properties.get('date_pattern', '') - - # Get timezone name (e.g., 'US/Eastern', 'US/Pacific', 'Europe/London') - timezone_value = custom_properties.get('timezone', 'UTC') - output_timezone_value = custom_properties.get('output_timezone', '') # Optional: display times in different timezone - program_duration = custom_properties.get('program_duration', 180) # Minutes - title_template = custom_properties.get('title_template', '') - subtitle_template = custom_properties.get('subtitle_template', '') - description_template = custom_properties.get('description_template', '') - - # Templates for upcoming/ended programs - upcoming_title_template = custom_properties.get('upcoming_title_template', '') - upcoming_description_template = custom_properties.get('upcoming_description_template', '') - ended_title_template = custom_properties.get('ended_title_template', '') - ended_description_template = custom_properties.get('ended_description_template', '') - - # Image URL templates - channel_logo_url_template = custom_properties.get('channel_logo_url', '') - program_poster_url_template = custom_properties.get('program_poster_url', '') - - # EPG metadata options - category_string = custom_properties.get('category', '') - # Split comma-separated categories and strip whitespace, filter out empty strings - categories = [cat.strip() for cat in category_string.split(',') if cat.strip()] if category_string else [] - include_date = custom_properties.get('include_date', True) - include_live = custom_properties.get('include_live', False) - include_new = custom_properties.get('include_new', False) - - # Parse timezone name - try: - source_tz = pytz.timezone(timezone_value) - logger.debug(f"Using timezone: {timezone_value} (DST will be handled automatically)") - except pytz.exceptions.UnknownTimeZoneError: - logger.warning(f"Unknown timezone: {timezone_value}, defaulting to UTC") - source_tz = pytz.utc - - # Parse output timezone if provided (for display purposes) - output_tz = None - if output_timezone_value: - try: - output_tz = pytz.timezone(output_timezone_value) - logger.debug(f"Using output timezone for display: {output_timezone_value}") - except pytz.exceptions.UnknownTimeZoneError: - logger.warning(f"Unknown output timezone: {output_timezone_value}, will use source timezone") - output_tz = None - - if not title_pattern: - logger.warning(f"No title_pattern in custom_properties, falling back to default") - return [] # Return empty, will use default - - logger.debug(f"Title pattern from DB: {repr(title_pattern)}") - - # Convert PCRE/JavaScript named groups (?) to Python format (?P) - # This handles patterns created with JavaScript regex syntax - # Use negative lookahead to avoid matching lookbehind (?<=) and negative lookbehind (?]+)>', r'(?P<\1>', title_pattern) - logger.debug(f"Converted title pattern: {repr(title_pattern)}") - - # Compile regex patterns using the enhanced regex module - # (supports variable-width lookbehinds like JavaScript) - try: - title_regex = regex.compile(title_pattern) - except Exception as e: - logger.error(f"Invalid title regex pattern after conversion: {e}") - logger.error(f"Pattern was: {repr(title_pattern)}") - return [] - - time_regex = None - if time_pattern: - # Convert PCRE/JavaScript named groups to Python format - # Use negative lookahead to avoid matching lookbehind (?<=) and negative lookbehind (?]+)>', r'(?P<\1>', time_pattern) - logger.debug(f"Converted time pattern: {repr(time_pattern)}") - try: - time_regex = regex.compile(time_pattern) - except Exception as e: - logger.warning(f"Invalid time regex pattern after conversion: {e}") - logger.warning(f"Pattern was: {repr(time_pattern)}") - - # Compile date regex if provided - date_regex = None - if date_pattern: - # Convert PCRE/JavaScript named groups to Python format - # Use negative lookahead to avoid matching lookbehind (?<=) and negative lookbehind (?]+)>', r'(?P<\1>', date_pattern) - logger.debug(f"Converted date pattern: {repr(date_pattern)}") - try: - date_regex = regex.compile(date_pattern) - except Exception as e: - logger.warning(f"Invalid date regex pattern after conversion: {e}") - logger.warning(f"Pattern was: {repr(date_pattern)}") - - # Try to match the channel name with the title pattern - # Use search() instead of match() to match JavaScript behavior where .match() searches anywhere in the string - title_match = title_regex.search(channel_name) - if not title_match: - logger.debug(f"Channel name '{channel_name}' doesn't match title pattern") - return [] # Return empty, will use default - - groups = title_match.groupdict() - logger.debug(f"Title pattern matched. Groups: {groups}") - - # Helper function to format template with matched groups - def format_template(template, groups, url_encode=False): - """Replace {groupname} placeholders with matched group values - - Args: - template: Template string with {groupname} placeholders - groups: Dict of group names to values - url_encode: If True, URL encode the group values for safe use in URLs - """ - if not template: - return '' - result = template - for key, value in groups.items(): - if url_encode and value: - # URL encode the value to handle spaces and special characters - from urllib.parse import quote - encoded_value = quote(str(value), safe='') - result = result.replace(f'{{{key}}}', encoded_value) - else: - result = result.replace(f'{{{key}}}', str(value) if value else '') - return result - - # Extract time from title if time pattern exists - time_info = None - time_groups = {} - if time_regex: - time_match = time_regex.search(channel_name) - if time_match: - time_groups = time_match.groupdict() - try: - hour = int(time_groups.get('hour')) - # Handle optional minute group - could be None if not captured - minute_value = time_groups.get('minute') - minute = int(minute_value) if minute_value is not None else 0 - ampm = time_groups.get('ampm') - ampm = ampm.lower() if ampm else None - - # Determine if this is 12-hour or 24-hour format - if ampm in ('am', 'pm'): - # 12-hour format: convert to 24-hour - if ampm == 'pm' and hour != 12: - hour += 12 - elif ampm == 'am' and hour == 12: - hour = 0 - logger.debug(f"Extracted time (12-hour): {hour}:{minute:02d} {ampm}") - else: - # 24-hour format: hour is already in 24-hour format - # Validate that it's actually a 24-hour time (0-23) - if hour > 23: - logger.warning(f"Invalid 24-hour time: {hour}. Must be 0-23.") - hour = hour % 24 # Wrap around just in case - logger.debug(f"Extracted time (24-hour): {hour}:{minute:02d}") - - time_info = {'hour': hour, 'minute': minute} - except (ValueError, TypeError) as e: - logger.warning(f"Error parsing time: {e}") - - # Extract date from title if date pattern exists - date_info = None - date_groups = {} - if date_regex: - date_match = date_regex.search(channel_name) - if date_match: - date_groups = date_match.groupdict() - try: - # Support various date group names: month, day, year - month_str = date_groups.get('month', '') - day_str = date_groups.get('day', '') - year_str = date_groups.get('year', '') - - # Parse day - default to current day if empty or invalid - day = int(day_str) if day_str else now.day - - # Parse year - default to current year if empty or invalid (matches frontend behavior) - year = int(year_str) if year_str else now.year - - # Parse month - can be numeric (1-12) or text (Jan, January, etc.) - month = None - if month_str: - if month_str.isdigit(): - month = int(month_str) - else: - # Try to parse text month names - import calendar - month_str_lower = month_str.lower() - # Check full month names - for i, month_name in enumerate(calendar.month_name): - if month_name.lower() == month_str_lower: - month = i - break - # Check abbreviated month names if not found - if month is None: - for i, month_abbr in enumerate(calendar.month_abbr): - if month_abbr.lower() == month_str_lower: - month = i - break - - # Default to current month if not extracted or invalid - if month is None: - month = now.month - - if month and 1 <= month <= 12 and 1 <= day <= 31: - date_info = {'year': year, 'month': month, 'day': day} - logger.debug(f"Extracted date: {year}-{month:02d}-{day:02d}") - else: - logger.warning(f"Invalid date values: month={month}, day={day}, year={year}") - except (ValueError, TypeError) as e: - logger.warning(f"Error parsing date: {e}") - - # Merge title groups, time groups, and date groups for template formatting - all_groups = {**groups, **time_groups, **date_groups} - - # Add normalized versions of all groups for cleaner URLs - # These remove all non-alphanumeric characters and convert to lowercase - for key, value in list(all_groups.items()): - if value: - # Remove all non-alphanumeric characters (except spaces temporarily) - # then replace spaces with nothing, and convert to lowercase - normalized = regex.sub(r'[^a-zA-Z0-9\s]', '', str(value)) - normalized = regex.sub(r'\s+', '', normalized).lower() - all_groups[f'{key}_normalize'] = normalized - - # Format channel logo URL if template provided (with URL encoding) - channel_logo_url = None - if channel_logo_url_template: - channel_logo_url = format_template(channel_logo_url_template, all_groups, url_encode=True) - logger.debug(f"Formatted channel logo URL: {channel_logo_url}") - - # Format program poster URL if template provided (with URL encoding) - program_poster_url = None - if program_poster_url_template: - program_poster_url = format_template(program_poster_url_template, all_groups, url_encode=True) - logger.debug(f"Formatted program poster URL: {program_poster_url}") - - # Add formatted time strings for better display (handles minutes intelligently) - if time_info: - hour_24 = time_info['hour'] - minute = time_info['minute'] - - # Determine the base date to use for placeholders - # If date was extracted, use it; otherwise use current date - if date_info: - base_date = datetime(date_info['year'], date_info['month'], date_info['day']) - else: - base_date = datetime.now() - - # If output_timezone is specified, convert the display time to that timezone - if output_tz: - # Create a datetime in the source timezone using the base date - temp_date = source_tz.localize(base_date.replace(hour=hour_24, minute=minute, second=0, microsecond=0)) - # Convert to output timezone - temp_date_output = temp_date.astimezone(output_tz) - # Extract converted hour and minute for display - hour_24 = temp_date_output.hour - minute = temp_date_output.minute - logger.debug(f"Converted display time from {source_tz} to {output_tz}: {hour_24}:{minute:02d}") - - # Add date placeholders based on the OUTPUT timezone - # This ensures {date}, {month}, {day}, {year} reflect the converted timezone - all_groups['date'] = temp_date_output.strftime('%Y-%m-%d') - all_groups['month'] = str(temp_date_output.month) - all_groups['day'] = str(temp_date_output.day) - all_groups['year'] = str(temp_date_output.year) - logger.debug(f"Converted date placeholders to {output_tz}: {all_groups['date']}") - else: - # No output timezone conversion - use source timezone for date - # Create temp date to get proper date in source timezone using the base date - temp_date_source = source_tz.localize(base_date.replace(hour=hour_24, minute=minute, second=0, microsecond=0)) - all_groups['date'] = temp_date_source.strftime('%Y-%m-%d') - all_groups['month'] = str(temp_date_source.month) - all_groups['day'] = str(temp_date_source.day) - all_groups['year'] = str(temp_date_source.year) - - # Format 24-hour start time string - only include minutes if non-zero - if minute > 0: - all_groups['starttime24'] = f"{hour_24}:{minute:02d}" - else: - all_groups['starttime24'] = f"{hour_24:02d}:00" - - # Convert 24-hour to 12-hour format for {starttime} placeholder - # Note: hour_24 is ALWAYS in 24-hour format at this point (converted earlier if needed) - ampm = 'AM' if hour_24 < 12 else 'PM' - hour_12 = hour_24 - if hour_24 == 0: - hour_12 = 12 - elif hour_24 > 12: - hour_12 = hour_24 - 12 - - # Format 12-hour start time string - only include minutes if non-zero - if minute > 0: - all_groups['starttime'] = f"{hour_12}:{minute:02d} {ampm}" - else: - all_groups['starttime'] = f"{hour_12} {ampm}" - - # Format long version that always includes minutes (e.g., "9:00 PM" instead of "9 PM") - all_groups['starttime_long'] = f"{hour_12}:{minute:02d} {ampm}" - - # Calculate end time based on program duration - # Create a datetime for calculations - temp_start = datetime.now(source_tz).replace(hour=hour_24, minute=minute, second=0, microsecond=0) - temp_end = temp_start + timedelta(minutes=program_duration) - - # Extract end time components (already in correct timezone if output_tz was applied above) - end_hour_24 = temp_end.hour - end_minute = temp_end.minute - - # Format 24-hour end time string - only include minutes if non-zero - if end_minute > 0: - all_groups['endtime24'] = f"{end_hour_24}:{end_minute:02d}" - else: - all_groups['endtime24'] = f"{end_hour_24:02d}:00" - - # Convert 24-hour to 12-hour format for {endtime} placeholder - end_ampm = 'AM' if end_hour_24 < 12 else 'PM' - end_hour_12 = end_hour_24 - if end_hour_24 == 0: - end_hour_12 = 12 - elif end_hour_24 > 12: - end_hour_12 = end_hour_24 - 12 - - # Format 12-hour end time string - only include minutes if non-zero - if end_minute > 0: - all_groups['endtime'] = f"{end_hour_12}:{end_minute:02d} {end_ampm}" - else: - all_groups['endtime'] = f"{end_hour_12} {end_ampm}" - - # Format long version that always includes minutes (e.g., "9:00 PM" instead of "9 PM") - all_groups['endtime_long'] = f"{end_hour_12}:{end_minute:02d} {end_ampm}" - - # Generate programs - programs = [] - - # If we have extracted time AND date, the event happens on a SPECIFIC date - # If we have time but NO date, generate for multiple days (existing behavior) - # All other days and times show "Upcoming" before or "Ended" after - event_happened = False - - # Determine how many iterations we need - if date_info and time_info: - # Specific date extracted - only generate for that one date - iterations = 1 - logger.debug(f"Date extracted, generating single event for specific date") - else: - # No specific date - use num_days (existing behavior) - iterations = num_days - - for day in range(iterations): - # Start from current time (like standard dummy) instead of midnight - # This ensures programs appear in the guide's current viewing window - day_start = now + timedelta(days=day) - day_end = day_start + timedelta(days=1) - - if time_info: - # We have an extracted event time - this is when the MAIN event starts - # The extracted time is in the SOURCE timezone (e.g., 8PM ET) - # We need to convert it to UTC for storage - - # Determine which date to use - if date_info: - # Use the extracted date from the channel title - current_date = datetime( - date_info['year'], - date_info['month'], - date_info['day'] - ).date() - logger.debug(f"Using extracted date: {current_date}") - else: - # No date extracted, use day offset from current time in SOURCE timezone - # This ensures we calculate "today" in the event's timezone, not UTC - # For example: 8:30 PM Central (1:30 AM UTC next day) for a 10 PM ET event - # should use today's date in ET, not tomorrow's date in UTC - now_in_source_tz = now.astimezone(source_tz) - current_date = (now_in_source_tz + timedelta(days=day)).date() - logger.debug(f"No date extracted, using day offset in {source_tz}: {current_date}") - - # Create a naive datetime (no timezone info) representing the event in source timezone - event_start_naive = datetime.combine( - current_date, - datetime.min.time().replace( - hour=time_info['hour'], - minute=time_info['minute'] - ) - ) - - # Use pytz to localize the naive datetime to the source timezone - # This automatically handles DST! - try: - event_start_local = source_tz.localize(event_start_naive) - # Convert to UTC - event_start_utc = event_start_local.astimezone(pytz.utc) - logger.debug(f"Converted {event_start_local} to UTC: {event_start_utc}") - except Exception as e: - logger.error(f"Error localizing time to {source_tz}: {e}") - # Fallback: treat as UTC - event_start_utc = django_timezone.make_aware(event_start_naive, pytz.utc) - - event_end_utc = event_start_utc + timedelta(minutes=program_duration) - - # Pre-generate the main event title and description for reuse - if title_template: - main_event_title = format_template(title_template, all_groups) - else: - title_parts = [] - if 'league' in all_groups and all_groups['league']: - title_parts.append(all_groups['league']) - if 'team1' in all_groups and 'team2' in all_groups: - title_parts.append(f"{all_groups['team1']} vs {all_groups['team2']}") - elif 'title' in all_groups and all_groups['title']: - title_parts.append(all_groups['title']) - main_event_title = ' - '.join(title_parts) if title_parts else channel_name - - if subtitle_template: - main_event_subtitle = format_template(subtitle_template, all_groups) - else: - main_event_subtitle = None - - if description_template: - main_event_description = format_template(description_template, all_groups) - else: - main_event_description = main_event_title - - - - # Determine if this day is before, during, or after the event - # Event only happens on day 0 (first day) - is_event_day = (day == 0) - - if is_event_day and not event_happened: - # This is THE day the event happens - # Fill programs BEFORE the event - current_time = day_start - - while current_time < event_start_utc: - program_start_utc = current_time - program_end_utc = min(current_time + timedelta(minutes=program_duration), event_start_utc) - - # Use custom upcoming templates if provided, otherwise use defaults - if upcoming_title_template: - upcoming_title = format_template(upcoming_title_template, all_groups) - else: - upcoming_title = main_event_title - - if upcoming_description_template: - upcoming_description = format_template(upcoming_description_template, all_groups) - else: - upcoming_description = f"Upcoming: {main_event_description}" - - # Build custom_properties for upcoming programs (only date, no category/live) - program_custom_properties = {} - - # Add date if requested (YYYY-MM-DD format from start time in event timezone) - if include_date: - # Convert UTC time to event timezone for date calculation - local_time = program_start_utc.astimezone(source_tz) - date_str = local_time.strftime('%Y-%m-%d') - program_custom_properties['date'] = date_str - - # Add program poster URL if provided - if program_poster_url: - program_custom_properties['icon'] = program_poster_url - - programs.append({ - "channel_id": channel_id, - "start_time": program_start_utc, - "end_time": program_end_utc, - "title": upcoming_title, - "sub_title": None, # No subtitle for filler programs - "description": upcoming_description, - "custom_properties": program_custom_properties, - "channel_logo_url": channel_logo_url, # Pass channel logo for EPG generation - }) - - current_time += timedelta(minutes=program_duration) - - # Add the MAIN EVENT at the extracted time - # Build custom_properties for main event (includes category and live) - main_event_custom_properties = {} - - # Add categories if provided - if categories: - main_event_custom_properties['categories'] = categories - - # Add date if requested (YYYY-MM-DD format from start time in event timezone) - if include_date: - # Convert UTC time to event timezone for date calculation - local_time = event_start_utc.astimezone(source_tz) - date_str = local_time.strftime('%Y-%m-%d') - main_event_custom_properties['date'] = date_str - - # Add live flag if requested - if include_live: - main_event_custom_properties['live'] = True - - # Add new flag if requested - if include_new: - main_event_custom_properties['new'] = True - - # Add program poster URL if provided - if program_poster_url: - main_event_custom_properties['icon'] = program_poster_url - - programs.append({ - "channel_id": channel_id, - "start_time": event_start_utc, - "end_time": event_end_utc, - "title": main_event_title, - "sub_title": main_event_subtitle, - "description": main_event_description, - "custom_properties": main_event_custom_properties, - "channel_logo_url": channel_logo_url, # Pass channel logo for EPG generation - }) - - event_happened = True - - # Fill programs AFTER the event until end of day - current_time = event_end_utc - - while current_time < day_end: - program_start_utc = current_time - program_end_utc = min(current_time + timedelta(minutes=program_duration), day_end) - - # Use custom ended templates if provided, otherwise use defaults - if ended_title_template: - ended_title = format_template(ended_title_template, all_groups) - else: - ended_title = main_event_title - - if ended_description_template: - ended_description = format_template(ended_description_template, all_groups) - else: - ended_description = f"Ended: {main_event_description}" - - # Build custom_properties for ended programs (only date, no category/live) - program_custom_properties = {} - - # Add date if requested (YYYY-MM-DD format from start time in event timezone) - if include_date: - # Convert UTC time to event timezone for date calculation - local_time = program_start_utc.astimezone(source_tz) - date_str = local_time.strftime('%Y-%m-%d') - program_custom_properties['date'] = date_str - - # Add program poster URL if provided - if program_poster_url: - program_custom_properties['icon'] = program_poster_url - - programs.append({ - "channel_id": channel_id, - "start_time": program_start_utc, - "end_time": program_end_utc, - "title": ended_title, - "sub_title": None, # No subtitle for filler programs - "description": ended_description, - "custom_properties": program_custom_properties, - "channel_logo_url": channel_logo_url, # Pass channel logo for EPG generation - }) - - current_time += timedelta(minutes=program_duration) - else: - # This day is either before the event (future days) or after the event happened - # Fill entire day with appropriate message - current_time = day_start - - # If event already happened, all programs show "Ended" - # If event hasn't happened yet (shouldn't occur with day 0 logic), show "Upcoming" - is_ended = event_happened - - while current_time < day_end: - program_start_utc = current_time - program_end_utc = min(current_time + timedelta(minutes=program_duration), day_end) - - # Use custom templates based on whether event has ended or is upcoming - if is_ended: - if ended_title_template: - program_title = format_template(ended_title_template, all_groups) - else: - program_title = main_event_title - - if ended_description_template: - program_description = format_template(ended_description_template, all_groups) - else: - program_description = f"Ended: {main_event_description}" - else: - if upcoming_title_template: - program_title = format_template(upcoming_title_template, all_groups) - else: - program_title = main_event_title - - if upcoming_description_template: - program_description = format_template(upcoming_description_template, all_groups) - else: - program_description = f"Upcoming: {main_event_description}" - - # Build custom_properties (only date for upcoming/ended filler programs) - program_custom_properties = {} - - # Add date if requested (YYYY-MM-DD format from start time in event timezone) - if include_date: - # Convert UTC time to event timezone for date calculation - local_time = program_start_utc.astimezone(source_tz) - date_str = local_time.strftime('%Y-%m-%d') - program_custom_properties['date'] = date_str - - # Add program poster URL if provided - if program_poster_url: - program_custom_properties['icon'] = program_poster_url - - programs.append({ - "channel_id": channel_id, - "start_time": program_start_utc, - "end_time": program_end_utc, - "title": program_title, - "sub_title": None, # No subtitle for filler programs - "description": program_description, - "custom_properties": program_custom_properties, - "channel_logo_url": channel_logo_url, - }) - - current_time += timedelta(minutes=program_duration) - else: - # No extracted time - fill entire day with regular intervals - # day_start and day_end are already in UTC, so no conversion needed - programs_per_day = max(1, int(24 / (program_duration / 60))) - - for program_num in range(programs_per_day): - program_start_utc = day_start + timedelta(minutes=program_num * program_duration) - program_end_utc = program_start_utc + timedelta(minutes=program_duration) - - if title_template: - title = format_template(title_template, all_groups) - else: - title_parts = [] - if 'league' in all_groups and all_groups['league']: - title_parts.append(all_groups['league']) - if 'team1' in all_groups and 'team2' in all_groups: - title_parts.append(f"{all_groups['team1']} vs {all_groups['team2']}") - elif 'title' in all_groups and all_groups['title']: - title_parts.append(all_groups['title']) - title = ' - '.join(title_parts) if title_parts else channel_name - - if subtitle_template: - subtitle = format_template(subtitle_template, all_groups) - else: - subtitle = None - - if description_template: - description = format_template(description_template, all_groups) - else: - description = title - - # Build custom_properties for this program - program_custom_properties = {} - - # Add categories if provided - if categories: - program_custom_properties['categories'] = categories - - # Add date if requested (YYYY-MM-DD format from start time in event timezone) - if include_date: - # Convert UTC time to event timezone for date calculation - local_time = program_start_utc.astimezone(source_tz) - date_str = local_time.strftime('%Y-%m-%d') - program_custom_properties['date'] = date_str - - # Add live flag if requested - if include_live: - program_custom_properties['live'] = True - - # Add new flag if requested - if include_new: - program_custom_properties['new'] = True - - # Add program poster URL if provided - if program_poster_url: - program_custom_properties['icon'] = program_poster_url - - programs.append({ - "channel_id": channel_id, - "start_time": program_start_utc, - "end_time": program_end_utc, - "title": title, - "sub_title": subtitle, - "description": description, - "custom_properties": program_custom_properties, - "channel_logo_url": channel_logo_url, # Pass channel logo for EPG generation - }) - - logger.info(f"Generated {len(programs)} custom dummy programs for {channel_name}") - return programs - - -def generate_dummy_epg( - channel_id, channel_name, xml_lines=None, num_days=1, program_length_hours=4 -): - """ - Generate dummy EPG programs for channels without EPG data. - Creates program blocks for a specified number of days. - - Args: - channel_id: The channel ID to use in the program entries - channel_name: The name of the channel to use in program titles - xml_lines: Optional list to append lines to, otherwise returns new list - num_days: Number of days to generate EPG data for (default: 1) - program_length_hours: Length of each program block in hours (default: 4) - - Returns: - List of XML lines for the dummy EPG entries - """ - if xml_lines is None: - xml_lines = [] - - for program in generate_dummy_programs(channel_id, channel_name, num_days=1, program_length_hours=4): - # Format times in XMLTV format - start_str = program['start_time'].strftime("%Y%m%d%H%M%S %z") - stop_str = program['end_time'].strftime("%Y%m%d%H%M%S %z") - - # Create program entry with escaped channel name - xml_lines.append( - f' ' - ) - xml_lines.append(f" {html.escape(program['title'])}") - - # Add subtitle if available - if program.get('sub_title'): - xml_lines.append(f" {html.escape(program['sub_title'])}") - - xml_lines.append(f" {html.escape(program['description'])}") - - # Add custom_properties if present - custom_data = program.get('custom_properties', {}) - - # Categories - if 'categories' in custom_data: - for cat in custom_data['categories']: - xml_lines.append(f" {html.escape(cat)}") - - # Date tag - if 'date' in custom_data: - xml_lines.append(f" {html.escape(custom_data['date'])}") - - # Live tag - if custom_data.get('live', False): - xml_lines.append(f" ") - - # New tag - if custom_data.get('new', False): - xml_lines.append(f" ") - - xml_lines.append(f" ") - - return xml_lines - - -def generate_epg(request, profile_name=None, user=None): - """ - Dynamically generate an XMLTV (EPG) file using streaming response to handle keep-alives. - Since the EPG data is stored independently of Channels, we group programmes - by their associated EPGData record. - This version filters data based on the 'days' parameter and sends keep-alives during processing. - """ - # Check cache for recent identical request (helps with double-GET from browsers) - from django.core.cache import cache - # Resolve all effective parameter values once here so they are reused for both - # the cache key and inside epg_generator() via closure. - # The cache key is built from resolved values only — not from the raw query string — - # so equivalent requests (e.g. days=7 via URL param vs. user default of 7) share - # the same cache entry regardless of how the value was supplied. - user_custom = (user.custom_properties or {}) if user else {} - try: - num_days = int(request.GET.get('days', user_custom.get('epg_days', 0))) - num_days = max(0, min(num_days, 365)) - except (ValueError, TypeError): - num_days = 0 - try: - prev_days = int(request.GET.get('prev_days', user_custom.get('epg_prev_days', 0))) - prev_days = max(0, min(prev_days, 30)) - except (ValueError, TypeError): - prev_days = 0 - use_cached_logos = request.GET.get('cachedlogos', 'true').lower() != 'false' - tvg_id_source = request.GET.get('tvg_id_source', 'channel_number').lower() - cache_params = ( - f"{profile_name or 'all'}:{user.username if user else 'anonymous'}" - f":d={num_days}:p={prev_days}:logos={use_cached_logos}:tvgid={tvg_id_source}" - ) - content_cache_key = f"epg_content:{cache_params}" - - cached_content = cache.get(content_cache_key) - if cached_content: - logger.debug("Serving EPG from cache") - response = HttpResponse(cached_content, content_type="application/xml") - response["Content-Disposition"] = 'attachment; filename="Dispatcharr.xml"' - response["Cache-Control"] = "no-cache" - return response - - def epg_generator(): - """Generator function that yields EPG data with keep-alives during processing.""" - - xml_lines = [] - xml_lines.append('') - xml_lines.append( - '' - ) - - # Get channels based on user/profile - if user is not None: - if user.user_level < 10: - user_profile_count = user.channel_profiles.count() - - # If user has ALL profiles or NO profiles, give unrestricted access - if user_profile_count == 0: - # No profile filtering - user sees all channels based on user_level - filters = {"user_level__lte": user.user_level} - # Hide adult content if user preference is set - if (user.custom_properties or {}).get('hide_adult_content', False): - filters["is_adult"] = False - base_qs = Channel.objects.filter(**filters).select_related('logo', 'epg_data__epg_source') - else: - # User has specific limited profiles assigned - filters = { - "channelprofilemembership__enabled": True, - "user_level__lte": user.user_level, - "channelprofilemembership__channel_profile__in": user.channel_profiles.all() - } - # Hide adult content if user preference is set - if (user.custom_properties or {}).get('hide_adult_content', False): - filters["is_adult"] = False - base_qs = Channel.objects.filter(**filters).select_related('logo', 'epg_data__epg_source').distinct() - else: - base_qs = Channel.objects.filter(user_level__lte=user.user_level).select_related('logo', 'epg_data__epg_source') - else: - if profile_name is not None: - try: - channel_profile = ChannelProfile.objects.get(name=profile_name) - except ChannelProfile.DoesNotExist: - logger.warning("Requested channel profile (%s) during epg generation does not exist", profile_name) - raise Http404(f"Channel profile '{profile_name}' not found") - base_qs = Channel.objects.filter( - channelprofilemembership__channel_profile=channel_profile, - channelprofilemembership__enabled=True, - ).select_related('logo', 'epg_data__epg_source') - else: - base_qs = Channel.objects.all().select_related('logo', 'epg_data__epg_source') - - # Resolve effective values at SQL level and exclude hidden channels - # so output ordering/display honors user overrides. - from apps.channels.managers import with_effective_values - channels = ( - with_effective_values(base_qs, select_related_fks=True) - .exclude(hidden_from_output=True) - .order_by("effective_channel_number") - .prefetch_related( - Prefetch('streams', queryset=Stream.objects.order_by('channelstream__order')) - ) - ) - - # For dummy EPG, use either the specified value or default to 3 days - dummy_days = num_days if num_days > 0 else 3 - - # Calculate cutoff dates for EPG data filtering - now = django_timezone.now() - cutoff_date = now + timedelta(days=num_days) if num_days > 0 else None - lookback_cutoff = now - timedelta(days=prev_days) - - # Build collision-free channel number mapping for XC clients (if user is authenticated) - # XC clients require integer channel numbers, so we need to ensure no conflicts - channel_num_map = {} - if user is not None: - # This is an XC client - build collision-free mapping - used_numbers = set() - - # First pass: assign integers for channels that already have integer numbers - for channel in channels: - effective_num = channel.effective_channel_number - if effective_num is not None and effective_num == int(effective_num): - num = int(effective_num) - channel_num_map[channel.id] = num - used_numbers.add(num) - - # Second pass: assign integers for channels with float numbers - for channel in channels: - effective_num = channel.effective_channel_number - if effective_num is not None and effective_num != int(effective_num): - candidate = int(effective_num) - while candidate in used_numbers: - candidate += 1 - channel_num_map[channel.id] = candidate - used_numbers.add(candidate) - - # Host/port/scheme are constant per request; precompute logo URL prefix once. - _base_url = build_absolute_uri_with_port(request, "") - _sample_logo_path = reverse("api:channels:logo-cache", args=[0]) - _logo_prefix_raw, _, _logo_suffix_raw = _sample_logo_path.partition("/0/") - _logo_url_prefix = _base_url + _logo_prefix_raw + "/" - _logo_url_suffix = "/" + _logo_suffix_raw - - dummy_epg_ids_for_program_check = set() - - # Process channels for the section - for channel in channels: - effective_name = channel.effective_name - effective_epg_data = channel.effective_epg_data_obj - effective_logo = channel.effective_logo_obj - effective_number = channel.effective_channel_number - - # user is set only for XC clients, which require integer channel numbers - if user is not None: - formatted_channel_number = channel_num_map[channel.id] - else: - formatted_channel_number = format_channel_number(effective_number) - - # Determine the channel ID based on the selected source - if tvg_id_source == 'tvg_id' and channel.effective_tvg_id: - channel_id = channel.effective_tvg_id - elif tvg_id_source == 'gracenote' and channel.effective_tvc_guide_stationid: - channel_id = channel.effective_tvc_guide_stationid - else: - channel_id = str(formatted_channel_number) if formatted_channel_number != "" else str(channel.id) - - tvg_logo = "" - - # Check if this is a custom dummy EPG with channel logo URL template - if effective_epg_data and effective_epg_data.epg_source and effective_epg_data.epg_source.source_type == 'dummy': - if channel.effective_epg_data_id: - dummy_epg_ids_for_program_check.add(channel.effective_epg_data_id) - epg_source = effective_epg_data.epg_source - if epg_source.custom_properties: - custom_props = epg_source.custom_properties - channel_logo_url_template = custom_props.get('channel_logo_url', '') - - if channel_logo_url_template: - pattern_match_name, _ = _pattern_match_name_from_custom_props( - channel, effective_name, custom_props - ) - - # Try to extract groups from the channel/stream name and build the logo URL - title_pattern = custom_props.get('title_pattern', '') - if title_pattern: - try: - # Convert PCRE/JavaScript named groups to Python format - title_pattern = regex.sub(r'\(\?<(?![=!])([^>]+)>', r'(?P<\1>', title_pattern) - title_regex = regex.compile(title_pattern) - title_match = title_regex.search(pattern_match_name) - - if title_match: - groups = title_match.groupdict() - - # Add normalized versions of all groups for cleaner URLs - for key, value in list(groups.items()): - if value: - # Remove all non-alphanumeric characters and convert to lowercase - normalized = regex.sub(r'[^a-zA-Z0-9\s]', '', str(value)) - normalized = regex.sub(r'\s+', '', normalized).lower() - groups[f'{key}_normalize'] = normalized - - # Format the logo URL template with the matched groups (with URL encoding) - from urllib.parse import quote - for key, value in groups.items(): - if value: - encoded_value = quote(str(value), safe='') - channel_logo_url_template = channel_logo_url_template.replace(f'{{{key}}}', encoded_value) - else: - channel_logo_url_template = channel_logo_url_template.replace(f'{{{key}}}', '') - tvg_logo = channel_logo_url_template - logger.debug(f"Built channel logo URL from template: {tvg_logo}") - except Exception as e: - logger.warning(f"Failed to build channel logo URL for {effective_name}: {e}") - - # If no custom dummy logo, use regular logo logic - if not tvg_logo and effective_logo: - if use_cached_logos: - tvg_logo = f"{_logo_url_prefix}{effective_logo.id}{_logo_url_suffix}" - else: - # Use direct URL if available, otherwise fall back to cached version - direct_logo = effective_logo.url if effective_logo.url.startswith(('http://', 'https://')) else None - if direct_logo: - tvg_logo = direct_logo - else: - tvg_logo = f"{_logo_url_prefix}{effective_logo.id}{_logo_url_suffix}" - display_name = effective_name - xml_lines.append(f' ') - xml_lines.append(f' {html.escape(display_name)}') - xml_lines.append(f' ') - xml_lines.append(" ") - - # Send all channel definitions - channel_xml = '\n'.join(xml_lines) + '\n' - yield channel_xml - xml_lines = [] # Clear to save memory - - dummy_epg_with_programs = set() - if dummy_epg_ids_for_program_check: - dummy_epg_with_programs = set( - ProgramData.objects.filter(epg_id__in=dummy_epg_ids_for_program_check) - .values_list('epg_id', flat=True) - .distinct() - ) - - # Pre-pass: categorize channels into dummy and real EPG groups - dummy_program_list = [] # (channel_id, pattern_match_name, epg_source_or_None) - real_epg_map = {} # epg_data_id -> [channel_id, ...] - - for channel in channels: - effective_name = channel.effective_name - effective_epg_data = channel.effective_epg_data_obj - effective_epg_data_id = channel.effective_epg_data_id - effective_number = channel.effective_channel_number - - # Determine channel_id (same logic as channel section) - if tvg_id_source == 'tvg_id' and channel.effective_tvg_id: - channel_id = channel.effective_tvg_id - elif tvg_id_source == 'gracenote' and channel.effective_tvc_guide_stationid: - channel_id = channel.effective_tvc_guide_stationid - else: - if user is not None: - formatted_channel_number = channel_num_map[channel.id] - else: - formatted_channel_number = format_channel_number(effective_number) - channel_id = str(formatted_channel_number) if formatted_channel_number != "" else str(channel.id) - - display_name = effective_epg_data.name if effective_epg_data else effective_name - pattern_match_name = effective_name - - # Check if we should use stream name instead of channel name - if effective_epg_data and effective_epg_data.epg_source: - epg_source = effective_epg_data.epg_source - if epg_source.custom_properties: - custom_props = epg_source.custom_properties - pattern_match_name, stream_lookup_failed = _pattern_match_name_from_custom_props( - channel, effective_name, custom_props - ) - if ( - custom_props.get('name_source') == 'stream' - and not stream_lookup_failed - and pattern_match_name != effective_name - ): - stream_index = custom_props.get('stream_index', 1) - 1 - logger.debug( - f"Using stream name for parsing: {pattern_match_name} " - f"(stream index: {stream_index})" - ) - elif stream_lookup_failed: - stream_index = custom_props.get('stream_index', 1) - 1 - logger.warning( - f"Stream index {stream_index} not found for channel " - f"{effective_name}, falling back to channel name" - ) - - if not effective_epg_data: - dummy_program_list.append((channel_id, pattern_match_name, None)) - else: - if effective_epg_data.epg_source and effective_epg_data.epg_source.source_type == 'dummy': - if effective_epg_data_id in dummy_epg_with_programs: - real_epg_map.setdefault(effective_epg_data_id, []).append(channel_id) - else: - dummy_program_list.append((channel_id, pattern_match_name, effective_epg_data.epg_source)) - continue - - real_epg_map.setdefault(effective_epg_data_id, []).append(channel_id) - - # Emit dummy programmes - for channel_id, pattern_match_name, epg_source in dummy_program_list: - program_length_hours = 4 - dummy_programs = generate_dummy_programs( - channel_id, pattern_match_name, - num_days=dummy_days, - program_length_hours=program_length_hours, - epg_source=epg_source - ) - for program in dummy_programs: - start_str = program['start_time'].strftime("%Y%m%d%H%M%S %z") - stop_str = program['end_time'].strftime("%Y%m%d%H%M%S %z") - yield f' \n' - yield f" {html.escape(program['title'])}\n" - if program.get('sub_title'): - yield f" {html.escape(program['sub_title'])}\n" - yield f" {html.escape(program['description'])}\n" - custom_data = program.get('custom_properties', {}) - if 'categories' in custom_data: - for cat in custom_data['categories']: - yield f" {html.escape(cat)}\n" - if 'date' in custom_data: - yield f" {html.escape(custom_data['date'])}\n" - if custom_data.get('live', False): - yield f" \n" - if custom_data.get('new', False): - yield f" \n" - if 'icon' in custom_data: - yield f' \n' - yield f" \n" - - # Emit real programmes: single bulk query, chunked to avoid server-side cursor issues. - all_epg_ids = list(real_epg_map.keys()) - if all_epg_ids: - if num_days > 0: - programs_qs = ProgramData.objects.filter( - epg_id__in=all_epg_ids, - end_time__gte=lookback_cutoff, - start_time__lt=cutoff_date, - ) - else: - programs_qs = ProgramData.objects.filter( - epg_id__in=all_epg_ids, - end_time__gte=lookback_cutoff, - ) - - programs_base_qs = programs_qs.order_by('epg_id', 'id').values( - 'id', 'epg_id', 'start_time', 'end_time', 'title', 'sub_title', - 'description', 'custom_properties', - ) - - current_epg_id = None - channel_ids_for_epg = None - is_multi = False - multi_buffer = [] - program_batch = [] - batch_size = 1000 - chunk_size = 5000 - # Keyset pagination: track last (epg_id, id) instead of OFFSET - # to avoid skipping/duplicating rows if the table changes mid-stream. - last_epg_id = 0 - last_id = 0 - _poster_url_base = build_absolute_uri_with_port(request, "/api/epg/programs/") - - while True: - program_chunk = list( - programs_base_qs.filter(epg_id__gte=last_epg_id) - .exclude(epg_id=last_epg_id, id__lte=last_id)[:chunk_size] - ) - - if not program_chunk: - break - - # Advance keyset cursor to last row in this chunk - last_row = program_chunk[-1] - last_epg_id = last_row['epg_id'] - last_id = last_row['id'] - - for prog in program_chunk: - epg_id = prog['epg_id'] - - # When epg_id changes, flush multi-channel buffer for previous group - if epg_id != current_epg_id: - if is_multi and multi_buffer: - escaped_primary = html.escape(channel_ids_for_epg[0]) - for extra_cid in channel_ids_for_epg[1:]: - escaped_extra = html.escape(extra_cid) - for xml_text in multi_buffer: - program_batch.append(xml_text.replace( - f'channel="{escaped_primary}"', - f'channel="{escaped_extra}"', - 1, - )) - if len(program_batch) >= batch_size: - yield '\n'.join(program_batch) + '\n' - program_batch = [] - multi_buffer = [] - - current_epg_id = epg_id - channel_ids_for_epg = real_epg_map[epg_id] - is_multi = len(channel_ids_for_epg) > 1 - - # Build programme XML for primary channel_id - primary_cid = channel_ids_for_epg[0] - start_str = prog['start_time'].strftime("%Y%m%d%H%M%S %z") - stop_str = prog['end_time'].strftime("%Y%m%d%H%M%S %z") - - program_xml = [f' '] - program_xml.append(f' {html.escape(prog["title"])}') - - if prog['sub_title']: - program_xml.append(f" {html.escape(prog['sub_title'])}") - - if prog['description']: - program_xml.append(f" {html.escape(prog['description'])}") - - custom_data = prog['custom_properties'] or {} - if custom_data: - - if "categories" in custom_data and custom_data["categories"]: - for category in custom_data["categories"]: - program_xml.append(f" {html.escape(category)}") - - if "keywords" in custom_data and custom_data["keywords"]: - for keyword in custom_data["keywords"]: - program_xml.append(f" {html.escape(keyword)}") - - # onscreen_episode takes priority over episode for the onscreen system - if "onscreen_episode" in custom_data: - program_xml.append(f' {html.escape(custom_data["onscreen_episode"])}') - elif "episode" in custom_data: - program_xml.append(f' E{custom_data["episode"]}') - - # Handle dd_progid format - if 'dd_progid' in custom_data: - program_xml.append(f' {html.escape(custom_data["dd_progid"])}') - - # Handle external database IDs - for system in ['thetvdb.com', 'themoviedb.org', 'imdb.com']: - if f'{system}_id' in custom_data: - program_xml.append(f' {html.escape(custom_data[f"{system}_id"])}') - - # Add season and episode numbers in xmltv_ns format if available - if "season" in custom_data and "episode" in custom_data: - season = ( - int(custom_data["season"]) - 1 - if str(custom_data["season"]).isdigit() - else 0 - ) - episode = ( - int(custom_data["episode"]) - 1 - if str(custom_data["episode"]).isdigit() - else 0 - ) - program_xml.append(f' {season}.{episode}.') - - if "language" in custom_data: - program_xml.append(f' {html.escape(custom_data["language"])}') - - if "original_language" in custom_data: - program_xml.append(f' {html.escape(custom_data["original_language"])}') - - if "length" in custom_data and isinstance(custom_data["length"], dict): - length_value = custom_data["length"].get("value", "") - length_units = custom_data["length"].get("units", "minutes") - program_xml.append(f' {html.escape(str(length_value))}') - - if "video" in custom_data and isinstance(custom_data["video"], dict): - program_xml.append(" ") - - if "audio" in custom_data and isinstance(custom_data["audio"], dict): - program_xml.append(" ") - - if "subtitles" in custom_data and isinstance(custom_data["subtitles"], list): - for subtitle in custom_data["subtitles"]: - if isinstance(subtitle, dict): - subtitle_type = subtitle.get("type", "") - type_attr = f' type="{html.escape(subtitle_type)}"' if subtitle_type else "" - program_xml.append(f" ") - if "language" in subtitle: - program_xml.append(f" {html.escape(subtitle['language'])}") - program_xml.append(" ") - - if "rating" in custom_data: - rating_system = custom_data.get("rating_system", "TV Parental Guidelines") - program_xml.append(f' ') - program_xml.append(f' {html.escape(custom_data["rating"])}') - program_xml.append(f" ") - - if "star_ratings" in custom_data and isinstance(custom_data["star_ratings"], list): - for star_rating in custom_data["star_ratings"]: - if isinstance(star_rating, dict) and "value" in star_rating: - system_attr = f' system="{html.escape(star_rating["system"])}"' if "system" in star_rating else "" - program_xml.append(f" ") - program_xml.append(f" {html.escape(star_rating['value'])}") - program_xml.append(" ") - - if "reviews" in custom_data and isinstance(custom_data["reviews"], list): - for review in custom_data["reviews"]: - if isinstance(review, dict) and "content" in review: - review_type = review.get("type", "text") - attrs = [f'type="{html.escape(review_type)}"'] - if "source" in review: - attrs.append(f'source="{html.escape(review["source"])}"') - if "reviewer" in review: - attrs.append(f'reviewer="{html.escape(review["reviewer"])}"') - attr_str = " ".join(attrs) - program_xml.append(f' {html.escape(review["content"])}') - - if "images" in custom_data and isinstance(custom_data["images"], list): - for image in custom_data["images"]: - if isinstance(image, dict) and "url" in image: - attrs = [] - for attr in ['type', 'size', 'orient', 'system']: - if attr in image: - attrs.append(f'{attr}="{html.escape(image[attr])}"') - attr_str = " " + " ".join(attrs) if attrs else "" - program_xml.append(f' {html.escape(image["url"])}') - - # Add enhanced credits handling - if "credits" in custom_data: - program_xml.append(" ") - credits = custom_data["credits"] - - for role in ['director', 'writer', 'adapter', 'producer', 'composer', 'editor', 'presenter', 'commentator', 'guest']: - if role in credits: - people = credits[role] - if isinstance(people, list): - for person in people: - program_xml.append(f" <{role}>{html.escape(person)}") - else: - program_xml.append(f" <{role}>{html.escape(people)}") - - # Handle actors separately to include role and guest attributes - if "actor" in credits: - actors = credits["actor"] - if isinstance(actors, list): - for actor in actors: - if isinstance(actor, dict): - name = actor.get("name", "") - role_attr = f' role="{html.escape(actor["role"])}"' if "role" in actor else "" - guest_attr = ' guest="yes"' if actor.get("guest") else "" - program_xml.append(f" {html.escape(name)}") - else: - program_xml.append(f" {html.escape(actor)}") - else: - program_xml.append(f" {html.escape(actors)}") - - program_xml.append(" ") - - if "date" in custom_data: - program_xml.append(f' {html.escape(custom_data["date"])}') - - if "country" in custom_data: - program_xml.append(f' {html.escape(custom_data["country"])}') - - if "icon" in custom_data: - program_xml.append(f' ') - elif "sd_icon" in custom_data: - program_xml.append(f' ') - - # Add special flags as proper tags with enhanced handling - if custom_data.get("previously_shown", False): - prev_shown_details = custom_data.get("previously_shown_details", {}) - attrs = [] - if "start" in prev_shown_details: - attrs.append(f'start="{html.escape(prev_shown_details["start"])}"') - if "channel" in prev_shown_details: - attrs.append(f'channel="{html.escape(prev_shown_details["channel"])}"') - attr_str = " " + " ".join(attrs) if attrs else "" - program_xml.append(f" ") - - if custom_data.get("premiere", False): - premiere_text = custom_data.get("premiere_text", "") - if premiere_text: - program_xml.append(f" {html.escape(premiere_text)}") - else: - program_xml.append(" ") - - if custom_data.get("last_chance", False): - last_chance_text = custom_data.get("last_chance_text", "") - if last_chance_text: - program_xml.append(f" {html.escape(last_chance_text)}") - else: - program_xml.append(" ") - - if custom_data.get("new", False): - program_xml.append(" ") - - if custom_data.get('live', False): - program_xml.append(' ') - - program_xml.append(" ") - - xml_text = '\n'.join(program_xml) - program_batch.append(xml_text) - - if is_multi: - multi_buffer.append(xml_text) - - if len(program_batch) >= batch_size: - yield '\n'.join(program_batch) + '\n' - program_batch = [] - - # Final flush of multi-channel buffer - if is_multi and multi_buffer: - escaped_primary = html.escape(channel_ids_for_epg[0]) - for extra_cid in channel_ids_for_epg[1:]: - escaped_extra = html.escape(extra_cid) - for xml_text in multi_buffer: - program_batch.append(xml_text.replace( - f'channel="{escaped_primary}"', - f'channel="{escaped_extra}"', - 1, - )) - - if program_batch: - yield '\n'.join(program_batch) + '\n' - - # Send final closing tag and completion message - yield "\n" - - # Log system event for EPG download after streaming completes (with deduplication based on client) - client_id, client_ip, user_agent = get_client_identifier(request) - event_cache_key = f"epg_download:{user.username if user else 'anonymous'}:{profile_name or 'all'}:{client_id}" - if not cache.get(event_cache_key): - # `len()` reuses the queryset's iteration cache populated above; - # `count()` would issue a separate SELECT COUNT(*). - log_system_event( - event_type='epg_download', - profile=profile_name or 'all', - user=user.username if user else 'anonymous', - channels=len(channels), - client_ip=client_ip, - user_agent=user_agent, - ) - cache.set(event_cache_key, True, 2) # Prevent duplicate events for 2 seconds - - # Wrapper generator that collects content for caching - def caching_generator(): - collected_content = [] - for chunk in epg_generator(): - collected_content.append(chunk) - yield chunk - # After streaming completes, cache the full content - full_content = ''.join(collected_content) - cache.set(content_cache_key, full_content, 300) - logger.debug("Cached EPG content (%d bytes)", len(full_content)) - - response = StreamingHttpResponse( - streaming_content=caching_generator(), - content_type="application/xml" - ) - response["Content-Disposition"] = 'attachment; filename="Dispatcharr.xml"' - response["Cache-Control"] = "no-cache" - return response - - def xc_get_user(request): username = request.GET.get("username") password = request.GET.get("password") diff --git a/core/tasks.py b/core/tasks.py index b562f335..a64469b4 100644 --- a/core/tasks.py +++ b/core/tasks.py @@ -398,12 +398,16 @@ def scan_and_process_files(): def _rebuild_programme_indices(): """Queue index builds for active EPG sources that are missing their DB index.""" try: + from django.db.models import Q from apps.epg.tasks import build_programme_index_task sources = EPGSource.objects.filter( is_active=True, - programme_index__isnull=True, - ).exclude(source_type__in=('dummy', 'schedules_direct')) + ).exclude( + source_type__in=('dummy', 'schedules_direct') + ).filter( + Q(index_record__isnull=True) | Q(index_record__data__isnull=True) + ) count = 0 for source in sources: diff --git a/core/tests.py b/core/tests.py index 7b2f1986..2af52f07 100644 --- a/core/tests.py +++ b/core/tests.py @@ -1,8 +1,8 @@ from unittest.mock import patch, MagicMock -from django.test import TestCase +from django.test import TestCase, SimpleTestCase -from apps.epg.models import EPGSource +from apps.epg.models import EPGSource, EPGSourceIndex from core.models import CoreSettings, DVR_SETTINGS_KEY, EPG_SETTINGS_KEY @@ -37,14 +37,10 @@ class DispatcharrUserAgentTests(TestCase): class ProgrammeIndexRebuildTests(TestCase): def test_startup_rebuild_does_not_lock_out_queued_build_task(self): - EPGSource.objects.update( - programme_index={"channels": {}, "interleaved_channels": []} - ) source = EPGSource.objects.create( name="Missing Index", source_type="xmltv", is_active=True, - programme_index=None, ) class FakeRedis: @@ -301,3 +297,14 @@ class DropDBCommandTlsTest(TestCase): host='localhost', port=5432, autocommit=True, ) + + +class MallocTrimTests(SimpleTestCase): + def test_trim_is_noop_when_libc_has_no_malloc_trim(self): + from core.utils import trim_c_allocator_heap + + fake_libc = MagicMock(spec=[]) + with patch('ctypes.util.find_library', return_value='libc.so.6'), patch( + 'ctypes.CDLL', return_value=fake_libc + ): + self.assertFalse(trim_c_allocator_heap()) diff --git a/core/utils.py b/core/utils.py index 29c613b3..44504b6a 100644 --- a/core/utils.py +++ b/core/utils.py @@ -546,6 +546,25 @@ def monitor_memory_usage(func): return result return wrapper +def trim_c_allocator_heap(): + """Return unused C heap pages to the OS where supported (glibc malloc_trim).""" + try: + import ctypes + import ctypes.util + + libc_name = ctypes.util.find_library("c") + if not libc_name: + return False + libc = ctypes.CDLL(libc_name) + if not hasattr(libc, "malloc_trim"): + return False + libc.malloc_trim(0) + return True + except Exception: + logger.debug("malloc_trim unavailable or failed", exc_info=True) + return False + + def cleanup_memory(log_usage=False, force_collection=True): """ Comprehensive memory cleanup function to reduce memory footprint @@ -589,6 +608,30 @@ def cleanup_memory(log_usage=False, force_collection=True): pass logger.trace("Memory cleanup complete for django") + +def spawn_memory_trim(close_connections=False): + """Reclaim a request's heap pages: GC, then return freed C pages to the OS. + + On gevent uWSGI workers the trim runs in a spawned greenlet so it never + blocks the caller; Celery prefork workers (no gevent hub) run it inline. + Set close_connections=True when called from a streaming generator's teardown + so the pooled DB connection is released first. + """ + def _run(): + cleanup_memory(force_collection=True) + trim_c_allocator_heap() + + if close_connections: + from django.db import close_old_connections + close_old_connections() + + if _is_gevent_monkey_patched(): + import gevent + gevent.spawn(_run) + else: + _run() + + def safe_upload_path(filename: str, base_dir) -> str: """Return a safe absolute path for an uploaded file within base_dir.