diff --git a/CHANGELOG.md b/CHANGELOG.md index 61b4eb47..ae471a00 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,10 +11,50 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Per-user IP/CIDR network allowlists**. Admins can now assign IP address and CIDR range restrictions to individual user accounts via the API & XC tab on the user edit form. When a user has one or more allowed ranges configured, requests from IPs outside that list are rejected with `403 Forbidden` regardless of the global network access policy; if no ranges are configured, the user inherits global settings unchanged. The existing `network_access_allowed()` utility is extended with an optional `user` argument so the per-user check is enforced at all access-controlled entry points (M3U/EPG, Streams, XC API, UI) without duplicating IP-matching logic. Per-user restrictions are stored in `custom_properties['allowed_networks']`; no model changes or migrations are required. — Thanks [@sethwv](https://github.com/sethwv) - **`reset_user_network` management command**: `manage.py reset_user_network ` clears the per-user `allowed_networks` restriction for the specified account, restoring it to global-policy inheritance. Useful for recovering a user locked out by a misconfigured allowlist. — Thanks [@sethwv](https://github.com/sethwv) +- **Auto-sync overhaul**: comprehensive rebuild of the M3U auto-channel-sync flow. Introduces a per-field override system, hide-from-output flag, range-bounded auto-numbering with a re-pack helper, multi-stream channel safety, multi-provider shared-range merging, and an across-the-board move from per-row writes to bulk operations. (Closes #1196) — Thanks [@CodeBormen](https://github.com/CodeBormen) + - **Per-field channel overrides for auto-synced channels.** A new `ChannelOverride` table (one-to-one with `Channel`) holds user-specified values for `name`, `channel_number`, `channel_group`, `logo`, `tvg_id`, `tvc_guide_stationid`, `epg_data`, and `stream_profile`. Sync writes only to `Channel.*` columns; the override row takes precedence at read time via a new `with_effective_values()` queryset helper that coalesces both sources at the SQL layer. Every output surface that consumes channel data reads through this helper so overrides surface consistently: HDHR (`/hdhr/lineup.json` and per-profile variants, `/hdhr/discover.json`), M3U (`/output/m3u`, per-profile variants), EPG (`/output/epg`, per-profile variants), XC API (`get_live_streams`, `xmltv.php`), the channels list/detail endpoints, and the TV Guide summary endpoint (`GET /api/channels/channels/summary/`). Channel API responses now include both the raw `name`/`channel_number`/etc. fields AND new `effective_*` annotations plus the `override` object, so frontend consumers can show both "what the user picked" and "what the provider sent" simultaneously. + - **Per-field reset-to-provider icon.** FK pickers (channel group, logo, EPG, stream profile) and scalar inputs (name, channel_number, tvg_id, tvc_guide_stationid) display a small reset icon ("Provider: " subtext + Undo2 button) when the field has an active override on an auto-synced channel. Clicking it sets the form value back to the provider value; the override for that field is cleared on save while other overrides remain intact. + - **Auto-created channel source attribution.** The channel edit form shows an "Auto-created from: / " label at the top of auto-synced channels so the user can see which M3U account and which stream produced the row. + - **Hide-from-output flag (`hidden_from_output`).** A user-toggleable boolean that excludes a channel from HDHR, M3U, EPG, and XC output without deleting it. Hidden channels are preserved across auto-sync refreshes and excluded from auto-cleanup. The channels table shows an EyeOff icon on hidden rows. + - **Channels table visibility filter and inline indicators.** The table header has a new visibility selector ("Active Only" / "Hidden Only" / "Show All") plus a "Has Overrides" filter that narrows to channels with an active override row. Each row's name cell renders a yellow Pencil icon when overrides are active (tooltip lists the overridden field names) and an EyeOff icon when the channel is hidden. + - **Bulk hide / unhide in the bulk edit modal.** Selecting any number of channels and toggling `hidden_from_output` in bulk edit applies the change in a single PATCH; auto-routes through the override-aware bulk endpoint so it works equivalently for auto-synced and manual channels. + - **Auto-sync channel ranges per group.** `ChannelGroupM3UAccount` accepts `auto_sync_channel_start` (lower bound, inclusive) and `auto_sync_channel_end` (upper bound, inclusive; nullable for unbounded fill). Channels created by auto-sync are assigned numbers within the configured range; streams that do not fit are surfaced in the failure detail modal with a typed "RANGE_EXHAUSTED" reason. + - **Per-group inline configuration in the M3U Live group filter.** Each group row now exposes a Sync toggle, a Numbering Mode selector (Provider / Next Available / Fixed), and Start/End channel-number inputs alongside the existing fields. The numbering modes control how each stream's channel number is chosen during sync: Provider tries the M3U-supplied number first then falls back to next-available; Next Available always picks the lowest free number from 1 upward; Fixed picks sequentially from the configured Start. + - **Per-group "Configure" modal.** A gear-icon button on each group row opens a `GroupConfigureModal` containing the full set of advanced options. Fields surfaced in the modal: Channel Sort Order (provider order / name / tvg_id / updated_at, with reverse toggle), Compact Numbering toggle and "Re-pack now" button, Group Override (re-route this group's auto-created channels into a different `ChannelGroup` so multiple provider groups merge into one logical group), Name Regex Find/Replace pattern, Exclude Regex pattern, Force Dummy EPG, and a live regex preview pane. + - **Live regex preview pane.** A new `GET /api/channels/streams/regex-preview/` endpoint scans the group's streams (capped at 5000) for the find / match / exclude patterns the user is editing and returns up to 10 sample matches per pattern plus accurate total counts. The frontend debounces the call (500ms) and runs it inside the configure modal so the user sees live evidence of what their patterns will match before saving. + - **Live overlap warning across rows.** As the user edits Start/End values across multiple groups in the same provider, an `AbortController`-debounced scan calls a new `GET /api/channels/channels/numbers-in-range/` endpoint to detect (a) cross-row range overlaps (in-memory, all group pairs) and (b) channels already occupying the configured range that are not the expected occupants for that group. An informational warning surfaces inline; configurations are not blocked. + - **Compact numbering with re-pack helper.** When `compact_numbering=True` on a group's account relation, sync packs visible auto-sync channels sequentially into the configured range. A "Re-pack now" button in the group config modal runs the pack manually via a new `POST /api/m3u/accounts/{id}/repack-group/` endpoint. Hidden channels do not occupy slots; un-hiding shifts visible numbers down to fill the gap. Override-pinned channel numbers are respected as fixed anchors. Single-channel hide / unhide via the post-save signal triggers an incremental `assign_compact_numbers_for_channels` pass without requiring a full sync. + - **Multi-stream channel safety in sync.** A user-attached second stream on an auto-created channel no longer causes the channel to be deleted when one of those streams disappears from the provider. The deletion path filters out channels where at least one current stream remains alive, and per-stream iteration mutates the same `Channel` instance so `bulk_update` writes the merged final state. + - **Orphan channel cleanup mode (3-state, account-scoped).** Stored under `M3UAccount.custom_properties.orphan_channel_cleanup` and surfaced as a SegmentedControl at the top of the M3U account's group-settings page. Three positions: `always` (default; removes every auto-channel whose source stream has disappeared), `preserve_customized` (removes orphans without a `ChannelOverride` row, preserves those with one), and `never` (preserves all orphans, intended for users with manual redundancy/failover stream setups). Hidden channels are universally preserved regardless of mode. + - **Failure modal grouped by reason.** The auto-sync completion notification's failure detail modal groups entries by typed reason (`RANGE_EXHAUSTED`, `INTEGRITY_ERROR`, `OTHER`) with collapsible sections and a per-group cap. The in-memory cap was raised from 50 to 1000 so realistic multi-provider failure sets are not truncated. The notification's auto-close timeout extends from 4s to 12s when failures are present so users have time to see and click into the modal. + - **Multi-provider shared-range merging.** Two M3U accounts can target the same group with overlapping channel-number ranges. Sync seeds `used_numbers` globally so the second provider's channels pick up where the first left off without colliding. The overlap warning in the group-settings UI surfaces this configuration as informational; the merge is intentional, not a hard error. + - **Selection summary in the bulk edit modal.** The bulk edit form shows `Selection: X auto-synced, Y manual` at the top so users can see how a single set of changes will route between override rows and direct writes. + - **Delete-Playlist preview.** A new `GET /api/m3u/accounts/{id}/auto-created-channels-count/` endpoint returns the count and up to 5 sample names of auto-created channels owned by the account. The Delete Playlist confirmation dialog uses this to show the user exactly how many channels will cascade away before they confirm. + - **Custom `Channel.objects` manager.** A thin manager wraps the existing `with_effective_values()` helper so new code can call `Channel.objects.with_effective_values(...)` directly. The module-level helper remains the canonical implementation; existing call sites are unchanged. ### Changed - **Global Network Access settings use tag-style inputs**: the IP/CIDR range fields in the Network Access settings panel now use tag-style chip inputs instead of a plain text field, making it easier to add, review, and remove individual addresses or ranges. — Thanks [@sethwv](https://github.com/sethwv) +- **Auto-sync overhaul** sync and channel behavior changes for the new override system. — Thanks [@CodeBormen](https://github.com/CodeBormen) + - **`Channel.channel_number` is now nullable.** Auto-sync may produce channels without an assigned number (for example, a sync run where the configured range was exhausted before this stream could be slotted, or a hidden channel in compact mode whose slot was released for visible channels). Existing channels are unchanged. HDHR / M3U / EPG / XC output endpoints filter out channels with `null` channel_number, so a number-less row is invisible to clients but visible in the channels page where the user can assign one. The 0037 auto-sync overhaul migration includes a reverse-direction backfill on the `channel_number` AlterField step so a rollback that re-imposes NOT NULL still succeeds even if NULLs are present. + - **M3U account delete always cascades auto-created channels.** The delete dialog is a single confirmation showing the count of affected channels (fetched from the new auto-created-channels-count endpoint). Auto-created channels are removed regardless of `hidden_from_output` or override state: override rows cascade via the one-to-one FK; hidden status does not preserve the channel because there is no provider left to populate it on the next sync. Manual channels are unaffected; their non-provider streams remain intact, and only streams owned by the deleted account are removed. The destroy response now returns `{"deleted_channels": N}` (HTTP 200) instead of an empty 204 so the confirmation toast can show the actual count. + - **Bulk channel edit auto-routes auto-created channels through the override path.** When a bulk edit includes auto-synced channels, the changed fields are written to each channel's `ChannelOverride` row instead of the raw `Channel.*` columns, so the edit survives the next sync. Manual channels in the same selection write directly to `Channel.*`. A "Clear all overrides" affordance pre-clears overrides on the selection before applying new edits in the same submit (clear runs before the routing PATCH so a same-submit clear cannot wipe just-written overrides). + - **Inline edits on the channels table route through the override row for auto-synced channels.** Editing a cell directly in the table (number, name, group, EPG, logo) now writes to `ChannelOverride.` for auto-synced rows so the edit survives the next refresh; manual rows continue to write directly. The save mechanic and validation are unchanged from the user's perspective. + - **Channel-number duplicates are explicitly allowed.** Sync's `used_numbers` seed still avoids assigning the same number to two newly-created auto channels in the same run, so accidental duplication during sync remains impossible. Manual or override-driven duplication is permitted; downstream client behavior on duplicates varies by client. + +### Fixed + +- **Bulk channel edit silently failed on API rejection.** The bulk-edit submit handler in `ChannelBatch.jsx` only wrote `console.error` when the PATCH was rejected; the user saw the spinner stop but received no notification, leading them to assume the save succeeded. The catch block now surfaces a red toast with the server-provided detail (or a generic fallback), and the form stays open with the in-progress selection intact so the user can correct and retry. — Thanks [@CodeBormen](https://github.com/CodeBormen) + +### Performance + +- **Auto-sync at scale**: the new override-aware sync flow is more capable than the prior path but the implementation choices below keep it viable on libraries with thousands of channels. — Thanks [@CodeBormen](https://github.com/CodeBormen) + - **Bulk writes throughout `sync_auto_channels`.** Per-row `Channel.objects.create()` and `.save()` calls were replaced with `bulk_create()` and `bulk_update()` paths that batch the entire group's create + update sets into single round-trips. The renumber pass collects all dirty channels into one list and flushes with a single `bulk_update` at the end of the loop. `ChannelStream.order` writes were similarly consolidated into a single `bulk_update`. + - **Single-pass collision detection via a global `used_numbers` set.** Choosing a free channel number was previously an `O(N)` DB query per stream. The new path seeds a `set()` of all reserved numbers (existing channels + override pins) once per run; `_next_available_number()` is `O(cluster size)` against that set. The seed deliberately excludes only this account's visible auto-created channels (the rows about to be reassigned), so cross-account and hidden-channel reservations are honored without extra queries. + - **SQL-level effective-value coalescing via `with_effective_values()`.** All channel-data consumers (HDHR, M3U, EPG, XC, channel list / detail, TV Guide summary) read effective values directly from a single annotated queryset that left-joins the override row at the database layer. Prevents N+1 `ChannelOverride` lookups across the channel scan and lets the same query serve the join. + - **EPG dispatch deduplication.** `bulk_create` and `bulk_update` bypass `post_save`, so the post-loop step explicitly dispatches `parse_programs_for_tvg_id.delay` once per unique `epg_data_id` actually changed in the run, not per channel. Avoids fan-out where one EPG source change otherwise queued thousands of redundant parse tasks. + - **Logo and EPGData lookup caching during sync.** `sync_auto_channels` now caches Logo and EPGData lookups in a per-run dict so repeated provider URLs / IDs across many streams do not produce duplicate `Logo.objects.get_or_create` or `EPGData.objects.filter` calls. + - **Override-aware bulk PATCH endpoint.** `update_channels_with_override_routing` partitions the bulk-edit selection into auto-created (override write) vs manual (direct write) once, then issues two bulk PATCHes instead of N single-row updates. ## [0.24.0] - 2026-05-03 diff --git a/apps/channels/api_views.py b/apps/channels/api_views.py index 5ef575a0..5eaeab35 100644 --- a/apps/channels/api_views.py +++ b/apps/channels/api_views.py @@ -8,8 +8,8 @@ from drf_spectacular.utils import extend_schema, OpenApiParameter, inline_serial from drf_spectacular.types import OpenApiTypes from rest_framework import serializers from django.shortcuts import get_object_or_404, get_list_or_404 -from django.db import transaction -from django.db.models import Count, F +from django.db import connection, transaction +from django.db.models import Count, F, Prefetch from django.db.models import Q import os, json, requests, logging, mimetypes, threading, time from datetime import timedelta @@ -200,6 +200,249 @@ class StreamViewSet(viewsets.ModelViewSet): # Return the response with the list of IDs return Response(list(stream_ids)) + @extend_schema( + parameters=[ + OpenApiParameter( + name="channel_group", + type=OpenApiTypes.STR, + location=OpenApiParameter.QUERY, + required=True, + description="Channel group name to scope the preview to.", + ), + OpenApiParameter( + name="find", + type=OpenApiTypes.STR, + location=OpenApiParameter.QUERY, + required=False, + description=( + "Find regex for the rename preview. When supplied, " + "the response includes find_matches and find_match_count." + ), + ), + OpenApiParameter( + name="replace", + type=OpenApiTypes.STR, + location=OpenApiParameter.QUERY, + required=False, + description=( + "Replacement string used with the find pattern. " + "Defaults to empty string when omitted." + ), + ), + OpenApiParameter( + name="match", + type=OpenApiTypes.STR, + location=OpenApiParameter.QUERY, + required=False, + description=( + "Filter regex for the include preview. When supplied, " + "the response includes filter_matches and filter_match_count." + ), + ), + OpenApiParameter( + name="exclude", + type=OpenApiTypes.STR, + location=OpenApiParameter.QUERY, + required=False, + description=( + "Filter regex for the exclude preview. When supplied, " + "the response includes exclude_matches and " + "exclude_match_count." + ), + ), + OpenApiParameter( + name="limit", + type=OpenApiTypes.INT, + location=OpenApiParameter.QUERY, + required=False, + description="Max preview entries per match list (default 10, capped at 50).", + ), + ], + responses={ + 200: inline_serializer( + name="StreamRegexPreviewResponse", + fields={ + "total_in_group": serializers.IntegerField(), + "total_scanned": serializers.IntegerField(), + "scan_limit_hit": serializers.BooleanField(), + "find_matches": serializers.ListField( + child=serializers.DictField(), required=False + ), + "find_match_count": serializers.IntegerField(required=False), + "filter_matches": serializers.ListField( + child=serializers.DictField(), required=False + ), + "filter_match_count": serializers.IntegerField(required=False), + "exclude_matches": serializers.ListField( + child=serializers.DictField(), required=False + ), + "exclude_match_count": serializers.IntegerField(required=False), + "find_error": serializers.CharField(required=False), + "match_error": serializers.CharField(required=False), + "exclude_error": serializers.CharField(required=False), + }, + ) + }, + description=( + "Returns regex preview info for a group's streams. Used by the " + "auto-sync gear modal so users can see how their find/replace " + "or filter pattern affects real stream names before saving. " + "Caps in-memory iteration at SCAN_CAP streams per call so the " + "endpoint stays bounded even on groups with tens of thousands " + "of streams; the caller surfaces total_in_group and " + "scan_limit_hit so users know whether the preview is complete." + ), + ) + @action(detail=False, methods=["get"], url_path="regex-preview") + def regex_preview(self, request, *args, **kwargs): + # `regex` (third-party) supports a per-call timeout that bounds + # catastrophic backtracking; paired with PATTERN_MAX_LEN to keep + # the endpoint safe under adversarial input. + import regex as re + + SCAN_CAP = 5000 + PATTERN_MAX_LEN = 512 + REGEX_TIMEOUT = 0.1 + + group_name = request.query_params.get("channel_group") + if not group_name: + return Response( + {"detail": "channel_group is required"}, + status=status.HTTP_400_BAD_REQUEST, + ) + + # Group names are not unique across M3U accounts (two providers + # can both publish a "Sports" group). Scope to the calling + # account so the sample reflects only the user's edits. + m3u_account_id = request.query_params.get("m3u_account_id") + if m3u_account_id is not None: + try: + m3u_account_id = int(m3u_account_id) + except (TypeError, ValueError): + return Response( + {"detail": "m3u_account_id must be an integer"}, + status=status.HTTP_400_BAD_REQUEST, + ) + + find_pat = request.query_params.get("find") or None + replace_pat = request.query_params.get("replace") or "" + match_pat = request.query_params.get("match") or None + exclude_pat = request.query_params.get("exclude") or None + for label, value in ( + ("find", find_pat), + ("replace", replace_pat), + ("match", match_pat), + ("exclude", exclude_pat), + ): + if value is not None and len(value) > PATTERN_MAX_LEN: + return Response( + {"detail": f"{label} exceeds {PATTERN_MAX_LEN} characters"}, + status=status.HTTP_400_BAD_REQUEST, + ) + try: + limit = int(request.query_params.get("limit", 10)) + except (TypeError, ValueError): + limit = 10 + limit = max(1, min(limit, 50)) + + find_re = None + match_re = None + exclude_re = None + find_error = None + match_error = None + exclude_error = None + if find_pat: + try: + find_re = re.compile(find_pat) + except re.error as e: + find_error = str(e) + if match_pat: + try: + match_re = re.compile(match_pat) + except re.error as e: + match_error = str(e) + if exclude_pat: + try: + exclude_re = re.compile(exclude_pat) + except re.error as e: + exclude_error = str(e) + + # Capped at SCAN_CAP to bound memory on huge groups; the + # separate COUNT lets the client surface scan_limit_hit when + # the preview covers only a sample. + base_qs = Stream.objects.filter(channel_group__name=group_name) + if m3u_account_id is not None: + base_qs = base_qs.filter(m3u_account_id=m3u_account_id) + names_iter = base_qs.values_list("name", flat=True)[:SCAN_CAP] + total_in_group = base_qs.count() + + find_matches = [] + filter_matches = [] + exclude_matches = [] + find_match_count = 0 + filter_match_count = 0 + exclude_match_count = 0 + total_scanned = 0 + # Abort a pattern on timeout to bound CPU; partial counts and + # an `*_error` field still flow back to the client. + for name in names_iter: + total_scanned += 1 + if find_re is not None: + try: + new_name = find_re.sub(replace_pat, name, timeout=REGEX_TIMEOUT) + except (TimeoutError, re.error) as e: + find_error = find_error or f"Pattern timed out: {e}" + find_re = None + continue + if new_name != name: + find_match_count += 1 + if len(find_matches) < limit: + find_matches.append({"before": name, "after": new_name}) + if match_re is not None: + try: + matched = match_re.search(name, timeout=REGEX_TIMEOUT) + except (TimeoutError, re.error) as e: + match_error = match_error or f"Pattern timed out: {e}" + match_re = None + continue + if matched: + filter_match_count += 1 + if len(filter_matches) < limit: + filter_matches.append({"name": name, "matches": True}) + if exclude_re is not None: + try: + matched = exclude_re.search(name, timeout=REGEX_TIMEOUT) + except (TimeoutError, re.error) as e: + exclude_error = exclude_error or f"Pattern timed out: {e}" + exclude_re = None + continue + if matched: + exclude_match_count += 1 + if len(exclude_matches) < limit: + exclude_matches.append({"name": name, "matches": True}) + + response_payload = { + "total_in_group": total_in_group, + "total_scanned": total_scanned, + "scan_limit_hit": total_in_group > SCAN_CAP, + } + if find_pat: + response_payload["find_matches"] = find_matches + response_payload["find_match_count"] = find_match_count + if find_error: + response_payload["find_error"] = find_error + if match_pat: + response_payload["filter_matches"] = filter_matches + response_payload["filter_match_count"] = filter_match_count + if match_error: + response_payload["match_error"] = match_error + if exclude_pat: + response_payload["exclude_matches"] = exclude_matches + response_payload["exclude_match_count"] = exclude_match_count + if exclude_error: + response_payload["exclude_error"] = exclude_error + return Response(response_payload) + @action(detail=False, methods=["get"], url_path="groups") def get_groups(self, request, *args, **kwargs): # Get unique ChannelGroup names that are linked to streams @@ -322,8 +565,56 @@ class ChannelGroupViewSet(viewsets.ModelViewSet): return [Authenticated()] def get_queryset(self): - """Return channel groups with prefetched relations for efficient counting""" - return ChannelGroup.objects.prefetch_related('channels', 'm3u_accounts').all() + # Annotate both counts at the SQL level so the serializer methods + # can read them from the object rather than issuing a COUNT per row. + # `distinct=True` is required when multiple reverse-FK annotations + # share the same queryset to avoid row-multiplication artifacts. + # m3u_accounts is still prefetched for the nested serializer data. + return ( + ChannelGroup.objects + .annotate( + channel_count=Count('channels', distinct=True), + m3u_account_count=Count('m3u_accounts', distinct=True), + ) + .prefetch_related('m3u_accounts') + .all() + ) + + def list(self, request, *args, **kwargs): + queryset = self.filter_queryset(self.get_queryset()) + + # Evaluate the queryset once so the annotations and prefetch cache are + # populated together, then extract IDs from the in-memory objects. + # A second .values_list() call would fire a separate SQL query. + groups = list(queryset) + group_ids = [g.id for g in groups] + + # Pre-aggregate stream counts for all (account, group) pairs in a + # single query so the nested ChannelGroupM3UAccountSerializer never + # fires a COUNT per row. + counts_qs = ( + Stream.objects.filter(channel_group_id__in=group_ids) + .values('m3u_account_id', 'channel_group_id') + .annotate(c=Count('id')) + ) + stream_counts = { + (row['m3u_account_id'], row['channel_group_id']): row['c'] + for row in counts_qs + } + + page = self.paginate_queryset(groups) + if page is not None: + serializer = self.get_serializer( + page, many=True, + context={**self.get_serializer_context(), 'stream_counts': stream_counts}, + ) + return self.get_paginated_response(serializer.data) + + serializer = self.get_serializer( + groups, many=True, + context={**self.get_serializer_context(), 'stream_counts': stream_counts}, + ) + return Response(serializer.data) def update(self, request, *args, **kwargs): """Override update to check M3U associations""" @@ -563,17 +854,31 @@ class ChannelViewSet(viewsets.ModelViewSet): return [Authenticated()] def get_queryset(self): - qs = ( - super() - .get_queryset() - .select_related( + # get_ids and summary only need the filter conditions, not the full + # object graph. Skipping the 5 select_related joins and 2 prefetch + # queries for those actions cuts their DB cost significantly. + action = getattr(self, "action", None) + qs = super().get_queryset() + + if action not in ("get_ids", "summary"): + qs = qs.select_related( "channel_group", "logo", "epg_data", "stream_profile", + "override", + "auto_created_by", + ).prefetch_related( + "streams", + # Default-attr prefetch shares the cache with M2M writes; + # a named `to_attr` would isolate it and trigger N+1. + Prefetch( + "channelstream_set", + queryset=ChannelStream.objects.select_related( + "stream__m3u_account" + ).order_by("order"), + ), ) - .prefetch_related("streams") - ) channel_group = self.request.query_params.get("channel_group") if channel_group: @@ -587,6 +892,8 @@ class ChannelViewSet(viewsets.ModelViewSet): show_disabled_param = self.request.query_params.get("show_disabled", None) only_streamless = self.request.query_params.get("only_streamless", None) only_stale = self.request.query_params.get("only_stale", None) + only_has_overrides = self.request.query_params.get("only_has_overrides", None) + visibility_filter = self.request.query_params.get("visibility_filter", "active") if channel_profile_id: try: @@ -609,6 +916,18 @@ class ChannelViewSet(viewsets.ModelViewSet): if only_stale: # Filter channels that have at least one related stream marked as stale q_filters &= Q(streams__is_stale=True) + if only_has_overrides: + q_filters &= Q(override__isnull=False) + + # Visibility filter applies to list-style reads only; retrieve / + # update / delete must still reach a hidden channel by id so the + # frontend can unhide. Summary powers the TV Guide and follows + # the same hidden semantic as downstream clients. + if self.action in ("list", "get_ids", "summary"): + if visibility_filter == "hidden": + q_filters &= Q(hidden_from_output=True) + elif visibility_filter != "all": + q_filters &= Q(hidden_from_output=False) if self.request.user.user_level < 10: filters["user_level__lte"] = self.request.user.user_level @@ -630,6 +949,14 @@ class ChannelViewSet(viewsets.ModelViewSet): self.request.query_params.get("include_streams", "false") == "true" ) context["include_streams"] = include_streams + # source_stream is only needed by the channel edit form. For get_ids + # and summary the channelstream_set prefetch is skipped entirely, so + # source_stream cannot be computed without hitting the DB per channel. + # For list and write/retrieve paths the prefetch is present, so we + # can populate it from memory without extra queries. + context["include_source_stream"] = action not in ( + "get_ids", "summary" + ) if (action := getattr(self, "action", None)) else False return context @extend_schema( @@ -756,14 +1083,44 @@ class ChannelViewSet(viewsets.ModelViewSet): status=status.HTTP_400_BAD_REQUEST, ) + # Capture override intent from the raw payload: presence of the + # key distinguishes "no change" from explicit null ("clear"), + # which `validated_data` would collapse. + override_intents = {} + for channel_data in data: + if "override" in channel_data: + cid = channel_data.get("id") + override_intents[cid] = channel_data["override"] + + # Capture hide / unhide transitions before setattr overwrites + # the in-memory channels. bulk_update skips post_save, so the + # compact-mode assign / release runs explicitly below. + unhide_transition_ids = [ + channel.id + for channel, validated_data in validated_updates + if validated_data.get("hidden_from_output") is False + and channel.hidden_from_output is True + ] + hide_transition_candidates = [ + channel + for channel, validated_data in validated_updates + if validated_data.get("hidden_from_output") is True + and channel.hidden_from_output is False + and channel.channel_number is not None + and channel.auto_created + and channel.auto_created_by_id + ] + # Apply all updates in a transaction with transaction.atomic(): streams_updates = [] for channel, validated_data in validated_updates: - # Pop streams before setattr loop — M2M fields can't be set via setattr + # Streams (M2M) and override (reverse OneToOne) cannot + # ride the setattr loop; handle each separately below. streams = validated_data.pop("streams", None) if streams is not None: streams_updates.append((channel, streams)) + validated_data.pop("override", None) for key, value in validated_data.items(): setattr(channel, key, value) @@ -783,6 +1140,179 @@ class ChannelViewSet(viewsets.ModelViewSet): batch_size=100 ) + # On unhide under compact mode, assign a number so the + # channel is immediately addressable by clients. + if unhide_transition_ids: + from .compact_numbering import ( + assign_compact_numbers_for_channels, + ) + assign_compact_numbers_for_channels(unhide_transition_ids) + + # On hide under compact mode, release the channel_number + # so the slot is reused. The bulk path bypasses the + # post_save signal that handles single-row hides. + if hide_transition_candidates: + from .compact_numbering import ( + get_group_relation_for_channel, + is_compact_group, + ) + ids_to_release = [] + for ch in hide_transition_candidates: + relation = get_group_relation_for_channel(ch) + if relation and is_compact_group(relation): + ids_to_release.append(ch.id) + if ids_to_release: + Channel.objects.filter(id__in=ids_to_release).update( + channel_number=None + ) + # Refresh in-memory copies so the response shape + # reflects the cleared numbers. + for ch in channels_to_update: + if ch.id in ids_to_release: + ch.channel_number = None + + # Override (reverse OneToOne) needs a separate write path. + if override_intents: + from apps.channels.models import ChannelOverride + from apps.channels.managers import OVERRIDABLE_FIELDS + override_fields = OVERRIDABLE_FIELDS + # Block override mutations on manual channels. Mixed + # selections with override:null are tolerated because + # clearing a non-existent row is a no-op. + manual_with_override = [] + for channel, _ in validated_updates: + if channel.id not in override_intents: + continue + intent = override_intents[channel.id] + if ( + intent is not None + and intent != {} + and not channel.auto_created + ): + manual_with_override.append(channel.id) + if manual_with_override: + return Response( + { + "errors": [ + { + "channel_id": cid, + "error": ( + "Cannot set override on a manual channel; " + "overrides only apply to auto-created channels." + ), + } + for cid in manual_with_override + ] + }, + status=status.HTTP_400_BAD_REQUEST, + ) + + channels_to_clear = [] + overrides_to_upsert = [] + for channel, _ in validated_updates: + if channel.id not in override_intents: + continue + intent = override_intents[channel.id] + if intent is None: + channels_to_clear.append(channel.id) + elif intent == {}: + # Empty dict means no override intent; treat as no-op. + continue + else: + defaults = { + f: intent.get(f) + for f in override_fields + if f in intent + } + # Coerce FK aliases (logo, channel_group, ...) to + # the *_id columns ChannelOverride actually stores. + for raw, mapped in ( + ("logo", "logo_id"), + ("channel_group", "channel_group_id"), + ("epg_data", "epg_data_id"), + ("stream_profile", "stream_profile_id"), + ): + if raw in intent and mapped not in defaults: + val = intent[raw] + defaults[mapped] = ( + val.id if hasattr(val, "id") else val + ) + overrides_to_upsert.append((channel.id, defaults)) + + if channels_to_clear: + ChannelOverride.objects.filter( + channel_id__in=channels_to_clear + ).delete() + + # Bulk upsert keeps a 1000-channel batch to two + # statements (one INSERT, one UPDATE) instead of the + # per-row SELECT + INSERT-or-UPDATE that update_or_create + # would generate. + if overrides_to_upsert: + existing_overrides = { + o.channel_id: o + for o in ChannelOverride.objects.filter( + channel_id__in=[ + cid for cid, _ in overrides_to_upsert + ] + ) + } + to_create = [] + to_update = [] + update_field_set = set() + for channel_id, defaults in overrides_to_upsert: + existing = existing_overrides.get(channel_id) + if existing: + for f, v in defaults.items(): + setattr(existing, f, v) + update_field_set.add(f) + to_update.append(existing) + else: + to_create.append( + ChannelOverride( + channel_id=channel_id, **defaults + ) + ) + if to_update: + ChannelOverride.objects.bulk_update( + to_update, + fields=list(update_field_set), + batch_size=200, + ) + if to_create: + ChannelOverride.objects.bulk_create( + to_create, batch_size=200 + ) + + # Drop override rows that ended up all-null; an empty + # override would falsely surface as active in the UI. + touched_ids = [cid for cid, _ in overrides_to_upsert] + empty_overrides = [ + o for o in ChannelOverride.objects.filter( + channel_id__in=touched_ids + ) + if not o.has_any_override() + ] + if empty_overrides: + ChannelOverride.objects.filter( + id__in=[o.id for o in empty_overrides] + ).delete() + + # Queryset writes leave the reverse-OneToOne cache stale; + # clear it so the serializer reads the new override state. + touched_channel_ids = {cid for cid, _ in overrides_to_upsert} + touched_channel_ids.update(channels_to_clear) + if touched_channel_ids: + for channel, _ in validated_updates: + if channel.id not in touched_channel_ids: + continue + try: + channel._state.fields_cache.pop("override", None) + except AttributeError: + pass + if hasattr(channel, "_channel_override_cache"): + delattr(channel, "_channel_override_cache") + # Handle streams M2M updates separately for channel, streams in streams_updates: normalized_ids = [ @@ -1032,21 +1562,158 @@ class ChannelViewSet(viewsets.ModelViewSet): @action(detail=False, methods=["get"], url_path="summary") def summary(self, request, *args, **kwargs): - """Return a lightweight list of channels with only the fields needed by the TV Guide.""" - queryset = self.filter_queryset(self.get_queryset()) - data = list( - queryset.values( - "id", - "name", - "logo_id", - "channel_number", - "uuid", - "epg_data_id", - "channel_group_id", - ) + """Return a lightweight list of channels with only the fields needed by the TV Guide. + + The TV Guide is a downstream output surface like HDHR / M3U / EPG / + XC and must reflect the user's overrides. Effective values are + coalesced at the SQL layer; the annotated columns are renamed + back to the raw field names on the way out so the response + shape stays unchanged for the frontend. + """ + from .managers import with_effective_values + + queryset = with_effective_values( + self.filter_queryset(self.get_queryset()) ) + data = [ + { + "id": row["id"], + "uuid": row["uuid"], + "name": row["effective_name"], + "logo_id": row["effective_logo_id"], + "channel_number": row["effective_channel_number"], + "epg_data_id": row["effective_epg_data_id"], + "channel_group_id": row["effective_channel_group_id"], + } + for row in queryset.values( + "id", + "uuid", + "effective_name", + "effective_logo_id", + "effective_channel_number", + "effective_epg_data_id", + "effective_channel_group_id", + ) + ] return Response(data) + @extend_schema( + parameters=[ + OpenApiParameter( + name="start", + type=OpenApiTypes.NUMBER, + location=OpenApiParameter.QUERY, + required=True, + description="Inclusive lower bound of the range to scan.", + ), + OpenApiParameter( + name="end", + type=OpenApiTypes.NUMBER, + location=OpenApiParameter.QUERY, + required=False, + description=( + "Inclusive upper bound. If omitted or equal to start, " + "behaves as a single-number lookup." + ), + ), + ], + responses={ + 200: inline_serializer( + name="ChannelsInRangeResponse", + fields={ + "occupants": serializers.ListField( + child=serializers.DictField() + ) + }, + ) + }, + description=( + "Returns the channels (including those whose effective number is " + "set via override) currently occupying numbers within the given " + "range. Used by the group settings form to surface inline range " + "conflict warnings. Capped at 50 entries to bound the response " + "payload; the frontend only needs to know whether any conflicts " + "exist after filtering, not the entire list." + ), + ) + @action(detail=False, methods=["get"], url_path="numbers-in-range") + def numbers_in_range(self, request, *args, **kwargs): + from .managers import with_effective_values + + raw_start = request.query_params.get("start") + raw_end = request.query_params.get("end") + if raw_start is None or raw_start == "": + return Response({"occupants": []}) + try: + start = float(raw_start) + except (TypeError, ValueError): + return Response( + {"detail": "Invalid start value"}, + status=status.HTTP_400_BAD_REQUEST, + ) + try: + end = float(raw_end) if raw_end not in (None, "") else start + except (TypeError, ValueError): + return Response( + {"detail": "Invalid end value"}, + status=status.HTTP_400_BAD_REQUEST, + ) + if end < start: + start, end = end, start + + queryset = ( + with_effective_values( + Channel.objects.all(), select_related_fks=True + ) + .filter( + effective_channel_number__gte=start, + effective_channel_number__lte=end, + ) + .order_by("effective_channel_number")[:50] + ) + + occupants = [] + for occupant in queryset: + effective_group = getattr( + occupant, "effective_channel_group_obj", None + ) + group_name = ( + getattr(effective_group, "name", None) + if effective_group is not None + else None + ) + effective_group_id = getattr( + occupant, "effective_channel_group_id", None + ) + override = getattr(occupant, "override", None) + override_sets_number = bool( + override is not None and override.channel_number is not None + ) + occupants.append( + { + "id": occupant.id, + "name": getattr( + occupant, "effective_name", occupant.name + ), + "channel_number": getattr( + occupant, + "effective_channel_number", + occupant.channel_number, + ), + "channel_group": group_name, + "channel_group_id": effective_group_id, + "auto_created": bool(occupant.auto_created), + "auto_created_by_account_id": ( + occupant.auto_created_by_id + if occupant.auto_created_by_id + else None + ), + "has_channel_number_override": override_sets_number, + } + ) + + return Response({"occupants": occupants}) + @extend_schema( methods=["POST"], description="Retrieve channels by a list of UUIDs using POST to avoid URL length limitations", diff --git a/apps/channels/compact_numbering.py b/apps/channels/compact_numbering.py new file mode 100644 index 00000000..f3d7183c --- /dev/null +++ b/apps/channels/compact_numbering.py @@ -0,0 +1,363 @@ +"""Compact channel numbering helpers. + +The per-group `compact_numbering` custom_property is opt-in. When enabled +on a ChannelGroupM3UAccount, the group's auto-created channels get packed +contiguously into the group's [start, end] range: + + * Visible without channel_number override: assigned sequentially + * Hidden without channel_number override: channel_number set to NULL + (released; the slot becomes available for visible channels) + * Override-pinned (any visibility): untouched; the override's + channel_number is treated as a global reservation that other channels + skip when packing + +Used by sync_auto_channels (full-pack pass), the post_save Channel signal +(single-channel unhide), the bulk-edit endpoint (bulk unhide), and a +manual per-group re-pack endpoint. + +Trade-off the user opts into: channel numbers may shift when hide / unhide +state changes. To pin a number through hide/unhide cycles, set a +channel_number override - the override is honored as a reservation. +""" + +import logging + +from django.db import transaction + +from .models import Channel, ChannelOverride, ChannelGroupM3UAccount +from apps.m3u.tasks import _custom_properties_as_dict, _next_available_number +from core.utils import ( + acquire_task_lock, + natural_sort_key, + release_task_lock, +) + +logger = logging.getLogger(__name__) + + +def is_compact_group(group_relation): + """Return True if the given ChannelGroupM3UAccount is in compact mode.""" + cp = _custom_properties_as_dict(group_relation.custom_properties) + return bool(cp.get("compact_numbering")) + + +def get_group_relation_for_channel(channel): + """Resolve the ChannelGroupM3UAccount that owns this auto-created + channel. Returns None for manual channels, or when the relation has + been deleted.""" + if ( + not channel.auto_created + or not channel.auto_created_by_id + or not channel.channel_group_id + ): + return None + try: + return ChannelGroupM3UAccount.objects.get( + m3u_account_id=channel.auto_created_by_id, + channel_group_id=channel.channel_group_id, + ) + except ChannelGroupM3UAccount.DoesNotExist: + return None + + +def build_reserved_set(exclude_channel_ids=None, range_start=None, range_end=None): + """Return the set of channel numbers currently 'claimed' system-wide + for the purposes of a compact pack: + + * Every ChannelOverride.channel_number value (overrides reserve + their effective number; duplicates across overrides are allowed + and collapse to a single reservation via set semantics) + * Every Channel.channel_number value EXCEPT for channels in the + passed-in exclude set (those are about to be reassigned) + + When the caller knows the target group has a bounded range, pass + `range_start` and `range_end` to scope the scan to numbers that + could possibly collide. Without scoping, a single signal-driven + unhide reads every channel_number in the database; with scoping it + reads at most the values within [start, end] which for typical + cable-style ranges is hundreds rather than tens of thousands. + + Float vs int normalization is unnecessary because Python treats + 50 and 50.0 as equal and produces the same hash, so set membership + works directly across both types. + """ + exclude_channel_ids = set(exclude_channel_ids or []) + override_qs = ChannelOverride.objects.filter(channel_number__isnull=False) + other_qs = Channel.objects.exclude(channel_number__isnull=True).exclude( + id__in=exclude_channel_ids + ) + if range_start is not None: + override_qs = override_qs.filter(channel_number__gte=range_start) + other_qs = other_qs.filter(channel_number__gte=range_start) + if range_end is not None: + override_qs = override_qs.filter(channel_number__lte=range_end) + other_qs = other_qs.filter(channel_number__lte=range_end) + reserved = set(override_qs.values_list("channel_number", flat=True)) + reserved.update(other_qs.values_list("channel_number", flat=True)) + reserved.discard(None) + return reserved + + +def _channel_has_number_override(channel): + try: + ov = channel.override + except ChannelOverride.DoesNotExist: + return False + return ov is not None and ov.channel_number is not None + + +def assign_compact_numbers_for_channels(channel_ids): + """For each channel ID in the input that became eligible for a number + (visible, auto-created, no number override, in a compact-mode group), + assign the next available channel number in the group's [start, end] + range. Channels whose group is not in compact mode are skipped silently + so callers can pass mixed batches without filtering up front. + + Used by both the post_save signal (single-channel unhide) and the bulk + edit endpoint (bulk unhide). Returns dict {channel_id: number_or_None}. + """ + if not channel_ids: + return {} + channels = list( + Channel.objects.filter( + id__in=channel_ids, + auto_created=True, + auto_created_by__isnull=False, + hidden_from_output=False, + channel_number__isnull=True, + ).select_related("override", "channel_group", "auto_created_by") + ) + if not channels: + return {} + + # Group channels by (account_id, group_id) so each unique pair's + # ChannelGroupM3UAccount is resolved with a single SELECT rather than + # one per channel. + by_pair = {} + for ch in channels: + if _channel_has_number_override(ch): + continue + key = (ch.auto_created_by_id, ch.channel_group_id) + by_pair.setdefault(key, []).append(ch) + if not by_pair: + return {} + + pair_keys = list(by_pair.keys()) + relations_by_pair = {} + relations_qs = ChannelGroupM3UAccount.objects.filter( + m3u_account_id__in={k[0] for k in pair_keys}, + channel_group_id__in={k[1] for k in pair_keys}, + ) + for rel in relations_qs: + relations_by_pair[(rel.m3u_account_id, rel.channel_group_id)] = rel + + by_relation = {} + for key, group_channels in by_pair.items(): + rel = relations_by_pair.get(key) + if rel is None or not is_compact_group(rel): + continue + by_relation[rel.id] = (rel, group_channels) + + # Group by account so writes share the `refresh_single_m3u_account` + # lock used by sync_auto_channels and the manual repack endpoint. + # If sync is in flight for an account, defer to it. + by_account = {} + for rel, group_channels in by_relation.values(): + by_account.setdefault(rel.m3u_account_id, []).append( + (rel, group_channels) + ) + + results = {} + for account_id, account_pairs in by_account.items(): + if not acquire_task_lock( + "refresh_single_m3u_account", account_id + ): + logger.info( + "Compact unhide deferred for account %s: refresh in progress; " + "next sync will assign numbers.", + account_id, + ) + for _, group_channels in account_pairs: + for ch in group_channels: + results[ch.id] = None + continue + + # try/finally release: the Redis lock is not transactional, so a + # transaction.on_commit release would leak the lock when an + # outer atomic rolls back, blocking subsequent syncs until TTL. + try: + with transaction.atomic(): + for rel, group_channels in account_pairs: + start = int(rel.auto_sync_channel_start or 1) + end = ( + int(rel.auto_sync_channel_end) + if rel.auto_sync_channel_end + else None + ) + reserved = build_reserved_set( + exclude_channel_ids=[c.id for c in group_channels], + range_start=start, + range_end=end, + ) + to_update = [] + for ch in group_channels: + next_num = _next_available_number( + reserved, start, end=end + ) + if next_num is None: + results[ch.id] = None + continue + ch.channel_number = next_num + reserved.add(next_num) + results[ch.id] = next_num + to_update.append(ch) + if to_update: + Channel.objects.bulk_update( + to_update, ["channel_number"], batch_size=100 + ) + finally: + try: + release_task_lock( + "refresh_single_m3u_account", account_id + ) + except Exception as e: + logger.warning( + "Failed to release compact-unhide lock for account " + "%s: %s", + account_id, + e, + ) + return results + + +def repack_group(group_relation): + """Renumber every auto-created channel in the given group+account. + + Visible non-override channels are assigned sequentially in + [start, end] using the group's configured channel_sort_order. + Hidden non-override channels have their channel_number set to + None (slot released). Override-pinned channels are untouched. + + Returns a dict with assigned/released/failed counts. ``failed`` + counts visible channels that could not fit because the range was + exhausted; their channel_number is set to None so the state is + unambiguous instead of stuck at a stale number. + + All writes run inside a single transaction so concurrent readers + (HDHR/M3U/EPG output paths) never observe a half-packed state. + """ + with transaction.atomic(): + return _repack_inner(group_relation) + + +def _repack_inner(group_relation): + account_id = group_relation.m3u_account_id + group_id = group_relation.channel_group_id + + cp = _custom_properties_as_dict(group_relation.custom_properties) + sort_order = cp.get("channel_sort_order") or "" + sort_reverse = bool(cp.get("channel_sort_reverse")) + + start = int(group_relation.auto_sync_channel_start or 1) + end = ( + int(group_relation.auto_sync_channel_end) + if group_relation.auto_sync_channel_end + else None + ) + + channels = list( + Channel.objects.filter( + auto_created=True, + auto_created_by_id=account_id, + channel_group_id=group_id, + ).select_related("override") + ) + + visible = [] + hidden = [] + pinned = [] + for ch in channels: + if _channel_has_number_override(ch): + pinned.append(ch) + elif ch.hidden_from_output: + hidden.append(ch) + else: + visible.append(ch) + + # Sort the visible set by the group's configured channel_sort_order. + # Provider order (the default) preserves DB-iteration order which is + # roughly creation order; treat unrecognized values the same way. + if sort_order == "name": + visible.sort( + key=lambda c: natural_sort_key(c.name or ""), + reverse=sort_reverse, + ) + elif sort_order == "tvg_id": + visible.sort( + key=lambda c: c.tvg_id or "", + reverse=sort_reverse, + ) + elif sort_order == "updated_at": + visible.sort( + key=lambda c: c.updated_at, + reverse=sort_reverse, + ) + + # Exclude every channel in this group: the pinned channel's raw value + # is irrelevant (the override is reserved globally and cleared below). + # Scope to the group's range when bounded to keep the set small. + affected_ids = [c.id for c in (visible + hidden + pinned)] + reserved = build_reserved_set( + exclude_channel_ids=affected_ids, + range_start=start, + range_end=end, + ) + + assigned_count = 0 + failed_count = 0 + visible_to_update = [] + for ch in visible: + next_num = _next_available_number(reserved, start, end=end) + if next_num is None: + failed_count += 1 + if ch.channel_number is not None: + ch.channel_number = None + visible_to_update.append(ch) + continue + if ch.channel_number != next_num: + ch.channel_number = next_num + visible_to_update.append(ch) + reserved.add(next_num) + assigned_count += 1 + + if visible_to_update: + Channel.objects.bulk_update( + visible_to_update, ["channel_number"], batch_size=100 + ) + + released_count = 0 + hidden_with_num = [c for c in hidden if c.channel_number is not None] + if hidden_with_num: + for c in hidden_with_num: + c.channel_number = None + Channel.objects.bulk_update( + hidden_with_num, ["channel_number"], batch_size=100 + ) + released_count = len(hidden_with_num) + + # Pinned channels: clear raw channel_number. The override controls + # their effective number; leaving a stale raw value would pollute + # uniqueness checks and could resurrect on override clear. + pinned_with_num = [c for c in pinned if c.channel_number is not None] + if pinned_with_num: + for c in pinned_with_num: + c.channel_number = None + Channel.objects.bulk_update( + pinned_with_num, ["channel_number"], batch_size=100 + ) + + return { + "assigned": assigned_count, + "released": released_count, + "failed": failed_count, + } diff --git a/apps/channels/managers.py b/apps/channels/managers.py new file mode 100644 index 00000000..ac5c8115 --- /dev/null +++ b/apps/channels/managers.py @@ -0,0 +1,55 @@ +""" +Queryset helpers that resolve effective Channel field values. + +Each Channel can optionally have a related ChannelOverride row carrying user +edits to any subset of its user-facing fields. Sync never touches the override +row; provider metadata flows directly into Channel.* and the override table +sits alongside with a nullable value per field. The helpers here coalesce the +two sources into `effective_*` annotations so output querysets can sort, +filter, and emit values correctly at SQL level (avoiding 23+ Python-side +resolutions across the codebase). +""" + +from django.db.models.functions import Coalesce + + +OVERRIDABLE_FIELDS = ( + "name", + "channel_number", + "channel_group_id", + "logo_id", + "tvg_id", + "tvc_guide_stationid", + "epg_data_id", + "stream_profile_id", +) + + +def with_effective_values(queryset, select_related_fks=False): + """ + Annotate the channels queryset with `effective_*` columns that resolve to + the override value when set, otherwise fall back to the channel's own + value. Always eagerly loads the override one-to-one to avoid N+1 when the + caller reads annotated attributes and then the related override. + + Pass `select_related_fks=True` when the output path will access FK objects + through the `effective_*_obj` Channel properties; this pulls the override's + logo, channel_group, epg_data, and stream_profile in the same query so + those accessors do not trigger per-row lookups. + """ + annotations = { + f"effective_{field}": Coalesce( + f"override__{field}", + field, + ) + for field in OVERRIDABLE_FIELDS + } + qs = queryset.select_related("override").annotate(**annotations) + if select_related_fks: + qs = qs.select_related( + "override__logo", + "override__channel_group", + "override__epg_data", + "override__stream_profile", + ) + return qs diff --git a/apps/channels/migrations/0037_auto_sync_overhaul.py b/apps/channels/migrations/0037_auto_sync_overhaul.py new file mode 100644 index 00000000..d93a1789 --- /dev/null +++ b/apps/channels/migrations/0037_auto_sync_overhaul.py @@ -0,0 +1,156 @@ +""" +Auto-sync overhaul (FR #1196): per-field channel overrides, hide-from-output +flag, configurable auto-sync number range, and nullable channel_number for +compact-numbering slot release. + +Bundled operations (in forward order; reversed on rollback): + 1. AddField Channel.hidden_from_output + 2. CreateModel ChannelOverride (one-to-one Channel) + 3. AddField ChannelGroupM3UAccount.auto_sync_channel_end + 4. RunPython backfill_auto_created_by_null (orphan re-attribution / demotion) + 5. AlterField Channel.channel_number (nullable) + 6. RunPython noop / reverse_backfill_channel_number_nulls + (rollback-safety hook; runs FIRST on un-apply, fills NULLs + so step 5 reverse can re-impose NOT NULL) +""" + +import django.db.models.deletion +from django.db import migrations, models +from django.db.models import Max + + +def backfill_auto_created_by_null(apps, schema_editor): + """ + Re-attribute or demote `auto_created=True, auto_created_by=NULL` rows. + + Sync only touches rows where `auto_created_by=account`, so orphans + accumulate indefinitely. Best-effort re-attribute via the channel's + streams' single owning account; otherwise demote to manual by clearing + `auto_created`. The channel and any user customization survive; sync + will not touch a demoted row again. + """ + Channel = apps.get_model("dispatcharr_channels", "Channel") + ChannelStream = apps.get_model("dispatcharr_channels", "ChannelStream") + + orphans = Channel.objects.filter(auto_created=True, auto_created_by__isnull=True) + total = orphans.count() + if total == 0: + return + + print(f"\n Found {total} auto_created channels with NULL auto_created_by") + reattributed = 0 + demoted = 0 + + for channel in orphans.iterator(chunk_size=200): + account_ids = set( + ChannelStream.objects.filter(channel=channel) + .values_list("stream__m3u_account_id", flat=True) + ) + account_ids.discard(None) + + if len(account_ids) == 1: + channel.auto_created_by_id = next(iter(account_ids)) + channel.save(update_fields=["auto_created_by"]) + reattributed += 1 + else: + channel.auto_created = False + channel.save(update_fields=["auto_created"]) + demoted += 1 + + print( + f" Re-attributed: {reattributed}, demoted to manual " + f"(ambiguous/no streams): {demoted}" + ) + + +def reverse_auto_created_by_null(apps, schema_editor): + # Forward decisions cannot be cleanly reverted (no record of the + # original NULL state). Leaving the re-attributions and demotions in + # place is safer than restoring NULLs the schema may not accept. + pass + + +def noop(apps, schema_editor): + pass + + +def reverse_backfill_channel_number_nulls(apps, schema_editor): + """ + Rollback-safety hook for the channel_number nullable AlterField. + + Runs FIRST on un-apply (operations reverse in list order) and assigns + sequential channel numbers above the current max to any NULL rows so + the AlterField reverse (nullable to NOT NULL) succeeds without a + constraint violation. The user can re-hide or re-number these + channels after they have rolled back. + """ + Channel = apps.get_model("dispatcharr_channels", "Channel") + null_qs = Channel.objects.filter(channel_number__isnull=True) + null_count = null_qs.count() + if null_count == 0: + return + + max_num = Channel.objects.aggregate(m=Max("channel_number"))["m"] or 0.0 + next_num = float(max_num) + 1.0 + print( + f"\n Backfilling channel_number on {null_count} NULL row(s) " + f"starting at {int(next_num)} so rollback can re-impose NOT NULL" + ) + for ch in null_qs.order_by("id"): + ch.channel_number = next_num + ch.save(update_fields=["channel_number"]) + next_num += 1.0 + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '022_default_user_limit_settings'), + ('dispatcharr_channels', '0036_alter_stream_name'), + ('epg', '0022_alter_epgdata_name'), + ] + + operations = [ + migrations.AddField( + model_name='channel', + name='hidden_from_output', + field=models.BooleanField( + db_index=True, + default=False, + help_text='Exclude this channel from downstream client output (HDHR, M3U, EPG, XC). Auto-sync still updates provider metadata.', + ), + ), + migrations.CreateModel( + name='ChannelOverride', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('name', models.CharField(blank=True, max_length=512, null=True)), + ('channel_number', models.FloatField(blank=True, null=True)), + ('tvg_id', models.CharField(blank=True, max_length=255, null=True)), + ('tvc_guide_stationid', models.CharField(blank=True, max_length=255, null=True)), + ('created_at', models.DateTimeField(auto_now_add=True)), + ('updated_at', models.DateTimeField(auto_now=True)), + ('channel', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='override', to='dispatcharr_channels.channel')), + ('channel_group', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to='dispatcharr_channels.channelgroup')), + ('epg_data', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to='epg.epgdata')), + ('logo', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to='dispatcharr_channels.logo')), + ('stream_profile', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='+', to='core.streamprofile')), + ], + ), + migrations.AddField( + model_name='channelgroupm3uaccount', + name='auto_sync_channel_end', + field=models.FloatField( + blank=True, + help_text='Optional upper bound for auto-created channel numbers in this group. Leave blank for unlimited fill. Overflow streams are skipped and reported in the completion notification.', + null=True, + ), + ), + migrations.RunPython(backfill_auto_created_by_null, reverse_auto_created_by_null), + migrations.AlterField( + model_name='channel', + name='channel_number', + field=models.FloatField(blank=True, db_index=True, null=True), + ), + migrations.RunPython(noop, reverse_backfill_channel_number_nulls), + ] diff --git a/apps/channels/models.py b/apps/channels/models.py index c0de0fe1..7bf53b73 100644 --- a/apps/channels/models.py +++ b/apps/channels/models.py @@ -291,9 +291,23 @@ class ChannelManager(models.Manager): def active(self): return self.all() + def with_effective_values(self, select_related_fks=False): + """ + Chainable shortcut for the override-aware queryset annotations, + delegating to the canonical helper in ``apps.channels.managers`` + so the function form (``with_effective_values(qs)``) and the + manager form (``Channel.objects.with_effective_values()``) are + both valid entry points and stay in sync. + """ + from apps.channels.managers import with_effective_values + + return with_effective_values( + self.get_queryset(), select_related_fks=select_related_fks + ) + class Channel(models.Model): - channel_number = models.FloatField(db_index=True) + channel_number = models.FloatField(db_index=True, null=True, blank=True) name = models.CharField(max_length=512) logo = models.ForeignKey( "Logo", @@ -360,6 +374,16 @@ class Channel(models.Model): help_text="The M3U account that auto-created this channel" ) + # Hidden channels are excluded from HDHR, M3U, EPG, and XC output queries. + # Auto-sync still recognizes them so they are not recreated when their + # underlying provider stream persists; this is an output-layer concern, not + # a sync-time flag. + hidden_from_output = models.BooleanField( + default=False, + db_index=True, + help_text="Exclude this channel from downstream client output (HDHR, M3U, EPG, XC). Auto-sync still updates provider metadata." + ) + created_at = models.DateTimeField( auto_now_add=True, help_text="Timestamp when this channel was created" @@ -369,6 +393,8 @@ class Channel(models.Model): help_text="Timestamp when this channel was last updated" ) + objects = ChannelManager() + def clean(self): # Enforce unique channel_number within a given group existing = Channel.objects.filter( @@ -384,11 +410,46 @@ class Channel(models.Model): @classmethod def get_next_available_channel_number(cls, starting_from=1): - used_numbers = set(cls.objects.all().values_list("channel_number", flat=True)) - n = starting_from - while n in used_numbers: - n += 1 - return n + # Both raw and override channel numbers are reserved. Handing out a + # raw number currently masked by an override would create a deferred + # collision once the override is cleared. + from apps.channels.compact_numbering import build_reserved_set + from apps.m3u.tasks import _next_available_number + + reserved = build_reserved_set() + return _next_available_number(reserved, starting_from) + + def _resolved_override(self): + """Return the related ChannelOverride or None, tolerant of no row.""" + try: + return self.override + except ChannelOverride.DoesNotExist: + return None + + def _resolve_effective_fk(self, field_name): + """Pick the override's FK object if set, otherwise the channel's own.""" + override = self._resolved_override() + if override is not None: + override_val = getattr(override, field_name, None) + if override_val is not None: + return override_val + return getattr(self, field_name) + + @property + def effective_logo_obj(self): + return self._resolve_effective_fk("logo") + + @property + def effective_channel_group_obj(self): + return self._resolve_effective_fk("channel_group") + + @property + def effective_epg_data_obj(self): + return self._resolve_effective_fk("epg_data") + + @property + def effective_stream_profile_obj(self): + return self._resolve_effective_fk("stream_profile") # @TODO: honor stream's stream profile def get_stream_profile(self): @@ -840,6 +901,76 @@ class Channel(models.Model): return True +class ChannelOverride(models.Model): + """ + Per-field user overrides for auto-synced channels. + + Each nullable column represents a user-provided value that takes precedence + over the matching field on the related Channel. Sync writes only to + Channel.* fields and never to this table, so provider metadata keeps + flowing while user customizations persist across refreshes. Output + querysets resolve the effective value via + `apps.channels.managers.with_effective_values`. + """ + channel = models.OneToOneField( + Channel, + on_delete=models.CASCADE, + related_name="override", + ) + name = models.CharField(max_length=512, null=True, blank=True) + channel_number = models.FloatField(null=True, blank=True) + channel_group = models.ForeignKey( + ChannelGroup, + on_delete=models.SET_NULL, + null=True, + blank=True, + related_name="+", + ) + logo = models.ForeignKey( + "Logo", + on_delete=models.SET_NULL, + null=True, + blank=True, + related_name="+", + ) + tvg_id = models.CharField(max_length=255, null=True, blank=True) + tvc_guide_stationid = models.CharField(max_length=255, null=True, blank=True) + epg_data = models.ForeignKey( + EPGData, + on_delete=models.SET_NULL, + null=True, + blank=True, + related_name="+", + ) + stream_profile = models.ForeignKey( + StreamProfile, + on_delete=models.SET_NULL, + null=True, + blank=True, + related_name="+", + ) + created_at = models.DateTimeField(auto_now_add=True) + updated_at = models.DateTimeField(auto_now=True) + + def __str__(self): + return f"Override for channel {self.channel_id}" + + def has_any_override(self) -> bool: + return any( + getattr(self, field) is not None + for field in ( + "name", + "channel_number", + "channel_group_id", + "logo_id", + "tvg_id", + "tvc_guide_stationid", + "epg_data_id", + "stream_profile_id", + ) + ) + + class ChannelProfile(models.Model): name = models.CharField(max_length=100, unique=True) @@ -887,6 +1018,12 @@ class ChannelGroupM3UAccount(models.Model): blank=True, help_text='Starting channel number for auto-created channels in this group' ) + # Optional upper bound; out-of-range streams fail. NULL = unlimited. + auto_sync_channel_end = models.FloatField( + null=True, + blank=True, + help_text='Optional upper bound for auto-created channel numbers in this group. Leave blank for unlimited fill. Overflow streams are skipped and reported in the completion notification.' + ) last_seen = models.DateTimeField( default=timezone.now, db_index=True, diff --git a/apps/channels/serializers.py b/apps/channels/serializers.py index 039386a3..2c7fee3e 100644 --- a/apps/channels/serializers.py +++ b/apps/channels/serializers.py @@ -6,6 +6,7 @@ from .models import ( Stream, Channel, ChannelGroup, + ChannelOverride, ChannelStream, ChannelGroupM3UAccount, Logo, @@ -17,6 +18,7 @@ from .models import ( from apps.epg.serializers import EPGDataSerializer from core.models import StreamProfile from apps.epg.models import EPGData +from django.db import connection, transaction from django.urls import reverse from rest_framework import serializers from django.utils import timezone @@ -154,12 +156,52 @@ class ChannelGroupM3UAccountSerializer(serializers.ModelSerializer): m3u_accounts = serializers.IntegerField(source="m3u_accounts.id", read_only=True) enabled = serializers.BooleanField() auto_channel_sync = serializers.BooleanField(default=False) - auto_sync_channel_start = serializers.FloatField(allow_null=True, required=False) + auto_sync_channel_start = serializers.FloatField( + allow_null=True, required=False, min_value=1 + ) + auto_sync_channel_end = serializers.FloatField( + allow_null=True, required=False, min_value=1 + ) custom_properties = serializers.JSONField(required=False) + # Provider stream count for this group+account. Lets users size an + # optional end-range without first running a blind sync. + stream_count = serializers.SerializerMethodField() class Meta: model = ChannelGroupM3UAccount - fields = ["m3u_accounts", "channel_group", "enabled", "auto_channel_sync", "auto_sync_channel_start", "custom_properties", "is_stale", "last_seen"] + fields = [ + "m3u_accounts", + "channel_group", + "enabled", + "auto_channel_sync", + "auto_sync_channel_start", + "auto_sync_channel_end", + "custom_properties", + "is_stale", + "last_seen", + "stream_count", + ] + + def get_stream_count(self, obj): + """ + Return the number of streams for this (m3u_account, channel_group) + pair. A parent serializer (e.g. M3UAccountSerializer) may seed + ``context["stream_counts"]`` with a pre-aggregated dict keyed by + ``(m3u_account_id, channel_group_id)`` to avoid one COUNT per row; + when present, it is used as the source of truth. The per-row + COUNT fallback is correct for stand-alone serialization (rare, + low-volume) and exists so direct ChannelGroupM3UAccount queries + do not require callers to know the seeding pattern. + """ + counts = self.context.get("stream_counts") + if counts is not None: + return counts.get((obj.m3u_account_id, obj.channel_group_id), 0) + from apps.channels.models import Stream + + return Stream.objects.filter( + m3u_account_id=obj.m3u_account_id, + channel_group_id=obj.channel_group_id, + ).count() def to_representation(self, instance): data = super().to_representation(instance) @@ -179,6 +221,26 @@ class ChannelGroupM3UAccountSerializer(serializers.ModelSerializer): return super().to_internal_value(data) + def validate(self, attrs): + # Partial PATCHes only carry submitted fields; fill missing + # start/end from the instance so the validator catches a PATCH + # that lowers end past the existing start. + start = attrs.get("auto_sync_channel_start") + end = attrs.get("auto_sync_channel_end") + if start is None and self.instance is not None: + start = self.instance.auto_sync_channel_start + if end is None and self.instance is not None: + end = self.instance.auto_sync_channel_end + if start is not None and end is not None and end < start: + raise serializers.ValidationError( + { + "auto_sync_channel_end": ( + "End must be greater than or equal to start." + ) + } + ) + return super().validate(attrs) + # # Channel Group # @@ -195,12 +257,14 @@ class ChannelGroupSerializer(serializers.ModelSerializer): fields = ["id", "name", "channel_count", "m3u_account_count", "m3u_accounts"] def get_channel_count(self, obj): - """Get count of channels in this group""" - return obj.channels.count() + # Use the queryset annotation when available (list path); fall back + # to a live query for retrieve/create/update where it isn't set. + v = getattr(obj, 'channel_count', None) + return v if v is not None else obj.channels.count() def get_m3u_account_count(self, obj): - """Get count of M3U accounts associated with this group""" - return obj.m3u_accounts.count() + v = getattr(obj, 'm3u_account_count', None) + return v if v is not None else obj.m3u_accounts.count() class ChannelProfileSerializer(serializers.ModelSerializer): @@ -240,6 +304,62 @@ class BulkChannelProfileMembershipSerializer(serializers.Serializer): return value +# +# Channel override +# +# Nullable per-field overrides resolved over the parent Channel in read +# paths. Embedded in ChannelSerializer so clients can upsert/clear in the +# same PATCH that targets direct channel fields. +class ChannelOverrideSerializer(serializers.ModelSerializer): + # HDHR clients reject negative GuideNumber and zero is not a real + # provider value, so reject both at the API boundary. + channel_number = serializers.FloatField( + allow_null=True, required=False, min_value=0.0001 + ) + channel_group_id = serializers.PrimaryKeyRelatedField( + queryset=ChannelGroup.objects.all(), + source="channel_group", + allow_null=True, + required=False, + ) + logo_id = serializers.PrimaryKeyRelatedField( + queryset=Logo.objects.all(), + source="logo", + allow_null=True, + required=False, + ) + epg_data_id = serializers.PrimaryKeyRelatedField( + queryset=EPGData.objects.all(), + source="epg_data", + allow_null=True, + required=False, + ) + stream_profile_id = serializers.PrimaryKeyRelatedField( + queryset=StreamProfile.objects.all(), + source="stream_profile", + allow_null=True, + required=False, + ) + + class Meta: + model = ChannelOverride + fields = [ + "name", + "channel_number", + "channel_group_id", + "logo_id", + "tvg_id", + "tvc_guide_stationid", + "epg_data_id", + "stream_profile_id", + ] + extra_kwargs = { + "name": {"allow_null": True, "required": False}, + "tvg_id": {"allow_null": True, "required": False}, + "tvc_guide_stationid": {"allow_null": True, "required": False}, + } + + # # Channel # @@ -280,6 +400,32 @@ class ChannelSerializer(serializers.ModelSerializer): ) auto_created_by_name = serializers.SerializerMethodField() + override = ChannelOverrideSerializer( + required=False, + allow_null=True, + help_text=( + "Per-field overrides for an auto-created channel. " + 'Send {"override": {"name": "ESPN"}} to upsert the listed ' + 'fields, {"override": {"name": null}} to clear specific fields ' + 'while leaving others, or {"override": null} to delete the ' + "override row entirely. Omitting the key leaves any existing " + "override unchanged. Only valid for auto_created=True channels. " + "Duplicate channel_number values across channels are permitted; " + "downstream client behavior on duplicates varies by client." + ), + ) + source_stream = serializers.SerializerMethodField() + # Effective fields coalesce override over channel column. Consumers + # display these; raw fields remain in the response so the edit form + # can show them as "Provider: X" subtext. + effective_name = serializers.SerializerMethodField() + effective_channel_number = serializers.SerializerMethodField() + effective_channel_group_id = serializers.SerializerMethodField() + effective_logo_id = serializers.SerializerMethodField() + effective_tvg_id = serializers.SerializerMethodField() + effective_tvc_guide_stationid = serializers.SerializerMethodField() + effective_epg_data_id = serializers.SerializerMethodField() + effective_stream_profile_id = serializers.SerializerMethodField() class Meta: model = Channel @@ -297,11 +443,88 @@ class ChannelSerializer(serializers.ModelSerializer): "logo_id", "user_level", "is_adult", + "hidden_from_output", "auto_created", "auto_created_by", "auto_created_by_name", + "override", + "source_stream", + "effective_name", + "effective_channel_number", + "effective_channel_group_id", + "effective_logo_id", + "effective_tvg_id", + "effective_tvc_guide_stationid", + "effective_epg_data_id", + "effective_stream_profile_id", ] + def _effective_value(self, obj, field_name): + override = getattr(obj, "_channel_override_cache", None) + if override is None: + try: + override = obj.override + except ChannelOverride.DoesNotExist: + override = None + obj._channel_override_cache = override + if override is not None: + value = getattr(override, field_name, None) + if value is not None: + return value + return getattr(obj, field_name, None) + + def get_effective_name(self, obj): + return self._effective_value(obj, "name") + + def get_effective_channel_number(self, obj): + return self._effective_value(obj, "channel_number") + + def get_effective_channel_group_id(self, obj): + return self._effective_value(obj, "channel_group_id") + + def get_effective_logo_id(self, obj): + return self._effective_value(obj, "logo_id") + + def get_effective_tvg_id(self, obj): + return self._effective_value(obj, "tvg_id") + + def get_effective_tvc_guide_stationid(self, obj): + return self._effective_value(obj, "tvc_guide_stationid") + + def get_effective_epg_data_id(self, obj): + return self._effective_value(obj, "epg_data_id") + + def get_effective_stream_profile_id(self, obj): + return self._effective_value(obj, "stream_profile_id") + + def get_source_stream(self, obj): + """ + Return the originating provider stream for an auto-created channel. + + Surfaces the provider stream's name and owning M3U account so the + frontend can render "Auto-created from: / " + in the channel edit form. Returns None for manual channels. + """ + if not self.context.get("include_source_stream", False): + return None + if not obj.auto_created: + return None + # Viewset prefetches `channelstream_set` ordered by `order`, so + # `.all()[0]` reuses the cache and returns the lowest-order entry. + prefetched_list = list(obj.channelstream_set.all()) + if not prefetched_list: + return None + cs = prefetched_list[0] + if not cs.stream: + return None + stream = cs.stream + return { + "id": stream.id, + "name": stream.name, + "account_id": stream.m3u_account_id, + "account_name": getattr(stream.m3u_account, "name", None), + } + def to_representation(self, instance): include_streams = self.context.get("include_streams", False) @@ -309,14 +532,14 @@ class ChannelSerializer(serializers.ModelSerializer): self.fields["streams"] = serializers.SerializerMethodField() return super().to_representation(instance) else: - # Fix: For PATCH/PUT responses, ensure streams are ordered + # Read from the prefetched channelstream_set (ordered by the + # viewset's Prefetch); chaining .order_by() rebuilds the + # queryset and fires one SELECT per row in list responses. representation = super().to_representation(instance) if "streams" in representation: - representation["streams"] = list( - instance.streams.all() - .order_by("channelstream__order") - .values_list("id", flat=True) - ) + representation["streams"] = [ + cs.stream_id for cs in instance.channelstream_set.all() + ] return representation def get_logo(self, obj): @@ -330,6 +553,7 @@ class ChannelSerializer(serializers.ModelSerializer): def create(self, validated_data): streams = validated_data.pop("streams", []) + override_data = validated_data.pop("override", None) channel_number = validated_data.pop( "channel_number", Channel.get_next_available_channel_number() ) @@ -341,60 +565,146 @@ class ChannelSerializer(serializers.ModelSerializer): default_group, _ = ChannelGroup.objects.get_or_create(name="Default Group") validated_data["channel_group"] = default_group - channel = Channel.objects.create(**validated_data) + # Atomic wrapper keeps the channel insert and its override row + # in the same transaction so a failure on either rolls both back. + with transaction.atomic(): + channel = Channel.objects.create(**validated_data) - # Add streams in the specified order - for index, stream in enumerate(streams): - ChannelStream.objects.create( - channel=channel, stream_id=stream.id, order=index - ) + # Add streams in the specified order + for index, stream in enumerate(streams): + ChannelStream.objects.create( + channel=channel, stream_id=stream.id, order=index + ) + + if override_data: + # Manual channels (auto_created=False) have no provider + # value to override; reject the override payload here so a + # programmatic client can't write a semantically meaningless + # row that the frontend would then surface as "Overrides + # active". + if not channel.auto_created: + raise serializers.ValidationError( + { + "override": ( + "Cannot set override on a manual channel; " + "overrides only apply to auto-created channels." + ) + } + ) + obj = ChannelOverride.objects.create(channel=channel, **override_data) + # Drop an all-null override row; an empty override would + # falsely surface as active in the UI. + if not obj.has_any_override(): + obj.delete() return channel def update(self, instance, validated_data): + """ + PATCH handler for Channel rows. The ``override`` key carries + per-field user overrides for auto-created channels and follows + these rules: + + * key absent from payload: no change to existing overrides + * ``{"override": {"field": value}}``: upsert those fields + * ``{"override": {"field": null}}``: clear those specific fields + * ``{"override": null}``: delete the override row entirely + + Key presence is what distinguishes "no change" from "delete"; + an explicit null means delete. Override mutations are rejected + on manual channels (auto_created=False) since there is no + provider value to override. + """ streams = validated_data.pop("streams", None) + has_override_key = "override" in self.initial_data + override_data = validated_data.pop("override", None) - # Update standard fields - for attr, value in validated_data.items(): - setattr(instance, attr, value) - - instance.save() - - if streams is not None: - # Normalize stream IDs - normalized_ids = [ - stream.id if hasattr(stream, "id") else stream for stream in streams - ] - - # Get current mapping of stream_id -> ChannelStream - current_links = { - cs.stream_id: cs for cs in instance.channelstream_set.all() - } - - # Track existing stream IDs - existing_ids = set(current_links.keys()) - new_ids = set(normalized_ids) - - # Delete any links not in the new list - to_remove = existing_ids - new_ids - if to_remove: - instance.channelstream_set.filter(stream_id__in=to_remove).delete() - - # Update or create with new order - to_update = [] - for order, stream_id in enumerate(normalized_ids): - if stream_id in current_links: - cs = current_links[stream_id] - if cs.order != order: - cs.order = order - to_update.append(cs) - else: - ChannelStream.objects.create( - channel=instance, stream_id=stream_id, order=order + # Block override mutations on manual channels (no provider + # value to override). Clearing is a tolerated no-op. + if ( + has_override_key + and override_data is not None + and override_data != {} + and not instance.auto_created + ): + raise serializers.ValidationError( + { + "override": ( + "Cannot set override on a manual channel; " + "overrides only apply to auto-created channels." ) + } + ) - if to_update: - ChannelStream.objects.bulk_update(to_update, ["order"]) + # Atomic so a failure on the override row rolls back the + # channel update too. + with transaction.atomic(): + # Skip save() when only override keys were submitted; a + # no-op UPDATE would bump updated_at and bust caches. + if validated_data: + for attr, value in validated_data.items(): + setattr(instance, attr, value) + instance.save() + + if has_override_key: + if override_data is None: + # Explicit null: remove the override row. + ChannelOverride.objects.filter(channel=instance).delete() + elif override_data == {}: + # Empty dict has no field intent; no-op. + pass + else: + obj, _ = ChannelOverride.objects.update_or_create( + channel=instance, defaults=override_data + ) + # Drop an all-null override; would falsely surface + # as active in the UI. + if not obj.has_any_override(): + obj.delete() + # Queryset writes leave the reverse-OneToOne cache stale; + # clear it so to_representation reads the new state. + try: + instance._state.fields_cache.pop("override", None) + except AttributeError: + pass + if hasattr(instance, "_channel_override_cache"): + delattr(instance, "_channel_override_cache") + + if streams is not None: + # Normalize stream IDs + normalized_ids = [ + stream.id if hasattr(stream, "id") else stream for stream in streams + ] + + # Get current mapping of stream_id -> ChannelStream + current_links = { + cs.stream_id: cs for cs in instance.channelstream_set.all() + } + + # Track existing stream IDs + existing_ids = set(current_links.keys()) + new_ids = set(normalized_ids) + + # Delete any links not in the new list + to_remove = existing_ids - new_ids + if to_remove: + instance.channelstream_set.filter(stream_id__in=to_remove).delete() + + # Update or create with new order + to_update = [] + for order, stream_id in enumerate(normalized_ids): + if stream_id in current_links: + cs = current_links[stream_id] + if cs.order != order: + cs.order = order + to_update.append(cs) + else: + ChannelStream.objects.create( + channel=instance, stream_id=stream_id, order=order + ) + + if to_update: + ChannelStream.objects.bulk_update(to_update, ["order"]) return instance diff --git a/apps/channels/signals.py b/apps/channels/signals.py index dd2ac158..2eecbebf 100644 --- a/apps/channels/signals.py +++ b/apps/channels/signals.py @@ -1,6 +1,6 @@ # apps/channels/signals.py -from django.db.models.signals import m2m_changed, pre_save, post_save, post_delete +from django.db.models.signals import m2m_changed, pre_save, post_save, post_delete, pre_delete from django.dispatch import receiver from django.utils.timezone import now, is_aware, make_aware from celery.result import AsyncResult @@ -60,6 +60,137 @@ def generate_custom_stream_hash(sender, instance, created, **kwargs): # Use update to avoid triggering signals again Stream.objects.filter(id=instance.id).update(stream_hash=instance.stream_hash) +@receiver(pre_delete, sender=Channel) +def stop_proxy_session_before_channel_delete(sender, instance, **kwargs): + """ + When a Channel is deleted, stop any active TS proxy session for it first. + + Without this, the proxy's Redis state (ts_proxy:channel:{uuid}:*) survives + the Channel row and the connected clients' "Stop" button hits + `ChannelService.stop_channel(uuid)`, which calls `Channel.objects.get(uuid=...)` + and crashes with DoesNotExist (reported as 'Channel not found' in the UI). + Users then cannot close the stream; source-side connection limits stay + consumed. Covers manual deletes, bulk deletes, and sync-driven deletes + via the same signal path. + """ + try: + from apps.proxy.ts_proxy.services.channel_service import ChannelService + + channel_uuid = str(instance.uuid) if instance.uuid else None + if not channel_uuid: + return + # Best-effort: if the channel has no active session the service + # returns a benign 'Channel not found' result, which is ignored. + ChannelService.stop_channel(channel_uuid) + except Exception as e: + # Never block a channel delete on proxy cleanup failure. Log and + # continue so at least the DB row is removed. + logger.warning( + "Failed to stop proxy session before deleting channel %s: %s", + getattr(instance, "id", ""), + e, + ) + + +@receiver(post_save, sender=Channel) +def assign_compact_number_on_unhide(sender, instance, created, **kwargs): + """When a channel transitions from hidden to visible under compact + numbering, immediately assign the next available number from its + group's range. Without this, an unhide would leave the channel at + NULL until the next M3U refresh, which is too long a delay for what + is meant to feel like an instant action. + + Bails out for the common cases where assignment is not appropriate: + a fresh channel (newly created), a channel that already has a + number, a hidden channel, a manual channel, or a channel whose group + is not in compact mode. The assignment helper does the same checks + defensively, but bailing here keeps the signal out of the helper + code path on every channel save. + """ + if created: + return + # Skip the signal when update_fields proves hidden_from_output was not + # touched. Sync sets update_fields on every save, so this keeps the + # signal off the sync hot path. + update_fields = kwargs.get("update_fields") + if update_fields is not None and "hidden_from_output" not in update_fields: + return + if instance.hidden_from_output: + return + if instance.channel_number is not None: + return + if not instance.auto_created or not instance.auto_created_by_id: + return + try: + from .compact_numbering import assign_compact_numbers_for_channels + + assign_compact_numbers_for_channels([instance.id]) + # The helper writes via queryset .update() which skips + # in-memory state. Reload so a same-request serializer + # response carries the assigned number, not stale None. + try: + instance.refresh_from_db(fields=["channel_number"]) + except Exception: + # Refresh failure (race with delete) is non-fatal. + pass + except Exception as e: + # Do not propagate. The save succeeded; the assignment is + # recoverable on next sync or manual repack. + logger.warning( + "Compact unhide assignment failed for channel %s: %s", + instance.id, + e, + ) + + +@receiver(post_save, sender=Channel) +def release_compact_number_on_hide(sender, instance, created, **kwargs): + """When a channel transitions from visible to hidden under compact + numbering, immediately release its channel_number slot. Without + this, the slot stays occupied until the next sync's repack pass, + so the user sees their hidden channels still consuming numbers in + the table for an indeterminate window. Mirror image of + `assign_compact_number_on_unhide` above. + + Bails out for the common cases where release is not appropriate: + a fresh channel (newly created), a non-hidden channel, a channel + that has no number to release, a manual channel, a channel without + a known auto_created_by, or a channel whose group is not in + compact mode. + """ + if created: + return + update_fields = kwargs.get("update_fields") + if update_fields is not None and "hidden_from_output" not in update_fields: + return + if not instance.hidden_from_output: + return + if instance.channel_number is None: + return + if not instance.auto_created or not instance.auto_created_by_id: + return + try: + from .compact_numbering import ( + get_group_relation_for_channel, + is_compact_group, + ) + + relation = get_group_relation_for_channel(instance) + if not relation or not is_compact_group(relation): + return + except Exception: + return + # Release the slot. Queryset .update() bypasses the post_save signal + # chain that this handler is itself running inside. + Channel.objects.filter(id=instance.id).update(channel_number=None) + # Refresh in-memory instance so the same-request serializer + # response surfaces channel_number=None instead of stale value. + try: + instance.refresh_from_db(fields=["channel_number"]) + except Exception: + pass + + @receiver(post_save, sender=Channel) def refresh_epg_programs(sender, instance, created, **kwargs): """ diff --git a/apps/channels/tests/test_channel_api.py b/apps/channels/tests/test_channel_api.py index bb245da1..052560a0 100644 --- a/apps/channels/tests/test_channel_api.py +++ b/apps/channels/tests/test_channel_api.py @@ -3,7 +3,7 @@ from django.contrib.auth import get_user_model from rest_framework.test import APIClient from rest_framework import status -from apps.channels.models import Channel, ChannelGroup +from apps.channels.models import Channel, ChannelGroup, ChannelOverride User = get_user_model() @@ -209,3 +209,133 @@ class ChannelBulkEditAPITests(TestCase): self.assertEqual(self.channel1.name, "Only Name Changed") self.assertEqual(self.channel1.channel_number, original_channel_number) self.assertEqual(self.channel1.tvg_id, original_tvg_id) + + def test_bulk_swap_clear_and_assign_same_number(self): + # User clears channel A's override (which currently pins #10) and + # in the same bulk request sets channel B's override.channel_number + # to #10. Both halves of the swap must succeed; the resulting + # state has A unpinned and B pinned at #10. + auto_a = Channel.objects.create( + channel_number=1.0, + name="Auto A", + tvg_id="auto_a", + channel_group=self.group1, + auto_created=True, + ) + ChannelOverride.objects.create(channel=auto_a, channel_number=10.0) + auto_b = Channel.objects.create( + channel_number=2.0, + name="Auto B", + tvg_id="auto_b", + channel_group=self.group1, + auto_created=True, + ) + + data = [ + {"id": auto_a.id, "override": None}, + {"id": auto_b.id, "override": {"channel_number": 10.0}}, + ] + response = self.client.patch(self.bulk_edit_url, data, format="json") + + self.assertEqual( + response.status_code, + status.HTTP_200_OK, + f"Expected 200; got {response.status_code} body={response.data}", + ) + self.assertFalse( + ChannelOverride.objects.filter(channel=auto_a).exists() + ) + b_override = ChannelOverride.objects.get(channel=auto_b) + self.assertEqual(b_override.channel_number, 10.0) + + +class ChannelSummaryEffectiveValuesTests(TestCase): + """ + The /api/channels/channels/summary/ endpoint feeds the TV Guide. + Like every downstream output surface, it must reflect the user's + overrides (name, channel_number, logo_id, epg_data_id, + channel_group_id) instead of the raw provider values, otherwise + the in-app guide would silently disagree with HDHR / M3U / EPG / + XC clients on the same channel set. + """ + + def setUp(self): + from django.contrib.auth import get_user_model + from rest_framework.test import APIClient + from apps.channels.models import ChannelOverride + + User = get_user_model() + self.user = User.objects.create_user( + username="summary_admin", password="x" + ) + self.user.user_level = 10 + self.user.save() + self.client = APIClient() + self.client.force_authenticate(user=self.user) + + self.group = ChannelGroup.objects.create(name="Summary Group") + self.other_group = ChannelGroup.objects.create(name="Other") + self.channel = Channel.objects.create( + channel_number=10.0, + name="Provider Name", + channel_group=self.group, + auto_created=True, + ) + ChannelOverride.objects.create( + channel=self.channel, + name="Override Name", + channel_number=99.0, + channel_group=self.other_group, + ) + + def test_summary_returns_effective_values(self): + response = self.client.get("/api/channels/channels/summary/") + self.assertEqual(response.status_code, 200) + row = next(r for r in response.data if r["id"] == self.channel.id) + self.assertEqual(row["name"], "Override Name") + self.assertEqual(row["channel_number"], 99.0) + self.assertEqual(row["channel_group_id"], self.other_group.id) + + +class ChannelManagerEffectiveValuesTests(TestCase): + """ + The chainable ``Channel.objects.with_effective_values()`` shortcut + must return rows with the same ``effective_*`` annotations the + module-level helper produces, since both forms are documented + entry points and a divergence would silently change output for + one set of callers. + """ + + def test_manager_shortcut_matches_module_helper(self): + from apps.channels.managers import with_effective_values + + group = ChannelGroup.objects.create(name="Manager Test") + channel = Channel.objects.create( + channel_number=42.0, + name="Original Name", + channel_group=group, + auto_created=True, + ) + ChannelOverride.objects.create( + channel=channel, + name="Renamed", + channel_number=99.0, + ) + + helper_row = with_effective_values( + Channel.objects.filter(id=channel.id) + ).get() + shortcut_row = ( + Channel.objects.with_effective_values() + .filter(id=channel.id) + .get() + ) + + self.assertEqual(helper_row.effective_name, "Renamed") + self.assertEqual(shortcut_row.effective_name, "Renamed") + self.assertEqual(helper_row.effective_channel_number, 99.0) + self.assertEqual(shortcut_row.effective_channel_number, 99.0) + self.assertEqual( + helper_row.effective_channel_group_id, + shortcut_row.effective_channel_group_id, + ) diff --git a/apps/channels/utils.py b/apps/channels/utils.py index 70358cee..f41c2b72 100644 --- a/apps/channels/utils.py +++ b/apps/channels/utils.py @@ -4,6 +4,18 @@ lock = threading.Lock() # Dictionary to track usage: {account_id: current_usage} active_streams_map = {} + +def format_channel_number(value, empty=""): + """Display formatting for an effective channel_number. Returns int for + whole-valued floats (so ``123.0`` renders as ``123``), the float as-is + for fractional values, or ``empty`` when the value is ``None``. + """ + if value is None: + return empty + if value == int(value): + return int(value) + return value + def increment_stream_count(account): with lock: current_usage = active_streams_map.get(account.id, 0) diff --git a/apps/hdhr/api_views.py b/apps/hdhr/api_views.py index b1d0e852..41c1a3d9 100644 --- a/apps/hdhr/api_views.py +++ b/apps/hdhr/api_views.py @@ -125,30 +125,38 @@ class LineupAPIView(APIView): if blocked is not None: return blocked + from apps.channels.managers import with_effective_values + from apps.channels.utils import format_channel_number + if profile is not None: channel_profile = ChannelProfile.objects.get(name=profile) - channels = Channel.objects.filter( + base_qs = Channel.objects.filter( channelprofilemembership__channel_profile=channel_profile, channelprofilemembership__enabled=True, - ).order_by("channel_number") + ) else: - channels = Channel.objects.all().order_by("channel_number") + base_qs = Channel.objects.all() + + channels = ( + with_effective_values(base_qs) + .exclude(hidden_from_output=True) + .order_by("effective_channel_number") + ) lineup = [] for ch in channels: - # Format channel number as integer if it has no decimal component - if ch.channel_number is not None: - if ch.channel_number == int(ch.channel_number): - formatted_channel_number = str(int(ch.channel_number)) - else: - formatted_channel_number = str(ch.channel_number) - else: - formatted_channel_number = "" + # HDHR clients reject lineup entries with empty/non-numeric + # GuideNumber and may drop the whole lineup. With nullable + # channel_number, skip rows that have no usable number. + formatted = format_channel_number(ch.effective_channel_number, empty=None) + if formatted is None: + continue + formatted_channel_number = str(formatted) lineup.append( { "GuideNumber": formatted_channel_number, - "GuideName": ch.name, + "GuideName": ch.effective_name, "URL": request.build_absolute_uri(f"/proxy/ts/stream/{ch.uuid}"), "Guide_ID": formatted_channel_number, "Station": formatted_channel_number, diff --git a/apps/hdhr/views.py b/apps/hdhr/views.py index f9dd42d2..6777bdbf 100644 --- a/apps/hdhr/views.py +++ b/apps/hdhr/views.py @@ -84,15 +84,27 @@ class LineupAPIView(APIView): description="Retrieve the available channel lineup", ) def get(self, request): - channels = Channel.objects.all().order_by("channel_number") - lineup = [ - { - "GuideNumber": str(ch.channel_number), - "GuideName": ch.name, - "URL": request.build_absolute_uri(f"/proxy/ts/stream/{ch.uuid}"), - } - for ch in channels - ] + from apps.channels.managers import with_effective_values + from apps.channels.utils import format_channel_number + + channels = ( + with_effective_values(Channel.objects.all()) + .exclude(hidden_from_output=True) + .order_by("effective_channel_number") + ) + lineup = [] + for ch in channels: + formatted = format_channel_number(ch.effective_channel_number, empty=None) + if formatted is None: + continue + formatted_channel_number = str(formatted) + lineup.append( + { + "GuideNumber": formatted_channel_number, + "GuideName": ch.effective_name, + "URL": request.build_absolute_uri(f"/proxy/ts/stream/{ch.uuid}"), + } + ) return JsonResponse(lineup, safe=False) diff --git a/apps/m3u/api_views.py b/apps/m3u/api_views.py index b8cb099a..c09b668f 100644 --- a/apps/m3u/api_views.py +++ b/apps/m3u/api_views.py @@ -17,6 +17,9 @@ from rest_framework.decorators import action from django.conf import settings from .tasks import refresh_m3u_groups import json +import logging + +logger = logging.getLogger(__name__) from .models import M3UAccount, M3UFilter, ServerGroup, M3UAccountProfile from core.models import UserAgent @@ -41,7 +44,7 @@ class M3UAccountViewSet(viewsets.ModelViewSet): queryset = M3UAccount.objects.select_related( "refresh_task__crontab", "refresh_task__interval" - ).prefetch_related("channel_group", "profiles") + ).prefetch_related("channel_group", "profiles", "filters") serializer_class = M3UAccountSerializer def get_permissions(self): @@ -50,6 +53,41 @@ class M3UAccountViewSet(viewsets.ModelViewSet): except KeyError: return [Authenticated()] + def list(self, request, *args, **kwargs): + queryset = self.filter_queryset(self.get_queryset()) + + # Pre-aggregate stream counts for all accounts in one query so the + # nested ChannelGroupM3UAccountSerializer never issues a COUNT per + # group row. The serializer checks for this key and skips its own + # per-instance query when it is present. + from apps.channels.models import Stream + from django.db.models import Count + + account_ids = list(queryset.values_list("id", flat=True)) + counts_qs = ( + Stream.objects.filter(m3u_account_id__in=account_ids) + .values("m3u_account_id", "channel_group_id") + .annotate(c=Count("id")) + ) + stream_counts = { + (row["m3u_account_id"], row["channel_group_id"]): row["c"] + for row in counts_qs + } + + page = self.paginate_queryset(queryset) + if page is not None: + serializer = self.get_serializer( + page, many=True, + context={**self.get_serializer_context(), "stream_counts": stream_counts}, + ) + return self.get_paginated_response(serializer.data) + + serializer = self.get_serializer( + queryset, many=True, + context={**self.get_serializer_context(), "stream_counts": stream_counts}, + ) + return Response(serializer.data) + def create(self, request, *args, **kwargs): # Handle file upload first, if any file_path = None @@ -225,6 +263,198 @@ class M3UAccountViewSet(viewsets.ModelViewSet): # Continue with regular partial update return super().partial_update(request, *args, **kwargs) + def destroy(self, request, *args, **kwargs): + """ + Delete an M3U account and all auto-created channels attributed + to it. Auto-created channels with no surviving provider have no + useful state (they cannot sync, their streams are about to + cascade away), so the delete is unconditional: the only + question for the user is whether to confirm. Manual channels + are untouched, even if they include streams from this account; + those streams cascade away independently and the channels + survive with their other streams. The legacy + ``?cleanup_channels`` query parameter is accepted for backward + compatibility but ignored. + """ + instance = self.get_object() + from apps.channels.models import Channel + from apps.proxy.ts_proxy.services.channel_service import ( + ChannelService, + ) + + # Snapshot channels so proxy sessions can be stopped outside + # the DB transaction. The pre_delete signal would otherwise + # fire ChannelService.stop_channel (Redis pub / hgetall / + # setex) per channel inside the atomic, holding the DB + # connection across thousands of blocking RPCs and gumming up + # the connection pool. + channels_to_delete = list( + Channel.objects.filter( + auto_created=True, + auto_created_by=instance, + ).values_list("id", "uuid") + ) + for _, channel_uuid in channels_to_delete: + if not channel_uuid: + continue + try: + ChannelService.stop_channel(str(channel_uuid)) + except Exception as e: + logger.warning( + "Failed to stop proxy session for channel %s " + "during account cleanup: %s", + channel_uuid, + e, + ) + + channel_ids = [cid for cid, _ in channels_to_delete] + # Channel + account writes share an atomic so an account + # delete failure rolls back the channel deletes too. The + # pre_delete signal will fire again here but its proxy stop + # is fast on already-stopped channels (a single Redis check + # returns "not found" immediately). + with transaction.atomic(): + if channel_ids: + _, per_model = Channel.objects.filter( + id__in=channel_ids + ).delete() + deleted_channels = per_model.get( + "dispatcharr_channels.Channel", 0 + ) + else: + deleted_channels = 0 + response = super().destroy(request, *args, **kwargs) + + # Surface the channel count alongside the standard 204; the + # confirmation toast renders the number to acknowledge what + # the cascade actually removed. + if response.status_code == status.HTTP_204_NO_CONTENT: + return Response( + {"deleted_channels": deleted_channels}, + status=status.HTTP_200_OK, + ) + return response + + @extend_schema( + responses={ + 200: { + "type": "object", + "properties": { + "count": {"type": "integer"}, + "sample_names": {"type": "array", "items": {"type": "string"}}, + }, + }, + }, + ) + @action(detail=True, methods=["get"], url_path="auto-created-channels-count") + def auto_created_channels_count(self, request, pk=None): + """ + Preview how many auto-created channels would be removed if the account + were deleted with cleanup_channels=true. The frontend calls this when + the user clicks Delete, to render a truthful confirmation dialog + ("Also delete N channels auto-created by this provider?"). + """ + account = self.get_object() + from apps.channels.models import Channel + + qs = Channel.objects.filter( + auto_created=True, auto_created_by=account + ) + count = qs.count() + sample_names = list(qs.values_list("name", flat=True)[:5]) + return Response({"count": count, "sample_names": sample_names}) + + @extend_schema( + parameters=[ + OpenApiParameter( + name="channel_group_id", + type=OpenApiTypes.INT, + location=OpenApiParameter.QUERY, + required=True, + description=( + "ID of the ChannelGroup whose auto-created channels " + "should be repacked." + ), + ), + ], + responses={ + 200: { + "type": "object", + "properties": { + "assigned": {"type": "integer"}, + "released": {"type": "integer"}, + "failed": {"type": "integer"}, + }, + }, + }, + ) + @action(detail=True, methods=["post"], url_path="repack-group") + def repack_group(self, request, pk=None): + """ + Manually re-pack visible channels in one of this account's + groups into the group's [start, end] range. Override-pinned + numbers are treated as reservations and skipped. Hidden channels + without overrides have their channel_number set to NULL. + + Useful when the user has just finished customizing channels + (setting overrides as pins, hiding unwanted streams) and wants + the result reflected immediately rather than on the next M3U + refresh. Also acts as a one-shot cleanup for groups that aren't + running in compact mode but have accumulated gaps. + """ + account = self.get_object() + group_id_raw = request.query_params.get("channel_group_id") + if not group_id_raw: + return Response( + {"detail": "channel_group_id is required"}, + status=status.HTTP_400_BAD_REQUEST, + ) + try: + group_id = int(group_id_raw) + except (TypeError, ValueError): + return Response( + {"detail": "channel_group_id must be an integer"}, + status=status.HTTP_400_BAD_REQUEST, + ) + + from apps.channels.models import ChannelGroupM3UAccount + from apps.channels.compact_numbering import repack_group as _repack + from core.utils import acquire_task_lock, release_task_lock + + # Share the lock that wraps the entire refresh-plus-sync pipeline + # (`refresh_single_m3u_account`). The narrower + # `refresh_m3u_account_groups` lock is released before + # `sync_auto_channels` runs, so it would not protect this writer + # from racing against the channel_number writes inside sync. + if not acquire_task_lock("refresh_single_m3u_account", account.id): + return Response( + {"detail": "An M3U refresh is in progress for this account."}, + status=status.HTTP_409_CONFLICT, + ) + try: + # Re-fetch under the lock so a sync that just released its lock + # cannot leave the cached group_relation reflecting pre-sync + # custom_properties (auto_sync_channel_start/end, etc.). + try: + group_relation = ChannelGroupM3UAccount.objects.get( + m3u_account=account, channel_group_id=group_id + ) + except ChannelGroupM3UAccount.DoesNotExist: + return Response( + {"detail": "Group is not associated with this account"}, + status=status.HTTP_404_NOT_FOUND, + ) + result = _repack(group_relation) + finally: + try: + release_task_lock("refresh_single_m3u_account", account.id) + except Exception as e: + logger.warning( + f"Failed to release repack lock for account " + f"{account.id}: {e}" + ) + return Response(result) + @action(detail=True, methods=["post"], url_path="refresh-vod") def refresh_vod(self, request, pk=None): """Trigger VOD content refresh for XtreamCodes accounts""" @@ -270,6 +500,33 @@ class M3UAccountViewSet(viewsets.ModelViewSet): category_settings = request.data.get("category_settings", []) try: + for setting in group_settings: + start = setting.get("auto_sync_channel_start") + end = setting.get("auto_sync_channel_end") + if (start is not None and start < 1) or ( + end is not None and end < 1 + ): + return Response( + { + "error": ( + f"Channel group {setting.get('channel_group')}: " + f"channel range must be >= 1." + ) + }, + status=status.HTTP_400_BAD_REQUEST, + ) + if start is not None and end is not None and end < start: + return Response( + { + "error": ( + f"Channel group {setting.get('channel_group')}: " + f"auto_sync_channel_end must be >= " + f"auto_sync_channel_start." + ) + }, + status=status.HTTP_400_BAD_REQUEST, + ) + with transaction.atomic(): group_objects = [ ChannelGroupM3UAccount( @@ -278,6 +535,7 @@ class M3UAccountViewSet(viewsets.ModelViewSet): enabled=setting.get("enabled", True), auto_channel_sync=setting.get("auto_channel_sync", False), auto_sync_channel_start=setting.get("auto_sync_channel_start"), + auto_sync_channel_end=setting.get("auto_sync_channel_end"), custom_properties=setting.get("custom_properties", {}), ) for setting in group_settings @@ -293,6 +551,7 @@ class M3UAccountViewSet(viewsets.ModelViewSet): "enabled", "auto_channel_sync", "auto_sync_channel_start", + "auto_sync_channel_end", "custom_properties", ], ) diff --git a/apps/m3u/models.py b/apps/m3u/models.py index c1cb5a99..6fc48bec 100644 --- a/apps/m3u/models.py +++ b/apps/m3u/models.py @@ -100,7 +100,6 @@ class M3UAccount(models.Model): default=0, help_text="Priority for VOD provider selection (higher numbers = higher priority). Used when multiple providers offer the same content.", ) - def __str__(self): return self.name diff --git a/apps/m3u/serializers.py b/apps/m3u/serializers.py index f72566cf..485a3687 100644 --- a/apps/m3u/serializers.py +++ b/apps/m3u/serializers.py @@ -193,6 +193,24 @@ class M3UAccountSerializer(serializers.ModelSerializer): } def to_representation(self, instance): + # When the list() view pre-aggregates stream counts for all accounts + # in a single query, it seeds "stream_counts" into the context before + # serialization. Avoid issuing a redundant per-instance COUNT in that + # case. The per-instance fallback handles direct serialization (e.g. + # retrieve, create) where only one account is in scope. + if "stream_counts" not in self.context: + from django.db.models import Count + from apps.channels.models import Stream + + counts_qs = ( + Stream.objects.filter(m3u_account_id=instance.id) + .values("channel_group_id") + .annotate(c=Count("id")) + ) + self.context["stream_counts"] = { + (instance.id, row["channel_group_id"]): row["c"] for row in counts_qs + } + data = super().to_representation(instance) # Parse custom_properties to get VOD preference and auto_enable_new_groups settings @@ -247,10 +265,17 @@ class M3UAccountSerializer(serializers.ModelSerializer): auto_enable_new_groups_vod = validated_data.pop("auto_enable_new_groups_vod", None) auto_enable_new_groups_series = validated_data.pop("auto_enable_new_groups_series", None) - # Get existing custom_properties - custom_props = instance.custom_properties or {} + # Merge client-supplied custom_properties over the existing blob + # so unrelated keys persist. The dedicated preference fields below + # overwrite their corresponding keys; clients should set those via + # the typed top-level fields rather than the custom_properties + # payload. + incoming_custom = validated_data.get("custom_properties") or {} + custom_props = { + **(instance.custom_properties or {}), + **incoming_custom, + } - # Update preferences if enable_vod is not None: custom_props["enable_vod"] = enable_vod if auto_enable_new_groups_live is not None: @@ -345,7 +370,9 @@ class M3UAccountSerializer(serializers.ModelSerializer): return instance def get_filters(self, obj): - filters = obj.filters.order_by("order") + # Sort over the prefetch cache; .order_by() would fire one SELECT + # per account (viewset prefetches "filters"). + filters = sorted(obj.filters.all(), key=lambda f: f.order) return M3UFilterSerializer(filters, many=True).data def get_earliest_expiration(self, obj): diff --git a/apps/m3u/tasks.py b/apps/m3u/tasks.py index 0a441e18..4599d688 100644 --- a/apps/m3u/tasks.py +++ b/apps/m3u/tasks.py @@ -1686,6 +1686,113 @@ def delete_m3u_refresh_task_by_id(account_id): return False +def _next_available_number(used_numbers, start, end=None): + """ + Return the smallest integer >= start that is not present in `used_numbers`. + + When `end` is provided (inclusive upper bound for range-constrained groups), + returns None if the search exceeds that bound instead of running + indefinitely. The search is O(cluster size) per call against the set; + maintaining the cursor monotonically across calls keeps the + "next_available" numbering mode from becoming O(N^2) on large groups. + """ + n = start + while n in used_numbers: + n += 1 + if end is not None and n > end: + return None + if end is not None and n > end: + return None + return n + + +def _pick_target_number( + mode, + stream, + used_numbers, + fixed_cursor, + fallback_start, + end_number=None, + range_start=None, +): + """ + Return the channel number a given stream should claim under the group's + numbering mode, or None if the configured range is exhausted. + + Shared by the existing-channel renumber pass and the new-channel create + pass so both honor identical mode semantics: provider-supplied number + when available and free, otherwise fall back; or always next-available; + or fixed-cursor sequential. + + `range_start`, when provided, is the inclusive lower bound for the + group's configured numbering range. Provider-supplied numbers below + this bound fall back to the next-available picker so freshly-created + channels never land outside the configured range. + """ + if mode == "provider": + chno = stream.stream_chno + if ( + chno is not None + and chno not in used_numbers + and (range_start is None or chno >= range_start) + and (end_number is None or chno <= end_number) + ): + return chno + # No usable provider number: walk from fallback_start, bumped up + # to range_start when set so the fallback never lands below the + # configured range. + effective_start = ( + max(fallback_start, range_start) + if range_start is not None + else fallback_start + ) + return _next_available_number(used_numbers, effective_start, end=end_number) + if mode == "next_available": + return _next_available_number(used_numbers, 1, end=end_number) + return _next_available_number(used_numbers, fixed_cursor, end=end_number) + + +def _custom_properties_as_dict(value): + """ + Normalize a JSONField-backed custom_properties value into a dict. + + Historical data has rows where the field holds a JSON-encoded string + instead of a dict. Django's JSONField serializes whatever it gets, so + `.get()` on one of those rows raises AttributeError and aborts the + entire sync. Treat string values as JSON to parse, and fall back to an + empty dict for anything that isn't a dict after parsing. + """ + import json + + if isinstance(value, dict): + return value + if isinstance(value, str): + try: + parsed = json.loads(value) + except (ValueError, TypeError): + logger.warning( + "custom_properties stored as non-JSON string; ignoring: %r", + value[:100], + ) + return {} + return parsed if isinstance(parsed, dict) else {} + return {} + + +def _classify_sync_failure(exc): + """ + Map an exception raised during per-stream sync to a coarse typed + reason used by the completion notification's grouped failure list. + Keeps the bucket count small so the modal stays readable; the + underlying exception text is preserved verbatim in ``error``. + """ + from django.db import IntegrityError + + if isinstance(exc, IntegrityError): + return "INTEGRITY_ERROR" + return "OTHER" + + @shared_task def sync_auto_channels(account_id, scan_start_time=None): """ @@ -1722,17 +1829,47 @@ def sync_auto_channels(account_id, scan_start_time=None): channels_created = 0 channels_updated = 0 channels_deleted = 0 + channels_failed = 0 + # Per-failure context for the completion notification. Each entry + # carries a typed ``reason`` so the modal can group counts by + # cause; the cap keeps the WebSocket payload bounded but is sized + # generously to cover realistic multi-provider failure sets. + # Full per-stream detail still goes to ``logger.warning`` for + # power-user diagnostics regardless of the cap. + failed_stream_details = [] + FAILURE_LOG_LIMIT = 1000 - # Get all channel numbers that are already in use by other channels (not auto-created by this account) + # Group range reservations (start+end) are advisory and NOT seeded + # here: two groups with overlapping ranges must cooperate, so only + # actually-occupied numbers constrain assignment. + # Hidden auto-created channels stay in the seed because the renumber + # loop iterates current provider streams (which excludes hidden + # ones); excluding them here would let sync reclaim their numbers. used_numbers = set( Channel.objects.exclude( - auto_created=True, auto_created_by=account + auto_created=True, + auto_created_by=account, + hidden_from_output=False, ).values_list("channel_number", flat=True) ) + # Override pins are global reservations: effective_channel_number + # uses the override, so the picker must treat those numbers as + # taken or sync can produce duplicate effective channel numbers. + from apps.channels.models import ChannelOverride + + used_numbers.update( + ChannelOverride.objects.filter( + channel_number__isnull=False + ).values_list("channel_number", flat=True) + ) + used_numbers.discard(None) for group_relation in auto_sync_groups: channel_group = group_relation.channel_group start_number = group_relation.auto_sync_channel_start or 1.0 + # Optional upper bound; _next_available_number returns None when + # exhausted, which the per-stream loop converts to a failure. + end_number = group_relation.auto_sync_channel_end # Get force_dummy_epg, group_override, and regex patterns from group custom_properties group_custom_props = {} @@ -1741,6 +1878,7 @@ def sync_auto_channels(account_id, scan_start_time=None): name_regex_pattern = None name_replace_pattern = None name_match_regex = None + name_match_exclude_regex = None channel_profile_ids = None channel_sort_order = None channel_sort_reverse = False @@ -1749,8 +1887,10 @@ def sync_auto_channels(account_id, scan_start_time=None): custom_epg_id = None # New option: select specific EPG source (takes priority over force_dummy_epg) channel_numbering_mode = "fixed" # Default mode channel_numbering_fallback = 1 # Default fallback for provider mode - if group_relation.custom_properties: - group_custom_props = group_relation.custom_properties + group_custom_props = _custom_properties_as_dict( + group_relation.custom_properties + ) + if group_custom_props: force_dummy_epg = group_custom_props.get("force_dummy_epg", False) override_group_id = group_custom_props.get("group_override") name_regex_pattern = group_custom_props.get("name_regex_pattern") @@ -1758,6 +1898,9 @@ def sync_auto_channels(account_id, scan_start_time=None): "name_replace_pattern" ) name_match_regex = group_custom_props.get("name_match_regex") + name_match_exclude_regex = group_custom_props.get( + "name_match_exclude_regex" + ) channel_profile_ids = group_custom_props.get("channel_profile_ids") custom_epg_id = group_custom_props.get("custom_epg_id") channel_sort_order = group_custom_props.get("channel_sort_order") @@ -1793,9 +1936,11 @@ def sync_auto_channels(account_id, scan_start_time=None): last_seen__gte=scan_start_time, ) - # --- FILTER STREAMS BY NAME MATCH REGEX IF SPECIFIED --- + # Pre-compile with Python re so an invalid pattern fails here + # rather than as a Postgres exception inside the per-stream loop. if name_match_regex: try: + re.compile(name_match_regex) current_streams = current_streams.filter( name__iregex=name_match_regex ) @@ -1804,6 +1949,19 @@ def sync_auto_channels(account_id, scan_start_time=None): f"Invalid name_match_regex '{name_match_regex}' for group '{channel_group.name}': {e}. Skipping name filter." ) + # Exclude regex runs after the include filter so the two + # compose: include narrows, exclude removes from what's left. + if name_match_exclude_regex: + try: + re.compile(name_match_exclude_regex) + current_streams = current_streams.exclude( + name__iregex=name_match_exclude_regex + ) + except re.error as e: + logger.warning( + f"Invalid name_match_exclude_regex '{name_match_exclude_regex}' for group '{channel_group.name}': {e}. Skipping exclude filter." + ) + # --- APPLY CHANNEL SORT ORDER --- streams_is_list = False # Track if we converted to list if channel_sort_order and channel_sort_order != "": @@ -1834,27 +1992,27 @@ def sync_auto_channels(account_id, scan_start_time=None): order_prefix = "-" if channel_sort_reverse else "" current_streams = current_streams.order_by(f"{order_prefix}id") - # Get existing auto-created channels for this account (regardless of current group) - # We'll find them by their stream associations instead of just group location - existing_channels = Channel.objects.filter( - auto_created=True, auto_created_by=account - ).select_related("logo", "epg_data") - - # Create mapping of existing channels by their associated stream - # This approach finds channels even if they've been moved to different groups + # Scoped to this group so the loop below runs in O(group size). + # Multi-stream channels are deduped by channel_id so every + # stream_id maps to the same in-memory Channel instance and + # post-loop bulk_update writes the merged state. existing_channel_map = {} - for channel in existing_channels: - # Get streams associated with this channel that belong to our M3U account and original group - channel_streams = ChannelStream.objects.filter( - channel=channel, + existing_channels_by_id = {} + existing_channel_streams = ( + ChannelStream.objects.filter( + channel__auto_created=True, + channel__auto_created_by=account, stream__m3u_account=account, - stream__channel_group=channel_group, # Match streams from the original group - ).select_related("stream") - - # Map each of our M3U account's streams to this channel - for channel_stream in channel_streams: - if channel_stream.stream: - existing_channel_map[channel_stream.stream.id] = channel + stream__channel_group=channel_group, + ) + .select_related("channel") + ) + for cs in existing_channel_streams: + if cs.stream_id and cs.channel_id: + canonical = existing_channels_by_id.setdefault( + cs.channel_id, cs.channel + ) + existing_channel_map[cs.stream_id] = canonical # Track which streams we've processed processed_stream_ids = set() @@ -1866,10 +2024,142 @@ def sync_auto_channels(account_id, scan_start_time=None): else current_streams.exists() ) + # Bulk pre-fetch collapses N+1 Logo/EPGData lookups into a + # pair of in_bulk() calls. + from apps.channels.models import Logo + from apps.epg.models import EPGSource + + # Resolve the group's custom EPG source once. + custom_epg_source = None + custom_dummy_epg_data = None + if custom_epg_id: + try: + custom_epg_source = EPGSource.objects.get(id=custom_epg_id) + if custom_epg_source.source_type == "dummy": + custom_dummy_epg_data = ( + EPGData.objects.filter( + epg_source=custom_epg_source + ).first() + ) + if not custom_dummy_epg_data: + logger.warning( + f"No EPGData found for dummy EPG source " + f"{custom_epg_source.name} (ID: {custom_epg_id})" + ) + except EPGSource.DoesNotExist: + logger.warning( + f"Custom EPG source with ID {custom_epg_id} not found " + f"for group '{channel_group.name}', falling back to " + f"auto-match" + ) + + # Resolve the group's custom logo once. + custom_logo = None + if custom_logo_id: + try: + custom_logo = Logo.objects.get(id=custom_logo_id) + except Logo.DoesNotExist: + logger.warning( + f"Custom logo with ID {custom_logo_id} not found for " + f"group '{channel_group.name}', falling back to stream " + f"logos" + ) + + logo_cache_by_url = {} + epg_cache_by_tvg_id = {} + if has_streams: + # Collect unique URLs / tvg_ids in one DB call each. + stream_iter = ( + current_streams + if streams_is_list + else list(current_streams.values("logo_url", "tvg_id")) + ) + unique_logo_urls = { + s.get("logo_url") if isinstance(s, dict) else getattr(s, "logo_url", None) + for s in stream_iter + } + unique_logo_urls.discard(None) + unique_logo_urls.discard("") + if unique_logo_urls: + logo_cache_by_url = { + lg.url: lg + for lg in Logo.objects.filter(url__in=unique_logo_urls) + } + + unique_tvg_ids = { + s.get("tvg_id") if isinstance(s, dict) else getattr(s, "tvg_id", None) + for s in stream_iter + } + unique_tvg_ids.discard(None) + unique_tvg_ids.discard("") + # Skip the EPG cache when force_dummy_epg with no + # custom source override; the resolver always returns None. + want_epg_cache = unique_tvg_ids and ( + not force_dummy_epg or custom_epg_id + ) + if want_epg_cache: + # Scope to the group's pinned source so foreign-source + # tvg_id matches do not leak in. + epg_q = EPGData.objects.filter(tvg_id__in=unique_tvg_ids) + if ( + custom_epg_source is not None + and custom_epg_source.source_type != "dummy" + ): + epg_q = epg_q.filter(epg_source=custom_epg_source) + epg_cache_by_tvg_id = {d.tvg_id: d for d in epg_q} + + def _resolve_logo_for_stream(stream): + """Return a Logo for stream.logo_url, creating it once if needed.""" + url = getattr(stream, "logo_url", None) + if not url: + return None + cached = logo_cache_by_url.get(url) + if cached is not None: + return cached + created, _ = Logo.objects.get_or_create( + url=url, + defaults={"name": stream.name or stream.tvg_id or "Unknown"}, + ) + logo_cache_by_url[url] = created + return created + + def _resolve_epg_for_stream(stream): + """Return the EPGData row that should be assigned to this + stream's channel. Encodes all four group-level EPG modes: + + 1. custom dummy source: single shared EPGData + 2. custom non-dummy source: cache lookup, scoped to + that source + 3. force_dummy_epg with no custom: None (clear EPG) + 4. default auto-match: cache lookup, any source + + The cache (epg_cache_by_tvg_id) is built once above with the + correct scope so the per-stream lookup is a dict access. + """ + if custom_epg_source is not None: + if custom_epg_source.source_type == "dummy": + return custom_dummy_epg_data + tvg_id = getattr(stream, "tvg_id", None) + if not tvg_id: + return None + return epg_cache_by_tvg_id.get(tvg_id) + if force_dummy_epg: + return None + tvg_id = getattr(stream, "tvg_id", None) + if not tvg_id: + return None + return epg_cache_by_tvg_id.get(tvg_id) + if not has_streams: logger.debug(f"No streams found in group {channel_group.name}") - # Delete all existing auto channels if no streams - channels_to_delete = [ch for ch in existing_channel_map.values()] + # No streams left in the group: drop the visible auto + # channels. Hidden channels are preserved so the hide + # flag survives temporary provider drops (event/PPV). + channels_to_delete = [ + ch + for ch in existing_channel_map.values() + if not ch.hidden_from_output + ] if channels_to_delete: deleted_count = len(channels_to_delete) Channel.objects.filter( @@ -1915,43 +2205,37 @@ def sync_auto_channels(account_id, scan_start_time=None): ) stream_profile_to_assign = None - # Process each current stream current_channel_number = start_number - # Always renumber all existing channels to match current sort order - # This ensures channels are always in the correct sequence + # Renumber existing channels to match sort order. Compact + # mode skips this; the end-of-iteration pack is the source + # of truth and would overwrite the renumber. + compact_mode = bool(group_custom_props.get("compact_numbering")) channels_to_renumber = [] temp_channel_number = start_number - for stream in current_streams: + for stream in current_streams if not compact_mode else []: if stream.id in existing_channel_map: channel = existing_channel_map[stream.id] - # Determine target number based on numbering mode - if channel_numbering_mode == "provider": - # Use provider number if available, otherwise use fallback with next available logic - if stream.stream_chno is not None: - target_number = stream.stream_chno - # If provider number is already used, find next available - if target_number in used_numbers: - target_number = channel_numbering_fallback - while target_number in used_numbers: - target_number += 1 - else: - # No provider number, use fallback and find next available - target_number = channel_numbering_fallback - while target_number in used_numbers: - target_number += 1 - elif channel_numbering_mode == "next_available": - # Find next available starting from 1 - target_number = 1 - while target_number in used_numbers: - target_number += 1 - else: # fixed mode (default) - # Find next available number starting from temp_channel_number - target_number = temp_channel_number - while target_number in used_numbers: - target_number += 1 + target_number = _pick_target_number( + channel_numbering_mode, + stream, + used_numbers, + temp_channel_number, + channel_numbering_fallback, + end_number=end_number, + range_start=start_number, + ) + + # Range exhausted: leave the channel at its existing + # number. The renumber pass is sort-optimization only; + # no failure record needed. + if target_number is None: + # Preserve the channel's current number in used_numbers + if channel.channel_number is not None: + used_numbers.add(channel.channel_number) + continue # Add this number to used_numbers so we don't reuse it in this batch used_numbers.add(target_number) @@ -1971,14 +2255,73 @@ def sync_auto_channels(account_id, scan_start_time=None): # Bulk update channel numbers if any need renumbering if channels_to_renumber: - Channel.objects.bulk_update(channels_to_renumber, ["channel_number"]) + Channel.objects.bulk_update( + channels_to_renumber, ["channel_number"], batch_size=500 + ) logger.info( f"Renumbered {len(channels_to_renumber)} channels to maintain sort order" ) + # When the group's configured range is narrower than its existing + # channels span, any non-hidden auto-created channel whose number + # falls outside [start, end] gets deleted. The new-channel + # creation loop below picks up the freed stream and re-creates + # the channel at a slot inside the new range, so the net user + # outcome is a renumber, not a failure. Counted in + # channels_deleted; the replacement counts in channels_created. + # Hidden channels are preserved. Runs BEFORE new-channel creation + # so slots freed by the deletions are available to incoming + # streams. + if end_number is not None: + overflow_delete_ids = [] + for stream_id, ch in list(existing_channel_map.items()): + if ch.hidden_from_output: + continue + num = ch.channel_number + if num is None: + continue + if num < start_number or num > end_number: + overflow_delete_ids.append(ch.id) + existing_channel_map.pop(stream_id, None) + used_numbers.discard(num) + if overflow_delete_ids: + deleted = Channel.objects.filter( + id__in=overflow_delete_ids + ).delete() + removed_count = ( + deleted[1].get("dispatcharr_channels.Channel", 0) + if isinstance(deleted, tuple) and len(deleted) > 1 + else len(overflow_delete_ids) + ) + channels_deleted += removed_count + logger.info( + f"Deleted {removed_count} channels outside the " + f"range {int(start_number)}-{int(end_number)} for " + f"group '{channel_group.name}'" + ) + # Reset channel number counter for processing new channels current_channel_number = start_number + # Per-channel changes are buffered and bulk_update'd once after the + # loop. update_fields is set explicitly so post_save signals only + # fire for receivers whose tracked field actually changed. + existing_dirty_channels = [] + existing_dirty_ids = set() + existing_dirty_field_set = set() + # Subset of channels whose epg_data actually changed in this + # pass. Used by the dispatcher below to fire the EPG parse + # task only for those, not for every channel in + # existing_dirty_channels. + epg_dirty_channel_ids = set() + + # New channels are buffered and bulk_create'd after the loop. + # bulk_create skips post_save, so the EPG parse task is dispatched + # once per unique epg_data_id below rather than per channel. + # Pairs are (Channel(), Stream) so the post-loop step can attach + # ChannelStream rows using the IDs Postgres returns. + new_channels_pending = [] + for stream in current_streams: processed_stream_ids.add(stream.id) try: @@ -2012,305 +2355,132 @@ def sync_auto_channels(account_id, scan_start_time=None): existing_channel = existing_channel_map.get(stream.id) if existing_channel: - # Update existing channel if needed (channel number already handled above) - channel_updated = False + # Track only the fields that actually changed, so the + # eventual UPDATE writes one column per change instead + # of every column on every channel. The dirty list is + # accumulated and bulk_update'd after the loop - + # which avoids issuing an UPDATE per channel and + # avoids firing the EPG post_save signal on saves + # that didn't touch epg_data. + dirty_fields = [] - # Use new_name instead of stream.name if existing_channel.name != new_name: existing_channel.name = new_name - channel_updated = True + dirty_fields.append("name") if existing_channel.tvg_id != stream.tvg_id: existing_channel.tvg_id = stream.tvg_id - channel_updated = True + dirty_fields.append("tvg_id") if existing_channel.tvc_guide_stationid != tvc_guide_stationid: existing_channel.tvc_guide_stationid = tvc_guide_stationid - channel_updated = True + dirty_fields.append("tvc_guide_stationid") - # Check if channel group needs to be updated (in case override was added/changed) - if existing_channel.channel_group != target_group: + # The group override may direct sync to a different + # ChannelGroup than the one currently on the row. + if existing_channel.channel_group_id != target_group.id: existing_channel.channel_group = target_group - channel_updated = True - logger.info( - f"Moved auto channel '{existing_channel.name}' from '{existing_channel.channel_group.name if existing_channel.channel_group else 'None'}' to '{target_group.name}'" - ) + dirty_fields.append("channel_group") - # Handle logo updates - current_logo = None - if custom_logo_id: - # Use the custom logo specified in group settings - from apps.channels.models import Logo - try: - current_logo = Logo.objects.get(id=custom_logo_id) - except Logo.DoesNotExist: - logger.warning( - f"Custom logo with ID {custom_logo_id} not found for existing channel, falling back to stream logo" - ) - # Fall back to stream logo if custom logo not found - if stream.logo_url: - current_logo, _ = Logo.objects.get_or_create( - url=stream.logo_url, - defaults={ - "name": stream.name or stream.tvg_id or "Unknown" - }, - ) - elif stream.logo_url: - # No custom logo configured, use stream logo - from apps.channels.models import Logo - - current_logo, _ = Logo.objects.get_or_create( - url=stream.logo_url, - defaults={ - "name": stream.name or stream.tvg_id or "Unknown" - }, - ) - - if existing_channel.logo != current_logo: + # Logo: custom group setting wins; otherwise stream logo + current_logo = ( + custom_logo + if custom_logo_id and custom_logo is not None + else _resolve_logo_for_stream(stream) + ) + current_logo_id = current_logo.id if current_logo else None + if existing_channel.logo_id != current_logo_id: existing_channel.logo = current_logo - channel_updated = True + dirty_fields.append("logo") - # Handle EPG data updates - current_epg_data = None - if custom_epg_id: - # Use the custom EPG specified in group settings (e.g., a dummy EPG) - from apps.epg.models import EPGSource - try: - epg_source = EPGSource.objects.get(id=custom_epg_id) - # For dummy EPGs, select the first (and typically only) EPGData entry from this source - if epg_source.source_type == 'dummy': - current_epg_data = EPGData.objects.filter( - epg_source=epg_source - ).first() - if not current_epg_data: - logger.warning( - f"No EPGData found for dummy EPG source {epg_source.name} (ID: {custom_epg_id})" - ) - else: - # For non-dummy sources, try to find existing EPGData by tvg_id - if stream.tvg_id: - current_epg_data = EPGData.objects.filter( - tvg_id=stream.tvg_id, - epg_source=epg_source - ).first() - except EPGSource.DoesNotExist: - logger.warning( - f"Custom EPG source with ID {custom_epg_id} not found for existing channel, falling back to auto-match" - ) - # Fall back to auto-match by tvg_id - if stream.tvg_id and not force_dummy_epg: - current_epg_data = EPGData.objects.filter( - tvg_id=stream.tvg_id - ).first() - elif stream.tvg_id and not force_dummy_epg: - # Auto-match EPG by tvg_id (original behavior) - current_epg_data = EPGData.objects.filter( - tvg_id=stream.tvg_id - ).first() - # If force_dummy_epg is True and no custom_epg_id, current_epg_data stays None - - if existing_channel.epg_data != current_epg_data: + # EPG: handled centrally by _resolve_epg_for_stream + current_epg_data = _resolve_epg_for_stream(stream) + current_epg_id = ( + current_epg_data.id if current_epg_data else None + ) + if existing_channel.epg_data_id != current_epg_id: existing_channel.epg_data = current_epg_data - channel_updated = True + dirty_fields.append("epg_data") + if current_epg_id is not None: + epg_dirty_channel_ids.add(existing_channel.id) - # Handle stream profile updates for the channel - if stream_profile_to_assign and existing_channel.stream_profile != stream_profile_to_assign: + # Stream profile: only set if group has one configured + if ( + stream_profile_to_assign is not None + and existing_channel.stream_profile_id + != stream_profile_to_assign.id + ): existing_channel.stream_profile = stream_profile_to_assign - channel_updated = True + dirty_fields.append("stream_profile") - if channel_updated: - existing_channel.save() - channels_updated += 1 - logger.debug( - f"Updated auto channel: {existing_channel.channel_number} - {existing_channel.name}" - ) - - # Update channel profile memberships for existing channels - current_memberships = set( - ChannelProfileMembership.objects.filter( - channel=existing_channel, enabled=True - ).values_list("channel_profile_id", flat=True) - ) - - target_profile_ids = set( - profile.id for profile in profiles_to_assign - ) - - # Only update if memberships have changed - if current_memberships != target_profile_ids: - # Disable all current memberships - ChannelProfileMembership.objects.filter( - channel=existing_channel - ).update(enabled=False) - - # Enable/create memberships for target profiles - for profile in profiles_to_assign: - membership, created = ( - ChannelProfileMembership.objects.get_or_create( - channel_profile=profile, - channel=existing_channel, - defaults={"enabled": True}, - ) - ) - if not created and not membership.enabled: - membership.enabled = True - membership.save() - - logger.debug( - f"Updated profile memberships for auto channel: {existing_channel.name}" - ) + if dirty_fields: + # Multi-stream channels appear once per stream; + # dedupe by id so bulk_update does not double-fire + # and channels_updated does not double-count. + if existing_channel.id not in existing_dirty_ids: + existing_dirty_channels.append(existing_channel) + existing_dirty_ids.add(existing_channel.id) + channels_updated += 1 + existing_dirty_field_set.update(dirty_fields) else: - # Create new channel - # Determine channel number based on numbering mode - if channel_numbering_mode == "provider": - # Use provider number if available, otherwise use fallback with next available logic - if stream.stream_chno is not None: - target_number = stream.stream_chno - # If provider number is already used, find next available from fallback - if target_number in used_numbers: - target_number = channel_numbering_fallback - while target_number in used_numbers: - target_number += 1 - else: - # No provider number, use fallback and find next available - target_number = channel_numbering_fallback - while target_number in used_numbers: - target_number += 1 - elif channel_numbering_mode == "next_available": - # Find next available starting from 1 - target_number = 1 - while target_number in used_numbers: - target_number += 1 - else: # fixed mode (default) - # Find next available channel number starting from current_channel_number - target_number = current_channel_number - while target_number in used_numbers: - target_number += 1 + # Range exhaustion is surfaced to the user via the + # completion notification, not swallowed. + target_number = _pick_target_number( + channel_numbering_mode, + stream, + used_numbers, + current_channel_number, + channel_numbering_fallback, + end_number=end_number, + range_start=start_number, + ) + + if target_number is None: + channels_failed += 1 + if len(failed_stream_details) < FAILURE_LOG_LIMIT: + failed_stream_details.append({ + "stream_name": stream.name, + "stream_id": stream.id, + "group": channel_group.name, + "reason": "RANGE_EXHAUSTED", + "error": ( + f"Channel number range " + f"{int(start_number)}-{int(end_number)} is full" + ), + }) + processed_stream_ids.add(stream.id) + continue # Add this number to used_numbers used_numbers.add(target_number) - channel = Channel.objects.create( - channel_number=target_number, - name=new_name, - tvg_id=stream.tvg_id, - tvc_guide_stationid=tvc_guide_stationid, - channel_group=target_group, - user_level=0, - auto_created=True, - auto_created_by=account, + # Resolve every FK BEFORE the create call so the + # initial INSERT carries the complete row. + new_logo = ( + custom_logo + if custom_logo_id and custom_logo is not None + else _resolve_logo_for_stream(stream) ) + new_epg_data = _resolve_epg_for_stream(stream) - # Associate the stream with the channel - ChannelStream.objects.create( - channel=channel, stream=stream, order=0 - ) - - # Assign to correct profiles - memberships = [ - ChannelProfileMembership( - channel_profile=profile, channel=channel, enabled=True + new_channels_pending.append( + ( + Channel( + channel_number=target_number, + name=new_name, + tvg_id=stream.tvg_id, + tvc_guide_stationid=tvc_guide_stationid, + channel_group=target_group, + user_level=0, + auto_created=True, + auto_created_by=account, + logo=new_logo, + epg_data=new_epg_data, + stream_profile=stream_profile_to_assign, + ), + stream, ) - for profile in profiles_to_assign - ] - if memberships: - ChannelProfileMembership.objects.bulk_create(memberships) - - # Try to match EPG data - if custom_epg_id: - # Use the custom EPG specified in group settings (e.g., a dummy EPG) - from apps.epg.models import EPGSource - try: - epg_source = EPGSource.objects.get(id=custom_epg_id) - # For dummy EPGs, select the first (and typically only) EPGData entry from this source - if epg_source.source_type == 'dummy': - epg_data = EPGData.objects.filter( - epg_source=epg_source - ).first() - if epg_data: - channel.epg_data = epg_data - channel.save(update_fields=["epg_data"]) - else: - logger.warning( - f"No EPGData found for dummy EPG source {epg_source.name} (ID: {custom_epg_id})" - ) - else: - # For non-dummy sources, try to find existing EPGData by tvg_id - if stream.tvg_id: - epg_data = EPGData.objects.filter( - tvg_id=stream.tvg_id, - epg_source=epg_source - ).first() - if epg_data: - channel.epg_data = epg_data - channel.save(update_fields=["epg_data"]) - except EPGSource.DoesNotExist: - logger.warning( - f"Custom EPG source with ID {custom_epg_id} not found, falling back to auto-match" - ) - # Fall back to auto-match by tvg_id - if stream.tvg_id and not force_dummy_epg: - epg_data = EPGData.objects.filter( - tvg_id=stream.tvg_id - ).first() - if epg_data: - channel.epg_data = epg_data - channel.save(update_fields=["epg_data"]) - elif stream.tvg_id and not force_dummy_epg: - # Auto-match EPG by tvg_id (original behavior) - epg_data = EPGData.objects.filter( - tvg_id=stream.tvg_id - ).first() - if epg_data: - channel.epg_data = epg_data - channel.save(update_fields=["epg_data"]) - elif force_dummy_epg: - # Force dummy EPG with no custom EPG selected (set to None) - channel.epg_data = None - channel.save(update_fields=["epg_data"]) - - # Handle logo - if custom_logo_id: - # Use the custom logo specified in group settings - from apps.channels.models import Logo - try: - custom_logo = Logo.objects.get(id=custom_logo_id) - channel.logo = custom_logo - channel.save(update_fields=["logo"]) - except Logo.DoesNotExist: - logger.warning( - f"Custom logo with ID {custom_logo_id} not found, falling back to stream logo" - ) - # Fall back to stream logo if custom logo not found - if stream.logo_url: - logo, _ = Logo.objects.get_or_create( - url=stream.logo_url, - defaults={ - "name": stream.name or stream.tvg_id or "Unknown" - }, - ) - channel.logo = logo - channel.save(update_fields=["logo"]) - elif stream.logo_url: - from apps.channels.models import Logo - - logo, _ = Logo.objects.get_or_create( - url=stream.logo_url, - defaults={ - "name": stream.name or stream.tvg_id or "Unknown" - }, - ) - channel.logo = logo - channel.save(update_fields=["logo"]) - - # Handle stream profile assignment - if stream_profile_to_assign: - channel.stream_profile = stream_profile_to_assign - channel.save(update_fields=['stream_profile']) - channels_created += 1 - logger.debug( - f"Created auto channel: {channel.channel_number} - {channel.name}" ) # Increment channel number for next iteration (only in fixed mode) @@ -2323,12 +2493,171 @@ def sync_auto_channels(account_id, scan_start_time=None): logger.error( f"Error processing auto channel for stream {stream.name}: {str(e)}" ) + channels_failed += 1 + if len(failed_stream_details) < FAILURE_LOG_LIMIT: + failed_stream_details.append({ + "stream_name": stream.name, + "stream_id": stream.id, + "group": channel_group.name, + "reason": _classify_sync_failure(e), + "error": str(e), + }) continue - # Delete channels for streams that no longer exist - channels_to_delete = [] + # Bulk-create channels, then dependent rows using the IDs + # Postgres returns. bulk_create skips post_save, so the EPG + # parse task is dispatched explicitly per-epg_data_id below + # to avoid flooding Celery at scale. + if new_channels_pending: + channel_objs = [pair[0] for pair in new_channels_pending] + streams_for_new = [pair[1] for pair in new_channels_pending] + Channel.objects.bulk_create(channel_objs, batch_size=500) + + ChannelStream.objects.bulk_create( + [ + ChannelStream( + channel_id=channel_objs[i].id, + stream_id=streams_for_new[i].id, + order=0, + ) + for i in range(len(channel_objs)) + ], + batch_size=500, + ) + + if profiles_to_assign: + ChannelProfileMembership.objects.bulk_create( + [ + ChannelProfileMembership( + channel_id=ch.id, + channel_profile_id=profile.id, + enabled=True, + ) + for ch in channel_objs + for profile in profiles_to_assign + ], + ignore_conflicts=True, + batch_size=500, + ) + + channels_created += len(channel_objs) + + # One EPG parse task per unique EPGData replaces the + # per-channel post_save dispatch bypassed by bulk_create. + from apps.epg.tasks import parse_programs_for_tvg_id + + unique_epg_ids = { + ch.epg_data_id for ch in channel_objs if ch.epg_data_id + } + for epg_id in unique_epg_ids: + parse_programs_for_tvg_id.delay(epg_id) + + logger.debug( + f"Bulk created {len(channel_objs)} channels in group " + f"'{channel_group.name}'; dispatched " + f"{len(unique_epg_ids)} unique EPG parse task(s)" + ) + + # bulk_update writes only the columns named in `fields` and + # bypasses post_save, so the EPG refresh signal cannot fire here. + # Dispatch one parse task per unique EPGData id when epg_data was + # in the dirty set, mirroring the new-channel path above. + if existing_dirty_channels: + Channel.objects.bulk_update( + existing_dirty_channels, + fields=list(existing_dirty_field_set), + batch_size=500, + ) + if epg_dirty_channel_ids: + from apps.epg.tasks import parse_programs_for_tvg_id + + # Dispatch only for channels whose epg_data_id changed. + # Other dirty channels would queue redundant parses. + unique_epg_ids = { + ch.epg_data_id + for ch in existing_dirty_channels + if ch.id in epg_dirty_channel_ids and ch.epg_data_id + } + for epg_id in unique_epg_ids: + parse_programs_for_tvg_id.delay(epg_id) + logger.debug( + f"Bulk updated {len(existing_dirty_channels)} existing " + f"channels (fields: {sorted(existing_dirty_field_set)})" + ) + + # Reconcile ChannelProfileMembership in two writes: one + # bulk_update for enable-flips, one bulk_create for missing + # rows. Avoids a per-channel save loop. + existing_channel_ids = [ + c.id for c in existing_channel_map.values() + ] + target_profile_ids = {p.id for p in profiles_to_assign} + if existing_channel_ids: + membership_rows = list( + ChannelProfileMembership.objects.filter( + channel_id__in=existing_channel_ids + ).only("id", "channel_id", "channel_profile_id", "enabled") + ) + memberships_by_channel = {} + for m in membership_rows: + memberships_by_channel.setdefault(m.channel_id, []).append(m) + + rows_to_flip = [] + rows_to_create = [] + for ch_id in existing_channel_ids: + rows = memberships_by_channel.get(ch_id, []) + have_for_target = set() + for m in rows: + if m.channel_profile_id in target_profile_ids: + have_for_target.add(m.channel_profile_id) + if not m.enabled: + m.enabled = True + rows_to_flip.append(m) + else: + if m.enabled: + m.enabled = False + rows_to_flip.append(m) + missing = target_profile_ids - have_for_target + for pid in missing: + rows_to_create.append( + ChannelProfileMembership( + channel_id=ch_id, + channel_profile_id=pid, + enabled=True, + ) + ) + + if rows_to_flip: + ChannelProfileMembership.objects.bulk_update( + rows_to_flip, ["enabled"], batch_size=500 + ) + if rows_to_create: + ChannelProfileMembership.objects.bulk_create( + rows_to_create, ignore_conflicts=True, batch_size=500 + ) + if rows_to_flip or rows_to_create: + logger.debug( + f"Reconciled memberships for " + f"{len(existing_channel_ids)} channels " + f"({len(rows_to_flip)} flipped, " + f"{len(rows_to_create)} created)" + ) + + # Delete channels whose streams have all disappeared. + # Hidden channels are preserved so event/PPV holds across + # provider drops. + channel_streams_in_group = {} for stream_id, channel in existing_channel_map.items(): - if stream_id not in processed_stream_ids: + channel_streams_in_group.setdefault(channel.id, []).append( + (stream_id, channel) + ) + channels_to_delete = [] + for ch_id, pairs in channel_streams_in_group.items(): + channel = pairs[0][1] + if channel.hidden_from_output: + continue + stream_ids = {sid for sid, _ in pairs} + if not (stream_ids & processed_stream_ids): channels_to_delete.append(channel) if channels_to_delete: @@ -2341,34 +2670,98 @@ def sync_auto_channels(account_id, scan_start_time=None): f"Deleted {deleted_count} auto channels for removed streams" ) - # Additional cleanup: Remove auto-created channels that no longer have any valid streams - # This handles the case where streams were deleted due to stale retention policy - orphaned_channels = Channel.objects.filter( - auto_created=True, - auto_created_by=account - ).exclude( - # Exclude channels that still have valid stream associations - id__in=ChannelStream.objects.filter( - stream__m3u_account=account, - stream__isnull=False - ).values_list('channel_id', flat=True) - ) + # Compact-mode pack: hidden channels release their number and + # visible channels pack contiguously into [start, end]. Runs + # after create/update/delete so the channel set is stable. + if compact_mode: + from apps.channels.compact_numbering import repack_group - deleted_total, _ = orphaned_channels.delete() - if deleted_total: - channels_deleted += deleted_total - logger.info( - f"Deleted {deleted_total} orphaned auto channels with no valid streams" + pack_result = repack_group(group_relation) + if pack_result["failed"]: + channels_failed += pack_result["failed"] + if ( + len(failed_stream_details) < FAILURE_LOG_LIMIT + ): + failed_stream_details.append( + { + "stream_name": None, + "stream_id": None, + "group": channel_group.name, + "reason": "RANGE_EXHAUSTED", + "error": ( + f"Compact pack: {pack_result['failed']} " + f"visible channel(s) could not fit in range " + f"{int(start_number)}" + + ( + f"-{int(end_number)}" + if end_number + else "+" + ) + ), + } + ) + logger.debug( + f"Compact pack for group '{channel_group.name}': " + f"{pack_result['assigned']} assigned, " + f"{pack_result['released']} released, " + f"{pack_result['failed']} failed" + ) + + # Cleanup mode read from account.custom_properties.orphan_channel_cleanup: + # "always" (default; key absent) removes every orphan auto channel; + # "preserve_customized" keeps those with a ChannelOverride row; + # "never" disables cleanup. Hidden channels are preserved across all + # modes so event/PPV channels that come and go are not silently lost. + cleanup_mode = (account.custom_properties or {}).get( + "orphan_channel_cleanup", "always" + ) + if cleanup_mode != "never": + orphaned_channels = Channel.objects.filter( + auto_created=True, + auto_created_by=account, + hidden_from_output=False, + ).exclude( + id__in=ChannelStream.objects.filter( + stream__m3u_account=account, + stream__isnull=False, + ).values_list("channel_id", flat=True) ) + if cleanup_mode == "preserve_customized": + orphaned_channels = orphaned_channels.filter(override__isnull=True) + + _, per_model = orphaned_channels.delete() + deleted_channels = per_model.get("dispatcharr_channels.Channel", 0) + if deleted_channels: + channels_deleted += deleted_channels + logger.info( + f"Deleted {deleted_channels} orphaned auto channels with no valid streams (mode={cleanup_mode})" + ) logger.info( - f"Auto channel sync complete for account {account.name}: {channels_created} created, {channels_updated} updated, {channels_deleted} deleted" + f"Auto channel sync complete for account {account.name}: " + f"{channels_created} created, {channels_updated} updated, " + f"{channels_deleted} deleted, {channels_failed} failed" ) - return f"Auto sync: {channels_created} channels created, {channels_updated} updated, {channels_deleted} deleted" + return { + "status": "ok", + "channels_created": channels_created, + "channels_updated": channels_updated, + "channels_deleted": channels_deleted, + "channels_failed": channels_failed, + "failed_stream_details": failed_stream_details, + } except Exception as e: logger.error(f"Error in auto channel sync for account {account_id}: {str(e)}") - return f"Auto sync error: {str(e)}" + return { + "status": "error", + "error": str(e), + "channels_created": 0, + "channels_updated": 0, + "channels_deleted": 0, + "channels_failed": 0, + "failed_stream_details": [], + } def get_transformed_credentials(account, profile=None): @@ -3074,15 +3467,34 @@ def _refresh_single_m3u_account_impl(account_id): # Run auto channel sync after successful refresh auto_sync_message = "" + auto_sync_result = {} try: - sync_result = sync_auto_channels( + auto_sync_result = sync_auto_channels( account_id, scan_start_time=str(refresh_start_timestamp) - ) + ) or {} logger.info( - f"Auto channel sync result for account {account_id}: {sync_result}" + f"Auto channel sync result for account {account_id}: {auto_sync_result}" ) - if sync_result and "created" in sync_result: - auto_sync_message = f" {sync_result}." + if auto_sync_result.get("status") == "ok": + created = auto_sync_result.get("channels_created", 0) + updated = auto_sync_result.get("channels_updated", 0) + deleted = auto_sync_result.get("channels_deleted", 0) + failed = auto_sync_result.get("channels_failed", 0) + if created or updated or deleted or failed: + parts = [] + if created: + parts.append(f"{created} channel(s) created") + if updated: + parts.append(f"{updated} updated") + if deleted: + parts.append(f"{deleted} deleted") + if failed: + parts.append(f"{failed} failed") + auto_sync_message = f" Auto-sync: {', '.join(parts)}." + elif auto_sync_result.get("status") == "error": + auto_sync_message = ( + f" Auto-sync error: {auto_sync_result.get('error', 'unknown')}." + ) except Exception as e: logger.error( f"Error running auto channel sync for account {account_id}: {str(e)}" @@ -3127,6 +3539,14 @@ def _refresh_single_m3u_account_impl(account_id): streams_created=streams_created, streams_updated=streams_updated, streams_deleted=streams_deleted, + # Structured auto-sync counts so the frontend can render a + # warning card when anything failed, without parsing the + # free-text last_message. + channels_created=auto_sync_result.get("channels_created", 0), + channels_updated=auto_sync_result.get("channels_updated", 0), + channels_deleted=auto_sync_result.get("channels_deleted", 0), + channels_failed=auto_sync_result.get("channels_failed", 0), + failed_stream_details=auto_sync_result.get("failed_stream_details", []), message=account.last_message, ) diff --git a/apps/m3u/tests/test_account_destroy.py b/apps/m3u/tests/test_account_destroy.py new file mode 100644 index 00000000..e3a46aa2 --- /dev/null +++ b/apps/m3u/tests/test_account_destroy.py @@ -0,0 +1,143 @@ +""" +DELETE /api/m3u/accounts/{id}/ behavior. + +The endpoint always cascade-deletes auto-created channels owned by the +account. Manual channels survive even if some of their streams were +owned by the deleted account; only the streams from that account go +away with the account, and the channel keeps any streams sourced from +other accounts. The legacy ``cleanup_channels`` query parameter is +accepted but ignored. +""" +from unittest.mock import patch + +from django.contrib.auth import get_user_model +from django.test import TestCase +from django.utils import timezone +from rest_framework import status +from rest_framework.test import APIClient + +from apps.channels.models import ( + Channel, + ChannelGroup, + ChannelStream, + Stream, +) +from apps.m3u.models import M3UAccount + +User = get_user_model() + + +class M3UAccountDestroyTests(TestCase): + def setUp(self): + self.user = User.objects.create_user( + username="destroyer", password="testpass123" + ) + self.user.user_level = 10 + self.user.save() + self.client = APIClient() + self.client.force_authenticate(user=self.user) + # Patching the proxy stop here keeps the Redis-backed call out + # of the test path. The endpoint loops over every auto-created + # channel before the DB transaction; the real implementation is + # exercised by integration tests, not unit tests. + self._stop_patch = patch( + "apps.proxy.ts_proxy.services.channel_service." + "ChannelService.stop_channel" + ) + self._stop_patch.start() + self.addCleanup(self._stop_patch.stop) + + self.account = M3UAccount.objects.create( + name="ProviderA", + server_url="http://example.com/a.m3u", + ) + self.other_account = M3UAccount.objects.create( + name="ProviderB", + server_url="http://example.com/b.m3u", + ) + self.group = ChannelGroup.objects.create(name="News") + + def _make_stream(self, account, name="ESPN"): + return Stream.objects.create( + name=name, + url=f"http://example.com/{name.lower()}.m3u8", + m3u_account=account, + channel_group=self.group, + tvg_id=name.lower(), + last_seen=timezone.now(), + ) + + def test_cascade_deletes_auto_created_channels(self): + # Two auto-created channels under the account being deleted. + for n in (101.0, 102.0): + ch = Channel.objects.create( + channel_number=n, + name=f"Auto {n}", + channel_group=self.group, + auto_created=True, + auto_created_by=self.account, + ) + ChannelStream.objects.create( + channel=ch, + stream=self._make_stream(self.account, name=f"Auto{int(n)}"), + order=0, + ) + + response = self.client.delete( + f"/api/m3u/accounts/{self.account.id}/" + ) + self.assertEqual(response.status_code, status.HTTP_200_OK) + self.assertEqual(response.data["deleted_channels"], 2) + self.assertFalse(M3UAccount.objects.filter(id=self.account.id).exists()) + self.assertFalse( + Channel.objects.filter(auto_created_by_id=self.account.id).exists() + ) + + def test_manual_channel_survives_with_other_provider_streams(self): + # Manual channel with one stream from each account. The provider + # account's stream goes away with the account; the other + # account's stream stays, and the channel survives. + manual = Channel.objects.create( + channel_number=200.0, + name="Manual", + channel_group=self.group, + auto_created=False, + ) + provider_stream = self._make_stream(self.account, name="ProviderA") + other_stream = self._make_stream(self.other_account, name="ProviderB") + ChannelStream.objects.create(channel=manual, stream=provider_stream, order=0) + ChannelStream.objects.create(channel=manual, stream=other_stream, order=1) + + response = self.client.delete( + f"/api/m3u/accounts/{self.account.id}/" + ) + self.assertEqual(response.status_code, status.HTTP_200_OK) + self.assertEqual(response.data["deleted_channels"], 0) + + manual.refresh_from_db() + remaining = list(manual.channelstream_set.values_list("stream__id", flat=True)) + self.assertEqual(remaining, [other_stream.id]) + + def test_legacy_cleanup_channels_param_is_ignored(self): + # Behavior must be identical with or without the deprecated + # ``cleanup_channels`` query parameter. + ch = Channel.objects.create( + channel_number=300.0, + name="Auto", + channel_group=self.group, + auto_created=True, + auto_created_by=self.account, + ) + response = self.client.delete( + f"/api/m3u/accounts/{self.account.id}/?cleanup_channels=false" + ) + self.assertEqual(response.status_code, status.HTTP_200_OK) + self.assertEqual(response.data["deleted_channels"], 1) + self.assertFalse(Channel.objects.filter(id=ch.id).exists()) + + def test_no_op_when_account_has_no_auto_created_channels(self): + response = self.client.delete( + f"/api/m3u/accounts/{self.account.id}/" + ) + self.assertEqual(response.status_code, status.HTTP_200_OK) + self.assertEqual(response.data["deleted_channels"], 0) diff --git a/apps/m3u/tests/test_sync_compound.py b/apps/m3u/tests/test_sync_compound.py new file mode 100644 index 00000000..5f7b4742 --- /dev/null +++ b/apps/m3u/tests/test_sync_compound.py @@ -0,0 +1,240 @@ +""" +Compound-fixture sync tests. + +Where individual sync tests cover one variation at a time (multi-stream, +hidden, override, manual), this module seeds all of them in the same +fixture and asserts the constraints still hold when sync sees the full +mix on a single account. The point is to catch interactions that pass +each isolated test but break when the conditions overlap. +""" + +from datetime import timedelta + +from django.test import TestCase +from django.utils import timezone + +from apps.channels.models import ( + Channel, + ChannelGroup, + ChannelGroupM3UAccount, + ChannelOverride, + ChannelStream, + Stream, +) +from apps.m3u.models import M3UAccount +from apps.m3u.tasks import sync_auto_channels + + +def _scan_start_time(): + return (timezone.now() - timedelta(minutes=1)).isoformat() + + +class CompoundFixtureSyncTests(TestCase): + """ + Single fixture covers: multi-stream auto channel, hidden auto channel, + auto channel with channel_number override, manual channel that shares + the group with auto-created rows. After sync runs, every channel must + end up in the state its individual test would have asserted. + """ + + def setUp(self): + self.account = M3UAccount.objects.create( + name="Compound Provider", + server_url="http://example.com/compound.m3u", + ) + self.group = ChannelGroup.objects.create(name="Compound Group") + self.relation = ChannelGroupM3UAccount.objects.create( + m3u_account=self.account, + channel_group=self.group, + enabled=True, + auto_channel_sync=True, + auto_sync_channel_start=100, + auto_sync_channel_end=199, + ) + now = timezone.now() + + # ── Multi-stream auto channel: two streams, one fresh, one stale. + # The channel must survive the stale stream's disappearance because + # the fresh one keeps the channel alive. + self.multi_stream_a = Stream.objects.create( + name="MultiCh HD", + url="http://example.com/multi-a.m3u8", + m3u_account=self.account, + channel_group=self.group, + tvg_id="multi", + last_seen=now, + ) + self.multi_stream_b = Stream.objects.create( + name="MultiCh HD", + url="http://example.com/multi-b.m3u8", + m3u_account=self.account, + channel_group=self.group, + tvg_id="multi", + last_seen=now - timedelta(days=2), + ) + self.multi_channel = Channel.objects.create( + name="MultiCh HD", + channel_number=100, + channel_group=self.group, + auto_created=True, + auto_created_by=self.account, + ) + ChannelStream.objects.create( + channel=self.multi_channel, stream=self.multi_stream_a, order=0 + ) + ChannelStream.objects.create( + channel=self.multi_channel, stream=self.multi_stream_b, order=1 + ) + + # ── Hidden auto channel: visible from the table's perspective only + # to admins with the Hidden filter on. Sync must not reuse its + # channel_number for a different channel just because it's hidden. + self.hidden_stream = Stream.objects.create( + name="HiddenCh", + url="http://example.com/hidden.m3u8", + m3u_account=self.account, + channel_group=self.group, + tvg_id="hidden", + last_seen=now, + ) + self.hidden_channel = Channel.objects.create( + name="HiddenCh", + channel_number=101, + channel_group=self.group, + auto_created=True, + auto_created_by=self.account, + hidden_from_output=True, + ) + ChannelStream.objects.create( + channel=self.hidden_channel, stream=self.hidden_stream, order=0 + ) + + # ── Overridden auto channel: user pinned channel_number to 150 via + # an override row. Sync must not clobber the override; the + # effective channel_number stays 150 even though the raw column + # may evolve. + self.overridden_stream = Stream.objects.create( + name="OverriddenCh", + url="http://example.com/overridden.m3u8", + m3u_account=self.account, + channel_group=self.group, + tvg_id="overridden", + last_seen=now, + ) + self.overridden_channel = Channel.objects.create( + name="OverriddenCh", + channel_number=102, + channel_group=self.group, + auto_created=True, + auto_created_by=self.account, + ) + ChannelStream.objects.create( + channel=self.overridden_channel, + stream=self.overridden_stream, + order=0, + ) + ChannelOverride.objects.create( + channel=self.overridden_channel, + channel_number=150, + name="My Pinned Name", + ) + + # ── Manual channel sharing the group: auto_created=False, user + # picked channel_number 175 themselves. Sync must not touch this + # row at all. + self.manual_channel = Channel.objects.create( + name="ManualCh", + channel_number=175, + channel_group=self.group, + auto_created=False, + ) + + # New stream that has no existing channel; sync should create a + # fresh channel for it within the configured range and skip + # 100-102 (in use), 150 (overridden), 175 (manual). + self.new_stream = Stream.objects.create( + name="FreshCh", + url="http://example.com/fresh.m3u8", + m3u_account=self.account, + channel_group=self.group, + tvg_id="fresh", + last_seen=now, + ) + + def test_compound_fixture_each_invariant_holds_after_sync(self): + sync_auto_channels(self.account.id, scan_start_time=_scan_start_time()) + + # Multi-stream channel survives. + self.assertTrue( + Channel.objects.filter(id=self.multi_channel.id).exists(), + "Multi-stream channel was deleted even though one stream is alive", + ) + + # Hidden channel survives and remains hidden. + self.hidden_channel.refresh_from_db() + self.assertTrue(self.hidden_channel.hidden_from_output) + + # Overridden channel: the override row is intact, not cleared by + # sync. The pinned channel_number persists. + override = ChannelOverride.objects.get(channel=self.overridden_channel) + self.assertEqual(override.channel_number, 150) + self.assertEqual(override.name, "My Pinned Name") + + # Manual channel is untouched. + self.manual_channel.refresh_from_db() + self.assertFalse(self.manual_channel.auto_created) + self.assertEqual(self.manual_channel.channel_number, 175) + + # The fresh stream becomes a new auto channel; its channel_number + # falls inside the configured range and must not collide with any + # existing number (100-102 used by the auto channels, 150 pinned + # by the override, 175 the manual channel). + fresh_stream_id = self.new_stream.id + new_channel_qs = Channel.objects.filter( + channelstream__stream_id=fresh_stream_id, + auto_created=True, + auto_created_by=self.account, + ).distinct() + self.assertEqual( + new_channel_qs.count(), + 1, + "Sync did not create exactly one channel for the new stream", + ) + new_number = new_channel_qs.first().channel_number + self.assertIsNotNone(new_number) + self.assertGreaterEqual(new_number, 100) + self.assertLessEqual(new_number, 199) + self.assertNotIn(new_number, {100, 101, 102, 150, 175}) + + def test_hidden_channel_number_not_reassigned_to_new_stream(self): + # Targeted assertion isolated from the broader fixture invariants + # so the failure mode is unambiguous when this single property + # regresses. + sync_auto_channels(self.account.id, scan_start_time=_scan_start_time()) + + new_channel = ( + Channel.objects.filter( + channelstream__stream_id=self.new_stream.id, + auto_created=True, + auto_created_by=self.account, + ) + .distinct() + .first() + ) + self.assertIsNotNone(new_channel) + self.assertNotEqual( + new_channel.channel_number, + self.hidden_channel.channel_number, + "Hidden channel's number was reassigned; hidden channels must " + "still occupy their slot in the used_numbers set", + ) + + def test_override_channel_number_preserved_through_sync(self): + # If a future regression caused sync to write to ChannelOverride + # (it must not), this test catches it because the override would + # change after refresh. + original_pin = 150 + sync_auto_channels(self.account.id, scan_start_time=_scan_start_time()) + + override = ChannelOverride.objects.get(channel=self.overridden_channel) + self.assertEqual(override.channel_number, original_pin) diff --git a/apps/m3u/tests/test_sync_correctness.py b/apps/m3u/tests/test_sync_correctness.py new file mode 100644 index 00000000..e89ac410 --- /dev/null +++ b/apps/m3u/tests/test_sync_correctness.py @@ -0,0 +1,2051 @@ +""" +Correctness reproductions and fixes for sync_auto_channels. + +Each test documents a specific bug in the unpatched code: reproduces the bug +first (fails on HEAD prior to the Tier 2 patch), then is flipped to assert +the correct post-fix behavior. Comments call out the failure mode and the +fix location. +""" +from django.test import TestCase +from django.utils import timezone + +from apps.channels.models import ( + Channel, + ChannelGroup, + ChannelGroupM3UAccount, + ChannelStream, + Stream, +) +from apps.m3u.models import M3UAccount +from apps.m3u.tasks import sync_auto_channels + + +def _make_account(name="Test Provider", custom_properties=None): + return M3UAccount.objects.create( + name=name, + server_url="http://example.com/test.m3u", + custom_properties=custom_properties, + ) + + +def _make_group(name="Sports"): + return ChannelGroup.objects.create(name=name) + + +def _attach_group_to_account(account, group, custom_properties=None): + return ChannelGroupM3UAccount.objects.create( + m3u_account=account, + channel_group=group, + enabled=True, + auto_channel_sync=True, + auto_sync_channel_start=100, + custom_properties=custom_properties, + ) + + +def _make_stream(account, group, name="ESPN", tvg_id="espn"): + return Stream.objects.create( + name=name, + url=f"http://example.com/{name.lower()}.m3u8", + m3u_account=account, + channel_group=group, + tvg_id=tvg_id, + last_seen=timezone.now(), + ) + + +def _sync(account): + # Use a scan_start_time prior to the freshly-created streams so + # `last_seen__gte=scan_start_time` includes them. + return sync_auto_channels( + account.id, + scan_start_time=(timezone.now() - timezone.timedelta(minutes=1)).isoformat(), + ) + + +class CustomPropertiesTypeHandlingTests(TestCase): + """ + `custom_properties` is a JSONField but at least one historical + code path stored a JSON string instead of a dict. When + `sync_auto_channels` reads `group_relation.custom_properties` and calls + `.get()` on it, AttributeError is raised because the value is a string. + + Failure mode on unpatched code: + AttributeError: 'str' object has no attribute 'get' + + The outer try/except in sync_auto_channels catches the exception and + returns an error string, aborting the sync for every other well-formed + group in the same account. + """ + + def test_group_custom_properties_as_string_does_not_abort_sync(self): + account = _make_account() + group = _make_group(name="Sports") + # Deliberately stored as a JSON-encoded string (reproducing #432). + _attach_group_to_account( + account, + group, + custom_properties='{"force_dummy_epg": true}', + ) + _make_stream(account, group) + + result = _sync(account) + + # Post-fix: sync does not abort with an error and the stream is + # processed normally. Pre-fix this returned "Auto sync error: ...". + self.assertEqual(result.get("status"), "ok") + + def test_group_custom_properties_as_string_with_streams_still_creates_channels(self): + # Tighter version of the first test: with a real stream attached to + # the group, the pre-fix failure aborts before creating the channel. + # Post-fix, the channel is created normally and the string-typed + # custom_properties is treated as an empty dict (no custom regex, + # no group_override, etc.). + account = _make_account() + group = _make_group(name="Sports") + _attach_group_to_account( + account, + group, + custom_properties='{"name_regex_pattern": "HD"}', + ) + _make_stream(account, group, name="ESPN HD", tvg_id="espn") + + result = _sync(account) + + self.assertEqual(result.get("status"), "ok") + self.assertEqual( + Channel.objects.filter(auto_created=True, auto_created_by=account).count(), + 1, + "Exactly one channel should have been created from the single stream", + ) + + +class NullAutoCreatedByOrphanTests(TestCase): + """ + Channels with `auto_created=True, auto_created_by=NULL` are never touched + by sync (it filters on `auto_created_by=account`). These rows accumulate + indefinitely. + + The fix is a backfill migration that either re-attributes the row by + matching its linked stream to an M3U account, or deletes it if + unattributable. Until that migration runs, sync has no way to clean them + up. This test confirms the current behavior so we can write the migration + with correct expectations. + """ + + def test_null_auto_created_by_rows_are_untouched_by_sync(self): + account = _make_account() + group = _make_group(name="Entertainment") + _attach_group_to_account(account, group) + + orphan = Channel.objects.create( + name="OrphanChannel", + channel_number=999, + channel_group=group, + auto_created=True, + auto_created_by=None, + ) + + # Run sync with no streams; normal orphan cleanup only deletes rows + # scoped to the account. The NULL-owner row should still be there. + _sync(account) + + orphan.refresh_from_db() + self.assertIsNotNone(orphan.id, "Pre-fix: NULL-owner rows are not cleaned") + + +class AccountDeleteCleanupTests(TestCase): + """ + Deleting an M3UAccount unconditionally cascades auto-created + channels owned by the account. An auto-created channel without a + surviving provider has no useful state (it cannot sync, its + streams cascade away), so the destroy endpoint always removes + them. The legacy ``cleanup_channels`` query parameter is accepted + for backward compatibility but ignored. + """ + + def test_destroy_cascades_auto_created_channels_unconditionally(self): + from rest_framework.test import APIClient + from apps.accounts.models import User + + admin = User.objects.create_superuser( + username="admin_delcleanup_a", + password="pw", + user_level=10, + ) + account = _make_account() + group = _make_group(name="Entertainment") + Channel.objects.create( + name="ShouldCascade", + channel_number=700, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + + client = APIClient() + client.force_authenticate(user=admin) + response = client.delete(f"/api/m3u/accounts/{account.id}/") + + # Cascade returns 200 with the deleted count body so the UI can + # toast the number of channels removed alongside the account. + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data.get("deleted_channels"), 1) + self.assertEqual( + Channel.objects.filter(auto_created=True).count(), 0 + ) + + def test_destroy_with_cleanup_removes_auto_created_channels(self): + from rest_framework.test import APIClient + from apps.accounts.models import User + + admin = User.objects.create_superuser( + username="admin_delcleanup_b", + password="pw", + user_level=10, + ) + account = _make_account() + group = _make_group(name="News") + Channel.objects.create( + name="Cleanme1", + channel_number=710, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + Channel.objects.create( + name="Cleanme2", + channel_number=711, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + + client = APIClient() + client.force_authenticate(user=admin) + response = client.delete( + f"/api/m3u/accounts/{account.id}/?cleanup_channels=true" + ) + + # Cleanup returns 200 with the deleted count body (so the UI can + # toast "Deleted N channels"). Without cleanup it would be 204. + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data.get("deleted_channels"), 2) + self.assertEqual( + Channel.objects.filter(auto_created=True).count(), + 0, + ) + + def test_auto_created_channels_count_endpoint(self): + from rest_framework.test import APIClient + from apps.accounts.models import User + + admin = User.objects.create_superuser( + username="admin_delcleanup_c", + password="pw", + user_level=10, + ) + account = _make_account() + group = _make_group(name="Sports") + for i in range(3): + Channel.objects.create( + name=f"AutoChan{i}", + channel_number=800 + i, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + + client = APIClient() + client.force_authenticate(user=admin) + response = client.get( + f"/api/m3u/accounts/{account.id}/auto-created-channels-count/" + ) + + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data["count"], 3) + self.assertEqual(len(response.data["sample_names"]), 3) + + +class ChannelDeleteStopsProxyTests(TestCase): + """ + Issue #870: When an auto-sync refresh deletes a channel that has an + active proxy session, the session's Redis state survives, making the UI + "Stop" button fail with 'Channel not found'. Fix is a pre_delete signal + on Channel that calls ChannelService.stop_channel first, covering + manual, bulk, and sync-triggered deletes uniformly. + """ + + def test_pre_delete_signal_calls_stop_channel(self): + from unittest.mock import patch + + account = _make_account() + group = _make_group(name="Sports") + channel = Channel.objects.create( + name="ESPN", + channel_number=1, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + channel_uuid = str(channel.uuid) + + with patch( + "apps.proxy.ts_proxy.services.channel_service.ChannelService.stop_channel" + ) as mock_stop: + channel.delete() + + mock_stop.assert_called_once_with(channel_uuid) + + def test_pre_delete_signal_swallows_stop_errors(self): + """Proxy failure must not block the DB delete.""" + from unittest.mock import patch + + account = _make_account() + group = _make_group(name="Sports") + channel = Channel.objects.create( + name="ESPN", + channel_number=1, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + + with patch( + "apps.proxy.ts_proxy.services.channel_service.ChannelService.stop_channel", + side_effect=Exception("proxy is down"), + ): + channel.delete() + + self.assertFalse(Channel.objects.filter(id=channel.id).exists()) + + +class RangeEnforcementTests(TestCase): + """ + Tier 4 feature: optional per-group `auto_sync_channel_end` caps the + number range a group can use. Streams that don't fit get surfaced in + the completion notification as failures rather than silently spilling + into a neighboring group's range. + """ + + def test_range_allows_streams_within_bounds(self): + account = _make_account() + group = _make_group(name="Sports") + rel = _attach_group_to_account(account, group) + rel.auto_sync_channel_end = 105 + rel.auto_sync_channel_start = 100 + rel.save() + + for i in range(3): + _make_stream(account, group, name=f"S{i}", tvg_id=f"s{i}") + + result = _sync(account) + + self.assertEqual(result["status"], "ok") + self.assertEqual(result["channels_created"], 3) + self.assertEqual(result["channels_failed"], 0) + + def test_range_exhaustion_surfaces_failures(self): + account = _make_account() + group = _make_group(name="Sports") + rel = _attach_group_to_account(account, group) + # Range holds 2 slots: 100, 101. + rel.auto_sync_channel_start = 100 + rel.auto_sync_channel_end = 101 + rel.save() + + for i in range(4): + _make_stream(account, group, name=f"S{i}", tvg_id=f"s{i}") + + result = _sync(account) + + self.assertEqual(result["status"], "ok") + self.assertEqual(result["channels_created"], 2) + self.assertEqual(result["channels_failed"], 2) + self.assertEqual(len(result["failed_stream_details"]), 2) + for detail in result["failed_stream_details"]: + self.assertIn("range", detail["error"].lower()) + self.assertEqual(detail["reason"], "RANGE_EXHAUSTED") + + def test_used_numbers_seed_includes_override_pinned_values(self): + # Channel A has override.channel_number=42 pinning effective #42 + # globally. Sync creating a new channel B in fixed mode starting + # at 1 must skip 42; otherwise B's raw channel_number=42 collides + # with A's effective number, producing duplicate output entries + # at the same channel number. + from apps.channels.models import ChannelOverride + + account = _make_account() + group = _make_group(name="News") + rel = _attach_group_to_account(account, group) + rel.auto_sync_channel_start = 1.0 + rel.save() + + # A: pre-existing manual channel with override pinning #42. + a = Channel.objects.create( + channel_number=999.0, + name="UserPinned", + tvg_id="user_pinned", + channel_group=group, + auto_created=False, + ) + ChannelOverride.objects.create(channel=a, channel_number=42.0) + + # Provide enough new streams to walk past 42 in fixed mode. + for i in range(45): + _make_stream(account, group, name=f"S{i}", tvg_id=f"s{i}") + + result = _sync(account) + self.assertEqual(result["status"], "ok") + + new_numbers = list( + Channel.objects.filter( + auto_created=True, auto_created_by=account + ).values_list("channel_number", flat=True) + ) + self.assertNotIn( + 42.0, + new_numbers, + "Sync must skip override-pinned numbers; assigning #42 to a " + "new auto-channel duplicates A's effective channel number.", + ) + + def test_provider_mode_fallback_respects_range_start(self): + # Provider-mode streams without a usable stream_chno fall back to + # the next-available picker. The fallback must honor the group's + # configured `auto_sync_channel_start` so freshly-created channels + # never land below the user's chosen range. Without this, a group + # configured for [100, 200] silently spawned channels at #1 when + # the provider omitted channel-number metadata. + account = _make_account() + group = _make_group(name="Sports") + rel = _attach_group_to_account(account, group) + rel.auto_sync_channel_start = 100 + rel.auto_sync_channel_end = 200 + rel.custom_properties = { + "channel_numbering_mode": "provider", + "channel_numbering_fallback": 1, + } + rel.save() + + _make_stream(account, group, name="NoChno", tvg_id="nc") + + result = _sync(account) + + self.assertEqual(result["status"], "ok") + self.assertEqual(result["channels_created"], 1) + created = Channel.objects.get(auto_created=True, auto_created_by=account) + self.assertGreaterEqual(created.channel_number, 100) + self.assertLessEqual(created.channel_number, 200) + + +class ReservationBehaviorTests(TestCase): + """ + Ranges with both `auto_sync_channel_start` and `auto_sync_channel_end` + set are advisory at the UI layer. Sync does not treat another group's + declared range as off-limits; only channels that are actually assigned + a number count toward used_numbers. This lets two groups carrying the + same category (e.g. "Entertainment" from two different providers) + share a number range by cooperative fill, which is the intended UX + for merged-provider setups. + + The UI surfaces overlap between ranges as an advisory heads-up rather + than a blocking error, and the sync task enforces only real occupancy. + """ + + def test_self_reservation_lets_group_use_its_own_range(self): + # A group with a range assigns within that range. This is the + # baseline case the UI advisory is built around. + account = _make_account() + group = _make_group(name="Sports") + rel = _attach_group_to_account(account, group) + rel.auto_sync_channel_start = 1000 + rel.auto_sync_channel_end = 1010 + rel.save() + + for i in range(3): + _make_stream(account, group, name=f"S{i}", tvg_id=f"s{i}") + + result = _sync(account) + + self.assertEqual(result["status"], "ok") + self.assertEqual(result["channels_created"], 3) + numbers = sorted( + Channel.objects.filter( + channel_group=group, + auto_created=True, + auto_created_by=account, + ).values_list("channel_number", flat=True) + ) + self.assertTrue( + all(1000 <= n <= 1010 for n in numbers), + f"Channels outside own range: {numbers}", + ) + + def test_overlapping_ranges_cooperate_when_no_existing_channels(self): + # Two groups with overlapping ranges but no pre-existing channels + # in the overlap should both be able to fill numbers in the shared + # range. Sync only avoids actual occupancy, not declared intent. + account_a = _make_account(name="Provider A") + account_b = _make_account(name="Provider B") + group_a = _make_group(name="Entertainment A") + group_b = _make_group(name="Entertainment B") + + rel_a = _attach_group_to_account(account_a, group_a) + rel_a.auto_sync_channel_start = 1 + rel_a.auto_sync_channel_end = 10 + rel_a.save() + + rel_b = _attach_group_to_account(account_b, group_b) + rel_b.auto_sync_channel_start = 1 + rel_b.auto_sync_channel_end = 10 + rel_b.save() + + # 3 streams in A, 3 in B. With shared range 1-10 and no prior + # occupancy, A fills some numbers, B fills the remaining free ones. + for i in range(3): + _make_stream(account_a, group_a, name=f"A{i}", tvg_id=f"a{i}") + _make_stream(account_b, group_b, name=f"B{i}", tvg_id=f"b{i}") + + _sync(account_a) + _sync(account_b) + + all_numbers = sorted( + Channel.objects.filter(auto_created=True).values_list( + "channel_number", flat=True + ) + ) + self.assertEqual(len(all_numbers), 6) + # All numbers are unique and all fit inside the shared range. + self.assertEqual(len(set(all_numbers)), 6) + self.assertTrue( + all(1 <= n <= 10 for n in all_numbers), + f"Channels outside the shared range: {all_numbers}", + ) + + +class NumbersInRangeLookupTests(TestCase): + """ + Backend support for the inline range conflict warning on the group + settings form. Returns every channel whose effective channel_number + falls within [start, end], with context fields the frontend uses to + classify each hit (auto-created from this group + account vs anything + else). Matching uses effective values so overrides participate. + """ + + def _client(self): + from rest_framework.test import APIClient + from apps.accounts.models import User + + admin = User.objects.create_superuser( + username="admin_inrange", + password="pw", + user_level=10, + ) + client = APIClient() + client.force_authenticate(user=admin) + return client + + def test_returns_occupants_within_range(self): + group = _make_group(name="Sports") + Channel.objects.create( + name="CNN", channel_number=100, channel_group=group + ) + Channel.objects.create( + name="Local 5", channel_number=105, channel_group=group + ) + client = self._client() + + response = client.get( + "/api/channels/channels/numbers-in-range/?start=100&end=110" + ) + + self.assertEqual(response.status_code, 200) + names = sorted(o["name"] for o in response.data["occupants"]) + self.assertEqual(names, ["CNN", "Local 5"]) + + def test_returns_empty_when_no_channels_in_range(self): + client = self._client() + response = client.get( + "/api/channels/channels/numbers-in-range/?start=900&end=999" + ) + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data["occupants"], []) + + def test_single_number_lookup_when_end_omitted(self): + # Mirrors the case where the user has set Start # but not End #. + # Endpoint should treat omitted end as a single-number lookup. + group = _make_group(name="Sports") + Channel.objects.create( + name="CNN", channel_number=100, channel_group=group + ) + client = self._client() + response = client.get( + "/api/channels/channels/numbers-in-range/?start=100" + ) + self.assertEqual(response.status_code, 200) + self.assertEqual(len(response.data["occupants"]), 1) + + def test_override_channel_number_is_treated_as_effective(self): + from apps.channels.models import ChannelOverride + + group = _make_group(name="News") + ch = Channel.objects.create( + name="Raw", + channel_number=500, + channel_group=group, + auto_created=True, + ) + ChannelOverride.objects.create(channel=ch, channel_number=777) + client = self._client() + + # Raw value 500 must NOT match - overrides are authoritative. + miss = client.get( + "/api/channels/channels/numbers-in-range/?start=500&end=500" + ) + self.assertEqual(miss.data["occupants"], []) + + hit = client.get( + "/api/channels/channels/numbers-in-range/?start=777&end=777" + ) + self.assertEqual(len(hit.data["occupants"]), 1) + self.assertTrue( + hit.data["occupants"][0]["has_channel_number_override"] + ) + + def test_response_includes_group_account_and_override_flags(self): + # The frontend uses these three fields to decide whether a hit is + # a genuine collision or the expected output of the group it's + # currently configuring. The endpoint must carry them through. + account = _make_account() + group = _make_group(name="Sports") + Channel.objects.create( + name="CNN", + channel_number=100, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + client = self._client() + + response = client.get( + "/api/channels/channels/numbers-in-range/?start=100&end=100" + ) + + occupant = response.data["occupants"][0] + self.assertEqual(occupant["channel_group_id"], group.id) + self.assertTrue(occupant["auto_created"]) + self.assertEqual( + occupant["auto_created_by_account_id"], account.id + ) + self.assertFalse(occupant["has_channel_number_override"]) + + def test_manual_channel_exposed_with_auto_created_false(self): + # Manual channels are always a real collision worth surfacing. + # The response must flag them with auto_created=False and a null + # account id so the frontend classifier warns. + group = _make_group(name="Sports") + Channel.objects.create( + name="MyManual", channel_number=100, channel_group=group + ) + client = self._client() + + response = client.get( + "/api/channels/channels/numbers-in-range/?start=100&end=100" + ) + + occupant = response.data["occupants"][0] + self.assertFalse(occupant["auto_created"]) + self.assertIsNone(occupant["auto_created_by_account_id"]) + + def test_results_are_capped_at_50_entries(self): + # The endpoint caps at 50 to keep payloads bounded; the frontend + # only needs to know whether any unfiltered occupants remain. + group = _make_group(name="Sports") + for i in range(60): + Channel.objects.create( + name=f"Ch{i}", + channel_number=1000 + i, + channel_group=group, + ) + client = self._client() + response = client.get( + "/api/channels/channels/numbers-in-range/?start=1000&end=1100" + ) + self.assertEqual(len(response.data["occupants"]), 50) + + +class RegexPreviewTests(TestCase): + """ + Backend support for the find/replace and filter regex previews in the + auto-sync gear modal. Returns matched names plus full-group counts so + the UI can show "12 matches across 5,000 streams" without loading the + whole stream list client-side. Hard-caps at SCAN_CAP=5000 names per + call to keep the endpoint bounded on huge groups. + """ + + def _client(self): + from rest_framework.test import APIClient + from apps.accounts.models import User + + admin = User.objects.create_superuser( + username="admin_regex_preview", + password="pw", + user_level=10, + ) + client = APIClient() + client.force_authenticate(user=admin) + return client + + def _make_account(self): + return _make_account(name="Regex Preview Provider") + + def test_find_replace_returns_only_changed_names(self): + account = self._make_account() + group = _make_group(name="Sports") + for name in ["ESPN HD", "Fox Sports HD", "CNN"]: + Stream.objects.create( + name=name, + url=f"http://example.com/{name}.m3u8", + m3u_account=account, + channel_group=group, + last_seen=timezone.now(), + ) + client = self._client() + + response = client.get( + "/api/channels/streams/regex-preview/" + "?channel_group=Sports&find=%20HD%24&replace=" + ) + + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data["find_match_count"], 2) + names = sorted(m["before"] for m in response.data["find_matches"]) + self.assertEqual(names, ["ESPN HD", "Fox Sports HD"]) + # Replacement is correctly applied in the after field. + self.assertEqual( + sorted(m["after"] for m in response.data["find_matches"]), + ["ESPN", "Fox Sports"], + ) + self.assertEqual(response.data["total_in_group"], 3) + self.assertEqual(response.data["total_scanned"], 3) + self.assertFalse(response.data["scan_limit_hit"]) + + def test_filter_returns_matched_names_with_count(self): + account = self._make_account() + group = _make_group(name="Sports") + for name in ["Sports Central", "News 24", "Sports Live"]: + Stream.objects.create( + name=name, + url=f"http://example.com/{name}.m3u8", + m3u_account=account, + channel_group=group, + last_seen=timezone.now(), + ) + client = self._client() + + response = client.get( + "/api/channels/streams/regex-preview/" + "?channel_group=Sports&match=%5ESports" + ) + + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data["filter_match_count"], 2) + matched_names = sorted( + m["name"] for m in response.data["filter_matches"] + ) + self.assertEqual(matched_names, ["Sports Central", "Sports Live"]) + + def test_returns_zero_match_counts_when_pattern_matches_nothing(self): + account = self._make_account() + group = _make_group(name="Sports") + for name in ["CNN", "MSNBC"]: + Stream.objects.create( + name=name, + url=f"http://example.com/{name}.m3u8", + m3u_account=account, + channel_group=group, + last_seen=timezone.now(), + ) + client = self._client() + + response = client.get( + "/api/channels/streams/regex-preview/" + "?channel_group=Sports&find=ESPN&replace=Whatever" + ) + + self.assertEqual(response.data["find_match_count"], 0) + self.assertEqual(response.data["find_matches"], []) + + def test_invalid_find_pattern_returns_error_field(self): + # The endpoint reports compile errors via response fields rather + # than 400 so the UI can surface them inline without the request + # being treated as a hard failure. + account = self._make_account() + group = _make_group(name="Sports") + Stream.objects.create( + name="ESPN", + url="http://example.com/espn.m3u8", + m3u_account=account, + channel_group=group, + last_seen=timezone.now(), + ) + client = self._client() + + response = client.get( + "/api/channels/streams/regex-preview/" + "?channel_group=Sports&find=(" + ) + + self.assertEqual(response.status_code, 200) + self.assertIn("find_error", response.data) + + def test_scan_limit_hit_when_group_exceeds_5000_streams(self): + # Build a group with 5050 streams so we cross the SCAN_CAP boundary. + # The endpoint must still return promptly and flag scan_limit_hit + # so the UI can disclose that the preview isn't full coverage. + account = self._make_account() + group = _make_group(name="Bigly") + Stream.objects.bulk_create( + [ + Stream( + name=f"Stream {i}", + url=f"http://example.com/{i}.m3u8", + m3u_account=account, + channel_group=group, + last_seen=timezone.now(), + ) + for i in range(5050) + ] + ) + client = self._client() + + response = client.get( + "/api/channels/streams/regex-preview/" + "?channel_group=Bigly&find=Stream" + ) + + self.assertEqual(response.status_code, 200) + self.assertTrue(response.data["scan_limit_hit"]) + self.assertEqual(response.data["total_in_group"], 5050) + self.assertEqual(response.data["total_scanned"], 5000) + + def test_channel_group_required(self): + client = self._client() + response = client.get("/api/channels/streams/regex-preview/") + self.assertEqual(response.status_code, 400) + + def test_exclude_returns_matched_names_with_count(self): + # Exclude pattern returns the streams it would remove, mirroring + # the include preview shape so the UI can render both side-by-side. + account = self._make_account() + group = _make_group(name="Sports") + for name in ["Sports Live", "Sports TEST", "Sports BACKUP"]: + Stream.objects.create( + name=name, + url=f"http://example.com/{name}.m3u8", + m3u_account=account, + channel_group=group, + last_seen=timezone.now(), + ) + client = self._client() + + response = client.get( + "/api/channels/streams/regex-preview/" + "?channel_group=Sports&exclude=TEST%7CBACKUP" + ) + + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data["exclude_match_count"], 2) + names = sorted(m["name"] for m in response.data["exclude_matches"]) + self.assertEqual(names, ["Sports BACKUP", "Sports TEST"]) + + +class ExcludeRegexSyncTests(TestCase): + """ + Sync respects the per-group `name_match_exclude_regex` custom property + by dropping any stream whose name matches the pattern, after any + include filter narrows the set. Invalid patterns are logged and + skipped rather than aborting the whole sync. + """ + + def test_exclude_pattern_skips_matching_streams(self): + account = _make_account() + group = _make_group(name="Sports") + rel = _attach_group_to_account( + account, + group, + custom_properties={"name_match_exclude_regex": "TEST|BACKUP"}, + ) + rel.auto_sync_channel_start = 100 + rel.save() + for name in ["Sports Live", "Sports TEST", "Sports BACKUP", "Sports Pro"]: + _make_stream( + account, group, name=name, tvg_id=name.replace(" ", "_") + ) + + result = _sync(account) + + self.assertEqual(result["status"], "ok") + # Only Sports Live and Sports Pro should have been turned into channels. + names = sorted( + Channel.objects.filter( + channel_group=group, + auto_created=True, + auto_created_by=account, + ).values_list("name", flat=True) + ) + self.assertEqual(names, ["Sports Live", "Sports Pro"]) + + def test_invalid_exclude_pattern_does_not_abort_sync(self): + # Bad regex must be logged + ignored, not crash the sync. The + # streams are imported as if no exclude filter was set. + account = _make_account() + group = _make_group(name="Sports") + rel = _attach_group_to_account( + account, + group, + custom_properties={"name_match_exclude_regex": "("}, + ) + rel.auto_sync_channel_start = 100 + rel.save() + _make_stream(account, group, name="ESPN", tvg_id="espn") + + result = _sync(account) + + self.assertEqual(result["status"], "ok") + self.assertEqual(result["channels_created"], 1) + + +class ChannelOverrideClearResponseTests(TestCase): + """ + Clearing all overrides on a channel via PATCH `override=null` must + return a response where `override` is null and every effective_* + value reflects the post-clear (provider) state. The previous + implementation used a queryset-level delete which left Django's + reverse-OneToOne cache pointing at the just-deleted row; serializing + the response on the same instance returned the stale override data + and kept the frontend's "Clear All Overrides" button stuck visible + after a successful clear. + """ + + def test_clear_response_carries_null_override(self): + from rest_framework.test import APIClient + from apps.accounts.models import User + from apps.channels.models import ChannelOverride + + admin = User.objects.create_superuser( + username="admin_clear_response", + password="pw", + user_level=10, + ) + group = _make_group(name="Sports") + ch = Channel.objects.create( + name="ProviderName", + channel_number=200, + channel_group=group, + auto_created=True, + ) + ChannelOverride.objects.create( + channel=ch, name="UserOverrideName" + ) + client = APIClient() + client.force_authenticate(user=admin) + + response = client.patch( + f"/api/channels/channels/{ch.id}/", + {"override": None}, + format="json", + ) + + self.assertEqual(response.status_code, 200) + self.assertIsNone(response.data["override"]) + # The effective name must come from the provider value now that + # the override is gone, not the cached override that was still + # attached to the in-memory instance. + self.assertEqual(response.data["effective_name"], "ProviderName") + + +class HiddenChannelPreservationTests(TestCase): + """ + Hidden channels (`hidden_from_output=True`) must be preserved across every + sync cleanup path so users can rely on the hide flag as "keep this + channel even if its source stream temporarily disappears". Common + case: event / PPV / seasonal channels whose provider stream comes + and goes between refreshes - if they get deleted, the user loses + the hide state, any overrides, and the channel's identity. + """ + + def test_hidden_channel_survives_when_its_stream_disappears(self): + # Build a channel auto-created from a stream that exists in the + # group, then mark it hidden, then run a sync where the stream is + # absent from the current scan window. The channel must remain. + account = _make_account() + group = _make_group(name="PPV") + rel = _attach_group_to_account(account, group) + rel.auto_sync_channel_start = 100 + rel.save() + + from datetime import timedelta + + old_stream_seen = timezone.now() - timedelta(days=1) + stream = Stream.objects.create( + name="Event A", + url="http://example.com/event-a.m3u8", + m3u_account=account, + channel_group=group, + tvg_id="event-a", + last_seen=old_stream_seen, + ) + ch = Channel.objects.create( + name="Event A", + channel_number=100, + channel_group=group, + auto_created=True, + auto_created_by=account, + hidden_from_output=True, + ) + ChannelStream.objects.create(channel=ch, stream=stream, order=0) + + # Sync's scan_start_time is now (stream's last_seen is older, + # so the stream is excluded from current_streams - simulating + # the provider dropping it). Without the hidden_from_output guard, the + # channel would be cleaned up here. + result = _sync(account) + + self.assertEqual(result["status"], "ok") + self.assertTrue( + Channel.objects.filter(id=ch.id).exists(), + "Hidden channel was deleted when its source stream " + "disappeared from the current sync window", + ) + + def test_visible_channel_for_disappeared_stream_is_still_deleted(self): + # The hide-preservation rule must not accidentally apply to + # visible channels - those should still get cleaned up when + # their source stream goes away. Otherwise sync would never + # remove anything. + account = _make_account() + group = _make_group(name="Sports") + rel = _attach_group_to_account(account, group) + rel.auto_sync_channel_start = 100 + rel.save() + + from datetime import timedelta + + stream = Stream.objects.create( + name="OldStream", + url="http://example.com/old.m3u8", + m3u_account=account, + channel_group=group, + tvg_id="old", + last_seen=timezone.now() - timedelta(days=1), + ) + ch = Channel.objects.create( + name="OldStream", + channel_number=100, + channel_group=group, + auto_created=True, + auto_created_by=account, + hidden_from_output=False, + ) + ChannelStream.objects.create(channel=ch, stream=stream, order=0) + + result = _sync(account) + + self.assertEqual(result["status"], "ok") + self.assertFalse( + Channel.objects.filter(id=ch.id).exists(), + "Visible channel should have been cleaned up when its " + "source stream disappeared", + ) + + +class CompactNumberingTests(TestCase): + """ + Per-group `compact_numbering` custom_property packs visible auto- + created channels into the group's [start, end] range; hidden channels + release their channel numbers; channel_number overrides act as + reservations that survive hide/unhide cycles. Exercised across full + sync, the post_save signal for single-channel unhide, and the + explicit re-pack endpoint. + """ + + def _compact_account_with_group(self, start=100, end=110, name="Sports"): + account = _make_account() + group = _make_group(name=name) + rel = _attach_group_to_account( + account, group, custom_properties={"compact_numbering": True} + ) + rel.auto_sync_channel_start = start + rel.auto_sync_channel_end = end + rel.save() + return account, group, rel + + def test_full_sync_packs_visible_and_releases_hidden(self): + # Five streams, three of which produce visible channels and two of + # which we mark hidden after the first sync. Compact pack should + # leave the three visible channels at 100/101/102 and NULL out the + # hidden ones. + account, group, rel = self._compact_account_with_group( + start=100, end=110 + ) + for i in range(5): + _make_stream(account, group, name=f"S{i}", tvg_id=f"s{i}") + + first = _sync(account) + self.assertEqual(first["status"], "ok") + self.assertEqual(first["channels_created"], 5) + + # Hide two of the resulting channels. + hidden_targets = list( + Channel.objects.filter( + auto_created=True, auto_created_by=account + ).order_by("id")[:2] + ) + Channel.objects.filter( + id__in=[c.id for c in hidden_targets] + ).update(hidden_from_output=True) + + # Re-run sync; compact pack runs at the end. + _sync(account) + + nums = sorted( + Channel.objects.filter( + auto_created=True, + auto_created_by=account, + hidden_from_output=False, + ).values_list("channel_number", flat=True) + ) + self.assertEqual(nums, [100, 101, 102]) + + hidden_nums = list( + Channel.objects.filter( + auto_created=True, + auto_created_by=account, + hidden_from_output=True, + ).values_list("channel_number", flat=True) + ) + self.assertTrue( + all(n is None for n in hidden_nums), + f"Hidden channels still hold numbers: {hidden_nums}", + ) + + def test_override_pinned_number_survives_compact_pass(self): + # Channel B has a channel_number override of 105. After compact + # pack, B's effective number stays at 105, and the other visible + # channels get packed around it (skipping 105 in the pool). + from apps.channels.models import ChannelOverride + + account, group, rel = self._compact_account_with_group( + start=100, end=110 + ) + for i in range(4): + _make_stream(account, group, name=f"S{i}", tvg_id=f"s{i}") + + _sync(account) + all_channels = list( + Channel.objects.filter( + auto_created=True, auto_created_by=account + ).order_by("id") + ) + ChannelOverride.objects.create( + channel=all_channels[1], channel_number=105 + ) + + _sync(account) + + # Override-pinned channel: raw cleared to None (override controls + # effective). The other three channels packed into 100/101/102 - + # skipping 105 because the override reserved it. + non_pinned = sorted( + float(n) + for n in Channel.objects.filter( + auto_created=True, + auto_created_by=account, + override__isnull=True, + ).values_list("channel_number", flat=True) + if n is not None + ) + self.assertEqual(non_pinned, [100.0, 101.0, 102.0]) + pinned = Channel.objects.get(id=all_channels[1].id) + self.assertIsNone(pinned.channel_number) + + def test_unhide_signal_assigns_immediately(self): + # Toggle hide → unhide on a single channel; the post_save signal + # under compact mode should give it a number from the range + # without requiring a sync pass to run. + account, group, rel = self._compact_account_with_group( + start=100, end=110 + ) + ch = Channel.objects.create( + name="Test", + channel_number=None, + channel_group=group, + auto_created=True, + auto_created_by=account, + hidden_from_output=True, + ) + + ch.hidden_from_output = False + ch.save() + ch.refresh_from_db() + + self.assertEqual(ch.channel_number, 100) + + def test_repack_endpoint_packs_around_overrides(self): + # Simulate a user who set overrides on a couple of channels to + # pin specific numbers and then clicks Re-pack to compact the + # rest. Override-pinned channels survive; everything else fills + # the remaining slots in order. + from rest_framework.test import APIClient + from apps.accounts.models import User + from apps.channels.models import ChannelOverride + + account, group, rel = self._compact_account_with_group( + start=100, end=110 + ) + # Six visible channels, no overrides + channels = [ + Channel.objects.create( + name=f"C{i}", + channel_number=200 + i, # arbitrary starting numbers + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + for i in range(6) + ] + # Pin two of them via override + ChannelOverride.objects.create(channel=channels[0], channel_number=110) + ChannelOverride.objects.create(channel=channels[1], channel_number=105) + + admin = User.objects.create_superuser( + username="admin_repack", password="pw", user_level=10 + ) + client = APIClient() + client.force_authenticate(user=admin) + response = client.post( + f"/api/m3u/accounts/{account.id}/repack-group/?channel_group_id={group.id}" + ) + + self.assertEqual(response.status_code, 200) + # 4 non-pinned channels assigned, no failures + self.assertEqual(response.data["assigned"], 4) + self.assertEqual(response.data["failed"], 0) + # Non-pinned channels get the lowest available numbers in the + # range, skipping 105 and 110 (override reservations). + non_pinned_nums = sorted( + Channel.objects.filter( + id__in=[c.id for c in channels[2:]] + ).values_list("channel_number", flat=True) + ) + self.assertEqual(non_pinned_nums, [100, 101, 102, 103]) + + def test_repack_reports_failed_when_range_too_small(self): + # Range covers 3 slots but visible channels need 5; the extra 2 + # are reported as failed and have their channel_number set to + # None so their state is unambiguous. + account, group, rel = self._compact_account_with_group( + start=100, end=102 + ) + for i in range(5): + Channel.objects.create( + name=f"C{i}", + channel_number=900 + i, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + + from apps.channels.compact_numbering import repack_group + + result = repack_group(rel) + self.assertEqual(result["assigned"], 3) + self.assertEqual(result["failed"], 2) + # Those that didn't fit got their channel_number nulled + nulled = Channel.objects.filter( + channel_group=group, + channel_number__isnull=True, + ).count() + self.assertEqual(nulled, 2) + + def test_assign_releases_lock_on_outer_atomic_rollback(self): + # The Redis-backed task lock is not transactional, so a release + # scheduled via `transaction.on_commit` is silently discarded + # when the caller's outer atomic rolls back. This left the lock + # held until its 5-minute TTL expired and silently blocked all + # subsequent syncs for the affected account. The function now + # releases via try/finally so the lock comes back regardless + # of outer transaction state. + from unittest.mock import patch + from django.db import transaction + from apps.channels.compact_numbering import ( + assign_compact_numbers_for_channels, + ) + from core.utils import acquire_task_lock, release_task_lock + + account, group, rel = self._compact_account_with_group( + start=100, end=105 + ) + ch = Channel.objects.create( + name="C1", + channel_group=group, + auto_created=True, + auto_created_by=account, + hidden_from_output=False, + ) + + # Make sure the lock starts free. + release_task_lock("refresh_single_m3u_account", account.id) + + try: + with transaction.atomic(): + assign_compact_numbers_for_channels([ch.id]) + # Simulate downstream work in the outer atomic blowing up + # AFTER the inner assignment succeeded. + raise RuntimeError("simulated downstream failure") + except RuntimeError: + pass + + # If the lock is still held, the next acquire returns False and + # subsequent syncs would silently skip auto-sync. + self.assertTrue( + acquire_task_lock("refresh_single_m3u_account", account.id), + "Lock leaked after outer atomic rollback", + ) + # Clean up so other tests in this class are not affected. + release_task_lock("refresh_single_m3u_account", account.id) + + +class OverrideChannelNumberValidationTests(TestCase): + """ + Override PATCH intentionally permits duplicate channel_number values: + users may want two entries at the same number (e.g., one of them + hidden from output). Sync's used_numbers set still avoids collisions + on its own writes via set membership, so cross-provider merge + behavior is unaffected by allowing user-set duplicates. + """ + + def test_override_channel_number_duplicate_is_allowed(self): + from rest_framework.test import APIClient + from apps.accounts.models import User + from apps.channels.models import ChannelOverride + + admin = User.objects.create_superuser( + username="admin_override_a", + password="pw", + user_level=10, + ) + group = _make_group(name="Sports") + Channel.objects.create( + name="Existing", + channel_number=500, + channel_group=group, + ) + target = Channel.objects.create( + name="Target", + channel_number=501, + channel_group=group, + auto_created=True, + ) + + client = APIClient() + client.force_authenticate(user=admin) + response = client.patch( + f"/api/channels/channels/{target.id}/", + {"override": {"channel_number": 500}}, + format="json", + ) + + self.assertEqual(response.status_code, 200) + self.assertEqual( + ChannelOverride.objects.get(channel=target).channel_number, 500 + ) + + def test_override_channel_number_reusing_own_is_allowed(self): + # Editing an override and re-submitting the same number the channel + # already effectively holds must succeed (no "conflicts with itself"). + from rest_framework.test import APIClient + from apps.accounts.models import User + + admin = User.objects.create_superuser( + username="admin_override_b", + password="pw", + user_level=10, + ) + group = _make_group(name="Sports") + ch = Channel.objects.create( + name="Target", + channel_number=700, + channel_group=group, + auto_created=True, + ) + + client = APIClient() + client.force_authenticate(user=admin) + response = client.patch( + f"/api/channels/channels/{ch.id}/", + {"override": {"channel_number": 700, "name": "Renamed"}}, + format="json", + ) + + self.assertEqual(response.status_code, 200) + + +class SyncPerformanceRegressionTests(TestCase): + """ + Query-count and throughput guards. The numbers below are ceilings for the + specific scenario, not tight lower bounds. They exist to catch regressions + where a future edit reintroduces N+1 lookups on logos, EPG, or + per-channel ChannelStream joins. If a ceiling needs to be raised, + investigate first (the original audit documented the cost of each pattern). + """ + + def test_sync_of_ten_streams_is_not_n_plus_one_in_logo_lookups(self): + from apps.channels.models import Logo + + account = _make_account() + group = _make_group(name="Sports") + _attach_group_to_account(account, group) + + # Seed 10 streams with 3 distinct logo URLs so the batch cache has + # an interesting ratio of cache hits to misses. + logo_urls = [ + "http://logos.example.com/a.png", + "http://logos.example.com/b.png", + "http://logos.example.com/c.png", + ] + for i in range(10): + Stream.objects.create( + name=f"Chan{i}", + url=f"http://example.com/{i}.m3u8", + m3u_account=account, + channel_group=group, + tvg_id=f"tvg{i}", + logo_url=logo_urls[i % 3], + last_seen=timezone.now(), + ) + + # Count Logo queries during the sync. Without batching, this would + # issue at least one SELECT per stream (10+) plus 10 separate INSERTs. + # With batching, a single SELECT populates the cache for all 3 URLs + # and each new Logo row requires one INSERT (the get_or_create miss + # path). Allow headroom for the initial cache-populate query and + # per-insert overhead, but keep the total well below the N+1 count. + from django.db import connection + from django.test.utils import CaptureQueriesContext + + with CaptureQueriesContext(connection) as ctx: + result = _sync(account) + + logo_queries = [ + q for q in ctx.captured_queries + if 'dispatcharr_channels_logo' in q['sql'].lower() + ] + self.assertEqual(result["status"], "ok") + self.assertEqual(result["channels_created"], 10) + # Target: at most ~6 Logo-related queries for 10 streams / 3 urls. + # Without the batch cache the count would be 10+ SELECTs + 3 INSERTs. + self.assertLessEqual( + len(logo_queries), + 8, + f"Logo queries: {len(logo_queries)} (expected <= 8 after batching)", + ) + + +class OrphanCleanupModeTests(TestCase): + """ + Account-level `custom_properties.orphan_channel_cleanup` is a 3-state + selector that governs how sync handles auto-created channels whose + source streams have disappeared. + + - "always" (default; absent key behaves the same): delete every orphan + auto-created channel. + - "preserve_customized": delete orphans without a ChannelOverride row; + preserve those with one. + - "never": preserve every orphan auto-created channel. + + Hidden channels (`hidden_from_output=True`) are universally preserved + regardless of mode, because the user has explicitly signaled "keep this + around but do not show clients". + """ + + def _set_mode(self, account, mode): + account.custom_properties = {"orphan_channel_cleanup": mode} + account.save() + + def test_default_mode_when_key_absent_is_always(self): + # Account with no custom_properties.orphan_channel_cleanup value + # behaves like "always": orphans get cleaned up. + account = _make_account() + group = _make_group(name="Sports") + _attach_group_to_account(account, group) + Channel.objects.create( + name="OldESPN", + channel_number=500, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + + result = _sync(account) + + self.assertEqual(result["status"], "ok") + self.assertEqual(result["channels_deleted"], 1) + self.assertEqual( + Channel.objects.filter( + auto_created=True, auto_created_by=account + ).count(), + 0, + "Default (absent key): orphans must be cleaned up", + ) + + def test_always_mode_removes_orphan_with_override(self): + from apps.channels.models import ChannelOverride + + account = _make_account() + self._set_mode(account, "always") + group = _make_group(name="Sports") + _attach_group_to_account(account, group) + ch = Channel.objects.create( + name="OldESPN", + channel_number=500, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + ChannelOverride.objects.create(channel=ch, name="My Custom ESPN") + + result = _sync(account) + + self.assertEqual(result["channels_deleted"], 1) + self.assertFalse( + Channel.objects.filter(id=ch.id).exists(), + "Always mode: even customized orphans get deleted", + ) + + def test_preserve_customized_mode_spares_overrides(self): + from apps.channels.models import ChannelOverride + + account = _make_account() + self._set_mode(account, "preserve_customized") + group = _make_group(name="Sports") + _attach_group_to_account(account, group) + plain = Channel.objects.create( + name="OldPlain", + channel_number=500, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + customized = Channel.objects.create( + name="OldCustomized", + channel_number=501, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + ChannelOverride.objects.create(channel=customized, name="My Renamed") + + _sync(account) + + self.assertFalse( + Channel.objects.filter(id=plain.id).exists(), + "Preserve-customized mode: orphan without override is removed", + ) + self.assertTrue( + Channel.objects.filter(id=customized.id).exists(), + "Preserve-customized mode: orphan WITH override is preserved", + ) + + def test_never_mode_preserves_all_orphans(self): + account = _make_account() + self._set_mode(account, "never") + group = _make_group(name="Sports") + _attach_group_to_account(account, group) + Channel.objects.create( + name="OldA", + channel_number=500, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + Channel.objects.create( + name="OldB", + channel_number=501, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + + result = _sync(account) + + self.assertEqual(result["channels_deleted"], 0) + self.assertEqual( + Channel.objects.filter( + auto_created=True, auto_created_by=account + ).count(), + 2, + "Never mode: every orphan survives", + ) + + def test_hidden_channels_universally_preserved(self): + # Hidden orphans must survive in every mode, including the + # default "always" mode. + for mode in ("always", "preserve_customized", "never"): + with self.subTest(mode=mode): + account = _make_account(name=f"Provider-{mode}") + self._set_mode(account, mode) + group = _make_group(name=f"Sports-{mode}") + _attach_group_to_account(account, group) + Channel.objects.create( + name=f"Hidden-{mode}", + channel_number=600, + channel_group=group, + auto_created=True, + auto_created_by=account, + hidden_from_output=True, + ) + + _sync(account) + + self.assertEqual( + Channel.objects.filter( + auto_created=True, + auto_created_by=account, + hidden_from_output=True, + ).count(), + 1, + f"Hidden channel must survive cleanup in mode={mode}", + ) + + +class MultiStreamChannelTests(TestCase): + """ + A user can manually attach more than one Stream to an auto-created + Channel (typical for backup feeds: same channel, two providers' + versions). Sync must not delete the channel just because ONE of + those streams disappears - other streams may still be valid. + + The deletion path was per-stream: for stream_id, channel in + existing_channel_map.items(): delete the channel if stream_id is not + in processed_stream_ids. With two map entries pointing at the same + channel, the first iteration would queue the channel for delete even + if the second stream is still alive. + + Even more subtle: existing_channel_map gets a fresh Channel instance + per ChannelStream row from the joined query. With two entries, + in-memory mutations during the per-stream iteration would land on + different Python instances and silently drop changes from later + iterations when bulk_update fires on only one of them. + """ + + def test_channel_with_two_streams_survives_when_one_disappears(self): + from datetime import timedelta + + account = _make_account() + group = _make_group(name="Movies") + rel = _attach_group_to_account(account, group) + rel.auto_sync_channel_start = 100 + rel.save() + + # Two streams in the same group, same account, attached to one + # auto-created channel. Stream A is current, Stream B is stale + # (provider dropped it). + stream_a = Stream.objects.create( + name="StarMovie HD", + url="http://example.com/star-a.m3u8", + m3u_account=account, + channel_group=group, + tvg_id="star", + last_seen=timezone.now(), + ) + stream_b = Stream.objects.create( + name="StarMovie HD", + url="http://example.com/star-b.m3u8", + m3u_account=account, + channel_group=group, + tvg_id="star", + last_seen=timezone.now() - timedelta(days=1), + ) + ch = Channel.objects.create( + name="StarMovie HD", + channel_number=100, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + ChannelStream.objects.create(channel=ch, stream=stream_a, order=0) + ChannelStream.objects.create(channel=ch, stream=stream_b, order=1) + + # Sync sees stream_a as current (live) and stream_b as gone. + # Stream_a means the channel is still wanted; the channel must + # survive. Direct DB check works on both old and new sync return + # shapes (baseline returns a string, overhaul returns a dict). + _sync(account) + + self.assertTrue( + Channel.objects.filter(id=ch.id).exists(), + "Multi-stream channel was deleted even though one of its " + "streams is still alive", + ) + + def test_channel_with_two_streams_metadata_consistent_after_sync(self): + # When two ChannelStream rows resolve to the same Channel, + # in-memory mutations from each iteration must land on the SAME + # Channel instance so the post-loop bulk_update writes the + # merged final state. With per-row Channel instances (the bug), + # later iterations would mutate a different in-memory copy and + # those changes would be lost. + + account = _make_account() + group = _make_group(name="Sports") + rel = _attach_group_to_account(account, group) + rel.auto_sync_channel_start = 100 + rel.save() + + # Two streams attached to the same channel, both currently + # served by the provider. Each carries different tvg_id so the + # sync code wants to update the channel's tvg_id field. + stream_a = Stream.objects.create( + name="SportsCh HD", + url="http://example.com/sports-a.m3u8", + m3u_account=account, + channel_group=group, + tvg_id="sports-a", + last_seen=timezone.now(), + ) + stream_b = Stream.objects.create( + name="SportsCh HD", + url="http://example.com/sports-b.m3u8", + m3u_account=account, + channel_group=group, + tvg_id="sports-b", + last_seen=timezone.now(), + ) + ch = Channel.objects.create( + name="OldName", + tvg_id="initial", + channel_number=100, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + ChannelStream.objects.create(channel=ch, stream=stream_a, order=0) + ChannelStream.objects.create(channel=ch, stream=stream_b, order=1) + + _sync(account) + + ch.refresh_from_db() + # The channel exists, was updated, and either tvg_id ('sports-a' + # or 'sports-b') is acceptable - the key invariant is that it is + # NOT 'initial' (the pre-sync value), which would mean updates + # were silently dropped because they targeted a different + # in-memory instance than the one bulk_update'd. + self.assertNotEqual( + ch.tvg_id, + "initial", + "tvg_id was not updated; in-memory mutations to the " + "channel were dropped (multi-stream identity bug)", + ) + + +class HiddenChannelNumberCollisionTests(TestCase): + """ + Sync seeds `used_numbers` from existing channels EXCEPT those owned by + the account being synced (those will be re-numbered). The original + pattern excluded ALL of the account's auto-created channels - including + HIDDEN ones, which the renumber loop never visits because they are not + in `current_streams`. Result: the hidden channel's number was free to + be re-assigned to a brand-new visible channel, producing two channels + with the same channel_number on disk. + """ + + def test_hidden_channels_keep_their_number_reserved_during_sync(self): + from datetime import timedelta + + account = _make_account() + group_a = _make_group(name="Hidden Group") + group_b = _make_group(name="New Group") + rel_a = _attach_group_to_account(account, group_a) + rel_a.auto_sync_channel_start = 100 + rel_a.save() + rel_b = _attach_group_to_account(account, group_b) + rel_b.auto_sync_channel_start = 100 + rel_b.save() + + # A hidden auto-created channel pinned at #100 in group_a. + # Its source stream is gone (last_seen old), but hidden_from_output=True + # protects the channel from cleanup. + old_seen = timezone.now() - timedelta(days=1) + Stream.objects.create( + name="HiddenStream", + url="http://example.com/hidden.m3u8", + m3u_account=account, + channel_group=group_a, + tvg_id="hid", + last_seen=old_seen, + ) + hidden_ch = Channel.objects.create( + name="HiddenChannel", + channel_number=100, + channel_group=group_a, + auto_created=True, + auto_created_by=account, + hidden_from_output=True, + ) + + # A NEW stream in group_b, no channel yet. Sync should create a + # channel for it and pick its number. If hidden_ch's number is + # free (the bug), this new channel would also get 100, colliding. + Stream.objects.create( + name="NewStream", + url="http://example.com/new.m3u8", + m3u_account=account, + channel_group=group_b, + tvg_id="new", + last_seen=timezone.now(), + ) + + _sync(account) + + new_ch = Channel.objects.filter( + auto_created=True, + auto_created_by=account, + channel_group=group_b, + ).first() + self.assertIsNotNone(new_ch, "Sync did not create a channel for the new stream") + + # Both rows must coexist with DIFFERENT channel_numbers. + hidden_ch.refresh_from_db() + self.assertNotEqual( + new_ch.channel_number, + hidden_ch.channel_number, + f"New channel #{new_ch.channel_number} collides with hidden " + f"channel #{hidden_ch.channel_number}; sync did not reserve " + f"the hidden channel's number", + ) + + +class HDHRLineupNullChannelNumberTests(TestCase): + """ + With nullable channel_number (migration 0039) it is now valid for a + channel to have effective_channel_number=None - typically a hidden + channel under compact numbering whose number was released. Both HDHR + lineup endpoints (legacy `apps/hdhr/views.py` and new + `apps/hdhr/api_views.py`) must skip those rows rather than emit + `"GuideNumber": "None"` or `"GuideNumber": ""`. HDHR clients reject + such entries and may drop the entire lineup. + """ + + def test_legacy_lineup_skips_channels_with_null_effective_number(self): + # Set up two channels: one with a number, one without. + account = _make_account() + group = _make_group(name="Legacy") + with_number = Channel.objects.create( + name="WithNum", + channel_number=42, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + Channel.objects.create( + name="NoNum", + channel_number=None, + channel_group=group, + auto_created=True, + auto_created_by=account, + ) + + from apps.accounts.models import User + from rest_framework.test import APIClient + + admin = User.objects.create_superuser( + username="hdhr_legacy_admin", password="x", user_level=10 + ) + client = APIClient() + client.force_authenticate(user=admin) + response = client.get("/hdhr/lineup.json") + + self.assertEqual(response.status_code, 200) + body = response.json() if hasattr(response, "json") else response.data + # Body is a list of {GuideNumber, GuideName, URL, ...} + guide_numbers = {entry.get("GuideNumber") for entry in body} + self.assertIn( + "42", guide_numbers, "Numbered channel must appear in lineup" + ) + self.assertNotIn( + "None", guide_numbers, + "Lineup must not contain literal 'None' GuideNumber", + ) + self.assertNotIn( + "", guide_numbers, + "Lineup must not contain empty GuideNumber", + ) + + +class DuplicateOverrideChannelNumberAllowedTests(TestCase): + """ + Override channel_number is intentionally allowed to duplicate an + existing channel's effective number. Downstream clients render two + entries at the same number; users decide whether that is desired + (e.g., one of the duplicates is hidden from output). Sync still + avoids collisions on its own writes via set-membership on + used_numbers, so the merge behavior across providers is unaffected. + """ + + def test_override_channel_number_duplicate_is_accepted(self): + from rest_framework.test import APIClient + from apps.accounts.models import User + from apps.channels.models import Channel as Ch + from apps.channels.models import ChannelOverride + + admin = User.objects.create_superuser( + username="dup_override_admin", password="x", user_level=10 + ) + + # Existing channel claiming #50 by raw channel_number. + Ch.objects.create( + name="Existing", + channel_number=50, + auto_created=False, + ) + editable = Ch.objects.create( + name="Editable", + channel_number=51, + auto_created=True, + auto_created_by=_make_account(), + ) + + client = APIClient() + client.force_authenticate(user=admin) + response = client.patch( + f"/api/channels/channels/{editable.id}/", + {"override": {"channel_number": 50}}, + format="json", + ) + + self.assertEqual( + response.status_code, + 200, + f"Expected 200; got {response.status_code} " + f"body={getattr(response, 'data', None)}", + ) + override = ChannelOverride.objects.get(channel=editable) + self.assertEqual(override.channel_number, 50) + + +class EPGDispatchExistingChannelTests(TestCase): + """ + The new-channel sync path manually dispatches `parse_programs_for_tvg_id` + once per unique epg_data_id (because bulk_create bypasses post_save). + The existing-channel path (bulk_update) ALSO bypasses post_save, so it + needs the same manual dispatch when `epg_data` is in the dirty set; + otherwise EPG re-parse never fires for channels whose epg link was + just changed by sync. + """ + + def test_existing_channel_epg_change_triggers_parse_dispatch(self): + from unittest.mock import patch + + account = _make_account() + group = _make_group(name="EPG Test") + rel = _attach_group_to_account(account, group) + rel.auto_sync_channel_start = 100 + rel.save() + + # First sync: create channel with one tvg_id. + s1 = Stream.objects.create( + name="EPGCh", + url="http://example.com/epg.m3u8", + m3u_account=account, + channel_group=group, + tvg_id="initial-tvg", + last_seen=timezone.now(), + ) + _sync(account) + + ch = Channel.objects.get( + auto_created=True, auto_created_by=account + ) + # Force epg_data divergence on the next sync by changing the + # stream's tvg_id (the resolver will see a different EPG link). + # Without changing the actual EPG resolution logic here we just + # set epg_data_id to something different in DB and verify the + # next sync detects the change via update_fields="epg_data" in + # the dirty set; the dispatch loop must fire. + ch.epg_data_id = None + ch.save(update_fields=["epg_data"]) + + # Update the stream's tvg_id so the resolver picks up a new + # EPGData on the next sync. + from apps.epg.models import EPGData, EPGSource + + src = EPGSource.objects.create( + name="TestSrc", source_type="manual", url="" + ) + new_epg = EPGData.objects.create( + tvg_id="initial-tvg", name="EPGCh", epg_source=src + ) + # Sync should now associate ch with new_epg via tvg_id match. + + with patch( + "apps.epg.tasks.parse_programs_for_tvg_id.delay" + ) as mock_dispatch: + _sync(account) + + # The exact number of calls is 1 per unique epg_data_id; for one + # changed channel that is at least 1 call. + called_for = {c.args[0] for c in mock_dispatch.call_args_list} + self.assertIn( + new_epg.id, + called_for, + f"parse_programs_for_tvg_id was not dispatched for the EPG " + f"id {new_epg.id} that the existing channel was just " + f"associated with. mock calls: {mock_dispatch.call_args_list}", + ) + + +class Migration0037DemoteOrphansTests(TestCase): + """ + The 0037 auto-sync overhaul migration's `backfill_auto_created_by_null` + step demotes orphaned `auto_created=True, auto_created_by=NULL` channels + to `auto_created=False` instead of deleting them, preserving the + channel and any overrides that may exist on it. + """ + + def test_orphan_with_no_streams_is_demoted_not_deleted(self): + # Create an orphaned auto-created channel with no streams. This + # is the case that a delete-on-orphan strategy would silently lose. + ch = Channel.objects.create( + name="OrphanGhost", + channel_number=999, + auto_created=True, + auto_created_by=None, + ) + + # The migration's backfill function takes (apps, schema_editor) + # where `apps` is normally the historical app registry. For this + # unit test we call with the live registry. The migration file + # name starts with a digit so it must be loaded via importlib. + from importlib import import_module + from django.apps import apps as django_apps + + module = import_module( + "apps.channels.migrations.0037_auto_sync_overhaul" + ) + + # The migration function takes (apps, schema_editor); apps is + # the historical app registry. For this unit test we call with + # the live registry. + module.backfill_auto_created_by_null(django_apps, None) + + ch.refresh_from_db() + self.assertFalse( + ch.auto_created, + "Orphaned auto-created channel with no streams must be " + "demoted to manual (auto_created=False), not left as " + "auto_created=True or deleted", + ) + self.assertIsNone(ch.auto_created_by) diff --git a/apps/output/views.py b/apps/output/views.py index a56640d6..16b15236 100644 --- a/apps/output/views.py +++ b/apps/output/views.py @@ -2,6 +2,7 @@ from django.http import HttpResponse, JsonResponse, Http404, HttpResponseForbidd from rest_framework.response import Response from django.urls import reverse from apps.channels.models import Channel, ChannelProfile, ChannelGroup, Stream +from apps.channels.utils import format_channel_number from django.db.models import Prefetch from django.views.decorators.csrf import csrf_exempt from django.views.decorators.http import require_http_methods @@ -138,7 +139,7 @@ def generate_m3u(request, profile_name=None, user=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).select_related('channel_group', 'logo').order_by("channel_number") + base_qs = Channel.objects.filter(**filters).select_related('channel_group', 'logo') else: # User has specific limited profiles assigned filters = { @@ -149,11 +150,9 @@ def generate_m3u(request, profile_name=None, user=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).select_related('channel_group', 'logo').distinct().order_by("channel_number") + base_qs = Channel.objects.filter(**filters).select_related('channel_group', 'logo').distinct() else: - channels = Channel.objects.filter(user_level__lte=user.user_level).select_related('channel_group', 'logo').order_by( - "channel_number" - ) + base_qs = Channel.objects.filter(user_level__lte=user.user_level).select_related('channel_group', 'logo') else: if profile_name is not None: @@ -162,12 +161,22 @@ def generate_m3u(request, profile_name=None, user=None): except ChannelProfile.DoesNotExist: logger.warning("Requested channel profile (%s) during m3u generation does not exist", profile_name) raise Http404(f"Channel profile '{profile_name}' not found") - channels = Channel.objects.filter( + base_qs = Channel.objects.filter( channelprofilemembership__channel_profile=channel_profile, channelprofilemembership__enabled=True - ).select_related('channel_group', 'logo').order_by('channel_number') + ).select_related('channel_group', 'logo') else: - channels = Channel.objects.select_related('channel_group', 'logo').order_by("channel_number") + base_qs = Channel.objects.select_related('channel_group', 'logo') + + # Resolve effective (override | provider) values at SQL level so ordering, + # naming, and logo resolution honor user overrides. `exclude(hidden_from_output=True)` + # is the consumer-facing hide guarantee. + from apps.channels.managers import with_effective_values + channels = ( + with_effective_values(base_qs, select_related_fks=True) + .exclude(hidden_from_output=True) + .order_by("effective_channel_number") + ) # Check if the request wants to use direct logo URLs instead of cache use_cached_logos = request.GET.get('cachedlogos', 'true').lower() != 'false' @@ -212,53 +221,55 @@ def generate_m3u(request, profile_name=None, user=None): m3u_content = f'#EXTM3U x-tvg-url="{epg_url}" url-tvg="{epg_url}"\n' # Start building M3U content + channel_count = 0 for channel in channels: - group_title = channel.channel_group.name if channel.channel_group else "Default" + channel_count += 1 + effective_group = channel.effective_channel_group_obj + effective_logo = channel.effective_logo_obj + effective_name = channel.effective_name + effective_tvg_id_val = channel.effective_tvg_id + effective_tvc_guide = channel.effective_tvc_guide_stationid + effective_number = channel.effective_channel_number - # Format channel number as integer if it has no decimal component - if channel.channel_number is not None: - if channel.channel_number == int(channel.channel_number): - formatted_channel_number = int(channel.channel_number) - else: - formatted_channel_number = channel.channel_number - else: - formatted_channel_number = "" + group_title = effective_group.name if effective_group else "Default" + + formatted_channel_number = format_channel_number(effective_number) # Determine the tvg-id based on the selected source - if tvg_id_source == 'tvg_id' and channel.tvg_id: - tvg_id = channel.tvg_id - elif tvg_id_source == 'gracenote' and channel.tvc_guide_stationid: - tvg_id = channel.tvc_guide_stationid + if tvg_id_source == 'tvg_id' and effective_tvg_id_val: + tvg_id = effective_tvg_id_val + elif tvg_id_source == 'gracenote' and effective_tvc_guide: + tvg_id = effective_tvc_guide else: # Default to channel number (original behavior) tvg_id = str(formatted_channel_number) if formatted_channel_number != "" else str(channel.id) - tvg_name = channel.name + tvg_name = effective_name tvg_logo = "" - if channel.logo: + if effective_logo: if use_cached_logos: # Use cached logo as before - tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[channel.logo.id])) + tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[effective_logo.id])) else: # Try to find direct logo URL from channel's streams - direct_logo = channel.logo.url if channel.logo.url.startswith(('http://', 'https://')) else None + direct_logo = effective_logo.url if effective_logo.url.startswith(('http://', 'https://')) else None # If direct logo found, use it; otherwise fall back to cached version if direct_logo: tvg_logo = direct_logo else: - tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[channel.logo.id])) + tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[effective_logo.id])) # create possible gracenote id insertion tvc_guide_stationid = "" - if channel.tvc_guide_stationid: + if effective_tvc_guide: tvc_guide_stationid = ( - f'tvc-guide-stationid="{channel.tvc_guide_stationid}" ' + f'tvc-guide-stationid="{effective_tvc_guide}" ' ) extinf_line = ( f'#EXTINF:-1 tvg-id="{tvg_id}" tvg-name="{tvg_name}" tvg-logo="{tvg_logo}" ' - f'tvg-chno="{formatted_channel_number}" {tvc_guide_stationid}group-title="{group_title}",{channel.name}\n' + f'tvg-chno="{formatted_channel_number}" {tvc_guide_stationid}group-title="{group_title}",{effective_name}\n' ) # Determine the stream URL based on request type @@ -293,7 +304,7 @@ def generate_m3u(request, profile_name=None, user=None): event_type='m3u_download', profile=profile_name or 'all', user=user.username if user else 'anonymous', - channels=channels.count(), + channels=channel_count, client_ip=client_ip, user_agent=user_agent, ) @@ -1328,7 +1339,7 @@ def generate_epg(request, profile_name=None, user=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).select_related('logo', 'epg_data__epg_source').order_by("channel_number") + base_qs = Channel.objects.filter(**filters).select_related('logo', 'epg_data__epg_source') else: # User has specific limited profiles assigned filters = { @@ -1339,11 +1350,9 @@ def generate_epg(request, profile_name=None, user=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).select_related('logo', 'epg_data__epg_source').distinct().order_by("channel_number") + base_qs = Channel.objects.filter(**filters).select_related('logo', 'epg_data__epg_source').distinct() else: - channels = Channel.objects.filter(user_level__lte=user.user_level).select_related('logo', 'epg_data__epg_source').order_by( - "channel_number" - ) + base_qs = Channel.objects.filter(user_level__lte=user.user_level).select_related('logo', 'epg_data__epg_source') else: if profile_name is not None: try: @@ -1351,12 +1360,21 @@ def generate_epg(request, profile_name=None, user=None): except ChannelProfile.DoesNotExist: logger.warning("Requested channel profile (%s) during epg generation does not exist", profile_name) raise Http404(f"Channel profile '{profile_name}' not found") - channels = Channel.objects.filter( + base_qs = Channel.objects.filter( channelprofilemembership__channel_profile=channel_profile, channelprofilemembership__enabled=True, - ).select_related('logo', 'epg_data__epg_source').order_by("channel_number") + ).select_related('logo', 'epg_data__epg_source') else: - channels = Channel.objects.all().select_related('logo', 'epg_data__epg_source').order_by("channel_number") + base_qs = Channel.objects.all().select_related('logo', 'epg_data__epg_source') + + # Resolve effective values at SQL level and exclude hidden channels + # so output ordering/display honors user overrides. + from apps.channels.managers import with_effective_values + channels = ( + with_effective_values(base_qs, select_related_fks=True) + .exclude(hidden_from_output=True) + .order_by("effective_channel_number") + ) # For dummy EPG, use either the specified value or default to 3 days @@ -1376,15 +1394,17 @@ def generate_epg(request, profile_name=None, user=None): # First pass: assign integers for channels that already have integer numbers for channel in channels: - if channel.channel_number == int(channel.channel_number): - num = int(channel.channel_number) + effective_num = channel.effective_channel_number + if effective_num is not None and effective_num == int(effective_num): + num = int(effective_num) channel_num_map[channel.id] = num used_numbers.add(num) # Second pass: assign integers for channels with float numbers for channel in channels: - if channel.channel_number != int(channel.channel_number): - candidate = int(channel.channel_number) + effective_num = channel.effective_channel_number + if effective_num is not None and effective_num != int(effective_num): + candidate = int(effective_num) while candidate in used_numbers: candidate += 1 channel_num_map[channel.id] = candidate @@ -1392,38 +1412,37 @@ def generate_epg(request, profile_name=None, user=None): # Process channels for the section for channel in channels: + effective_name = channel.effective_name + effective_epg_data = channel.effective_epg_data_obj + effective_logo = channel.effective_logo_obj + effective_number = channel.effective_channel_number + # user is set only for XC clients, which require integer channel numbers if user is not None: formatted_channel_number = channel_num_map[channel.id] else: - if channel.channel_number is not None: - if channel.channel_number == int(channel.channel_number): - formatted_channel_number = int(channel.channel_number) - else: - formatted_channel_number = channel.channel_number - else: - formatted_channel_number = "" + formatted_channel_number = format_channel_number(effective_number) # Determine the channel ID based on the selected source - if tvg_id_source == 'tvg_id' and channel.tvg_id: - channel_id = channel.tvg_id - elif tvg_id_source == 'gracenote' and channel.tvc_guide_stationid: - channel_id = channel.tvc_guide_stationid + if tvg_id_source == 'tvg_id' and channel.effective_tvg_id: + channel_id = channel.effective_tvg_id + elif tvg_id_source == 'gracenote' and channel.effective_tvc_guide_stationid: + channel_id = channel.effective_tvc_guide_stationid else: channel_id = str(formatted_channel_number) if formatted_channel_number != "" else str(channel.id) tvg_logo = "" # Check if this is a custom dummy EPG with channel logo URL template - if channel.epg_data and channel.epg_data.epg_source and channel.epg_data.epg_source.source_type == 'dummy': - epg_source = channel.epg_data.epg_source + if effective_epg_data and effective_epg_data.epg_source and effective_epg_data.epg_source.source_type == 'dummy': + epg_source = effective_epg_data.epg_source if epg_source.custom_properties: custom_props = epg_source.custom_properties channel_logo_url_template = custom_props.get('channel_logo_url', '') if channel_logo_url_template: # Determine which name to use for pattern matching (same logic as program generation) - pattern_match_name = channel.name + pattern_match_name = effective_name name_source = custom_props.get('name_source') if name_source == 'stream': @@ -1465,20 +1484,20 @@ def generate_epg(request, profile_name=None, user=None): tvg_logo = channel_logo_url_template logger.debug(f"Built channel logo URL from template: {tvg_logo}") except Exception as e: - logger.warning(f"Failed to build channel logo URL for {channel.name}: {e}") + logger.warning(f"Failed to build channel logo URL for {effective_name}: {e}") # If no custom dummy logo, use regular logo logic - if not tvg_logo and channel.logo: + if not tvg_logo and effective_logo: if use_cached_logos: - tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[channel.logo.id])) + tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[effective_logo.id])) else: # Use direct URL if available, otherwise fall back to cached version - direct_logo = channel.logo.url if channel.logo.url.startswith(('http://', 'https://')) else None + direct_logo = effective_logo.url if effective_logo.url.startswith(('http://', 'https://')) else None if direct_logo: tvg_logo = direct_logo else: - tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[channel.logo.id])) - display_name = channel.name + tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[effective_logo.id])) + display_name = effective_name xml_lines.append(f' ') xml_lines.append(f' {html.escape(display_name)}') xml_lines.append(f' ') @@ -1495,30 +1514,29 @@ def generate_epg(request, profile_name=None, user=None): dummy_epg_checked = {} # epg_data_id -> bool (has stored programs) for channel in channels: + effective_name = channel.effective_name + effective_epg_data = channel.effective_epg_data_obj + effective_epg_data_id = channel.effective_epg_data_id + effective_number = channel.effective_channel_number + # Determine channel_id (same logic as channel section) - if tvg_id_source == 'tvg_id' and channel.tvg_id: - channel_id = channel.tvg_id - elif tvg_id_source == 'gracenote' and channel.tvc_guide_stationid: - channel_id = channel.tvc_guide_stationid + if tvg_id_source == 'tvg_id' and channel.effective_tvg_id: + channel_id = channel.effective_tvg_id + elif tvg_id_source == 'gracenote' and channel.effective_tvc_guide_stationid: + channel_id = channel.effective_tvc_guide_stationid else: if user is not None: formatted_channel_number = channel_num_map[channel.id] else: - if channel.channel_number is not None: - if channel.channel_number == int(channel.channel_number): - formatted_channel_number = int(channel.channel_number) - else: - formatted_channel_number = channel.channel_number - else: - formatted_channel_number = "" + formatted_channel_number = format_channel_number(effective_number) channel_id = str(formatted_channel_number) if formatted_channel_number != "" else str(channel.id) - display_name = channel.epg_data.name if channel.epg_data else channel.name - pattern_match_name = channel.name + display_name = effective_epg_data.name if effective_epg_data else effective_name + pattern_match_name = effective_name # Check if we should use stream name instead of channel name - if channel.epg_data and channel.epg_data.epg_source: - epg_source = channel.epg_data.epg_source + if effective_epg_data and effective_epg_data.epg_source: + epg_source = effective_epg_data.epg_source if epg_source.custom_properties: custom_props = epg_source.custom_properties name_source = custom_props.get('name_source') @@ -1532,22 +1550,21 @@ def generate_epg(request, profile_name=None, user=None): pattern_match_name = stream.name logger.debug(f"Using stream name for parsing: {pattern_match_name} (stream index: {stream_index})") else: - logger.warning(f"Stream index {stream_index} not found for channel {channel.name}, falling back to channel name") + logger.warning(f"Stream index {stream_index} not found for channel {effective_name}, falling back to channel name") - if not channel.epg_data: + if not effective_epg_data: dummy_program_list.append((channel_id, pattern_match_name, None)) else: - if channel.epg_data.epg_source and channel.epg_data.epg_source.source_type == 'dummy': - epg_data_id = channel.epg_data_id - if epg_data_id not in dummy_epg_checked: - dummy_epg_checked[epg_data_id] = channel.epg_data.programs.exists() - if dummy_epg_checked[epg_data_id]: - real_epg_map.setdefault(epg_data_id, []).append(channel_id) + if effective_epg_data.epg_source and effective_epg_data.epg_source.source_type == 'dummy': + if effective_epg_data_id not in dummy_epg_checked: + dummy_epg_checked[effective_epg_data_id] = effective_epg_data.programs.exists() + if dummy_epg_checked[effective_epg_data_id]: + real_epg_map.setdefault(effective_epg_data_id, []).append(channel_id) else: - dummy_program_list.append((channel_id, pattern_match_name, channel.epg_data.epg_source)) + dummy_program_list.append((channel_id, pattern_match_name, effective_epg_data.epg_source)) continue - real_epg_map.setdefault(channel.epg_data_id, []).append(channel_id) + real_epg_map.setdefault(effective_epg_data_id, []).append(channel_id) # Emit dummy programmes for channel_id, pattern_match_name, epg_source in dummy_program_list: @@ -1880,11 +1897,13 @@ def generate_epg(request, profile_name=None, user=None): client_id, client_ip, user_agent = get_client_identifier(request) event_cache_key = f"epg_download:{user.username if user else 'anonymous'}:{profile_name or 'all'}:{client_id}" if not cache.get(event_cache_key): + # `len()` reuses the queryset's iteration cache populated above; + # `count()` would issue a separate SELECT COUNT(*). log_system_event( event_type='epg_download', profile=profile_name or 'all', user=user.username if user else 'anonymous', - channels=channels.count(), + channels=len(channels), client_ip=client_ip, user_agent=user_agent, ) @@ -2103,8 +2122,18 @@ def xc_xmltv(request): def xc_get_live_categories(user): from django.db.models import Min + from django.db.models.functions import Coalesce + response = [] + # Rank categories by the minimum EFFECTIVE channel number across their + # visible (not hidden_from_output) channels so overridden numbers drive the + # ordering, not the underlying provider values. + effective_min = Min( + Coalesce("channels__override__channel_number", "channels__channel_number") + ) + hidden_exclusion = {"channels__hidden_from_output": False} + if user.user_level < 10: user_profile_count = user.channel_profiles.count() @@ -2112,20 +2141,25 @@ def xc_get_live_categories(user): if user_profile_count == 0: # No profile filtering - user sees all channel groups channel_groups = ChannelGroup.objects.filter( - channels__isnull=False, channels__user_level__lte=user.user_level - ).distinct().annotate(min_channel_number=Min('channels__channel_number')).order_by('min_channel_number') + channels__isnull=False, + channels__user_level__lte=user.user_level, + **hidden_exclusion, + ).distinct().annotate(min_channel_number=effective_min).order_by('min_channel_number') else: # User has specific limited profiles assigned filters = { "channels__channelprofilemembership__enabled": True, "channels__user_level": 0, - "channels__channelprofilemembership__channel_profile__in": user.channel_profiles.all() + "channels__channelprofilemembership__channel_profile__in": user.channel_profiles.all(), + **hidden_exclusion, } - channel_groups = ChannelGroup.objects.filter(**filters).distinct().annotate(min_channel_number=Min('channels__channel_number')).order_by('min_channel_number') + channel_groups = ChannelGroup.objects.filter(**filters).distinct().annotate(min_channel_number=effective_min).order_by('min_channel_number') else: channel_groups = ChannelGroup.objects.filter( - channels__isnull=False, channels__user_level__lte=user.user_level - ).distinct().annotate(min_channel_number=Min('channels__channel_number')).order_by('min_channel_number') + channels__isnull=False, + channels__user_level__lte=user.user_level, + **hidden_exclusion, + ).distinct().annotate(min_channel_number=effective_min).order_by('min_channel_number') for group in channel_groups: response.append( @@ -2140,6 +2174,8 @@ def xc_get_live_categories(user): def xc_get_live_streams(request, user, category_id=None): + from apps.channels.managers import with_effective_values + streams = [] if user.user_level < 10: @@ -2154,7 +2190,7 @@ def xc_get_live_streams(request, user, category_id=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).select_related('channel_group', 'logo').order_by("channel_number") + base_qs = Channel.objects.filter(**filters).select_related('channel_group', 'logo') else: # User has specific limited profiles assigned filters = { @@ -2167,14 +2203,20 @@ def xc_get_live_streams(request, user, category_id=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).select_related('channel_group', 'logo').distinct().order_by("channel_number") + base_qs = Channel.objects.filter(**filters).select_related('channel_group', 'logo').distinct() else: if not category_id: - channels = Channel.objects.filter(user_level__lte=user.user_level).select_related('channel_group', 'logo').order_by("channel_number") + base_qs = Channel.objects.filter(user_level__lte=user.user_level).select_related('channel_group', 'logo') else: - channels = Channel.objects.filter( + base_qs = Channel.objects.filter( channel_group__id=category_id, user_level__lte=user.user_level - ).select_related('channel_group', 'logo').order_by("channel_number") + ).select_related('channel_group', 'logo') + + channels = ( + with_effective_values(base_qs, select_related_fks=True) + .exclude(hidden_from_output=True) + .order_by("effective_channel_number") + ) # Resolve the fallback group ID once to avoid a get_or_create query per null-group channel _default_group_id = None @@ -2190,21 +2232,21 @@ def xc_get_live_streams(request, user, category_id=None): channel_num_map = {} # Maps channel.id -> integer channel number for XC used_numbers = set() # Track all assigned integer channel numbers - # First pass: assign integers for channels that already have integer numbers + # First pass: assign integers for channels that already have integer effective numbers for channel in channels: - if channel.channel_number == int(channel.channel_number): - # Already an integer, use it directly - num = int(channel.channel_number) + effective_num = channel.effective_channel_number + if effective_num is not None and effective_num == int(effective_num): + num = int(effective_num) channel_num_map[channel.id] = num used_numbers.add(num) # Second pass: assign integers for channels with float numbers # Find next available number to avoid collisions for channel in channels: - if channel.channel_number != int(channel.channel_number): + effective_num = channel.effective_channel_number + if effective_num is not None and effective_num != int(effective_num): # Has decimal component, need to find available integer - # Start from truncated value and increment until we find an unused number - candidate = int(channel.channel_number) + candidate = int(effective_num) while candidate in used_numbers: candidate += 1 channel_num_map[channel.id] = candidate @@ -2213,26 +2255,28 @@ def xc_get_live_streams(request, user, category_id=None): # Build the streams list with the collision-free channel numbers for channel in channels: channel_num_int = channel_num_map[channel.id] + effective_logo = channel.effective_logo_obj + effective_group = channel.effective_channel_group_obj streams.append( { "num": channel_num_int, - "name": channel.name, + "name": channel.effective_name, "stream_type": "live", "stream_id": channel.id, "stream_icon": ( None - if not channel.logo + if not effective_logo else build_absolute_uri_with_port( request, - reverse("api:channels:logo-cache", args=[channel.logo.id]) + reverse("api:channels:logo-cache", args=[effective_logo.id]) ) ), "epg_channel_id": str(channel_num_int), "added": str(int(channel.created_at.timestamp())), "is_adult": int(channel.is_adult), - "category_id": str(channel.channel_group.id if channel.channel_group else _get_default_group_id()), - "category_ids": [channel.channel_group.id if channel.channel_group else _get_default_group_id()], + "category_id": str(effective_group.id if effective_group else _get_default_group_id()), + "category_ids": [effective_group.id if effective_group else _get_default_group_id()], "custom_sid": None, "tv_archive": 0, "direct_source": "", @@ -2244,11 +2288,19 @@ def xc_get_live_streams(request, user, category_id=None): def xc_get_epg(request, user, short=False): + from apps.channels.managers import with_effective_values + channel_id = request.GET.get('stream_id') if not channel_id: raise Http404() channel = None + # Apply effective-value annotation + hidden-exclusion at every channel + # resolution path so a single channel lookup honors the same visibility + # rules as xc_get_live_streams. + def _annotate(qs): + return with_effective_values(qs, select_related_fks=True).exclude(hidden_from_output=True) + if user.user_level < 10: user_profile_count = user.channel_profiles.count() @@ -2262,7 +2314,7 @@ def xc_get_epg(request, user, short=False): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channel = Channel.objects.filter(**filters).select_related('epg_data__epg_source').first() + channel = _annotate(Channel.objects.filter(**filters).select_related('epg_data__epg_source')).first() else: # User has specific limited profiles assigned filters = { @@ -2274,44 +2326,58 @@ def xc_get_epg(request, user, short=False): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channel = Channel.objects.filter(**filters).select_related('epg_data__epg_source').distinct().first() + channel = _annotate(Channel.objects.filter(**filters).select_related('epg_data__epg_source').distinct()).first() if not channel: raise Http404() else: - channel = get_object_or_404(Channel.objects.select_related('epg_data__epg_source'), id=channel_id) + channel = _annotate(Channel.objects.filter(id=channel_id).select_related('epg_data__epg_source')).first() + if not channel: + raise Http404() if not channel: raise Http404() # Calculate the collision-free integer channel number for this channel - # This must match the logic in xc_get_live_streams to ensure consistency - # Get all channels in the same category for collision detection - category_channels = Channel.objects.filter( - channel_group=channel.channel_group - ).order_by("channel_number") + # This must match the logic in xc_get_live_streams to ensure consistency. + # The category channels must be filtered by the channel's EFFECTIVE group + # (an override can move a channel into a different group), then annotated + # so the comparison runs on effective numbers. + effective_group = channel.effective_channel_group_obj + category_channels = ( + with_effective_values( + Channel.objects.filter(channel_group=effective_group) if effective_group else Channel.objects.none() + ) + .exclude(hidden_from_output=True) + .order_by("effective_channel_number") + ) channel_num_map = {} used_numbers = set() - # First pass: assign integers for channels that already have integer numbers + # First pass: assign integers for channels that already have integer effective numbers for ch in category_channels: - if ch.channel_number == int(ch.channel_number): - num = int(ch.channel_number) + effective_num = ch.effective_channel_number + if effective_num is not None and effective_num == int(effective_num): + num = int(effective_num) channel_num_map[ch.id] = num used_numbers.add(num) - # Second pass: assign integers for channels with float numbers + # Second pass: assign integers for channels with float effective numbers for ch in category_channels: - if ch.channel_number != int(ch.channel_number): - candidate = int(ch.channel_number) + effective_num = ch.effective_channel_number + if effective_num is not None and effective_num != int(effective_num): + candidate = int(effective_num) while candidate in used_numbers: candidate += 1 channel_num_map[ch.id] = candidate used_numbers.add(candidate) # Get the mapped integer for this specific channel - channel_num_int = channel_num_map.get(channel.id, int(channel.channel_number)) + channel_num_int = channel_num_map.get( + channel.id, + int(channel.effective_channel_number) if channel.effective_channel_number is not None else 0, + ) limit = int(request.GET.get('limit', 4)) user_custom = user.custom_properties or {} @@ -2328,25 +2394,28 @@ def xc_get_epg(request, user, short=False): now = django_timezone.now() lookback_cutoff = now - timedelta(days=prev_days) forward_cutoff = now + timedelta(days=num_days) if num_days > 0 else None - if channel.epg_data: + effective_epg_data = channel.effective_epg_data_obj + effective_name = channel.effective_name + + if effective_epg_data: # Check if this is a dummy EPG that generates on-demand - if channel.epg_data.epg_source and channel.epg_data.epg_source.source_type == 'dummy': - if not channel.epg_data.programs.exists(): + if effective_epg_data.epg_source and effective_epg_data.epg_source.source_type == 'dummy': + if not effective_epg_data.programs.exists(): # Generate on-demand using custom patterns programs = generate_dummy_programs( channel_id=channel_id, - channel_name=channel.name, - epg_source=channel.epg_data.epg_source + channel_name=effective_name, + epg_source=effective_epg_data.epg_source ) else: # Has stored programs, use them if short: # Short EPG: current and upcoming only (never historical), limited count - programs = channel.epg_data.programs.filter( + programs = effective_epg_data.programs.filter( end_time__gt=now ).order_by('start_time')[:limit] else: - qs = channel.epg_data.programs.filter(end_time__gt=lookback_cutoff) + qs = effective_epg_data.programs.filter(end_time__gt=lookback_cutoff) if forward_cutoff: qs = qs.filter(start_time__lt=forward_cutoff) programs = qs.order_by('start_time') @@ -2354,17 +2423,17 @@ def xc_get_epg(request, user, short=False): # Regular EPG with stored programs if short: # Short EPG: current and upcoming only (never historical), limited count - programs = channel.epg_data.programs.filter( + programs = effective_epg_data.programs.filter( end_time__gt=now ).order_by('start_time')[:limit] else: - qs = channel.epg_data.programs.filter(end_time__gt=lookback_cutoff) + qs = effective_epg_data.programs.filter(end_time__gt=lookback_cutoff) if forward_cutoff: qs = qs.filter(start_time__lt=forward_cutoff) programs = qs.order_by('start_time') else: # No EPG data assigned, generate default dummy - programs = generate_dummy_programs(channel_id=channel_id, channel_name=channel.name, epg_source=None) + programs = generate_dummy_programs(channel_id=channel_id, channel_name=effective_name, epg_source=None) output = {"epg_listings": []} @@ -2385,7 +2454,7 @@ def xc_get_epg(request, user, short=False): # epg_id refers to the EPG source/channel mapping in XC panels # Use the actual EPGData ID when available, otherwise fall back to 0 - epg_id = str(channel.epg_data.id) if channel.epg_data else "0" + epg_id = str(effective_epg_data.id) if effective_epg_data else "0" program_output = { "id": program_id, diff --git a/frontend/src/api.js b/frontend/src/api.js index ff080c88..6690b07c 100644 --- a/frontend/src/api.js +++ b/frontend/src/api.js @@ -271,6 +271,77 @@ export default class API { } } + // Repack visible auto-created channels into [start, end]; override + // pins are reservations and hidden non-pinned channels release their + // number. + static async repackGroupChannels(accountId, channelGroupId) { + try { + const params = new URLSearchParams({ + channel_group_id: String(channelGroupId), + }); + const url = `${host}/api/m3u/accounts/${accountId}/repack-group/?${params.toString()}`; + return await request(url, { method: 'POST' }); + } catch (e) { + errorNotification('Failed to re-pack group channels', e); + return null; + } + } + + // Returns occupants whose effective channel_number falls in [start, end]. + // Pass `signal` from an AbortController on per-keystroke calls so an + // out-of-order response cannot overwrite newer state. + static async getChannelsInRange(start, end, { signal } = {}) { + try { + const params = new URLSearchParams({ start: String(start) }); + if (end !== undefined && end !== null && end !== '') { + params.set('end', String(end)); + } + const url = `${host}/api/channels/channels/numbers-in-range/?${params.toString()}`; + return await request(url, { signal }); + } catch (e) { + if (e?.name === 'AbortError') { + throw e; + } + // Silent failure is correct here: the warning is purely advisory and + // should not block the user from saving when the server is flaky. + return { occupants: [] }; + } + } + + // Server-side regex preview for a group's streams. Returns find_matches, + // filter_matches, and exclude_matches plus accurate counts across the + // whole group (capped at 5000 scanned streams server-side). Used by the + // auto-sync gear modal so the user sees real matches and totals rather + // than a small client-side sample. The three patterns are independent; + // any combination can be supplied per call. + static async getStreamsRegexPreview( + channelGroupName, + { find, replace, match, exclude, limit = 10, signal, m3uAccountId } = {} + ) { + try { + const params = new URLSearchParams({ + channel_group: channelGroupName, + }); + if (m3uAccountId !== undefined && m3uAccountId !== null) { + params.set('m3u_account_id', String(m3uAccountId)); + } + if (find) params.set('find', find); + if (replace !== undefined && replace !== null) { + params.set('replace', replace); + } + if (match) params.set('match', match); + if (exclude) params.set('exclude', exclude); + params.set('limit', String(limit)); + const url = `${host}/api/channels/streams/regex-preview/?${params.toString()}`; + return await request(url, { signal }); + } catch (e) { + if (e?.name === 'AbortError') { + throw e; + } + return null; + } + } + static async queryChannels(params) { try { API.lastQueryParams = params; @@ -1341,14 +1412,34 @@ export default class API { } static async deletePlaylist(id) { + // Cascade-deletes auto-created channels owned by the account; the + // response includes the deleted count for the confirmation toast. try { - await request(`${host}/api/m3u/accounts/${id}/`, { + const response = await request(`${host}/api/m3u/accounts/${id}/`, { method: 'DELETE', }); - usePlaylistsStore.getState().removePlaylists([id]); + return response || {}; } catch (e) { errorNotification(`Failed to delete playlist ${id}`, e); + throw e; + } + } + + // Used by the Delete Playlist confirmation dialog to render an accurate + // "Also delete N auto-created channels?" option before the user commits. + static async getPlaylistAutoCreatedChannelsCount(id) { + try { + const response = await request( + `${host}/api/m3u/accounts/${id}/auto-created-channels-count/` + ); + return response; + } catch (e) { + console.error( + `Failed to fetch auto-created channel count for playlist ${id}`, + e + ); + return { count: 0, sample_names: [] }; } } diff --git a/frontend/src/components/M3URefreshNotification.jsx b/frontend/src/components/M3URefreshNotification.jsx index bd6319bf..8e24bdb3 100644 --- a/frontend/src/components/M3URefreshNotification.jsx +++ b/frontend/src/components/M3URefreshNotification.jsx @@ -5,7 +5,15 @@ import useStreamsStore from '../store/streams'; import useChannelsStore from '../store/channels'; import useEPGsStore from '../store/epgs'; import useVODStore from '../store/useVODStore'; -import { Stack, Button, Group } from '@mantine/core'; +import { + Stack, + Button, + Group, + Modal, + ScrollArea, + Text, + Code, +} from '@mantine/core'; import API from '../api'; import { useNavigate } from 'react-router-dom'; import { CircleCheck } from 'lucide-react'; @@ -43,6 +51,47 @@ const M3uSetupSuccess = ({ data }) => { ); }; +// One-line outcome summary for the notification body. +const buildAutoSyncSummary = (data) => { + const created = data.channels_created || 0; + const updated = data.channels_updated || 0; + const deleted = data.channels_deleted || 0; + const failed = data.channels_failed || 0; + if (!created && !updated && !deleted && !failed) return null; + const parts = []; + if (created) parts.push(`${created} created`); + if (updated) parts.push(`${updated} updated`); + if (deleted) parts.push(`${deleted} deleted`); + if (failed) parts.push(`${failed} failed`); + return `Auto-sync: ${parts.join(', ')}.`; +}; + +// Human labels for the typed failure reasons attached to each +// failed_stream_details entry. Unknown reasons fall back to the raw +// key so a future reason added on the backend still surfaces in the +// modal without a code change here. +const FAILURE_REASON_LABELS = { + RANGE_EXHAUSTED: 'Channel number range exhausted', + INTEGRITY_ERROR: 'Database integrity error', + OTHER: 'Other errors', +}; + +const FAILURES_PER_GROUP_LIMIT = 50; + +// Group entries by their reason key, preserving insertion order so +// section ordering reflects the sync's encounter order. Entries +// without a reason field bucket into OTHER so a backend that does +// not classify failures still renders correctly. +const groupFailuresByReason = (entries) => { + const buckets = new Map(); + for (const entry of entries) { + const key = entry?.reason || 'OTHER'; + if (!buckets.has(key)) buckets.set(key, []); + buckets.get(key).push(entry); + } + return buckets; +}; + export default function M3URefreshNotification() { const playlists = usePlaylistsStore((s) => s.playlists); const refreshProgress = usePlaylistsStore((s) => s.refreshProgress); @@ -54,6 +103,9 @@ export default function M3URefreshNotification() { const fetchCategories = useVODStore((s) => s.fetchCategories); const [notificationStatus, setNotificationStatus] = useState({}); + // Modal payload for the "Click for details" affordance on syncs that + // produced failed_stream_details. Null when the modal is closed. + const [failureModal, setFailureModal] = useState(null); const handleM3UUpdate = (data) => { // Skip if status hasn't changed @@ -148,7 +200,7 @@ export default function M3URefreshNotification() { const handleProgressNotification = (playlist, data) => { const baseMessage = getActionMessage(data.action); - const message = + let message = data.progress == 0 ? `${baseMessage} starting...` : `${baseMessage} complete!`; @@ -157,11 +209,49 @@ export default function M3URefreshNotification() { triggerPostCompletionFetches(data.action); } + let body = message; + let autoClose = 2000; + // Surface auto-sync counts attached to the parsing-complete event + // so the channel-side outcome appears in the notification body. + if (data.progress == 100 && data.action === 'parsing') { + const autoSyncSummary = buildAutoSyncSummary(data); + const failed = data.channels_failed || 0; + const failedDetails = Array.isArray(data.failed_stream_details) + ? data.failed_stream_details + : []; + if (autoSyncSummary) { + body = ( + + {message} + + {autoSyncSummary} + + {failed > 0 && failedDetails.length > 0 && ( + + )} + + ); + autoClose = failed > 0 ? 12000 : 4000; + } + } + showNotification({ title: `M3U Processing: ${playlist.name}`, - message, + message: body, loading: data.progress == 0, - autoClose: 2000, + autoClose, icon: data.progress == 100 ? : null, }); }; @@ -182,5 +272,61 @@ export default function M3URefreshNotification() { Object.values(refreshProgress).map((data) => handleM3UUpdate(data)); }, [playlists, refreshProgress]); - return <>; + return ( + setFailureModal(null)} + title={ + failureModal + ? `Auto-sync failures: ${failureModal.playlistName}` + : 'Auto-sync failures' + } + size="lg" + > + + + The following streams could not be synced. Failures are grouped by + cause so the most common issues surface first. + + + + {Array.from( + groupFailuresByReason(failureModal?.failedDetails || []).entries() + ).map(([reasonKey, entries]) => { + const label = FAILURE_REASON_LABELS[reasonKey] || reasonKey; + const visible = entries.slice(0, FAILURES_PER_GROUP_LIMIT); + const hidden = entries.length - visible.length; + return ( + + + {label} ({entries.length}) + + {visible.map((entry, idx) => ( + + {[ + entry.stream_name && `Stream: ${entry.stream_name}`, + entry.group && `Group: ${entry.group}`, + entry.error && `Error: ${entry.error}`, + ] + .filter(Boolean) + .join('\n')} + + ))} + {hidden > 0 && ( + + Showing first {visible.length} of {entries.length}. + Remaining {hidden} entries are recorded in the server log. + + )} + + ); + })} + + + + + ); } diff --git a/frontend/src/components/__tests__/M3URefreshNotification.test.jsx b/frontend/src/components/__tests__/M3URefreshNotification.test.jsx index 43375fa7..734d83e4 100644 --- a/frontend/src/components/__tests__/M3URefreshNotification.test.jsx +++ b/frontend/src/components/__tests__/M3URefreshNotification.test.jsx @@ -52,6 +52,19 @@ vi.mock('@mantine/core', async () => { Button: ({ children, onClick }) => ( ), + // Stub for the auto-sync failure-details modal. + Modal: ({ children, opened, onClose, title }) => + opened ? ( +
+ + {children} +
+ ) : null, + ScrollArea: ({ children }) =>
{children}
, + Text: ({ children }) => {children}, + Code: ({ children }) => 
{children}
, }; }); @@ -714,4 +727,128 @@ describe('M3URefreshNotification', () => { ); }); }); + + // The parsing-complete event payload carries auto-sync counts + // (channels_created/updated/deleted/failed) and a failed_stream_details + // array. The component surfaces the counts inline in the notification + // body and exposes a "Click for details" affordance when failures exist. + describe('Auto-sync count rendering on parsing complete', () => { + it('inlines auto-sync summary when counts are present', async () => { + mockPlaylistsStore.refreshProgress = { + 1: { + account: 1, + action: 'parsing', + progress: 100, + status: 'success', + channels_created: 12, + channels_updated: 3, + channels_deleted: 1, + channels_failed: 0, + failed_stream_details: [], + }, + }; + + renderWithProviders(); + + await waitFor(() => { + expect(showNotification).toHaveBeenCalled(); + }); + // The message arg is JSX; render it and confirm both the + // base message and the auto-sync summary text appear. + const call = showNotification.mock.calls.find( + (c) => typeof c[0]?.message === 'object' + ); + expect(call).toBeDefined(); + const { container } = render(<>{call[0].message}); + expect(container.textContent).toContain('Stream parsing complete!'); + expect(container.textContent).toContain('12 created'); + expect(container.textContent).toContain('3 updated'); + expect(container.textContent).toContain('1 deleted'); + }); + + it('shows "Click for details" button when failures exist', async () => { + mockPlaylistsStore.refreshProgress = { + 1: { + account: 1, + action: 'parsing', + progress: 100, + status: 'success', + channels_created: 5, + channels_updated: 0, + channels_deleted: 0, + channels_failed: 2, + failed_stream_details: [ + { + stream_name: 'BadStream1', + group: 'Sports', + error: 'Range exhausted', + }, + { + stream_name: 'BadStream2', + group: 'News', + error: 'Channel number conflict', + }, + ], + }, + }; + + renderWithProviders(); + + await waitFor(() => { + expect(showNotification).toHaveBeenCalled(); + }); + const call = showNotification.mock.calls.find( + (c) => typeof c[0]?.message === 'object' + ); + expect(call).toBeDefined(); + const { container } = render(<>{call[0].message}); + expect(container.textContent).toContain('2 failed'); + expect(container.textContent).toContain('Click for details'); + }); + + it('falls back to plain string body when no auto-sync counts arrive', async () => { + // Older payload shape (or non-parsing actions) get the original + // simple "X complete!" string body, no JSX wrapping. + mockPlaylistsStore.refreshProgress = { + 1: { + account: 1, + action: 'downloading', + progress: 100, + status: 'success', + }, + }; + + renderWithProviders(); + + await waitFor(() => { + expect(showNotification).toHaveBeenCalled(); + }); + const call = showNotification.mock.calls[0]; + expect(typeof call[0].message).toBe('string'); + }); + + it('uses extended autoClose when failures are present', async () => { + mockPlaylistsStore.refreshProgress = { + 1: { + account: 1, + action: 'parsing', + progress: 100, + status: 'success', + channels_created: 1, + channels_failed: 1, + failed_stream_details: [{ stream_name: 'X', group: 'Y', error: 'Z' }], + }, + }; + + renderWithProviders(); + + await waitFor(() => { + expect(showNotification).toHaveBeenCalled(); + }); + const call = showNotification.mock.calls[0]; + // 12000ms when failures > 0; 4000ms when summary present but no + // failures; 2000ms when no auto-sync counts. + expect(call[0].autoClose).toBe(12000); + }); + }); }); diff --git a/frontend/src/components/forms/Channel.jsx b/frontend/src/components/forms/Channel.jsx index cc503209..90b95a42 100644 --- a/frontend/src/components/forms/Channel.jsx +++ b/frontend/src/components/forms/Channel.jsx @@ -33,7 +33,7 @@ import { UnstyledButton, useMantineTheme, } from '@mantine/core'; -import { ListOrdered, SquarePlus, X, Zap } from 'lucide-react'; +import { ListOrdered, SquarePlus, Undo2, X, Zap } from 'lucide-react'; import useEPGsStore from '../../store/epgs'; import { FixedSizeList as List } from 'react-window'; import { USER_LEVEL_LABELS, USER_LEVELS } from '../../constants'; @@ -43,11 +43,18 @@ import { } from '../../utils/notificationUtils.js'; import { addChannel, + clearChannelOverrides, createLogo, getChannelFormDefaultValues, + getFkProviderHint, getFormattedValues, + getProviderFormValue, + getProviderHint, handleEpgUpdate, + isFormFieldOverridden, matchChannelEpg, + OVERRIDABLE_FIELDS, + OVERRIDE_FIELD_LABELS, requeryChannels, } from '../../utils/forms/ChannelUtils.js'; @@ -56,13 +63,48 @@ const validationSchema = Yup.object({ channel_group_id: Yup.string().required('Channel group is required'), }); -const ChannelForm = ({ channel = null, isOpen, onClose }) => { +// Provider hint plus a reset-to-provider icon for auto-synced +// channels; rendered as the field's `description` prop. +const ProviderHintRow = ({ channel, field, formValue, hintText, onReset }) => { + if (!hintText) return null; + const overridden = isFormFieldOverridden(channel, field, formValue); + return ( + + + {hintText} + + {overridden && ( + + + + + + )} + + ); +}; + +const ChannelForm = ({ channel: channelProp = null, isOpen, onClose }) => { const theme = useMantineTheme(); const listRef = useRef(null); const logoListRef = useRef(null); const groupListRef = useRef(null); + // Local copy so in-modal mutations (clear overrides, etc.) update the + // form immediately. Reset on each open from `channelProp`; mutated in + // place with API responses. + const [channel, setChannel] = useState(channelProp); + useEffect(() => { + setChannel(channelProp); + }, [channelProp]); + const channelGroups = useChannelsStore((s) => s.channelGroups); const { @@ -309,7 +351,45 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { resolver: yupResolver(validationSchema), }); + const clearOverrides = async () => { + if (!channel) return; + try { + const updated = await clearChannelOverrides(channel.id); + // Update local state first so the form reflects the cleared + // overrides immediately; the table-store refresh is best-effort. + if (updated && typeof updated === 'object') { + setChannel(updated); + } + requeryChannels(); + showNotification({ + title: 'Overrides Cleared', + message: 'Channel values now follow the provider.', + color: 'green', + }); + } catch (error) { + showNotification({ + title: 'Clear Failed', + message: + error?.body?.detail || error?.message || 'Could not clear overrides.', + color: 'red', + }); + } + }; + + // Computed from live form values so per-field resets update the + // Clear-all button immediately, before submit. + const watchedFormValues = watch(); + const overriddenFieldLabels = useMemo(() => { + if (!channel) return []; + return OVERRIDABLE_FIELDS.filter((field) => + isFormFieldOverridden(channel, field, watchedFormValues[field]) + ).map((field) => OVERRIDE_FIELD_LABELS[field]); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [channel, JSON.stringify(watchedFormValues)]); + const hasAnyOverride = overriddenFieldLabels.length > 0; + const onSubmit = async (values) => { + let saveFailed = false; try { const formattedValues = getFormattedValues(values); @@ -324,8 +404,29 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { } } catch (error) { console.error('Error saving channel:', error); + saveFailed = true; + showNotification({ + title: 'Save Failed', + message: + error?.body?.detail || error?.message || 'Failed to save channel.', + color: 'red', + }); } + if (saveFailed) { + // Keep the form open with the user's edits intact so they can correct + // a validation error without retyping. + return; + } + + showNotification({ + title: 'Saved', + message: channel + ? `Channel "${values.name}" updated.` + : `Channel "${values.name}" created.`, + color: 'green', + }); + reset(); requeryChannels(); @@ -408,6 +509,17 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { styles={{ content: { '--mantine-color-body': '#27272A' } }} >
+ {channel?.auto_created && channel?.source_stream && ( + + Auto-created from:{' '} + + {channel.source_stream.account_name || 'Unknown provider'} + + {channel.source_stream.name + ? ` / ${channel.source_stream.name}` + : ''} + + )} { )} } + description={ + + setValue('name', getProviderFormValue(channel, 'name'), { + shouldDirty: true, + }) + } + /> + } {...register('name')} error={errors.name?.message} size="xs" @@ -448,6 +573,24 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { id="channel_group_id" name="channel_group_id" label="Channel Group" + description={ + + setValue( + 'channel_group_id', + getProviderFormValue(channel, 'channel_group_id') + ) + } + /> + } readOnly value={ channelGroups[watch('channel_group_id')] @@ -535,6 +678,27 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { id="stream_profile_id" label="Stream Profile" name="stream_profile_id" + description={ + { + acc[p.id] = p; + return acc; + }, {}) + )} + onReset={() => + setValue( + 'stream_profile_id', + getProviderFormValue(channel, 'stream_profile_id') + ) + } + /> + } value={watch('stream_profile_id')} onChange={(value) => { setValue('stream_profile_id', value); @@ -605,6 +769,24 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { )} } + description={ + + setValue( + 'logo_id', + getProviderFormValue(channel, 'logo_id') + ) + } + /> + } readOnly value={channelLogos[watch('logo_id')]?.name || 'Default'} onClick={() => { @@ -738,6 +920,38 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { /> + + + + setValue('hidden_from_output', event.currentTarget.checked) + } + size="md" + /> + + + {channel?.auto_created && hasAnyOverride && ( + + + + )} @@ -747,6 +961,21 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { id="channel_number" name="channel_number" label="Channel # (blank to auto-assign)" + description={ + + setValue( + 'channel_number', + getProviderFormValue(channel, 'channel_number'), + { shouldDirty: true } + ) + } + /> + } value={watch('channel_number')} onChange={(value) => setValue('channel_number', value)} error={errors.channel_number?.message} @@ -775,6 +1004,21 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { )} } + description={ + + setValue( + 'tvg_id', + getProviderFormValue(channel, 'tvg_id'), + { shouldDirty: true } + ) + } + /> + } {...register('tvg_id')} error={errors.tvg_id?.message} size="xs" @@ -784,6 +1028,21 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { id="tvc_guide_stationid" name="tvc_guide_stationid" label="Gracenote StationId" + description={ + + setValue( + 'tvc_guide_stationid', + getProviderFormValue(channel, 'tvc_guide_stationid'), + { shouldDirty: true } + ) + } + /> + } {...register('tvc_guide_stationid')} error={errors.tvc_guide_stationid?.message} size="xs" @@ -829,6 +1088,24 @@ const ChannelForm = ({ channel = null, isOpen, onClose }) => { } + description={ + + setValue( + 'epg_data_id', + getProviderFormValue(channel, 'epg_data_id') + ) + } + /> + } readOnly value={(() => { const tvg = tvgsById[watch('epg_data_id')]; diff --git a/frontend/src/components/forms/ChannelBatch.jsx b/frontend/src/components/forms/ChannelBatch.jsx index a227b20c..e0355ea0 100644 --- a/frontend/src/components/forms/ChannelBatch.jsx +++ b/frontend/src/components/forms/ChannelBatch.jsx @@ -54,6 +54,7 @@ import { setChannelNamesFromEpg, setChannelTvgIdsFromEpg, updateChannels, + updateChannelsWithOverrideRouting, } from '../../utils/forms/ChannelBatchUtils.js'; const ChannelBatchForm = ({ channelIds, isOpen, onClose }) => { @@ -123,14 +124,40 @@ const ChannelBatchForm = ({ channelIds, isOpen, onClose }) => { stream_profile_id: '-1', user_level: '-1', is_adult: '-1', + hidden_from_output: '-1', + clear_overrides: '-1', }, }); + // Surfaces auto-vs-manual routing in the selection. Falls back to a + // single total when the table store only has a partial view (e.g. + // cross-page selects). Kept separate from getConfirmationMessage so + // the line does not count against the no-changes guard. + const getSelectionSummary = () => { + const channelsById = useChannelsTableStore + .getState() + .channels.reduce((acc, c) => { + acc[c.id] = c; + return acc; + }, {}); + let autoCount = 0; + let manualCount = 0; + for (const id of channelIds) { + const c = channelsById[id]; + if (!c) continue; + if (c.auto_created) autoCount++; + else manualCount++; + } + const resolved = autoCount + manualCount; + return resolved === channelIds.length + ? `Selection: ${autoCount} auto-synced, ${manualCount} manual` + : `Selection: ${channelIds.length} channels`; + }; + // Build confirmation message based on selected changes const getConfirmationMessage = () => { const values = form.getValues(); - - return [ + const lines = [ getRegexNameChange(regexFind, regexReplace), getChannelGroupChange(selectedChannelGroup, channelGroups), getLogoChange(selectedLogoId, channelLogos), @@ -138,7 +165,18 @@ const ChannelBatchForm = ({ channelIds, isOpen, onClose }) => { getUserLevelChange(values.user_level, USER_LEVEL_LABELS), getMatureContentChange(values.is_adult), getEpgChange(selectedDummyEpgId, epgs), - ].filter(Boolean); + ]; + if (values.hidden_from_output && values.hidden_from_output !== '-1') { + lines.push( + `• Hidden: ${values.hidden_from_output === 'true' ? 'Yes' : 'No'}` + ); + } + if (values.clear_overrides === 'clear') { + lines.push( + '• Clear all overrides on auto-synced channels in selection, then apply the edits above as new overrides' + ); + } + return lines.filter(Boolean); }; const handleSubmit = () => { @@ -167,14 +205,37 @@ const ChannelBatchForm = ({ channelIds, isOpen, onClose }) => { setIsSubmitting(true); try { + const formValues = form.getValues(); + const shouldClearOverrides = formValues.clear_overrides === 'clear'; + const values = buildSubmitValues( - form.getValues(), + formValues, selectedChannelGroup, selectedLogoId ); + // Clear runs before the routing PATCH (not in parallel) so a + // late-landing clear cannot wipe the freshly-written override + // fields. + if (shouldClearOverrides && channelIds.length > 0) { + await updateChannels(channelIds, { override: null }); + } + if (Object.keys(values).length > 0) { - await updateChannels(channelIds, values); + // Route auto-created channels to override.X (survives sync) + // and manual channels to direct Channel.X writes; auto_created + // is read from the table store. + const channelsById = useChannelsTableStore + .getState() + .channels.reduce((acc, c) => { + acc[c.id] = c; + return acc; + }, {}); + await updateChannelsWithOverrideRouting( + channelIds, + values, + channelsById + ); } if (regexFind.trim().length > 0) { @@ -197,7 +258,16 @@ const ChannelBatchForm = ({ channelIds, isOpen, onClose }) => { ]); onClose(); } catch (error) { - console.error('Failed to update channels:', error); + // Keep the form open with the user's edits intact so they can correct + // a validation error without retyping the bulk selection. + showNotification({ + title: 'Bulk Update Failed', + message: + error?.body?.detail || + error?.message || + 'Failed to apply changes to the selected channels.', + color: 'red', + }); } finally { setIsSubmitting(false); } @@ -457,6 +527,9 @@ const ChannelBatchForm = ({ channelIds, isOpen, onClose }) => { styles={{ hannontent: { '--mantine-color-body': '#27272A' } }} > + + {getSelectionSummary()} + @@ -799,6 +872,31 @@ const ChannelBatchForm = ({ channelIds, isOpen, onClose }) => { { value: 'false', label: 'No' }, ]} /> + + @@ -905,6 +1003,9 @@ This action cannot be undone.`} style={{ backgroundColor: 'rgba(0, 0, 0, 0.2)' }} > + + {getSelectionSummary()} + {getConfirmationMessage().map((change, index) => ( { + if (!group) return null; + const streamCount = + typeof group.stream_count === 'number' ? group.stream_count : null; + + return ( + + Configure: {group.name} + {streamCount !== null && ( + + ({streamCount} stream{streamCount === 1 ? '' : 's'} available) + + )} + + } + styles={{ content: { '--mantine-color-body': '#27272A' } }} + > + {children} + {/* Done keeps in-memory edits routed into the parent's groupStates + (parent's Save and Refresh persists them). Cancel reverts to + the open-time snapshot. */} + + + + + + ); +}; + +export default GroupConfigureModal; diff --git a/frontend/src/components/forms/LiveGroupFilter.jsx b/frontend/src/components/forms/LiveGroupFilter.jsx index 4cfdaca6..b1e87da7 100644 --- a/frontend/src/components/forms/LiveGroupFilter.jsx +++ b/frontend/src/components/forms/LiveGroupFilter.jsx @@ -1,5 +1,4 @@ -// Modal.js -import React, { useState, useEffect, forwardRef } from 'react'; +import React, { useEffect, useRef, useState } from 'react'; import { TextInput, Button, @@ -20,8 +19,19 @@ import { ScrollArea, Center, SegmentedControl, + ActionIcon, + Switch, } from '@mantine/core'; -import { Info, CircleCheck, CircleX } from 'lucide-react'; +import { + Info, + CircleCheck, + CircleX, + Settings as Cog, + AlertTriangle, + RefreshCw, +} from 'lucide-react'; +import GroupConfigureModal from './GroupConfigureModal'; +import { notifications } from '@mantine/notifications'; import useChannelsStore from '../../store/channels'; import useStreamProfilesStore from '../../store/streamProfiles'; import { useChannelLogoSelection } from '../../hooks/useSmartLogos'; @@ -30,17 +40,7 @@ import LazyLogo from '../LazyLogo'; import LogoForm from './Logo'; import logo from '../../images/logo.png'; import API from '../../api'; - -// Custom item component for MultiSelect with tooltip -const OptionWithTooltip = forwardRef( - ({ label, description, ...others }, ref) => ( - -
- {label} -
- - ) -); +import { getGroupReservation } from '../../utils/forms/GroupSyncUtils'; const LiveGroupFilter = ({ playlist, @@ -57,7 +57,6 @@ const LiveGroupFilter = ({ const [statusFilter, setStatusFilter] = useState('all'); const [epgSources, setEpgSources] = useState([]); - // Logo selection functionality const { logos: channelLogos, ensureLogosLoaded, @@ -65,6 +64,373 @@ const LiveGroupFilter = ({ } = useChannelLogoSelection(); const [logoModalOpen, setLogoModalOpen] = useState(false); const [currentEditingGroupId, setCurrentEditingGroupId] = useState(null); + const [configuringGroupId, setConfiguringGroupId] = useState(null); + // Snapshot of the configuring group's state taken when the Configure + // modal opens. Cancel restores from this; Done discards it. + const configureSnapshotRef = useRef(null); + // Merged per-group conflict state: { id: { hasChannelConflict: bool } } + // sourced from the debounced /numbers-in-range/ scan plus an in-memory + // overlap check against other groups' ranges in modal state. + const [groupConflicts, setGroupConflicts] = useState({}); + const conflictTimersRef = useRef({}); + // Aborts the previous /numbers-in-range/ call so a slow response cannot + // overwrite newer state. + const conflictAbortRef = useRef({}); + // Conflict state split by source ('occupant' DB scan vs 'form' overlap). + // The render-time `hasChannelConflict` is `occupant || form`; tracking + // both lets the sweep refresh form-overlap synchronously while only + // firing the DB scan when a group's own range changes. + const conflictSourcesRef = useRef({}); + // Signature of each group's conflict-relevant fields from the last sweep. + // The sweep skips the (debounced) DB scan when the signature is + // unchanged, so unrelated state changes do not fan out HTTP requests. + const lastConflictSigRef = useRef({}); + // Per-group regex preview state mirroring the /streams/regex-preview/ + // payload (find/filter results, counts, scan_limit_hit). Cached by + // pattern args; cache lifetime = modal lifetime. + const [regexPreviewState, setRegexPreviewState] = useState({}); + const regexPreviewTimersRef = useRef({}); + const regexPreviewCacheRef = useRef({}); + // Aborts the previous regex preview request so out-of-order responses + // cannot stomp newer state. + const regexPreviewAbortRef = useRef({}); + const configuringGroup = configuringGroupId + ? groupStates.find((g) => g.channel_group === configuringGroupId) + : null; + const applyGroupChange = (nextGroupState) => { + setGroupStates((prev) => + prev.map((state) => + state.channel_group === nextGroupState.channel_group + ? nextGroupState + : state + ) + ); + }; + + // "Expected" occupants are this group's own auto-sync output: + // auto_created, in this group on this account, no channel_number + // override. Channels from any other provider, group, or with a user + // pin all surface as a warning so the user is aware their range + // overlaps with existing assignments. Sync still merges shared + // ranges across providers, so the warning is informational rather + // than blocking. + const isExpectedOccupantForGroup = (occupant, groupChannelGroupId) => { + if (!occupant) return false; + if (!occupant.auto_created) return false; + if (occupant.has_channel_number_override) return false; + if ( + occupant.channel_group_id !== undefined && + occupant.channel_group_id !== groupChannelGroupId + ) + return false; + if ( + occupant.auto_created_by_account_id !== undefined && + playlist?.id !== undefined && + occupant.auto_created_by_account_id !== playlist.id + ) + return false; + return true; + }; + + // Update one source ('occupant' or 'form') of a group's conflict + // tracking and re-merge into the public `groupConflicts` state. + const setConflictSource = (groupId, source, value) => { + const prev = conflictSourcesRef.current[groupId] || { + occupant: false, + form: false, + }; + if (prev[source] === value) return; + const next = { ...prev, [source]: value }; + conflictSourcesRef.current[groupId] = next; + setGroupConflicts((prevState) => ({ + ...prevState, + [groupId]: { hasChannelConflict: next.occupant || next.form }, + })); + }; + + // Debounced /numbers-in-range/ scan; sets `occupant` conflict source + // when any returned channel is not this group's own auto-sync output. + // + // Design: three refs (timer, abort, signature) cooperate to keep the + // request volume tied to user intent rather than render frequency. + // The timer debounces fast keystrokes; the abort controller cancels + // any in-flight request so a slow response cannot stomp newer state; + // and the parent sweep effect skips this scheduler entirely when a + // group's start/end signature has not changed since the last sweep. + // The conflict result is split into 'occupant' (DB scan) and 'form' + // (in-memory range overlap with sibling groups) sources so the sweep + // can refresh form-overlap synchronously without firing HTTP for + // groups that did not change. + const scheduleConflictScan = (groupId, rawStart, rawEnd) => { + if (conflictTimersRef.current[groupId]) { + clearTimeout(conflictTimersRef.current[groupId]); + } + if (conflictAbortRef.current[groupId]) { + conflictAbortRef.current[groupId].abort(); + } + const start = Number(rawStart); + const end = + rawEnd === null || rawEnd === undefined || rawEnd === '' + ? start + : Number(rawEnd); + if (!Number.isFinite(start) || start <= 0) { + setConflictSource(groupId, 'occupant', false); + return; + } + conflictTimersRef.current[groupId] = setTimeout(async () => { + const controller = new AbortController(); + conflictAbortRef.current[groupId] = controller; + try { + const result = await API.getChannelsInRange(start, end, { + signal: controller.signal, + }); + const occupants = Array.isArray(result?.occupants) + ? result.occupants + : []; + const unexpected = occupants.filter( + (o) => !isExpectedOccupantForGroup(o, groupId) + ); + setConflictSource(groupId, 'occupant', unexpected.length > 0); + } catch (e) { + // Aborted by a newer keystroke; the newer call will replace state. + if (e?.name === 'AbortError') return; + throw e; + } + }, 300); + }; + + useEffect(() => { + // Clear pending timers and abort in-flight conflict-scan requests on + // unmount so a late response cannot setState on an unmounted component. + return () => { + Object.values(conflictTimersRef.current).forEach((t) => clearTimeout(t)); + conflictTimersRef.current = {}; + Object.values(conflictAbortRef.current).forEach((c) => { + try { + c.abort(); + } catch { + // ignore + } + }); + conflictAbortRef.current = {}; + }; + }, []); + + // Sweep effect: recomputes form-overlap in-memory for every group + // (cheap). The HTTP-bound DB scan only runs for groups whose own + // range fields changed since the last sweep. + useEffect(() => { + const rangeFor = (g) => { + if (!g.enabled || !g.auto_channel_sync) return null; + const mode = g.custom_properties?.channel_numbering_mode || 'fixed'; + if (mode === 'next_available') return null; + const startRaw = + mode === 'provider' + ? (g.custom_properties?.channel_numbering_fallback ?? 1) + : (g.auto_sync_channel_start ?? 1); + const start = Number(startRaw); + if (!Number.isFinite(start)) return null; + const endRaw = g.auto_sync_channel_end; + const end = + endRaw === null || endRaw === undefined || endRaw === '' + ? start + : Number(endRaw); + return { start, end, startRaw }; + }; + + const ranges = new Map(); + for (const g of groupStates) { + const r = rangeFor(g); + if (r) ranges.set(g.channel_group, r); + } + + for (const g of groupStates) { + const range = ranges.get(g.channel_group); + if (!range) { + // Group out of scope (disabled, mode flipped, or start blanked). + // Abort any in-flight scan so its late response cannot stamp a + // stale 'occupant' value onto the cleared state. + if (conflictTimersRef.current[g.channel_group]) { + clearTimeout(conflictTimersRef.current[g.channel_group]); + delete conflictTimersRef.current[g.channel_group]; + } + if (conflictAbortRef.current[g.channel_group]) { + conflictAbortRef.current[g.channel_group].abort(); + delete conflictAbortRef.current[g.channel_group]; + } + setConflictSource(g.channel_group, 'form', false); + setConflictSource(g.channel_group, 'occupant', false); + delete lastConflictSigRef.current[g.channel_group]; + continue; + } + + let hasFormConflict = false; + for (const [otherId, otherRange] of ranges) { + if (otherId === g.channel_group) continue; + if (range.start <= otherRange.end && otherRange.start <= range.end) { + hasFormConflict = true; + break; + } + } + setConflictSource(g.channel_group, 'form', hasFormConflict); + + const sig = `${range.start}|${range.end}`; + if (lastConflictSigRef.current[g.channel_group] !== sig) { + lastConflictSigRef.current[g.channel_group] = sig; + scheduleConflictScan( + g.channel_group, + range.startRaw, + g.auto_sync_channel_end + ); + } + } + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [groupStates]); + + // Debounced regex preview fetcher. Each call computes a cache key from + // the group + pattern args; identical arg sets reuse the cached result + // instantly. Distinct keys schedule a backend round-trip 500ms after + // the last change so the user can finish typing before the request + // fires. Backend caps in-memory iteration at 5000 streams per call so + // groups with tens of thousands of streams stay performant. Three + // independent patterns are supported per call: find/replace, include + // filter, exclude filter. + const scheduleRegexPreview = (group, opts) => { + const groupId = group.channel_group; + const find = opts.find || ''; + const replace = opts.replace ?? ''; + const match = opts.match || ''; + const exclude = opts.exclude || ''; + const emptyState = { + findResult: null, + filterResult: null, + excludeResult: null, + loading: false, + }; + // Clear any pending request whenever the inputs settle on a state that + // does not require a backend round-trip (all-empty or cache hit). + // Otherwise a 500ms-old timer would still fire and stomp the new state. + const cancelPending = () => { + if (regexPreviewTimersRef.current[groupId]) { + clearTimeout(regexPreviewTimersRef.current[groupId]); + regexPreviewTimersRef.current[groupId] = null; + } + if (regexPreviewAbortRef.current[groupId]) { + regexPreviewAbortRef.current[groupId].abort(); + regexPreviewAbortRef.current[groupId] = null; + } + }; + if (!find && !match && !exclude) { + cancelPending(); + setRegexPreviewState((prev) => ({ ...prev, [groupId]: emptyState })); + return; + } + // Account ID in the cache key so previews stay correct when the + // user switches between accounts that share a group name. + const accountId = playlist?.id ?? ''; + const cacheKey = `${accountId}|${groupId}|${find}|${replace}|${match}|${exclude}`; + const cached = regexPreviewCacheRef.current[cacheKey]; + if (cached) { + cancelPending(); + setRegexPreviewState((prev) => ({ + ...prev, + [groupId]: { ...cached, loading: false }, + })); + return; + } + if (regexPreviewTimersRef.current[groupId]) { + clearTimeout(regexPreviewTimersRef.current[groupId]); + } + if (regexPreviewAbortRef.current[groupId]) { + regexPreviewAbortRef.current[groupId].abort(); + } + setRegexPreviewState((prev) => ({ + ...prev, + [groupId]: { + ...(prev[groupId] || { + findResult: null, + filterResult: null, + excludeResult: null, + }), + loading: true, + }, + })); + regexPreviewTimersRef.current[groupId] = setTimeout(async () => { + const controller = new AbortController(); + regexPreviewAbortRef.current[groupId] = controller; + let response; + try { + response = await API.getStreamsRegexPreview(group.name, { + find: find || undefined, + replace: find ? replace : undefined, + match: match || undefined, + exclude: exclude || undefined, + limit: 10, + signal: controller.signal, + m3uAccountId: playlist?.id, + }); + } catch (e) { + if (e?.name === 'AbortError') return; + throw e; + } + if (!response) { + setRegexPreviewState((prev) => ({ ...prev, [groupId]: emptyState })); + return; + } + const buildResult = (key, errorKey) => ({ + matches: response[`${key}_matches`] || [], + match_count: response[`${key}_match_count`] || 0, + total_in_group: response.total_in_group || 0, + total_scanned: response.total_scanned || 0, + scan_limit_hit: !!response.scan_limit_hit, + error: response[errorKey] || null, + }); + const next = { + findResult: find ? buildResult('find', 'find_error') : null, + filterResult: match ? buildResult('filter', 'match_error') : null, + excludeResult: exclude ? buildResult('exclude', 'exclude_error') : null, + loading: false, + }; + regexPreviewCacheRef.current[cacheKey] = next; + setRegexPreviewState((prev) => ({ + ...prev, + [groupId]: next, + })); + }, 500); + }; + + useEffect(() => { + return () => { + Object.values(regexPreviewTimersRef.current).forEach((t) => + clearTimeout(t) + ); + regexPreviewTimersRef.current = {}; + Object.values(regexPreviewAbortRef.current).forEach((c) => { + try { + c.abort(); + } catch { + // ignore + } + }); + regexPreviewAbortRef.current = {}; + }; + }, []); + + // When the gear modal opens (or its open group changes), trigger a + // preview fetch using whatever patterns are already saved on that + // group. Subsequent edits to the patterns trigger their own scheduled + // fetches via the field handlers. + useEffect(() => { + if (!configuringGroup) return; + const cp = configuringGroup.custom_properties || {}; + scheduleRegexPreview(configuringGroup, { + find: cp.name_regex_pattern || '', + replace: cp.name_replace_pattern ?? '', + match: cp.name_match_regex || '', + exclude: cp.name_match_exclude_regex || '', + }); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [configuringGroup?.channel_group]); // Ensure logos are loaded when component mounts useEffect(() => { @@ -91,16 +457,28 @@ const LiveGroupFilter = ({ fetchEPGSources(); }, []); + // Build group state once per playlist, not on every prop reference change. + // The parent re-renders this component on WebSocket sync-progress updates, + // which would otherwise blow away in-progress edits while the modal is open. + const lastInitKey = useRef(null); useEffect(() => { if (Object.keys(channelGroups).length === 0) { return; } + const groupIds = (playlist.channel_groups || []) + .map((g) => g.channel_group) + .sort() + .join(','); + const initKey = `${playlist.id}:${groupIds}`; + if (lastInitKey.current === initKey) { + return; + } + lastInitKey.current = initKey; setGroupStates( playlist.channel_groups - .filter((group) => channelGroups[group.channel_group]) // Filter out groups that don't exist + .filter((group) => channelGroups[group.channel_group]) .map((group) => { - // Parse custom_properties if present let customProps = {}; if (group.custom_properties) { try { @@ -117,6 +495,7 @@ const LiveGroupFilter = ({ name: channelGroups[group.channel_group].name, auto_channel_sync: group.auto_channel_sync || false, auto_sync_channel_start: group.auto_sync_channel_start || 1.0, + auto_sync_channel_end: group.auto_sync_channel_end ?? null, custom_properties: customProps, original_enabled: group.enabled, }; @@ -125,8 +504,8 @@ const LiveGroupFilter = ({ }, [playlist, channelGroups]); const toggleGroupEnabled = (id) => { - setGroupStates( - groupStates.map((state) => ({ + setGroupStates((prev) => + prev.map((state) => ({ ...state, enabled: state.channel_group == id ? !state.enabled : state.enabled, })) @@ -134,32 +513,52 @@ const LiveGroupFilter = ({ }; const toggleAutoSync = (id) => { - setGroupStates( - groupStates.map((state) => ({ - ...state, - auto_channel_sync: - state.channel_group == id - ? !state.auto_channel_sync - : state.auto_channel_sync, - })) - ); - }; + setGroupStates((prev) => + prev.map((state) => { + if (state.channel_group != id) return state; + const turningOn = !state.auto_channel_sync; + const next = { ...state, auto_channel_sync: turningOn }; + if (!turningOn) return next; - const updateChannelStart = (id, value) => { - setGroupStates( - groupStates.map((state) => ({ - ...state, - auto_sync_channel_start: - state.channel_group == id ? value : state.auto_sync_channel_start, - })) + // Pick a sensible start when enabling auto-sync: max of other + // groups' end (or start) plus 1, so multiple groups don't all + // default to 1. Skipped if a non-default start is already set. + const currentStart = state.auto_sync_channel_start; + if (currentStart && currentStart > 1) return next; + + let proposedStart = 1; + for (const other of prev) { + if (other.channel_group == id) continue; + if (!other.enabled || !other.auto_channel_sync) continue; + const otherMode = + other.custom_properties?.channel_numbering_mode || 'fixed'; + if (otherMode === 'next_available') continue; + const otherStart = Number( + otherMode === 'provider' + ? (other.custom_properties?.channel_numbering_fallback ?? 1) + : (other.auto_sync_channel_start ?? 1) + ); + if (!Number.isFinite(otherStart)) continue; + const otherEnd = + other.auto_sync_channel_end === null || + other.auto_sync_channel_end === undefined || + other.auto_sync_channel_end === '' + ? otherStart + : Number(other.auto_sync_channel_end); + const upper = Math.max(otherStart, otherEnd); + if (upper + 1 > proposedStart) proposedStart = upper + 1; + } + next.auto_sync_channel_start = proposedStart; + return next; + }) ); }; // Handle logo selection from LogoForm const handleLogoSuccess = ({ logo }) => { if (logo && logo.id && currentEditingGroupId !== null) { - setGroupStates( - groupStates.map((state) => { + setGroupStates((prev) => + prev.map((state) => { if (state.channel_group === currentEditingGroupId) { return { ...state, @@ -172,7 +571,7 @@ const LiveGroupFilter = ({ return state; }) ); - ensureLogosLoaded(); // Refresh logos + ensureLogosLoaded(); } setLogoModalOpen(false); setCurrentEditingGroupId(null); @@ -190,8 +589,8 @@ const LiveGroupFilter = ({ }; const selectAll = () => { - setGroupStates( - groupStates.map((state) => ({ + setGroupStates((prev) => + prev.map((state) => ({ ...state, enabled: isVisible(state) ? true : state.enabled, })) @@ -199,14 +598,1012 @@ const LiveGroupFilter = ({ }; const deselectAll = () => { - setGroupStates( - groupStates.map((state) => ({ + setGroupStates((prev) => + prev.map((state) => ({ ...state, enabled: isVisible(state) ? false : state.enabled, })) ); }; + // Returns {name, start, end}[] for groups whose declared ranges + // intersect this group's range, or [] when there is no overlap. + const computeRangeOverlapsFor = (group) => { + const myReservation = getGroupReservation(group); + if (!myReservation) return []; + const [myStart, myEnd] = myReservation; + const overlaps = []; + for (const other of groupStates) { + if (other.channel_group === group.channel_group) continue; + const otherReservation = getGroupReservation(other); + if (!otherReservation) continue; + const [oStart, oEnd] = otherReservation; + if (myStart <= oEnd && oStart <= myEnd) { + overlaps.push({ name: other.name, start: oStart, end: oEnd }); + } + } + return overlaps; + }; + + // Inline Start/End range inputs for Fixed and Provider modes. Start + // writes to auto_sync_channel_start (Fixed) or + // custom_properties.channel_numbering_fallback (Provider); End always + // writes to auto_sync_channel_end. Next Available shows a one-line + // explanation because a range contradicts pack-anywhere semantics. + const renderNumberingRange = (group) => { + const mode = group.custom_properties?.channel_numbering_mode || 'fixed'; + if (mode === 'next_available') { + return ( + + Channels receive the lowest available numbers starting at 1. + + ); + } + const startValue = + mode === 'provider' + ? (group.custom_properties?.channel_numbering_fallback ?? 1) + : (group.auto_sync_channel_start ?? 1); + const endValue = group.auto_sync_channel_end; + // Caps pathological pasted values like "1e308" at the input layer. + const MAX_CHANNEL_NUMBER = 999999; + const clampChannelNumber = (n) => + Math.max(1, Math.min(MAX_CHANNEL_NUMBER, Math.floor(Number(n) || 1))); + const updateStart = (value) => { + const normalized = + value === '' || value === null || value === undefined + ? 1 + : clampChannelNumber(value); + if (mode === 'provider') { + applyGroupChange({ + ...group, + custom_properties: { + ...(group.custom_properties || {}), + channel_numbering_fallback: normalized, + }, + }); + } else { + // If End is set and the new Start exceeds it, drop End so the user + // is not left holding an invalid range silently. + const next = { ...group, auto_sync_channel_start: normalized }; + if ( + endValue !== null && + endValue !== undefined && + normalized > endValue + ) { + next.auto_sync_channel_end = null; + } + applyGroupChange(next); + } + // Sweep effect picks up the state change and dispatches the scan. + }; + const updateEnd = (value) => { + const normalized = + value === '' || value === null || value === undefined + ? null + : clampChannelNumber(value); + applyGroupChange({ + ...group, + auto_sync_channel_end: normalized, + }); + }; + + const streamCount = + typeof group.stream_count === 'number' ? group.stream_count : null; + const metaParts = []; + if (endValue) { + metaParts.push(`Range: ${Math.max(endValue - startValue + 1, 0)}`); + } + if (streamCount !== null) { + metaParts.push(`Streams: ${streamCount}`); + } + const metaText = metaParts.join(' · '); + + const conflict = groupConflicts[group.channel_group]; + const hasChannelConflict = !!conflict?.hasChannelConflict; + const overlaps = computeRangeOverlapsFor(group); + + // Channel-level conflicts get a generic Channels-page pointer (count + // can be large); range-level overlaps stay specific to the modal. + const tooltipSections = []; + if (hasChannelConflict) { + tooltipSections.push( + 'Range conflicts with configured channels.\nView the Channels page to inspect.' + ); + } + if (overlaps.length > 0) { + const overlapLines = overlaps + .map((o) => `${o.name} (${o.start}-${o.end})`) + .join('\n '); + tooltipSections.push( + overlaps.length === 1 + ? `Range overlaps with: ${overlaps[0].name} (${overlaps[0].start}-${overlaps[0].end})` + : `Range overlaps with:\n ${overlapLines}` + ); + } + const showWarning = tooltipSections.length > 0; + const tooltipBody = tooltipSections.join('\n\n'); + + return ( + + + + + + + + + + + + {metaText} + + {showWarning && ( + + + + )} + + + ); + }; + + // Header line for the preview box. Adds a scan-cap suffix when the + // backend only scanned the first SCAN_CAP streams of the group. + const formatPreviewSummary = (label, result) => { + if (!result) return null; + const { match_count, total_in_group, total_scanned, scan_limit_hit } = + result; + const matchWord = `match${match_count === 1 ? '' : 'es'}`; + if (scan_limit_hit) { + return `${match_count} ${matchWord} in first ${total_scanned.toLocaleString()} streams scanned (of ${total_in_group.toLocaleString()} total)`; + } + return `${match_count} ${label} ${matchWord} in ${total_scanned.toLocaleString()} stream${total_scanned === 1 ? '' : 's'}`; + }; + + // Find/replace regex preview backed by /streams/regex-preview/, so + // counts reflect the whole group (or up to SCAN_CAP) rather than a + // small client-side sample. + const renderRegexPreview = (group) => { + const find = group.custom_properties?.name_regex_pattern || ''; + if (!find) return null; + const state = regexPreviewState[group.channel_group] || {}; + const result = state.findResult; + const loading = state.loading; + return ( + + + {result ? formatPreviewSummary('rename', result) : 'Preview'} + + {result?.error && ( + + Invalid regex: {result.error} + + )} + {loading && !result && ( + + Scanning streams... + + )} + {result && !result.error && result.matches.length === 0 && ( + + {result.total_in_group === 0 + ? 'No streams in this group yet. Run an M3U refresh first to populate streams.' + : 'No streams matched this pattern.'} + + )} + {result?.matches?.map((row, idx) => ( + + + {row.before} + + + {' -> '} + + + {row.after} + + + ))} + + ); + }; + + // Shared preview box for include and exclude filters. The marker and + // color reflect whether matched names are kept (teal check) or dropped + // (red x); empty/loading/error states mirror the find preview. + const renderFilterPreview = (group, kind) => { + const pattern = + kind === 'exclude' + ? group.custom_properties?.name_match_exclude_regex || '' + : group.custom_properties?.name_match_regex || ''; + if (!pattern) return null; + const state = regexPreviewState[group.channel_group] || {}; + const result = + kind === 'exclude' ? state.excludeResult : state.filterResult; + const loading = state.loading; + const summaryLabel = kind === 'exclude' ? 'exclude' : 'filter'; + const placeholderLabel = + kind === 'exclude' ? 'Exclude preview' : 'Filter preview'; + const markerChar = kind === 'exclude' ? '✗' : '✓'; + const markerColor = kind === 'exclude' ? 'red.4' : 'teal.4'; + const emptyText = + kind === 'exclude' + ? 'No streams matched this pattern (nothing would be excluded).' + : 'No streams matched this pattern.'; + return ( + + + {result + ? formatPreviewSummary(summaryLabel, result) + : placeholderLabel} + + {result?.error && ( + + Invalid regex: {result.error} + + )} + {loading && !result && ( + + Scanning streams... + + )} + {result && !result.error && result.matches.length === 0 && ( + + {result.total_in_group === 0 + ? 'No streams in this group yet. Run an M3U refresh first to populate streams.' + : emptyText} + + )} + {result?.matches?.map((row, idx) => ( + + + {markerChar} + + + {row.name} + + + ))} + + ); + }; + + const renderMatchPreview = (group) => renderFilterPreview(group, 'include'); + const renderExcludePreview = (group) => renderFilterPreview(group, 'exclude'); + + // Advanced Options form rendered inside the gear modal. A field's + // presence in custom_properties activates it; blanking returns the + // group to default behavior. + const renderAdvancedOptions = (group) => { + const cp = group.custom_properties || {}; + const setCp = (patch, clears = []) => { + const next = { ...cp, ...patch }; + clears.forEach((k) => delete next[k]); + applyGroupChange({ ...group, custom_properties: next }); + }; + + // --- Name Transforms --- + + const findValue = cp.name_regex_pattern ?? ''; + const replaceValue = cp.name_replace_pattern ?? ''; + const filterValue = cp.name_match_regex ?? ''; + const excludeValue = cp.name_match_exclude_regex ?? ''; + const updateFind = (val) => { + if (!val && !replaceValue) { + setCp({}, ['name_regex_pattern', 'name_replace_pattern']); + } else { + setCp({ + name_regex_pattern: val, + name_replace_pattern: replaceValue, + }); + } + scheduleRegexPreview(group, { + find: val, + replace: replaceValue, + match: filterValue, + exclude: excludeValue, + }); + }; + const updateReplace = (val) => { + if (!val && !findValue) { + setCp({}, ['name_regex_pattern', 'name_replace_pattern']); + } else { + setCp({ + name_regex_pattern: findValue, + name_replace_pattern: val, + }); + } + scheduleRegexPreview(group, { + find: findValue, + replace: val, + match: filterValue, + exclude: excludeValue, + }); + }; + const updateFilter = (val) => { + if (!val) setCp({}, ['name_match_regex']); + else setCp({ name_match_regex: val }); + scheduleRegexPreview(group, { + find: findValue, + replace: replaceValue, + match: val, + exclude: excludeValue, + }); + }; + const updateExclude = (val) => { + if (!val) setCp({}, ['name_match_exclude_regex']); + else setCp({ name_match_exclude_regex: val }); + scheduleRegexPreview(group, { + find: findValue, + replace: replaceValue, + match: filterValue, + exclude: val, + }); + }; + + // --- EPG --- + + const epgValue = (() => { + if (cp.custom_epg_id !== undefined && cp.custom_epg_id !== null) { + return cp.custom_epg_id.toString(); + } + if (cp.force_dummy_epg) return '0'; + return ''; + })(); + const updateEpg = (value) => { + const next = { ...cp }; + delete next.custom_epg_id; + delete next.force_dummy_epg; + delete next.force_epg_selected; + if (value === '0') { + next.force_dummy_epg = true; + } else if (value) { + next.custom_epg_id = parseInt(value); + } + applyGroupChange({ ...group, custom_properties: next }); + }; + + // --- Channel Assignment --- + + const groupOverrideValue = cp.group_override + ? cp.group_override.toString() + : ''; + const updateGroupOverride = (value) => { + if (!value) setCp({}, ['group_override']); + else setCp({ group_override: parseInt(value) }); + }; + + const profileValue = cp.channel_profile_ids ?? []; + const updateProfiles = (value) => { + if (!value || value.length === 0) { + setCp({}, ['channel_profile_ids']); + } else { + setCp({ channel_profile_ids: value }); + } + }; + + const streamProfileValue = cp.stream_profile_id + ? cp.stream_profile_id.toString() + : ''; + const updateStreamProfile = (value) => { + if (!value) setCp({}, ['stream_profile_id']); + else setCp({ stream_profile_id: parseInt(value) }); + }; + + const sortOrderValue = cp.channel_sort_order ?? '__default__'; + const sortReverseEnabled = cp.channel_sort_order !== undefined; + const updateSortOrder = (value) => { + if (!value || value === '__default__') { + setCp({}, ['channel_sort_order', 'channel_sort_reverse']); + } else { + setCp({ + channel_sort_order: value, + channel_sort_reverse: cp.channel_sort_reverse ?? false, + }); + } + }; + const updateSortReverse = (checked) => { + setCp({ channel_sort_reverse: checked }); + }; + + // --- Custom Logo --- + + const logoValue = cp.custom_logo_id; + + return ( + + + + Name Transforms + + } + labelPosition="left" + size="sm" + color="gray.6" + /> + + + + updateFind(e.currentTarget.value)} + size="xs" + style={{ flex: 1 }} + /> + updateReplace(e.currentTarget.value)} + size="xs" + style={{ flex: 1 }} + /> + + {findValue && renderRegexPreview(group)} + + + + + + updateFilter(e.currentTarget.value)} + size="xs" + /> + {filterValue && renderMatchPreview(group)} + + + + + updateExclude(e.currentTarget.value)} + size="xs" + /> + {excludeValue && renderExcludePreview(group)} + + + + + + + + EPG & Logo + + } + labelPosition="left" + size="sm" + color="gray.6" + /> + + + ({ + value: g.id.toString(), + label: g.name, + }))} + clearable + searchable + size="xs" + style={{ flex: 1 }} + /> + + + ({ + value: profile.id.toString(), + label: profile.name, + }))} + clearable + searchable + size="xs" + style={{ flex: 1 }} + /> + + + + + + {sortReverseEnabled && ( + + updateSortReverse(event.currentTarget.checked) + } + size="xs" + mt="xs" + /> + )} + + + + + + + { + if (event.currentTarget.checked) { + setCp({ compact_numbering: true }); + } else { + setCp({}, ['compact_numbering']); + } + }} + size="xs" + /> + + + + + + + + + ); + }; + + // Local state mirrors the persisted mode so the SegmentedControl + // reflects clicks immediately even when the parent's playlist prop is + // a stale snapshot from before the PATCH lands. + const [orphanCleanupMode, setOrphanCleanupMode] = useState( + (playlist?.custom_properties || {}).orphan_channel_cleanup || 'always' + ); + + useEffect(() => { + setOrphanCleanupMode( + (playlist?.custom_properties || {}).orphan_channel_cleanup || 'always' + ); + }, [playlist?.id, playlist?.custom_properties?.orphan_channel_cleanup]); + + const handleOrphanCleanupChange = async (mode) => { + if (!playlist?.id) return; + const previousMode = orphanCleanupMode; + setOrphanCleanupMode(mode); + const nextProps = { + ...(playlist.custom_properties || {}), + orphan_channel_cleanup: mode, + }; + try { + await API.updatePlaylist({ + id: playlist.id, + custom_properties: nextProps, + }); + } catch (err) { + setOrphanCleanupMode(previousMode); + notifications.show({ + title: 'Failed to update cleanup mode', + message: err?.body?.detail || err?.message || 'Please try again.', + color: 'red', + }); + } + }; + return ( } color="blue" variant="light"> @@ -228,6 +1625,40 @@ const LiveGroupFilter = ({ description="When disabled, new groups from the M3U source will be created but disabled by default. You can enable them manually later." /> + + + + + Auto-sync orphan cleanup + + + + + + {orphanCleanupMode === 'always' && + 'Removes any auto-synced channel whose source stream is gone from this provider.'} + {orphanCleanupMode === 'preserve_customized' && + 'Removes orphaned auto-synced channels except those with active overrides.'} + {orphanCleanupMode === 'never' && + 'Keeps all orphaned auto-synced channels. You can clean up manually from the channels page.'} + + + - + - + toggleAutoSync(group.channel_group)} size="xs" /> + {group.auto_channel_sync && group.enabled && ( + + { + // Snapshot at open time so Cancel can restore + // pre-edit state. custom_properties needs a + // one-level clone since the rest of group + // state is flat. + configureSnapshotRef.current = { + ...group, + custom_properties: { + ...(group.custom_properties || {}), + }, + }; + setConfiguringGroupId(group.channel_group); + }} + aria-label="Configure group" + > + + + + )} {group.auto_channel_sync && group.enabled && ( @@ -352,1030 +1810,60 @@ const LiveGroupFilter = ({ w={280} openDelay={500} > - { - setGroupStates( - groupStates.map((state) => { - if ( - state.channel_group === group.channel_group - ) { - return { - ...state, - custom_properties: { - ...state.custom_properties, - channel_sort_order: value || '', - }, - }; - } - return state; - }) - ); - }} - data={[ - { - value: '', - label: 'Provider Order (Default)', - }, - { value: 'name', label: 'Name' }, - { value: 'tvg_id', label: 'TVG ID' }, - { - value: 'updated_at', - label: 'Updated At', - }, - ]} - clearable - searchable - size="xs" - /> - - {/* Add reverse sort checkbox when sort order is selected (including default) */} - {group.custom_properties?.channel_sort_order !== - undefined && ( - - { - setGroupStates( - groupStates.map((state) => { - if ( - state.channel_group === - group.channel_group - ) { - return { - ...state, - custom_properties: { - ...state.custom_properties, - channel_sort_reverse: - event.target.checked, - }, - }; - } - return state; - }) - ); - }} - size="xs" - /> - - )} - - )} - - {/* Show profile selection only if profile_assignment is selected */} - {group.custom_properties?.channel_profile_ids !== - undefined && ( - - { - setGroupStates( - groupStates.map((state) => { - if ( - state.channel_group === group.channel_group - ) { - return { - ...state, - custom_properties: { - ...state.custom_properties, - channel_profile_ids: value || [], - }, - }; - } - return state; - }) - ); - }} - data={Object.values(profiles).map((profile) => ({ - value: profile.id.toString(), - label: profile.name, - }))} - clearable - searchable - size="xs" - /> - - )} - - {/* Show group select only if group_override is selected */} - {group.custom_properties?.group_override !== - undefined && ( - - { - const newValue = value ? parseInt(value) : null; - setGroupStates( - groupStates.map((state) => { - if ( - state.channel_group === group.channel_group - ) { - return { - ...state, - custom_properties: { - ...state.custom_properties, - stream_profile_id: newValue, - }, - }; - } - return state; - }) - ); - }} - data={streamProfiles.map((profile) => ({ - value: profile.id.toString(), - label: profile.name, - }))} - clearable - searchable - size="xs" - /> - - )} - - {/* Show regex fields only if name_regex is selected */} - {(group.custom_properties?.name_regex_pattern !== - undefined || - group.custom_properties?.name_replace_pattern !== - undefined) && ( - <> - - { - const val = e.currentTarget.value; - setGroupStates( - groupStates.map((state) => - state.channel_group === group.channel_group - ? { - ...state, - custom_properties: { - ...state.custom_properties, - name_regex_pattern: val, - }, - } - : state - ) - ); - }} - size="xs" - /> - - - { - const val = e.currentTarget.value; - setGroupStates( - groupStates.map((state) => - state.channel_group === group.channel_group - ? { - ...state, - custom_properties: { - ...state.custom_properties, - name_replace_pattern: val, - }, - } - : state - ) - ); - }} - size="xs" - /> - - - )} - - {/* Show name_match_regex field only if selected */} - {group.custom_properties?.name_match_regex !== - undefined && ( - - { - const val = e.currentTarget.value; - setGroupStates( - groupStates.map((state) => - state.channel_group === group.channel_group - ? { - ...state, - custom_properties: { - ...state.custom_properties, - name_match_regex: val, - }, - } - : state - ) - ); - }} - size="xs" - /> - - )} - - {/* Show logo selector only if custom_logo is selected */} - {group.custom_properties?.custom_logo_id !== - undefined && ( - - - { - setGroupStates( - groupStates.map((state) => { - if ( - state.channel_group === - group.channel_group - ) { - return { - ...state, - logoPopoverOpened: opened, - }; - } - return state; - }) - ); - if (opened) { - ensureLogosLoaded(); - } - }} - withArrow - > - - { - setGroupStates( - groupStates.map((state) => { - if ( - state.channel_group === - group.channel_group - ) { - return { - ...state, - logoPopoverOpened: true, - }; - } - return { - ...state, - logoPopoverOpened: false, - }; - }) - ); - }} - size="xs" - /> - - - e.stopPropagation()} - > - - { - const val = e.currentTarget.value; - setGroupStates( - groupStates.map((state) => - state.channel_group === - group.channel_group - ? { - ...state, - logoFilter: val, - } - : state - ) - ); - }} - /> - {logosLoading && ( - - Loading... - - )} - - - - {(() => { - const logoOptions = [ - { id: '0', name: 'Default' }, - ...Object.values(channelLogos), - ]; - const filteredLogos = logoOptions.filter( - (logo) => - logo.name - .toLowerCase() - .includes( - ( - group.logoFilter || '' - ).toLowerCase() - ) - ); - - if (filteredLogos.length === 0) { - return ( -
- - {group.logoFilter - ? 'No logos match your filter' - : 'No logos available'} - -
- ); - } - - return ( - - {({ index, style }) => { - const logoItem = filteredLogos[index]; - return ( -
{ - setGroupStates( - groupStates.map((state) => { - if ( - state.channel_group === - group.channel_group - ) { - return { - ...state, - custom_properties: { - ...state.custom_properties, - custom_logo_id: - logoItem.id, - }, - logoPopoverOpened: false, - }; - } - return state; - }) - ); - }} - onMouseEnter={(e) => { - e.currentTarget.style.backgroundColor = - 'rgb(68, 68, 68)'; - }} - onMouseLeave={(e) => { - e.currentTarget.style.backgroundColor = - 'transparent'; - }} - > -
- {logoItem.name { - if (e.target.src !== logo) { - e.target.src = logo; - } - }} - /> - - {logoItem.name || 'Default'} - -
-
- ); - }} - - ); - })()} - - - - - - - - - - - - )} - - {/* Show EPG selector when force_epg is selected */} - {(group.custom_properties?.custom_epg_id !== undefined || - group.custom_properties?.force_dummy_epg || - group.custom_properties?.force_epg_selected) && ( - - - {label} - - ), + Switch: ({ label, checked, onChange }) => { + // The form renders multiple switches (Mature Content, Hide from + // Clients, and future flags), so derive a unique testid per label. + // The Mature Content switch keeps "switch-mature" so tests that look + // it up by that legacy id continue to work. + const key = + typeof label === 'string' && label.toLowerCase().includes('mature') + ? 'switch-mature' + : `switch-${String(label || '') + .toLowerCase() + .replace(/\s+/g, '-')}`; + return ( + + ); + }, Text: ({ children, size, c, style }) => ( {children} @@ -1014,17 +1039,29 @@ describe('ChannelForm', () => { }); }); - it('still calls onClose when submission throws', async () => { - vi.mocked(ChannelUtils.addChannel).mockRejectedValue( - new Error('Server error') - ); + it('does NOT close the form when submission throws (preserves user edits)', async () => { + // Regression guard: a save failure (validation error from the + // backend, network failure, etc.) must not call reset() + onClose() + // unconditionally; doing so drops the user's in-progress edits. The + // form early-returns on saveFailed and surfaces an error notification + // so the user can correct the error and retry without retyping. + const error = new Error('Server error'); + error.body = { detail: 'channel_number must be unique' }; + vi.mocked(ChannelUtils.addChannel).mockRejectedValue(error); setupMocks(); const onClose = vi.fn(); render(); fireEvent.click(screen.getByText('Submit')); await waitFor(() => { - expect(onClose).toHaveBeenCalled(); + expect(vi.mocked(ChannelUtils.addChannel)).toHaveBeenCalled(); }); + expect(onClose).not.toHaveBeenCalled(); + expect(showNotification).toHaveBeenCalledWith( + expect.objectContaining({ + color: 'red', + message: 'channel_number must be unique', + }) + ); }); }); @@ -1080,4 +1117,70 @@ describe('ChannelForm', () => { expect(epgInput).toHaveValue('EPG Source 1 - ESPN'); }); }); + + // ── Override reactivity ──────────────────────────────────────────────────── + // + // The "Currently overriding" tooltip and "Clear All Overrides (N)" button + // are derived state computed from the form's watched values. The derivation + // must recompute when any watched value changes, otherwise the affordance + // becomes stale and the user loses both the override count and the clear + // action between edits. + + describe('override reactivity', () => { + const autoCreatedChannel = () => + makeChannel({ id: 'ch-auto', auto_created: true }); + + it('updates "Clear All Overrides" affordance when a watched value diverges from provider', () => { + const channel = autoCreatedChannel(); + // Initial render: no field is overridden, so the Clear All button + // must NOT be visible. + vi.mocked(ChannelUtils.isFormFieldOverridden).mockReturnValue(false); + setupMocks({ channel }); + const { rerender } = render( + + ); + expect(screen.queryByText(/Clear All Overrides/)).not.toBeInTheDocument(); + + // Simulate the user typing into the name field. The mocked watch() + // now returns a different name, AND isFormFieldOverridden now + // reports the name field as overridden. The derived state must + // recompute and surface the Clear All affordance. + vi.mocked(ChannelUtils.isFormFieldOverridden).mockImplementation( + (_ch, field) => field === 'name' + ); + setupMocks({ + channel, + formOverrides: { watchValues: { name: 'My Override' } }, + }); + rerender(); + + expect(screen.getByText(/Clear All Overrides \(1\)/)).toBeInTheDocument(); + }); + + it('hides "Clear All Overrides" when the last divergent field reverts to provider', () => { + const channel = autoCreatedChannel(); + vi.mocked(ChannelUtils.isFormFieldOverridden).mockImplementation( + (_ch, field) => field === 'name' + ); + setupMocks({ + channel, + formOverrides: { watchValues: { name: 'My Override' } }, + }); + const { rerender } = render( + + ); + expect(screen.getByText(/Clear All Overrides \(1\)/)).toBeInTheDocument(); + + // User types the provider value back into the field. No fields are + // overridden anymore, so the affordance must disappear. + vi.mocked(ChannelUtils.isFormFieldOverridden).mockReturnValue(false); + setupMocks({ + channel, + formOverrides: { watchValues: { name: 'Test Channel' } }, + }); + rerender(); + + expect(screen.queryByText(/Clear All Overrides/)).not.toBeInTheDocument(); + }); + }); }); diff --git a/frontend/src/components/forms/__tests__/ChannelBatch.test.jsx b/frontend/src/components/forms/__tests__/ChannelBatch.test.jsx index b31b7d7c..9d503098 100644 --- a/frontend/src/components/forms/__tests__/ChannelBatch.test.jsx +++ b/frontend/src/components/forms/__tests__/ChannelBatch.test.jsx @@ -41,6 +41,7 @@ vi.mock('../../../utils/forms/ChannelBatchUtils.js', () => ({ setChannelNamesFromEpg: vi.fn(), setChannelTvgIdsFromEpg: vi.fn(), updateChannels: vi.fn(), + updateChannelsWithOverrideRouting: vi.fn(), })); // ── Sub-component mocks ──────────────────────────────────────────────────────── @@ -260,6 +261,9 @@ const setupMocks = (overrides = {}) => { vi.mocked(useChannelsTableStore).mockImplementation((sel) => sel({ channels: overrides.pageChannels ?? [] }) ); + useChannelsTableStore.getState = vi.fn(() => ({ + channels: overrides.pageChannels ?? [], + })); vi.mocked(useStreamProfilesStore).mockImplementation((sel) => sel({ profiles: overrides.profiles ?? makeProfiles() }) @@ -304,6 +308,9 @@ const setupMocks = (overrides = {}) => { }); vi.mocked(ChannelBatchUtils.buildEpgAssociations).mockResolvedValue(null); vi.mocked(ChannelBatchUtils.updateChannels).mockResolvedValue(undefined); + vi.mocked( + ChannelBatchUtils.updateChannelsWithOverrideRouting + ).mockResolvedValue(undefined); vi.mocked(ChannelBatchUtils.bulkRegexRenameChannels).mockResolvedValue( undefined ); @@ -434,6 +441,53 @@ describe('ChannelBatchForm', () => { screen.queryByTestId('confirmation-dialog') ).not.toBeInTheDocument(); }); + + it('runs clear-overrides BEFORE the routing PATCH so a same-submit clear cannot wipe just-written overrides', async () => { + // If clear and routing fired in parallel via Promise.all, the + // server could process them in either order. When clear lands + // last, the routing PATCH's freshly-written override fields are + // wiped silently. Awaiting clear before routing makes the user + // intent deterministic. + const callOrder = []; + const isWarningSuppressed = vi + .fn() + .mockImplementation((key) => key === 'batch-update-channels'); + setupMocks({ + isWarningSuppressed, + formValues: { clear_overrides: 'clear' }, + }); + vi.mocked(ChannelBatchUtils.buildSubmitValues).mockReturnValue({ + name: 'Renamed', + }); + // clear PATCH resolves slowly; routing PATCH resolves fast. If + // they were fired in parallel, routing would `await` first; the + // sequential code awaits clear first regardless of resolution + // speed. + vi.mocked(ChannelBatchUtils.updateChannels).mockImplementation( + () => + new Promise((resolve) => { + setTimeout(() => { + callOrder.push('clear'); + resolve(undefined); + }, 30); + }) + ); + vi.mocked( + ChannelBatchUtils.updateChannelsWithOverrideRouting + ).mockImplementation(async () => { + callOrder.push('routing'); + return undefined; + }); + + const onClose = vi.fn(); + renderForm({ onClose }); + fireEvent.click(screen.getByText('Submit')); + + await waitFor(() => { + expect(onClose).toHaveBeenCalled(); + }); + expect(callOrder).toEqual(['clear', 'routing']); + }); }); // ── Confirmation dialog ──────────────────────────────────────────────────── @@ -455,7 +509,7 @@ describe('ChannelBatchForm', () => { ).not.toBeInTheDocument(); }); - it('calls updateChannels on confirm', async () => { + it('routes confirm through updateChannelsWithOverrideRouting so auto-created channels write to override.X', async () => { setupMocks({ formValues: { stream_profile_id: '1', // survives: not '-1' or '0', so kept as-is after parseInt... @@ -474,10 +528,9 @@ describe('ChannelBatchForm', () => { fireEvent.click(screen.getByTestId('dialog-confirm')); await waitFor(() => { - expect(ChannelBatchUtils.updateChannels).toHaveBeenCalledWith( - CHANNEL_IDS, - { stream_profile_id: '1' } - ); + expect( + ChannelBatchUtils.updateChannelsWithOverrideRouting + ).toHaveBeenCalledWith(CHANNEL_IDS, { stream_profile_id: '1' }, {}); }); }); @@ -859,6 +912,64 @@ describe('ChannelBatchForm', () => { waitFor(() => fireEvent.click(screen.getByTestId('dialog-confirm'))) ).resolves.not.toThrow(); }); + + it('selection summary reflects current channelIds when prop changes', () => { + // Reactivity guard: the selection summary at the top of the form + // must update when the parent passes a different channelIds prop. + // A stale summary misleads the user about how many channels (and + // of which kind) are about to be edited. + setupMocks({ + pageChannels: [ + { id: 1, auto_created: true }, + { id: 2, auto_created: false }, + { id: 3, auto_created: true }, + { id: 4, auto_created: false }, + ], + }); + const { rerender } = renderForm({ channelIds: [1, 2] }); + // 1 auto + 1 manual. + expect(screen.getByText(/1 auto-synced, 1 manual/)).toBeInTheDocument(); + + rerender( + + ); + // After prop change: 2 auto + 1 manual. + expect(screen.getByText(/2 auto-synced, 1 manual/)).toBeInTheDocument(); + }); + + it('keeps the form open and surfaces an error notification when batch submit rejects', async () => { + // A bulk PATCH rejection (server validation, network) must not close + // the form silently; the user needs the error message to correct the + // issue and retry without losing the in-progress selection. + setupMocks(); + vi.mocked(ChannelBatchUtils.getChannelGroupChange).mockReturnValue( + '• Channel Group: Sports' + ); + vi.mocked( + ChannelBatchUtils.updateChannelsWithOverrideRouting + ).mockRejectedValue(new Error('Server error')); + const onClose = vi.fn(); + + renderForm({ onClose }); + fireEvent.click(screen.getByText('Submit')); + fireEvent.click(screen.getByTestId('dialog-confirm')); + + await waitFor(() => { + expect( + ChannelBatchUtils.updateChannelsWithOverrideRouting + ).toHaveBeenCalled(); + }); + + expect(onClose).not.toHaveBeenCalled(); + expect(screen.getByTestId('modal')).toBeInTheDocument(); + expect(showNotification).toHaveBeenCalledWith( + expect.objectContaining({ color: 'red' }) + ); + }); }); // ── Batch update confirmation message ────────────────────────────────────── diff --git a/frontend/src/components/forms/__tests__/GroupConfigureModal.test.jsx b/frontend/src/components/forms/__tests__/GroupConfigureModal.test.jsx new file mode 100644 index 00000000..26e8e44a --- /dev/null +++ b/frontend/src/components/forms/__tests__/GroupConfigureModal.test.jsx @@ -0,0 +1,95 @@ +import { render, screen, fireEvent } from '@testing-library/react'; +import { describe, it, expect, vi } from 'vitest'; + +// Contract test for the per-group Configure modal. Covers the footer +// affordances added so users have an unambiguous bottom-of-modal action: +// Done keeps in-memory edits, Cancel triggers the parent's revert path. +// The corner X is gone; Esc / click-outside route the modal's onClose +// to onCancel. + +vi.mock('@mantine/core', () => ({ + Modal: ({ opened, onClose, withCloseButton, children, title }) => + opened ? ( +
{ + if (e.target.dataset.testid === 'modal-overlay') onClose?.(); + }} + > +
{title}
+ {children} +
+ ) : null, + Stack: ({ children }) =>
{children}
, + Group: ({ children }) =>
{children}
, + Text: ({ children }) => {children}, + Button: ({ children, onClick }) => ( + + ), +})); + +import GroupConfigureModal from '../GroupConfigureModal'; + +const baseGroup = { channel_group: 1, name: 'Sports', stream_count: 42 }; + +const renderModal = (overrides = {}) => { + const onDone = vi.fn(); + const onCancel = vi.fn(); + render( + +
advanced options content
+ + ); + return { onDone, onCancel }; +}; + +describe('GroupConfigureModal footer affordances', () => { + it('returns null when no group is provided', () => { + const { container } = render( + + child + + ); + expect(container.firstChild).toBeNull(); + }); + + it('renders Done and Cancel buttons in the footer', () => { + renderModal(); + expect(screen.getByText('Done')).toBeInTheDocument(); + expect(screen.getByText('Cancel')).toBeInTheDocument(); + }); + + it('removes the corner X close button so the footer is the only action', () => { + renderModal(); + expect(screen.getByTestId('modal')).toHaveAttribute( + 'data-with-close-button', + 'false' + ); + }); + + it('clicking Done calls onDone and not onCancel', () => { + const { onDone, onCancel } = renderModal(); + fireEvent.click(screen.getByText('Done')); + expect(onDone).toHaveBeenCalledTimes(1); + expect(onCancel).not.toHaveBeenCalled(); + }); + + it('clicking Cancel calls onCancel and not onDone', () => { + const { onDone, onCancel } = renderModal(); + fireEvent.click(screen.getByText('Cancel')); + expect(onCancel).toHaveBeenCalledTimes(1); + expect(onDone).not.toHaveBeenCalled(); + }); + + it('renders the children passed in (advanced options content)', () => { + renderModal(); + expect(screen.getByTestId('advanced-options')).toBeInTheDocument(); + }); +}); diff --git a/frontend/src/components/forms/__tests__/LiveGroupFilter.test.jsx b/frontend/src/components/forms/__tests__/LiveGroupFilter.test.jsx new file mode 100644 index 00000000..35549856 --- /dev/null +++ b/frontend/src/components/forms/__tests__/LiveGroupFilter.test.jsx @@ -0,0 +1,251 @@ +import { render, screen, fireEvent, waitFor } from '@testing-library/react'; +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +// Targeted Vitest for the orphan-cleanup SegmentedControl that lives at +// the top of the M3U account's group-settings page. Covers default-mode +// resolution, click-to-PATCH, and optimistic-update / error-revert +// behavior. Other parts of LiveGroupFilter (per-group inline config, +// gear modal, regex preview, overlap warning) are not unit-tested here +// because they depend on external APIs and live data; they are exercised +// via the in-repo manual UI walkthrough. + +// ── Store mocks ──────────────────────────────────────────────────────────────── +vi.mock('../../../store/channels', () => ({ default: vi.fn() })); +vi.mock('../../../store/streamProfiles', () => ({ default: vi.fn() })); + +// ── Hook mocks ───────────────────────────────────────────────────────────────── +vi.mock('../../../hooks/useSmartLogos', () => ({ + useChannelLogoSelection: vi.fn(() => ({ + logos: {}, + ensureLogosLoaded: vi.fn(), + isLoading: false, + })), +})); + +// ── Module mocks ─────────────────────────────────────────────────────────────── +vi.mock('@mantine/notifications', () => ({ + notifications: { show: vi.fn() }, +})); + +vi.mock('../../../api', () => ({ + default: { + updatePlaylist: vi.fn(() => Promise.resolve({})), + getEPGs: vi.fn(() => Promise.resolve([])), + getChannelsInRange: vi.fn(() => Promise.resolve({ data: [] })), + getStreamsRegexPreview: vi.fn(() => Promise.resolve({ data: {} })), + repackGroupChannels: vi.fn(() => Promise.resolve({})), + }, +})); + +vi.mock('../../../utils/forms/GroupSyncUtils', () => ({ + getGroupReservation: vi.fn(() => null), +})); + +// ── Child component mocks ────────────────────────────────────────────────────── +vi.mock('../GroupConfigureModal', () => ({ + default: ({ children }) =>
{children}
, +})); + +vi.mock('../Logo', () => ({ default: () => null })); +vi.mock('../../LazyLogo', () => ({ default: () => null })); +vi.mock('../../../images/logo.png', () => ({ default: 'default-logo.png' })); + +vi.mock('react-window', () => ({ + FixedSizeList: ({ children, itemCount }) => ( +
+ {Array.from({ length: itemCount }, (_, index) => + children({ index, style: {} }) + )} +
+ ), +})); + +vi.mock('lucide-react', () => ({ + Info: () => , + CircleCheck: () => , + CircleX: () => , + Settings: () => , + AlertTriangle: () => , + RefreshCw: () => , +})); + +// ── @mantine/core minimal mocks ──────────────────────────────────────────────── +vi.mock('@mantine/core', () => ({ + TextInput: ({ value, onChange, placeholder }) => ( + + ), + Button: ({ children, onClick }) => , + Checkbox: ({ label, checked, onChange }) => ( + + ), + Flex: ({ children }) =>
{children}
, + Select: ({ value, onChange, data }) => ( + + ), + Stack: ({ children }) =>
{children}
, + Group: ({ children }) =>
{children}
, + SimpleGrid: ({ children }) =>
{children}
, + Text: ({ children }) => {children}, + NumberInput: ({ value, onChange }) => ( + onChange?.(Number(e.target.value))} + /> + ), + Divider: ({ label }) =>
, + Alert: ({ children }) =>
{children}
, + Box: ({ children }) =>
{children}
, + MultiSelect: ({ value, onChange }) => ( + + ), + Tooltip: ({ children }) => <>{children}, + Popover: ({ children }) =>
{children}
, + ScrollArea: ({ children }) =>
{children}
, + Center: ({ children }) =>
{children}
, + SegmentedControl: ({ value, onChange, data }) => ( +
+ {data.map((opt) => ( + + ))} +
+ ), + ActionIcon: ({ children, onClick }) => , + Switch: ({ label, checked, onChange }) => ( + + ), +})); + +// ── Imports after mocks ──────────────────────────────────────────────────────── +import LiveGroupFilter from '../LiveGroupFilter'; +import API from '../../../api'; +import { notifications } from '@mantine/notifications'; +import useChannelsStore from '../../../store/channels'; +import useStreamProfilesStore from '../../../store/streamProfiles'; + +// ────────────────────────────────────────────────────────────────────────────── + +const makePlaylist = (overrides = {}) => ({ + id: 7, + channel_groups: [], + custom_properties: null, + ...overrides, +}); + +const setupStores = () => { + vi.mocked(useChannelsStore).mockImplementation((sel) => + sel({ channelGroups: {}, profiles: [] }) + ); + vi.mocked(useStreamProfilesStore).mockImplementation((sel) => + sel({ profiles: [], fetchProfiles: vi.fn() }) + ); +}; + +const renderFilter = (playlistOverrides = {}) => + render( + + ); + +// The orphan-cleanup SegmentedControl shares its mocked testid with the +// status-filter SegmentedControl that lives elsewhere in the form, so +// disambiguate by walking from a uniquely-testided child button up to +// its parent SegmentedControl element. +const findCleanupControl = () => + screen.getByTestId('segmented-always').closest('[data-testid="segmented-control"]'); + +describe('LiveGroupFilter orphan-cleanup SegmentedControl', () => { + beforeEach(() => { + vi.clearAllMocks(); + setupStores(); + }); + + it('defaults to "always" when custom_properties is null', () => { + renderFilter({ custom_properties: null }); + expect(findCleanupControl()).toHaveAttribute('data-value', 'always'); + }); + + it('defaults to "always" when the orphan_channel_cleanup key is absent', () => { + renderFilter({ custom_properties: { compact_numbering: true } }); + expect(findCleanupControl()).toHaveAttribute('data-value', 'always'); + }); + + it('reads the persisted mode from custom_properties on mount', () => { + renderFilter({ + custom_properties: { orphan_channel_cleanup: 'preserve_customized' }, + }); + expect(findCleanupControl()).toHaveAttribute( + 'data-value', + 'preserve_customized' + ); + }); + + it('PATCHes the playlist with merged custom_properties on click and updates the displayed value', async () => { + renderFilter({ + custom_properties: { compact_numbering: true }, + }); + fireEvent.click(screen.getByTestId('segmented-never')); + + await waitFor(() => { + expect(API.updatePlaylist).toHaveBeenCalledWith({ + id: 7, + custom_properties: { + compact_numbering: true, + orphan_channel_cleanup: 'never', + }, + }); + }); + expect(findCleanupControl()).toHaveAttribute('data-value', 'never'); + }); + + it('reverts to the previous mode and surfaces an error toast when the PATCH fails', async () => { + vi.mocked(API.updatePlaylist).mockRejectedValueOnce( + new Error('Server error') + ); + renderFilter({ + custom_properties: { orphan_channel_cleanup: 'always' }, + }); + fireEvent.click(screen.getByTestId('segmented-never')); + + await waitFor(() => { + expect(notifications.show).toHaveBeenCalledWith( + expect.objectContaining({ color: 'red' }) + ); + }); + expect(findCleanupControl()).toHaveAttribute('data-value', 'always'); + }); +}); diff --git a/frontend/src/components/tables/ChannelsTable.jsx b/frontend/src/components/tables/ChannelsTable.jsx index 48b84531..e97c1caa 100644 --- a/frontend/src/components/tables/ChannelsTable.jsx +++ b/frontend/src/components/tables/ChannelsTable.jsx @@ -38,7 +38,10 @@ import { ArrowUpDown, ArrowDownWideNarrow, Search, + EyeOff, + Pencil, } from 'lucide-react'; +import { listOverriddenFields } from '../../utils/forms/ChannelUtils.js'; import { Box, TextInput, @@ -241,11 +244,10 @@ const ChannelRowActions = React.memo( ); }, - // Custom comparator: only re-render when the actual channel changes. - // The row object is a new TanStack Table reference on each render, but - // row.original.id is stable. Callbacks read fresh data at call time. - (prevProps, nextProps) => - prevProps.row.original.id === nextProps.row.original.id + // Custom comparator: skip re-render when the channel's data object hasn't + // changed. row.original is stable when the underlying channel hasn't been + // updated; it becomes a new reference when the store replaces that channel. + (prevProps, nextProps) => prevProps.row.original === nextProps.row.original ); const ChannelsTable = ({ onReady }) => { @@ -326,6 +328,9 @@ const ChannelsTable = ({ onReady }) => { const [showOnlyStreamlessChannels, setShowOnlyStreamlessChannels] = useState(false); const [showOnlyStaleChannels, setShowOnlyStaleChannels] = useState(false); + const [showOnlyOverriddenChannels, setShowOnlyOverriddenChannels] = + useState(false); + const [visibilityFilter, setVisibilityFilter] = useState('active'); const [paginationString, setPaginationString] = useState(''); const [filters, setFilters] = useState({ @@ -429,6 +434,14 @@ const ChannelsTable = ({ onReady }) => { if (showOnlyStaleChannels === true) { params.append('only_stale', true); } + if (showOnlyOverriddenChannels === true) { + params.append('only_has_overrides', true); + } + // The backend defaults to "active"; send other choices explicitly so + // hidden rows surface when the user opts into "Hidden Only" or "Show All". + if (visibilityFilter && visibilityFilter !== 'active') { + params.append('visibility_filter', visibilityFilter); + } // Apply sorting if (sorting.length > 0) { @@ -525,6 +538,8 @@ const ChannelsTable = ({ onReady }) => { selectedProfileId, showOnlyStreamlessChannels, showOnlyStaleChannels, + showOnlyOverriddenChannels, + visibilityFilter, ]); const stopPropagation = useCallback((e) => { @@ -559,36 +574,30 @@ const ChannelsTable = ({ onReady }) => { })); }; - const editChannel = async (ch = null, opts = {}) => { - // If forceAdd is set, always open a blank form + const editChannel = useCallback(async (ch = null, opts = {}) => { if (opts.forceAdd) { setChannel(null); setChannelModalOpen(true); return; } - // Use table's selected state instead of store state to avoid stale selections - const currentSelection = table ? table.selectedTableIds : []; - console.log('editChannel called with:', { - ch, - currentSelection, - tableExists: !!table, - }); + const currentSelection = + useChannelsTableStore.getState().selectedChannelIds; + console.log('editChannel called with:', { ch, currentSelection }); if (currentSelection.length > 1) { setChannelBatchModalOpen(true); } else { - // If no channel object is passed but we have a selection, get the selected channel let channelToEdit = ch; if (!channelToEdit && currentSelection.length === 1) { const selectedId = currentSelection[0]; - - // Use table data since that's what's currently displayed - channelToEdit = data.find((d) => d.id === selectedId); + channelToEdit = useChannelsTableStore + .getState() + .channels.find((d) => d.id === selectedId); } setChannel(channelToEdit); setChannelModalOpen(true); } - }; + }, []); const deleteChannel = async (id) => { console.log(`Deleting channel with ID: ${id}`); @@ -914,7 +923,10 @@ const ChannelsTable = ({ onReady }) => { }, { id: 'channel_number', - accessorKey: 'channel_number', + // Prefer the backend-resolved effective_channel_number so overrides + // show through to the table. Inline save still writes to the + // override row via buildInlinePatch in EditableCell. + accessorFn: (row) => row.effective_channel_number ?? row.channel_number, size: columnSizing.channel_number || 40, minSize: 30, maxSize: 100, @@ -922,16 +934,53 @@ const ChannelsTable = ({ onReady }) => { }, { id: 'name', - accessorKey: 'name', + accessorFn: (row) => row.effective_name ?? row.name, size: columnSizing.name || 200, minSize: 100, grow: true, - cell: (props) => , + cell: (props) => { + const row = props.row?.original || {}; + const overriddenLabels = listOverriddenFields(row); + return ( + + + + + {overriddenLabels.length > 0 && ( + + + + + )} + {row.hidden_from_output && ( + + + + + )} + + ); + }, }, { id: 'epg', header: 'EPG', - accessorKey: 'epg_data_id', + // Effective EPG id so overridden EPG assignments show in the table. + accessorFn: (row) => row.effective_epg_data_id ?? row.epg_data_id, cell: (props) => ( { }, { id: 'channel_group', - accessorFn: (row) => - channelGroups[row.channel_group_id] - ? channelGroups[row.channel_group_id].name - : '', + accessorFn: (row) => { + const effectiveGroupId = + row.effective_channel_group_id ?? row.channel_group_id; + return channelGroups[effectiveGroupId] + ? channelGroups[effectiveGroupId].name + : ''; + }, cell: (props) => ( ), @@ -957,10 +1009,7 @@ const ChannelsTable = ({ onReady }) => { }, { id: 'logo', - accessorFn: (row) => { - // Just pass the logo_id directly, not the full logo object - return row.logo_id; - }, + accessorFn: (row) => row.effective_logo_id ?? row.logo_id, size: 75, minSize: 50, maxSize: 120, @@ -1001,7 +1050,7 @@ const ChannelsTable = ({ onReady }) => { // from the store, so we don't need to recreate columns when logos load. // Note: tvgsLoaded is intentionally excluded - EditableEPGCell handles loading state internally // eslint-disable-next-line react-hooks/exhaustive-deps - [selectedProfileId, channelGroups, theme, tvgsById, epgs] + [selectedProfileId, channelGroups, theme, tvgsById, epgs, editChannel] ); const renderHeaderCell = (header) => { @@ -1515,6 +1564,10 @@ const ChannelsTable = ({ onReady }) => { setShowOnlyStreamlessChannels={setShowOnlyStreamlessChannels} showOnlyStaleChannels={showOnlyStaleChannels} setShowOnlyStaleChannels={setShowOnlyStaleChannels} + showOnlyOverriddenChannels={showOnlyOverriddenChannels} + setShowOnlyOverriddenChannels={setShowOnlyOverriddenChannels} + visibilityFilter={visibilityFilter} + setVisibilityFilter={setVisibilityFilter} /> {/* Table or ghost empty state inside Paper */} diff --git a/frontend/src/components/tables/ChannelsTable/ChannelTableHeader.jsx b/frontend/src/components/tables/ChannelsTable/ChannelTableHeader.jsx index cda9b496..fcef2479 100644 --- a/frontend/src/components/tables/ChannelsTable/ChannelTableHeader.jsx +++ b/frontend/src/components/tables/ChannelsTable/ChannelTableHeader.jsx @@ -118,6 +118,10 @@ const ChannelTableHeader = ({ setShowOnlyStreamlessChannels, showOnlyStaleChannels, setShowOnlyStaleChannels, + showOnlyOverriddenChannels, + setShowOnlyOverriddenChannels, + visibilityFilter, + setVisibilityFilter, }) => { const theme = useMantineTheme(); @@ -237,6 +241,12 @@ const ChannelTableHeader = ({ } }; + const toggleShowOnlyOverriddenChannels = () => { + if (setShowOnlyOverriddenChannels) { + setShowOnlyOverriddenChannels(!showOnlyOverriddenChannels); + } + }; + const toggleHeaderPinned = () => { setHeaderPinned(!headerPinned); }; @@ -327,11 +337,55 @@ const ChannelTableHeader = ({ : + showOnlyStaleChannels ? ( + + ) : ( + + ) } > Has Stale Streams + + + ) : ( + + ) + } + > + Has Overrides + + + + + Visibility + + + {[ + { value: 'active', label: 'Active Only' }, + { value: 'hidden', label: 'Hidden Only' }, + { value: 'all', label: 'Show All' }, + ].map(({ value, label }) => ( + + setVisibilityFilter && setVisibilityFilter(value) + } + leftSection={ + visibilityFilter === value ? ( + + ) : ( + + ) + } + > + {label} + + ))} diff --git a/frontend/src/components/tables/ChannelsTable/EditableCell.jsx b/frontend/src/components/tables/ChannelsTable/EditableCell.jsx index 42d99e73..55bd44e3 100644 --- a/frontend/src/components/tables/ChannelsTable/EditableCell.jsx +++ b/frontend/src/components/tables/ChannelsTable/EditableCell.jsx @@ -18,6 +18,54 @@ import { import API from '../../../api'; import useChannelsTableStore from '../../../store/channelsTable'; import useLogosStore from '../../../store/logos'; +import { + OVERRIDABLE_FIELDS, + normalizeFieldValue, +} from '../../../utils/forms/ChannelUtils.js'; +import { showNotification } from '../../../utils/notificationUtils.js'; + +// Surfaces server-side validation failures so the user knows the +// inline edit was rejected (otherwise the cell silently reverts). +// Exported so unit tests can verify the message composition without +// mounting the component. +export const notifyInlineSaveError = (columnId, error) => { + const detail = + error?.body?.detail || + error?.body?.[columnId]?.[0] || + error?.body?.error || + error?.message || + 'Server rejected the change'; + showNotification({ + title: 'Edit not saved', + message: String(detail), + color: 'red', + autoClose: 5000, + }); +}; + +// Inline edits on auto-synced channels route into the override row so +// sync cannot overwrite them. If the new value matches the provider's, +// clear that field's override instead of writing a duplicate. Manual +// channels keep direct Channel.* writes. +const buildInlinePatch = (rowOriginal, fieldId, newValue) => { + if (rowOriginal?.auto_created && OVERRIDABLE_FIELDS.includes(fieldId)) { + // Normalize both sides so a stringified form value compares + // cleanly against the typed provider value. + const formValue = normalizeFieldValue(fieldId, newValue); + const providerValue = normalizeFieldValue(fieldId, rowOriginal[fieldId]); + const overrideFieldValue = formValue === providerValue ? null : formValue; + return { + id: rowOriginal.id, + override: { [fieldId]: overrideFieldValue }, + }; + } + const normalized = + newValue === undefined || newValue === '' ? null : newValue; + return { + id: rowOriginal.id, + [fieldId]: normalized, + }; +}; // Lightweight wrapper that only renders full editable cell when unlocked // This prevents 250+ heavy component instances when table is locked @@ -103,10 +151,9 @@ const EditableTextCellInner = ({ row, column, getValue, onBlur }) => { } try { - const response = await API.updateChannel({ - id: row.original.id, - [column.id]: newValue || null, - }); + const response = await API.updateChannel( + buildInlinePatch(row.original, column.id, newValue) + ); previousValue.current = newValue; // Update the table store to reflect the change @@ -114,11 +161,13 @@ const EditableTextCellInner = ({ row, column, getValue, onBlur }) => { useChannelsTableStore.getState().updateChannel(response); } } catch (error) { - // Revert on error + // Surface server-side errors (e.g. max_length=512, validator + // rejection) so the user knows the change was not saved. + notifyInlineSaveError(column.id, error); setValue(previousValue.current || ''); } }, - [row.original.id, column.id] + [row.original, column.id] ); useEffect(() => { @@ -249,10 +298,9 @@ const EditableNumberCellInner = ({ row, column, getValue, onBlur }) => { } try { - const response = await API.updateChannel({ - id: row.original.id, - [column.id]: newValue, - }); + const response = await API.updateChannel( + buildInlinePatch(row.original, column.id, newValue) + ); previousValue.current = newValue; // Update the table store to reflect the change @@ -266,11 +314,13 @@ const EditableNumberCellInner = ({ row, column, getValue, onBlur }) => { } } } catch (error) { - // Revert on error + // Surface server-side errors (channel_number out-of-range, + // collision, etc.) so the user knows the change was not saved. + notifyInlineSaveError(column.id, error); setValue(previousValue.current); } }, - [row.original.id, column.id, onBlur] + [row.original, column.id, onBlur] ); useEffect(() => { @@ -366,10 +416,13 @@ const EditableGroupCellInner = ({ } try { - const response = await API.updateChannel({ - id: row.original.id, - channel_group_id: parseInt(newGroupId, 10), - }); + const response = await API.updateChannel( + buildInlinePatch( + row.original, + 'channel_group_id', + parseInt(newGroupId, 10) + ) + ); previousGroupId.current = newGroupId; // Update the table store to reflect the change @@ -378,9 +431,10 @@ const EditableGroupCellInner = ({ } } catch (error) { console.error('Failed to update channel group:', error); + notifyInlineSaveError(column.id, error); } }, - [row.original.id] + [row.original] ); const handleChange = (newGroupId) => { @@ -537,11 +591,13 @@ const EditableEPGCellInner = ({ } try { - const response = await API.updateChannel({ - id: row.original.id, - epg_data_id: - newEpgDataId === 'null' ? null : parseInt(newEpgDataId, 10), - }); + const response = await API.updateChannel( + buildInlinePatch( + row.original, + 'epg_data_id', + newEpgDataId === 'null' ? null : parseInt(newEpgDataId, 10) + ) + ); previousEpgDataId.current = newEpgDataId; // Update the table store to reflect the change @@ -550,9 +606,10 @@ const EditableEPGCellInner = ({ } } catch (error) { console.error('Failed to update EPG:', error); + notifyInlineSaveError(column.id, error); } }, - [row.original.id] + [row.original] ); const handleChange = (newEpgDataId) => { @@ -704,10 +761,13 @@ const EditableLogoCellInner = ({ row, logoId, onBlur }) => { } try { - const response = await API.updateChannel({ - id: row.original.id, - logo_id: newLogoId === 'null' ? null : parseInt(newLogoId, 10), - }); + const response = await API.updateChannel( + buildInlinePatch( + row.original, + 'logo_id', + newLogoId === 'null' ? null : parseInt(newLogoId, 10) + ) + ); previousLogoId.current = newLogoId; // Update the table store to reflect the change @@ -716,9 +776,10 @@ const EditableLogoCellInner = ({ row, logoId, onBlur }) => { } } catch (error) { console.error('Failed to update logo:', error); + notifyInlineSaveError(column.id, error); } }, - [row.original.id] + [row.original] ); const handleChange = (newLogoId) => { diff --git a/frontend/src/components/tables/ChannelsTable/__tests__/EditableCell.test.jsx b/frontend/src/components/tables/ChannelsTable/__tests__/EditableCell.test.jsx new file mode 100644 index 00000000..a67cbe35 --- /dev/null +++ b/frontend/src/components/tables/ChannelsTable/__tests__/EditableCell.test.jsx @@ -0,0 +1,82 @@ +// @vitest-environment jsdom +import { describe, it, expect, beforeEach, vi } from 'vitest'; + +// Mock the notification utility before importing the module under test +// so the helper picks up the mock. +vi.mock('../../../../utils/notificationUtils.js', () => ({ + showNotification: vi.fn(), +})); + +// Mock other heavy dependencies so the import doesn't pull in stores. +vi.mock('../../../../api', () => ({ default: {} })); +vi.mock('../../../../store/channelsTable', () => ({ default: { getState: vi.fn() } })); +vi.mock('../../../../store/logos', () => ({ default: vi.fn() })); +vi.mock('../../../../utils/forms/ChannelUtils.js', () => ({ + OVERRIDABLE_FIELDS: new Set(['name', 'channel_number', 'tvg_id']), + normalizeFieldValue: (v) => v, +})); + +import { notifyInlineSaveError } from '../EditableCell.jsx'; +import { showNotification } from '../../../../utils/notificationUtils.js'; + +describe('EditableCell.notifyInlineSaveError (F-12 regression guard)', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it('surfaces a red notification when an inline save fails (the fix)', () => { + // Regression guard: the previous implementation silently reverted + // the field on save error with no user feedback. The user could + // see their input vanish without knowing why. This helper must + // produce a visible error notification. + const error = new Error('Bad request'); + notifyInlineSaveError('name', error); + + expect(showNotification).toHaveBeenCalledTimes(1); + const arg = showNotification.mock.calls[0][0]; + expect(arg.color).toBe('red'); + expect(arg.title).toMatch(/not saved|error/i); + expect(arg.message).toBeTruthy(); + }); + + it('extracts a per-field validator message from a structured DRF response', () => { + // DRF serializer errors come back as { field: ["error string"] }. + // The helper must dig into the field-keyed array and surface the + // first message rather than rendering "[object Object]" or + // generic fallback. + const apiError = { + message: 'Request failed', + body: { + channel_number: ['Channel number 0 is below the allowed minimum.'], + }, + }; + notifyInlineSaveError('channel_number', apiError); + + const arg = showNotification.mock.calls[0][0]; + expect(arg.message).toContain('below the allowed minimum'); + }); + + it('extracts a top-level "detail" key when DRF emits a non-field error', () => { + const apiError = { + message: 'Request failed', + body: { detail: 'Channel name exceeds max_length=512' }, + }; + notifyInlineSaveError('name', apiError); + + const arg = showNotification.mock.calls[0][0]; + expect(arg.message).toContain('max_length'); + }); + + it('falls back to the error.message when no body shape is available', () => { + notifyInlineSaveError('name', new Error('Network error')); + const arg = showNotification.mock.calls[0][0]; + expect(arg.message).toContain('Network error'); + }); + + it('uses a generic fallback when nothing useful is on the error', () => { + notifyInlineSaveError('name', {}); + const arg = showNotification.mock.calls[0][0]; + expect(arg.message).toBeTruthy(); + expect(typeof arg.message).toBe('string'); + }); +}); diff --git a/frontend/src/components/tables/M3UsTable.jsx b/frontend/src/components/tables/M3UsTable.jsx index 2b0e6168..6384adc5 100644 --- a/frontend/src/components/tables/M3UsTable.jsx +++ b/frontend/src/components/tables/M3UsTable.jsx @@ -135,6 +135,12 @@ const M3UTable = () => { const [confirmDeleteOpen, setConfirmDeleteOpen] = useState(false); const [deleteTarget, setDeleteTarget] = useState(null); const [playlistToDelete, setPlaylistToDelete] = useState(null); + // Auto-created channel preview shown in the delete confirmation so the + // user sees what cascades along with the account. + const [autoChannelsInfo, setAutoChannelsInfo] = useState({ + count: 0, + sample_names: [], + }); const [data, setData] = useState([]); const [sorting, setSorting] = useState([{ id: 'name', desc: '' }]); const [deleting, setDeleting] = useState(false); @@ -404,8 +410,32 @@ const M3UTable = () => { setPlaylistToDelete(playlist); setDeleteTarget(id); - // Skip warning if it's been suppressed - if (isWarningSuppressed('delete-m3u')) { + // Fetch how many auto-created channels this playlist owns. Populates the + // confirmation message so the user can decide whether to also delete + // them. On failure, surface "unknown" so the user is not misled into + // thinking there are zero auto-created channels. + let info; + try { + const result = await API.getPlaylistAutoCreatedChannelsCount(id); + info = result || { count: 0, sample_names: [] }; + } catch { + info = { + count: null, + sample_names: [], + countUnavailable: true, + }; + } + setAutoChannelsInfo(info); + + // Skip the warning when it has been suppressed AND the account has + // no auto-created channels. When the account did create channels (or + // the count could not be resolved), the dialog still opens so the + // user sees and confirms what cascades. + if ( + isWarningSuppressed('delete-m3u') && + info.count === 0 && + !info.countUnavailable + ) { return executeDeletePlaylist(id); } @@ -421,6 +451,7 @@ const M3UTable = () => { setDeleting(false); setIsLoading(false); setConfirmDeleteOpen(false); + setAutoChannelsInfo({ count: 0, sample_names: [] }); } }; @@ -1028,15 +1059,49 @@ const M3UTable = () => { title="Confirm M3U Account Deletion" message={ playlistToDelete ? ( -
- {`Are you sure you want to delete the following M3U account? +
+
+ {`Delete the following M3U account? Name: ${playlistToDelete.name} Type: ${playlistToDelete.account_type === 'XC' ? 'Xtream Codes' : 'Standard'} Server: ${playlistToDelete.server_url || 'Local file'} -This will remove all related streams and may affect channels using these streams. +Streams owned by this provider will be removed. Manual channels that include those streams will lose them, but the channels and any other streams on them survive. + This action cannot be undone.`} +
+ {autoChannelsInfo.countUnavailable ? ( +
+ + Auto-synced channel count is unavailable; any channels + auto-created by this provider will be deleted with the + account. + +
+ ) : autoChannelsInfo.count > 0 ? ( +
+ + {`${autoChannelsInfo.count} auto-synced channel${autoChannelsInfo.count === 1 ? '' : 's'} created by this provider will also be deleted.`} + +
+ ) : null}
) : ( 'Are you sure you want to delete this M3U account? This action cannot be undone.' diff --git a/frontend/src/utils/forms/ChannelBatchUtils.js b/frontend/src/utils/forms/ChannelBatchUtils.js index 04cd17a5..cebe5990 100644 --- a/frontend/src/utils/forms/ChannelBatchUtils.js +++ b/frontend/src/utils/forms/ChannelBatchUtils.js @@ -49,6 +49,45 @@ export const updateChannels = (channelIds, values) => { return API.updateChannels(channelIds, values); }; +// Auto-created channels route override-able fields to override.* so they +// survive the next sync; manual channels write to the raw Channel columns. +// Non-overridable fields (status/permission flags) always write raw. +export const updateChannelsWithOverrideRouting = async ( + channelIds, + values, + channelsById +) => { + const { OVERRIDABLE_FIELDS } = await import('./ChannelUtils.js'); + const overrideKeys = new Set(OVERRIDABLE_FIELDS); + + const rawValues = {}; + const overrideValues = {}; + for (const [key, val] of Object.entries(values)) { + if (overrideKeys.has(key)) { + overrideValues[key] = val; + } else { + rawValues[key] = val; + } + } + + const body = []; + for (const id of channelIds) { + const channel = channelsById?.[id]; + const isAuto = !!channel?.auto_created; + const item = { id, ...rawValues }; + if (Object.keys(overrideValues).length > 0) { + if (isAuto) { + item.override = { ...overrideValues }; + } else { + Object.assign(item, overrideValues); + } + } + body.push(item); + } + + return API.bulkUpdateChannels(body); +}; + export const bulkRegexRenameChannels = ( channelIds, regexFind, @@ -152,6 +191,17 @@ export const buildSubmitValues = ( values.is_adult = values.is_adult === 'true'; } + if (values.hidden_from_output === '-1' || values.hidden_from_output === undefined) { + delete values.hidden_from_output; + } else { + values.hidden_from_output = values.hidden_from_output === 'true'; + } + + // clear_overrides is a UI-only flag; the caller splits it out and routes a + // follow-up PATCH to just the auto-created subset. Strip it from the main + // PATCH body. + delete values.clear_overrides; + return values; }; diff --git a/frontend/src/utils/forms/ChannelUtils.js b/frontend/src/utils/forms/ChannelUtils.js index 4a2a250f..467ff1e7 100644 --- a/frontend/src/utils/forms/ChannelUtils.js +++ b/frontend/src/utils/forms/ChannelUtils.js @@ -1,5 +1,30 @@ import API from '../../api.js'; +// Fields that users can override on an auto-synced channel. Mirrors +// `OVERRIDABLE_FIELDS` in `apps/channels/managers.py`. Keep the two in sync. +export const OVERRIDABLE_FIELDS = [ + 'name', + 'channel_number', + 'channel_group_id', + 'logo_id', + 'tvg_id', + 'tvc_guide_stationid', + 'epg_data_id', + 'stream_profile_id', +]; + +// Display labels for the override fields above. +export const OVERRIDE_FIELD_LABELS = { + name: 'Name', + channel_number: 'Channel Number', + channel_group_id: 'Channel Group', + logo_id: 'Logo', + tvg_id: 'TVG-ID', + tvc_guide_stationid: 'Gracenote Station ID', + epg_data_id: 'EPG', + stream_profile_id: 'Stream Profile', +}; + export const matchChannelEpg = (channel) => { return API.matchChannelEpg(channel.id); }; @@ -19,27 +44,165 @@ export const requeryChannels = () => { API.requeryChannels(); }; +// PATCH semantic: `override: null` deletes the override row; per-field +// nulls only clear the matching field. +export const clearChannelOverrides = (channelId) => { + return API.updateChannel({ id: channelId, override: null }); +}; + +// Coerce a form value to the backend's storage shape so equality +// comparisons against channel data don't drift on type mismatches. +export const normalizeFieldValue = (field, value) => { + if (value === '' || value === null || value === undefined || value === '-1') { + return null; + } + // The stream_profile and logo pickers encode "(use default)" as '0', + // which is semantically a null FK on the Channel row. + if ((field === 'stream_profile_id' || field === 'logo_id') && value === '0') { + return null; + } + if (field === 'channel_number') { + const n = parseFloat(value); + return Number.isFinite(n) ? n : null; + } + if ( + field === 'channel_group_id' || + field === 'logo_id' || + field === 'epg_data_id' || + field === 'stream_profile_id' + ) { + const n = parseInt(value, 10); + return Number.isFinite(n) ? n : null; + } + return value; +}; + +// Per-field form shape for FK pickers (string for popovers, raw for +// the EPG select); used by the reset-to-provider affordance. +const PROVIDER_FORM_VALUE_BUILDERS = { + channel_group_id: (channel) => + channel?.channel_group_id != null ? `${channel.channel_group_id}` : '', + stream_profile_id: (channel) => + channel?.stream_profile_id != null ? `${channel.stream_profile_id}` : '0', + logo_id: (channel) => (channel?.logo_id != null ? `${channel.logo_id}` : ''), + epg_data_id: (channel) => channel?.epg_data_id ?? '', +}; + +export const getProviderFormValue = (channel, field) => { + const builder = PROVIDER_FORM_VALUE_BUILDERS[field]; + if (builder) return builder(channel); + return channel?.[field] ?? ''; +}; + +// Form value differs from the channel's provider value. Manual +// channels always return false (no provider value to compare to). +export const isFormFieldOverridden = (channel, field, formValue) => { + if (!channel?.auto_created) return false; + const normalizedForm = normalizeFieldValue(field, formValue); + const normalizedProvider = normalizeFieldValue(field, channel[field]); + return normalizedForm !== normalizedProvider; +}; + +// Human labels for the table's overrides indicator tooltip. +export const listOverriddenFields = (channel) => { + if (!channel?.override) return []; + return OVERRIDABLE_FIELDS.filter((field) => { + const value = channel.override[field]; + return value !== null && value !== undefined; + }).map((field) => OVERRIDE_FIELD_LABELS[field]); +}; + +// "Provider: " subtext for auto-synced channels (null for manual). +export const getProviderHint = (channel, field) => { + if (!channel?.auto_created) return null; + const providerValue = channel[field]; + const display = + providerValue === null || + providerValue === undefined || + providerValue === '' + ? '(empty)' + : providerValue; + return `Provider: ${display}`; +}; + +// FK provider hint that resolves the ID to a display name via lookup. +export const getFkProviderHint = (channel, field, lookup) => { + if (!channel?.auto_created) return null; + const providerId = channel[field]; + if (providerId === null || providerId === undefined) { + return 'Provider: (none)'; + } + const entry = lookup?.[providerId]; + const display = entry?.name || entry?.tvg_id || String(providerId); + return `Provider: ${display}`; +}; + +// Build the override PATCH payload by diffing form values against +// provider values. Matching fields become null (clear); diverging +// fields carry the form value. +export const buildOverridePayload = (channel, formattedValues) => { + if (!channel) return undefined; + const payload = {}; + let anyOverride = false; + + for (const field of OVERRIDABLE_FIELDS) { + const formValue = normalizeFieldValue(field, formattedValues[field]); + const providerValue = normalizeFieldValue(field, channel[field]); + if (formValue === null && providerValue === null) continue; + if (formValue !== providerValue) { + payload[field] = formValue; + if (formValue !== null) anyOverride = true; + } else { + payload[field] = null; + } + } + + if (!anyOverride) { + // Every field matches provider; explicit null deletes the row. + return null; + } + return payload; +}; + +// Prefer the backend-resolved effective_* so the form loads with the +// overridden value; fall back to the raw field otherwise. +const effective = (channel, field) => { + if (!channel) return undefined; + const effKey = `effective_${field}`; + if (effKey in channel && channel[effKey] !== undefined) { + return channel[effKey]; + } + return channel[field]; +}; + export const getChannelFormDefaultValues = (channel, channelGroups) => { + const name = effective(channel, 'name') ?? ''; + const channelNumber = effective(channel, 'channel_number'); + const groupId = effective(channel, 'channel_group_id'); + const streamProfileId = effective(channel, 'stream_profile_id'); + const tvgId = effective(channel, 'tvg_id'); + const gracenoteId = effective(channel, 'tvc_guide_stationid'); + const epgDataId = effective(channel, 'epg_data_id'); + const logoId = effective(channel, 'logo_id'); return { - name: channel?.name || '', + name: name || '', channel_number: - channel?.channel_number !== null && channel?.channel_number !== undefined - ? channel.channel_number + channelNumber !== null && channelNumber !== undefined + ? channelNumber : '', - channel_group_id: channel?.channel_group_id - ? `${channel.channel_group_id}` + channel_group_id: groupId + ? `${groupId}` : Object.keys(channelGroups).length > 0 ? Object.keys(channelGroups)[0] : '', - stream_profile_id: channel?.stream_profile_id - ? `${channel.stream_profile_id}` - : '0', - tvg_id: channel?.tvg_id || '', - tvc_guide_stationid: channel?.tvc_guide_stationid || '', - epg_data_id: channel?.epg_data_id ?? '', - logo_id: channel?.logo_id ? `${channel.logo_id}` : '', + stream_profile_id: streamProfileId ? `${streamProfileId}` : '0', + tvg_id: tvgId || '', + tvc_guide_stationid: gracenoteId || '', + epg_data_id: epgDataId ?? '', + logo_id: logoId ? `${logoId}` : '', user_level: `${channel?.user_level ?? '0'}`, is_adult: channel?.is_adult ?? false, + hidden_from_output: channel?.hidden_from_output ?? false, }; }; @@ -70,15 +233,32 @@ export const handleEpgUpdate = async ( formattedValues, channelStreams ) => { - // If there's an EPG to set, use our enhanced endpoint + // Auto-synced channels route identity edits into the override row. Sync + // keeps writing provider values to Channel.* unmodified, so the override + // is what actually persists user changes across refreshes. `hidden_from_output` + // stays as a direct Channel field even for auto-created channels because + // it is a status flag, not a value replacement. + if (channel.auto_created) { + const overridePayload = buildOverridePayload(channel, formattedValues); + const payload = { + id: channel.id, + hidden_from_output: formattedValues.hidden_from_output, + }; + if (overridePayload !== undefined) { + payload.override = overridePayload; + } + await updateChannel(payload); + return; + } + + // Manual channels: existing behavior preserved. When the EPG has changed, + // the dedicated set-EPG endpoint triggers an EPG refresh; other field + // updates go through the regular PATCH and are skipped entirely when + // there is nothing besides epg_data_id to update. if (values.epg_data_id !== (channel.epg_data_id ?? '')) { - // Use the special endpoint to set EPG and trigger refresh await setChannelEPG(channel, values); - // Remove epg_data_id from values since we've handled it separately const { epg_data_id: _epg_data_id, ...otherValues } = formattedValues; - - // Update other channel fields if needed if (Object.keys(otherValues).length > 0) { await updateChannel({ id: channel.id, @@ -87,7 +267,6 @@ export const handleEpgUpdate = async ( }); } } else { - // No EPG change, regular update await updateChannel({ id: channel.id, ...formattedValues, diff --git a/frontend/src/utils/forms/GroupSyncUtils.js b/frontend/src/utils/forms/GroupSyncUtils.js new file mode 100644 index 00000000..64a002ac --- /dev/null +++ b/frontend/src/utils/forms/GroupSyncUtils.js @@ -0,0 +1,70 @@ +// Utilities for per-group auto-sync configuration in LiveGroupFilter / +// M3UGroupFilter. Range reservations (groups with both start and end set +// on Fixed or Provider numbering) claim channel numbers exclusively, so +// overlapping reservations across groups would produce conflicts during +// sync. Validation runs client-side at save-time to block obvious +// misconfigurations before the request is sent; the backend enforces +// the same rule across accounts as a safety net. + +const MODE_WITHOUT_RANGE = 'next_available'; + +// Returns null if the group does not participate in reservations (disabled, +// not auto-syncing, unbounded, or in Next Available mode), otherwise a +// [start, end] pair of integers with start <= end. +export const getGroupReservation = (group) => { + if (!group || !group.enabled || !group.auto_channel_sync) return null; + const mode = group.custom_properties?.channel_numbering_mode || 'fixed'; + if (mode === MODE_WITHOUT_RANGE) return null; + const endRaw = group.auto_sync_channel_end; + if (endRaw === null || endRaw === undefined || endRaw === '') return null; + const startRaw = + mode === 'provider' + ? (group.custom_properties?.channel_numbering_fallback ?? 1) + : (group.auto_sync_channel_start ?? 1); + const start = Math.floor(Number(startRaw)); + const end = Math.floor(Number(endRaw)); + if (!Number.isFinite(start) || !Number.isFinite(end)) return null; + if (end < start) return null; + return [start, end]; +}; + +// Given an array of groupStates from the M3U Group Filter modal, returns +// overlap records describing pairs of groups whose reservations intersect. +// Shape: [{ a: { channel_group, name, start, end }, b: {...} }, ...] +// Callers surface these as user-facing errors and block submit. +export const detectGroupReservationOverlaps = (groupStates) => { + const reservations = []; + for (const g of groupStates || []) { + const range = getGroupReservation(g); + if (!range) continue; + reservations.push({ + channel_group: g.channel_group, + name: g.name, + start: range[0], + end: range[1], + }); + } + const conflicts = []; + for (let i = 0; i < reservations.length; i += 1) { + for (let j = i + 1; j < reservations.length; j += 1) { + const a = reservations[i]; + const b = reservations[j]; + if (a.start <= b.end && b.start <= a.end) { + conflicts.push({ a, b }); + } + } + } + return conflicts; +}; + +// Human-readable summary of an overlap list for a notification body. +export const formatOverlapMessage = (conflicts) => { + if (!conflicts || conflicts.length === 0) return ''; + const lines = conflicts.slice(0, 5).map(({ a, b }) => { + return `"${a.name}" [${a.start}-${a.end}] overlaps "${b.name}" [${b.start}-${b.end}]`; + }); + if (conflicts.length > 5) { + lines.push(`... and ${conflicts.length - 5} more.`); + } + return lines.join('\n'); +}; diff --git a/frontend/src/utils/forms/__tests__/ChannelBatchUtils.bulkOverrideRouting.test.js b/frontend/src/utils/forms/__tests__/ChannelBatchUtils.bulkOverrideRouting.test.js new file mode 100644 index 00000000..00e4f907 --- /dev/null +++ b/frontend/src/utils/forms/__tests__/ChannelBatchUtils.bulkOverrideRouting.test.js @@ -0,0 +1,105 @@ +// @vitest-environment jsdom +import { describe, it, expect, beforeEach, vi } from 'vitest'; + +// Mock API.bulkUpdateChannels so we can assert on the body shape the +// helper sends. We're testing routing logic, not network behavior. +vi.mock('../../../api.js', () => ({ + default: { + bulkUpdateChannels: vi.fn(async (body) => ({ ok: true, body })), + updateChannels: vi.fn(), + }, +})); + +import { updateChannelsWithOverrideRouting } from '../ChannelBatchUtils.js'; +import API from '../../../api.js'; + +describe('updateChannelsWithOverrideRouting (manual finding #4)', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it('routes override-able fields to override.X for auto-created channels', async () => { + // User-reported: bulk-edit "name" on auto-channels lost the change + // on next sync because it wrote Channel.name (raw) instead of + // override.name. The Pencil indicator never appeared because no + // override row was created. + const channelsById = { + 1: { id: 1, auto_created: true }, + 2: { id: 2, auto_created: true }, + }; + await updateChannelsWithOverrideRouting( + [1, 2], + { name: 'BulkRename' }, + channelsById, + ); + + expect(API.bulkUpdateChannels).toHaveBeenCalledTimes(1); + const body = API.bulkUpdateChannels.mock.calls[0][0]; + expect(body).toEqual([ + { id: 1, override: { name: 'BulkRename' } }, + { id: 2, override: { name: 'BulkRename' } }, + ]); + }); + + it('keeps raw Channel.X writes for manual channels (auto_created=false)', async () => { + const channelsById = { + 1: { id: 1, auto_created: false }, + }; + await updateChannelsWithOverrideRouting( + [1], + { name: 'ManualRename' }, + channelsById, + ); + const body = API.bulkUpdateChannels.mock.calls[0][0]; + expect(body).toEqual([{ id: 1, name: 'ManualRename' }]); + }); + + it('splits a mixed selection: auto-created → override, manual → raw', async () => { + const channelsById = { + 1: { id: 1, auto_created: true }, + 2: { id: 2, auto_created: false }, + 3: { id: 3, auto_created: true }, + }; + await updateChannelsWithOverrideRouting( + [1, 2, 3], + { name: 'Mixed', tvg_id: 'mixed.tvg' }, + channelsById, + ); + const body = API.bulkUpdateChannels.mock.calls[0][0]; + expect(body).toEqual([ + { id: 1, override: { name: 'Mixed', tvg_id: 'mixed.tvg' } }, + { id: 2, name: 'Mixed', tvg_id: 'mixed.tvg' }, + { id: 3, override: { name: 'Mixed', tvg_id: 'mixed.tvg' } }, + ]); + }); + + it('always-raw fields (hidden_from_output, user_level, is_adult) bypass override routing', async () => { + // hidden_from_output is a status flag, not a value override - it goes to + // Channel.hidden_from_output directly even on auto-created channels. + const channelsById = { + 1: { id: 1, auto_created: true }, + }; + await updateChannelsWithOverrideRouting( + [1], + { hidden_from_output: true, name: 'Renamed' }, + channelsById, + ); + const body = API.bulkUpdateChannels.mock.calls[0][0]; + expect(body).toEqual([ + { id: 1, hidden_from_output: true, override: { name: 'Renamed' } }, + ]); + }); + + it('falls back to raw when channelsById lookup is missing the row', async () => { + // Defensive: if the channel store doesn't have the row (paginated + // off, recent creation), default behavior should not crash and + // should not silently drop the change. + await updateChannelsWithOverrideRouting( + [99], + { name: 'Defensive' }, + {}, // empty lookup + ); + const body = API.bulkUpdateChannels.mock.calls[0][0]; + expect(body).toEqual([{ id: 99, name: 'Defensive' }]); + }); +}); diff --git a/frontend/src/utils/forms/__tests__/ChannelUtils.test.js b/frontend/src/utils/forms/__tests__/ChannelUtils.test.js index 390f26cd..73c550ee 100644 --- a/frontend/src/utils/forms/__tests__/ChannelUtils.test.js +++ b/frontend/src/utils/forms/__tests__/ChannelUtils.test.js @@ -8,6 +8,12 @@ import { getChannelFormDefaultValues, getFormattedValues, handleEpgUpdate, + getProviderHint, + getFkProviderHint, + normalizeFieldValue, + buildOverridePayload, + listOverriddenFields, + clearChannelOverrides, } from '../ChannelUtils.js'; // ── API mock ─────────────────────────────────────────────────────────────────── @@ -134,6 +140,7 @@ describe('ChannelUtils', () => { logo_id: '10', user_level: '1', is_adult: false, + hidden_from_output: false, }); }); @@ -230,6 +237,7 @@ describe('ChannelUtils', () => { logo_id: '', user_level: '0', is_adult: false, + hidden_from_output: false, }); }); }); @@ -458,4 +466,336 @@ describe('ChannelUtils', () => { ).rejects.toThrow('Update error'); }); }); + + // ── normalizeFieldValue ────────────────────────────────────────────────────── + + describe('normalizeFieldValue', () => { + it('returns null for empty string', () => { + expect(normalizeFieldValue('name', '')).toBeNull(); + }); + + it('returns null for null', () => { + expect(normalizeFieldValue('name', null)).toBeNull(); + }); + + it('returns null for undefined', () => { + expect(normalizeFieldValue('name', undefined)).toBeNull(); + }); + + it('returns null for the "-1" sentinel', () => { + expect(normalizeFieldValue('name', '-1')).toBeNull(); + }); + + it('coerces channel_number string "10" to numeric 10', () => { + expect(normalizeFieldValue('channel_number', '10')).toBe(10); + }); + + it('coerces channel_number "5.5" to 5.5 (preserves decimal)', () => { + expect(normalizeFieldValue('channel_number', '5.5')).toBe(5.5); + }); + + it('coerces logo_id string "10" to integer 10', () => { + expect(normalizeFieldValue('logo_id', '10')).toBe(10); + }); + + it('coerces channel_group_id string "3" to integer 3', () => { + expect(normalizeFieldValue('channel_group_id', '3')).toBe(3); + }); + + it('coerces epg_data_id string "7" to integer 7', () => { + expect(normalizeFieldValue('epg_data_id', '7')).toBe(7); + }); + + it('coerces stream_profile_id string "2" to integer 2', () => { + expect(normalizeFieldValue('stream_profile_id', '2')).toBe(2); + }); + + it('treats stream_profile_id "0" as null (the "use default" sentinel)', () => { + expect(normalizeFieldValue('stream_profile_id', '0')).toBeNull(); + }); + + it('treats logo_id "0" as null (the "Default" picker option)', () => { + expect(normalizeFieldValue('logo_id', '0')).toBeNull(); + }); + + it('returns string field unchanged', () => { + expect(normalizeFieldValue('name', 'ESPN HD')).toBe('ESPN HD'); + }); + + it('returns null for non-numeric channel_number', () => { + expect(normalizeFieldValue('channel_number', 'abc')).toBeNull(); + }); + + it('returns null for non-integer FK id input', () => { + // parseInt('abc') is NaN, which is_not_finite, returns null. + expect(normalizeFieldValue('logo_id', 'abc')).toBeNull(); + }); + }); + + // ── normalizeFieldValue: sentinel battery ──────────────────────────────────── + // Every overridable form field gets walked through the sentinel matrix so + // future field additions inherit coverage automatically. Add the field to + // OVERRIDABLE_FIELDS in ChannelUtils.js, then add its row to the matrix + // below. + + describe('normalizeFieldValue: sentinel battery', () => { + // Sentinels common to all fields that must always normalize to null. + const universalNullSentinels = ['', null, undefined, '-1']; + + // (field, expectations) pairs for every overridable form field. + // Each row documents what the field accepts as input and what + // normalizeFieldValue must return for the canonical inputs. + const matrix = [ + { + field: 'name', + kind: 'string', + passthrough: [['ESPN HD', 'ESPN HD']], + zeroSentinelIsNull: false, + }, + { + field: 'tvg_id', + kind: 'string', + passthrough: [['hbo.us', 'hbo.us']], + zeroSentinelIsNull: false, + }, + { + field: 'tvc_guide_stationid', + kind: 'string', + passthrough: [['hbo-station', 'hbo-station']], + zeroSentinelIsNull: false, + }, + { + field: 'channel_number', + kind: 'numeric', + passthrough: [ + ['10', 10], + ['5.5', 5.5], + ['0', 0], + ], + zeroSentinelIsNull: false, + }, + { + field: 'channel_group_id', + kind: 'fk-int', + passthrough: [ + ['3', 3], + ['0', 0], + ], + zeroSentinelIsNull: false, + }, + { + field: 'epg_data_id', + kind: 'fk-int', + passthrough: [ + ['7', 7], + ['0', 0], + ], + zeroSentinelIsNull: false, + }, + { + field: 'logo_id', + kind: 'fk-int', + passthrough: [['10', 10]], + // '0' is a domain sentinel: the picker's "Default" option. + zeroSentinelIsNull: true, + }, + { + field: 'stream_profile_id', + kind: 'fk-int', + passthrough: [['2', 2]], + // '0' is a domain sentinel: "(use default)" in the form. + zeroSentinelIsNull: true, + }, + ]; + + matrix.forEach(({ field, passthrough, zeroSentinelIsNull }) => { + describe(`field: ${field}`, () => { + universalNullSentinels.forEach((sentinel) => { + it(`normalizes ${JSON.stringify(sentinel)} to null`, () => { + expect(normalizeFieldValue(field, sentinel)).toBeNull(); + }); + }); + + if (zeroSentinelIsNull) { + it(`treats "0" as the domain sentinel and returns null`, () => { + expect(normalizeFieldValue(field, '0')).toBeNull(); + }); + } + + passthrough.forEach(([input, expected]) => { + it(`coerces ${JSON.stringify(input)} to ${JSON.stringify(expected)}`, () => { + expect(normalizeFieldValue(field, input)).toBe(expected); + }); + }); + }); + }); + }); + + // ── getProviderHint / getFkProviderHint ────────────────────────────────────── + + describe('getProviderHint', () => { + it('returns null for null channel', () => { + expect(getProviderHint(null, 'name')).toBeNull(); + }); + + it('returns null for manual (non-auto) channel', () => { + const ch = makeChannel({ auto_created: false }); + expect(getProviderHint(ch, 'name')).toBeNull(); + }); + + it('returns "Provider: " for auto-created channel with value', () => { + const ch = makeChannel({ auto_created: true, name: 'ESPN' }); + expect(getProviderHint(ch, 'name')).toBe('Provider: ESPN'); + }); + + it('renders "(empty)" placeholder for null/empty provider value', () => { + const ch = makeChannel({ auto_created: true, tvg_id: null }); + expect(getProviderHint(ch, 'tvg_id')).toBe('Provider: (empty)'); + }); + + it('renders "(empty)" for empty string provider value', () => { + const ch = makeChannel({ auto_created: true, tvg_id: '' }); + expect(getProviderHint(ch, 'tvg_id')).toBe('Provider: (empty)'); + }); + }); + + describe('getFkProviderHint', () => { + it('returns null for null channel', () => { + expect(getFkProviderHint(null, 'channel_group_id', {})).toBeNull(); + }); + + it('returns null for manual channel', () => { + const ch = makeChannel({ auto_created: false }); + expect(getFkProviderHint(ch, 'channel_group_id', {})).toBeNull(); + }); + + it('returns "(none)" when provider FK id is null', () => { + const ch = makeChannel({ auto_created: true, channel_group_id: null }); + expect(getFkProviderHint(ch, 'channel_group_id', {})).toBe( + 'Provider: (none)' + ); + }); + + it('returns the lookup name when FK id resolves', () => { + const ch = makeChannel({ auto_created: true, channel_group_id: 5 }); + const lookup = { 5: { name: 'Sports' } }; + expect(getFkProviderHint(ch, 'channel_group_id', lookup)).toBe( + 'Provider: Sports' + ); + }); + + it('falls back to tvg_id when name is missing', () => { + const ch = makeChannel({ auto_created: true, epg_data_id: 9 }); + const lookup = { 9: { tvg_id: 'espn.us' } }; + expect(getFkProviderHint(ch, 'epg_data_id', lookup)).toBe( + 'Provider: espn.us' + ); + }); + + it('falls back to stringified id when lookup misses', () => { + const ch = makeChannel({ auto_created: true, logo_id: 99 }); + expect(getFkProviderHint(ch, 'logo_id', {})).toBe('Provider: 99'); + }); + }); + + // ── listOverriddenFields ───────────────────────────────────────────────────── + + describe('listOverriddenFields', () => { + it('returns [] for channel without override', () => { + const ch = makeChannel({ override: null }); + expect(listOverriddenFields(ch)).toEqual([]); + }); + + it('returns [] for null channel', () => { + expect(listOverriddenFields(null)).toEqual([]); + }); + + it('returns labels for all fields where override has a value', () => { + const ch = makeChannel({ + override: { name: 'ESPN HD', channel_number: 100, logo_id: null }, + }); + const labels = listOverriddenFields(ch); + expect(labels).toContain('Name'); + expect(labels).toContain('Channel Number'); + expect(labels).not.toContain('Logo'); + }); + }); + + // ── buildOverridePayload ───────────────────────────────────────────────────── + + describe('buildOverridePayload', () => { + it('returns undefined for null channel', () => { + expect(buildOverridePayload(null, {})).toBeUndefined(); + }); + + it('returns null when every form value matches provider', () => { + // No override needed => signal "delete the override row" via null. + const ch = makeChannel(); + const formattedValues = { + name: ch.name, + channel_number: ch.channel_number, + channel_group_id: ch.channel_group_id, + logo_id: ch.logo_id, + tvg_id: ch.tvg_id, + tvc_guide_stationid: ch.tvc_guide_stationid, + epg_data_id: ch.epg_data_id, + stream_profile_id: ch.stream_profile_id, + }; + expect(buildOverridePayload(ch, formattedValues)).toBeNull(); + }); + + it('emits the diverging field with form value, others as null (clear)', () => { + const ch = makeChannel(); + const formattedValues = { + name: 'ESPN-NEW', + channel_number: ch.channel_number, + channel_group_id: ch.channel_group_id, + logo_id: ch.logo_id, + tvg_id: ch.tvg_id, + tvc_guide_stationid: ch.tvc_guide_stationid, + epg_data_id: ch.epg_data_id, + stream_profile_id: ch.stream_profile_id, + }; + const payload = buildOverridePayload(ch, formattedValues); + expect(payload).not.toBeNull(); + expect(payload.name).toBe('ESPN-NEW'); + // Fields that match provider are emitted as null (clear that + // override field; ensures returning to provider value is cleanly + // expressed through one PATCH). + expect(payload.channel_number).toBeNull(); + expect(payload.tvg_id).toBeNull(); + }); + + it('coerces FK id "10" string to int 10 before comparing against provider 10', () => { + // Without normalization, "10" !== 10 would falsely emit logo_id as + // a divergence; the test pins normalizeFieldValue's role in the + // diff path. + const ch = makeChannel({ logo_id: 10 }); + const formattedValues = { + name: ch.name, + channel_number: ch.channel_number, + channel_group_id: ch.channel_group_id, + logo_id: '10', + tvg_id: ch.tvg_id, + tvc_guide_stationid: ch.tvc_guide_stationid, + epg_data_id: ch.epg_data_id, + stream_profile_id: ch.stream_profile_id, + }; + // No actual divergence; helper should return null. + expect(buildOverridePayload(ch, formattedValues)).toBeNull(); + }); + }); + + // ── clearChannelOverrides ──────────────────────────────────────────────────── + + describe('clearChannelOverrides', () => { + it('PATCHes the channel with override:null', () => { + API.updateChannel.mockResolvedValue({ id: 1, override: null }); + clearChannelOverrides(7); + expect(API.updateChannel).toHaveBeenCalledWith({ + id: 7, + override: null, + }); + }); + }); }); diff --git a/frontend/src/utils/forms/__tests__/GroupSyncUtils.test.js b/frontend/src/utils/forms/__tests__/GroupSyncUtils.test.js new file mode 100644 index 00000000..0c42fada --- /dev/null +++ b/frontend/src/utils/forms/__tests__/GroupSyncUtils.test.js @@ -0,0 +1,178 @@ +import { describe, it, expect } from 'vitest'; +import { + getGroupReservation, + detectGroupReservationOverlaps, + formatOverlapMessage, +} from '../GroupSyncUtils'; + +const makeGroup = (overrides = {}) => ({ + channel_group: 1, + name: 'Entertainment', + enabled: true, + auto_channel_sync: true, + auto_sync_channel_start: 100, + auto_sync_channel_end: 110, + custom_properties: { channel_numbering_mode: 'fixed' }, + ...overrides, +}); + +describe('getGroupReservation', () => { + it('returns [start, end] for a bounded Fixed group', () => { + expect(getGroupReservation(makeGroup())).toEqual([100, 110]); + }); + + it('returns null when the group is disabled', () => { + expect(getGroupReservation(makeGroup({ enabled: false }))).toBeNull(); + }); + + it('returns null when auto_channel_sync is off', () => { + expect( + getGroupReservation(makeGroup({ auto_channel_sync: false })) + ).toBeNull(); + }); + + it('returns null when end is missing (unbounded group does not reserve)', () => { + expect( + getGroupReservation(makeGroup({ auto_sync_channel_end: null })) + ).toBeNull(); + }); + + it('returns null for Next Available mode regardless of bounds', () => { + expect( + getGroupReservation( + makeGroup({ + custom_properties: { channel_numbering_mode: 'next_available' }, + }) + ) + ).toBeNull(); + }); + + it('reads fallback as Start for Provider mode', () => { + expect( + getGroupReservation( + makeGroup({ + auto_sync_channel_start: null, + auto_sync_channel_end: 250, + custom_properties: { + channel_numbering_mode: 'provider', + channel_numbering_fallback: 200, + }, + }) + ) + ).toEqual([200, 250]); + }); + + it('returns null when end < start', () => { + expect( + getGroupReservation( + makeGroup({ auto_sync_channel_start: 500, auto_sync_channel_end: 100 }) + ) + ).toBeNull(); + }); +}); + +describe('detectGroupReservationOverlaps', () => { + it('returns no conflicts when ranges are disjoint', () => { + const a = makeGroup({ + channel_group: 1, + name: 'A', + auto_sync_channel_start: 1, + auto_sync_channel_end: 100, + }); + const b = makeGroup({ + channel_group: 2, + name: 'B', + auto_sync_channel_start: 101, + auto_sync_channel_end: 200, + }); + expect(detectGroupReservationOverlaps([a, b])).toEqual([]); + }); + + it('flags a straight overlap', () => { + const a = makeGroup({ + channel_group: 1, + name: 'A', + auto_sync_channel_start: 1, + auto_sync_channel_end: 100, + }); + const b = makeGroup({ + channel_group: 2, + name: 'B', + auto_sync_channel_start: 50, + auto_sync_channel_end: 150, + }); + const conflicts = detectGroupReservationOverlaps([a, b]); + expect(conflicts).toHaveLength(1); + expect(conflicts[0].a.name).toBe('A'); + expect(conflicts[0].b.name).toBe('B'); + }); + + it('flags an enclosed range (one reservation inside another)', () => { + const outer = makeGroup({ + channel_group: 1, + name: 'Outer', + auto_sync_channel_start: 1, + auto_sync_channel_end: 1000, + }); + const inner = makeGroup({ + channel_group: 2, + name: 'Inner', + auto_sync_channel_start: 500, + auto_sync_channel_end: 600, + }); + expect(detectGroupReservationOverlaps([outer, inner])).toHaveLength(1); + }); + + it('ignores groups without reservations', () => { + const unbounded = makeGroup({ + channel_group: 1, + name: 'Unbounded', + auto_sync_channel_end: null, + }); + const next_avail = makeGroup({ + channel_group: 2, + name: 'Next', + custom_properties: { channel_numbering_mode: 'next_available' }, + }); + expect(detectGroupReservationOverlaps([unbounded, next_avail])).toEqual([]); + }); + + it('returns all pairwise overlaps when three groups collide', () => { + const groups = [1, 2, 3].map((id) => + makeGroup({ + channel_group: id, + name: `G${id}`, + auto_sync_channel_start: 1, + auto_sync_channel_end: 100, + }) + ); + expect(detectGroupReservationOverlaps(groups)).toHaveLength(3); + }); +}); + +describe('formatOverlapMessage', () => { + it('produces one line per conflict', () => { + const conflicts = [ + { + a: { name: 'A', start: 1, end: 100 }, + b: { name: 'B', start: 50, end: 150 }, + }, + ]; + const msg = formatOverlapMessage(conflicts); + expect(msg).toContain('"A" [1-100]'); + expect(msg).toContain('"B" [50-150]'); + }); + + it('truncates at five entries with a "... and N more" suffix', () => { + const conflicts = Array.from({ length: 8 }, (_, i) => ({ + a: { name: `A${i}`, start: 1, end: 10 }, + b: { name: `B${i}`, start: 5, end: 15 }, + })); + const msg = formatOverlapMessage(conflicts); + expect(msg).toContain('and 3 more'); + }); + + it('returns empty string for no conflicts', () => { + expect(formatOverlapMessage([])).toBe(''); + }); +});