Mirrors/Dispatcharr
1
0
Fork 1
mirror of https://github.com/Dispatcharr/Dispatcharr.git synced 2026-07-18 00:55:50 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

Merge remote-tracking branch 'upstream/dev' into feat/timeshift-native-v2

Browse source 
# Conflicts:
#	apps/m3u/tasks.py
This commit is contained in:
Cédric Marcoux 2026-06-10 08:42:33 +02:00
commit 19acdcc29d
62 changed files with 9218 additions and 2246 deletions
Download patch file Download diff file Expand all files Collapse all files

66
CHANGELOG.md
View file

@ -9,6 +9,56 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **Manual Server Groups for shared M3U connection limits.** The existing `ServerGroup` model and account FK are now wired into live and VOD playback. Accounts assigned to the same group share a credential-scoped Redis counter when their provider logins match (hashed fingerprint); unrelated logins in the same group keep separate counters. Enforcement uses each profile's `max_streams` - not a group-wide cap - and profiles with `max_streams=0` skip credential pooling for that profile while still rotating on their own per-profile counter. New `apps/m3u/connection_pool.py` centralizes reserve/release, profile rotation, and credential moves on profile switch. (Closes #1137) — Thanks [@Goldenfreddy0703](https://github.com/Goldenfreddy0703)
- **Server Groups manager UI** on the M3U Accounts page: create, rename, and delete groups with account counts and delete confirmation. Groups load on login via a new Zustand store and REST helpers (`/api/m3u/server-groups/`).
- **M3U account form** adds a Server Group picker (including inline “add group”), reorganized into three columns (source/auth, connection limits, sync/content), and a Manage server groups shortcut.
- **VOD profile selection** uses `pool_has_capacity_for_profile()` so grouped accounts respect shared credential limits before a profile is chosen (non-grouped accounts behave as before).
- **Live profile switches** move the shared credential counter when the new profile uses a different provider login; same-login switches leave the credential counter unchanged.
- **Credential release keys** stored at reserve time allow counters to be released even if the M3U profile row is deleted afterward.
### Changed
- **Live and VOD connection slot logic now routes through `connection_pool`.** `Channel.get_stream()`, direct `Stream.get_stream()`, VOD profile selection, and the multi-worker VOD connection manager all call `reserve_profile_slot()` / `release_profile_slot()` instead of inline Redis INCR/DECR. VOD profile selection also checks `pool_has_capacity_for_profile()` before choosing a profile.
- **Live XC upstream URLs use current credentials.** `_resolve_live_stream_url()` builds `/live/{user}/{pass}/{stream_id}.ts` from transformed account credentials and the stream's provider `stream_id`, so playback stays aligned with the active login after credential or profile changes instead of reusing a stale `stream.url` from sync.
- **Channel stream switches check pooled capacity.** `get_stream_info_for_switch()` uses `profile_available_for_channel_switch()` when targeting a specific stream, and releases a reserved slot if URL assembly fails after `get_stream()` allocated one.
- **Live proxy init reads Redis assignment once.** After `generate_stream_url()` reserves a slot, the stream handler reads `channel_stream` / `stream_profile` from Redis instead of calling `get_stream()` again, avoiding double INCR under concurrent release.
- **EPG auto-match overhaul** — matching logic moved to `apps/channels/epg_matching.py`; Celery tasks in `tasks.py` are thin wrappers.
- Single-channel auto-match is now asynchronous: the API returns `202 Accepted` and pushes the result over WebSocket (`single_channel_epg_match`), so large EPG libraries no longer hit the previous 30-second HTTP timeout.
- Progress, bulk completion, and single-channel results use `send_websocket_update` instead of `async_to_sync(channel_layer.group_send)`, so notifications work reliably under gevent-patched uWSGI and Celery workers.
- Single-channel and selected-channel auto-match always run, even when the channel already has EPG assigned; match-all (no channel IDs) still only processes channels without EPG.
- Rematching to the same EPG no longer re-saves the channel or queues program-parse tasks; only assignments that actually change are written and refreshed.
### Performance
- **EPG auto-match memory and throughput improvements.**
- Single-channel matching streams active EPG rows and keeps only the best match plus the top 20 candidates in memory; ML validates at most 21 names per channel instead of embedding the full catalog.
- Strong fuzzy matches (≥75% single channel, ≥80% bulk) skip ML entirely, avoiding a ~500MB PyTorch load when the fuzzy result is already reliable.
- Bulk matching uses a single fuzzy pass per channel instead of scanning the full catalog twice for best match and top candidates.
- Bulk exact `tvg_id` / Gracenote matching uses an in-memory index built alongside the EPG catalog (`build_epg_matching_catalog()`), giving O(1) lookups with no extra database queries.
- Bulk match apply uses batched queries (two fetches plus `bulk_update`) instead of one `EPGData.objects.get()` per matched channel.
- EPG normalization settings are cached once per matching run, avoiding repeated `CoreSettings` reads when normalizing thousands of names.
### Fixed
- **Stale `channel_stream` Redis keys after a channel stopped could skip connection accounting on retune.** On `dev`, `get_stream()` reused any existing `channel_stream` / `stream_profile` assignment without reserving a new slot. If those keys were left behind after a stop, the next tune-in could reach the provider without incrementing Redis counters. `get_stream()` now releases stale assignments when proxy metadata shows the channel is inactive, then reserves fresh slots.
- **EPG auto-match reliability fixes.**
- Memory could spike to multiple GB on large EPG sources when building a full in-memory catalog before fuzzy matching; single-channel matching now streams rows and bounds ML work to a small candidate set.
- Wrong channel assignments from global ML similarity; ML validation now checks the fuzzy best match (or top fuzzy candidates as a last resort) instead of scoring the entire catalog.
- Channel form auto-match spinner could stick after errors or early task exits; all single-channel outcomes now push a WebSocket result, and the UI clears loading state after a 3-minute timeout.
- Bulk auto-match completion no longer calls `batch-set-epg` from the WebSocket handler, which had been re-applying every match and queueing redundant `parse_programs_for_tvg_id` tasks even when assignments were unchanged.
## [0.26.0] - 2026-06-07
### Added
- **EPG logo auto-apply for XMLTV and Schedules Direct.** Channel logos are applied from `EPGData.icon_url` through a shared helper used by bulk "Set Logos from EPG", optional per-source auto-apply on refresh (`auto_apply_epg_logos` in source `custom_properties`), and a new `epg_source_id` option on the set-logos-from-epg API. Large libraries are processed in 500-channel chunks without loading every mapped row into memory.
- **Schedules Direct EPG integration.** Dispatcharr now supports Schedules Direct as a first-class EPG source type alongside XMLTV and dummy EPG, with credential auth, lineup management, and guide refresh through the existing EPG pipeline and WebSocket progress. (Closes #1246) — Thanks [@Shokkstokk](https://github.com/Shokkstokk)
- Lineup manager in EPG source settings: search by postal code, add/remove up to four active lineups, with SD's six-adds-per-24-hours limit and midnight-UTC reset surfaced in the UI.
- MD5 delta refresh skips unchanged schedule and program downloads; a two-hour minimum interval between full refreshes is enforced (bypassable via force refresh).
- Station logos in dark, light, gray, or white variants (`logo_style`), stored in `EPGData.icon_url` alongside XMLTV channel icons.
- Optional program poster fetch (off by default): configurable style preference (SD Recommended via Gracenote's `primary` flag, or portrait/landscape banner/iconic variants with fallbacks), served through a backend proxy cached by nginx on first view.
- XMLTV-compatible cast output (`role` for character names, `guest` for guest stars).
- Lightweight stations-only fetch on source creation so EPG entries exist for auto-matching before the first full schedule pull.
- **Live EPG program preview in the channel create/edit modal.** When selecting an EPG channel in the channel form, a "Current Program" card appears showing the currently-airing program title, description, and a progress bar. The backend builds a byte-offset index over the raw XMLTV file after each EPG refresh so lookups seek directly to the relevant file positions rather than parsing the full file - returning results in 1-10ms regardless of source size. If the index has not yet been built for a source, the API dispatches an async background build and the frontend retries up to 20 times over a 3-minute window. The `ProgramPreview` component was extracted into a shared component reused by the Stats page. The `programme_index` field on `EPGSource` is excluded from all API list responses to avoid returning multi-MB blobs. — Thanks [@FiveBoroughs](https://github.com/FiveBoroughs)
- **Public IP display in the sidebar is now blurred by default and reveals on click.** Prevents accidental exposure in screenshots and screen shares. A new toggle in Settings > System Settings lets users disable IP and geolocation fetching entirely. A `DISPATCHARR_ENABLE_IP_LOOKUP` environment variable provides a container-level override; when set to `false` the toggle is hidden from the UI and cannot be changed. (Closes #1302) — Thanks [@sethwv](https://github.com/sethwv)
- **Configurable per-page count and sticky pagination footer in Plugin Browse.** The pagination controls now live in a fixed footer bar at the bottom of the page. A page-size selector (9 / 18 / 27 / 36) sits alongside the pagination widget and an item range readout (`X to Y of Z`). The selected page size is persisted in `localStorage` so it survives page navigations. — Thanks [@sethwv](https://github.com/sethwv)
@ -16,18 +66,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- **`send_websocket_update` now detects Celery worker context when gevent is monkey-patched.** EPG refresh progress (including Schedules Direct) uses the shared `send_epg_update` helper instead of a duplicate synchronous Redis sender. In Celery prefork workers that inherit gevent patching, WebSocket messages are delivered synchronously via the existing `_gevent_ws_send` Redis path; uWSGI continues to use `gevent.spawn` as before.
- **Schedules Direct EPG delete confirmation shows username only.** The delete dialog no longer surfaces password or API key fields for credential-based SD sources.
- **Channel edit form reorganized into three semantic columns.** Fields are now grouped as Identity (name, number, group, logo), Guide Data (TVG-ID, Gracenote StationId, EPG picker, current program preview), and Behavior/Access (stream profile, user level, mature content, hidden). The EPG section having its own column gives the `ProgramPreview` card enough width to truncate long titles correctly rather than expanding the column.
- **IP lookup result delivered via WebSocket push.** When the background lookup completes, an `ip_lookup_complete` event is pushed to all connected clients so the sidebar IP field populates without polling. A `Skeleton` placeholder is shown while the lookup is in progress.
- **`get_host_and_port` and `build_absolute_uri_with_port` moved from `apps/output/views.py` to `core/utils.py`.** Both helpers have no dependencies on anything in `apps/output` and are now used in `apps/channels/serializers.py` as well. Moving them to `core/utils` eliminates the need for a local import inside `LogoSerializer` and makes them available to the rest of the codebase without circular-import risk. `LogoSerializer.get_cache_url()` was also updated to use `build_absolute_uri_with_port` instead of `request.build_absolute_uri()`, so logo cache URLs now correctly include non-standard ports (fixing port-stripping for logo URLs behind reverse proxies, matching the existing fix applied to M3U and EPG URLs).
- **nginx logo/poster proxy cache moved to `/data/logo_cache`.** `proxy_cache_path` now uses the persistent data volume instead of `/app/logo_cache`, and the init script ensures the directory exists with correct ownership on container start.
- **`debian_install.sh` switched from Gunicorn to uWSGI with gevent workers.** The Debian/LXC bare-metal installer now deploys Dispatcharr under the same uWSGI + gevent stack used by the Docker image, eliminating a class of compatibility differences between the two deployment paths. The installer writes a `uwsgi-debian.ini` next to the app and manages uWSGI via a systemd service. The nginx site config now uses `uwsgi_pass` + `include uwsgi_params` instead of `proxy_pass`, which correctly populates `SERVER_PORT` in the WSGI environ so M3U playlist URLs include the configured port number (fixing the port-stripping bug from #1267 for bare-metal installs). Python 3.13 is now provisioned through uv's managed runtime so the install works on Debian 12 and Ubuntu 24.04 LTS regardless of the system Python version.
- **`get_vod_streams` XC API response was missing metadata fields available from basic sync.** When a provider includes `director`, `cast`, `release_date`, `plot`, `genre`, or `year` in its `get_vod_streams` response, Dispatcharr stores those fields during the basic sync pass but was not including them in its own `get_vod_streams` XC output. The endpoint now outputs all six fields. The `trailer` key in the response also mapped to the wrong internal key (`trailer` instead of `youtube_trailer`) following the storage key rename, so the trailer field was always empty until an advanced refresh ran. Both issues are corrected. Users with existing libraries should trigger a VOD provider refresh to populate the missing fields. (Fixes #1228)
- **Docker base image now uses a multi-stage build.** The builder stage installs compilers and dev headers (`gcc`, `g++`, `gfortran`, `build-essential`, `libopenblas-dev`, `libpcre3-dev`, `python3.13-dev`, `ninja-build`) to create the virtual environment and compile the legacy NumPy wheel (cpu-baseline=none for old hardware). The final runtime image starts from the same ffmpeg base, copies only the prebuilt venv and wheel from the builder, and installs only the runtime libraries needed at runtime - eliminating compiler binaries from production containers and reducing final image size. — Thanks [@kensac](https://github.com/kensac)
- **`libpq-dev` removed from both build and runtime stages.** The project uses `psycopg[binary]` (psycopg3), which bundles its own statically linked copy of libpq inside the wheel. No system libpq headers or shared library are needed at build or runtime.
- **Database driver upgraded from `psycopg2` to `psycopg3` (`psycopg[binary]`).** psycopg3 rewrote its network I/O layer in Python, so `gevent`'s `monkey.patch_all()` makes it gevent-cooperative without any additional patching. The `psycogreen` dependency and its driver-patching block in `gevent_patch.py` have been removed. Django's native psycopg3 connection pool is explicitly disabled (`pool: False`) so `django-db-geventpool` remains in sole control of connection lifecycle. The `dropdb` management command was updated to the `psycopg` API (`psycopg.connect`, `psycopg.sql`).
- **Frontend unit tests extended to additional form components and the Guide page.** `ProgramRecordingModal`, `Recording`, `RecordingDetailsModal`, `RecurringRuleModal`, `ScheduleInput`, `SeriesRecordingModal`, `SeriesRuleEditorModal`, `Stream`, `StreamProfile`, `SuperuserForm`, `User`, `UserAgent`, and `VODCategoryFilter` now have Vitest + Testing Library test suites. Business logic was extracted from these components into corresponding utility modules and is exercised through the new tests. The Guide page was similarly refactored with extracted utils, and `dateTimeUtils.js` was extended to support the new test coverage. — Thanks [@nick4810](https://github.com/nick4810)
- **Official plugin repository manifest URL moved to GitHub Pages.** The default `OFFICIAL_REPO_URL` now points to `https://dispatcharr.github.io/Plugins/manifest.json` instead of the raw GitHub content URL. GitHub Pages avoids opaque caching on raw content that could serve stale manifests without indication. A data migration updates existing `PluginRepo` rows marked `is_official=True` in place; fresh installs pick up the new URL from the model default after the seed migration runs. — Thanks [@sethwv](https://github.com/sethwv)
### Performance
- **M3U and EPG channel logo URLs no longer call `build_absolute_uri_with_port()` per channel.** The host/port/scheme base URL and logo path prefix/suffix are computed once per request; each row appends only the logo ID. M3U proxy stream URLs use the same pattern for channel UUIDs.
- **`get_vod_streams` and `get_series` XC API response times reduced significantly for large libraries.** Both endpoints previously loaded all active relations per title (one row per account that carries the same movie/series), then discarded duplicates in Python. With large libraries across multiple active accounts this fetched a multiple of N rows. Both queries now use a single `DISTINCT ON (movie_id/series_id)` pass over the relation table ordered by account priority, returning exactly one row per title. Logo URL construction (`reverse()` + `build_absolute_uri_with_port()`) is also computed once and reused via prefix/suffix string concatenation, matching the existing pattern in `get_live_streams`. The `m3u_account` JOIN in `get_series` was also removed (it was fetched but never read in the loop). Additionally, `get_series` previously had no `order_by` on `M3USeriesRelation`, so output order was non-deterministic (physical storage order). Both endpoints now sort alphabetically by title. Observed improvements: 18s → 12s for 48k movies; 9.5s → 3.5s for 11k series.
- **Database connections are now managed by a persistent per-worker pool.** Previously each request opened a fresh TCP connection to PostgreSQL, paid a full authentication handshake, and closed the connection at request end. `django-db-geventpool` now maintains a pool of warm connections per uWSGI worker; requests borrow a connection and return it when done, eliminating connection-setup overhead on every request. Pool size is bounded (`MAX_CONNS=8` per worker, `REUSE_CONNS=3` warm connections kept idle) to stay comfortably within PostgreSQL's default `max_connections=100` across all uWSGI workers, Celery workers, and Daphne. — Thanks [@JCBird1012](https://github.com/JCBird1012)
- **Reduced Redis round-trips on the Stats page channel status endpoint.** `get_basic_channel_info` was making up to 6 individual `HGET` calls per connected client plus a redundant `HGET` for `TOTAL_BYTES` (already present in the preceding `HGETALL` result). Client metadata is now fetched with a single `HMGET` per client, and `TOTAL_BYTES` is read from the already-fetched hash. Under load with many active streams this significantly reduces the time each uWSGI worker holds the GIL servicing the stats endpoint, reducing the chance of concurrent requests from other pages timing out with a 503. The same `HGET`-to-`HMGET` consolidation was applied to `stream_ts` and `get_user_active_connections`. The Stats page frontend was also fixed to fire the initial fetch only once on mount (previously two `useEffect` hooks both triggered an immediate fetch on load). — Thanks [@JCBird1012](https://github.com/JCBird1012)
@ -35,9 +90,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Security
- **M3U endpoint no longer reflects POST body content in error responses.** The error message for disallowed POST requests previously echoed the raw request body back to the caller in a `text/html` response, which could be used for reflected XSS. The body is no longer included in the response. - Thanks [@sebastiondev](https://github.com/sebastiondev)
- Updated frontend npm dependencies to resolve 5 audit vulnerabilities (2 moderate, 2 high, 1 critical):
- Updated `react-router` and `react-router-dom` 7.13.0 → 7.17.0, resolving **high** unauthenticated RCE via turbo-stream TYPE_ERROR deserialization ([GHSA-49rj-9fvp-4h2h](https://github.com/advisories/GHSA-49rj-9fvp-4h2h)), **high** open redirect via protocol-relative `//` paths ([GHSA-2j2x-hqr9-3h42](https://github.com/advisories/GHSA-2j2x-hqr9-3h42)), **high** XSS in unstable RSC redirect handling via `javascript:` targets ([GHSA-8646-j5j9-6r62](https://github.com/advisories/GHSA-8646-j5j9-6r62)), **high** stored XSS via unescaped `Location` header in prerendered redirect HTML ([GHSA-f22v-gfqf-p8f3](https://github.com/advisories/GHSA-f22v-gfqf-p8f3)), **high** DoS via unbounded path expansion in the `__manifest` endpoint ([GHSA-8x6r-g9mw-2r78](https://github.com/advisories/GHSA-8x6r-g9mw-2r78)), and **high** DoS via reflected user input in single-fetch ([GHSA-rxv8-25v2-qmq8](https://github.com/advisories/GHSA-rxv8-25v2-qmq8))
- Updated `vitest` 3.2.4 → 4.1.8, resolving **critical** arbitrary file read and execution when the Vitest UI server is listening ([GHSA-5xrq-8626-4rwp](https://github.com/advisories/GHSA-5xrq-8626-4rwp))
- Updated `brace-expansion` 5.0.5 → 5.0.6, resolving **moderate** DoS when large numeric ranges defeat the documented `max` limit ([GHSA-jxxr-4gwj-5jf2](https://github.com/advisories/GHSA-jxxr-4gwj-5jf2))
- Updated `ws` 8.19.0 → 8.21.0, resolving **moderate** uninitialized memory disclosure ([GHSA-58qx-3vcg-4xpx](https://github.com/advisories/GHSA-58qx-3vcg-4xpx))
### Fixed
- **HDHR lineup and discovery URLs now use the shared port-aware URI builder.** `lineup.json`, `discover.json`, and `device.xml` previously called Django's `request.build_absolute_uri()`, which drops non-standard ports behind reverse proxies and on dev installs. They now use `build_absolute_uri_with_port()` from `core/utils.py`, matching M3U, EPG, XC, and logo cache URLs.
- **EPG channel list did not refresh after a source finished parsing channels.** The `epg_refresh` WebSocket handler closed over a stale `epgs` snapshot from when the socket connected. When a newly created source was not found in that snapshot, the handler updated progress and returned early without calling `fetchEPGData()`, so the channel picker and EPG assignment UI stayed empty until a full page reload. The handler now reads the current store via `getState()` and still calls `fetchEPGData()` when `parsing_channels` completes.
- **DVR recordings no longer stop immediately when FFmpeg exits mid-stream.** If FFmpeg crashes, stalls, or loses the source before the scheduled end time, `run_recording` now restarts it in-process instead of going straight to HLS→MKV concat and marking the recording complete. Each restart reuses the same HLS working directory and continues segment numbering (`append_list`, `-start_number`) so the final MKV is a single continuous file. Retries are bounded by a per-outage time window matching live-proxy client tolerance (`STREAM_TIMEOUT` + `FAILOVER_GRACE_PERIOD`, default 80s); the window resets whenever new segments resume, so a long recording can survive multiple separate outages. Reconnect pacing between attempts follows the same `min(0.25 × attempt, 3s)` backoff used by the live-proxy `StreamManager`. Input demuxing also uses `-err_detect ignore_err` to tolerate minor TS corruption without aborting the process. If the outage window expires without recovery, the recording is saved as `interrupted` with `ffmpeg_outage_window_exhausted` rather than falsely reported as `completed`. (Closes #1170)
- **DVR HLS→MKV concat now tolerates timestamp splices and corrupt segments.** The finalize step (and startup recovery concat) previously used a bare `ffmpeg -f concat -c copy`, which could fail on truncated tail segments or PTS discontinuities from FFmpeg restarts mid-recording. Concat now uses a shared `_dvr_build_hls_concat_cmd` helper with `-fflags +genpts+igndts+discardcorrupt`, `-err_detect ignore_err`, and `-avoid_negative_ts make_zero`; the existing MP4-intermediate fallback path uses the same tolerant input flags.
- **DVR FFmpeg restart no longer skips HLS segment numbers.** On restart, `-start_number` was set to the existing segment count while `append_list` also incremented the internal sequence for each segment reloaded from `index.m3u8`, so the first post-restart segment could land at roughly double the expected index (e.g. `seg_00028.ts` after 14 segments). Restarts now pass `-start_number 0` when the playlist already lists segments and let `append_list` continue numbering; loose `.ts` files without a playlist still seed from the highest filename index + 1.
- **Null TS keepalive packets during channel initialization caused Emby (and Jellyfin) to force a deinterlace transcode or fail playback entirely.** While waiting for a channel to become ready, the generator sent a null TS packet (PID `0x1FFF`) every 0.5 s to keep the HTTP connection alive. Emby probes the initial bytes of the response via ffprobe; receiving a null packet instead of real stream data produced incorrect codec metadata and triggered a forced transcode. The null-packet yield has been replaced with a plain `gevent.sleep(0.1)`. (Fixes #1280)
- **VOD proxy stream URLs survive M3U import refresh cycles.** When `process_movie_batch` / `process_series_batch` create duplicate content records during a refresh, existing `M3UMovieRelation` / `M3UEpisodeRelation` rows are repointed at the new records, orphaning the old UUIDs. External players (Emby, Jellyfin, ChannelsDVR) that cached `.strm` URLs with the dead UUID would then 404 even though the same request carried a stable `stream_id` that maps to an active relation. The proxy now falls back to stream_id resolution when the UUID lookup misses, using strictest-match-first logic (account-scoped relation preferred, then highest-priority account). A `[STREAMID-FALLBACK]` WARNING log keeps the underlying import churn. — Thanks [@R3XCHRIS](https://github.com/R3XCHRIS)
- **IP lookup no longer blocks the settings page load.** The `environment` endpoint previously made up to three sequential HTTP calls (ipify, ipapi.co, ip-api.com) with 5s timeouts each, blocking the page for up to 15s if any were unreachable. The lookups now run in a background thread on first request and results are cached in Redis for 1 hour, so the endpoint returns immediately on every call. — Thanks [@sethwv](https://github.com/sethwv)
@ -51,6 +116,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Restoring a backup from an older version left the database with missing schema.** The restore task ran `pg_restore` which replaced the entire database (including the `django_migrations` table) but did not run migrations afterward. If the backup predated a schema migration, the restored database was missing tables and columns added by those migrations, causing 500 errors on every API call. `migrate --noinput` now runs automatically after every restore. The success notification also now recommends a restart to clear stale service state.
- **Cast and actors lists were silently truncated to the first name.** When a provider returns `cast` or `actors` as a JSON array, the helper used during both movie and series basic sync and movie advanced refresh only extracted the first element. All names in the array are now joined into a comma-separated string. Providers that return cast as a plain string are unaffected.
- **Channel Group Override interactions in compact numbering and override display.** Two related bugs surfaced when Channel Group Override was used with auto-synced channels. First, compact numbering silently failed: with an override on the source `ChannelGroupM3UAccount`, sync stores channels under the override target group's id (not the source group id), so hide/unhide/repack operations all missed their channels and slot accounting broke silently (hidden channels kept their numbers, unhides got none, repack saw zero channels). The compact paths now include an override-aware fallback to match channels under both source and override-target group ids. Second, the clear-override reset button disappeared when an override's stored value coincidentally matched the provider value. The frontend `isFormFieldOverridden` function was value-based only; it now also checks for a persisted override row regardless of value, so the reset button remains available to clear any override. (Fixes #1263, Fixes #1276) — Thanks [@CodeBormen](https://github.com/CodeBormen)
- **Compact numbering repack is now idempotent.** Auto-synced channel numbers reshuffled within their configured range on every sync, even when the provider returned no changes. The compact repack queried channels with no `ORDER BY`, so packing followed PostgreSQL's physical row order, which drifts after each repack's UPDATEs and autovacuum. The channel query now uses `.order_by("id")` (creation order tracks provider stream order for the default "provider" sort), and explicit name / tvg_id / updated_at sorts carry `c.id` as a secondary tiebreaker so equal values keep a stable relative order instead of churning. (Fixes #1321) — Thanks [@CodeBormen](https://github.com/CodeBormen)
## [0.25.1] - 2026-05-23

89
apps/channels/api_views.py
View file

@ -1525,32 +1525,52 @@ class ChannelViewSet(viewsets.ModelViewSet):
@action(detail=False, methods=["post"], url_path="set-logos-from-epg")
def set_logos_from_epg(self, request):
"""
Trigger a Celery task to set channel logos from EPG data
Trigger a Celery task to set channel logos from EPG data.
Provide channel_ids or epg_source_id (not both).
"""
from .tasks import set_channels_logos_from_epg
data = request.data
channel_ids = data.get("channel_ids", [])
channel_ids = data.get("channel_ids")
epg_source_id = data.get("epg_source_id")
if not channel_ids:
if channel_ids and epg_source_id:
return Response(
{"error": "channel_ids is required"},
{"error": "Provide either channel_ids or epg_source_id, not both"},
status=status.HTTP_400_BAD_REQUEST,
)
if not isinstance(channel_ids, list):
if not channel_ids and not epg_source_id:
return Response(
{"error": "channel_ids must be a list"},
{"error": "channel_ids or epg_source_id is required"},
status=status.HTTP_400_BAD_REQUEST,
)
# Start the Celery task
task = set_channels_logos_from_epg.delay(channel_ids)
if channel_ids is not None:
if not isinstance(channel_ids, list):
return Response(
{"error": "channel_ids must be a list"},
status=status.HTTP_400_BAD_REQUEST,
)
if not channel_ids:
return Response(
{"error": "channel_ids cannot be empty"},
status=status.HTTP_400_BAD_REQUEST,
)
task = set_channels_logos_from_epg.delay(channel_ids=channel_ids)
channel_count = len(channel_ids)
else:
from .utils import channels_with_epg_icon_queryset
task = set_channels_logos_from_epg.delay(epg_source_id=epg_source_id)
channel_count = channels_with_epg_icon_queryset(
epg_source_id=epg_source_id,
).count()
return Response({
"message": f"Started EPG logo setting task for {len(channel_ids)} channels",
"message": f"Started EPG logo setting task for {channel_count} channels",
"task_id": task.id,
"channel_count": len(channel_ids)
"channel_count": channel_count,
})
@action(detail=False, methods=["post"], url_path="set-tvg-ids-from-epg")
@ -2067,7 +2087,7 @@ class ChannelViewSet(viewsets.ModelViewSet):
fields={
'channel_ids': serializers.ListField(
child=serializers.IntegerField(),
help_text='List of channel IDs to process. If empty or not provided, all channels without EPG will be processed.',
help_text='List of channel IDs to process (includes channels that already have EPG). If empty or not provided, only channels without EPG are processed.',
required=False,
)
}
@ -2100,23 +2120,15 @@ class ChannelViewSet(viewsets.ModelViewSet):
def match_channel_epg(self, request, pk=None):
channel = self.get_object()
# Import the matching logic
from apps.channels.tasks import match_single_channel_epg
try:
# Try to match this specific channel - call synchronously for immediate response
result = match_single_channel_epg.apply_async(args=[channel.id]).get(timeout=30)
# Refresh the channel from DB to get any updates
channel.refresh_from_db()
return Response({
"message": result.get("message", "Channel matching completed"),
"matched": result.get("matched", False),
"channel": self.get_serializer(channel).data
})
except Exception as e:
return Response({"error": str(e)}, status=400)
match_single_channel_epg.delay(channel.id)
return Response(
{
"message": f"EPG matching started for channel '{channel.name}'",
"accepted": True,
"channel_id": channel.id,
},
status=status.HTTP_202_ACCEPTED,
)
# ─────────────────────────────────────────────────────────
# 7) Set EPG and Refresh
@ -2301,7 +2313,6 @@ class ChannelViewSet(viewsets.ModelViewSet):
# Extract channel IDs upfront
channel_updates = {}
unique_epg_ids = set()
for assoc in associations:
channel_id = assoc.get("channel_id")
@ -2311,24 +2322,28 @@ class ChannelViewSet(viewsets.ModelViewSet):
continue
channel_updates[channel_id] = epg_data_id
if epg_data_id:
unique_epg_ids.add(epg_data_id)
# Batch fetch all channels (single query)
channels_dict = {
c.id: c for c in Channel.objects.filter(id__in=channel_updates.keys())
}
# Collect channels to update
# Collect channels whose EPG assignment actually changes
channels_to_update = []
changed_epg_ids = set()
for channel_id, epg_data_id in channel_updates.items():
if channel_id not in channels_dict:
logger.error(f"Channel with ID {channel_id} not found")
continue
channel = channels_dict[channel_id]
if channel.epg_data_id == epg_data_id:
continue
channel.epg_data_id = epg_data_id
channels_to_update.append(channel)
if epg_data_id:
changed_epg_ids.add(epg_data_id)
# Bulk update all channels (single query)
if channels_to_update:
@ -2341,25 +2356,25 @@ class ChannelViewSet(viewsets.ModelViewSet):
channels_updated = len(channels_to_update)
# Trigger program refresh for unique EPG data IDs (skip dummy EPGs)
# Trigger program refresh only for EPG ids newly assigned (skip dummy/SD)
from apps.epg.tasks import parse_programs_for_tvg_id
from apps.epg.models import EPGData
# Batch fetch EPG data (single query)
epg_data_dict = {
epg.id: epg
for epg in EPGData.objects.filter(id__in=unique_epg_ids).select_related('epg_source')
for epg in EPGData.objects.filter(id__in=changed_epg_ids).select_related('epg_source')
}
programs_refreshed = 0
for epg_id in unique_epg_ids:
for epg_id in changed_epg_ids:
epg_data = epg_data_dict.get(epg_id)
if not epg_data:
logger.error(f"EPGData with ID {epg_id} not found")
continue
# Only refresh non-dummy EPG sources
if epg_data.epg_source.source_type != 'dummy':
source_type = epg_data.epg_source.source_type if epg_data.epg_source else None
if source_type not in ('dummy', 'schedules_direct'):
parse_programs_for_tvg_id.delay(epg_id)
programs_refreshed += 1

18
apps/channels/compact_numbering.py
View file

@ -324,12 +324,17 @@ def _repack_inner(group_relation):
# into the SAME target group, their channels are indistinguishable
# here (channels carry no source-group back-reference), so each repack
# renumbers the shared target's channels into its own range.
# order_by("id") makes the pack deterministic. Without it the query
# returns rows in unspecified physical order, which shifts after the
# renumber's own UPDATEs and autovacuum, so the default "provider" sort
# below would repack channels into different numbers on every sync.
# id order is creation order, which tracks the provider stream order.
channels = list(
Channel.objects.filter(
auto_created=True,
auto_created_by_id=account_id,
channel_group_id__in=group_ids,
).select_related("override")
).select_related("override").order_by("id")
)
visible = []
@ -344,21 +349,22 @@ def _repack_inner(group_relation):
visible.append(ch)
# Sort the visible set by the group's configured channel_sort_order.
# Provider order (the default) preserves DB-iteration order which is
# roughly creation order; treat unrecognized values the same way.
# Provider order (the default) keeps the id order from the query above.
# Each explicit sort carries c.id as a secondary key so equal values
# (e.g. blank tvg_id) break ties deterministically instead of churning.
if sort_order == "name":
visible.sort(
key=lambda c: natural_sort_key(c.name or ""),
key=lambda c: (natural_sort_key(c.name or ""), c.id),
reverse=sort_reverse,
)
elif sort_order == "tvg_id":
visible.sort(
key=lambda c: c.tvg_id or "",
key=lambda c: (c.tvg_id or "", c.id),
reverse=sort_reverse,
)
elif sort_order == "updated_at":
visible.sort(
key=lambda c: c.updated_at,
key=lambda c: (c.updated_at, c.id),
reverse=sort_reverse,
)

937
apps/channels/epg_matching.py Normal file
View file

@ -0,0 +1,937 @@
"""
EPG channel matching: fuzzy scoring, optional ML validation, and UI notifications.
Celery tasks in tasks.py call into this module; keep orchestration here and
task wiring thin so matching logic stays testable without a worker.
"""
import gc
import heapq
import logging
import os
import re
from rapidfuzz import fuzz
from apps.epg.models import EPGData
from core.models import CoreSettings
from core.utils import send_websocket_update
logger = logging.getLogger(__name__)
_ml_model_cache = {'sentence_transformer': None}
_normalize_settings_cache = None
ML_CANDIDATE_LIMIT = 20
SINGLE_CHANNEL_MATCH_TIMEOUT_MS = 180_000
COMMON_EXTRANEOUS_WORDS = [
"tv", "channel", "network", "television",
"east", "west", "hd", "uhd", "24/7",
"1080p", "720p", "540p", "480p",
"film", "movie", "movies",
]
def release_ml_models():
"""Unload sentence transformer and encourage PyTorch to release memory."""
if _ml_model_cache['sentence_transformer'] is None:
return
logger.info("Cleaning up ML models from memory")
model = _ml_model_cache['sentence_transformer']
_ml_model_cache['sentence_transformer'] = None
del model
try:
import torch
if hasattr(torch, 'cuda') and torch.cuda.is_available():
torch.cuda.empty_cache()
except ImportError:
pass
gc.collect()
def clear_normalize_settings_cache():
"""Reset cached normalization settings after a matching run."""
global _normalize_settings_cache
_normalize_settings_cache = None
def cleanup_after_matching():
"""Release ML models and normalization cache after a matching run."""
release_ml_models()
clear_normalize_settings_cache()
def get_sentence_transformer():
"""Lazy load the sentence transformer model only when needed."""
if _ml_model_cache['sentence_transformer'] is None:
try:
from sentence_transformers import SentenceTransformer
from sentence_transformers import util
model_name = "sentence-transformers/all-MiniLM-L6-v2"
cache_dir = "/data/models"
disable_downloads = os.environ.get('DISABLE_ML_DOWNLOADS', 'false').lower() == 'true'
if disable_downloads:
hf_model_path = os.path.join(cache_dir, f"models--{model_name.replace('/', '--')}")
if not os.path.exists(hf_model_path):
logger.warning(
"ML model not found and downloads disabled (DISABLE_ML_DOWNLOADS=true). "
"Skipping ML matching."
)
return None, None
os.makedirs(cache_dir, exist_ok=True)
logger.info(f"Loading sentence transformer model (cache: {cache_dir})")
_ml_model_cache['sentence_transformer'] = SentenceTransformer(
model_name,
cache_folder=cache_dir,
)
return _ml_model_cache['sentence_transformer'], util
except ImportError:
logger.warning("sentence-transformers not available - ML-enhanced matching disabled")
return None, None
except Exception as e:
logger.error(f"Failed to load sentence transformer: {e}")
return None, None
from sentence_transformers import util
return _ml_model_cache['sentence_transformer'], util
def normalize_name(name: str) -> str:
"""Normalize a channel/EPG name for fuzzy matching."""
if not name:
return ""
global _normalize_settings_cache
if _normalize_settings_cache is None:
prefixes = []
suffixes = []
custom_strings = []
try:
settings = CoreSettings.get_epg_settings()
mode = settings.get("epg_match_mode", "default")
if mode == "advanced":
prefixes = settings.get("epg_match_ignore_prefixes", [])
suffixes = settings.get("epg_match_ignore_suffixes", [])
custom_strings = settings.get("epg_match_ignore_custom", [])
if not isinstance(prefixes, list):
prefixes = []
if not isinstance(suffixes, list):
suffixes = []
if not isinstance(custom_strings, list):
custom_strings = []
except Exception as e:
logger.debug(f"Could not load EPG matching settings: {e}")
_normalize_settings_cache = (prefixes, suffixes, custom_strings)
prefixes, suffixes, custom_strings = _normalize_settings_cache
result = name
for prefix in prefixes:
if not prefix or not isinstance(prefix, str):
continue
if result.startswith(prefix):
result = result[len(prefix):]
break
for suffix in suffixes:
if not suffix or not isinstance(suffix, str):
continue
if result.endswith(suffix):
result = result[:-len(suffix)]
break
for custom in custom_strings:
if not custom or not isinstance(custom, str):
continue
try:
result = result.replace(custom, "")
except Exception as e:
logger.debug(f"Failed to remove custom string '{custom}': {e}")
norm = result.lower()
norm = re.sub(r"\[.*?\]", "", norm)
call_sign_match = re.search(r"\(([A-Z]{3,5})\)", name)
preserved_call_sign = ""
if call_sign_match:
preserved_call_sign = " " + call_sign_match.group(1).lower()
norm = re.sub(r"\(.*?\)", "", norm)
norm = norm + preserved_call_sign
norm = re.sub(r"[^\w\s]", "", norm)
tokens = [t for t in norm.split() if t not in COMMON_EXTRANEOUS_WORDS]
return " ".join(tokens).strip()
def send_epg_matching_progress(total_channels, matched_channels, current_channel_name="", stage="matching"):
"""Send bulk EPG matching progress via WebSocket."""
matched_count = (
len(matched_channels) if isinstance(matched_channels, list) else matched_channels
)
send_websocket_update(
'updates',
'update',
{
'type': 'epg_matching_progress',
'total': total_channels,
'matched': matched_count,
'remaining': total_channels - matched_count,
'current_channel': current_channel_name,
'stage': stage,
'progress_percent': round(matched_count / total_channels * 100, 1) if total_channels > 0 else 0,
},
)
def send_single_channel_epg_match_result(channel_id, matched, message, channel=None, epg_data=None):
"""Notify the UI that a single-channel EPG match attempt has finished."""
try:
from apps.channels.serializers import ChannelSerializer
payload = {
"type": "single_channel_epg_match",
"channel_id": channel_id,
"matched": matched,
"message": message,
}
if channel is not None:
payload["channel"] = ChannelSerializer(channel).data
if epg_data is not None:
payload["epg_id"] = epg_data.id
payload["epg_name"] = epg_data.name
send_websocket_update('updates', 'update', payload)
except Exception as e:
logger.warning(f"Failed to send single channel EPG match result: {e}")
def _compute_fuzzy_score(chan_norm, row, region_code=None):
"""Compute fuzzy match score with optional region bonus/penalty."""
if not row.get("norm_name"):
return 0
base_score = fuzz.ratio(chan_norm, row["norm_name"])
bonus = 0
if region_code and row.get("tvg_id"):
combined_text = row["tvg_id"].lower() + " " + row["name"].lower()
dot_regions = re.findall(r'\.([a-z]{2})', combined_text)
if dot_regions:
bonus = 15 if region_code in dot_regions else -15
elif region_code in combined_text:
bonus = 10
return base_score + bonus
def _ml_cosine_similarities(st_model, util, query_text, candidate_texts):
"""Encode only the query plus candidate texts (not the full EPG database)."""
if not candidate_texts:
return []
texts = [query_text] + list(candidate_texts)
embeddings = st_model.encode(texts, convert_to_tensor=True, show_progress_bar=False)
sim_scores = util.cos_sim(embeddings[0:1], embeddings[1:])[0]
return [float(s) for s in sim_scores]
def _active_epg_lookup_queryset():
"""Lightweight queryset for exact EPG lookups (includes nameless entries)."""
return (
EPGData.objects
.filter(epg_source__is_active=True)
.values('id', 'tvg_id', 'name', 'epg_source_id', 'epg_source__priority')
)
def _active_epg_fuzzy_queryset():
"""Lightweight queryset for fuzzy EPG matching (requires a display name)."""
return (
_active_epg_lookup_queryset()
.filter(name__isnull=False)
.exclude(name='')
)
def _row_from_epg_values(values_row):
tvg_id = values_row.get('tvg_id') or ''
normalized_tvg_id = tvg_id.strip().lower() if tvg_id else ''
return {
'id': values_row['id'],
'tvg_id': normalized_tvg_id,
'original_tvg_id': tvg_id,
'name': values_row['name'],
'epg_source_id': values_row['epg_source_id'],
'epg_source_priority': values_row.get('epg_source__priority') or 0,
}
def lookup_epg_by_tvg_id(tvg_id):
"""Exact tvg_id lookup without loading the full EPG catalog into memory."""
if not tvg_id:
return None
values_row = _active_epg_lookup_queryset().filter(tvg_id__iexact=tvg_id.strip()).first()
return _row_from_epg_values(values_row) if values_row else None
def build_epg_matching_catalog():
"""
Build the in-memory EPG catalog for bulk matching using a streaming DB cursor.
Returns (epg_data, tvg_id_index): the full catalog plus an O(1) in-memory
tvg_id lookup table (no extra DB queries). The index prefers the first entry
per tvg_id after priority sorting.
"""
epg_data = []
for values_row in _active_epg_fuzzy_queryset().iterator(chunk_size=500):
row = _row_from_epg_values(values_row)
row['norm_name'] = normalize_name(row['name'])
epg_data.append(row)
epg_data.sort(key=lambda x: x['epg_source_priority'], reverse=True)
return epg_data, build_epg_tvg_id_index(epg_data)
def build_epg_tvg_id_index(epg_data):
"""
Build an in-memory tvg_id -> row index from an EPG catalog (no DB queries).
epg_data must be sorted by source priority (highest first) so the first
entry wins when multiple sources share the same tvg_id.
"""
index = {}
for row in epg_data:
tvg_id = row.get("tvg_id")
if tvg_id and tvg_id not in index:
index[tvg_id] = row
return index
def _dispatch_program_parse_for_epg_assignments(changed_associations):
"""
Queue parse_programs once per unique EPG id newly assigned to a channel.
bulk_update bypasses post_save, so callers must invoke this when epg_data
actually changes (mirrors the M3U sync path).
"""
if not changed_associations:
return 0
from apps.epg.tasks import parse_programs_for_tvg_id
epg_ids = {
assoc["epg_data_id"]
for assoc in changed_associations
if assoc.get("epg_data_id")
}
if not epg_ids:
return 0
dispatched = 0
for epg in EPGData.objects.filter(id__in=epg_ids).select_related("epg_source"):
source_type = epg.epg_source.source_type if epg.epg_source else None
if source_type in ("dummy", "schedules_direct"):
continue
parse_programs_for_tvg_id.delay(epg.id)
dispatched += 1
return dispatched
def _log_unchanged_epg_assignment(chan, epg_id, epg_name, epg_tvg_id, match_method):
chan_name = chan.get("name") or f"id={chan['id']}"
chan_tvg = chan.get("original_tvg_id") or chan.get("tvg_id") or ""
logger.debug(
f"Channel '{chan_name}' (id={chan['id']}, tvg_id={chan_tvg!r}) "
f"unchanged - already on EPG '{epg_name or '?'}' "
f"(id={epg_id}, tvg_id={(epg_tvg_id or '?')!r}, via {match_method})"
)
def _record_epg_match(
chan,
epg_id,
*,
epg_name,
epg_tvg_id,
match_method,
channels_to_update,
matched_channels,
unchanged_channels,
):
"""Record a match result; skip channels_to_update when assignment is already correct."""
if chan.get("current_epg_data_id") == epg_id:
unchanged_channels.append((chan["id"], chan.get("name") or "", epg_tvg_id or ""))
_log_unchanged_epg_assignment(chan, epg_id, epg_name, epg_tvg_id, match_method)
return
chan_name = chan.get("name") or f"id={chan['id']}"
chan_tvg = chan.get("original_tvg_id") or chan.get("tvg_id") or ""
fallback_name = chan.get("fallback_name") or chan_name
chan["epg_data_id"] = epg_id
channels_to_update.append(chan)
matched_channels.append((chan["id"], fallback_name, epg_tvg_id or ""))
logger.info(
f"Channel '{chan_name}' (id={chan['id']}, tvg_id={chan_tvg!r}) "
f"=> EPG '{epg_name or '?'}' (id={epg_id}, tvg_id={(epg_tvg_id or '?')!r}, via {match_method})"
)
def apply_matched_epg_to_channels(channels_to_update_dicts):
"""
Assign matched EPG rows to channels using two DB queries (channels + EPG).
Skips channels that already have the matched EPG. Returns association dicts
for channels whose epg_data assignment actually changed, and dispatches
program-parse tasks only for those new assignments.
"""
from apps.channels.models import Channel
if not channels_to_update_dicts:
return []
channel_ids = [d["id"] for d in channels_to_update_dicts]
epg_mapping = {d["id"]: d["epg_data_id"] for d in channels_to_update_dicts}
epg_ids = {epg_id for epg_id in epg_mapping.values() if epg_id}
epg_by_id = {epg.id: epg for epg in EPGData.objects.filter(id__in=epg_ids)}
channels_list = list(Channel.objects.filter(id__in=channel_ids))
changed_associations = []
channels_to_bulk = []
for channel_obj in channels_list:
epg_data_id = epg_mapping.get(channel_obj.id)
if not epg_data_id:
continue
if channel_obj.epg_data_id == epg_data_id:
epg_row = epg_by_id.get(epg_data_id)
_log_unchanged_epg_assignment(
{
"id": channel_obj.id,
"name": channel_obj.name,
"original_tvg_id": channel_obj.tvg_id,
},
epg_data_id,
epg_row.name if epg_row else None,
epg_row.tvg_id if epg_row else None,
"apply",
)
continue
epg_data_obj = epg_by_id.get(epg_data_id)
if epg_data_obj:
channel_obj.epg_data = epg_data_obj
channels_to_bulk.append(channel_obj)
changed_associations.append(
{"channel_id": channel_obj.id, "epg_data_id": epg_data_id}
)
else:
logger.error(f"EPG data {epg_data_id} not found for channel {channel_obj.id}")
if channels_to_bulk:
Channel.objects.bulk_update(channels_to_bulk, ["epg_data"])
parse_dispatched = _dispatch_program_parse_for_epg_assignments(changed_associations)
if parse_dispatched:
logger.info(
f"Dispatched {parse_dispatched} EPG program parse task(s) for changed assignments"
)
return changed_associations
def get_preferred_region_code():
try:
region_obj = CoreSettings.objects.get(key="preferred-region")
return region_obj.value.strip().lower()
except CoreSettings.DoesNotExist:
return None
def _fuzzy_scan_core(chan_norm, rows, region_code=None, candidate_limit=ML_CANDIDATE_LIMIT):
"""
Single-pass fuzzy scan: track best match and top-K candidates.
Rows must already include norm_name when scanning an in-memory catalog.
"""
best_score = 0
best_epg = None
top_heap = []
seq = 0
scanned = 0
for row in rows:
if not row.get("norm_name"):
continue
scanned += 1
score = _compute_fuzzy_score(chan_norm, row, region_code)
if score <= 0:
continue
if score > 50:
logger.debug(f" EPG '{row['name']}' (norm: '{row['norm_name']}') => score: {score}")
priority = row['epg_source_priority']
if score > best_score or (
score == best_score
and priority > (best_epg.get('epg_source_priority', 0) if best_epg else -1)
):
best_score = score
best_epg = row
seq += 1
if len(top_heap) < candidate_limit:
heapq.heappush(top_heap, (score, priority, seq, row))
else:
smallest_score, smallest_priority, _, _ = top_heap[0]
if score > smallest_score or (score == smallest_score and priority > smallest_priority):
heapq.heapreplace(top_heap, (score, priority, seq, row))
top_candidates = sorted(top_heap, key=lambda item: (item[0], item[1]), reverse=True)
return best_score, best_epg, [(score, row) for score, _, _, row in top_candidates], scanned
def fuzzy_scan_epg_list(chan_norm, epg_data, region_code=None, candidate_limit=ML_CANDIDATE_LIMIT):
"""Fuzzy scan over a pre-built in-memory EPG catalog (bulk matching)."""
logger.debug(f"Fuzzy matching '{chan_norm}' against EPG entries...")
return _fuzzy_scan_core(chan_norm, epg_data, region_code, candidate_limit)
def stream_fuzzy_epg_scan(chan_norm, region_code=None, candidate_limit=ML_CANDIDATE_LIMIT):
"""Stream fuzzy scan over active EPG entries (single-channel matching)."""
def row_iterator():
for values_row in _active_epg_fuzzy_queryset().iterator(chunk_size=500):
row = _row_from_epg_values(values_row)
row['norm_name'] = normalize_name(row['name'])
yield row
logger.debug(f"Fuzzy matching '{chan_norm}' against EPG entries...")
return _fuzzy_scan_core(chan_norm, row_iterator(), region_code, candidate_limit)
def _get_epg_match_thresholds(is_bulk_matching):
if is_bulk_matching:
return {
'FUZZY_HIGH_CONFIDENCE': 90,
'FUZZY_SKIP_ML': 80,
'FUZZY_MEDIUM_CONFIDENCE': 70,
'ML_HIGH_CONFIDENCE': 0.75,
'ML_LAST_RESORT': 0.65,
'FUZZY_LAST_RESORT_MIN': 50,
}
return {
'FUZZY_HIGH_CONFIDENCE': 85,
'FUZZY_SKIP_ML': 75,
'FUZZY_MEDIUM_CONFIDENCE': 40,
'ML_HIGH_CONFIDENCE': 0.65,
'ML_LAST_RESORT': 0.50,
'FUZZY_LAST_RESORT_MIN': 20,
}
def try_epg_name_match(chan, best_score, best_epg, top_candidates, is_bulk_matching,
use_ml=True, ml_state=None):
"""
Apply fuzzy/ML thresholds to a channel's best fuzzy result.
Returns the matched EPG row dict, or None.
"""
if not best_epg:
return None
thresholds = _get_epg_match_thresholds(is_bulk_matching)
fuzzy_high = thresholds['FUZZY_HIGH_CONFIDENCE']
fuzzy_skip_ml = thresholds['FUZZY_SKIP_ML']
fuzzy_medium = thresholds['FUZZY_MEDIUM_CONFIDENCE']
ml_high = thresholds['ML_HIGH_CONFIDENCE']
ml_last_resort = thresholds['ML_LAST_RESORT']
fuzzy_last_resort_min = thresholds['FUZZY_LAST_RESORT_MIN']
if best_score >= fuzzy_high:
logger.info(
f"Channel {chan['id']} '{chan['name']}' => matched tvg_id={best_epg['tvg_id']} "
f"(score={best_score})"
)
return best_epg
if best_score >= fuzzy_skip_ml:
logger.info(
f"Channel {chan['id']} '{chan['name']}' => matched tvg_id={best_epg['tvg_id']} "
f"(fuzzy={best_score}, ML skipped)"
)
return best_epg
if ml_state is None:
ml_state = {}
st_model = ml_state.get('st_model')
util = ml_state.get('util')
if best_score >= fuzzy_medium and use_ml:
if st_model is None:
st_model, util = get_sentence_transformer()
ml_state['st_model'] = st_model
ml_state['util'] = util
if st_model:
try:
logger.info("Validating fuzzy best match with ML model (single candidate)")
sims = _ml_cosine_similarities(st_model, util, chan["norm_chan"], [best_epg["norm_name"]])
top_value = sims[0] if sims else 0.0
if top_value >= ml_high - 1e-9:
logger.info(
f"Channel {chan['id']} '{chan['name']}' => matched EPG tvg_id={best_epg['tvg_id']} "
f"(fuzzy={best_score}, ML-sim={top_value:.2f})"
)
return best_epg
if top_value >= ml_last_resort - 1e-9:
logger.info(
f"Channel {chan['id']} '{chan['name']}' => LAST RESORT match EPG "
f"tvg_id={best_epg['tvg_id']} (fuzzy={best_score}, ML-sim={top_value:.2f})"
)
return best_epg
logger.info(
f"Channel {chan['id']} '{chan['name']}' => fuzzy={best_score}, "
f"ML-sim={top_value:.2f} < {ml_last_resort}, skipping"
)
except Exception as e:
logger.warning(f"ML matching failed for channel {chan['id']}: {e}")
logger.info(
f"Channel {chan['id']} '{chan['name']}' => fuzzy score {best_score} below threshold, skipping"
)
return None
if best_score >= fuzzy_last_resort_min and use_ml:
if st_model is None:
st_model, util = get_sentence_transformer()
ml_state['st_model'] = st_model
ml_state['util'] = util
if st_model and top_candidates:
try:
logger.info(
f"Channel {chan['id']} '{chan['name']}' => trying ML last resort against "
f"top {len(top_candidates)} fuzzy candidates (fuzzy={best_score})"
)
candidate_rows = [row for _, row in top_candidates]
sims = _ml_cosine_similarities(
st_model,
util,
chan["norm_chan"],
[row["norm_name"] for row in candidate_rows],
)
top_index = max(range(len(sims)), key=lambda i: sims[i])
top_value = sims[top_index]
matched_epg = candidate_rows[top_index]
if top_value >= ml_last_resort - 1e-9:
logger.info(
f"Channel {chan['id']} '{chan['name']}' => DESPERATE LAST RESORT match "
f"EPG tvg_id={matched_epg['tvg_id']} (fuzzy={best_score}, ML-sim={top_value:.2f})"
)
return matched_epg
logger.info(
f"Channel {chan['id']} '{chan['name']}' => desperate last resort "
f"ML-sim {top_value:.2f} < {ml_last_resort}, giving up"
)
except Exception as e:
logger.warning(f"Last resort ML matching failed for channel {chan['id']}: {e}")
logger.info(
f"Channel {chan['id']} '{chan['name']}' => best fuzzy score={best_score} "
f"< {fuzzy_medium}, giving up"
)
return None
logger.info(
f"Channel {chan['id']} '{chan['name']}' => best fuzzy score={best_score} "
f"< {fuzzy_medium}, no ML fallback available"
)
return None
def prepare_channel_match_data(channel):
"""Build the channel dict used by matching logic."""
normalized_tvg_id = channel.tvg_id.strip().lower() if channel.tvg_id else ""
normalized_gracenote_id = (
channel.tvc_guide_stationid.strip().lower() if channel.tvc_guide_stationid else ""
)
return {
"id": channel.id,
"name": channel.name,
"tvg_id": normalized_tvg_id,
"original_tvg_id": channel.tvg_id,
"gracenote_id": normalized_gracenote_id,
"original_gracenote_id": channel.tvc_guide_stationid,
"fallback_name": normalized_tvg_id if normalized_tvg_id else channel.name,
"norm_chan": normalize_name(channel.name),
"current_epg_data_id": channel.epg_data_id,
}
def match_channels_to_epg(
channels_data,
epg_data,
region_code=None,
use_ml=True,
send_progress=True,
epg_tvg_id_index=None,
):
"""
Match channels to EPG rows using exact ID, fuzzy, and optional ML strategies.
epg_tvg_id_index: optional pre-built tvg_id -> row map from build_epg_matching_catalog().
"""
channels_to_update = []
matched_channels = []
unchanged_channels = []
total_channels = len(channels_data)
if send_progress:
send_epg_matching_progress(total_channels, 0, stage="starting")
is_bulk_matching = len(channels_data) > 1
ml_state = {}
epg_by_tvg_id = epg_tvg_id_index if epg_tvg_id_index is not None else build_epg_tvg_id_index(epg_data)
if is_bulk_matching:
logger.info(f"Using conservative thresholds for bulk matching ({total_channels} channels)")
else:
logger.info("Using aggressive thresholds for single channel matching")
for index, chan in enumerate(channels_data):
normalized_tvg_id = chan.get("tvg_id", "")
fallback_name = chan["tvg_id"].strip() if chan["tvg_id"] else chan["name"]
resolved_count = len(matched_channels) + len(unchanged_channels)
if send_progress and (index < 5 or index % 5 == 0 or index == total_channels - 1):
send_epg_matching_progress(
total_channels,
resolved_count,
current_channel_name=chan["name"][:50],
stage="matching",
)
if normalized_tvg_id:
epg_row = epg_by_tvg_id.get(normalized_tvg_id)
if epg_row:
_record_epg_match(
chan,
epg_row["id"],
epg_name=epg_row.get("name"),
epg_tvg_id=epg_row.get("original_tvg_id") or epg_row.get("tvg_id"),
match_method="exact tvg_id",
channels_to_update=channels_to_update,
matched_channels=matched_channels,
unchanged_channels=unchanged_channels,
)
continue
normalized_gracenote_id = chan.get("gracenote_id", "")
if normalized_gracenote_id:
epg_by_gracenote_id = epg_by_tvg_id.get(normalized_gracenote_id)
if epg_by_gracenote_id:
_record_epg_match(
chan,
epg_by_gracenote_id["id"],
epg_name=epg_by_gracenote_id.get("name"),
epg_tvg_id=epg_by_gracenote_id.get("original_tvg_id")
or epg_by_gracenote_id.get("tvg_id"),
match_method="exact gracenote_id",
channels_to_update=channels_to_update,
matched_channels=matched_channels,
unchanged_channels=unchanged_channels,
)
continue
if not chan["norm_chan"]:
logger.debug(f"Channel {chan['id']} '{chan['name']}' => empty after normalization, skipping")
continue
best_score, best_epg, top_candidates, _scanned = fuzzy_scan_epg_list(
chan["norm_chan"], epg_data, region_code
)
if not best_epg:
logger.debug(f"Channel {chan['id']} '{chan['name']}' => no EPG entries with valid norm_name found")
continue
matched_epg = try_epg_name_match(
chan,
best_score,
best_epg,
top_candidates,
is_bulk_matching,
use_ml=use_ml,
ml_state=ml_state,
)
if matched_epg:
_record_epg_match(
chan,
matched_epg["id"],
epg_name=matched_epg.get("name"),
epg_tvg_id=matched_epg.get("original_tvg_id") or matched_epg.get("tvg_id"),
match_method=f"fuzzy (score={best_score})",
channels_to_update=channels_to_update,
matched_channels=matched_channels,
unchanged_channels=unchanged_channels,
)
if send_progress:
send_epg_matching_progress(
total_channels,
len(matched_channels) + len(unchanged_channels),
stage="completed",
)
return {
"channels_to_update": channels_to_update,
"matched_channels": matched_channels,
"unchanged_channels": unchanged_channels,
}
def run_single_channel_epg_match(channel_id):
"""
Match one channel to EPG data. Always notifies the UI via WebSocket before returning.
"""
from apps.channels.models import Channel
channel = None
try:
logger.info(f"Starting integrated single channel EPG matching for channel ID {channel_id}")
try:
channel = Channel.objects.get(id=channel_id)
except Channel.DoesNotExist:
message = "Channel not found"
send_single_channel_epg_match_result(channel_id, False, message)
return {"matched": False, "message": message}
channel_data = prepare_channel_match_data(channel)
logger.info(
f"Channel data prepared: name='{channel.name}', tvg_id='{channel_data['tvg_id']}', "
f"gracenote_id='{channel_data['gracenote_id']}', norm_chan='{channel_data['norm_chan']}'"
)
send_epg_matching_progress(1, 0, current_channel_name=channel.name, stage="matching")
region_code = get_preferred_region_code()
fallback_name = channel_data["tvg_id"] if channel_data["tvg_id"] else channel.name
matched_epg_row = None
match_via = None
if channel_data["tvg_id"]:
matched_epg_row = lookup_epg_by_tvg_id(channel_data["tvg_id"])
if matched_epg_row:
match_via = matched_epg_row["tvg_id"]
logger.info(
f"Channel {channel.id} '{fallback_name}' => EPG found by exact tvg_id={match_via}"
)
if not matched_epg_row and channel_data["gracenote_id"]:
matched_epg_row = lookup_epg_by_tvg_id(channel_data["gracenote_id"])
if matched_epg_row:
match_via = f"gracenote:{matched_epg_row['tvg_id']}"
logger.info(
f"Channel {channel.id} '{fallback_name}' => EPG found by exact "
f"gracenote_id={channel_data['gracenote_id']}"
)
if not matched_epg_row and channel_data["norm_chan"]:
best_score, best_epg, top_candidates, scanned = stream_fuzzy_epg_scan(
channel_data["norm_chan"], region_code
)
logger.info(
f"Matching single channel '{channel.name}' against {scanned} EPG entries"
)
if best_epg:
logger.info(
f"Channel {channel.id} '{channel.name}' => best match: '{best_epg['name']}' "
f"(score: {best_score})"
)
matched_epg_row = try_epg_name_match(
channel_data,
best_score,
best_epg,
top_candidates,
is_bulk_matching=False,
use_ml=True,
)
if matched_epg_row:
match_via = matched_epg_row["tvg_id"]
elif not channel_data["norm_chan"]:
logger.debug(f"Channel {channel.id} '{channel.name}' => empty after normalization, skipping")
if not matched_epg_row:
has_fuzzy_epg = _active_epg_fuzzy_queryset().exists()
if not has_fuzzy_epg and not channel_data["tvg_id"] and not channel_data["gracenote_id"]:
message = "No EPG data available for matching (from active sources)"
send_epg_matching_progress(1, 0, current_channel_name=channel.name, stage="completed")
send_single_channel_epg_match_result(channel.id, False, message, channel=channel)
return {"matched": False, "message": message}
if matched_epg_row:
try:
matched_epg_id = matched_epg_row["id"]
epg_data = (
channel.epg_data
if channel.epg_data_id == matched_epg_id
else EPGData.objects.get(id=matched_epg_id)
)
if channel.epg_data_id == matched_epg_id:
success_msg = (
f"Channel '{channel.name}' already matched with EPG '{epg_data.name}'"
)
if match_via:
success_msg += f" (matched via: {match_via})"
logger.info(success_msg)
send_epg_matching_progress(1, 1, current_channel_name=channel.name, stage="completed")
send_single_channel_epg_match_result(
channel.id, True, success_msg, channel=channel, epg_data=epg_data
)
return {
"matched": True,
"unchanged": True,
"message": success_msg,
"epg_name": epg_data.name,
"epg_id": epg_data.id,
}
channel.epg_data = epg_data
channel.save(update_fields=["epg_data"])
success_msg = f"Channel '{channel.name}' matched with EPG '{epg_data.name}'"
if match_via:
success_msg += f" (matched via: {match_via})"
logger.info(success_msg)
send_epg_matching_progress(1, 1, current_channel_name=channel.name, stage="completed")
channel.refresh_from_db()
send_single_channel_epg_match_result(
channel.id, True, success_msg, channel=channel, epg_data=epg_data
)
return {
"matched": True,
"message": success_msg,
"epg_name": epg_data.name,
"epg_id": epg_data.id,
}
except EPGData.DoesNotExist:
message = "Matched EPG data not found"
send_single_channel_epg_match_result(channel.id, False, message, channel=channel)
return {"matched": False, "message": message}
send_epg_matching_progress(1, 0, current_channel_name=channel.name, stage="completed")
message = f"No suitable EPG match found for channel '{channel.name}'"
send_single_channel_epg_match_result(channel.id, False, message, channel=channel)
return {"matched": False, "message": message}
except Exception as e:
logger.error(f"Error in integrated single channel EPG matching: {e}", exc_info=True)
message = f"Error during matching: {str(e)}"
send_single_channel_epg_match_result(
channel_id,
False,
message,
channel=channel,
)
return {"matched": False, "message": message}
finally:
cleanup_after_matching()

216
apps/channels/models.py
View file

@ -4,7 +4,7 @@ from django.conf import settings
from core.models import StreamProfile, CoreSettings
from core.utils import RedisClient
from apps.proxy.live_proxy.redis_keys import RedisKeys
from apps.proxy.live_proxy.constants import ChannelMetadataField
from apps.proxy.live_proxy.constants import ChannelMetadataField, ChannelState
import logging
import uuid
from django.utils import timezone
@ -17,6 +17,7 @@ logger = logging.getLogger(__name__)
# If you have an M3UAccount model in apps.m3u, you can still import it:
from apps.m3u.models import M3UAccount
from apps.m3u.connection_pool import reserve_profile_slot, release_profile_slot
# Add fallback functions if Redis isn't available
@ -222,13 +223,14 @@ class Stream(models.Model):
Finds an available profile for this stream and reserves a connection slot.
Returns:
Tuple[Optional[int], Optional[int], Optional[str]]: (stream_id, profile_id, error_reason)
Tuple[Optional[int], Optional[int], Optional[str], bool]:
(stream_id, profile_id, error_reason, slot_reserved)
"""
redis_client = RedisClient.get_client()
profile_id = redis_client.get(f"stream_profile:{self.id}")
if profile_id:
profile_id = int(profile_id)
return self.id, profile_id, None
return self.id, profile_id, None, False
# Retrieve the M3U account associated with the stream.
m3u_account = self.m3u_account
@ -239,29 +241,22 @@ class Stream(models.Model):
]
for profile in profiles:
logger.info(profile)
logger.debug("Evaluating profile %s for stream %s", profile.id, self.id)
# Skip inactive profiles
if profile.is_active == False:
continue
# Atomic slot reservation: INCR first, check, rollback if over capacity
if profile.max_streams == 0:
reserved = True
else:
profile_connections_key = f"profile_connections:{profile.id}"
new_count = redis_client.incr(profile_connections_key)
if new_count <= profile.max_streams:
reserved = True
else:
redis_client.decr(profile_connections_key)
reserved = False
# Atomic slot reservation via shared connection pool helper
reserved, _count, _failure_reason = reserve_profile_slot(
profile, redis_client
)
if reserved:
redis_client.set(f"channel_stream:{self.id}", self.id)
redis_client.set(f"stream_profile:{self.id}", profile.id)
return self.id, profile.id, None
return self.id, profile.id, None, True
return None, None, "All active M3U profiles have reached maximum connection limits"
return None, None, "All active M3U profiles have reached maximum connection limits", False
def release_stream(self):
"""
@ -290,12 +285,7 @@ class Stream(models.Model):
f"Stream {stream_id}: found profile_id={profile_id}"
)
profile_connections_key = f"profile_connections:{profile_id}"
# Only decrement if the profile had a max_connections limit
current_count = int(redis_client.get(profile_connections_key) or 0)
if current_count > 0:
redis_client.decr(profile_connections_key)
release_profile_slot(profile_id, redis_client)
return True
@ -625,44 +615,65 @@ class Channel(models.Model):
return victim_id
def _check_and_reserve_profile_slot(self, profile, redis_client):
def _channel_proxy_is_active(self, redis_client) -> bool:
"""True when live proxy metadata shows this channel is still running."""
metadata_key = RedisKeys.channel_metadata(str(self.uuid))
if not redis_client.exists(metadata_key):
return False
state = redis_client.hget(metadata_key, ChannelMetadataField.STATE)
if state is None:
return False
if isinstance(state, bytes):
state = state.decode()
return state in (
ChannelState.ACTIVE,
ChannelState.WAITING_FOR_CLIENTS,
ChannelState.BUFFERING,
ChannelState.INITIALIZING,
ChannelState.CONNECTING,
)
def _stream_assignment_is_reusable(self, redis_client, stream_id: int) -> bool:
"""
Atomically check and reserve a connection slot for the given profile.
Return True when an existing channel_stream assignment should be reused.
Uses an INCR-first-then-check pattern to eliminate the TOCTOU race
condition where separate GET + check + INCR operations could allow
concurrent requests to both pass the capacity check.
For profiles with max_streams=0 (unlimited), no reservation is needed.
Args:
profile: M3UAccountProfile instance
redis_client: Redis client instance
Returns:
tuple: (reserved: bool, current_count: int)
Reuse when the proxy is active, or when metadata is not written yet
(between get_stream() reserving slots and initialize_channel() starting).
When metadata exists but the proxy is inactive, the assignment is stale.
"""
if profile.max_streams == 0:
return (True, 0)
if self._channel_proxy_is_active(redis_client):
return True
profile_connections_key = f"profile_connections:{profile.id}"
metadata_key = RedisKeys.channel_metadata(str(self.uuid))
if not redis_client.exists(metadata_key):
return redis_client.get(f"stream_profile:{stream_id}") is not None
# Atomically increment first — this is a single Redis command
new_count = redis_client.incr(profile_connections_key)
return False
if new_count <= profile.max_streams:
return (True, new_count)
def _release_stale_stream_assignment(self, redis_client, stream_id: int) -> None:
"""Release pool counters and remove stale channel/stream assignment keys."""
profile_id_bytes = redis_client.get(f"stream_profile:{stream_id}")
if profile_id_bytes:
try:
profile_id = int(profile_id_bytes)
release_profile_slot(profile_id, redis_client)
except (ValueError, TypeError):
logger.debug(
"Invalid profile ID for stale assignment on stream %s: %s",
stream_id,
profile_id_bytes,
)
# Over capacity — roll back the increment
redis_client.decr(profile_connections_key)
return (False, new_count - 1)
redis_client.delete(f"channel_stream:{self.id}")
redis_client.delete(f"stream_profile:{stream_id}")
def get_stream(self, requester=None):
"""
Finds an available stream for the requested channel and returns the selected stream and profile.
Returns:
Tuple[Optional[int], Optional[int], Optional[str]]: (stream_id, profile_id, error_reason)
Tuple[Optional[int], Optional[int], Optional[str], bool]:
(stream_id, profile_id, error_reason, slot_reserved)
"""
redis_client = RedisClient.get_client()
error_reason = None
@ -670,26 +681,42 @@ class Channel(models.Model):
# Check if this channel has any streams
if not self.streams.exists():
error_reason = "No streams assigned to channel"
return None, None, error_reason
return None, None, error_reason, False
# Check if a stream is already active for this channel
# Reuse assignment only when this channel is still active in the proxy.
# Stale channel_stream keys after stop/disconnect skip INCR and break pool
# accounting, which lets a second stream reach the provider and fail validation.
stream_id_bytes = redis_client.get(f"channel_stream:{self.id}")
if stream_id_bytes:
try:
stream_id = int(stream_id_bytes)
profile_id_bytes = redis_client.get(f"stream_profile:{stream_id}")
if profile_id_bytes:
try:
profile_id = int(profile_id_bytes)
return stream_id, profile_id, None
except (ValueError, TypeError):
logger.debug(
f"Invalid profile ID retrieved from Redis: {profile_id_bytes}"
)
except (ValueError, TypeError):
logger.debug(
f"Invalid stream ID retrieved from Redis: {stream_id_bytes}"
)
stream_id = None
if stream_id is not None:
if self._stream_assignment_is_reusable(redis_client, stream_id):
profile_id_bytes = redis_client.get(f"stream_profile:{stream_id}")
if profile_id_bytes:
try:
profile_id = int(profile_id_bytes)
logger.debug(
f"Channel {self.uuid}: reusing stream assignment "
f"stream={stream_id} profile={profile_id}"
)
return stream_id, profile_id, None, False
except (ValueError, TypeError):
logger.debug(
f"Invalid profile ID retrieved from Redis: {profile_id_bytes}"
)
else:
logger.info(
f"Channel {self.uuid}: releasing stale stream assignment "
f"(stream={stream_id}, proxy not active)"
)
self._release_stale_stream_assignment(redis_client, stream_id)
# No existing active stream, attempt to assign a new one
has_streams_but_maxed_out = False
@ -723,7 +750,7 @@ class Channel(models.Model):
has_active_profiles = True
# Atomically check and reserve a slot (INCR-first pattern)
reserved, current_count = self._check_and_reserve_profile_slot(
reserved, current_count, failure_reason = reserve_profile_slot(
profile, redis_client
)
@ -731,11 +758,16 @@ class Channel(models.Model):
# Slot reserved — assign stream to this channel
redis_client.set(f"channel_stream:{self.id}", stream.id)
redis_client.set(f"stream_profile:{stream.id}", profile.id)
logger.info(
f"Channel {self.uuid}: assigned stream {stream.id} "
f"profile {profile.id} ({profile.name})"
)
return (
stream.id,
profile.id,
None,
True,
) # Return newly assigned stream and matched profile
else:
# At capacity: try to preempt a lower-impact channel on this profile
@ -749,12 +781,21 @@ class Channel(models.Model):
logger.info(f"Preempting channel {victim_channel_id} for new stream on profile {profile.id}")
# return self.id, profile.id, victim_channel_id
# This profile is at max connections
has_streams_but_maxed_out = True
logger.debug(
f"Profile {profile.id} at max connections: "
f"{current_count}/{profile.max_streams}"
)
if failure_reason == "profile_full":
logger.info(
f"Profile {profile.id} at max connections: "
f"{current_count}/{profile.max_streams}, trying next profile"
)
elif failure_reason == "credential_full":
logger.info(
f"Profile {profile.id} shared login pool full, trying next profile"
)
else:
logger.debug(
f"Profile {profile.id} reservation failed: "
f"{current_count}/{profile.max_streams}"
)
# No available streams - determine specific reason
if has_streams_but_maxed_out:
@ -764,7 +805,7 @@ class Channel(models.Model):
else:
error_reason = "No active profiles found for any assigned stream"
return None, None, error_reason
return None, None, error_reason, False
def release_stream(self):
"""
@ -808,12 +849,7 @@ class Channel(models.Model):
ChannelMetadataField.M3U_PROFILE,
)
profile_connections_key = f"profile_connections:{profile_id}"
current_count = int(
redis_client.get(profile_connections_key) or 0
)
if current_count > 0:
redis_client.decr(profile_connections_key)
release_profile_slot(profile_id, redis_client)
return True
logger.debug(
@ -858,12 +894,7 @@ class Channel(models.Model):
f"stream {stream_id}"
)
profile_connections_key = f"profile_connections:{profile_id}"
# Only decrement if the profile had a max_connections limit
current_count = int(redis_client.get(profile_connections_key) or 0)
if current_count > 0:
redis_client.decr(profile_connections_key)
release_profile_slot(profile_id, redis_client)
# Clear metadata fields so duplicate release_stream() calls
# (e.g. from _clean_redis_keys or ChannelService.stop_channel)
@ -909,10 +940,31 @@ class Channel(models.Model):
if current_profile_id == new_profile_id:
return True
# Use pipeline for atomic profile switch to prevent counter drift
# if an exception occurs between DECR and INCR
old_profile_connections_key = f"profile_connections:{current_profile_id}"
new_profile_connections_key = f"profile_connections:{new_profile_id}"
from apps.m3u.connection_pool import (
move_credential_slot_on_profile_switch,
profile_connections_key,
)
from apps.m3u.models import M3UAccountProfile
old_profile = M3UAccountProfile.objects.select_related(
"m3u_account__server_group"
).get(id=current_profile_id)
new_profile = M3UAccountProfile.objects.select_related(
"m3u_account__server_group"
).get(id=new_profile_id)
if not move_credential_slot_on_profile_switch(
old_profile, new_profile, redis_client
):
logger.warning(
"Shared login pool full for profile %s during stream profile switch",
new_profile_id,
)
return False
# Profile counters always move on switch; credential totals move only when login changes.
old_profile_connections_key = profile_connections_key(current_profile_id)
new_profile_connections_key = profile_connections_key(new_profile_id)
old_count = int(redis_client.get(old_profile_connections_key) or 0)
pipe = redis_client.pipeline()

17
apps/channels/serializers.py
View file

@ -19,6 +19,7 @@ from apps.epg.serializers import EPGDataSerializer
from core.models import StreamProfile
from apps.epg.models import EPGData
from django.db import connection, transaction
from django.urls import reverse
from rest_framework import serializers
from django.utils import timezone
from core.utils import validate_flexible_url, build_absolute_uri_with_port
@ -56,12 +57,18 @@ class LogoSerializer(serializers.ModelSerializer):
return instance
def get_cache_url(self, obj):
# Cache-busting: append a short hash of the logo's source URL so the browser
# fetches fresh when the logo changes (e.g., M3U logo replaced by SD logo).
# The backend ignores the 'v' parameter — it's purely for browser cache invalidation.
# See SD integration PR notes for context on why this was added.
import hashlib
url_hash = hashlib.md5((obj.url or '').encode()).hexdigest()[:8]
base_path = reverse("api:channels:logo-cache", args=[obj.id])
cache_url = f"{base_path}?v={url_hash}"
request = self.context.get("request")
if not request:
return f"/api/channels/logos/{obj.id}/cache/"
if not hasattr(self, "_cache_url_prefix"):
self._cache_url_prefix = build_absolute_uri_with_port(request, "")
return f"{self._cache_url_prefix}/api/channels/logos/{obj.id}/cache/"
if request:
return build_absolute_uri_with_port(request, cache_url)
return cache_url
def get_channel_count(self, obj):
"""Get the number of channels using this logo"""

9
apps/channels/signals.py
View file

@ -201,10 +201,19 @@ def refresh_epg_programs(sender, instance, created, **kwargs):
if not created and kwargs.get('update_fields') and 'epg_data' in kwargs['update_fields']:
logger.info(f"Channel {instance.id} ({instance.name}) EPG data updated, refreshing program data")
if instance.epg_data:
# Skip XMLTV program parser for SD sources — program data is handled
# by fetch_schedules_direct() directly.
if instance.epg_data.epg_source and instance.epg_data.epg_source.source_type == 'schedules_direct':
logger.info(f"Skipping XMLTV parse for SD source {instance.epg_data.tvg_id}")
return
logger.info(f"Triggering EPG program refresh for {instance.epg_data.tvg_id}")
parse_programs_for_tvg_id.delay(instance.epg_data.id)
# For new channels with EPG data, also refresh
elif created and instance.epg_data:
# Skip XMLTV program parser for SD sources
if instance.epg_data.epg_source and instance.epg_data.epg_source.source_type == 'schedules_direct':
logger.info(f"Skipping XMLTV parse for SD source {instance.epg_data.tvg_id}")
return
logger.info(f"New channel {instance.id} ({instance.name}) created with EPG data, refreshing program data")
parse_programs_for_tvg_id.delay(instance.epg_data.id)

1907
apps/channels/tasks.py
View file

File diff suppressed because it is too large Load diff

12
apps/channels/tests/test_db_retry.py
View file

@ -196,15 +196,17 @@ class InitialConnectionRetryTests(TestCase):
base URL before falling back to the next candidate."""
def test_reconnect_max_constant_exists_in_run_recording(self):
"""run_recording must define a max-reconnect limit to prevent
infinite retries on the same broken base URL."""
"""run_recording must use a time-bounded FFmpeg outage window to prevent
infinite restarts when the source stream is permanently down."""
import inspect
from apps.channels.tasks import run_recording
from apps.channels.tasks import run_recording, _dvr_ffmpeg_retry_window_seconds
source = inspect.getsource(run_recording)
# The reconnection counter pattern must be present
self.assertGreater(_dvr_ffmpeg_retry_window_seconds(), 0)
self.assertIn("reconnect", source.lower(),
"run_recording must contain reconnection logic")
"run_recording must contain input reconnection flags")
self.assertIn("_ffmpeg_outage_started", source)
self.assertIn("_ffmpeg_retry_window", source)
# ---------------------------------------------------------------------------

272
apps/channels/tests/test_epg_logo_apply.py Normal file
View file

@ -0,0 +1,272 @@
from unittest.mock import patch
from django.contrib.auth import get_user_model
from django.test import TestCase
from rest_framework import status
from rest_framework.test import APIClient
from apps.channels.models import Channel, Logo
from apps.channels.utils import (
apply_logos_from_epg_icon_url,
apply_logos_from_epg_for_source,
auto_apply_epg_logos_enabled,
maybe_auto_apply_epg_logos,
)
from apps.epg.models import EPGData, EPGSource
User = get_user_model()
class AutoApplyEpgLogosEnabledTests(TestCase):
def test_enabled_when_flag_true(self):
self.assertTrue(
auto_apply_epg_logos_enabled({'auto_apply_epg_logos': True})
)
def test_disabled_when_flag_false_or_missing(self):
self.assertFalse(
auto_apply_epg_logos_enabled({'auto_apply_epg_logos': False})
)
self.assertFalse(auto_apply_epg_logos_enabled({}))
self.assertFalse(auto_apply_epg_logos_enabled(None))
class ApplyLogosFromEpgIconUrlTests(TestCase):
def setUp(self):
self.source = EPGSource.objects.create(
name='XML EPG',
source_type='xmltv',
url='http://example.com/epg.xml',
)
self.epg_one = EPGData.objects.create(
tvg_id='ch.one',
name='Channel One',
icon_url='https://example.com/one.png',
epg_source=self.source,
)
self.epg_two = EPGData.objects.create(
tvg_id='ch.two',
name='Channel Two',
icon_url='https://example.com/one.png',
epg_source=self.source,
)
self.channel_one = Channel.objects.create(
channel_number=1,
name='Channel One',
tvg_id='ch.one',
epg_data=self.epg_one,
)
self.channel_two = Channel.objects.create(
channel_number=2,
name='Channel Two',
tvg_id='ch.two',
epg_data=self.epg_two,
)
def test_creates_logo_and_updates_channels(self):
channels = Channel.objects.filter(
id__in=[self.channel_one.id, self.channel_two.id],
).select_related('epg_data', 'logo')
stats = apply_logos_from_epg_icon_url(channels)
self.assertEqual(stats['updated_count'], 2)
self.assertEqual(stats['created_logos_count'], 1)
self.assertEqual(Logo.objects.count(), 1)
self.channel_one.refresh_from_db()
self.channel_two.refresh_from_db()
self.assertEqual(self.channel_one.logo.url, 'https://example.com/one.png')
self.assertEqual(self.channel_two.logo_id, self.channel_one.logo_id)
def test_skips_channels_already_using_icon_url(self):
existing_logo = Logo.objects.create(
name='Existing',
url='https://example.com/one.png',
)
self.channel_one.logo = existing_logo
self.channel_one.save(update_fields=['logo'])
channels = Channel.objects.filter(
id__in=[self.channel_one.id, self.channel_two.id],
).select_related('epg_data', 'logo')
stats = apply_logos_from_epg_icon_url(channels)
self.assertEqual(stats['updated_count'], 1)
self.assertEqual(stats['created_logos_count'], 0)
def test_skips_channels_without_icon_url(self):
self.epg_one.icon_url = None
self.epg_one.save(update_fields=['icon_url'])
channels = Channel.objects.filter(
id__in=[self.channel_one.id, self.channel_two.id],
).select_related('epg_data', 'logo')
stats = apply_logos_from_epg_icon_url(channels)
self.assertEqual(stats['updated_count'], 1)
self.channel_one.refresh_from_db()
self.assertIsNone(self.channel_one.logo_id)
class ApplyLogosForSourceTests(TestCase):
def setUp(self):
self.source = EPGSource.objects.create(
name='XML EPG',
source_type='xmltv',
url='http://example.com/epg.xml',
)
self.other_source = EPGSource.objects.create(
name='Other EPG',
source_type='xmltv',
url='http://example.com/other.xml',
)
self.epg = EPGData.objects.create(
tvg_id='mapped',
name='Mapped',
icon_url='https://example.com/mapped.png',
epg_source=self.source,
)
self.other_epg = EPGData.objects.create(
tvg_id='other',
name='Other',
icon_url='https://example.com/other.png',
epg_source=self.other_source,
)
self.mapped_channel = Channel.objects.create(
channel_number=1,
name='Mapped',
tvg_id='mapped',
epg_data=self.epg,
)
Channel.objects.create(
channel_number=2,
name='Other',
tvg_id='other',
epg_data=self.other_epg,
)
def test_only_updates_channels_mapped_to_source(self):
stats = apply_logos_from_epg_for_source(self.source)
self.assertEqual(stats['updated_count'], 1)
self.mapped_channel.refresh_from_db()
self.assertEqual(
self.mapped_channel.logo.url,
'https://example.com/mapped.png',
)
def test_processes_source_in_batches(self):
stats = apply_logos_from_epg_for_source(self.source, batch_size=1)
self.assertEqual(stats['updated_count'], 1)
self.mapped_channel.refresh_from_db()
self.assertEqual(
self.mapped_channel.logo.url,
'https://example.com/mapped.png',
)
class MaybeAutoApplyEpgLogosTests(TestCase):
def setUp(self):
self.source = EPGSource.objects.create(
name='XML EPG',
source_type='xmltv',
url='http://example.com/epg.xml',
custom_properties={'auto_apply_epg_logos': True},
)
self.epg = EPGData.objects.create(
tvg_id='mapped',
name='Mapped',
icon_url='https://example.com/mapped.png',
epg_source=self.source,
)
self.channel = Channel.objects.create(
channel_number=1,
name='Mapped',
tvg_id='mapped',
epg_data=self.epg,
)
def test_runs_when_enabled(self):
stats = maybe_auto_apply_epg_logos(self.source)
self.assertEqual(stats['updated_count'], 1)
self.channel.refresh_from_db()
self.assertIsNotNone(self.channel.logo_id)
def test_skips_when_disabled(self):
self.source.custom_properties = {'auto_apply_epg_logos': False}
self.source.save(update_fields=['custom_properties'])
stats = maybe_auto_apply_epg_logos(self.source)
self.assertIsNone(stats)
self.channel.refresh_from_db()
self.assertIsNone(self.channel.logo_id)
class SetLogosFromEpgApiTests(TestCase):
def setUp(self):
self.user = User.objects.create_user(username='testuser', password='testpass123')
self.user.user_level = 10
self.user.save()
self.client = APIClient()
self.client.force_authenticate(user=self.user)
self.url = '/api/channels/channels/set-logos-from-epg/'
self.source = EPGSource.objects.create(
name='XML EPG',
source_type='xmltv',
url='http://example.com/epg.xml',
)
self.epg = EPGData.objects.create(
tvg_id='mapped',
name='Mapped',
icon_url='https://example.com/mapped.png',
epg_source=self.source,
)
self.channel = Channel.objects.create(
channel_number=1,
name='Mapped',
tvg_id='mapped',
epg_data=self.epg,
)
@patch('apps.channels.tasks.set_channels_logos_from_epg.delay')
def test_accepts_channel_ids(self, mock_delay):
mock_delay.return_value.id = 'task-1'
response = self.client.post(
self.url,
{'channel_ids': [self.channel.id]},
format='json',
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
mock_delay.assert_called_once_with(channel_ids=[self.channel.id])
@patch('apps.channels.tasks.set_channels_logos_from_epg.delay')
def test_accepts_epg_source_id(self, mock_delay):
mock_delay.return_value.id = 'task-2'
response = self.client.post(
self.url,
{'epg_source_id': self.source.id},
format='json',
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
mock_delay.assert_called_once_with(epg_source_id=self.source.id)
self.assertEqual(response.data['channel_count'], 1)
def test_rejects_both_parameters(self):
response = self.client.post(
self.url,
{
'channel_ids': [self.channel.id],
'epg_source_id': self.source.id,
},
format='json',
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)

58
apps/channels/tests/test_epg_match_apply.py Normal file
View file

@ -0,0 +1,58 @@
"""Tests for applying EPG auto-match results to channels."""
from unittest.mock import patch
from django.test import TestCase
from apps.channels.epg_matching import apply_matched_epg_to_channels
from apps.channels.models import Channel
from apps.epg.models import EPGData, EPGSource
class ApplyMatchedEpgToChannelsTests(TestCase):
def setUp(self):
self.source = EPGSource.objects.create(
name="XML EPG",
source_type="xmltv",
url="http://example.com/epg.xml",
)
self.epg_one = EPGData.objects.create(
tvg_id="ch.one",
name="Channel One",
epg_source=self.source,
)
self.epg_two = EPGData.objects.create(
tvg_id="ch.two",
name="Channel Two",
epg_source=self.source,
)
self.channel = Channel.objects.create(
channel_number=1,
name="Channel One",
tvg_id="ch.one",
epg_data=self.epg_one,
)
@patch("apps.epg.tasks.parse_programs_for_tvg_id.delay")
def test_skips_unchanged_assignment(self, mock_delay):
changed = apply_matched_epg_to_channels(
[{"id": self.channel.id, "epg_data_id": self.epg_one.id}]
)
self.assertEqual(changed, [])
mock_delay.assert_not_called()
self.channel.refresh_from_db()
self.assertEqual(self.channel.epg_data_id, self.epg_one.id)
@patch("apps.epg.tasks.parse_programs_for_tvg_id.delay")
def test_updates_changed_assignment_and_dispatches_parse(self, mock_delay):
changed = apply_matched_epg_to_channels(
[{"id": self.channel.id, "epg_data_id": self.epg_two.id}]
)
self.assertEqual(
changed,
[{"channel_id": self.channel.id, "epg_data_id": self.epg_two.id}],
)
mock_delay.assert_called_once_with(self.epg_two.id)
self.channel.refresh_from_db()
self.assertEqual(self.channel.epg_data_id, self.epg_two.id)

117
apps/channels/tests/test_epg_name_normalize.py Normal file
View file

@ -0,0 +1,117 @@
"""Tests for EPG channel name normalization (prefix/suffix/custom ignore rules)."""
from django.test import TestCase
from apps.channels.epg_matching import (
build_epg_tvg_id_index,
clear_normalize_settings_cache,
normalize_name,
)
from core.models import CoreSettings, EPG_SETTINGS_KEY
class NormalizeNameSettingsTest(TestCase):
def _set_epg_settings(self, **kwargs):
obj, _ = CoreSettings.objects.get_or_create(
key=EPG_SETTINGS_KEY,
defaults={"name": "EPG Settings", "value": {}},
)
current = obj.value if isinstance(obj.value, dict) else {}
current.update(kwargs)
obj.value = current
obj.save()
clear_normalize_settings_cache()
def test_default_mode_does_not_apply_ignore_lists(self):
self._set_epg_settings(
epg_match_mode="default",
epg_match_ignore_prefixes=["HD:"],
epg_match_ignore_suffixes=[" 4K"],
epg_match_ignore_custom=["Plus"],
)
result_default = normalize_name("HD:HBO Plus East 4K")
self._set_epg_settings(
epg_match_mode="advanced",
epg_match_ignore_prefixes=["HD:"],
epg_match_ignore_suffixes=[" 4K"],
epg_match_ignore_custom=["Plus"],
)
result_advanced = normalize_name("HD:HBO Plus East 4K")
self.assertNotEqual(result_default, result_advanced)
self.assertEqual(result_advanced, "hbo east")
def test_advanced_mode_strips_prefix(self):
self._set_epg_settings(
epg_match_mode="advanced",
epg_match_ignore_prefixes=["HD:"],
)
self.assertEqual(
normalize_name("HD:ABC 7 (WXYZ) - Springfield"),
"abc 7 springfield wxyz",
)
def test_advanced_mode_strips_suffix(self):
self._set_epg_settings(
epg_match_mode="advanced",
epg_match_ignore_suffixes=[" 4K"],
)
self.assertEqual(
normalize_name("NBC 5 (KABC) - Metro 4K"),
"nbc 5 metro kabc",
)
def test_advanced_mode_removes_custom_strings(self):
self._set_epg_settings(
epg_match_mode="advanced",
epg_match_ignore_custom=["Plus"],
)
self.assertEqual(
normalize_name("HBO Plus East"),
"hbo east",
)
def test_advanced_mode_applies_prefix_suffix_and_custom_in_order(self):
self._set_epg_settings(
epg_match_mode="advanced",
epg_match_ignore_prefixes=["Sling:"],
epg_match_ignore_suffixes=[" HD"],
epg_match_ignore_custom=["Plus"],
)
self.assertEqual(
normalize_name("Sling:HBO Plus East HD"),
"hbo east",
)
def test_only_first_matching_prefix_is_removed(self):
self._set_epg_settings(
epg_match_mode="advanced",
epg_match_ignore_prefixes=["HD:", "SD:"],
)
self.assertEqual(normalize_name("HD:SD:Channel 5"), "sd channel 5")
def test_call_sign_preserved_from_original_name(self):
self._set_epg_settings(epg_match_mode="default")
self.assertEqual(
normalize_name("NBC 5 (KABC) - Metro"),
"nbc 5 metro kabc",
)
def test_tvg_id_index_prefers_first_entry_when_catalog_sorted_by_priority(self):
# Catalog from build_epg_matching_catalog() is highest-priority first.
epg_data = [
{"id": 2, "tvg_id": "abc.us", "epg_source_priority": 50, "name": "High"},
{"id": 1, "tvg_id": "abc.us", "epg_source_priority": 10, "name": "Low"},
]
index = build_epg_tvg_id_index(epg_data)
self.assertEqual(index["abc.us"]["id"], 2)
def test_settings_cache_refresh_picks_up_new_rules(self):
self._set_epg_settings(epg_match_mode="default")
self.assertEqual(normalize_name("HD:ABC"), "hdabc")
self._set_epg_settings(
epg_match_mode="advanced",
epg_match_ignore_prefixes=["HD:"],
)
self.assertEqual(normalize_name("HD:ABC"), "abc")

173
apps/channels/tests/test_get_stream_assignment.py Normal file
View file

@ -0,0 +1,173 @@
"""Tests for Channel.get_stream() assignment reuse and stale cleanup."""
from unittest.mock import patch
from django.test import TestCase
from apps.channels.models import Channel, ChannelStream, Stream
from apps.m3u.models import M3UAccount, M3UAccountProfile
from apps.proxy.live_proxy.constants import ChannelMetadataField, ChannelState
from apps.proxy.live_proxy.redis_keys import RedisKeys
class FakeAssignmentRedis:
"""In-memory Redis for channel_stream assignment tests."""
def __init__(self):
self._strings = {}
self._hashes = {}
def _decode(self, value):
if isinstance(value, bytes):
return value.decode()
return value
def get(self, key):
value = self._strings.get(key)
if value is None:
return None
if isinstance(value, int):
return str(value).encode()
return str(value).encode()
def set(self, key, value):
self._strings[key] = value
def delete(self, key):
self._strings.pop(key, None)
self._hashes.pop(key, None)
def exists(self, key):
return key in self._strings or key in self._hashes
def hget(self, key, field):
return self._hashes.get(key, {}).get(field)
def hset(self, key, mapping=None, **kwargs):
bucket = self._hashes.setdefault(key, {})
if mapping:
bucket.update(mapping)
bucket.update(kwargs)
def incr(self, key):
current = int(self._decode(self.get(key)) or 0)
current += 1
self._strings[key] = current
return current
def decr(self, key):
current = int(self._decode(self.get(key)) or 0)
current -= 1
self._strings[key] = current
return current
class ChannelGetStreamAssignmentTests(TestCase):
def setUp(self):
self.redis = FakeAssignmentRedis()
self.account = M3UAccount.objects.create(
name="assignment-test",
account_type="XC",
username="user",
password="pass",
max_streams=5,
)
self.profile = M3UAccountProfile.objects.get(
m3u_account=self.account, is_default=True
)
self.profile.max_streams = 2
self.profile.save()
self.stream = Stream.objects.create(
name="Test Stream",
url="http://example.com/live/user/pass/1.ts",
m3u_account=self.account,
)
self.channel = Channel.objects.create(channel_number=501, name="Assignment Ch")
ChannelStream.objects.create(channel=self.channel, stream=self.stream, order=0)
self.metadata_key = RedisKeys.channel_metadata(str(self.channel.uuid))
def _seed_assignment(self):
self.redis.set(f"channel_stream:{self.channel.id}", self.stream.id)
self.redis.set(f"stream_profile:{self.stream.id}", self.profile.id)
@patch("apps.channels.models.RedisClient.get_client")
@patch("apps.channels.models.reserve_profile_slot")
def test_reuses_assignment_when_proxy_active(
self, mock_reserve, mock_get_client
):
mock_get_client.return_value = self.redis
self._seed_assignment()
self.redis.hset(
self.metadata_key,
{ChannelMetadataField.STATE: ChannelState.ACTIVE},
)
stream_id, profile_id, error, slot_reserved = self.channel.get_stream()
self.assertEqual(stream_id, self.stream.id)
self.assertEqual(profile_id, self.profile.id)
self.assertIsNone(error)
self.assertFalse(slot_reserved)
mock_reserve.assert_not_called()
@patch("apps.channels.models.RedisClient.get_client")
@patch("apps.channels.models.reserve_profile_slot")
def test_reuses_assignment_during_init_before_metadata(
self, mock_reserve, mock_get_client
):
mock_get_client.return_value = self.redis
self._seed_assignment()
stream_id, profile_id, error, slot_reserved = self.channel.get_stream()
self.assertEqual(stream_id, self.stream.id)
self.assertEqual(profile_id, self.profile.id)
self.assertIsNone(error)
self.assertFalse(slot_reserved)
mock_reserve.assert_not_called()
@patch("apps.channels.models.RedisClient.get_client")
@patch("apps.channels.models.release_profile_slot")
@patch("apps.channels.models.reserve_profile_slot")
def test_releases_stale_assignment_when_proxy_stopped(
self, mock_reserve, mock_release, mock_get_client
):
mock_get_client.return_value = self.redis
mock_reserve.return_value = (True, 1, None)
self._seed_assignment()
self.redis.hset(
self.metadata_key,
{ChannelMetadataField.STATE: ChannelState.STOPPED},
)
stream_id, profile_id, error, slot_reserved = self.channel.get_stream()
mock_release.assert_called_once_with(self.profile.id, self.redis)
mock_reserve.assert_called_once()
self.assertEqual(stream_id, self.stream.id)
self.assertEqual(profile_id, self.profile.id)
self.assertTrue(slot_reserved)
@patch("apps.channels.models.RedisClient.get_client")
def test_stream_assignment_is_reusable_during_init_pending(self, mock_get_client):
mock_get_client.return_value = self.redis
self._seed_assignment()
self.assertTrue(
self.channel._stream_assignment_is_reusable(self.redis, self.stream.id)
)
@patch("apps.channels.models.RedisClient.get_client")
def test_stream_assignment_not_reusable_when_stopped(self, mock_get_client):
mock_get_client.return_value = self.redis
self._seed_assignment()
self.redis.hset(
self.metadata_key,
{ChannelMetadataField.STATE: ChannelState.STOPPED},
)
self.assertFalse(
self.channel._stream_assignment_is_reusable(self.redis, self.stream.id)
)

150
apps/channels/tests/test_recording_pipeline.py
View file

@ -365,46 +365,44 @@ class RecordingStatusLifecycleTests(TestCase):
# =========================================================================
class ConcatFlagsTests(TestCase):
"""Verify that the finalize phase uses error-tolerant ffmpeg flags
when concatenating pre-restart segments."""
"""Verify error-tolerant FFmpeg flags on the HLS segment concat command."""
def test_concat_command_includes_error_tolerant_flags(self):
"""Inspect the source code to confirm error-tolerant flags are present.
This is a static analysis test — no ffmpeg execution needed."""
def test_hls_concat_cmd_includes_error_tolerant_flags(self):
from apps.channels.tasks import _dvr_build_hls_concat_cmd
cmd = _dvr_build_hls_concat_cmd("/data/concat.txt", "/data/out.mkv")
self.assertIn("+genpts+igndts+discardcorrupt", cmd)
self.assertIn("-err_detect", cmd)
self.assertEqual(cmd[cmd.index("-err_detect") + 1], "ignore_err")
self.assertIn("-avoid_negative_ts", cmd)
self.assertEqual(cmd[cmd.index("-avoid_negative_ts") + 1], "make_zero")
self.assertIn("concat", cmd)
self.assertEqual(cmd[-1], "/data/out.mkv")
def test_hls_concat_cmd_supports_mp4_fallback_extra_args(self):
from apps.channels.tasks import _dvr_build_hls_concat_cmd
cmd = _dvr_build_hls_concat_cmd(
"/data/concat.txt",
"/data/intermediate.mp4",
extra_args=["-bsf:a", "aac_adtstoasc"],
)
self.assertIn("aac_adtstoasc", cmd)
self.assertEqual(cmd[-1], "/data/intermediate.mp4")
def test_run_recording_uses_hls_concat_helper(self):
import inspect
from apps.channels.tasks import run_recording
source = inspect.getsource(run_recording)
self.assertIn("_dvr_build_hls_concat_cmd", source)
# The concat subprocess.run call must include these flags
self.assertIn("+genpts+igndts+discardcorrupt", source,
"Concat must use +genpts+igndts+discardcorrupt fflags")
self.assertIn("ignore_err", source,
"Concat must use -err_detect ignore_err")
self.assertIn("-f", source)
self.assertIn("concat", source)
def test_concat_goes_directly_to_mkv(self):
"""Concat must produce MKV directly (not intermediate .ts) to
preserve timestamp boundaries and avoid playback freeze at splice."""
def test_recover_recordings_uses_hls_concat_helper(self):
import inspect
from apps.channels.tasks import run_recording
source = inspect.getsource(run_recording)
from apps.channels.tasks import recover_recordings_on_startup
# Must contain reset_timestamps for proper segment boundary handling
self.assertIn("reset_timestamps", source,
"Concat must use -reset_timestamps 1 for seamless seeking")
# Must write directly to final_path (MKV), not an intermediate .ts
self.assertIn("_concat_did_remux", source,
"Concat path must set flag to skip separate remux step")
def test_segment_time_metadata_present(self):
"""Verify concat uses -segment_time_metadata for boundary awareness."""
import inspect
from apps.channels.tasks import run_recording
source = inspect.getsource(run_recording)
self.assertIn("segment_time_metadata", source,
"Concat must use -segment_time_metadata 1 for segment boundary handling")
source = inspect.getsource(recover_recordings_on_startup)
self.assertIn("_dvr_build_hls_concat_cmd", source)
# =========================================================================
@ -486,6 +484,92 @@ class RecoverySkipListTests(TestCase):
mock_run.apply_async.assert_not_called()
# =========================================================================
# 7. FFmpeg in-process retry loop
# =========================================================================
class FfmpegRetryTests(TestCase):
"""Verify FFmpeg restart logic for mid-recording crashes and stalls."""
def test_ffmpeg_retry_constants_and_helpers_exist(self):
from apps.channels import tasks as dvr_tasks
self.assertGreater(dvr_tasks._dvr_ffmpeg_retry_window_seconds(), 0)
self.assertEqual(dvr_tasks._dvr_count_hls_segments(None), 0)
self.assertEqual(dvr_tasks._dvr_count_hls_segments("/nonexistent"), 0)
self.assertEqual(dvr_tasks._dvr_ffmpeg_retry_backoff_seconds(1), 0.25)
self.assertEqual(dvr_tasks._dvr_ffmpeg_retry_backoff_seconds(12), 3.0)
@patch("apps.proxy.live_proxy.config_helper.ConfigHelper.stream_timeout", return_value=60)
@patch("apps.proxy.live_proxy.config_helper.ConfigHelper.failover_grace_period", return_value=20)
def test_retry_window_matches_live_proxy_timeouts(self, _grace, _stream):
from apps.channels.tasks import _dvr_ffmpeg_retry_window_seconds
self.assertEqual(_dvr_ffmpeg_retry_window_seconds(), 80.0)
def test_hls_start_number_zero_when_playlist_exists(self):
import tempfile
from apps.channels.tasks import _dvr_hls_start_number
with tempfile.TemporaryDirectory() as tmp:
m3u8 = os.path.join(tmp, "index.m3u8")
open(os.path.join(tmp, "seg_00000.ts"), "wb").write(b"\x00")
open(os.path.join(tmp, "seg_00013.ts"), "wb").write(b"\x00")
with open(m3u8, "w") as f:
f.write("#EXTM3U\n#EXT-X-TARGETDURATION:4\n")
f.write("seg_00000.ts\nseg_00013.ts\n")
# append_list reloads playlist entries; start_number must stay 0.
self.assertEqual(_dvr_hls_start_number(tmp, m3u8), 0)
def test_hls_start_number_from_max_index_without_playlist(self):
import tempfile
from apps.channels.tasks import _dvr_hls_start_number
with tempfile.TemporaryDirectory() as tmp:
open(os.path.join(tmp, "seg_00000.ts"), "wb").write(b"\x00")
open(os.path.join(tmp, "seg_00013.ts"), "wb").write(b"\x00")
self.assertEqual(_dvr_hls_start_number(tmp, os.path.join(tmp, "index.m3u8")), 14)
def test_hls_start_number_zero_on_fresh_dir(self):
import tempfile
from apps.channels.tasks import _dvr_hls_start_number
with tempfile.TemporaryDirectory() as tmp:
self.assertEqual(_dvr_hls_start_number(tmp, os.path.join(tmp, "index.m3u8")), 0)
def test_build_ffmpeg_cmd_continues_hls_numbering(self):
from apps.channels.tasks import _dvr_build_ffmpeg_cmd
cmd = _dvr_build_ffmpeg_cmd(
"http://127.0.0.1:5656/proxy/ts/stream/uuid",
71,
"/data/recordings/.dvr_71_hls/index.m3u8",
"/data/recordings/.dvr_71_hls/seg_%05d.ts",
42,
)
self.assertIn("-start_number", cmd)
self.assertEqual(cmd[cmd.index("-start_number") + 1], "42")
hls_flags = cmd[cmd.index("-hls_flags") + 1]
self.assertIn("append_list", hls_flags)
self.assertIn("omit_endlist", hls_flags)
self.assertIn("-err_detect", cmd)
self.assertEqual(cmd[cmd.index("-err_detect") + 1], "ignore_err")
def test_run_recording_has_retry_loop(self):
import inspect
from apps.channels.tasks import run_recording
source = inspect.getsource(run_recording)
self.assertIn("_ffmpeg_retry_count", source)
self.assertIn("_ffmpeg_outage_started", source)
self.assertIn("_ffmpeg_retry_window", source)
self.assertIn("_break_reason", source)
self.assertIn("ffmpeg_outage_window_exhausted", source)
self.assertIn("_dvr_build_ffmpeg_cmd", source)
self.assertIn("_dvr_hls_start_number", source)
self.assertIn("_ffmpeg_retry_count = 0", source)
# =========================================================================
# 6. Frontend red-dot filter (guideUtils.mapRecordingsByProgramId)
# =========================================================================

190
apps/channels/utils.py
View file

@ -1,5 +1,12 @@
import logging
import threading
logger = logging.getLogger(__name__)
# Bound memory/DB work per chunk for large libraries (20k+ channels).
EPG_LOGO_APPLY_BATCH_SIZE = 500
EPG_LOGO_APPLY_MAX_ERRORS = 100
lock = threading.Lock()
# Dictionary to track usage: {account_id: current_usage}
active_streams_map = {}
@ -57,3 +64,186 @@ def decrement_stream_count(account):
active_streams_map[account.id] = current_usage
account.active_streams = current_usage
account.save(update_fields=['active_streams'])
def auto_apply_epg_logos_enabled(custom_properties):
"""Return whether channel logos should be auto-applied after EPG refresh."""
return bool((custom_properties or {}).get('auto_apply_epg_logos', False))
def _empty_logo_apply_stats():
return {
'updated_count': 0,
'created_logos_count': 0,
'error_count': 0,
'errors': [],
}
def _merge_logo_apply_stats(accumulated, batch_stats):
accumulated['updated_count'] += batch_stats['updated_count']
accumulated['created_logos_count'] += batch_stats['created_logos_count']
accumulated['error_count'] += batch_stats['error_count']
remaining = EPG_LOGO_APPLY_MAX_ERRORS - len(accumulated['errors'])
if remaining > 0:
accumulated['errors'].extend(batch_stats['errors'][:remaining])
return accumulated
def apply_logos_from_epg_icon_url(channels):
"""
Set channel.logo from epg_data.icon_url for the given channels.
Expects channels to be pre-filtered with select_related('epg_data', 'logo').
Uses bulk logo lookup/create and a single channel bulk_update for efficiency.
"""
from .models import Channel, Logo
work = []
url_to_meta = {}
for channel in channels:
if not channel.epg_data:
continue
icon_url = (channel.epg_data.icon_url or '').strip()
if not icon_url:
continue
if channel.logo and channel.logo.url == icon_url:
continue
work.append((channel, icon_url))
if icon_url not in url_to_meta:
url_to_meta[icon_url] = (
channel.epg_data.name,
channel.epg_data.tvg_id,
)
if not work:
return _empty_logo_apply_stats()
unique_urls = list(url_to_meta.keys())
logo_by_url = {
logo.url: logo
for logo in Logo.objects.filter(url__in=unique_urls)
}
missing_urls = [url for url in unique_urls if url not in logo_by_url]
created_logos_count = 0
if missing_urls:
logos_to_create = [
Logo(
name=(url_to_meta[url][0] or f"Logo for {url_to_meta[url][1]}"),
url=url,
)
for url in missing_urls
]
created_logos_count = len(logos_to_create)
Logo.objects.bulk_create(logos_to_create, ignore_conflicts=True)
for logo in Logo.objects.filter(url__in=unique_urls):
logo_by_url[logo.url] = logo
channels_to_update = []
errors = []
for channel, icon_url in work:
logo = logo_by_url.get(icon_url)
if not logo:
errors.append(f"Channel {channel.id}: Logo not found for {icon_url}")
continue
if channel.logo_id != logo.id:
channel.logo = logo
channels_to_update.append(channel)
if channels_to_update:
Channel.objects.bulk_update(channels_to_update, ['logo'], batch_size=500)
return {
'updated_count': len(channels_to_update),
'created_logos_count': created_logos_count,
'error_count': len(errors),
'errors': errors,
}
def channels_with_epg_icon_queryset(*, epg_source=None, epg_source_id=None):
"""Channels mapped to a source that have a non-empty EPG icon URL."""
from .models import Channel
qs = Channel.objects.filter(epg_data__isnull=False)
if epg_source is not None:
qs = qs.filter(epg_data__epg_source=epg_source)
elif epg_source_id is not None:
qs = qs.filter(epg_data__epg_source_id=epg_source_id)
else:
raise ValueError("epg_source or epg_source_id is required")
return qs.exclude(
epg_data__icon_url__isnull=True,
).exclude(
epg_data__icon_url='',
)
def apply_logos_from_epg_queryset(channels_qs, *, batch_size=EPG_LOGO_APPLY_BATCH_SIZE):
"""
Apply logos for a potentially large queryset without loading every row at once.
Streams channel IDs from the database and processes fixed-size chunks.
"""
from .models import Channel
stats = _empty_logo_apply_stats()
batch_ids = []
id_stream = channels_qs.order_by('id').values_list('id', flat=True).iterator(
chunk_size=batch_size,
)
for channel_id in id_stream:
batch_ids.append(channel_id)
if len(batch_ids) >= batch_size:
batch = Channel.objects.filter(
id__in=batch_ids,
).select_related('epg_data', 'logo')
_merge_logo_apply_stats(stats, apply_logos_from_epg_icon_url(batch))
batch_ids = []
if batch_ids:
batch = Channel.objects.filter(
id__in=batch_ids,
).select_related('epg_data', 'logo')
_merge_logo_apply_stats(stats, apply_logos_from_epg_icon_url(batch))
return stats
def apply_logos_from_epg_for_source(epg_source, *, batch_size=EPG_LOGO_APPLY_BATCH_SIZE):
"""Apply EPG icon URLs to all channels mapped to the given EPG source."""
channels_qs = channels_with_epg_icon_queryset(epg_source=epg_source)
return apply_logos_from_epg_queryset(channels_qs, batch_size=batch_size)
def maybe_auto_apply_epg_logos(epg_source):
"""Auto-apply logos after refresh when enabled on the source. Non-fatal on error."""
if not auto_apply_epg_logos_enabled(epg_source.custom_properties):
return None
try:
stats = apply_logos_from_epg_for_source(epg_source)
if stats['updated_count'] or stats['created_logos_count']:
logger.info(
"Auto-applied EPG logos for source %s: updated %s channels, "
"created %s logos.",
epg_source.name,
stats['updated_count'],
stats['created_logos_count'],
)
else:
logger.info(
"Auto-apply EPG logos for source %s: all matched channels already current.",
epg_source.name,
)
return stats
except Exception as logo_error:
logger.warning(
"EPG logo auto-apply failed for source %s (non-fatal): %s",
epg_source.name,
logo_error,
exc_info=True,
)
return None

447
apps/epg/api_views.py
View file

@ -1,6 +1,11 @@
import logging, os, re
import hashlib
import logging
import os
import re
import time
from rest_framework import viewsets, status, serializers
from rest_framework.pagination import PageNumberPagination
from rest_framework.permissions import AllowAny
from rest_framework.response import Response
from rest_framework.views import APIView
from rest_framework.decorators import action
@ -22,6 +27,7 @@ from .tasks import refresh_epg_data, find_current_program_for_tvg_id
from .query_utils import parse_text_query
from apps.accounts.permissions import (
Authenticated,
IsAdmin,
IsStandardUser,
permission_classes_by_action,
permission_classes_by_method,
@ -47,6 +53,10 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
try:
return [perm() for perm in permission_classes_by_action[self.action]]
except KeyError:
if self.action in ('sd_lineups', 'sd_lineups_search'):
if self.request.method == 'GET':
return [IsStandardUser()]
return [IsAdmin()]
return [Authenticated()]
def get_queryset(self):
@ -110,6 +120,337 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
return super().partial_update(request, *args, **kwargs)
def _sd_authenticate(self, source):
"""
Authenticate with Schedules Direct using stored credentials.
Returns (token, None) on success or (None, Response) on failure.
"""
import hashlib
import requests as http_requests
from apps.epg.tasks import SD_BASE_URL
from version import __version__ as dispatcharr_version
username = (source.username or '').strip()
password = (source.password or '').strip()
if not username or not password:
return None, Response(
{"error": "Username and password are required."},
status=status.HTTP_400_BAD_REQUEST
)
sha1_password = hashlib.sha1(password.encode('utf-8')).hexdigest()
try:
auth_response = http_requests.post(
f"{SD_BASE_URL}/token",
json={'username': username, 'password': sha1_password},
headers={'Content-Type': 'application/json', 'User-Agent': f'Dispatcharr/{dispatcharr_version}'},
timeout=15,
)
auth_response.raise_for_status()
token = auth_response.json().get('token')
if not token:
return None, Response(
{"error": "Authentication failed. Check your credentials."},
status=status.HTTP_401_UNAUTHORIZED
)
return token, None
except http_requests.exceptions.RequestException as e:
return None, Response(
{"error": f"Authentication failed: {str(e)}"},
status=status.HTTP_502_BAD_GATEWAY
)
def _get_sd_reset_at(self, source):
"""Retrieve stored reset timestamp from EPGSource model field."""
reset_at_str = (source.custom_properties or {}).get('sd_changes_reset_at')
return reset_at_str
def _get_sd_changes_remaining(self, source):
"""
Retrieve stored changesRemaining from EPGSource model field.
If a reset timestamp exists and has passed (midnight UTC), clears the
lockout automatically so the user can make adds again.
"""
from django.utils import timezone
cp = source.custom_properties or {}
changes_remaining = cp.get('sd_changes_remaining')
reset_at_str = cp.get('sd_changes_reset_at')
from django.utils.dateparse import parse_datetime
reset_at = parse_datetime(reset_at_str) if reset_at_str else None
# If we have a reset timestamp and it has passed, clear the lockout
if changes_remaining == 0 and reset_at:
if timezone.now() >= reset_at:
cp = source.custom_properties or {}
cp.pop('sd_changes_remaining', None)
cp.pop('sd_changes_reset_at', None)
source.custom_properties = cp
source.save(update_fields=['custom_properties'])
return None
return changes_remaining
def _save_sd_changes_remaining(self, source, changes_remaining):
"""Persist changesRemaining to EPGSource model field."""
cp = source.custom_properties or {}
cp['sd_changes_remaining'] = changes_remaining
source.custom_properties = cp
source.save(update_fields=['custom_properties'])
def _save_sd_lockout(self, source):
"""
Persist a hard lockout to EPGSource custom_properties when SD returns
4100 MAX_LINEUP_CHANGES_REACHED. SD lineup change counters reset at
00:00Z (midnight UTC) per SD's documented behavior — error 4100 states
"lineup changes for today" and all SD rate counters reset at midnight UTC.
Lockout clears automatically when the next midnight UTC passes.
"""
from django.utils import timezone
from datetime import datetime, timedelta, timezone as dt_timezone
now = timezone.now()
# Calculate next midnight UTC — SD resets at 00:00Z not on a rolling window
tomorrow = (now + timedelta(days=1)).replace(
hour=0, minute=0, second=0, microsecond=0,
tzinfo=dt_timezone.utc
)
reset_at = tomorrow
cp = source.custom_properties or {}
cp['sd_changes_remaining'] = 0
cp['sd_changes_reset_at'] = reset_at.isoformat()
source.custom_properties = cp
source.save(update_fields=['custom_properties'])
logger.warning(
f"SD source {source.id}: daily add limit reached (4100). "
f"Lockout set until {reset_at.isoformat()}."
)
@action(detail=True, methods=["get", "post", "delete"], url_path="sd-lineups")
def sd_lineups(self, request, pk=None):
"""
GET — list lineups currently on the SD account
POST — add a lineup (body: {"lineup": "USA-NJ29486-X"})
DELETE — remove a lineup (body: {"lineup": "USA-NJ29486-X"})
"""
import requests as http_requests
from apps.epg.tasks import SD_BASE_URL
from version import __version__ as dispatcharr_version
source = self.get_object()
if source.source_type != 'schedules_direct':
return Response(
{"error": "This action is only available for Schedules Direct sources."},
status=status.HTTP_400_BAD_REQUEST
)
token, error = self._sd_authenticate(source)
if error:
return error
headers = {
'Content-Type': 'application/json',
'User-Agent': f'Dispatcharr/{dispatcharr_version}',
'token': token,
}
if request.method == "GET":
try:
resp = http_requests.get(
f"{SD_BASE_URL}/lineups",
headers=headers,
timeout=15,
)
if resp.status_code == 400:
sd_data = resp.json()
sd_code = sd_data.get('code')
if sd_code == 4102:
return Response({
"lineups": [],
"max_lineups": 4,
"changes_remaining": self._get_sd_changes_remaining(source),
"changes_reset_at": self._get_sd_reset_at(source),
"notice": "No lineups are currently configured on this Schedules Direct account. Use the search below to add one.",
})
resp.raise_for_status()
data = resp.json()
lineups = [l for l in data.get('lineups', []) if not l.get('isDeleted', False)]
return Response({
"lineups": lineups,
"max_lineups": 4,
"changes_remaining": self._get_sd_changes_remaining(source),
"changes_reset_at": self._get_sd_reset_at(source),
})
except http_requests.exceptions.RequestException as e:
return Response(
{"error": f"Failed to fetch lineups: {str(e)}"},
status=status.HTTP_502_BAD_GATEWAY
)
elif request.method == "POST":
lineup_id = request.data.get('lineup')
if not lineup_id:
return Response({"error": "lineup field is required."}, status=status.HTTP_400_BAD_REQUEST)
try:
resp = http_requests.put(
f"{SD_BASE_URL}/lineups/{lineup_id}",
headers=headers,
timeout=15,
)
sd_data = resp.json()
sd_code = sd_data.get('code')
if resp.status_code == 400 or resp.status_code == 403:
if sd_code == 4100:
self._save_sd_lockout(source)
return Response({
"error": "daily_limit_reached",
"message": "You have reached your daily Schedules Direct lineup addition limit. SD allows 6 adds per 24-hour period. Resets at midnight UTC.",
"changes_remaining": 0,
"docs_url": "https://github.com/SchedulesDirect/JSON-Service/wiki/API-20141201#tasks-your-client-must-perform",
}, status=status.HTTP_200_OK)
if sd_code == 4101:
return Response({
"error": "max_lineups_reached",
"message": "Your Schedules Direct account has reached the maximum of 4 lineups. Remove one before adding another.",
"changes_remaining": self._get_sd_changes_remaining(source),
}, status=status.HTTP_200_OK)
if sd_code == 2100:
return Response({
"error": "duplicate_lineup",
"message": "This lineup is already on your Schedules Direct account.",
"changes_remaining": self._get_sd_changes_remaining(source),
}, status=status.HTTP_200_OK)
return Response({
"error": sd_data.get('message', 'Failed to add lineup.'),
"changes_remaining": self._get_sd_changes_remaining(source),
}, status=status.HTTP_200_OK)
resp.raise_for_status()
# Persist changesRemaining to custom_properties
changes_remaining = sd_data.get('changesRemaining')
if changes_remaining is not None:
self._save_sd_changes_remaining(source, changes_remaining)
logger.info(
f"SD lineup added for source {source.id}: {lineup_id}. "
f"changesRemaining: {changes_remaining}"
)
# Re-fetch stations so the new lineup's stations are available for matching
from apps.epg.tasks import fetch_schedules_direct_stations
fetch_schedules_direct_stations.delay(source.id)
return Response({
**sd_data,
"changes_remaining": changes_remaining,
})
except http_requests.exceptions.RequestException as e:
return Response(
{"error": f"Failed to add lineup: {str(e)}"},
status=status.HTTP_502_BAD_GATEWAY
)
elif request.method == "DELETE":
lineup_id = request.data.get('lineup')
if not lineup_id:
return Response({"error": "lineup field is required."}, status=status.HTTP_400_BAD_REQUEST)
try:
resp = http_requests.delete(
f"{SD_BASE_URL}/lineups/{lineup_id}",
headers=headers,
timeout=15,
)
if resp.status_code == 400:
sd_data = resp.json()
sd_code = sd_data.get('code')
if sd_code == 2103:
return Response({
"response": "OK",
"code": 0,
"message": "Lineup not found on account — already removed.",
"changes_remaining": self._get_sd_changes_remaining(source),
})
resp.raise_for_status()
sd_data = resp.json()
# SD returns changesRemaining on deletes — persist it
changes_remaining = sd_data.get('changesRemaining')
if changes_remaining is not None:
self._save_sd_changes_remaining(source, changes_remaining)
logger.info(f"SD lineup deleted for source {source.id}: {lineup_id}")
return Response({
**sd_data,
"changes_remaining": self._get_sd_changes_remaining(source),
})
except http_requests.exceptions.RequestException as e:
return Response(
{"error": f"Failed to remove lineup: {str(e)}"},
status=status.HTTP_502_BAD_GATEWAY
)
@action(detail=True, methods=["post"], url_path="sd-lineups/search")
def sd_lineups_search(self, request, pk=None):
"""
Search available headends/lineups by country and postal code.
Body: {"country": "USA", "postalcode": "07030"}
Returns a flat list of lineups across all matching headends.
"""
import requests as http_requests
from apps.epg.tasks import SD_BASE_URL
from version import __version__ as dispatcharr_version
source = self.get_object()
if source.source_type != 'schedules_direct':
return Response(
{"error": "This action is only available for Schedules Direct sources."},
status=status.HTTP_400_BAD_REQUEST
)
country = request.data.get('country', '').strip()
postalcode = request.data.get('postalcode', '').strip()
if not country or not postalcode:
return Response(
{"error": "country and postalcode are required."},
status=status.HTTP_400_BAD_REQUEST
)
token, error = self._sd_authenticate(source)
if error:
return error
headers = {
'Content-Type': 'application/json',
'User-Agent': f'Dispatcharr/{dispatcharr_version}',
'token': token,
}
try:
resp = http_requests.get(
f"{SD_BASE_URL}/headends",
params={'country': country, 'postalcode': postalcode},
headers=headers,
timeout=15,
)
resp.raise_for_status()
headends = resp.json()
lineups = []
for headend in headends:
for lineup in headend.get('lineups', []):
lineups.append({
'lineup': lineup.get('lineup'),
'name': lineup.get('name'),
'transport': headend.get('transport'),
'location': headend.get('location'),
'headend': headend.get('headend'),
})
return Response({"lineups": lineups})
except http_requests.exceptions.RequestException as e:
return Response(
{"error": f"Failed to search headends: {str(e)}"},
status=status.HTTP_502_BAD_GATEWAY
)
# ─────────────────────────────
# 2) Program API (CRUD)
# ─────────────────────────────
@ -125,7 +466,13 @@ class ProgramViewSet(viewsets.ModelViewSet):
queryset = ProgramData.objects.select_related("epg").all()
serializer_class = ProgramDataSerializer
# Per-source in-memory caches (token and error state)
_sd_poster_token_cache: dict = {}
_sd_poster_error_cache: dict = {}
def get_permissions(self):
if self.action == 'poster':
return [AllowAny()]
try:
return [perm() for perm in permission_classes_by_action[self.action]]
except KeyError:
@ -141,6 +488,101 @@ class ProgramViewSet(viewsets.ModelViewSet):
serializer = self.get_serializer(instance)
return Response(serializer.data)
@action(detail=True, methods=['get'], url_path='poster', permission_classes=[AllowAny])
def poster(self, request, pk=None):
"""Proxy endpoint for SD program poster images. Nginx caches the response."""
import requests as http_requests
from apps.epg.tasks import SD_BASE_URL
program = self.get_object()
poster_sd_url = (program.custom_properties or {}).get('sd_icon')
if not poster_sd_url:
return Response(status=status.HTTP_404_NOT_FOUND)
source = program.epg.epg_source if program.epg else None
if not source or source.source_type != 'schedules_direct':
return Response(status=status.HTTP_404_NOT_FOUND)
error_cache = ProgramViewSet._sd_poster_error_cache.get(source.id)
if error_cache and time.time() < error_cache['until']:
return Response(
{'error': f"SD temporarily unavailable: {error_cache['reason']}"},
status=status.HTTP_503_SERVICE_UNAVAILABLE,
)
cached = ProgramViewSet._sd_poster_token_cache.get(source.id)
token = cached['token'] if cached and time.time() < cached['expires'] else None
if not token:
sha1_password = hashlib.sha1(source.password.encode('utf-8')).hexdigest()
try:
from version import __version__ as dispatcharr_version
auth_resp = http_requests.post(
f"{SD_BASE_URL}/token",
json={'username': source.username, 'password': sha1_password},
headers={
'Content-Type': 'application/json',
'User-Agent': f'Dispatcharr/{dispatcharr_version}',
},
timeout=10,
)
auth_data = auth_resp.json()
token = auth_data.get('token')
if not token:
ProgramViewSet._sd_poster_error_cache[source.id] = {
'until': time.time() + 3600,
'reason': auth_data.get('message', 'Authentication failed'),
}
return Response(status=status.HTTP_502_BAD_GATEWAY)
token_expires = auth_data.get('tokenExpires', time.time() + 86400)
ProgramViewSet._sd_poster_token_cache[source.id] = {
'token': token,
'expires': token_expires,
}
except http_requests.exceptions.RequestException:
ProgramViewSet._sd_poster_error_cache[source.id] = {
'until': time.time() + 300,
'reason': 'Network error reaching Schedules Direct',
}
return Response(status=status.HTTP_502_BAD_GATEWAY)
try:
img_resp = http_requests.get(
poster_sd_url,
headers={'token': token},
timeout=15,
allow_redirects=True,
)
if img_resp.status_code in (401, 403):
ProgramViewSet._sd_poster_token_cache.pop(source.id, None)
ProgramViewSet._sd_poster_error_cache[source.id] = {
'until': time.time() + 3600,
'reason': f'SD returned {img_resp.status_code}',
}
return Response(status=status.HTTP_502_BAD_GATEWAY)
if img_resp.status_code == 400:
try:
err_code = img_resp.json().get('code')
except Exception:
err_code = None
if err_code == 5002:
ProgramViewSet._sd_poster_error_cache[source.id] = {
'until': time.time() + 3600,
'reason': 'Daily image download limit reached (SD error 5002)',
}
return Response(status=status.HTTP_429_TOO_MANY_REQUESTS)
if img_resp.status_code != 200:
return Response(status=status.HTTP_502_BAD_GATEWAY)
from django.http import HttpResponse
content_type = img_resp.headers.get('Content-Type', 'image/jpeg')
response = HttpResponse(img_resp.content, content_type=content_type)
response['Cache-Control'] = 'public, max-age=86400'
return response
except http_requests.exceptions.RequestException:
return Response(status=status.HTTP_502_BAD_GATEWAY)
def list(self, request, *args, **kwargs):
logger.debug("Listing all EPG programs.")
return super().list(request, *args, **kwargs)
@ -665,6 +1107,7 @@ class EPGImportAPIView(APIView):
def post(self, request, format=None):
logger.info("EPGImportAPIView: Received request to import EPG data.")
epg_id = request.data.get("id", None)
force = bool(request.data.get("force", False))
# Check if this is a dummy EPG source
try:
@ -679,7 +1122,7 @@ class EPGImportAPIView(APIView):
except EPGSource.DoesNotExist:
pass # Let the task handle the missing source
refresh_epg_data.delay(epg_id) # Trigger Celery task
refresh_epg_data.delay(epg_id, force=force) # Trigger Celery task
logger.info("EPGImportAPIView: Task dispatched to refresh EPG data.")
return Response(
{"success": True, "message": "EPG data refresh initiated."},

65
apps/epg/migrations/0024_remove_epgsource_api_key_epgsource_password_and_more.py Normal file
View file

@ -0,0 +1,65 @@
# Generated by Django 6.0.5 on 2026-05-31 16:48
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('epg', '0023_epgsource_programme_index'),
]
operations = [
migrations.RemoveField(
model_name='epgsource',
name='api_key',
),
migrations.AddField(
model_name='epgsource',
name='password',
field=models.CharField(blank=True, help_text='Password for credential-based EPG sources (e.g. Schedules Direct)', max_length=255, null=True),
),
migrations.AddField(
model_name='epgsource',
name='username',
field=models.CharField(blank=True, help_text='Username for credential-based EPG sources (e.g. Schedules Direct)', max_length=255, null=True),
),
migrations.AddField(
model_name='programdata',
name='program_id',
field=models.CharField(blank=True, help_text='Schedules Direct programID (e.g. EP123456789). Null for XMLTV sources.', max_length=64, null=True),
),
migrations.AlterField(
model_name='epgsource',
name='custom_properties',
field=models.JSONField(blank=True, default=dict, help_text='Custom properties for source-specific configuration', null=True),
),
migrations.CreateModel(
name='SDProgramMD5',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('program_id', models.CharField(help_text='Schedules Direct programID (e.g. EP123456789)', max_length=64)),
('md5', models.CharField(help_text='MD5 hash of the program metadata from Schedules Direct', max_length=22)),
('epg_source', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='sd_program_md5s', to='epg.epgsource')),
],
options={
'unique_together': {('epg_source', 'program_id')},
},
),
migrations.CreateModel(
name='SDScheduleMD5',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('station_id', models.CharField(help_text='Schedules Direct stationID', max_length=20)),
('date', models.DateField(help_text='Schedule date (UTC)')),
('md5', models.CharField(help_text='MD5 hash of the schedule for this station/date from Schedules Direct', max_length=22)),
('last_modified', models.DateTimeField(help_text='Last modified timestamp from Schedules Direct')),
('epg_source', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='sd_schedule_md5s', to='epg.epgsource')),
],
options={
'indexes': [models.Index(fields=['epg_source', 'station_id'], name='epg_sdsched_epg_sou_0d700e_idx')],
'unique_together': {('epg_source', 'station_id', 'date')},
},
),
]

94
apps/epg/models.py
View file

@ -30,7 +30,10 @@ class EPGSource(models.Model):
name = models.CharField(max_length=255, unique=True)
source_type = models.CharField(max_length=20, choices=SOURCE_TYPE_CHOICES)
url = models.URLField(max_length=1000, blank=True, null=True) # For XMLTV
api_key = models.CharField(max_length=255, blank=True, null=True) # For Schedules Direct
username = models.CharField(max_length=255, blank=True, null=True,
help_text='Username for credential-based EPG sources (e.g. Schedules Direct)')
password = models.CharField(max_length=255, blank=True, null=True,
help_text='Password for credential-based EPG sources (e.g. Schedules Direct)')
is_active = models.BooleanField(default=True)
file_path = models.CharField(max_length=1024, blank=True, null=True)
extracted_file_path = models.CharField(max_length=1024, blank=True, null=True,
@ -43,7 +46,7 @@ class EPGSource(models.Model):
default=dict,
blank=True,
null=True,
help_text="Custom properties for dummy EPG configuration (regex patterns, timezone, duration, etc.)"
help_text="Custom properties for source-specific configuration"
)
priority = models.PositiveIntegerField(
default=0,
@ -80,18 +83,13 @@ class EPGSource(models.Model):
def get_cache_file(self):
import mimetypes
# Use a temporary extension for initial download
# The actual extension will be determined after content inspection
file_ext = ".tmp"
# If file_path is already set and contains an extension, use that
# This handles cases where we've already detected the proper type
if self.file_path and os.path.exists(self.file_path):
_, existing_ext = os.path.splitext(self.file_path)
if existing_ext:
file_ext = existing_ext
else:
# Try to detect the MIME type and map to extension
mime_type, _ = mimetypes.guess_type(self.file_path)
if mime_type:
if mime_type == 'application/gzip' or mime_type == 'application/x-gzip':
@ -100,48 +98,33 @@ class EPGSource(models.Model):
file_ext = '.zip'
elif mime_type == 'application/xml' or mime_type == 'text/xml':
file_ext = '.xml'
# For files without mime type detection, try peeking at content
else:
try:
with open(self.file_path, 'rb') as f:
header = f.read(4)
# Check for gzip magic number (1f 8b)
if header[:2] == b'\x1f\x8b':
file_ext = '.gz'
# Check for zip magic number (PK..)
elif header[:2] == b'PK':
file_ext = '.zip'
# Check for XML
elif header[:5] == b'<?xml' or header[:5] == b'<tv>':
file_ext = '.xml'
except Exception as e:
# If we can't read the file, just keep the default extension
except Exception:
pass
filename = f"{self.id}{file_ext}"
# Build full path in MEDIA_ROOT/cached_epg
cache_dir = os.path.join(settings.MEDIA_ROOT, "cached_epg")
# Create directory if it doesn't exist
os.makedirs(cache_dir, exist_ok=True)
cache = os.path.join(cache_dir, filename)
return cache
def save(self, *args, **kwargs):
# Prevent auto_now behavior by handling updated_at manually
if 'update_fields' in kwargs and 'updated_at' not in kwargs['update_fields']:
# Don't modify updated_at for regular updates
kwargs.setdefault('update_fields', [])
if 'updated_at' in kwargs['update_fields']:
kwargs['update_fields'].remove('updated_at')
super().save(*args, **kwargs)
class EPGData(models.Model):
# Removed the Channel foreign key. We now just store the original tvg_id
# and a name (which might simply be the tvg_id if no real channel exists).
tvg_id = models.CharField(max_length=255, null=True, blank=True, db_index=True)
name = models.CharField(max_length=512)
icon_url = models.URLField(max_length=500, null=True, blank=True)
@ -160,7 +143,6 @@ class EPGData(models.Model):
return f"EPG Data for {self.name}"
class ProgramData(models.Model):
# Each programme is associated with an EPGData record.
epg = models.ForeignKey(EPGData, on_delete=models.CASCADE, related_name="programs")
start_time = models.DateTimeField()
end_time = models.DateTimeField()
@ -168,7 +150,71 @@ class ProgramData(models.Model):
sub_title = models.TextField(blank=True, null=True)
description = models.TextField(blank=True, null=True)
tvg_id = models.CharField(max_length=255, null=True, blank=True)
program_id = models.CharField(max_length=64, null=True, blank=True, help_text='Schedules Direct programID (e.g. EP123456789). Null for XMLTV sources.')
custom_properties = models.JSONField(default=dict, blank=True, null=True)
def __str__(self):
return f"{self.title} ({self.start_time} - {self.end_time})"
class SDScheduleMD5(models.Model):
"""
Caches per-station per-date MD5 hashes from Schedules Direct.
Used to detect schedule changes and avoid unnecessary re-downloads,
minimizing API calls against SD's rate-limited endpoints.
"""
epg_source = models.ForeignKey(
EPGSource,
on_delete=models.CASCADE,
related_name="sd_schedule_md5s",
)
station_id = models.CharField(
max_length=20,
help_text="Schedules Direct stationID"
)
date = models.DateField(
help_text="Schedule date (UTC)"
)
md5 = models.CharField(
max_length=22,
help_text="MD5 hash of the schedule for this station/date from Schedules Direct"
)
last_modified = models.DateTimeField(
help_text="Last modified timestamp from Schedules Direct"
)
class Meta:
unique_together = ('epg_source', 'station_id', 'date')
indexes = [
models.Index(fields=['epg_source', 'station_id']),
]
def __str__(self):
return f"SDScheduleMD5: {self.station_id} / {self.date} ({self.epg_source.name})"
class SDProgramMD5(models.Model):
"""
Caches per-program MD5 hashes from Schedules Direct.
Keyed by epg_source + program_id (SD's programID e.g. EP123456789).
Used for program-level delta detection to avoid re-downloading unchanged
program metadata, minimizing API calls against SD's rate-limited endpoints.
"""
epg_source = models.ForeignKey(
EPGSource,
on_delete=models.CASCADE,
related_name="sd_program_md5s",
)
program_id = models.CharField(
max_length=64,
help_text="Schedules Direct programID (e.g. EP123456789)"
)
md5 = models.CharField(
max_length=22,
help_text="MD5 hash of the program metadata from Schedules Direct"
)
class Meta:
unique_together = ('epg_source', 'program_id')
def __str__(self):
return f"SDProgramMD5: {self.program_id} ({self.epg_source.name})"

33
apps/epg/serializers.py
View file

@ -1,4 +1,4 @@
from core.utils import validate_flexible_url
from core.utils import validate_flexible_url, build_absolute_uri_with_port
from rest_framework import serializers
from .models import EPGSource, EPGData, ProgramData
from apps.channels.models import Channel, Stream
@ -22,7 +22,8 @@ class EPGSourceSerializer(serializers.ModelSerializer):
'name',
'source_type',
'url',
'api_key',
'username',
'password',
'is_active',
'file_path',
'refresh_interval',
@ -36,6 +37,7 @@ class EPGSourceSerializer(serializers.ModelSerializer):
'epg_data_count',
'has_channels',
]
extra_kwargs = {'password': {'write_only': True}}
def get_epg_data_count(self, obj):
"""Return the count of EPG data entries instead of all IDs to prevent large payloads"""
@ -66,6 +68,8 @@ class EPGSourceSerializer(serializers.ModelSerializer):
cron_expr = f'{ct.minute} {ct.hour} {ct.day_of_month} {ct.month_of_year} {ct.day_of_week}'
instance._cron_expression = cron_expr
for attr, value in validated_data.items():
if attr == 'password' and not value:
continue
setattr(instance, attr, value)
instance.save()
return instance
@ -142,6 +146,20 @@ class ProgramDetailSerializer(ProgramDataSerializer):
previously_shown = cp.get('previously_shown_details') or {}
data['original_air_date'] = previously_shown.get('start')
# Content advisory (SD)
data['content_advisory'] = cp.get('content_advisory') or []
# Full content ratings array (SD — all regional ratings)
data['content_ratings'] = cp.get('content_ratings') or []
# Sports event details (SD)
data['event_details'] = cp.get('event_details')
# Runtime (duration without commercials)
length = cp.get('length') or {}
data['runtime'] = length.get('value') if length else None
data['runtime_units'] = length.get('units') if length else None
# External IDs
data['imdb_id'] = cp.get('imdb.com_id')
data['tmdb_id'] = cp.get('themoviedb.org_id')
@ -151,6 +169,17 @@ class ProgramDetailSerializer(ProgramDataSerializer):
data['icon'] = cp.get('icon')
data['images'] = cp.get('images') or []
# SD poster: expose as absolute proxy URL so frontend/img tags never need SD auth
if cp.get('sd_icon'):
poster_path = f"/api/epg/programs/{obj.id}/poster/"
request = self.context.get('request')
if request:
data['poster_url'] = build_absolute_uri_with_port(request, poster_path)
else:
data['poster_url'] = poster_path
else:
data['poster_url'] = None
return data

13
apps/epg/signals.py
View file

@ -1,7 +1,7 @@
from django.db.models.signals import post_save, post_delete, pre_save
from django.dispatch import receiver
from .models import EPGSource, EPGData
from .tasks import refresh_epg_data, delete_epg_refresh_task_by_id
from .tasks import refresh_epg_data, delete_epg_refresh_task_by_id, fetch_schedules_direct_stations
from core.scheduling import create_or_update_periodic_task, delete_periodic_task
from core.utils import is_protected_path, send_websocket_update
import json
@ -12,9 +12,16 @@ logger = logging.getLogger(__name__)
@receiver(post_save, sender=EPGSource)
def trigger_refresh_on_new_epg_source(sender, instance, created, **kwargs):
# Trigger refresh only if the source is newly created, active, and not a dummy EPG
# Trigger refresh only if the source is newly created, active, and not a dummy EPG.
# For Schedules Direct sources, run a stations-only fetch on creation so the user
# can run Auto-match EPG before committing to a full schedule/program fetch.
# A short countdown gives the frontend time to receive the API response and
# populate the store before WebSocket updates arrive.
if created and instance.is_active and instance.source_type != 'dummy':
refresh_epg_data.delay(instance.id)
if instance.source_type == 'schedules_direct':
fetch_schedules_direct_stations.apply_async((instance.id,), countdown=3)
else:
refresh_epg_data.delay(instance.id)
@receiver(post_save, sender=EPGSource)
def create_dummy_epg_data(sender, instance, created, **kwargs):

1440
apps/epg/tasks.py
View file

File diff suppressed because it is too large Load diff

503
apps/epg/tests/test_schedules_direct.py Normal file
View file

@ -0,0 +1,503 @@
"""
Tests for the Schedules Direct EPG integration.
Covers:
- EPGSource model: username field presence and help text
- EPGSource serializer: username field included in output
- fetch_schedules_direct: credential validation
- fetch_schedules_direct: SHA1 password hashing and token exchange
- fetch_schedules_direct: graceful error handling on auth failure
- parse_schedules_direct_time: correct UTC parsing
- EPG signals: SD sources skip the XMLTV program parser
"""
import hashlib
from datetime import datetime
from unittest.mock import MagicMock, patch
from django.test import TestCase
from django.utils import timezone
from apps.epg.models import EPGSource, EPGData
from apps.epg.serializers import EPGSourceSerializer
# ---------------------------------------------------------------------------
# Model tests
# ---------------------------------------------------------------------------
class EPGSourceUsernameFieldTests(TestCase):
"""EPGSource.username must exist, be nullable, and carry help text."""
def test_username_field_exists(self):
source = EPGSource.objects.create(
name='SD Test',
source_type='schedules_direct',
username='testuser',
password='testpass',
)
source.refresh_from_db()
self.assertEqual(source.username, 'testuser')
def test_username_nullable(self):
source = EPGSource.objects.create(
name='SD Nullable',
source_type='schedules_direct',
)
source.refresh_from_db()
self.assertIsNone(source.username)
def test_username_help_text(self):
field = EPGSource._meta.get_field('username')
self.assertIn('Schedules Direct', field.help_text)
# ---------------------------------------------------------------------------
# Serializer tests
# ---------------------------------------------------------------------------
class EPGSourceSerializerSDTests(TestCase):
"""EPGSourceSerializer must include the username field."""
def test_username_in_serializer_fields(self):
source = EPGSource.objects.create(
name='SD Serializer Test',
source_type='schedules_direct',
username='sduser',
password='sdpass',
)
data = EPGSourceSerializer(source).data
self.assertIn('username', data)
self.assertEqual(data['username'], 'sduser')
def test_password_not_in_serializer_output(self):
source = EPGSource.objects.create(
name='SD API Key Test',
source_type='schedules_direct',
password='secret',
)
data = EPGSourceSerializer(source).data
self.assertNotIn('password', data)
# ---------------------------------------------------------------------------
# fetch_schedules_direct tests
# ---------------------------------------------------------------------------
class FetchSchedulesDirectCredentialTests(TestCase):
"""fetch_schedules_direct must reject sources missing credentials."""
def _make_source(self, username=None, password=None):
return EPGSource.objects.create(
name='SD Cred Test',
source_type='schedules_direct',
username=username,
password=password,
)
def test_missing_username_sets_error_status(self):
from apps.epg.tasks import fetch_schedules_direct
source = self._make_source(username=None, password='pass')
fetch_schedules_direct(source)
source.refresh_from_db()
self.assertEqual(source.status, EPGSource.STATUS_ERROR)
def test_missing_password_sets_error_status(self):
from apps.epg.tasks import fetch_schedules_direct
source = self._make_source(username='user', password=None)
fetch_schedules_direct(source)
source.refresh_from_db()
self.assertEqual(source.status, EPGSource.STATUS_ERROR)
def test_empty_username_sets_error_status(self):
from apps.epg.tasks import fetch_schedules_direct
source = self._make_source(username=' ', password='pass')
fetch_schedules_direct(source)
source.refresh_from_db()
self.assertEqual(source.status, EPGSource.STATUS_ERROR)
class FetchSchedulesDirectAuthTests(TestCase):
"""fetch_schedules_direct must SHA1-hash the password before sending."""
@patch('apps.epg.tasks.requests.post')
@patch('apps.epg.tasks.requests.get')
def test_password_sha1_hashed_in_token_request(self, mock_get, mock_post):
"""The token POST body must contain the SHA1 hash of the plaintext password."""
plaintext = 'mysecretpassword'
expected_hash = hashlib.sha1(plaintext.encode('utf-8')).hexdigest()
# Auth succeeds, status check returns empty data, lineups returns empty
mock_post.return_value = MagicMock(
status_code=200,
json=MagicMock(return_value={'code': 0, 'token': 'tok123'}),
)
mock_get.return_value = MagicMock(
status_code=200,
json=MagicMock(return_value={}),
)
from apps.epg.tasks import fetch_schedules_direct
source = EPGSource.objects.create(
name='SD Hash Test',
source_type='schedules_direct',
username='sduser',
password=plaintext,
)
with patch('apps.epg.tasks.send_epg_update'):
fetch_schedules_direct(source)
# Verify the POST was called and the body contained the hash
self.assertTrue(mock_post.called)
call_kwargs = mock_post.call_args
posted_json = call_kwargs[1].get('json') or call_kwargs[0][1]
self.assertEqual(posted_json.get('password'), expected_hash)
self.assertEqual(posted_json.get('username'), 'sduser')
@patch('apps.epg.tasks.requests.post')
def test_auth_failure_sets_error_status(self, mock_post):
"""A non-zero SD response code must set STATUS_ERROR on the source."""
mock_post.return_value = MagicMock(
status_code=200,
json=MagicMock(return_value={
'code': 3000,
'message': 'Invalid credentials',
}),
)
from apps.epg.tasks import fetch_schedules_direct
source = EPGSource.objects.create(
name='SD Auth Fail',
source_type='schedules_direct',
username='baduser',
password='badpass',
)
with patch('apps.epg.tasks.send_epg_update'):
fetch_schedules_direct(source)
source.refresh_from_db()
self.assertEqual(source.status, EPGSource.STATUS_ERROR)
@patch('apps.epg.tasks.requests.post')
def test_network_error_sets_error_status(self, mock_post):
"""A network-level exception must set STATUS_ERROR and not crash."""
import requests as req_lib
mock_post.side_effect = req_lib.exceptions.ConnectionError('timeout')
from apps.epg.tasks import fetch_schedules_direct
source = EPGSource.objects.create(
name='SD Network Error',
source_type='schedules_direct',
username='user',
password='pass',
)
with patch('apps.epg.tasks.send_epg_update'):
fetch_schedules_direct(source) # Must not raise
source.refresh_from_db()
self.assertEqual(source.status, EPGSource.STATUS_ERROR)
class FetchSchedulesDirectStationsOnlyTests(TestCase):
"""stations_only fetch must signal channel parsing completion to the frontend."""
@patch('apps.epg.tasks.send_epg_update')
@patch('apps.epg.tasks.requests.get')
@patch('apps.epg.tasks.requests.post')
def test_stations_only_sends_parsing_channels_complete(
self, mock_post, mock_get, mock_send_epg_update
):
from apps.epg.tasks import fetch_schedules_direct
mock_post.return_value = MagicMock(
status_code=200,
json=MagicMock(return_value={'code': 0, 'token': 'tok123'}),
)
def get_side_effect(url, **kwargs):
if url.endswith('/status'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={'systemStatus': [{'status': 'Online'}]}),
)
if url.endswith('/lineups'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={
'lineups': [{'lineupID': 'USA-TEST-X'}],
}),
)
if '/lineups/USA-TEST-X' in url:
return MagicMock(
status_code=200,
json=MagicMock(return_value={
'stations': [{
'stationID': '10001',
'name': 'Test Station',
'callsign': 'TEST',
}],
}),
)
raise AssertionError(f'Unexpected GET URL: {url}')
mock_get.side_effect = get_side_effect
source = EPGSource.objects.create(
name='SD Stations Only',
source_type='schedules_direct',
username='sduser',
password='sdpass',
)
fetch_schedules_direct(source, stations_only=True)
source.refresh_from_db()
self.assertEqual(source.status, EPGSource.STATUS_SUCCESS)
self.assertEqual(EPGData.objects.filter(epg_source=source).count(), 1)
parsing_channel_complete = [
c
for c in mock_send_epg_update.call_args_list
if c[0][1] == 'parsing_channels' and c[0][2] == 100
]
self.assertEqual(len(parsing_channel_complete), 1)
complete_call = parsing_channel_complete[0]
self.assertEqual(complete_call[0][0], source.id)
self.assertEqual(complete_call[1]['status'], 'success')
self.assertEqual(complete_call[1]['channels_count'], 1)
# ---------------------------------------------------------------------------
# parse_schedules_direct_time tests
# ---------------------------------------------------------------------------
class ParseSchedulesDirectTimeTests(TestCase):
"""parse_schedules_direct_time must parse SD ISO timestamps to UTC-aware datetimes."""
def test_parses_valid_timestamp(self):
from apps.epg.tasks import parse_schedules_direct_time
result = parse_schedules_direct_time('2026-05-16T20:00:00Z')
self.assertEqual(result.year, 2026)
self.assertEqual(result.month, 5)
self.assertEqual(result.day, 16)
self.assertEqual(result.hour, 20)
self.assertIsNotNone(result.tzinfo)
def test_result_is_utc_aware(self):
from apps.epg.tasks import parse_schedules_direct_time
result = parse_schedules_direct_time('2026-01-01T00:00:00Z')
# Should be timezone-aware
self.assertIsNotNone(result.tzinfo)
def test_raises_on_invalid_format(self):
from apps.epg.tasks import parse_schedules_direct_time
with self.assertRaises(Exception):
parse_schedules_direct_time('not-a-timestamp')
# ---------------------------------------------------------------------------
# Signal tests
# ---------------------------------------------------------------------------
class SDSourceSignalTests(TestCase):
"""SD EPG sources must skip the XMLTV program parser signal."""
@patch('apps.channels.signals.parse_programs_for_tvg_id')
def test_sd_source_skips_xmltv_parse_on_channel_create(self, mock_parse):
"""Creating a channel linked to an SD EPG source must not trigger
the XMLTV program parser — SD data is handled by fetch_schedules_direct."""
from apps.epg.models import EPGData
from apps.channels.models import Channel
sd_source = EPGSource.objects.create(
name='SD Signal Test',
source_type='schedules_direct',
username='u',
password='p',
)
epg_data = EPGData.objects.create(
tvg_id='sd-test-station',
name='SD Test Station',
epg_source=sd_source,
)
Channel.objects.create(
name='SD Channel',
epg_data=epg_data,
)
mock_parse.delay.assert_not_called()
# ---------------------------------------------------------------------------
# Poster selection tests
# ---------------------------------------------------------------------------
class SDPosterSelectionTests(TestCase):
"""_sd_pick_poster_url must honour style preference with sensible fallbacks."""
def _images(self):
return [
{
'uri': 'assets/iconic_portrait.jpg',
'width': '960',
'aspect': '2x3',
'category': 'Iconic',
},
{
'uri': 'assets/banner_portrait.jpg',
'width': '360',
'aspect': '2x3',
'category': 'Banner-L1',
},
{
'uri': 'assets/iconic_landscape.jpg',
'width': '1920',
'aspect': '16x9',
'category': 'Iconic',
},
{
'uri': 'assets/banner_landscape.jpg',
'width': '1280',
'aspect': '16x9',
'category': 'Banner-L1',
},
]
def test_portrait_iconic_prefers_iconic_over_banner(self):
from apps.epg.tasks import _sd_pick_poster_url
self.assertEqual(
_sd_pick_poster_url(self._images(), 'portrait_iconic'),
'assets/iconic_portrait.jpg',
)
def test_portrait_banner_prefers_banner(self):
from apps.epg.tasks import _sd_pick_poster_url
self.assertEqual(
_sd_pick_poster_url(self._images(), 'portrait_banner'),
'assets/banner_portrait.jpg',
)
def test_landscape_iconic_prefers_landscape_iconic(self):
from apps.epg.tasks import _sd_pick_poster_url
self.assertEqual(
_sd_pick_poster_url(self._images(), 'landscape_iconic'),
'assets/iconic_landscape.jpg',
)
def test_landscape_falls_back_to_portrait_when_unavailable(self):
from apps.epg.tasks import _sd_pick_poster_url
images = [img for img in self._images() if img['aspect'] in ('2x3', '3x4')]
self.assertEqual(
_sd_pick_poster_url(images, 'landscape_iconic'),
'assets/iconic_portrait.jpg',
)
def test_unknown_style_defaults_to_sd_recommended(self):
from apps.epg.tasks import _sd_pick_poster_url
self.assertEqual(
_sd_pick_poster_url(self._images(), 'not_a_real_style'),
'assets/iconic_portrait.jpg',
)
def test_prefers_primary_when_category_and_aspect_match(self):
from apps.epg.tasks import _sd_pick_poster_url
images = [
{
'uri': 'assets/banner_small.jpg',
'width': '120',
'aspect': '2x3',
'category': 'Banner-L1',
},
{
'uri': 'assets/banner_primary.jpg',
'width': '360',
'aspect': '2x3',
'category': 'Banner-L1',
'primary': 'true',
},
]
self.assertEqual(
_sd_pick_poster_url(images, 'portrait_banner'),
'assets/banner_primary.jpg',
)
def test_sd_recommended_uses_primary_poster_category(self):
from apps.epg.tasks import _sd_pick_poster_url
images = [
{
'uri': 'assets/cast_primary.jpg',
'width': '500',
'aspect': '3x4',
'category': 'Cast in Character',
'primary': 'true',
},
{
'uri': 'assets/iconic_primary.jpg',
'width': '300',
'aspect': '16x9',
'category': 'Iconic',
'primary': 'true',
},
]
self.assertEqual(
_sd_pick_poster_url(images, 'sd_recommended'),
'assets/iconic_primary.jpg',
)
def test_sd_recommended_falls_back_to_portrait_iconic(self):
from apps.epg.tasks import _sd_pick_poster_url
self.assertEqual(
_sd_pick_poster_url(self._images(), 'sd_recommended'),
'assets/iconic_portrait.jpg',
)
def test_default_style_is_sd_recommended(self):
from apps.epg.tasks import _sd_pick_poster_url, SD_POSTER_STYLE_DEFAULT
self.assertEqual(SD_POSTER_STYLE_DEFAULT, 'sd_recommended')
images = [
{
'uri': 'assets/primary.jpg',
'width': '960',
'aspect': '16x9',
'category': 'Iconic',
'primary': 'true',
},
]
self.assertEqual(_sd_pick_poster_url(images), 'assets/primary.jpg')
def test_style_fallback_uses_primary_before_cross_orientation(self):
from apps.epg.tasks import _sd_pick_poster_url
images = [
{
'uri': 'assets/iconic_portrait.jpg',
'width': '960',
'aspect': '2x3',
'category': 'Iconic',
},
{
'uri': 'assets/landscape_primary.jpg',
'width': '1920',
'aspect': '16x9',
'category': 'Iconic',
'primary': 'true',
},
]
# square_iconic has no 1x1 images; should pick SD primary before portrait iconic fallback
self.assertEqual(
_sd_pick_poster_url(images, 'square_iconic'),
'assets/landscape_primary.jpg',
)

16
apps/hdhr/api_views.py
View file

@ -19,6 +19,7 @@ from django.utils.decorators import method_decorator
from django.contrib.auth.decorators import login_required
from django.views.decorators.csrf import csrf_exempt
from dispatcharr.utils import network_access_allowed
from core.utils import build_absolute_uri_with_port
# Configure logger
logger = logging.getLogger(__name__)
@ -75,7 +76,7 @@ class DiscoverAPIView(APIView):
uri_parts.append("output_profile")
uri_parts.append(str(output_profile_id))
base_url = request.build_absolute_uri(f'/{"/".join(uri_parts)}/').rstrip("/")
base_url = build_absolute_uri_with_port(request, f'/{"/".join(uri_parts)}/').rstrip("/")
device = HDHRDevice.objects.first()
from apps.m3u.utils import calculate_tuner_count
@ -166,6 +167,13 @@ class LineupAPIView(APIView):
resolved_output_profile_id = _resolve_hdhr_output_profile_id(output_profile_id)
_stream_url_prefix = build_absolute_uri_with_port(request, "/proxy/ts/stream/")
_output_profile_qs = (
f"?output_profile={resolved_output_profile_id}"
if resolved_output_profile_id is not None
else ""
)
lineup = []
for ch in channels:
formatted = format_channel_number(ch.effective_channel_number, empty=None)
@ -173,9 +181,7 @@ class LineupAPIView(APIView):
continue
formatted_channel_number = str(formatted)
stream_url = request.build_absolute_uri(f"/proxy/ts/stream/{ch.uuid}")
if resolved_output_profile_id is not None:
stream_url += f"?output_profile={resolved_output_profile_id}"
stream_url = f"{_stream_url_prefix}{ch.uuid}{_output_profile_qs}"
lineup.append(
{
@ -224,7 +230,7 @@ class HDHRDeviceXMLAPIView(APIView):
if blocked is not None:
return blocked
base_url = request.build_absolute_uri("/hdhr/").rstrip("/")
base_url = build_absolute_uri_with_port(request, "/hdhr/").rstrip("/")
xml_response = f"""<?xml version="1.0" encoding="utf-8"?>
<root>

9
apps/hdhr/views.py
View file

@ -15,6 +15,7 @@ from django.views import View
from django.utils.decorators import method_decorator
from django.contrib.auth.decorators import login_required
from django.views.decorators.csrf import csrf_exempt
from core.utils import build_absolute_uri_with_port
@login_required
@ -46,7 +47,7 @@ class DiscoverAPIView(APIView):
description="Retrieve HDHomeRun device discovery information",
)
def get(self, request):
base_url = request.build_absolute_uri("/hdhr/").rstrip("/")
base_url = build_absolute_uri_with_port(request, "/hdhr/").rstrip("/")
device = HDHRDevice.objects.first()
if not device:
@ -92,6 +93,8 @@ class LineupAPIView(APIView):
.exclude(hidden_from_output=True)
.order_by("effective_channel_number")
)
_stream_url_prefix = build_absolute_uri_with_port(request, "/proxy/ts/stream/")
lineup = []
for ch in channels:
formatted = format_channel_number(ch.effective_channel_number, empty=None)
@ -102,7 +105,7 @@ class LineupAPIView(APIView):
{
"GuideNumber": formatted_channel_number,
"GuideName": ch.effective_name,
"URL": request.build_absolute_uri(f"/proxy/ts/stream/{ch.uuid}"),
"URL": f"{_stream_url_prefix}{ch.uuid}",
}
)
return JsonResponse(lineup, safe=False)
@ -133,7 +136,7 @@ class HDHRDeviceXMLAPIView(APIView):
description="Retrieve the HDHomeRun device XML configuration",
)
def get(self, request):
base_url = request.build_absolute_uri("/hdhr/").rstrip("/")
base_url = build_absolute_uri_with_port(request, "/hdhr/").rstrip("/")
xml_response = f"""<?xml version="1.0" encoding="utf-8"?>
<root>

323
apps/m3u/connection_pool.py Normal file
View file

@ -0,0 +1,323 @@
"""
Shared connection pool enforcement for M3U accounts in the same ServerGroup.
Profile selection rotates across M3UAccountProfile rows using each profile's own
Redis counter (the pre-pool behavior). When an account belongs to a ServerGroup, a credential-scoped counter is checked on reserve/release
so accounts sharing the same provider login share one limit without blocking
unrelated logins on the same group. Account profiles with max_streams=0 skip
credential enforcement for that profile.
"""
from __future__ import annotations
import hashlib
import logging
import re
from typing import Literal, Optional, Tuple
logger = logging.getLogger(__name__)
ReserveFailureReason = Literal["profile_full", "credential_full"]
PROFILE_CONNECTIONS_KEY = "profile_connections:{profile_id}"
PROFILE_CREDENTIAL_RELEASE_KEY = "profile_credential_release:{profile_id}"
SERVER_GROUP_CONNECTIONS_KEY = "server_group_connections:{group_id}:{fingerprint}"
_XC_URL_CREDENTIALS_RE = re.compile(
r"/(?:live|movie|series)/([^/]+)/([^/]+)/",
re.IGNORECASE,
)
def profile_connections_key(profile_id: int) -> str:
return PROFILE_CONNECTIONS_KEY.format(profile_id=profile_id)
def profile_credential_release_key(profile_id: int) -> str:
"""Redis key storing the credential counter to release when the profile row is gone."""
return PROFILE_CREDENTIAL_RELEASE_KEY.format(profile_id=profile_id)
def server_group_connections_key(group_id: int, fingerprint: str) -> str:
"""Redis key for per-credential usage within a ServerGroup."""
return SERVER_GROUP_CONNECTIONS_KEY.format(
group_id=group_id,
fingerprint=fingerprint[:16],
)
def compute_credential_fingerprint(username: str, password: str) -> Optional[str]:
"""Return a stable hash for grouping accounts with the same IPTV login."""
if not username or not password:
return None
normalized = f"{username.strip().lower()}\0{password.strip()}"
return hashlib.sha256(normalized.encode("utf-8")).hexdigest()
def extract_credentials_from_stream_url(url: str) -> Tuple[Optional[str], Optional[str]]:
"""Parse username/password embedded in an Xtream-style stream URL."""
if not url:
return None, None
match = _XC_URL_CREDENTIALS_RE.search(url)
if not match:
return None, None
return match.group(1), match.group(2)
def _fingerprint_from_profile_stream_url(profile) -> Optional[str]:
"""STD/M3U: fingerprint from a sample stream URL after profile rewrite."""
from apps.channels.models import Stream
sample_url = (
Stream.objects.filter(m3u_account=profile.m3u_account)
.exclude(url="")
.values_list("url", flat=True)
.first()
)
if not sample_url:
return None
try:
from apps.proxy.live_proxy.url_utils import transform_url
transformed = transform_url(
sample_url,
profile.search_pattern or "",
profile.replace_pattern or "",
)
url_user, url_pass = extract_credentials_from_stream_url(
transformed or sample_url
)
return compute_credential_fingerprint(url_user or "", url_pass or "")
except Exception as exc:
logger.debug(
"Could not derive profile %s fingerprint from stream URL: %s",
profile.pk,
exc,
)
return None
def get_profile_credential_fingerprint(profile) -> Optional[str]:
"""Fingerprint for credentials this profile uses at playback time."""
m3u_account = profile.m3u_account
if m3u_account.account_type == "XC":
try:
from apps.m3u.tasks import get_transformed_credentials
_url, username, password = get_transformed_credentials(m3u_account, profile)
fingerprint = compute_credential_fingerprint(username or "", password or "")
if fingerprint:
return fingerprint
except Exception as exc:
logger.debug(
"Could not resolve transformed credentials for profile %s: %s",
profile.pk,
exc,
)
fingerprint = _fingerprint_from_profile_stream_url(profile)
if fingerprint:
return fingerprint
return compute_credential_fingerprint(
m3u_account.username or "",
m3u_account.password or "",
)
def get_enforced_server_group_for_profile(profile):
"""Return the ServerGroup for credential pooling when the account is assigned to one."""
group = profile.m3u_account.server_group
if group:
return group
return None
def _credential_counter_key(profile, group) -> Optional[str]:
fingerprint = get_profile_credential_fingerprint(profile)
if not fingerprint:
return None
return server_group_connections_key(group.id, fingerprint)
def get_profile_connection_count(profile, redis_client) -> int:
return int(redis_client.get(profile_connections_key(profile.id)) or 0)
def get_credential_connection_count(profile, redis_client) -> int:
group = get_enforced_server_group_for_profile(profile)
if not group:
return 0
cred_key = _credential_counter_key(profile, group)
if not cred_key:
return 0
return int(redis_client.get(cred_key) or 0)
def profile_has_capacity_for_selection(profile, redis_client) -> bool:
"""Per-profile capacity check used when rotating across profiles on one account."""
if profile.max_streams == 0:
return True
return get_profile_connection_count(profile, redis_client) < profile.max_streams
def group_has_capacity_for_profile(profile, redis_client) -> bool:
# Profiles with max_streams=0 skip credential enforcement entirely. An unlimited
# profile in a pooled group can still stream while other accounts share the login.
group = get_enforced_server_group_for_profile(profile)
if not group or profile.max_streams == 0:
return True
cred_key = _credential_counter_key(profile, group)
if not cred_key:
return True
return int(redis_client.get(cred_key) or 0) < profile.max_streams
def pool_has_capacity_for_profile(profile, redis_client) -> bool:
"""Non-mutating check before reserve: profile slot and credential slot if applicable."""
return profile_has_capacity_for_selection(profile, redis_client) and group_has_capacity_for_profile(
profile, redis_client
)
def profile_available_for_channel_switch(
profile, redis_client, *, channel_already_on_profile: bool
) -> bool:
"""
Non-mutating capacity check when selecting a profile for an in-flight channel.
If the channel already holds this profile's slots, skip re-checking capacity.
"""
if channel_already_on_profile:
return True
return pool_has_capacity_for_profile(profile, redis_client)
def move_credential_slot_on_profile_switch(
old_profile, new_profile, redis_client
) -> bool:
"""
Move the shared credential counter when switching to a different provider login.
Profile counters are managed separately by Channel.update_stream_profile().
Returns False when the new profile's credential pool is full.
"""
old_fp = get_profile_credential_fingerprint(old_profile)
new_fp = get_profile_credential_fingerprint(new_profile)
if old_fp == new_fp:
return True
_release_credential_slot_by_profile_id(old_profile.id, redis_client)
cred_reserved, cred_key = _reserve_server_group_slot_for_profile(
new_profile, redis_client
)
if not cred_reserved:
restore_reserved, restore_key = _reserve_server_group_slot_for_profile(
old_profile, redis_client
)
if restore_reserved and restore_key:
_remember_credential_release_key(
old_profile.id, restore_key, redis_client
)
return False
if cred_key:
_remember_credential_release_key(new_profile.id, cred_key, redis_client)
return True
def _safe_decr(redis_client, key: str) -> None:
current = int(redis_client.get(key) or 0)
if current <= 0:
return
new_count = redis_client.decr(key)
if new_count < 0:
redis_client.set(key, 0)
def _remember_credential_release_key(
profile_id: int, cred_key: str, redis_client
) -> None:
redis_client.set(profile_credential_release_key(profile_id), cred_key)
def _release_credential_slot_by_profile_id(profile_id: int, redis_client) -> bool:
"""Release a reserved credential counter using the key stored at reserve time."""
release_key = profile_credential_release_key(profile_id)
cred_key = redis_client.get(release_key)
if not cred_key:
return False
if isinstance(cred_key, bytes):
cred_key = cred_key.decode()
_safe_decr(redis_client, cred_key)
redis_client.delete(release_key)
return True
def _reserve_server_group_slot_for_profile(
profile, redis_client
) -> Tuple[bool, Optional[str]]:
group = get_enforced_server_group_for_profile(profile)
if not group or profile.max_streams == 0:
return True, None
cred_key = _credential_counter_key(profile, group)
if not cred_key:
return True, None
cred_count = redis_client.incr(cred_key)
if cred_count <= profile.max_streams:
return True, cred_key
redis_client.decr(cred_key)
return False, None
def reserve_profile_slot(
profile, redis_client
) -> Tuple[bool, int, Optional[ReserveFailureReason]]:
"""
Atomically reserve profile + optional credential slots (INCR-first).
Returns (reserved, profile_count_after_attempt, failure_reason).
failure_reason is set when reserved is False.
"""
profile_key = profile_connections_key(profile.id)
profile_count = 0
if profile.max_streams > 0:
profile_count = redis_client.incr(profile_key)
if profile_count > profile.max_streams:
redis_client.decr(profile_key)
return False, profile_count - 1, "profile_full"
cred_reserved, cred_key = _reserve_server_group_slot_for_profile(
profile, redis_client
)
if not cred_reserved:
if profile.max_streams > 0:
redis_client.decr(profile_key)
return (
False,
profile_count - 1 if profile.max_streams > 0 else 0,
"credential_full",
)
if cred_key:
_remember_credential_release_key(profile.id, cred_key, redis_client)
return True, profile_count, None
def release_profile_slot(profile_id: int, redis_client) -> None:
"""Release profile and shared credential slots after a stream end."""
_release_credential_slot_by_profile_id(profile_id, redis_client)
profile_key = profile_connections_key(profile_id)
current = int(redis_client.get(profile_key) or 0)
if current > 0:
redis_client.decr(profile_key)

8
apps/m3u/models.py
View file

@ -194,7 +194,13 @@ class M3UFilter(models.Model):
class ServerGroup(models.Model):
"""Represents a logical grouping of servers or channels."""
"""
Groups M3U accounts that share provider credentials.
Accounts assigned to the same server group share credential-scoped connection
counters when their logins match. Limits come from each account profile's
max_streams, not from the group itself.
"""
name = models.CharField(
max_length=100, unique=True, help_text="Unique name for this server group."

17
apps/m3u/tasks.py
View file

@ -990,7 +990,7 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys):
existing_streams = {
s.stream_hash: s
for s in Stream.objects.filter(stream_hash__in=stream_hashes.keys()).select_related('m3u_account').only(
'id', 'stream_hash', 'name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'last_seen', 'updated_at', 'm3u_account', 'stream_id', 'stream_chno'
'id', 'stream_hash', 'name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'last_seen', 'updated_at', 'm3u_account', 'stream_id', 'stream_chno', 'channel_group_id'
)
}
@ -1006,7 +1006,8 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys):
obj.custom_properties != stream_props["custom_properties"] or
obj.is_adult != stream_props["is_adult"] or
obj.stream_id != stream_props["stream_id"] or
obj.stream_chno != stream_props["stream_chno"]
obj.stream_chno != stream_props["stream_chno"] or
obj.channel_group_id != stream_props["channel_group_id"]
)
if changed:
@ -1042,7 +1043,7 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys):
# Simplified bulk update for better performance
Stream.objects.bulk_update(
streams_to_update,
['name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'is_adult', 'last_seen', 'updated_at', 'is_stale', 'stream_id', 'stream_chno', 'is_catchup', 'catchup_days'],
['name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'is_adult', 'last_seen', 'updated_at', 'is_stale', 'stream_id', 'stream_chno', 'channel_group_id', 'is_catchup', 'catchup_days'],
batch_size=150 # Smaller batch size for XC processing
)
@ -1224,7 +1225,7 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys):
existing_streams = {
s.stream_hash: s
for s in Stream.objects.filter(stream_hash__in=stream_hashes.keys()).select_related('m3u_account').only(
'id', 'stream_hash', 'name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'last_seen', 'updated_at', 'm3u_account', 'stream_id', 'stream_chno'
'id', 'stream_hash', 'name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'last_seen', 'updated_at', 'm3u_account', 'stream_id', 'stream_chno', 'channel_group_id'
)
}
@ -1240,7 +1241,8 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys):
obj.custom_properties != stream_props["custom_properties"] or
obj.is_adult != stream_props["is_adult"] or
obj.stream_id != stream_props["stream_id"] or
obj.stream_chno != stream_props["stream_chno"]
obj.stream_chno != stream_props["stream_chno"] or
obj.channel_group_id != stream_props["channel_group_id"]
)
# Always update last_seen
@ -1256,6 +1258,7 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys):
obj.is_adult = stream_props["is_adult"]
obj.stream_id = stream_props["stream_id"]
obj.stream_chno = stream_props["stream_chno"]
obj.channel_group_id = stream_props["channel_group_id"]
obj.updated_at = timezone.now()
# Always mark as not stale since we saw it in this refresh
@ -1278,7 +1281,7 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys):
# Update all streams in a single bulk operation
Stream.objects.bulk_update(
streams_to_update,
['name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'is_adult', 'last_seen', 'updated_at', 'is_stale', 'stream_id', 'stream_chno', 'is_catchup', 'catchup_days'],
['name', 'url', 'logo_url', 'tvg_id', 'custom_properties', 'is_adult', 'last_seen', 'updated_at', 'is_stale', 'stream_id', 'stream_chno', 'channel_group_id', 'is_catchup', 'catchup_days'],
batch_size=200
)
except Exception as e:
@ -3024,7 +3027,7 @@ def refresh_account_profiles(account_id):
existing_props = profile.custom_properties or {}
existing_props.update(profile_account_info)
profile.custom_properties = existing_props
profile.save(update_fields=['custom_properties', 'exp_date'])
profile.save(update_fields=['custom_properties'])
profiles_updated += 1
logger.info(f"Updated account information for profile '{profile.name}' ({profiles_updated}/{profiles.count()})")

594
apps/m3u/tests/test_connection_pool.py Normal file
View file

@ -0,0 +1,594 @@
"""Tests for shared ServerGroup connection pools (#1137)."""
from django.test import TestCase
from unittest.mock import patch
from apps.m3u.connection_pool import (
extract_credentials_from_stream_url,
get_credential_connection_count,
get_enforced_server_group_for_profile,
get_profile_connection_count,
get_profile_credential_fingerprint,
group_has_capacity_for_profile,
pool_has_capacity_for_profile,
profile_has_capacity_for_selection,
profile_connections_key,
profile_credential_release_key,
release_profile_slot,
reserve_profile_slot,
server_group_connections_key,
)
from apps.m3u.models import M3UAccount, M3UAccountProfile, ServerGroup
class FakeRedis:
"""Minimal in-memory Redis stand-in for counter tests."""
def __init__(self):
self._data = {}
def get(self, key):
val = self._data.get(key)
if val is None:
return None
if isinstance(val, str):
return val.encode()
return str(val).encode()
def set(self, key, value, ex=None):
try:
self._data[key] = int(value)
except (ValueError, TypeError):
self._data[key] = value
def incr(self, key):
self._data[key] = self._data.get(key, 0) + 1
return self._data[key]
def decr(self, key):
self._data[key] = self._data.get(key, 0) - 1
return self._data[key]
def delete(self, key):
self._data.pop(key, None)
def pipeline(self):
return FakeRedisPipeline(self)
class FakeRedisPipeline:
def __init__(self, redis):
self.redis = redis
self._ops = []
def decr(self, key):
self._ops.append(("decr", key))
return self
def incr(self, key):
self._ops.append(("incr", key))
return self
def set(self, key, value):
self._ops.append(("set", key, value))
return self
def execute(self):
for op in self._ops:
if op[0] == "decr":
self.redis.decr(op[1])
elif op[0] == "incr":
self.redis.incr(op[1])
elif op[0] == "set":
self.redis.set(op[1], op[2])
self._ops = []
class ExtractCredentialsTests(TestCase):
def test_extract_credentials_from_xc_style_url(self):
url = "http://example.com/live/alice/secret123/99999.ts"
user, password = extract_credentials_from_stream_url(url)
self.assertEqual(user, "alice")
self.assertEqual(password, "secret123")
class ManualServerGroupTests(TestCase):
def test_group_enforced_when_account_assigned(self):
group = ServerGroup.objects.create(name="provider-a")
account = M3UAccount.objects.create(
name="Account A",
account_type="XC",
username="user",
password="pass",
server_group=group,
)
profile = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
self.assertEqual(get_enforced_server_group_for_profile(profile), group)
def test_accounts_in_same_group_share_credential_counter(self):
group = ServerGroup.objects.create(name="shared")
account1 = M3UAccount.objects.create(
name="XC Account",
account_type="XC",
username="user",
password="pass",
server_url="http://xc.example.com",
server_group=group,
max_streams=5,
)
account2 = M3UAccount.objects.create(
name="M3U Account",
account_type="STD",
username="user",
password="pass",
server_group=group,
max_streams=5,
)
profile1 = M3UAccountProfile.objects.get(m3u_account=account1, is_default=True)
profile2 = M3UAccountProfile.objects.get(m3u_account=account2, is_default=True)
profile1.max_streams = 1
profile1.save()
profile2.max_streams = 1
profile2.save()
redis = FakeRedis()
reserved1, _, _ = reserve_profile_slot(profile1, redis)
self.assertTrue(reserved1)
reserved2, _, _ = reserve_profile_slot(profile2, redis)
self.assertFalse(reserved2)
self.assertFalse(group_has_capacity_for_profile(profile2, redis))
def test_profile_rotation_when_default_profile_full(self):
account = M3UAccount.objects.create(
name="Multi-profile",
account_type="XC",
max_streams=1,
)
default = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
default.max_streams = 1
default.save()
alt = M3UAccountProfile.objects.create(
m3u_account=account,
name="alt_profile",
is_default=False,
is_active=True,
max_streams=1,
search_pattern="",
replace_pattern="",
)
redis = FakeRedis()
reserved, _, _ = reserve_profile_slot(default, redis)
self.assertTrue(reserved)
self.assertFalse(profile_has_capacity_for_selection(default, redis))
self.assertTrue(profile_has_capacity_for_selection(alt, redis))
class PoolEnforcementTests(TestCase):
def setUp(self):
self.redis = FakeRedis()
self.group = ServerGroup.objects.create(name="test-pool")
self.account = M3UAccount.objects.create(
name="Test Account",
account_type="XC",
username="user",
password="pass",
server_url="http://xc.example.com",
server_group=self.group,
max_streams=5,
)
self.profile = M3UAccountProfile.objects.get(
m3u_account=self.account, is_default=True
)
self.profile.max_streams = 1
self.profile.save()
def test_group_has_capacity_reads_credential_counter_directly(self):
cred_key = server_group_connections_key(
self.group.id,
get_profile_credential_fingerprint(self.profile),
)
self.redis.set(cred_key, 1)
self.assertFalse(group_has_capacity_for_profile(self.profile, self.redis))
self.redis.set(cred_key, 0)
self.assertTrue(group_has_capacity_for_profile(self.profile, self.redis))
def test_reserve_and_release_both_counters(self):
reserved, count, _ = reserve_profile_slot(self.profile, self.redis)
self.assertTrue(reserved)
self.assertEqual(count, 1)
cred_key = server_group_connections_key(
self.group.id,
get_profile_credential_fingerprint(self.profile),
)
profile_key = profile_connections_key(self.profile.id)
self.assertEqual(self.redis._data[cred_key], 1)
self.assertEqual(self.redis._data[profile_key], 1)
release_profile_slot(self.profile.id, self.redis)
self.assertEqual(self.redis._data[cred_key], 0)
self.assertEqual(self.redis._data[profile_key], 0)
def test_same_credential_capped_at_profile_max(self):
"""Shared credential counter is capped by each profile's max_streams."""
account2 = M3UAccount.objects.create(
name="Second Account",
account_type="XC",
username="user",
password="pass",
server_url="http://xc.example.com",
server_group=self.group,
max_streams=5,
)
profile2 = M3UAccountProfile.objects.get(m3u_account=account2, is_default=True)
profile2.max_streams = 1
profile2.save()
self.assertTrue(reserve_profile_slot(self.profile, self.redis)[0])
self.assertFalse(reserve_profile_slot(profile2, self.redis)[0])
fp = get_profile_credential_fingerprint(self.profile)
cred_key = server_group_connections_key(self.group.id, fp)
self.assertEqual(self.redis._data[cred_key], 1)
def test_different_logins_both_stream_in_same_group(self):
"""Different provider logins keep separate credential counters in one group."""
account = M3UAccount.objects.create(
name="Grouped multi-login",
account_type="XC",
username="login_a",
password="pass_a",
server_group=self.group,
max_streams=5,
)
default = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
default.max_streams = 1
default.save()
alt = M3UAccountProfile.objects.create(
m3u_account=account,
name="alt_login",
is_default=False,
is_active=True,
max_streams=1,
search_pattern="",
replace_pattern="",
)
fp_a = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
fp_b = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
with patch(
"apps.m3u.connection_pool.get_profile_credential_fingerprint",
side_effect=lambda profile: fp_a if profile.id == default.id else fp_b,
):
self.assertTrue(reserve_profile_slot(default, self.redis)[0])
self.assertTrue(reserve_profile_slot(alt, self.redis)[0])
key_a = server_group_connections_key(self.group.id, fp_a)
key_b = server_group_connections_key(self.group.id, fp_b)
self.assertEqual(self.redis._data[key_a], 1)
self.assertEqual(self.redis._data[key_b], 1)
def test_no_fingerprint_skips_credential_counter(self):
account = M3UAccount.objects.create(
name="No creds",
account_type="STD",
server_group=self.group,
max_streams=5,
)
profile = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
profile.max_streams = 1
profile.save()
with patch(
"apps.m3u.connection_pool.get_profile_credential_fingerprint",
return_value=None,
):
self.assertTrue(reserve_profile_slot(profile, self.redis)[0])
self.assertEqual(get_credential_connection_count(profile, self.redis), 0)
self.assertEqual(get_credential_connection_count(profile, self.redis), 0)
def test_release_when_profile_row_deleted(self):
profile_id = self.profile.id
fp = get_profile_credential_fingerprint(self.profile)
cred_key = server_group_connections_key(self.group.id, fp)
reserved, _, failure_reason = reserve_profile_slot(
self.profile, self.redis
)
self.assertTrue(reserved)
self.assertIsNone(failure_reason)
self.profile.delete()
release_profile_slot(profile_id, self.redis)
self.assertEqual(self.redis._data[profile_connections_key(profile_id)], 0)
self.assertEqual(self.redis._data[cred_key], 0)
self.assertNotIn(profile_credential_release_key(profile_id), self.redis._data)
def test_release_uses_stored_credential_key_without_db_lookup(self):
profile_id = self.profile.id
cred_key = server_group_connections_key(
self.group.id,
get_profile_credential_fingerprint(self.profile),
)
self.assertTrue(reserve_profile_slot(self.profile, self.redis)[0])
with patch("apps.m3u.models.M3UAccountProfile.objects.get") as mock_get:
release_profile_slot(profile_id, self.redis)
mock_get.assert_not_called()
self.assertEqual(self.redis._data[profile_connections_key(profile_id)], 0)
self.assertEqual(self.redis._data[cred_key], 0)
def test_reserve_returns_failure_reason_without_extra_checks(self):
account2 = M3UAccount.objects.create(
name="Reason Account",
account_type="XC",
username="user",
password="pass",
server_url="http://xc.example.com",
server_group=self.group,
max_streams=5,
)
profile2 = M3UAccountProfile.objects.get(m3u_account=account2, is_default=True)
profile2.max_streams = 1
profile2.save()
self.assertTrue(reserve_profile_slot(self.profile, self.redis)[0])
reserved, _count, reason = reserve_profile_slot(profile2, self.redis)
self.assertFalse(reserved)
self.assertEqual(reason, "credential_full")
class StaleAssignmentTests(TestCase):
def test_stale_assignment_releases_counters(self):
from apps.channels.models import Channel, Stream
redis = FakeRedis()
group = ServerGroup.objects.create(name="stale-group")
account = M3UAccount.objects.create(
name="Stale Account",
account_type="XC",
username="user",
password="pass",
server_url="http://xc.example.com",
server_group=group,
max_streams=5,
)
profile = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
profile.max_streams = 1
profile.save()
stream = Stream.objects.create(name="Stale Stream", m3u_account=account)
channel = Channel.objects.create(channel_number=502, name="Stale Channel")
channel.streams.add(stream)
reserve_profile_slot(profile, redis)
redis.set(f"channel_stream:{channel.id}", stream.id)
redis.set(f"stream_profile:{stream.id}", profile.id)
profile_key = profile_connections_key(profile.id)
cred_key = server_group_connections_key(
group.id, get_profile_credential_fingerprint(profile)
)
self.assertEqual(redis._data[profile_key], 1)
self.assertEqual(redis._data[cred_key], 1)
with patch("core.utils.RedisClient.get_client", return_value=redis):
channel._release_stale_stream_assignment(redis, stream.id)
self.assertNotIn(f"channel_stream:{channel.id}", redis._data)
self.assertNotIn(f"stream_profile:{stream.id}", redis._data)
self.assertEqual(redis._data[profile_key], 0)
self.assertEqual(redis._data[cred_key], 0)
class UpdateStreamProfileTests(TestCase):
def test_switch_updates_profile_counters_when_group_assigned(self):
from apps.channels.models import Channel, Stream
redis = FakeRedis()
group = ServerGroup.objects.create(name="switch-group")
account = M3UAccount.objects.create(
name="Switch Account",
account_type="XC",
username="user",
password="pass",
server_url="http://xc.example.com",
server_group=group,
max_streams=5,
)
profile_a = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
profile_a.max_streams = 1
profile_a.save()
profile_b = M3UAccountProfile.objects.create(
m3u_account=account,
name="alt",
is_default=False,
is_active=True,
max_streams=1,
search_pattern="",
replace_pattern="",
)
stream = Stream.objects.create(name="Test Stream", m3u_account=account)
channel = Channel.objects.create(channel_number=501, name="Switch Channel")
channel.streams.add(stream)
reserve_profile_slot(profile_a, redis)
redis.set(f"channel_stream:{channel.id}", stream.id)
redis.set(f"stream_profile:{stream.id}", profile_a.id)
cred_key = server_group_connections_key(
group.id, get_profile_credential_fingerprint(profile_a)
)
cred_before = redis._data[cred_key]
with patch("core.utils.RedisClient.get_client", return_value=redis):
self.assertTrue(channel.update_stream_profile(profile_b.id))
self.assertEqual(int(redis.get(f"stream_profile:{stream.id}")), profile_b.id)
self.assertEqual(redis._data[profile_connections_key(profile_a.id)], 0)
self.assertEqual(redis._data[profile_connections_key(profile_b.id)], 1)
self.assertEqual(redis._data[cred_key], cred_before)
def test_switch_moves_credential_counter_when_login_changes(self):
from apps.channels.models import Channel, Stream
redis = FakeRedis()
group = ServerGroup.objects.create(name="cred-switch-group")
account = M3UAccount.objects.create(
name="Cred Switch Account",
account_type="XC",
username="login_a",
password="pass_a",
server_url="http://xc.example.com",
server_group=group,
max_streams=5,
)
profile_a = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
profile_a.max_streams = 1
profile_a.save()
profile_b = M3UAccountProfile.objects.create(
m3u_account=account,
name="alt_login",
is_default=False,
is_active=True,
max_streams=1,
search_pattern="",
replace_pattern="",
)
fp_a = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
fp_b = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
key_a = server_group_connections_key(group.id, fp_a)
key_b = server_group_connections_key(group.id, fp_b)
stream = Stream.objects.create(name="Cred Switch Stream", m3u_account=account)
channel = Channel.objects.create(channel_number=503, name="Cred Switch Channel")
channel.streams.add(stream)
with patch(
"apps.m3u.connection_pool.get_profile_credential_fingerprint",
side_effect=lambda profile: fp_a if profile.id == profile_a.id else fp_b,
):
reserve_profile_slot(profile_a, redis)
redis.set(f"channel_stream:{channel.id}", stream.id)
redis.set(f"stream_profile:{stream.id}", profile_a.id)
self.assertEqual(redis._data[key_a], 1)
self.assertNotIn(key_b, redis._data)
with patch("core.utils.RedisClient.get_client", return_value=redis):
self.assertTrue(channel.update_stream_profile(profile_b.id))
self.assertEqual(redis._data[key_a], 0)
self.assertEqual(redis._data[key_b], 1)
class VodProfileSelectionTests(TestCase):
def test_get_m3u_profile_skips_default_when_profile_full(self):
from apps.proxy.vod_proxy.views import _get_m3u_profile
account = M3UAccount.objects.create(
name="VOD multi-profile",
account_type="XC",
username="xc_user_a",
password="xc_pass_a",
server_url="http://xc.example.com",
max_streams=1,
)
default = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
default.max_streams = 1
default.save()
alt = M3UAccountProfile.objects.create(
m3u_account=account,
name="alt_profile",
is_default=False,
is_active=True,
max_streams=1,
search_pattern="",
replace_pattern="",
)
redis = FakeRedis()
self.assertTrue(reserve_profile_slot(default, redis)[0])
with patch("core.utils.RedisClient.get_client", return_value=redis):
result = _get_m3u_profile(account, None, None)
self.assertIsNotNone(result)
selected, _connections = result
self.assertEqual(selected.id, alt.id)
def test_get_m3u_profile_skips_default_when_credential_pool_full(self):
from apps.proxy.vod_proxy.views import _get_m3u_profile
group = ServerGroup.objects.create(name="vod-cred-pool")
account = M3UAccount.objects.create(
name="VOD pooled",
account_type="XC",
username="shared_user",
password="shared_pass",
server_url="http://xc.example.com",
server_group=group,
max_streams=5,
)
default = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
default.max_streams = 1
default.save()
alt = M3UAccountProfile.objects.create(
m3u_account=account,
name="alt_login",
is_default=False,
is_active=True,
max_streams=1,
search_pattern="",
replace_pattern="",
)
other_account = M3UAccount.objects.create(
name="Other pooled account",
account_type="XC",
username="shared_user",
password="shared_pass",
server_url="http://xc.example.com",
server_group=group,
max_streams=5,
)
other_profile = M3UAccountProfile.objects.get(
m3u_account=other_account, is_default=True
)
other_profile.max_streams = 1
other_profile.save()
fp_shared = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
fp_alt = "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd"
redis = FakeRedis()
with patch(
"apps.m3u.connection_pool.get_profile_credential_fingerprint",
side_effect=lambda profile: (
fp_shared
if profile.id in (default.id, other_profile.id)
else fp_alt
),
):
self.assertTrue(reserve_profile_slot(other_profile, redis)[0])
with patch("core.utils.RedisClient.get_client", return_value=redis):
result = _get_m3u_profile(account, None, None)
self.assertIsNotNone(result)
selected, _connections = result
self.assertEqual(selected.id, alt.id)

107
apps/m3u/tests/test_sync_correctness.py
View file

@ -6,7 +6,10 @@ first (fails on HEAD prior to the Tier 2 patch), then is flipped to assert
the correct post-fix behavior. Comments call out the failure mode and the
fix location.
"""
from django.test import TestCase
from unittest import skipUnless
from django.db import connection
from django.test import TestCase, TransactionTestCase
from django.utils import timezone
from apps.channels.models import (
@ -2195,3 +2198,105 @@ class CompactNumberingWithGroupOverrideTests(TestCase):
large,
f"repack query count scaled with channel count: {small} -> {large}",
)
@skipUnless(
connection.vendor == "postgresql",
"Idempotency repro forces a physical heap reorder via CLUSTER, which is "
"PostgreSQL-specific (the suite's target DB).",
)
class CompactNumberingIdempotencyTests(TransactionTestCase):
"""
A compact repack must be idempotent: with no change to hide state or
overrides, repacking again must leave every channel on the same number.
The unpatched _repack_inner read its channels with no ORDER BY, so the
pack followed PostgreSQL's physical row order. That order drifts after
the UPDATEs each repack issues (and after autovacuum), so successive
syncs packed the same channels into different numbers. That is the daily
channel-number churn users reported.
This test forces the divergence deterministically. After the first pack
it rewrites every channel_number to the reverse of id order, then
physically clusters the table on that column so the heap order becomes
the reverse of id order. An unordered SELECT then returns the rows in the
opposite order from the first pass. Unpatched, the second pack assigns
numbers in that reversed order and the channel->number mapping flips;
patched, .order_by("id") keeps both packs identical.
Fail signature: channel->number mapping differs between the two repacks
= _repack_inner is following physical row order instead of id order.
Fix location: apps/channels/compact_numbering.py (_repack_inner channel
query .order_by("id")).
"""
# TransactionTestCase commits its rows (TestCase's savepoint rollback
# would hide them from CLUSTER, which also cannot run inside the
# transaction block TestCase wraps each test in).
def _mapping(self, account):
return {
c.id: c.channel_number
for c in Channel.objects.filter(
auto_created=True, auto_created_by=account
)
}
def test_repack_is_idempotent_under_physical_reorder(self):
from apps.channels.compact_numbering import repack_group
account = _make_account()
group = _make_group(name="Sports")
rel = _attach_group_to_account(
account, group, custom_properties={"compact_numbering": True}
)
rel.auto_sync_channel_start = 8000
rel.auto_sync_channel_end = 8099
rel.save()
# Eight visible auto channels; ascending id is creation order.
channels = [
Channel.objects.create(
name=f"C{i}",
channel_group=group,
auto_created=True,
auto_created_by=account,
)
for i in range(8)
]
repack_group(rel)
first = self._mapping(account)
# Provider-order pack (the default) assigns by id, so the lowest id
# takes the range start.
lowest_id = min(c.id for c in channels)
self.assertEqual(first[lowest_id], 8000)
# Set channel_number to the reverse of id order, then cluster the
# heap on that column so physical order becomes reverse-id order.
# Values sit above the range so they cannot collide with the pack.
table = Channel._meta.db_table
with connection.cursor() as cur:
for pos, ch in enumerate(channels):
cur.execute(
f"UPDATE {table} SET channel_number = %s WHERE id = %s",
[9000 - pos, ch.id],
)
cur.execute(
f"CREATE INDEX IF NOT EXISTS churn_cn_idx "
f"ON {table} (channel_number)"
)
cur.execute(f"CLUSTER {table} USING churn_cn_idx")
cur.execute("DROP INDEX IF EXISTS churn_cn_idx")
repack_group(rel)
second = self._mapping(account)
self.assertEqual(
first,
second,
"Repack is not idempotent: channel numbers changed on a second "
"pass with no hide or override change. _repack_inner is following "
"physical row order instead of id order.",
)

37
apps/output/views.py
View file

@ -207,11 +207,11 @@ def generate_m3u(request, profile_name=None, user=None):
xc_username = request.GET.get('username')
xc_password = request.GET.get('password')
is_xc_request = user is not None and xc_username and xc_password
_base_url = build_absolute_uri_with_port(request, '')
if is_xc_request:
# This is an XC API request - use XC-style EPG URL
base_url = build_absolute_uri_with_port(request, '')
epg_url = f"{base_url}/xmltv.php?username={xc_username}&password={xc_password}"
epg_url = f"{_base_url}/xmltv.php?username={xc_username}&password={xc_password}"
# Build the query-string suffix for stream URLs once - it's the same for every channel
xc_qs = {}
if output_profile_id:
@ -241,6 +241,13 @@ def generate_m3u(request, profile_name=None, user=None):
# Add x-tvg-url and url-tvg attribute for EPG URL
m3u_content = f'#EXTM3U x-tvg-url="{epg_url}" url-tvg="{epg_url}"\n'
# Host/port/scheme are constant per request; precompute URL prefixes once.
_stream_url_prefix = None if is_xc_request else f"{_base_url}/proxy/ts/stream/"
_sample_logo_path = reverse("api:channels:logo-cache", args=[0])
_logo_prefix_raw, _, _logo_suffix_raw = _sample_logo_path.partition("/0/")
_logo_url_prefix = _base_url + _logo_prefix_raw + "/"
_logo_url_suffix = "/" + _logo_suffix_raw
# Start building M3U content
channel_count = 0
for channel in channels:
@ -270,8 +277,7 @@ def generate_m3u(request, profile_name=None, user=None):
tvg_logo = ""
if effective_logo:
if use_cached_logos:
# Use cached logo as before
tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[effective_logo.id]))
tvg_logo = f"{_logo_url_prefix}{effective_logo.id}{_logo_url_suffix}"
else:
# Try to find direct logo URL from channel's streams
direct_logo = effective_logo.url if effective_logo.url.startswith(('http://', 'https://')) else None
@ -279,7 +285,7 @@ def generate_m3u(request, profile_name=None, user=None):
if direct_logo:
tvg_logo = direct_logo
else:
tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[effective_logo.id]))
tvg_logo = f"{_logo_url_prefix}{effective_logo.id}{_logo_url_suffix}"
# create possible gracenote id insertion
tvc_guide_stationid = ""
@ -295,7 +301,7 @@ def generate_m3u(request, profile_name=None, user=None):
# Determine the stream URL based on request type
if is_xc_request:
stream_url = f"{base_url}/live/{xc_username}/{xc_password}/{channel.id}{xc_qs_suffix}"
stream_url = f"{_base_url}/live/{xc_username}/{xc_password}/{channel.id}{xc_qs_suffix}"
elif use_direct_urls:
# Try to get the first stream's direct URL
all_streams = channel.streams.all()
@ -305,11 +311,10 @@ def generate_m3u(request, profile_name=None, user=None):
stream_url = first_stream.url
else:
# Fall back to proxy URL if no direct URL available
stream_url = build_absolute_uri_with_port(request, f"/proxy/ts/stream/{channel.uuid}")
stream_url = f"{_stream_url_prefix}{channel.uuid}"
else:
# Standard behavior - use proxy URL
base_stream_url = build_absolute_uri_with_port(request, f"/proxy/ts/stream/{channel.uuid}")
stream_url = f"{base_stream_url}{proxy_qs_suffix}"
stream_url = f"{_stream_url_prefix}{channel.uuid}{proxy_qs_suffix}"
m3u_content += extinf_line + stream_url + "\n"
@ -1450,6 +1455,13 @@ def generate_epg(request, profile_name=None, user=None):
channel_num_map[channel.id] = candidate
used_numbers.add(candidate)
# Host/port/scheme are constant per request; precompute logo URL prefix once.
_base_url = build_absolute_uri_with_port(request, "")
_sample_logo_path = reverse("api:channels:logo-cache", args=[0])
_logo_prefix_raw, _, _logo_suffix_raw = _sample_logo_path.partition("/0/")
_logo_url_prefix = _base_url + _logo_prefix_raw + "/"
_logo_url_suffix = "/" + _logo_suffix_raw
# Process channels for the <channel> section
for channel in channels:
effective_name = channel.effective_name
@ -1529,14 +1541,14 @@ def generate_epg(request, profile_name=None, user=None):
# If no custom dummy logo, use regular logo logic
if not tvg_logo and effective_logo:
if use_cached_logos:
tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[effective_logo.id]))
tvg_logo = f"{_logo_url_prefix}{effective_logo.id}{_logo_url_suffix}"
else:
# Use direct URL if available, otherwise fall back to cached version
direct_logo = effective_logo.url if effective_logo.url.startswith(('http://', 'https://')) else None
if direct_logo:
tvg_logo = direct_logo
else:
tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[effective_logo.id]))
tvg_logo = f"{_logo_url_prefix}{effective_logo.id}{_logo_url_suffix}"
display_name = effective_name
xml_lines.append(f' <channel id="{html.escape(channel_id)}">')
xml_lines.append(f' <display-name>{html.escape(display_name)}</display-name>')
@ -1668,6 +1680,7 @@ def generate_epg(request, profile_name=None, user=None):
# to avoid skipping/duplicating rows if the table changes mid-stream.
last_epg_id = 0
last_id = 0
_poster_url_base = build_absolute_uri_with_port(request, "/api/epg/programs/")
while True:
program_chunk = list(
@ -1871,6 +1884,8 @@ def generate_epg(request, profile_name=None, user=None):
if "icon" in custom_data:
program_xml.append(f' <icon src="{html.escape(custom_data["icon"])}" />')
elif "sd_icon" in custom_data:
program_xml.append(f' <icon src="{html.escape(_poster_url_base)}{prog["id"]}/poster/" />')
# Add special flags as proper tags with enhanced handling
if custom_data.get("previously_shown", False):

25
apps/plugins/migrations/0003_update_official_repo_url.py Normal file
View file

@ -0,0 +1,25 @@
from django.db import migrations
def update_url(apps, schema_editor):
PluginRepo = apps.get_model("plugins", "PluginRepo")
PluginRepo.objects.filter(is_official=True).update(
url="https://dispatcharr.github.io/Plugins/manifest.json"
)
def revert_url(apps, schema_editor):
PluginRepo = apps.get_model("plugins", "PluginRepo")
PluginRepo.objects.filter(is_official=True).update(
url="https://raw.githubusercontent.com/Dispatcharr/Plugins/releases/manifest.json"
)
class Migration(migrations.Migration):
dependencies = [
("plugins", "0002_pluginrepo"),
]
operations = [
migrations.RunPython(update_url, revert_url),
]

4
apps/plugins/models.py
View file

@ -36,9 +36,7 @@ class PluginConfig(models.Model):
return f"{self.name} ({self.key})"
OFFICIAL_REPO_URL = (
"https://raw.githubusercontent.com/Dispatcharr/Plugins/releases/manifest.json"
)
OFFICIAL_REPO_URL = "https://dispatcharr.github.io/Plugins/manifest.json"
class PluginRepo(models.Model):

176
apps/proxy/live_proxy/url_utils.py
View file

@ -8,6 +8,10 @@ from typing import Optional, Tuple, List
from django.shortcuts import get_object_or_404
from apps.channels.models import Channel, Stream
from apps.m3u.models import M3UAccount, M3UAccountProfile
from apps.m3u.connection_pool import (
get_profile_connection_count,
profile_available_for_channel_switch,
)
from core.models import UserAgent, CoreSettings, StreamProfile
from .utils import get_logger
from uuid import UUID
@ -15,6 +19,35 @@ import requests
logger = get_logger()
def _resolve_live_stream_url(stream, m3u_account, m3u_profile):
"""
Build the upstream URL for live playback.
XC accounts use current transformed credentials plus provider stream_id so
playback matches the account login (not a stale stream.url from an old sync).
STD/M3U accounts keep using the URL stored on the stream row.
"""
if (
m3u_account.account_type == M3UAccount.Types.XC
and stream.stream_id
):
from apps.m3u.tasks import get_transformed_credentials
server_url, username, password = get_transformed_credentials(
m3u_account, m3u_profile
)
if server_url and username and password:
base = server_url.rstrip("/")
return f"{base}/live/{username}/{password}/{stream.stream_id}.ts"
return transform_url(
stream.url or "",
m3u_profile.search_pattern,
m3u_profile.replace_pattern,
)
def get_stream_object(id: str):
try:
logger.info(f"Fetching channel ID {id}")
@ -24,15 +57,14 @@ def get_stream_object(id: str):
logger.info(f"Fetching stream hash {id}")
return get_object_or_404(Stream, stream_hash=id)
def generate_stream_url(channel_id: str) -> Tuple[str, str, bool, Optional[int]]:
def generate_stream_url(
channel_id: str,
) -> Tuple[str, str, bool, Optional[int], bool, Optional[str]]:
"""
Generate the appropriate stream URL for a channel or stream based on its profile settings.
Args:
channel_id: The UUID of the channel or stream hash
Returns:
Tuple[str, str, bool, Optional[int]]: (stream_url, user_agent, transcode_flag, profile_id)
Tuple: (stream_url, user_agent, transcode_flag, profile_id, slot_reserved, error_reason)
"""
try:
channel_or_stream = get_stream_object(channel_id)
@ -44,15 +76,12 @@ def generate_stream_url(channel_id: str) -> Tuple[str, str, bool, Optional[int]]
if not stream.m3u_account:
logger.error(f"Stream {stream.id} has no M3U account")
return None, None, False, None
return None, None, False, None, False, "Stream has no M3U account"
# Use get_stream() to atomically reserve a slot and write the
# channel_stream / stream_profile Redis keys, matching the channel
# path so stream_name and stream_stats work correctly.
stream_id, profile_id, error_reason = stream.get_stream()
stream_id, profile_id, error_reason, slot_reserved = stream.get_stream()
if not stream_id or not profile_id:
logger.error(f"No profile available for stream {stream.id}: {error_reason}")
return None, None, False, None
return None, None, False, None, False, error_reason
try:
profile = M3UAccountProfile.objects.get(id=profile_id)
@ -63,7 +92,7 @@ def generate_stream_url(channel_id: str) -> Tuple[str, str, bool, Optional[int]]
stream_user_agent = UserAgent.objects.get(id=CoreSettings.get_default_user_agent_id())
logger.debug(f"No user agent found for account, using default: {stream_user_agent}")
stream_url = transform_url(stream.url, profile.search_pattern, profile.replace_pattern)
stream_url = _resolve_live_stream_url(stream, m3u_account, profile)
stream_profile = stream.get_stream_profile()
logger.debug(f"Using stream profile: {stream_profile.name}")
@ -71,23 +100,23 @@ def generate_stream_url(channel_id: str) -> Tuple[str, str, bool, Optional[int]]
transcode = not stream_profile.is_proxy()
stream_profile_id = stream_profile.id
return stream_url, stream_user_agent, transcode, stream_profile_id
return stream_url, stream_user_agent, transcode, stream_profile_id, slot_reserved, None
except Exception as e:
logger.error(f"Error generating stream URL for stream {stream.id}: {e}")
stream.release_stream()
return None, None, False, None
if slot_reserved:
stream.release_stream()
return None, None, False, None, False, str(e)
# Handle channel preview (existing logic)
channel = channel_or_stream
# Get stream and profile for this channel
# Note: get_stream now returns 3 values (stream_id, profile_id, error_reason)
stream_id, profile_id, error_reason = channel.get_stream()
stream_id, profile_id, error_reason, slot_reserved = channel.get_stream()
if not stream_id or not profile_id:
logger.error(f"No stream available for channel {channel_id}: {error_reason}")
return None, None, False, None
return None, None, False, None, False, error_reason
# get_stream() allocated a connection slot - ensure it's released on any error
try:
@ -106,9 +135,7 @@ def generate_stream_url(channel_id: str) -> Tuple[str, str, bool, Optional[int]]
stream_user_agent = UserAgent.objects.get(id=CoreSettings.get_default_user_agent_id())
logger.debug(f"No user agent found for account, using default: {stream_user_agent}")
# Generate stream URL based on the selected profile
input_url = stream.url
stream_url = transform_url(input_url, m3u_profile.search_pattern, m3u_profile.replace_pattern)
stream_url = _resolve_live_stream_url(stream, m3u_account, m3u_profile)
# Check if transcoding is needed
stream_profile = channel.get_stream_profile()
@ -119,15 +146,16 @@ def generate_stream_url(channel_id: str) -> Tuple[str, str, bool, Optional[int]]
stream_profile_id = stream_profile.id
return stream_url, stream_user_agent, transcode, stream_profile_id
return stream_url, stream_user_agent, transcode, stream_profile_id, slot_reserved, None
except Exception as e:
logger.error(f"Error generating stream URL for channel {channel_id}: {e}")
if not channel.release_stream():
logger.warning(f"Failed to release stream for channel {channel_id} after URL generation error")
return None, None, False, None
if slot_reserved:
if not channel.release_stream():
logger.warning(f"Failed to release stream for channel {channel_id} after URL generation error")
return None, None, False, None, False, str(e)
except Exception as e:
logger.error(f"Error generating stream URL: {e}")
return None, None, False, None
return None, None, False, None, False, str(e)
def transform_url(input_url: str, search_pattern: str, replace_pattern: str) -> str:
"""
@ -175,6 +203,8 @@ def get_stream_info_for_switch(channel_id: str, target_stream_id: Optional[int]
Returns:
dict: Stream information including URL, user agent and transcode flag
"""
slot_reserved = False
channel = None
try:
from core.utils import RedisClient
@ -204,35 +234,34 @@ def get_stream_info_for_switch(channel_id: str, target_stream_id: Optional[int]
selected_profile = None
for profile in profiles:
# Check connection availability
if redis_client:
profile_connections_key = f"profile_connections:{profile.id}"
current_connections = int(redis_client.get(profile_connections_key) or 0)
# Check if this channel is already using this profile
channel_using_profile = False
existing_stream_id = redis_client.get(f"channel_stream:{channel.id}")
if existing_stream_id:
# Decode bytes to string/int for proper Redis key lookup
existing_stream_id = existing_stream_id
existing_profile_id = redis_client.get(f"stream_profile:{existing_stream_id}")
existing_profile_id = redis_client.get(
f"stream_profile:{existing_stream_id}"
)
if existing_profile_id and int(existing_profile_id) == profile.id:
channel_using_profile = True
logger.debug(f"Channel {channel.id} already using profile {profile.id}")
# Calculate effective connections (subtract 1 if channel already using this profile)
effective_connections = current_connections - (1 if channel_using_profile else 0)
# Check if profile has available slots
if profile.max_streams == 0 or effective_connections < profile.max_streams:
if profile_available_for_channel_switch(
profile,
redis_client,
channel_already_on_profile=channel_using_profile,
):
current_connections = get_profile_connection_count(
profile, redis_client
)
selected_profile = profile
logger.debug(f"Selected profile {profile.id} with {effective_connections}/{profile.max_streams} effective connections (current: {current_connections}, already using: {channel_using_profile})")
logger.debug(
f"Selected profile {profile.id} with "
f"{current_connections}/{profile.max_streams} connections"
)
break
else:
logger.debug(f"Profile {profile.id} at max connections: {effective_connections}/{profile.max_streams} (current: {current_connections}, already using: {channel_using_profile})")
logger.debug(
f"Profile {profile.id} unavailable for channel switch"
)
else:
# No Redis available, assume first active profile is okay
selected_profile = profile
break
@ -241,29 +270,18 @@ def get_stream_info_for_switch(channel_id: str, target_stream_id: Optional[int]
m3u_profile_id = selected_profile.id
else:
stream_id, m3u_profile_id, error_reason = channel.get_stream()
stream_id, m3u_profile_id, error_reason, slot_reserved = channel.get_stream()
if stream_id is None or m3u_profile_id is None:
return {'error': error_reason or 'No stream assigned to channel'}
# Get the stream and profile objects directly
stream = get_object_or_404(Stream, pk=stream_id)
profile = get_object_or_404(M3UAccountProfile, pk=m3u_profile_id)
# Check connections left
m3u_account = M3UAccount.objects.get(id=profile.m3u_account.id)
#connections_left = get_connections_left(m3u_profile_id)
#if connections_left <= 0:
#logger.warning(f"No connections left for M3U account {m3u_account.id}")
#return {'error': 'No connections left'}
# Get the user agent from the M3U account
user_agent = m3u_account.get_user_agent().user_agent
# Generate URL using the transform function directly
stream_url = transform_url(stream.url, profile.search_pattern, profile.replace_pattern)
stream_url = _resolve_live_stream_url(stream, m3u_account, profile)
# Get transcode info from the channel's stream profile
stream_profile = channel.get_stream_profile()
transcode = not (stream_profile.is_proxy() or stream_profile is None)
profile_value = stream_profile.id
@ -278,6 +296,8 @@ def get_stream_info_for_switch(channel_id: str, target_stream_id: Optional[int]
'stream_name': stream.name,
}
except Exception as e:
if slot_reserved and channel is not None:
channel.release_stream()
logger.error(f"Error getting stream info for switch: {e}", exc_info=True)
return {'error': f'Error: {str(e)}'}
@ -344,34 +364,38 @@ def get_alternate_streams(channel_id: str, current_stream_id: Optional[int] = No
selected_profile = None
for profile in profiles:
# Check connection availability
if redis_client:
profile_connections_key = f"profile_connections:{profile.id}"
current_connections = int(redis_client.get(profile_connections_key) or 0)
# Check if this channel is already using this profile
channel_using_profile = False
existing_stream_id = redis_client.get(f"channel_stream:{channel.id}")
if existing_stream_id:
# Decode bytes to string/int for proper Redis key lookup
existing_stream_id = existing_stream_id
existing_profile_id = redis_client.get(f"stream_profile:{existing_stream_id}")
existing_profile_id = redis_client.get(
f"stream_profile:{existing_stream_id}"
)
if existing_profile_id and int(existing_profile_id) == profile.id:
channel_using_profile = True
logger.debug(f"Channel {channel.id} already using profile {profile.id}")
logger.debug(
f"Channel {channel.id} already using profile {profile.id}"
)
# Calculate effective connections (subtract 1 if channel already using this profile)
effective_connections = current_connections - (1 if channel_using_profile else 0)
# Check if profile has available slots
if profile.max_streams == 0 or effective_connections < profile.max_streams:
if profile_available_for_channel_switch(
profile,
redis_client,
channel_already_on_profile=channel_using_profile,
):
current_connections = get_profile_connection_count(
profile, redis_client
)
selected_profile = profile
logger.debug(f"Found available profile {profile.id} for stream {stream.id}: {effective_connections}/{profile.max_streams} effective (current: {current_connections}, already using: {channel_using_profile})")
logger.debug(
f"Found available profile {profile.id} for stream {stream.id}: "
f"{current_connections}/{profile.max_streams} "
f"(already using: {channel_using_profile})"
)
break
else:
logger.debug(f"Profile {profile.id} at max connections: {effective_connections}/{profile.max_streams} (current: {current_connections}, already using: {channel_using_profile})")
logger.debug(
f"Profile {profile.id} unavailable for alternate stream {stream.id}"
)
else:
# No Redis available, assume first active profile is okay
selected_profile = profile
break

34
apps/proxy/live_proxy/views.py
View file

@ -213,6 +213,7 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
stream_user_agent = None
transcode = False
profile_value = None
slot_reserved = False
error_reason = None
attempt = 0
should_retry = True
@ -220,9 +221,14 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
# Try to get a stream with fixed interval retries
while should_retry and time.time() - wait_start_time < retry_timeout:
attempt += 1
stream_url, stream_user_agent, transcode, profile_value = (
generate_stream_url(channel_id)
)
(
stream_url,
stream_user_agent,
transcode,
profile_value,
slot_reserved,
error_reason,
) = generate_stream_url(channel_id)
if stream_url is not None:
logger.info(
@ -232,7 +238,6 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
# On first failure, check if the error is retryable
if attempt == 1:
_, _, error_reason = channel.get_stream()
if error_reason and "maximum connection limits" not in error_reason:
logger.warning(
f"[{client_id}] Can't retry - error not related to connection limits: {error_reason}"
@ -265,18 +270,21 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
logger.info(
f"[{client_id}] Making final attempt {attempt} at timeout boundary"
)
stream_url, stream_user_agent, transcode, profile_value = (
generate_stream_url(channel_id)
)
(
stream_url,
stream_user_agent,
transcode,
profile_value,
slot_reserved,
error_reason,
) = generate_stream_url(channel_id)
if stream_url is not None:
logger.info(
f"[{client_id}] Successfully obtained stream on final attempt for channel {channel_id}"
)
if stream_url is None:
# Release any connection slot that may have been allocated
# by the error-checking get_stream() call during retries
if not channel.release_stream():
if slot_reserved and not channel.release_stream():
logger.debug(f"[{client_id}] release_stream found no keys during failed init cleanup")
# Get the specific error message if available
@ -295,7 +303,7 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
# generate_stream_url() called get_stream() which allocated a connection
# slot (INCR'd profile_connections) - track this for cleanup on error
if needs_initialization:
if needs_initialization and slot_reserved:
connection_allocated = True
# Read stream assignment from Redis (already set by generate_stream_url → get_stream).
@ -378,8 +386,8 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
logger.warning(
f"[{client_id}] Alternate stream #{alt['stream_id']} failed validation: {message}"
)
# Release stream lock before redirecting
if not channel.release_stream():
# Release stream lock before redirecting only if we reserved a slot
if connection_allocated and not channel.release_stream():
logger.warning(f"[{client_id}] Failed to release stream before redirect")
connection_allocated = False
# Final decision based on validation results

82
apps/proxy/vod_proxy/multi_worker_connection_manager.py
View file

@ -735,68 +735,35 @@ class MultiWorkerVODConnectionManager:
"""Get Redis key for tracking connections per profile - STANDARDIZED with TS proxy"""
return f"profile_connections:{profile_id}"
def _check_profile_limits(self, m3u_profile) -> bool:
"""Check if profile has available connection slots"""
if m3u_profile.max_streams == 0: # Unlimited
return True
try:
profile_connections_key = self._get_profile_connections_key(m3u_profile.id)
current_connections = int(self.redis_client.get(profile_connections_key) or 0)
logger.info(f"[PROFILE-CHECK] Profile {m3u_profile.id} has {current_connections}/{m3u_profile.max_streams} connections")
return current_connections < m3u_profile.max_streams
except Exception as e:
logger.error(f"Error checking profile limits: {e}")
return False
def _check_and_reserve_profile_slot(self, m3u_profile) -> bool:
"""
Atomically check and reserve a connection slot for the given profile.
Uses an INCR-first-then-check pattern to eliminate the TOCTOU race
condition where separate GET > check > INCR operations could allow
concurrent requests to both pass the capacity check.
For profiles with max_streams=0 (unlimited), no reservation is needed.
Returns:
bool: True if slot was reserved (or unlimited), False if at capacity
"""
if m3u_profile.max_streams == 0: # Unlimited
return True
from apps.m3u.connection_pool import reserve_profile_slot
try:
profile_connections_key = self._get_profile_connections_key(m3u_profile.id)
# Atomically increment first — single Redis command eliminates race window
new_count = self.redis_client.incr(profile_connections_key)
if new_count <= m3u_profile.max_streams:
logger.info(f"[PROFILE-RESERVE] Profile {m3u_profile.id} slot reserved: {new_count}/{m3u_profile.max_streams}")
return True
# Over capacity — roll back the increment
self.redis_client.decr(profile_connections_key)
logger.info(f"[PROFILE-RESERVE] Profile {m3u_profile.id} at capacity: {new_count - 1}/{m3u_profile.max_streams}")
return False
reserved, new_count, _failure_reason = reserve_profile_slot(
m3u_profile, self.redis_client
)
if reserved:
logger.info(
f"[PROFILE-RESERVE] Profile {m3u_profile.id} slot reserved: "
f"{new_count}/{m3u_profile.max_streams}"
)
else:
logger.info(
f"[PROFILE-RESERVE] Profile {m3u_profile.id} at capacity: "
f"{new_count}/{m3u_profile.max_streams}"
)
return reserved
except Exception as e:
logger.error(f"Error reserving profile slot: {e}")
return False
def _increment_profile_connections(self, m3u_profile):
"""Increment profile connection count"""
try:
profile_connections_key = self._get_profile_connections_key(m3u_profile.id)
new_count = self.redis_client.incr(profile_connections_key)
logger.info(f"[PROFILE-INCR] Profile {m3u_profile.id} connections: {new_count}")
return new_count
except Exception as e:
logger.error(f"Error incrementing profile connections: {e}")
return None
def _trigger_vod_stats_update(self):
"""Trigger a VOD stats WebSocket update in a background thread."""
threading.Thread(target=self._do_vod_stats_update, daemon=True).start()
@ -859,21 +826,14 @@ class MultiWorkerVODConnectionManager:
logger.error(f"Failed to trigger VOD stats update: {e}")
def _decrement_profile_connections(self, m3u_profile_id: int):
"""Decrement profile connection count.
"""Decrement profile and shared pool connection counters."""
from apps.m3u.connection_pool import release_profile_slot
Uses a single atomic DECR (no GET-before-DECR) to avoid the race condition
where two concurrent decrements both pass a >0 guard and both fire, sending
the counter negative. If the counter would go below zero it is clamped to 0.
"""
try:
profile_connections_key = self._get_profile_connections_key(m3u_profile_id)
new_count = self.redis_client.decr(profile_connections_key)
if new_count < 0:
self.redis_client.set(profile_connections_key, 0)
new_count = 0
logger.warning(f"[PROFILE-DECR] Profile {m3u_profile_id} counter went negative, clamped to 0")
else:
logger.info(f"[PROFILE-DECR] Profile {m3u_profile_id} connections: {new_count}")
release_profile_slot(m3u_profile_id, self.redis_client)
profile_key = self._get_profile_connections_key(m3u_profile_id)
new_count = int(self.redis_client.get(profile_key) or 0)
logger.info(f"[PROFILE-DECR] Profile {m3u_profile_id} connections: {new_count}")
return new_count
except Exception as e:
logger.error(f"Error decrementing profile connections: {e}")

31
apps/proxy/vod_proxy/views.py
View file

@ -260,6 +260,10 @@ def _get_m3u_profile(m3u_account, profile_id, session_id=None):
"""
try:
from core.utils import RedisClient
from apps.m3u.connection_pool import (
get_profile_connection_count,
pool_has_capacity_for_profile,
)
redis_client = RedisClient.get_client()
if not redis_client:
@ -296,7 +300,11 @@ def _get_m3u_profile(m3u_account, profile_id, session_id=None):
except Exception as e:
logger.warning(f"[PROFILE-SELECTION] Error checking existing profile for session {session_id}: {e}")
else:
logger.debug(f"[PROFILE-SELECTION] Session {session_id} exists but has no profile ID stored") # If specific profile requested, try to use it
logger.debug(
f"[PROFILE-SELECTION] Session {session_id} exists but has no profile ID stored"
)
# If specific profile requested, try to use it
if profile_id:
try:
profile = M3UAccountProfile.objects.get(
@ -306,13 +314,12 @@ def _get_m3u_profile(m3u_account, profile_id, session_id=None):
)
# Check Redis-based current connections
profile_connections_key = f"profile_connections:{profile.id}"
current_connections = int(redis_client.get(profile_connections_key) or 0)
current_connections = get_profile_connection_count(profile, redis_client)
if profile.max_streams == 0 or current_connections < profile.max_streams:
if pool_has_capacity_for_profile(profile, redis_client):
logger.info(f"[PROFILE-SELECTION] Using requested profile {profile.id}: {current_connections}/{profile.max_streams} connections")
return (profile, current_connections)
else:
logger.warning(f"[PROFILE-SELECTION] Requested profile {profile.id} is at capacity: {current_connections}/{profile.max_streams}")
logger.warning(f"[PROFILE-SELECTION] Requested profile {profile.id} is at capacity: {current_connections}/{profile.max_streams}")
except M3UAccountProfile.DoesNotExist:
logger.warning(f"[PROFILE-SELECTION] Requested profile {profile_id} not found")
@ -331,15 +338,16 @@ def _get_m3u_profile(m3u_account, profile_id, session_id=None):
profiles = [default_profile] + list(m3u_profiles.filter(is_default=False))
for profile in profiles:
profile_connections_key = f"profile_connections:{profile.id}"
current_connections = int(redis_client.get(profile_connections_key) or 0)
current_connections = get_profile_connection_count(profile, redis_client)
# Check if profile has available connection slots
if profile.max_streams == 0 or current_connections < profile.max_streams:
if pool_has_capacity_for_profile(profile, redis_client):
logger.info(f"[PROFILE-SELECTION] Selected profile {profile.id} ({profile.name}): {current_connections}/{profile.max_streams} connections")
return (profile, current_connections)
else:
logger.debug(f"[PROFILE-SELECTION] Profile {profile.id} at capacity: {current_connections}/{profile.max_streams}")
logger.debug(
f"[PROFILE-SELECTION] Profile {profile.id} unavailable "
f"(profile={current_connections}/{profile.max_streams})"
)
# All profiles are at capacity - return None to trigger error response
logger.error(f"[PROFILE-SELECTION] All profiles at capacity for M3U account {m3u_account.id}, rejecting request")
@ -906,13 +914,14 @@ def build_vod_stats_data(redis_client):
if m3u_profile_id:
try:
from apps.m3u.models import M3UAccountProfile
profile = M3UAccountProfile.objects.select_related('m3u_account').get(id=m3u_profile_id)
m3u_profile_info = {
'profile_name': profile.name,
'account_name': profile.m3u_account.name,
'account_id': profile.m3u_account.id,
'max_streams': profile.m3u_account.max_streams,
'm3u_profile_id': int(m3u_profile_id)
'm3u_profile_id': int(m3u_profile_id),
}
except Exception as e:
logger.warning(f"Could not fetch M3U profile {m3u_profile_id}: {e}")

61
core/utils.py
View file

@ -312,6 +312,34 @@ class TaskLockRenewer:
return False
def _is_gevent_monkey_patched():
try:
import gevent.monkey
return gevent.monkey.is_module_patched('threading')
except Exception:
return False
def _is_celery_worker_context():
"""True when executing inside an active Celery task (prefork worker)."""
try:
from celery import current_task
request = getattr(current_task, 'request', None)
return bool(request and getattr(request, 'id', None))
except Exception:
return False
def _should_use_sync_websocket_send():
"""
Use synchronous Redis delivery when gevent is monkey-patched but no gevent
hub is driving the process — e.g. Celery prefork workers that inherit
gevent patching from uWSGI imports. gevent.spawn in that context schedules
coroutines that never run.
"""
return _is_gevent_monkey_patched() and _is_celery_worker_context()
def _gevent_ws_send(group_name, message):
"""
Publishes a WebSocket group message synchronously through Redis.
@ -363,6 +391,12 @@ def _gevent_ws_send(group_name, message):
logger.warning(f"Failed to send WebSocket update: {e}")
def send_websocket_update_sync(group_name, event_type, data):
"""Send a WebSocket group message synchronously via Redis (channels_redis wire format)."""
message = {'type': event_type, 'data': data}
_gevent_ws_send(group_name, message)
def send_websocket_update(group_name, event_type, data, collect_garbage=False):
"""
Sends a WebSocket group message.
@ -370,28 +404,25 @@ def send_websocket_update(group_name, event_type, data, collect_garbage=False):
In gevent-patched uWSGI workers, asyncio event loop creation fails because
monkey-patching removes select.epoll. For those contexts a synchronous Redis
path is used instead, matching the channels_redis 4.x wire format.
Celery prefork workers may inherit gevent monkey-patching without a running
gevent hub; in that case gevent.spawn would never execute, so delivery is
synchronous via Redis instead.
"""
channel_layer = get_channel_layer()
message = {'type': event_type, 'data': data}
def _do_send():
if _should_use_sync_websocket_send():
_gevent_ws_send(group_name, message)
elif _is_gevent_monkey_patched():
import gevent
gevent.spawn(_gevent_ws_send, group_name, message)
else:
# Not gevent-patched (plain Celery, tests) — use asyncio channel layer
try:
async_to_sync(channel_layer.group_send)(group_name, message)
async_to_sync(get_channel_layer().group_send)(group_name, message)
except Exception as e:
logger.warning(f"Failed to send WebSocket update: {e}")
try:
import gevent.monkey
if gevent.monkey.is_module_patched('threading'):
import gevent
gevent.spawn(_gevent_ws_send, group_name, message)
return
except Exception:
pass
# Not in a gevent-patched environment (Celery, tests) — use asyncio path
_do_send()
if collect_garbage:
gc.collect()

2
dispatcharr/celery.py
View file

@ -95,6 +95,8 @@ def cleanup_task_memory(**kwargs):
'apps.epg.tasks.parse_programs_for_source',
'apps.epg.tasks.parse_programs_for_tvg_id',
'apps.channels.tasks.match_epg_channels',
'apps.channels.tasks.match_selected_channels_epg',
'apps.channels.tasks.match_single_channel_epg',
'core.tasks.rehash_streams'
]

2
docker/init/03-init-dispatcharr.sh
View file

@ -7,6 +7,7 @@
DATA_DIRS=(
"/data/backups"
"/data/logos"
"/data/logo_cache"
"/data/recordings"
"/data/uploads/m3us"
"/data/uploads/epgs"
@ -19,7 +20,6 @@ DATA_DIRS=(
# APP_DIRS live on the image layer and are always locally writable.
APP_DIRS=(
"/app/logo_cache"
"/app/media"
"/app/static"
)

10
docker/nginx.conf
View file

@ -1,4 +1,4 @@
proxy_cache_path /app/logo_cache levels=1:2 keys_zone=logo_cache:10m
proxy_cache_path /data/logo_cache levels=1:2 keys_zone=logo_cache:10m
inactive=24h use_temp_path=off;
server {
@ -58,6 +58,14 @@ server {
proxy_cache_use_stale error timeout updating; # Serve stale if Django is slow
}
location ~ ^/api/epg/programs/(?<prog_id>\d+)/poster/ {
proxy_pass http://127.0.0.1:5656;
proxy_cache logo_cache;
proxy_cache_key "$scheme$request_uri";
proxy_cache_valid 200 24h;
proxy_cache_use_stale error timeout updating;
}
# admin disabled when not in dev mode
location ~ ^/admin/?$ {
return 301 /login;

365
frontend/package-lock.json generated
View file

@ -58,7 +58,7 @@
"jsdom": "^27.0.0",
"prettier": "^3.5.3",
"vite": "^7.1.7",
"vitest": "^3.2.4"
"vitest": "^4.1.8"
}
},
"node_modules/@acemir/cssom": {
@ -1840,6 +1840,13 @@
"win32"
]
},
"node_modules/@standard-schema/spec": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@standard-schema/spec/-/spec-1.1.0.tgz",
"integrity": "sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==",
"dev": true,
"license": "MIT"
},
"node_modules/@standard-schema/utils": {
"version": "0.3.0",
"resolved": "https://registry.npmjs.org/@standard-schema/utils/-/utils-0.3.0.tgz",
@ -2403,39 +2410,40 @@
}
},
"node_modules/@vitest/expect": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.4.tgz",
"integrity": "sha512-Io0yyORnB6sikFlt8QW5K7slY4OjqNX9jmJQ02QDda8lyM6B5oNgVWoSoKPac8/kgnCUzuHQKrSLtu/uOqqrig==",
"version": "4.1.8",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.8.tgz",
"integrity": "sha512-h3nDO677RDLEGlBxyQ5CW8RlMThSKSRLUePLOx09gNIWRL40edgA1GCZSZgf1W55MFAG6/Sw14KeaAnqv0NKdQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@standard-schema/spec": "^1.1.0",
"@types/chai": "^5.2.2",
"@vitest/spy": "3.2.4",
"@vitest/utils": "3.2.4",
"chai": "^5.2.0",
"tinyrainbow": "^2.0.0"
"@vitest/spy": "4.1.8",
"@vitest/utils": "4.1.8",
"chai": "^6.2.2",
"tinyrainbow": "^3.1.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/mocker": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-3.2.4.tgz",
"integrity": "sha512-46ryTE9RZO/rfDd7pEqFl7etuyzekzEhUbTW3BvmeO/BcCMEgq59BKhek3dXDWgAj4oMK6OZi+vRr1wPW6qjEQ==",
"version": "4.1.8",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.8.tgz",
"integrity": "sha512-LEiN/xe4OSIbKe9HQIp5OC24agGD9J5CnmMgsLohVVoOPWL9a2sBoR6VBx43jQZb7Kr1l4RCuyCJzcAa0+dojw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/spy": "3.2.4",
"@vitest/spy": "4.1.8",
"estree-walker": "^3.0.3",
"magic-string": "^0.30.17"
"magic-string": "^0.30.21"
},
"funding": {
"url": "https://opencollective.com/vitest"
},
"peerDependencies": {
"msw": "^2.4.9",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0"
"vite": "^6.0.0 || ^7.0.0 || ^8.0.0"
},
"peerDependenciesMeta": {
"msw": {
@ -2447,42 +2455,42 @@
}
},
"node_modules/@vitest/pretty-format": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.4.tgz",
"integrity": "sha512-IVNZik8IVRJRTr9fxlitMKeJeXFFFN0JaB9PHPGQ8NKQbGpfjlTx9zO4RefN8gp7eqjNy8nyK3NZmBzOPeIxtA==",
"version": "4.1.8",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.8.tgz",
"integrity": "sha512-9GasEBxpZ1VYIpqHf/0+YGg121uSNwCKOJqIrTwWP/TB7DmFCiaBpNl3aPZzoLWfWkuqhbH8vJIVobZkvdo2cA==",
"dev": true,
"license": "MIT",
"dependencies": {
"tinyrainbow": "^2.0.0"
"tinyrainbow": "^3.1.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/runner": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-3.2.4.tgz",
"integrity": "sha512-oukfKT9Mk41LreEW09vt45f8wx7DordoWUZMYdY/cyAk7w5TWkTRCNZYF7sX7n2wB7jyGAl74OxgwhPgKaqDMQ==",
"version": "4.1.8",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.8.tgz",
"integrity": "sha512-EmVxeBAfMJvycdjd6Hm+RbFBbA9fKvo0Kx37hNpBYoYeavH3RNsBXWDooR1mgD52dCrxIIuP7UotpfiwOikvcg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/utils": "3.2.4",
"pathe": "^2.0.3",
"strip-literal": "^3.0.0"
"@vitest/utils": "4.1.8",
"pathe": "^2.0.3"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/snapshot": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-3.2.4.tgz",
"integrity": "sha512-dEYtS7qQP2CjU27QBC5oUOxLE/v5eLkGqPE0ZKEIDGMs4vKWe7IjgLOeauHsR0D5YuuycGRO5oSRXnwnmA78fQ==",
"version": "4.1.8",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.8.tgz",
"integrity": "sha512-acfZboRmAIf05DEKcBQy33VXojFJjtUdLyo7oOmV9kebb2xdU01UknNiPuPZoJZQyO7DF0gZdTGTpeAzET9QPQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "3.2.4",
"magic-string": "^0.30.17",
"@vitest/pretty-format": "4.1.8",
"@vitest/utils": "4.1.8",
"magic-string": "^0.30.21",
"pathe": "^2.0.3"
},
"funding": {
@ -2490,33 +2498,37 @@
}
},
"node_modules/@vitest/spy": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.4.tgz",
"integrity": "sha512-vAfasCOe6AIK70iP5UD11Ac4siNUNJ9i/9PZ3NKx07sG6sUxeag1LWdNrMWeKKYBLlzuK+Gn65Yd5nyL6ds+nw==",
"version": "4.1.8",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.8.tgz",
"integrity": "sha512-6EevtBp6OZOPF7bmz36HrGMeP3txgVSrgebWxHOafDXGkhIzfXK14f8KF6MuFfgXXUeHxmpD3BQxkV00/3s5mA==",
"dev": true,
"license": "MIT",
"dependencies": {
"tinyspy": "^4.0.3"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/utils": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.4.tgz",
"integrity": "sha512-fB2V0JFrQSMsCo9HiSq3Ezpdv4iYaXRG1Sx8edX3MwxfyNn83mKiGzOcH+Fkxt4MHxr3y42fQi1oeAInqgX2QA==",
"version": "4.1.8",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.8.tgz",
"integrity": "sha512-uOJamYALNhfJ6iolExyQM40yIQwDqYnkKtQ5VCiSe17E33H0aQ/u+1GlRuz4LZBk6Mm3sg90G9hEbmEt37C1Zg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "3.2.4",
"loupe": "^3.1.4",
"tinyrainbow": "^2.0.0"
"@vitest/pretty-format": "4.1.8",
"convert-source-map": "^2.0.0",
"tinyrainbow": "^3.1.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/utils/node_modules/convert-source-map": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz",
"integrity": "sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==",
"dev": true,
"license": "MIT"
},
"node_modules/@xmldom/xmldom": {
"version": "0.8.13",
"resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.13.tgz",
@ -2704,9 +2716,9 @@
}
},
"node_modules/brace-expansion": {
"version": "5.0.5",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.5.tgz",
"integrity": "sha512-VZznLgtwhn+Mact9tfiwx64fA9erHH/MCXEUfB/0bX/6Fz6ny5EGTXYltMocqg4xFAQZtnO3DHWWXi8RiuN7cQ==",
"version": "5.0.6",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.6.tgz",
"integrity": "sha512-kLpxurY4Z4r9sgMsyG0Z9uzsBlgiU/EFKhj/h91/8yHu0edo7XuixOIH3VcJ8kkxs6/jPzoI6U9Vj3WqbMQ94g==",
"dev": true,
"license": "MIT",
"dependencies": {
@ -2716,16 +2728,6 @@
"node": "18 || 20 || >=22"
}
},
"node_modules/cac": {
"version": "6.7.14",
"resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz",
"integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/callsites": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz",
@ -2736,18 +2738,11 @@
}
},
"node_modules/chai": {
"version": "5.3.3",
"resolved": "https://registry.npmjs.org/chai/-/chai-5.3.3.tgz",
"integrity": "sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==",
"version": "6.2.2",
"resolved": "https://registry.npmjs.org/chai/-/chai-6.2.2.tgz",
"integrity": "sha512-NUPRluOfOiTKBKvWPtSD4PhFvWCqOi0BGStNWs57X9js7XGTprSmFoz5F0tWhR4WPjNeR9jXqdC7/UpSJTnlRg==",
"dev": true,
"license": "MIT",
"dependencies": {
"assertion-error": "^2.0.1",
"check-error": "^2.1.1",
"deep-eql": "^5.0.1",
"loupe": "^3.1.0",
"pathval": "^2.0.0"
},
"engines": {
"node": ">=18"
}
@ -2769,16 +2764,6 @@
"url": "https://github.com/chalk/chalk?sponsor=1"
}
},
"node_modules/check-error": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/check-error/-/check-error-2.1.3.tgz",
"integrity": "sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 16"
}
},
"node_modules/classnames": {
"version": "2.5.1",
"resolved": "https://registry.npmjs.org/classnames/-/classnames-2.5.1.tgz",
@ -3097,16 +3082,6 @@
"integrity": "sha512-qIMFpTMZmny+MMIitAB6D7iVPEorVw6YQRWkvarTkT4tBeSLLiHzcwj6q0MmYSFCiVpiqPJTJEYIrpcPzVEIvg==",
"license": "MIT"
},
"node_modules/deep-eql": {
"version": "5.0.2",
"resolved": "https://registry.npmjs.org/deep-eql/-/deep-eql-5.0.2.tgz",
"integrity": "sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6"
}
},
"node_modules/deep-is": {
"version": "0.1.4",
"resolved": "https://registry.npmjs.org/deep-is/-/deep-is-0.1.4.tgz",
@ -3175,9 +3150,9 @@
}
},
"node_modules/es-module-lexer": {
"version": "1.7.0",
"resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz",
"integrity": "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==",
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-2.1.0.tgz",
"integrity": "sha512-n27zTYMjYu1aj4MjCWzSP7G9r75utsaoc8m61weK+W8JMBGGQybd43GstCXZ3WNmSFtGT9wi59qQTW6mhTR5LQ==",
"dev": true,
"license": "MIT"
},
@ -4022,13 +3997,6 @@
"loose-envify": "cli.js"
}
},
"node_modules/loupe": {
"version": "3.2.1",
"resolved": "https://registry.npmjs.org/loupe/-/loupe-3.2.1.tgz",
"integrity": "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==",
"dev": true,
"license": "MIT"
},
"node_modules/lru-cache": {
"version": "11.2.6",
"resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.2.6.tgz",
@ -4210,6 +4178,20 @@
"node": ">=0.10.0"
}
},
"node_modules/obug": {
"version": "2.1.2",
"resolved": "https://registry.npmjs.org/obug/-/obug-2.1.2.tgz",
"integrity": "sha512-AWGB9WFcRXOQs48Z/udjI5ZcZMHXwX8XPByNpOydgcGsDLIzjGizhoMWJyKAWze7AVW/2W1i+/gPX4YtKe5cyg==",
"dev": true,
"funding": [
"https://github.com/sponsors/sxzz",
"https://opencollective.com/debug"
],
"license": "MIT",
"engines": {
"node": ">=12.20.0"
}
},
"node_modules/optionator": {
"version": "0.9.4",
"resolved": "https://registry.npmjs.org/optionator/-/optionator-0.9.4.tgz",
@ -4345,16 +4327,6 @@
"dev": true,
"license": "MIT"
},
"node_modules/pathval": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/pathval/-/pathval-2.0.1.tgz",
"integrity": "sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 14.16"
}
},
"node_modules/picocolors": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
@ -4666,9 +4638,9 @@
}
},
"node_modules/react-router": {
"version": "7.13.0",
"resolved": "https://registry.npmjs.org/react-router/-/react-router-7.13.0.tgz",
"integrity": "sha512-PZgus8ETambRT17BUm/LL8lX3Of+oiLaPuVTRH3l1eLvSPpKO3AvhAEb5N7ihAFZQrYDqkvvWfFh9p0z9VsjLw==",
"version": "7.17.0",
"resolved": "https://registry.npmjs.org/react-router/-/react-router-7.17.0.tgz",
"integrity": "sha512-FDELK7rTMlCHO5+reyXsPlmfr7N1F91lPHsWYfMEGQm/KQ+F4JFM8jGoeQDmDvdTs93Fw9aSilH+uKRb4/jXvQ==",
"license": "MIT",
"dependencies": {
"cookie": "^1.0.1",
@ -4688,12 +4660,12 @@
}
},
"node_modules/react-router-dom": {
"version": "7.13.0",
"resolved": "https://registry.npmjs.org/react-router-dom/-/react-router-dom-7.13.0.tgz",
"integrity": "sha512-5CO/l5Yahi2SKC6rGZ+HDEjpjkGaG/ncEP7eWFTvFxbHP8yeeI0PxTDjimtpXYlR3b3i9/WIL4VJttPrESIf2g==",
"version": "7.17.0",
"resolved": "https://registry.npmjs.org/react-router-dom/-/react-router-dom-7.17.0.tgz",
"integrity": "sha512-fyU2yjGups/hE6Xz0I5ZYbVL8Gx29eCjgpHaRaTaVU+OOAdfRX05KsvyRm0GO8YQwOkhpU3MurW1jyMUJn+zSw==",
"license": "MIT",
"dependencies": {
"react-router": "7.13.0"
"react-router": "7.17.0"
},
"engines": {
"node": ">=20.0.0"
@ -5051,9 +5023,9 @@
"license": "MIT"
},
"node_modules/std-env": {
"version": "3.10.0",
"resolved": "https://registry.npmjs.org/std-env/-/std-env-3.10.0.tgz",
"integrity": "sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==",
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/std-env/-/std-env-4.1.0.tgz",
"integrity": "sha512-Rq7ybcX2RuC55r9oaPVEW7/xu3tj8u4GeBYHBWCychFtzMIr86A7e3PPEBPT37sHStKX3+TiX/Fr/ACmJLVlLQ==",
"dev": true,
"license": "MIT"
},
@ -5083,26 +5055,6 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/strip-literal": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/strip-literal/-/strip-literal-3.1.0.tgz",
"integrity": "sha512-8r3mkIM/2+PpjHoOtiAW8Rg3jJLHaV7xPwG+YRGrv6FP0wwk/toTpATxWYOW0BKdWwl82VT2tFYi5DlROa0Mxg==",
"dev": true,
"license": "MIT",
"dependencies": {
"js-tokens": "^9.0.1"
},
"funding": {
"url": "https://github.com/sponsors/antfu"
}
},
"node_modules/strip-literal/node_modules/js-tokens": {
"version": "9.0.1",
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-9.0.1.tgz",
"integrity": "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ==",
"dev": true,
"license": "MIT"
},
"node_modules/stylis": {
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/stylis/-/stylis-4.2.0.tgz",
@ -5167,11 +5119,14 @@
"license": "MIT"
},
"node_modules/tinyexec": {
"version": "0.3.2",
"resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-0.3.2.tgz",
"integrity": "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==",
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.2.4.tgz",
"integrity": "sha512-SHf/r48b7vOrjve9PxJo3MN5v5yuyjHvdUcrQffT3WXMUfnGmHDVbC4k3sHJaJTgZCwpUplIaAo5ANtMyp3YHg==",
"dev": true,
"license": "MIT"
"license": "MIT",
"engines": {
"node": ">=18"
}
},
"node_modules/tinyglobby": {
"version": "0.2.15",
@ -5190,30 +5145,10 @@
"url": "https://github.com/sponsors/SuperchupuDev"
}
},
"node_modules/tinypool": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/tinypool/-/tinypool-1.1.1.tgz",
"integrity": "sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==",
"dev": true,
"license": "MIT",
"engines": {
"node": "^18.0.0 || >=20.0.0"
}
},
"node_modules/tinyrainbow": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-2.0.0.tgz",
"integrity": "sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/tinyspy": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/tinyspy/-/tinyspy-4.0.4.tgz",
"integrity": "sha512-azl+t0z7pw/z958Gy9svOTuzqIk6xq+NSheJzn5MMWtWTFywIacg2wUlzKFGtt3cthx0r2SxMK0yzJOR0IES7Q==",
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-3.1.0.tgz",
"integrity": "sha512-Bf+ILmBgretUrdJxzXM0SgXLZ3XfiaUuOj/IKQHuTXip+05Xn+uyEYdVg0kYDipTBcLrCVyUzAPz7QmArb0mmw==",
"dev": true,
"license": "MIT",
"engines": {
@ -5564,89 +5499,80 @@
}
}
},
"node_modules/vite-node": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/vite-node/-/vite-node-3.2.4.tgz",
"integrity": "sha512-EbKSKh+bh1E1IFxeO0pg1n4dvoOTt0UDiXMd/qn++r98+jPO1xtJilvXldeuQ8giIB5IkpjCgMleHMNEsGH6pg==",
"dev": true,
"license": "MIT",
"dependencies": {
"cac": "^6.7.14",
"debug": "^4.4.1",
"es-module-lexer": "^1.7.0",
"pathe": "^2.0.3",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0"
},
"bin": {
"vite-node": "vite-node.mjs"
},
"engines": {
"node": "^18.0.0 || ^20.0.0 || >=22.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/vitest": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-3.2.4.tgz",
"integrity": "sha512-LUCP5ev3GURDysTWiP47wRRUpLKMOfPh+yKTx3kVIEiu5KOMeqzpnYNsKyOoVrULivR8tLcks4+lga33Whn90A==",
"version": "4.1.8",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.8.tgz",
"integrity": "sha512-flY6ScbCIt9HThs+C5HS7jvGOB560DJtk/Z15IQROTA6zEy49Nh8T/dofWTQL+n3vswqn87sbJNiuqw1SDp5Ig==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/chai": "^5.2.2",
"@vitest/expect": "3.2.4",
"@vitest/mocker": "3.2.4",
"@vitest/pretty-format": "^3.2.4",
"@vitest/runner": "3.2.4",
"@vitest/snapshot": "3.2.4",
"@vitest/spy": "3.2.4",
"@vitest/utils": "3.2.4",
"chai": "^5.2.0",
"debug": "^4.4.1",
"expect-type": "^1.2.1",
"magic-string": "^0.30.17",
"@vitest/expect": "4.1.8",
"@vitest/mocker": "4.1.8",
"@vitest/pretty-format": "4.1.8",
"@vitest/runner": "4.1.8",
"@vitest/snapshot": "4.1.8",
"@vitest/spy": "4.1.8",
"@vitest/utils": "4.1.8",
"es-module-lexer": "^2.0.0",
"expect-type": "^1.3.0",
"magic-string": "^0.30.21",
"obug": "^2.1.1",
"pathe": "^2.0.3",
"picomatch": "^4.0.2",
"std-env": "^3.9.0",
"picomatch": "^4.0.3",
"std-env": "^4.0.0-rc.1",
"tinybench": "^2.9.0",
"tinyexec": "^0.3.2",
"tinyglobby": "^0.2.14",
"tinypool": "^1.1.1",
"tinyrainbow": "^2.0.0",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0",
"vite-node": "3.2.4",
"tinyexec": "^1.0.2",
"tinyglobby": "^0.2.15",
"tinyrainbow": "^3.1.0",
"vite": "^6.0.0 || ^7.0.0 || ^8.0.0",
"why-is-node-running": "^2.3.0"
},
"bin": {
"vitest": "vitest.mjs"
},
"engines": {
"node": "^18.0.0 || ^20.0.0 || >=22.0.0"
"node": "^20.0.0 || ^22.0.0 || >=24.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
},
"peerDependencies": {
"@edge-runtime/vm": "*",
"@types/debug": "^4.1.12",
"@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0",
"@vitest/browser": "3.2.4",
"@vitest/ui": "3.2.4",
"@opentelemetry/api": "^1.9.0",
"@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0",
"@vitest/browser-playwright": "4.1.8",
"@vitest/browser-preview": "4.1.8",
"@vitest/browser-webdriverio": "4.1.8",
"@vitest/coverage-istanbul": "4.1.8",
"@vitest/coverage-v8": "4.1.8",
"@vitest/ui": "4.1.8",
"happy-dom": "*",
"jsdom": "*"
"jsdom": "*",
"vite": "^6.0.0 || ^7.0.0 || ^8.0.0"
},
"peerDependenciesMeta": {
"@edge-runtime/vm": {
"optional": true
},
"@types/debug": {
"@opentelemetry/api": {
"optional": true
},
"@types/node": {
"optional": true
},
"@vitest/browser": {
"@vitest/browser-playwright": {
"optional": true
},
"@vitest/browser-preview": {
"optional": true
},
"@vitest/browser-webdriverio": {
"optional": true
},
"@vitest/coverage-istanbul": {
"optional": true
},
"@vitest/coverage-v8": {
"optional": true
},
"@vitest/ui": {
@ -5657,6 +5583,9 @@
},
"jsdom": {
"optional": true
},
"vite": {
"optional": false
}
}
},
@ -5756,9 +5685,9 @@
}
},
"node_modules/ws": {
"version": "8.19.0",
"resolved": "https://registry.npmjs.org/ws/-/ws-8.19.0.tgz",
"integrity": "sha512-blAT2mjOEIi0ZzruJfIhb3nps74PRWTCz1IjglWEEpQl5XS/UNama6u2/rjFkDDouqr4L67ry+1aGIALViWjDg==",
"version": "8.21.0",
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.0.tgz",
"integrity": "sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==",
"dev": true,
"license": "MIT",
"engines": {

6
frontend/package.json
View file

@ -16,6 +16,7 @@
"@dnd-kit/modifiers": "^9.0.0",
"@dnd-kit/sortable": "^10.0.0",
"@dnd-kit/utilities": "^3.2.2",
"@hookform/resolvers": "^5.2.2",
"@mantine/charts": "~8.0.1",
"@mantine/core": "~8.0.1",
"@mantine/dates": "~8.0.1",
@ -23,17 +24,16 @@
"@mantine/form": "~8.0.1",
"@mantine/hooks": "~8.0.1",
"@mantine/notifications": "~8.0.1",
"@hookform/resolvers": "^5.2.2",
"@tanstack/react-table": "^8.21.2",
"allotment": "^1.20.4",
"dayjs": "^1.11.13",
"hls.js": "^1.5.20",
"react-hook-form": "^7.70.0",
"lucide-react": "^0.511.0",
"mpegts.js": "^1.8.0",
"react": "^19.1.0",
"react-dom": "^19.1.0",
"react-draggable": "^4.4.6",
"react-hook-form": "^7.70.0",
"react-pro-sidebar": "^1.1.0",
"react-router-dom": "^7.3.0",
"react-virtualized": "^9.22.6",
@ -62,7 +62,7 @@
"jsdom": "^27.0.0",
"prettier": "^3.5.3",
"vite": "^7.1.7",
"vitest": "^3.2.4"
"vitest": "^4.1.8"
},
"resolutions": {
"vite": "7.1.7",

160
frontend/src/WebSocket.jsx
View file

@ -50,7 +50,6 @@ export const WebsocketProvider = ({ children }) => {
const epgs = useEPGsStore((s) => s.epgs);
const updateEPG = useEPGsStore((s) => s.updateEPG);
const updateEPGProgress = useEPGsStore((s) => s.updateEPGProgress);
const updatePlaylist = usePlaylistsStore((s) => s.updatePlaylist);
@ -367,19 +366,28 @@ export const WebsocketProvider = ({ children }) => {
fetchEPGData();
break;
case 'single_channel_epg_match': {
const matchResult = parsedEvent.data;
if (matchResult.channel) {
useChannelsStore.getState().updateChannel(matchResult.channel);
}
window.dispatchEvent(
new CustomEvent('single-channel-epg-match', {
detail: matchResult,
})
);
break;
}
case 'epg_match':
notifications.show({
message: parsedEvent.data.message || 'EPG match is complete!',
color: 'green.5',
});
// Check if we have associations data and use the more efficient batch API
if (
parsedEvent.data.associations &&
parsedEvent.data.associations.length > 0
) {
API.batchSetEPG(parsedEvent.data.associations);
}
// Celery already applied assignments server-side; refresh local state.
fetchEPGData();
API.requeryChannels();
break;
case 'epg_matching_progress': {
@ -635,81 +643,91 @@ export const WebsocketProvider = ({ children }) => {
}
break;
case 'epg_refresh':
// If we have source/account info, check if EPG exists before processing
if (parsedEvent.data.source || parsedEvent.data.account) {
const sourceId =
parsedEvent.data.source || parsedEvent.data.account;
const epg = epgs[sourceId];
case 'epg_refresh': {
const sourceId =
parsedEvent.data.source || parsedEvent.data.account;
if (!sourceId) break;
// Only update progress if the EPG still exists in the store
// This prevents crashes when receiving updates for deleted EPGs
if (epg) {
// Update the store with progress information
updateEPGProgress(parsedEvent.data);
} else {
// EPG was deleted, ignore this update
console.debug(
`Ignoring EPG refresh update for deleted EPG ${sourceId}`
// Read from the store directly. connectWebSocket closes over a stale
// epgs snapshot, so a newly created source is missed and the old early-
// return path never reached fetchEPGData on parsing_channels completion.
let {
epgs: epgsState,
updateEPG,
updateEPGProgress,
fetchEPGs,
fetchEPGData,
} = useEPGsStore.getState();
if (!epgsState[sourceId]) {
try {
await fetchEPGs();
} catch (e) {
console.warn(
'Failed to refresh EPG sources for progress update:',
e
);
break;
}
epgsState = useEPGsStore.getState().epgs;
}
if (epg) {
// Check for any indication of an error (either via status or error field)
const hasError =
parsedEvent.data.status === 'error' ||
!!parsedEvent.data.error ||
(parsedEvent.data.message &&
parsedEvent.data.message.toLowerCase().includes('error'));
updateEPGProgress(parsedEvent.data);
if (hasError) {
// Handle error state
const errorMessage =
parsedEvent.data.error ||
parsedEvent.data.message ||
'Unknown error occurred';
const epg = epgsState[sourceId];
if (!epg) break;
updateEPG({
...epg,
status: 'error',
last_message: errorMessage,
});
const hasError =
parsedEvent.data.status === 'error' ||
!!parsedEvent.data.error ||
(parsedEvent.data.message &&
parsedEvent.data.message.toLowerCase().includes('error'));
// Show notification for the error
notifications.show({
title: 'EPG Refresh Error',
message: errorMessage,
color: 'red.5',
});
}
// Update status on completion only if no errors
else if (parsedEvent.data.progress === 100) {
updateEPG({
...epg,
status: parsedEvent.data.status || 'success',
last_message:
parsedEvent.data.message || epg.last_message,
// Use the timestamp from the backend if provided
...(parsedEvent.data.updated_at && {
updated_at: parsedEvent.data.updated_at,
}),
});
if (hasError) {
const errorMessage =
parsedEvent.data.error ||
parsedEvent.data.message ||
'Unknown error occurred';
// Only show success notification if we've finished parsing programs and had no errors
if (parsedEvent.data.action === 'parsing_programs') {
notifications.show({
title: 'EPG Processing Complete',
message: 'EPG data has been updated successfully',
color: 'green.5',
});
updateEPG({
...epg,
status: 'error',
last_message: errorMessage,
});
fetchEPGData();
}
}
notifications.show({
title: 'EPG Refresh Error',
message: errorMessage,
color: 'red.5',
});
} else if (parsedEvent.data.progress === 100) {
updateEPG({
...epg,
status: parsedEvent.data.status || 'success',
last_message: parsedEvent.data.message || epg.last_message,
...(parsedEvent.data.updated_at && {
updated_at: parsedEvent.data.updated_at,
}),
});
if (parsedEvent.data.action === 'parsing_channels') {
notifications.show({
message: 'EPG channels updated!',
color: 'green.5',
});
await fetchEPGData();
} else if (parsedEvent.data.action === 'parsing_programs') {
notifications.show({
title: 'EPG Processing Complete',
message: 'EPG data has been updated successfully',
color: 'green.5',
});
await fetchEPGData();
}
}
break;
}
case 'epg_sources_changed':
// A plugin or backend process signaled that the EPG sources changed

148
frontend/src/api.js
View file

@ -3,6 +3,7 @@ import useAuthStore from './store/auth';
import useChannelsStore from './store/channels';
import useLogosStore from './store/logos';
import useUserAgentsStore from './store/userAgents';
import useServerGroupsStore from './store/serverGroups';
import usePlaylistsStore from './store/playlists';
import useEPGsStore from './store/epgs';
import useStreamsStore from './store/streams';
@ -1305,6 +1306,59 @@ export default class API {
}
}
static async getServerGroups() {
try {
const response = await request(`${host}/api/m3u/server-groups/`);
return response;
} catch (e) {
errorNotification('Failed to retrieve server groups', e);
}
}
static async addServerGroup(values) {
try {
const response = await request(`${host}/api/m3u/server-groups/`, {
method: 'POST',
body: values,
});
useServerGroupsStore.getState().addServerGroup(response);
return response;
} catch (e) {
errorNotification('Failed to create server group', e);
}
}
static async updateServerGroup(values) {
try {
const { id, ...payload } = values;
const response = await request(`${host}/api/m3u/server-groups/${id}/`, {
method: 'PUT',
body: payload,
});
useServerGroupsStore.getState().updateServerGroup(response);
return response;
} catch (e) {
errorNotification('Failed to update server group', e);
}
}
static async deleteServerGroup(id) {
try {
await request(`${host}/api/m3u/server-groups/${id}/`, {
method: 'DELETE',
});
useServerGroupsStore.getState().removeServerGroups([id]);
} catch (e) {
errorNotification('Failed to delete server group', e);
}
}
static async getPlaylist(id) {
try {
const response = await request(`${host}/api/m3u/accounts/${id}/`);
@ -1530,13 +1584,10 @@ export default class API {
}
static async getCurrentProgramForEpg(epgId) {
const response = await request(
`${host}/api/epg/current-programs/`,
{
method: 'POST',
body: { epg_data_ids: [epgId] },
}
);
const response = await request(`${host}/api/epg/current-programs/`, {
method: 'POST',
body: { epg_data_ids: [epgId] },
});
if (response && response.length > 0) {
if (response[0].parsing) {
@ -1650,11 +1701,11 @@ export default class API {
}
}
static async refreshEPG(id) {
static async refreshEPG(id, force = false) {
try {
const response = await request(`${host}/api/epg/import/`, {
method: 'POST',
body: { id },
body: { id, force },
});
return response;
@ -3837,4 +3888,83 @@ export default class API {
errorNotification('Failed to fetch connect logs', e);
}
}
static async getSDLineups(sourceId) {
try {
const response = await request(
`${host}/api/epg/sources/${sourceId}/sd-lineups/`
);
return response;
} catch (e) {
errorNotification('Failed to retrieve Schedules Direct lineups', e);
}
}
static async addSDLineup(sourceId, lineup) {
try {
const response = await request(
`${host}/api/epg/sources/${sourceId}/sd-lineups/`,
{
method: 'POST',
body: { lineup },
}
);
return response;
} catch (e) {
errorNotification(`Failed to add lineup ${lineup}`, e);
}
}
static async deleteSDLineup(sourceId, lineup) {
try {
const response = await request(
`${host}/api/epg/sources/${sourceId}/sd-lineups/`,
{
method: 'DELETE',
body: { lineup },
}
);
return response;
} catch (e) {
errorNotification(`Failed to remove lineup ${lineup}`, e);
}
}
static async updateEpgSourceSettings(sourceId, settings) {
try {
// Read current custom_properties from the store to merge, not replace
const epgs = useEPGsStore.getState().epgs;
const source = epgs[sourceId];
const cp = { ...(source?.custom_properties || {}), ...settings };
const response = await request(`${host}/api/epg/sources/${sourceId}/`, {
method: 'PATCH',
body: { custom_properties: cp },
});
useEPGsStore.getState().updateEPG(response);
return response;
} catch (e) {
errorNotification('Failed to update EPG source settings', e);
}
}
static async updateSDSettings(sourceId, settings) {
return API.updateEpgSourceSettings(sourceId, settings);
}
static async searchSDLineups(sourceId, country, postalcode) {
try {
const response = await request(
`${host}/api/epg/sources/${sourceId}/sd-lineups/search/`,
{
method: 'POST',
body: { country, postalcode },
}
);
return response;
} catch (e) {
errorNotification('Failed to search Schedules Direct lineups', e);
}
}
}

65
frontend/src/components/ProgramDetailModal.jsx
View file

@ -40,11 +40,19 @@ function formatDurationMinutes(startTime, endTime) {
return `${minutes}m`;
}
function resolveApiUrl(url) {
if (!url || url.startsWith('http')) return url;
const apiHost = import.meta.env.DEV
? `http://${window.location.hostname}:5656`
: '';
return `${apiHost}${url}`;
}
function resolveImageUrl(detail) {
if (detail?.tmdb_poster_url) return detail.tmdb_poster_url;
if (detail?.poster_url) return detail.poster_url;
if (detail?.images?.length > 0) return detail.images[0].url;
if (detail?.icon) return detail.icon;
if (detail?.poster_url) return resolveApiUrl(detail.poster_url);
if (detail?.images?.length > 0) return resolveApiUrl(detail.images[0].url);
if (detail?.icon) return resolveApiUrl(detail.icon);
return null;
}
@ -298,6 +306,37 @@ export default function ProgramDetailModal({
</>
)}
{d.content_advisory?.length > 0 && (
<Text size="xs" c="orange" fs="italic">
{d.content_advisory.join(', ')}
</Text>
)}
{d.event_details && (
<>
<Divider color="#333" />
<Stack gap={4}>
{d.event_details.venue100 && (
<Text size="sm" c="dimmed">
<Text span fw={600}>Venue: </Text>
{d.event_details.venue100}
</Text>
)}
{d.event_details.teams?.length > 0 && (
<Text size="sm" c="dimmed">
<Text span fw={600}>Teams: </Text>
{d.event_details.teams.map((t, i) => (
<Text span key={i}>
{i > 0 ? ' vs ' : ''}
{t.name}{t.isHome ? ' (Home)' : ''}
</Text>
))}
</Text>
)}
</Stack>
</>
)}
{hasCredits && (
<>
<Divider color="#333" />
@ -330,21 +369,13 @@ export default function ProgramDetailModal({
</>
)}
{(d.country ||
d.language ||
{(d.language ||
d.original_air_date ||
(d.production_date && d.is_previously_shown) ||
starRatings.length > 0) && (
<>
<Divider color="#333" />
<Group gap="md" wrap="wrap">
{d.country && (
<Text size="sm" c="dimmed">
<Text span fw={600}>
Country:{' '}
</Text>
{d.country}
</Text>
)}
{d.language && (
<Text size="sm" c="dimmed">
<Text span fw={600}>
@ -353,6 +384,14 @@ export default function ProgramDetailModal({
{d.language}
</Text>
)}
{d.production_date && d.is_previously_shown && (
<Text size="sm" c="dimmed">
<Text span fw={600}>
First aired:{' '}
</Text>
{d.production_date}
</Text>
)}
{d.original_air_date && (
<Text size="sm" c="dimmed">
<Text span fw={600}>

28
frontend/src/components/ServerGroupsManagerModal.jsx Normal file
View file

@ -0,0 +1,28 @@
import { Modal } from '@mantine/core';
import ServerGroupsTable from './tables/ServerGroupsTable';
const ServerGroupsManagerModal = ({
isOpen,
onClose,
onGroupCreated,
openCreateOnMount = false,
}) => {
return (
<Modal
opened={isOpen}
onClose={onClose}
title="Server Groups"
size="md"
centered
>
{isOpen ? (
<ServerGroupsTable
onGroupCreated={onGroupCreated}
openCreateOnMount={openCreateOnMount}
/>
) : null}
</Modal>
);
};
export default ServerGroupsManagerModal;

61
frontend/src/components/__tests__/ServerGroupsManagerModal.test.jsx Normal file
View file

@ -0,0 +1,61 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen, waitFor } from '@testing-library/react';
import { MantineProvider } from '@mantine/core';
import ServerGroupsManagerModal from '../ServerGroupsManagerModal';
vi.mock('../../api', () => ({
default: {
deleteServerGroup: vi.fn(),
},
}));
vi.mock('../../store/serverGroups', () => ({
default: (selector) =>
selector({
serverGroups: [
{ id: 1, name: 'Pool A' },
{ id: 2, name: 'Pool B' },
],
isLoading: false,
error: null,
fetchServerGroups: vi.fn().mockResolvedValue(undefined),
}),
}));
vi.mock('../../store/playlists', () => ({
default: (selector) =>
selector({
playlists: [{ id: 10, server_group: 2 }],
}),
}));
vi.mock('../forms/ServerGroup', () => ({
default: () => null,
}));
vi.mock('../../hooks/useLocalStorage', () => ({
default: () => ['default', vi.fn()],
}));
const renderModal = (props = {}) =>
render(
<MantineProvider>
<ServerGroupsManagerModal isOpen onClose={vi.fn()} {...props} />
</MantineProvider>
);
describe('ServerGroupsManagerModal', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('opens without Mantine Tooltip errors', async () => {
expect(() => renderModal()).not.toThrow();
await waitFor(() => {
expect(screen.getByText('Pool A')).toBeInTheDocument();
expect(screen.getByText('Pool B')).toBeInTheDocument();
expect(screen.getByText('Accounts')).toBeInTheDocument();
});
});
});

62
frontend/src/components/forms/Channel.jsx
View file

@ -163,12 +163,24 @@ const ChannelForm = ({ channel: channelProp = null, isOpen, onClose }) => {
}
setAutoMatchLoading(true);
let accepted = false;
try {
const response = await matchChannelEpg(channel);
if (response?.accepted) {
accepted = true;
showNotification({
title: 'Matching in Progress',
message:
response.message ||
'EPG auto-match is running. Results will appear when complete.',
color: 'blue',
});
return;
}
if (response.matched) {
// Update the form with the new EPG data
if (response.channel && response.channel.epg_data_id) {
if (response.channel?.epg_data_id) {
setValue('epg_data_id', response.channel.epg_data_id);
}
@ -192,7 +204,9 @@ const ChannelForm = ({ channel: channelProp = null, isOpen, onClose }) => {
});
console.error('Auto-match error:', error);
} finally {
setAutoMatchLoading(false);
if (!accepted) {
setAutoMatchLoading(false);
}
}
};
@ -354,6 +368,48 @@ const ChannelForm = ({ channel: channelProp = null, isOpen, onClose }) => {
resolver: yupResolver(validationSchema),
});
useEffect(() => {
const onMatchResult = (event) => {
const data = event.detail;
if (!channel?.id || String(data.channel_id) !== String(channel.id)) {
return;
}
if (data.matched && data.channel?.epg_data_id) {
setValue('epg_data_id', data.channel.epg_data_id);
}
showNotification({
title: data.matched ? 'Success' : 'No Match Found',
message: data.message,
color: data.matched ? 'green' : 'orange',
});
setAutoMatchLoading(false);
};
window.addEventListener('single-channel-epg-match', onMatchResult);
return () =>
window.removeEventListener('single-channel-epg-match', onMatchResult);
}, [channel?.id, setValue]);
useEffect(() => {
if (!autoMatchLoading) {
return undefined;
}
const timeoutId = window.setTimeout(() => {
setAutoMatchLoading(false);
showNotification({
title: 'Matching Timed Out',
message:
'EPG auto-match is taking longer than expected. Check back shortly or try again.',
color: 'orange',
});
}, 180_000);
return () => window.clearTimeout(timeoutId);
}, [autoMatchLoading]);
const clearOverrides = async () => {
if (!channel) return;
try {

889
frontend/src/components/forms/EPG.jsx
View file

File diff suppressed because it is too large Load diff

276
frontend/src/components/forms/M3U.jsx
View file

@ -1,11 +1,11 @@
// Modal.js
import React, { useEffect, useState } from 'react';
import useUserAgentsStore from '../../store/userAgents';
import useServerGroupsStore from '../../store/serverGroups';
import M3UProfiles from './M3UProfiles';
import {
Box,
Button,
Checkbox,
Divider,
FileInput,
Flex,
@ -35,6 +35,7 @@ import {
prepareSubmitValues,
updatePlaylist,
} from '../../utils/forms/M3uUtils.js';
import ServerGroupsManagerModal from '../ServerGroupsManagerModal';
const M3U = ({
m3uAccount = null,
@ -43,6 +44,7 @@ const M3U = ({
playlistCreated = false,
}) => {
const userAgents = useUserAgentsStore((s) => s.userAgents);
const serverGroups = useServerGroupsStore((s) => s.serverGroups);
const fetchChannelGroups = useChannelsStore((s) => s.fetchChannelGroups);
const fetchEPGs = useEPGsStore((s) => s.fetchEPGs);
const fetchCategories = useVODStore((s) => s.fetchCategories);
@ -54,6 +56,9 @@ const M3U = ({
const [groupFilterModalOpen, setGroupFilterModalOpen] = useState(false);
const [filterModalOpen, setFilterModalOpen] = useState(false);
const [scheduleType, setScheduleType] = useState('interval');
const [serverGroupsManagerOpen, setServerGroupsManagerOpen] = useState(false);
const [serverGroupsCreateOnOpen, setServerGroupsCreateOnOpen] =
useState(false);
const form = useForm({
mode: 'uncontrolled',
@ -61,6 +66,7 @@ const M3U = ({
name: '',
server_url: '',
user_agent: '0',
server_group: '0',
is_active: true,
max_streams: 0,
refresh_interval: 24,
@ -88,6 +94,9 @@ const M3U = ({
server_url: m3uAccount.server_url,
max_streams: m3uAccount.max_streams,
user_agent: m3uAccount.user_agent ? `${m3uAccount.user_agent}` : '0',
server_group: m3uAccount.server_group
? `${m3uAccount.server_group}`
: '0',
is_active: m3uAccount.is_active,
refresh_interval: m3uAccount.refresh_interval,
cron_expression: m3uAccount.cron_expression || '',
@ -203,7 +212,7 @@ const M3U = ({
return (
<>
<Modal
size={700}
size={960}
opened={isOpen}
onClose={close}
title="M3U Account"
@ -216,10 +225,9 @@ const M3U = ({
<LoadingOverlay visible={form.submitting} overlayBlur={2} />
<form onSubmit={form.onSubmit(onSubmit)}>
<Group justify="space-between" align="top">
<Stack gap="5" style={{ flex: 1, minWidth: 0 }}>
<Group align="flex-start" gap="md" wrap="nowrap">
<Stack gap="xs" style={{ flex: 1, minWidth: 0 }}>
<TextInput
style={{ width: '100%' }}
id="name"
name="name"
label="Name"
@ -228,7 +236,6 @@ const M3U = ({
key={form.key('name')}
/>
<TextInput
style={{ width: '100%' }}
id="server_url"
name="server_url"
label="URL"
@ -236,7 +243,6 @@ const M3U = ({
{...form.getInputProps('server_url')}
key={form.key('server_url')}
/>
<Select
id="account_type"
name="account_type"
@ -248,49 +254,15 @@ const M3U = ({
</>
}
data={[
{
value: 'STD',
label: 'Standard',
},
{
value: 'XC',
label: 'Xtream Codes',
},
{ value: 'STD', label: 'Standard' },
{ value: 'XC', label: 'Xtream Codes' },
]}
key={form.key('account_type')}
{...form.getInputProps('account_type')}
/>
{form.getValues().account_type == 'XC' && (
<Box>
{!m3uAccount && (
<Group justify="space-between">
<Box>Create EPG</Box>
<Switch
id="create_epg"
name="create_epg"
description="Automatically create matching EPG source for this Xtream account"
key={form.key('create_epg')}
{...form.getInputProps('create_epg', {
type: 'checkbox',
})}
/>
</Group>
)}
<Group justify="space-between">
<Box>Enable VOD Scanning</Box>
<Switch
id="enable_vod"
name="enable_vod"
description="Scan and import VOD content (movies/series) from this Xtream account"
key={form.key('enable_vod')}
{...form.getInputProps('enable_vod', {
type: 'checkbox',
})}
/>
</Group>
<>
<TextInput
id="username"
name="username"
@ -298,7 +270,6 @@ const M3U = ({
description="Username for Xtream Codes authentication"
{...form.getInputProps('username')}
/>
<PasswordInput
id="password"
name="password"
@ -306,7 +277,7 @@ const M3U = ({
description="Password for Xtream Codes authentication (leave empty to keep existing)"
{...form.getInputProps('password')}
/>
</Box>
</>
)}
{form.getValues().account_type != 'XC' && (
@ -326,7 +297,6 @@ const M3U = ({
},
}}
/>
<DateTimePicker
label="Expiration Date"
description="Set an expiration date to receive a warning notification"
@ -342,9 +312,8 @@ const M3U = ({
<Divider size="sm" orientation="vertical" />
<Stack gap="5" style={{ flex: 1 }}>
<Stack gap="xs" style={{ flex: 1, minWidth: 0 }}>
<NumberInput
style={{ width: '100%' }}
id="max_streams"
name="max_streams"
label="Max Streams"
@ -354,7 +323,41 @@ const M3U = ({
{...form.getInputProps('max_streams')}
key={form.key('max_streams')}
/>
<Select
id="server_group"
name="server_group"
label="Server Group"
description="Share login limits across accounts in a server group. Set max streams on each profile (unlimited profiles skip group enforcement)."
key={form.key('server_group')}
value={form.getValues().server_group}
onChange={(value) => {
if (value === '__new__') {
setServerGroupsCreateOnOpen(true);
setServerGroupsManagerOpen(true);
return;
}
form.setFieldValue('server_group', value);
}}
data={[
{ value: '0', label: '(None)' },
...serverGroups.map((group) => ({
label: group.name,
value: `${group.id}`,
})),
{ value: '__new__', label: '+ Add server group...' },
]}
/>
<Button
variant="subtle"
size="compact-xs"
onClick={() => {
setServerGroupsCreateOnOpen(false);
setServerGroupsManagerOpen(true);
}}
style={{ alignSelf: 'flex-start' }}
>
Manage server groups
</Button>
<Select
id="user_agent"
name="user_agent"
@ -369,7 +372,11 @@ const M3U = ({
}))
)}
/>
</Stack>
<Divider size="sm" orientation="vertical" />
<Stack gap="xs" style={{ flex: 1, minWidth: 0 }}>
<ScheduleInput
scheduleType={scheduleType}
onScheduleTypeChange={setScheduleType}
@ -390,7 +397,6 @@ const M3U = ({
</>
}
/>
<NumberInput
min={0}
max={365}
@ -399,74 +405,114 @@ const M3U = ({
{...form.getInputProps('stale_stream_days')}
/>
<NumberInput
min={0}
max={999}
label="VOD Priority"
description="Priority for VOD provider selection (higher numbers = higher priority). Used when multiple providers offer the same content."
{...form.getInputProps('priority')}
key={form.key('priority')}
/>
{form.getValues().account_type == 'XC' && (
<Box>
<NumberInput
min={0}
max={999}
label="VOD Priority"
description="Priority for VOD provider selection (higher numbers = higher priority). Used when multiple providers offer the same content."
{...form.getInputProps('priority')}
key={form.key('priority')}
/>
<Group justify="space-between">
<Box>Enable VOD Scanning</Box>
<Switch
id="enable_vod"
name="enable_vod"
description="Scan and import VOD content (movies/series) from this Xtream account"
key={form.key('enable_vod')}
{...form.getInputProps('enable_vod', {
type: 'checkbox',
})}
/>
</Group>
{!m3uAccount && (
<Group justify="space-between">
<Box>Create EPG</Box>
<Switch
id="create_epg"
name="create_epg"
description="Automatically create matching EPG source for this Xtream account"
key={form.key('create_epg')}
{...form.getInputProps('create_epg', {
type: 'checkbox',
})}
/>
</Group>
)}
</Box>
)}
</Stack>
</Group>
<Divider my="md" />
<Flex gap="xl" wrap="wrap">
<Checkbox
<Flex
gap="md"
justify="space-between"
align="center"
wrap="wrap"
mih={50}
>
<Switch
id="is_active"
name="is_active"
label="Is Active"
description="Enable or disable this M3U account"
{...form.getInputProps('is_active', { type: 'checkbox' })}
key={form.key('is_active')}
{...form.getInputProps('is_active', { type: 'checkbox' })}
/>
</Flex>
<Flex mih={50} gap="xs" justify="flex-end" align="flex-end">
{playlist && (
<>
<Button
variant="filled"
size="sm"
onClick={() => setFilterModalOpen(true)}
>
Filters
</Button>
<Button
variant="filled"
// color={theme.custom.colors.buttonPrimary}
size="sm"
onClick={() => {
// If this is an XC account with VOD enabled, fetch VOD categories
if (
m3uAccount?.account_type === 'XC' &&
m3uAccount?.enable_vod
) {
fetchCategories();
}
setGroupFilterModalOpen(true);
}}
>
Groups
</Button>
<Button
variant="filled"
// color={theme.custom.colors.buttonPrimary}
size="sm"
onClick={() => setProfileModalOpen(true)}
>
Profiles
</Button>
</>
)}
<Flex gap="xs" align="center">
{playlist && (
<>
<Button
variant="filled"
size="sm"
onClick={() => setFilterModalOpen(true)}
>
Filters
</Button>
<Button
variant="filled"
// color={theme.custom.colors.buttonPrimary}
size="sm"
onClick={() => {
// If this is an XC account with VOD enabled, fetch VOD categories
if (
m3uAccount?.account_type === 'XC' &&
m3uAccount?.enable_vod
) {
fetchCategories();
}
setGroupFilterModalOpen(true);
}}
>
Groups
</Button>
<Button
variant="filled"
// color={theme.custom.colors.buttonPrimary}
size="sm"
onClick={() => setProfileModalOpen(true)}
>
Profiles
</Button>
</>
)}
<Button
type="submit"
variant="filled"
disabled={form.submitting}
size="sm"
>
Save
</Button>
<Button
type="submit"
variant="filled"
disabled={form.submitting}
size="sm"
>
Save
</Button>
</Flex>
</Flex>
</form>
</Modal>
@ -489,6 +535,20 @@ const M3U = ({
/>
</>
)}
<ServerGroupsManagerModal
isOpen={serverGroupsManagerOpen}
onClose={() => {
setServerGroupsManagerOpen(false);
setServerGroupsCreateOnOpen(false);
}}
openCreateOnMount={serverGroupsCreateOnOpen}
onGroupCreated={(group) => {
if (group?.id) {
form.setFieldValue('server_group', `${group.id}`);
}
}}
/>
</>
);
};

86
frontend/src/components/forms/ServerGroup.jsx Normal file
View file

@ -0,0 +1,86 @@
import React, { useEffect, useMemo } from 'react';
import { useForm } from 'react-hook-form';
import { yupResolver } from '@hookform/resolvers/yup';
import * as Yup from 'yup';
import API from '../../api';
import { Button, Flex, Modal, TextInput } from '@mantine/core';
const schema = Yup.object({
name: Yup.string().required('Name is required'),
});
const ServerGroupForm = ({
serverGroup = null,
isOpen,
onClose,
onSaved,
}) => {
const defaultValues = useMemo(
() => ({
name: serverGroup?.name || '',
}),
[serverGroup]
);
const {
register,
handleSubmit,
formState: { errors, isSubmitting },
reset,
} = useForm({
defaultValues,
resolver: yupResolver(schema),
});
const onSubmit = async (values) => {
let response;
if (serverGroup?.id) {
response = await API.updateServerGroup({ id: serverGroup.id, ...values });
} else {
response = await API.addServerGroup(values);
}
if (response) {
onSaved?.(response);
}
reset();
onClose();
};
useEffect(() => {
reset(defaultValues);
}, [defaultValues, reset]);
if (!isOpen) {
return null;
}
return (
<Modal
opened={isOpen}
onClose={onClose}
title="Server Group"
centered
withinPortal
zIndex={400}
>
<form onSubmit={handleSubmit(onSubmit)}>
<TextInput
label="Name"
description="Accounts in this group share connection limits when they use the same provider login. Limits come from each account profile's max streams."
{...register('name')}
error={errors.name?.message}
/>
<Flex mih={50} gap="xs" justify="flex-end" align="flex-end">
<Button size="small" type="submit" disabled={isSubmitting}>
Submit
</Button>
</Flex>
</form>
</Modal>
);
};
export default ServerGroupForm;

19
frontend/src/components/forms/__tests__/Channel.test.jsx
View file

@ -703,6 +703,25 @@ describe('ChannelForm', () => {
expect(autoMatch).not.toBeDisabled();
});
it('shows in-progress notification when matchChannelEpg returns accepted', async () => {
const channel = makeChannel();
vi.mocked(ChannelUtils.matchChannelEpg).mockResolvedValue({
accepted: true,
message: 'EPG matching started',
});
setupMocks({ channel });
render(<ChannelForm {...defaultProps({ channel })} />);
fireEvent.click(screen.getByText('Auto Match'));
await waitFor(() => {
expect(showNotification).toHaveBeenCalledWith(
expect.objectContaining({
title: 'Matching in Progress',
color: 'blue',
})
);
});
});
it('calls matchChannelEpg with the channel on click', async () => {
const channel = makeChannel();
vi.mocked(ChannelUtils.matchChannelEpg).mockResolvedValue({

75
frontend/src/components/forms/__tests__/EPG.test.jsx
View file

@ -2,6 +2,20 @@ import { render, screen, fireEvent, waitFor } from '@testing-library/react';
import { describe, it, expect, vi, beforeEach } from 'vitest';
// ── Utility mocks ──────────────────────────────────────────────────────────────
global.fetch = vi.fn().mockResolvedValue({
json: () => Promise.resolve({}),
});
vi.mock('../../../api.js', () => ({
default: {
getSDLineups: vi.fn().mockResolvedValue([]),
addSDLineup: vi.fn().mockResolvedValue({ success: true }),
deleteSDLineup: vi.fn().mockResolvedValue({ success: true }),
searchSDLineups: vi.fn().mockResolvedValue([]),
updateSDSettings: vi.fn().mockResolvedValue({}),
},
}));
vi.mock('../../../utils/notificationUtils.js', () => ({
showNotification: vi.fn(),
}));
@ -184,6 +198,62 @@ vi.mock('@mantine/core', async () => ({
{error && <span data-testid="input-error">{error}</span>}
</div>
),
Alert: ({ children, title, color, icon }) => (
<div data-testid="alert" data-color={color}>
{title && <div data-testid="alert-title">{title}</div>}
{children}
</div>
),
Select: ({ label, value, onChange, data, placeholder }) => (
<select
aria-label={label}
data-testid={`select-${label?.toString().toLowerCase().replace(/\s+/g, '-')}`}
value={value ?? ''}
onChange={(e) => onChange?.(e.target.value)}
>
<option value="">{placeholder}</option>
{(data ?? []).map((opt) => {
const val = typeof opt === 'string' ? opt : opt.value;
const lbl = typeof opt === 'string' ? opt : opt.label;
return <option key={val} value={val}>{lbl}</option>;
})}
</select>
),
Loader: ({ size }) => <div data-testid="loader" data-size={size} />,
Badge: ({ children, color }) => <span data-testid="badge" data-color={color}>{children}</span>,
ScrollArea: ({ children }) => <div data-testid="scroll-area">{children}</div>,
Table: ({ children }) => <table>{children}</table>,
Tooltip: ({ children }) => <div>{children}</div>,
Switch: ({ label, checked, onChange, disabled, description }) => (
<label>
<input
type="checkbox"
data-testid={`switch-${label?.toString().toLowerCase().replace(/\s+/g, '-')}`}
checked={checked ?? false}
onChange={(e) => onChange?.(e)}
disabled={disabled}
/>
{label}
</label>
),
UnstyledButton: ({ children, onClick, ...props }) => (
<button type="button" onClick={onClick} {...props}>{children}</button>
),
Alert: ({ children, title, color, icon }) => (
<div data-testid="alert" data-color={color}><strong>{title}</strong>{children}</div>
),
Stack: ({ children, gap }) => <div data-testid="stack">{children}</div>,
Text: ({ children, ...props }) => <span>{children}</span>,
TextInput: ({ label, value, onChange, placeholder, ...props }) => (
<input
type="text"
aria-label={label}
data-testid={`input-${label?.toString().toLowerCase().replace(/\s+/g, '-')}`}
value={value || ''}
onChange={onChange}
placeholder={placeholder}
/>
),
}));
// ── Imports after mocks ────────────────────────────────────────────────────────
@ -373,10 +443,11 @@ describe('EPG', () => {
expect(screen.queryByTestId('input-api-key')).not.toBeInTheDocument();
});
it('shows API key input when source type requires it', () => {
it('shows username and password inputs when source type is schedules_direct', () => {
const epg = makeEPG({ source_type: 'schedules_direct' });
render(<EPG {...defaultProps({ epg })} />);
expect(screen.getByTestId('input-api-key')).toBeInTheDocument();
expect(screen.getByTestId('input-username')).toBeInTheDocument();
expect(screen.getByTestId('input-password')).toBeInTheDocument();
});
});

313
frontend/src/components/tables/EPGsTable.jsx
View file

@ -99,52 +99,23 @@ const RowActions = ({ tableSize, row, editEPG, deleteEPG, refreshEPG }) => {
);
};
const EPGsTable = () => {
const [epg, setEPG] = useState(null);
const [epgModalOpen, setEPGModalOpen] = useState(false);
const [dummyEpgModalOpen, setDummyEpgModalOpen] = useState(false);
const [rowSelection, setRowSelection] = useState([]);
const [confirmDeleteOpen, setConfirmDeleteOpen] = useState(false);
const [deleteTarget, setDeleteTarget] = useState(null);
const [epgToDelete, setEpgToDelete] = useState(null);
const [data, setData] = useState([]);
const [deleting, setDeleting] = useState(false);
const epgs = useEPGsStore((s) => s.epgs);
const refreshProgress = useEPGsStore((s) => s.refreshProgress);
const EPGStatusCell = ({ epg }) => {
// Direct Zustand subscription scoped to this source only.
// This component re-renders whenever its source's progress changes,
// independent of the parent table's render cycle.
const progress = useEPGsStore((s) => s.refreshProgress[epg.id]);
const theme = useMantineTheme();
const { fullDateTimeFormat } = useDateTimeFormat();
const [tableSize] = useLocalStorage('table-size', 'default');
const isWarningSuppressed = useWarningsStore((s) => s.isWarningSuppressed);
const suppressWarning = useWarningsStore((s) => s.suppressWarning);
const toggleActive = async (epg) => {
try {
// Validate that epg is a valid object with an id
if (!epg || typeof epg !== 'object' || !epg.id) {
console.error('toggleActive called with invalid epg:', epg);
return;
}
// Send only the is_active field to trigger our special handling
await API.updateEPG(
{
id: epg.id,
is_active: !epg.is_active,
},
true
); // Add a new parameter to indicate this is just a toggle
} catch (error) {
console.error('Error toggling active state:', error);
}
};
const buildProgressDisplay = (data) => {
const progress = refreshProgress[data.id] || null;
if (!progress) return null;
const isDummyEPG = epg.source_type === 'dummy';
if (isDummyEPG) return null;
// Show progress bar if an active fetch is in progress
if (
progress &&
(progress.progress < 100 ||
progress.status === 'in_progress' ||
(progress.action === 'parsing_channels' && epg.status === 'parsing'))
) {
let label = '';
switch (progress.action) {
case 'downloading':
@ -163,7 +134,6 @@ const EPGsTable = () => {
return null;
}
// Build additional info string from progress data
let additionalInfo = '';
if (progress.message) {
additionalInfo = progress.message;
@ -201,6 +171,96 @@ const EPGsTable = () => {
)}
</Stack>
);
}
// Show error message
if (epg.status === 'error' && epg.last_message) {
return (
<Tooltip label={epg.last_message} multiline width={300}>
<Text
c="dimmed"
size="xs"
lineClamp={2}
style={{ color: theme.colors.red[6], lineHeight: 1.3 }}
>
{epg.last_message}
</Text>
</Tooltip>
);
}
// Show success message
if (epg.status === 'success') {
const successMessage =
epg.last_message || 'EPG data refreshed successfully';
return (
<Tooltip label={successMessage} multiline width={300}>
<Text
c="dimmed"
size="xs"
lineClamp={2}
style={{ color: theme.colors.green[6], lineHeight: 1.3 }}
>
{successMessage}
</Text>
</Tooltip>
);
}
// Show idle message
if (epg.status === 'idle' && epg.last_message) {
return (
<Tooltip label={epg.last_message} multiline width={300}>
<Text c="dimmed" size="xs" lineClamp={2} style={{ lineHeight: 1.3 }}>
{epg.last_message}
</Text>
</Tooltip>
);
}
return null;
};
const EPGsTable = () => {
const [epg, setEPG] = useState(null);
const [epgModalOpen, setEPGModalOpen] = useState(false);
const [dummyEpgModalOpen, setDummyEpgModalOpen] = useState(false);
const [rowSelection, setRowSelection] = useState([]);
const [confirmDeleteOpen, setConfirmDeleteOpen] = useState(false);
const [deleteTarget, setDeleteTarget] = useState(null);
const [epgToDelete, setEpgToDelete] = useState(null);
const [data, setData] = useState([]);
const [deleting, setDeleting] = useState(false);
const [confirmSDRefreshOpen, setConfirmSDRefreshOpen] = useState(false);
const [sdRefreshTarget, setSDRefreshTarget] = useState(null);
const epgs = useEPGsStore((s) => s.epgs);
const theme = useMantineTheme();
const { fullDateTimeFormat } = useDateTimeFormat();
const [tableSize] = useLocalStorage('table-size', 'default');
const isWarningSuppressed = useWarningsStore((s) => s.isWarningSuppressed);
const suppressWarning = useWarningsStore((s) => s.suppressWarning);
const toggleActive = async (epg) => {
try {
// Validate that epg is a valid object with an id
if (!epg || typeof epg !== 'object' || !epg.id) {
console.error('toggleActive called with invalid epg:', epg);
return;
}
// Send only the is_active field to trigger our special handling
await API.updateEPG(
{
id: epg.id,
is_active: !epg.is_active,
},
true
); // Add a new parameter to indicate this is just a toggle
} catch (error) {
console.error('Error toggling active state:', error);
}
};
const columns = useMemo(
@ -214,21 +274,42 @@ const EPGsTable = () => {
{
header: 'Type',
accessorKey: 'source_type',
size: 100,
size: 130,
cell: ({ cell }) => {
const typeMap = {
xmltv: 'XMLTV',
schedules_direct: 'Schedules Direct',
dummy: 'Custom Dummy',
};
return typeMap[cell.getValue()] || cell.getValue();
},
},
{
header: 'URL / API Key / File Path',
header: 'Source / Credentials / File Path',
accessorKey: 'url',
enableSorting: false,
minSize: 250,
cell: ({ cell, row }) => {
const value =
cell.getValue() ||
row.original.api_key ||
row.original.file_path ||
'';
const sourceType = row.original.source_type;
let value = '';
let tooltip = '';
if (sourceType === 'schedules_direct') {
// Never expose credentials — show username only
const username = row.original.username || '';
value = username ? `User: ${username}` : '(credentials set)';
tooltip = value;
} else {
value =
cell.getValue() ||
row.original.password ||
row.original.file_path ||
'';
tooltip = value;
}
return (
<Tooltip label={value} disabled={!value}>
<Tooltip label={tooltip} disabled={!tooltip}>
<div
style={{
whiteSpace: 'nowrap',
@ -267,76 +348,7 @@ const EPGsTable = () => {
enableSorting: false,
minSize: 250,
grow: true,
cell: ({ row }) => {
const data = row.original;
const isDummyEPG = data.source_type === 'dummy';
// Dummy EPGs don't have status messages
if (isDummyEPG) {
return null;
}
// Check if there's an active progress for this EPG - show progress first if active
if (
refreshProgress[data.id] &&
refreshProgress[data.id].progress < 100
) {
return buildProgressDisplay(data);
}
// Show error message when status is error
if (data.status === 'error' && data.last_message) {
return (
<Tooltip label={data.last_message} multiline width={300}>
<Text
c="dimmed"
size="xs"
lineClamp={2}
style={{ color: theme.colors.red[6], lineHeight: 1.3 }}
>
{data.last_message}
</Text>
</Tooltip>
);
}
// Show success message for successful sources
if (data.status === 'success') {
const successMessage =
data.last_message || 'EPG data refreshed successfully';
return (
<Tooltip label={successMessage} multiline width={300}>
<Text
c="dimmed"
size="xs"
lineClamp={2}
style={{ color: theme.colors.green[6], lineHeight: 1.3 }}
>
{successMessage}
</Text>
</Tooltip>
);
}
// Show last_message for idle sources (from previous refresh)
if (data.status === 'idle' && data.last_message) {
return (
<Tooltip label={data.last_message} multiline width={300}>
<Text
c="dimmed"
size="xs"
lineClamp={2}
style={{ lineHeight: 1.3 }}
>
{data.last_message}
</Text>
</Tooltip>
);
}
// Otherwise return empty cell
return null;
},
cell: ({ row }) => <EPGStatusCell epg={row.original} />,
},
{
header: 'Updated',
@ -380,14 +392,15 @@ const EPGsTable = () => {
size: tableSize == 'compact' ? 75 : 100,
},
],
[refreshProgress, fullDateTimeFormat]
[fullDateTimeFormat]
);
const [isLoading, setIsLoading] = useState(true);
const [sorting, setSorting] = useState([]);
const editEPG = async (epg = null) => {
setEPG(epg);
const freshEpg = epg?.id ? epgs[epg.id] || epg : epg;
setEPG(freshEpg);
// Open the appropriate modal based on source type
if (epg?.source_type === 'dummy') {
setDummyEpgModalOpen(true);
@ -430,13 +443,27 @@ const EPGsTable = () => {
}
};
const refreshEPG = async (id) => {
await API.refreshEPG(id);
const refreshEPG = async (id, force = false) => {
await API.refreshEPG(id, force);
notifications.show({
title: 'EPG refresh initiated',
});
};
const handleRefreshEPG = (id) => {
const epgObj = epgs[id];
if (
epgObj?.source_type === 'schedules_direct' &&
epgObj?.updated_at &&
Date.now() - new Date(epgObj.updated_at).getTime() < 2 * 60 * 60 * 1000
) {
setSDRefreshTarget(id);
setConfirmSDRefreshOpen(true);
return;
}
refreshEPG(id);
};
const closeEPGForm = () => {
setEPG(null);
setEPGModalOpen(false);
@ -469,7 +496,7 @@ const EPGsTable = () => {
row={row}
editEPG={editEPG}
deleteEPG={deleteEPG}
refreshEPG={refreshEPG}
refreshEPG={handleRefreshEPG}
/>
);
}
@ -677,6 +704,28 @@ const EPGsTable = () => {
onClose={closeDummyEPGForm}
/>
<ConfirmationDialog
opened={confirmSDRefreshOpen}
onClose={() => setConfirmSDRefreshOpen(false)}
onConfirm={() => {
setConfirmSDRefreshOpen(false);
refreshEPG(sdRefreshTarget, true);
}}
title="Refresh Schedules Direct Early?"
message={
<div>
<p>This source was refreshed less than 2 hours ago.</p>
<p>
Schedules Direct rate-limits requests per account. Refreshing too
frequently may cause your account to be temporarily blocked.
</p>
<p>Are you sure you want to force a refresh now?</p>
</div>
}
confirmLabel="Refresh Anyway"
cancelLabel="Cancel"
/>
<ConfirmationDialog
opened={confirmDeleteOpen}
onClose={() => setConfirmDeleteOpen(false)}
@ -691,10 +740,12 @@ const EPGsTable = () => {
Name: ${epgToDelete.name}
Source Type: ${epgToDelete.source_type}
${
epgToDelete.url
? `URL: ${epgToDelete.url}`
: epgToDelete.api_key
? `API Key: ${epgToDelete.api_key}`
epgToDelete.source_type === 'schedules_direct'
? epgToDelete.username
? `Username: ${epgToDelete.username}`
: '(credentials set)'
: epgToDelete.url
? `URL: ${epgToDelete.url}`
: epgToDelete.file_path
? `File Path: ${epgToDelete.file_path}`
: ''

47
frontend/src/components/tables/M3UsTable.jsx
View file

@ -8,6 +8,7 @@ import React, {
import API from '../../api';
import usePlaylistsStore from '../../store/playlists';
import M3UForm from '../forms/M3U';
import ServerGroupsManagerModal from '../ServerGroupsManagerModal';
import { TableHelper } from '../../helpers';
import {
useMantineTheme,
@ -144,6 +145,7 @@ const M3UTable = () => {
const [data, setData] = useState([]);
const [sorting, setSorting] = useState([{ id: 'name', desc: '' }]);
const [deleting, setDeleting] = useState(false);
const [serverGroupsManagerOpen, setServerGroupsManagerOpen] = useState(false);
const playlists = usePlaylistsStore((s) => s.playlists);
const refreshProgress = usePlaylistsStore((s) => s.refreshProgress);
@ -988,21 +990,31 @@ const M3UTable = () => {
>
M3U Accounts
</Text>
<Button
leftSection={<SquarePlus size={14} />}
variant="light"
size="xs"
onClick={() => editPlaylist()}
p={5}
color="green"
style={{
borderWidth: '1px',
borderColor: 'green',
color: 'white',
}}
>
Add M3U
</Button>
<Flex gap={6}>
<Button
variant="light"
size="xs"
onClick={() => setServerGroupsManagerOpen(true)}
p={5}
>
Server Groups
</Button>
<Button
leftSection={<SquarePlus size={14} />}
variant="light"
size="xs"
onClick={() => editPlaylist()}
p={5}
color="green"
style={{
borderWidth: '1px',
borderColor: 'green',
color: 'white',
}}
>
Add M3U
</Button>
</Flex>
</Flex>
<Paper
@ -1051,6 +1063,11 @@ const M3UTable = () => {
playlistCreated={playlistCreated}
/>
<ServerGroupsManagerModal
isOpen={serverGroupsManagerOpen}
onClose={() => setServerGroupsManagerOpen(false)}
/>
<ConfirmationDialog
opened={confirmDeleteOpen}
onClose={() => setConfirmDeleteOpen(false)}

298
frontend/src/components/tables/ServerGroupsTable.jsx Normal file
View file

@ -0,0 +1,298 @@
import { useEffect, useMemo, useRef, useState } from 'react';
import API from '../../api';
import useServerGroupsStore from '../../store/serverGroups';
import usePlaylistsStore from '../../store/playlists';
import useWarningsStore from '../../store/warnings';
import ServerGroupForm from '../forms/ServerGroup';
import ConfirmationDialog from '../ConfirmationDialog';
import {
ActionIcon,
Box,
Button,
Center,
Flex,
Stack,
Text,
} from '@mantine/core';
import { SquareMinus, SquarePen, SquarePlus } from 'lucide-react';
import { CustomTable, useTable } from './CustomTable';
import './table.css';
const TABLE_WIDTH = 360;
const ACTIONS_COLUMN_SIZE = 76;
const ACCOUNTS_COLUMN_SIZE = 72;
const RowActions = ({ row, editServerGroup, deleteServerGroup }) => (
<Flex gap={4} justify="center" w="100%">
<ActionIcon
variant="transparent"
size="sm"
color="yellow.5"
onClick={() => editServerGroup(row.original)}
>
<SquarePen size="18" />
</ActionIcon>
<ActionIcon
variant="transparent"
size="sm"
color="red.9"
onClick={() => deleteServerGroup(row.original.id)}
>
<SquareMinus size="18" />
</ActionIcon>
</Flex>
);
const ServerGroupsTable = ({ onGroupCreated, openCreateOnMount = false }) => {
const [serverGroup, setServerGroup] = useState(null);
const [serverGroupModalOpen, setServerGroupModalOpen] = useState(false);
const [confirmDeleteOpen, setConfirmDeleteOpen] = useState(false);
const [deleteTarget, setDeleteTarget] = useState(null);
const [groupToDelete, setGroupToDelete] = useState(null);
const [deleting, setDeleting] = useState(false);
const openedCreateOnMount = useRef(false);
const isWarningSuppressed = useWarningsStore((s) => s.isWarningSuppressed);
const suppressWarning = useWarningsStore((s) => s.suppressWarning);
const serverGroups = useServerGroupsStore((state) => state.serverGroups);
const isLoading = useServerGroupsStore((state) => state.isLoading);
const error = useServerGroupsStore((state) => state.error);
const fetchServerGroups = useServerGroupsStore(
(state) => state.fetchServerGroups
);
const playlists = usePlaylistsStore((state) => state.playlists);
const tableData = useMemo(
() =>
serverGroups.map((group) => ({
...group,
accountCount: playlists.filter(
(playlist) => playlist.server_group === group.id
).length,
})),
[serverGroups, playlists]
);
const columns = useMemo(
() => [
{
header: 'Name',
accessorKey: 'name',
grow: true,
minSize: 100,
cell: ({ cell }) => (
<Text size="sm" truncate>
{cell.getValue()}
</Text>
),
},
{
header: 'Accounts',
accessorKey: 'accountCount',
size: ACCOUNTS_COLUMN_SIZE,
minSize: 65,
cell: ({ cell }) => (
<Center w="100%">
<Text size="sm">{cell.getValue() ?? 0}</Text>
</Center>
),
},
{
id: 'actions',
header: 'Actions',
size: ACTIONS_COLUMN_SIZE,
minSize: 65,
},
],
[]
);
const editServerGroup = (group = null) => {
setServerGroup(group);
setServerGroupModalOpen(true);
};
const executeDeleteServerGroup = async (id) => {
setDeleting(true);
try {
await API.deleteServerGroup(id);
} finally {
setDeleting(false);
setConfirmDeleteOpen(false);
}
};
const deleteServerGroup = (id) => {
const group = tableData.find((item) => item.id === id);
setGroupToDelete(group);
setDeleteTarget(id);
if (isWarningSuppressed('delete-server-group')) {
return executeDeleteServerGroup(id);
}
setConfirmDeleteOpen(true);
};
const closeServerGroupForm = () => {
setServerGroup(null);
setServerGroupModalOpen(false);
};
const handleServerGroupSaved = (savedGroup) => {
if (!serverGroup?.id) {
onGroupCreated?.(savedGroup);
}
};
useEffect(() => {
fetchServerGroups();
}, [fetchServerGroups]);
useEffect(() => {
if (!openCreateOnMount) {
openedCreateOnMount.current = false;
return;
}
if (!isLoading && !openedCreateOnMount.current) {
openedCreateOnMount.current = true;
editServerGroup();
}
}, [openCreateOnMount, isLoading]);
const renderHeaderCell = (header) => (
<Text size="sm" name={header.id}>
{header.column.columnDef.header}
</Text>
);
const renderBodyCell = ({ row }) => (
<RowActions
row={row}
editServerGroup={editServerGroup}
deleteServerGroup={deleteServerGroup}
/>
);
const table = useTable({
columns,
data: tableData,
allRowIds: tableData.map((group) => group.id),
bodyCellRenderFns: {
actions: renderBodyCell,
},
headerCellRenderFns: {
name: renderHeaderCell,
accountCount: renderHeaderCell,
actions: renderHeaderCell,
},
});
if (isLoading) {
return (
<Center py="md">
<Text size="sm">Loading server groups...</Text>
</Center>
);
}
if (error) {
return (
<Center py="md">
<Text size="sm" c="red">
{error}
</Text>
</Center>
);
}
return (
<Stack gap="md">
<Text size="sm" c="dimmed">
Group accounts that share the same provider login so their connection
limits are enforced together. Assign a group when editing an M3U
account.
</Text>
<Flex justify="center">
<Stack gap="xs" w={TABLE_WIDTH} maw="100%">
<Flex justify="flex-end">
<Button
leftSection={<SquarePlus size={18} />}
variant="light"
size="xs"
onClick={() => editServerGroup()}
p={5}
color="green"
title="Create a shared connection pool for multiple accounts"
style={{
borderWidth: '1px',
borderColor: 'green',
color: 'white',
}}
>
Add Server Group
</Button>
</Flex>
<Box
style={{
maxHeight: 280,
overflowY: 'auto',
overflowX: 'auto',
border: 'solid 1px rgb(68,68,68)',
borderRadius: 'var(--mantine-radius-default)',
}}
>
{tableData.length === 0 ? (
<Center py="lg" px="xl">
<Text size="sm" c="dimmed">
No server groups yet.
</Text>
</Center>
) : (
<CustomTable table={table} />
)}
</Box>
</Stack>
</Flex>
<ServerGroupForm
serverGroup={serverGroup}
isOpen={serverGroupModalOpen}
onClose={closeServerGroupForm}
onSaved={handleServerGroupSaved}
/>
<ConfirmationDialog
opened={confirmDeleteOpen}
onClose={() => setConfirmDeleteOpen(false)}
onConfirm={() => executeDeleteServerGroup(deleteTarget)}
loading={deleting}
title="Confirm Server Group Deletion"
message={
groupToDelete ? (
<div style={{ whiteSpace: 'pre-line' }}>
{`Are you sure you want to delete the following server group?
Name: ${groupToDelete.name}
Accounts: ${groupToDelete.accountCount ?? 0}
Accounts in this group will no longer share connection limits. This action cannot be undone.`}
</div>
) : (
'Are you sure you want to delete this server group? This action cannot be undone.'
)
}
confirmLabel="Delete"
cancelLabel="Cancel"
actionKey="delete-server-group"
onSuppressChange={suppressWarning}
zIndex={401}
/>
</Stack>
);
};
export default ServerGroupsTable;

83
frontend/src/components/tables/__tests__/ServerGroupsTable.test.jsx Normal file
View file

@ -0,0 +1,83 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { render, screen, waitFor } from '@testing-library/react';
import { MantineProvider } from '@mantine/core';
import ServerGroupsTable from '../ServerGroupsTable';
const mockServerGroupsState = vi.hoisted(() => ({
serverGroups: [
{ id: 1, name: 'Pool A' },
{ id: 2, name: 'Pool B' },
],
isLoading: false,
error: null,
fetchServerGroups: vi.fn().mockResolvedValue(undefined),
}));
const renderTable = () =>
render(
<MantineProvider>
<ServerGroupsTable />
</MantineProvider>
);
vi.mock('../../../api', () => ({
default: {
deleteServerGroup: vi.fn(),
},
}));
vi.mock('../../../store/serverGroups', () => ({
default: (selector) => selector(mockServerGroupsState),
}));
vi.mock('../../../store/playlists', () => ({
default: (selector) =>
selector({
playlists: [
{ id: 10, server_group: 2 },
{ id: 11, server_group: 1 },
],
}),
}));
vi.mock('../../forms/ServerGroup', () => ({
default: () => null,
}));
describe('ServerGroupsTable', () => {
beforeEach(() => {
vi.clearAllMocks();
mockServerGroupsState.serverGroups = [
{ id: 1, name: 'Pool A' },
{ id: 2, name: 'Pool B' },
];
mockServerGroupsState.isLoading = false;
mockServerGroupsState.error = null;
mockServerGroupsState.fetchServerGroups = vi
.fn()
.mockResolvedValue(undefined);
});
it('renders server groups without tooltip errors', async () => {
expect(() => renderTable()).not.toThrow();
await waitFor(() => {
expect(screen.getByText('Pool A')).toBeInTheDocument();
expect(screen.getByText('Pool B')).toBeInTheDocument();
expect(screen.getByText('Accounts')).toBeInTheDocument();
});
});
it('shows store error message when fetch fails', async () => {
mockServerGroupsState.serverGroups = [];
mockServerGroupsState.error = 'Failed to load server groups.';
renderTable();
await waitFor(() => {
expect(
screen.getByText('Failed to load server groups.')
).toBeInTheDocument();
});
});
});

2
frontend/src/store/auth.jsx
View file

@ -6,6 +6,7 @@ import useEPGsStore from './epgs';
import useStreamProfilesStore from './streamProfiles';
import useOutputProfilesStore from './outputProfiles';
import useUserAgentsStore from './userAgents';
import useServerGroupsStore from './serverGroups';
import useUsersStore from './users';
import API from '../api';
import { USER_LEVELS } from '../constants';
@ -124,6 +125,7 @@ const useAuthStore = create((set, get) => ({
useStreamProfilesStore.getState().fetchProfiles(),
useOutputProfilesStore.getState().fetchProfiles(),
useUserAgentsStore.getState().fetchUserAgents(),
useServerGroupsStore.getState().fetchServerGroups(),
useChannelsStore.getState().fetchChannelIds(),
]);

40
frontend/src/store/serverGroups.jsx Normal file
View file

@ -0,0 +1,40 @@
import { create } from 'zustand';
import api from '../api';
const useServerGroupsStore = create((set) => ({
serverGroups: [],
isLoading: false,
error: null,
fetchServerGroups: async () => {
set({ isLoading: true, error: null });
try {
const serverGroups = await api.getServerGroups();
set({ serverGroups: serverGroups || [], isLoading: false });
} catch (error) {
console.error('Failed to fetch server groups:', error);
set({ error: 'Failed to load server groups.', isLoading: false });
}
},
addServerGroup: (serverGroup) =>
set((state) => ({
serverGroups: [...state.serverGroups, serverGroup],
})),
updateServerGroup: (serverGroup) =>
set((state) => ({
serverGroups: state.serverGroups.map((group) =>
group.id === serverGroup.id ? serverGroup : group
),
})),
removeServerGroups: (serverGroupIds) =>
set((state) => ({
serverGroups: state.serverGroups.filter(
(group) => !serverGroupIds.includes(group.id)
),
})),
}));
export default useServerGroupsStore;

4
frontend/src/utils/forms/M3uUtils.js
View file

@ -50,5 +50,9 @@ export const prepareSubmitValues = (values, expDate) => {
prepared.user_agent = null;
}
if (prepared.server_group == '0') {
prepared.server_group = null;
}
return prepared;
};

71
tests/test_websocket_utils.py Normal file
View file

@ -0,0 +1,71 @@
"""Tests for WebSocket delivery context detection in core.utils."""
from unittest.mock import MagicMock, patch
from django.test import SimpleTestCase
class WebSocketContextDetectionTests(SimpleTestCase):
def test_sync_not_used_without_gevent_patch(self):
from core.utils import _should_use_sync_websocket_send
with patch("core.utils._is_gevent_monkey_patched", return_value=False), patch(
"core.utils._is_celery_worker_context", return_value=True
):
self.assertFalse(_should_use_sync_websocket_send())
def test_sync_not_used_in_gevent_uwsgi_without_celery(self):
from core.utils import _should_use_sync_websocket_send
with patch("core.utils._is_gevent_monkey_patched", return_value=True), patch(
"core.utils._is_celery_worker_context", return_value=False
):
self.assertFalse(_should_use_sync_websocket_send())
def test_sync_used_in_celery_with_gevent_patch(self):
from core.utils import _should_use_sync_websocket_send
with patch("core.utils._is_gevent_monkey_patched", return_value=True), patch(
"core.utils._is_celery_worker_context", return_value=True
):
self.assertTrue(_should_use_sync_websocket_send())
class SendWebsocketUpdateRoutingTests(SimpleTestCase):
def test_celery_gevent_uses_sync_redis_path(self):
from core.utils import send_websocket_update
with patch("core.utils._should_use_sync_websocket_send", return_value=True), patch(
"core.utils._gevent_ws_send"
) as mock_sync, patch("core.utils.get_channel_layer") as mock_layer:
send_websocket_update("updates", "update", {"type": "epg_refresh"})
mock_sync.assert_called_once()
mock_layer.assert_not_called()
def test_gevent_uwsgi_uses_spawn(self):
from core.utils import send_websocket_update
with patch("core.utils._should_use_sync_websocket_send", return_value=False), patch(
"core.utils._is_gevent_monkey_patched", return_value=True
), patch("gevent.spawn") as mock_spawn, patch(
"core.utils.get_channel_layer"
) as mock_layer:
send_websocket_update("updates", "update", {"type": "epg_refresh"})
mock_spawn.assert_called_once()
mock_layer.assert_not_called()
def test_plain_context_uses_channel_layer(self):
from core.utils import send_websocket_update
mock_layer = MagicMock()
with patch("core.utils._should_use_sync_websocket_send", return_value=False), patch(
"core.utils._is_gevent_monkey_patched", return_value=False
), patch("core.utils.get_channel_layer", return_value=mock_layer), patch(
"core.utils.async_to_sync", side_effect=lambda fn: fn
) as mock_async_to_sync:
send_websocket_update("updates", "update", {"type": "epg_refresh"})
mock_async_to_sync.assert_called_once()
mock_layer.group_send.assert_called_once()

2
version.py
View file

@ -1,5 +1,5 @@
"""
Dispatcharr version information.
"""
__version__ = '0.25.1' # Follow semantic versioning (MAJOR.MINOR.PATCH)
__version__ = '0.26.0' # Follow semantic versioning (MAJOR.MINOR.PATCH)
__timestamp__ = None # Set during CI/CD build process