Mirrors/Dispatcharr
1
0
Fork 1
mirror of https://github.com/Dispatcharr/Dispatcharr.git synced 2026-07-22 01:30:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

Merge branch 'dev' of https://github.com/Dispatcharr/Dispatcharr into pr/sethwv/1333

Browse source
This commit is contained in:
SergeantPanda 2026-07-02 15:01:57 -05:00
commit 0b29edbef7
205 changed files with 35993 additions and 6054 deletions
Download patch file Download diff file Expand all files Collapse all files

145
CHANGELOG.md
View file

@ -9,27 +9,155 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **XC catch-up (timeshift) support**. Adds a native `/timeshift/{user}/{pass}/{stream_id}/{timestamp}/{duration}.ts` endpoint that proxies replay sessions from an Xtream Codes provider to clients such as iPlayTV (Apple TV) and TiviMate. Auth reuses the same `xc_password` custom-property the live `/live/.../{id}.ts` endpoint already uses. Catch-up sessions appear on `/stats` with a violet `TIMESHIFT` badge alongside live sessions and respect per-channel access rules. (Closes #133) — Thanks [@cedric-marcoux](https://github.com/cedric-marcoux)
- **Catch-up flags** (`tv_archive`, `tv_archive_duration`) denormalized at import time for zero-cost output queries; end-of-refresh SQL rollup updates channels linked to the refreshed account and self-heals stale flags on those channels only (e.g. after catch-up streams are removed or an account is deactivated).
- **Strictly-UTC API surface, automatic per-provider timezone.** `server_info.timezone` is always `UTC` and the XC EPG `start`/`end` strings are emitted in UTC; the proxy converts the requested instant to the serving provider's own reported timezone (`server_info.timezone` captured on account refresh) at request time. No timezone to configure.
- **Multi-provider failover.** Catch-up walks the channel's catch-up streams in order (like live playback): if one provider cannot serve the archive the next is tried, each attempt with its own account context (credentials, reported timezone, user-agent). Accounts that fail with an auth/ban-class status are not retried via their other streams.
- **Provider pool accounting.** Catch-up reserves a provider profile slot (`connection_pool`) before connecting upstream — same contract as live and VOD — and releases it when the session ends. When the default profile is at capacity it walks the account's alternate profiles with their own resolved credentials, and answers `503` when every profile is full.
- **Per-client session pool.** The first request without `session_id` receives a `301` redirect to the same URL with a stable `?session_id=` query parameter; that ID becomes the client identity for stats, stop keys, and pool coordination. Each viewer gets their own Redis-backed pool entry even when watching the same programme; idle sessions can be reclaimed via fingerprint matching (same user and programme, IP + user-agent score). Real scrubs within a session stop the in-flight stream through the standard stop-key mechanism; parallel startup probes (full-file, open-ended, and EOF tail `Range` requests) are deferred with `503` instead of opening extra upstream connections. Stream-limit exemptions for `ignore_same_channel_connections` apply only to sibling requests from the same `session_id`.
- **`xmltv_prev_days_override` under `Settings → EPG`** (default `0` = auto-detect). Verbose timeshift logging follows the standard logger DEBUG level.
- **EPG XMLTV `prev_days` auto-detection** from the provider's largest `tv_archive_duration` (capped at 30 days), overridable via setting or `?prev_days=` URL parameter.
- **Byte path streams directly via `iter_content` + generator yield** (same pattern as the VOD proxy), with an upfront MPEG-TS sync peek that strips any PHP-warning preamble before bytes reach strict demuxers. Throughput stays comfortably above the typical 5 Mbps FHD bitrate so clients don't buffer.
- **Frontend unit tests extended to table components.** `ChannelsTable`, `StreamsTable`, `M3UsTable`, `EPGsTable`, `LogosTable`, `CustomTable`, and related header/editable subcomponents now have Vitest + Testing Library suites. Table business logic was extracted into `frontend/src/utils/tables/*` (with shared `M3uTableUtils` for M3U/EPG sort headers) and covered by dedicated util tests. `dateTimeUtils.formatDuration` gained a `human` precision mode for readable content lengths. — Thanks [@nick4810](https://github.com/nick4810)
### Changed
- **VOD proxy now fails over across M3U accounts when the selected account is at capacity.** `stream_vod()` and `head_vod()` previously picked the highest-priority relation and returned HTTP 503 if that account's profile pool was full, without trying other accounts carrying the same title. The proxy now materialises active relations in a single query, walks them in priority order (preferred stream/account first), and streams from the first account with spare capacity—matching live-channel failover. (Closes #1385) — Thanks [@francescodg89-crypto](https://github.com/francescodg89-crypto)
### Performance
- **M3U/XC stream refresh is faster on large accounts.** Steady-state refreshes split `bulk_update` into a lightweight touch pass (`last_seen` / `is_stale` only) for unchanged streams and a full column update only when provider metadata or catch-up fields actually change.
- **M3U stream filters compile once per refresh.** Account filters are regex-compiled before batch workers start; accounts with no filters skip per-stream filter checks entirely.
- **M3U refresh releases parse catalogs sooner.** Standard accounts stream-parse the on-disk M3U instead of loading the full file into RAM; `extinf_data` is dropped immediately after batch DB work, before stale cleanup and auto-sync.
- **XC live refresh avoids redundant work during catalog filtering.** `collect_xc_streams` skips disabled categories before building entries and uses a shared URL prefix instead of formatting each stream URL separately. Auto-sync releases per-group logo/EPG caches after each group iteration.
- **Celery workers return RSS after memory-intensive tasks.** `cleanup_memory()` accepts an optional `trim_heap` flag (glibc `malloc_trim`); Celery `task_postrun` enables it after `close_old_connections()` for M3U account/group refresh, EPG, VOD, and channel-matching tasks so worker memory drops back toward baseline instead of ratcheting across successive large jobs.
### Fixed
- **Celery workers no longer use django-db-geventpool (fixes Postgres protocol errors under concurrent tasks).** Celery's default queue runs prefork (`--autoscale=6,1`), not gevent, but it was sharing the same `django-db-geventpool` backend as uWSGI. That pool is a process-wide singleton of warm connections; `fork()` (including autoscale spawning new children on demand) duplicated those sockets across processes and corrupted Postgres session state (`the last operation didn't produce records (command status: SET/BEGIN/ROLLBACK)`, `connection ... in transaction status INTRANS`, and matching server-side `there is already a transaction in progress` on one backend). Celery workers and beat now use Django's standard PostgreSQL backend (`CONN_MAX_AGE=0`, no warm pool); uWSGI keeps geventpool. Plugin discovery moved from `worker_ready` (arbiter) to `worker_process_init` (each child after `connections.close_all()`), so the prefork parent no longer opens DB handles that children would inherit.
- **EPG channel parsing progress no longer exceeds 100%.** During `parsing_channels`, progress used the pre-parse database channel count as the denominator while the numerator counted channels found in the XML. When the guide grew (or on sources where the file had more channels than the DB), the UI could show values well above 100% (e.g. ~300%). The estimate now bumps when the XML exceeds the DB baseline, progress is capped at 98% until final cleanup, and update frequency adapts to the known total (~20 ticks on steady-state refreshes; coarse every-100-channel updates for first import or when the file outgrows the estimate). A final in-loop update reports the true parsed count before completion.
- **EPG refresh and per-channel parses no longer race.** Full-source refresh holds an `epg_source_file` lock through download, channel parse, and bulk programme swap; programme index builds acquire the same lock separately (after refresh releases it). Per-channel `parse_programs_for_tvg_id` tasks run concurrently for different tvg-ids (no file lock between them) but defer while a source refresh is in progress. Refresh waits until in-flight per-channel parses finish (Redis counter) before downloading. Channels matched mid-refresh are excluded from the bulk-parse snapshot but receive a follow-up per-channel parse when refresh finishes; orphan cleanup at swap time uses live channel assignments so mid-refresh matches are not wiped. Duplicate tasks for the same `epg_id` are deduped via `parse_epg_programs` lock with a log noting how many channels map to that EPG. Busy file/index deferrals retry for up to two hours (15s interval).
- **Per-channel EPG parse no longer wipes guide data on failure.** `parse_programs_for_tvg_id` used to delete existing programmes before streaming the XML and flush new rows in batches as it parsed, so any failure partway through left a mapped channel with no guide data. It also re-fetched the `EPGData` row mid-task instead of reusing the instance loaded at task start. The task now reuses the initial row and replaces programmes in one transaction at the end (delete old, insert new), so a parse failure leaves the previous guide intact.
- **M3U refresh completion counts now reflect actual stream changes.** The "updated" count previously included every existing stream because the summary treated `last_seen` touch-only rows as updates; it now counts only streams whose provider metadata changed. "Total processed" includes unchanged streams separately. The completion message, WebSocket payload, and parsing notification now also report how many streams were **marked stale** (missing from this refresh, pending retention-gated deletion) versus **removed** (deleted this run).
- **M3U group processing retries on poisoned DB connections.** `_db_query_with_retry` now treats psycopg desync errors (`DatabaseError`, e.g. `lost synchronization with server`) as transient; `process_groups` relationship loading uses it so a stale Celery worker connection resets once instead of failing the whole refresh.
- **Auto Channel Sync's Find and Replace preview now matches the rename it performs.** The preview rendered the literal `$1` instead of substituting numbered capture groups, and compiled patterns with a stricter engine than the rename, so patterns like `^*` previewed a change the sync silently skipped. The live rename now uses the same `regex` engine as the preview (with a substitution timeout to bound catastrophic backtracking), both apply the `$1`→`\1` conversion through one shared helper, and a rename whose result exceeds the channel-name column length is truncated instead of aborting the whole sync. (Fixes #1332) — Thanks [@CodeBormen](https://github.com/CodeBormen)
- **XC refresh no longer wipes auto-created channels on an empty provider fetch.** When `collect_xc_streams()` returned no live streams (transient upstream failure, fetch exception, or no enabled category matched), the refresh used to continue into stale-marking and auto-sync, which deleted the account's entire auto-created channel lineup. An empty XC result now aborts before those steps, sets the account to `ERROR`, and preserves the existing lineup—matching the standard M3U path's empty/failed-download guards. (Fixes #1377) — Thanks [@Jacob-Lasky](https://github.com/Jacob-Lasky)
- **`ChannelUtils.requeryChannels()` returns the API promise again** so bulk channel delete, drag reorder, and inline channel-number saves await the table refetch before clearing selection or continuing. — Thanks [@nick4810](https://github.com/nick4810)
- **VOD connection card Content Duration badge** shows human-readable lengths (`45m`, `2h 0m`) again on the Stats page instead of bare minute integers. — Thanks [@nick4810](https://github.com/nick4810)
- **Frontend unit tests pass on Node 25+ again.** Node 25+ exposes a native `localStorage` stub on `globalThis` that lacks Storage API methods (`getItem`, `clear`, etc.), which shadows jsdom's implementation and breaks tests that touch `localStorage` (e.g. `TypeError: localStorage.clear is not a function`). `setupTests.js` now installs an in-memory shim on `globalThis` and `window` when those methods are missing; the shim is skipped automatically once Node provides a working implementation. — Thanks [@nick4810](https://github.com/nick4810)
## [0.27.2] - 2026-06-30
### Added
- **New proxy setting: Client Connect Grace Period (`channel_client_wait_period`, default 5s).** Adds a dedicated timeout for channels that have filled their buffer but still have no viewers (`waiting_for_clients`). Previously that window reused `channel_shutdown_delay` (default 0s), so those channels were torn down almost immediately.
### Changed
- **Proxy grace-period settings are now split into three distinct timeouts.** The cleanup watchdog already applied `channel_init_grace_period` while a channel was still connecting (buffer not ready) and reused `channel_shutdown_delay` once `connection_ready_time` was set, including for `waiting_for_clients` with zero viewers. With the default `channel_shutdown_delay` of 0s, a buffered channel waiting for its first viewer was stopped almost immediately; raising shutdown delay was the only workaround, but that also delayed teardown after real disconnects. Behaviour is now:
- **`channel_init_grace_period` (default 60s, max 300s):** how long the proxy may spend connecting and cycling failover streams before giving up on startup.
- **`channel_client_wait_period` (default 5s):** how long a ready channel with no viewers stays up waiting for the first client (the original grace-period use case).
- **`channel_shutdown_delay` (default 0s):** delay after the last client disconnects only; no longer applies when the buffer is ready but no viewer has connected yet.
- **Migration 0026 bumps `channel_init_grace_period` to 60s when the stored value is below 60.** Existing installs on the old 5s default (or any custom value under 60) are raised automatically. Values already at 60s or higher are unchanged. If you previously raised `channel_shutdown_delay` to keep buffered channels alive with no viewers, set `channel_client_wait_period` instead (Settings → Proxy → Advanced).
- **Proxy settings UI: less-used options moved under Advanced.** Settings → Proxy now shows the day-to-day tuning fields by default (`buffering_timeout`, `buffering_speed`, `channel_shutdown_delay`, `new_client_behind_seconds`). **Buffer Chunk TTL**, **Channel Initialization Timeout**, and **Client Connect Grace Period** are tucked under **Show Advanced Settings**.
## [0.27.1] - 2026-06-25
### Security
- Updated `Django` 6.0.5 → 6.0.6, resolving the following CVEs:
- **CVE-2026-6873**: Signed cookie salt namespace collision in `HttpRequest.get_signed_cookie()`.
- **CVE-2026-7666**: Potential unencrypted email transmission via STARTTLS in the SMTP backend.
- **CVE-2026-8404**: Potential private data exposure via case-sensitive `Cache-Control` directives in `UpdateCacheMiddleware`.
- **CVE-2026-35193**: Potential private data exposure via missing `Vary: Authorization` in `UpdateCacheMiddleware`.
- **CVE-2026-48587**: Potential private data exposure via whitespace padding in the `Vary` header.
- Updated frontend npm dependencies to resolve 4 audit vulnerabilities (1 low, 2 moderate, 1 high):
- Updated `vite` 7.3.2 → 7.3.5, resolving **moderate** NTLMv2 hash disclosure via UNC path handling on Windows ([GHSA-v6wh-96g9-6wx3](https://github.com/advisories/GHSA-v6wh-96g9-6wx3)) and **high** `server.fs.deny` bypass on Windows alternate paths ([GHSA-fx2h-pf6j-xcff](https://github.com/advisories/GHSA-fx2h-pf6j-xcff))
- Updated `js-yaml` 4.1.1 → 5.1.0, resolving **moderate** quadratic-complexity DoS in merge key handling via repeated aliases ([GHSA-h67p-54hq-rp68](https://github.com/advisories/GHSA-h67p-54hq-rp68))
- Updated `esbuild` 0.27.3 → 0.28.1, resolving **low** arbitrary file read when running the development server on Windows ([GHSA-g7r4-m6w7-qqqr](https://github.com/advisories/GHSA-g7r4-m6w7-qqqr))
### Added
- **Isolated backend test settings (`dispatcharr.settings_test`).** `python manage.py test` now switches to this module automatically (via `manage.py`). It creates an empty PostgreSQL `test_<dbname>` database (same engine as production), uses the standard Postgres backend instead of geventpool so `TestCase` transactions isolate correctly, and leaves Celery tasks queued (no eager `post_save` signal runs during tests). Set `TEST_USE_SQLITE=1` for an in-memory SQLite fallback when Postgres is unavailable.
### Performance
- **`get_vod_streams` and `get_series` XC API endpoints are faster and no longer exhaust Docker `/dev/shm`.** Large libraries (e.g. 125k movies) previously ran one wide `DISTINCT ON` query with parallel workers, which could fail with `could not resize shared memory segment … No space left on device` on the default 64MB container shm. Both endpoints now fetch display columns via `.values()` (no ORM model instantiation per row). Redundant `category` and `logo` joins were dropped in favor of FK ids; alphabetical sort runs in SQL. Typical full-library response time drops from ~23–28s to ~8–10s with stable shm usage.
- **XMLTV EPG export is faster and no longer balloons worker memory.** `generate_epg()` was reworked end-to-end for large guides. (Fixes #1366)
- Streams incrementally: on a cache miss each chunk is pushed to a Redis list as it is yielded (no `''.join()` in the worker); repeat requests within 300s stream chunks back from Redis. `malloc_trim` runs after cold builds.
- Channel streams are prefetched once (only `id`/`name`) instead of one query per custom-dummy channel; dummy `EPGData` programme existence is bulk-checked in a single query.
- The primary channel id is escaped once per `epg_id` group instead of once per programme (~750k fewer `html.escape` calls on a large guide).
- The channel query no longer JOINs multi-MB `programme_index` blobs per channel (~13s saved on a ~2000-channel guide; indices live in `EPGSourceIndex`).
- Programme export uses `(epg_id, id)` keyset pagination with a per-source `start_time` sort; a matching composite index on `ProgramData` (created `CONCURRENTLY` on PostgreSQL) lets each chunk use an ordered index range scan instead of re-sorting every chunk.
- **EPG grid endpoint releases its payload memory back to the OS.** `/api/epg/grid/` drops the redundant full-list copy when appending dummy programmes and runs `malloc_trim` once the response is sent, so worker RSS no longer ratchets up ~20MB per request.
### Changed
- **`programme_index` moved off `EPGSource` into a dedicated `EPGSourceIndex` table.** The multi-MB byte-offset index was repeatedly pulled into web and Celery workers by ordinary `EPGSource` queries and `select_related` JOINs (which ignore manager-level `defer()`). It now lives in a one-to-one `EPGSourceIndex` row, read only when explicitly accessed through the `EPGSource.programme_index` property, so no list, detail, or JOIN query can load it by accident. Migration `0026` copies existing indices across. No API or index-build behavior change.
- **EPG generation extracted into `apps/output/epg.py`.** All XMLTV output logic (`generate_epg`, `generate_dummy_programs`, `generate_custom_dummy_programs`, `generate_dummy_epg`, and supporting helpers) moved from `apps/output/views.py` into a dedicated module. `views.py` retains the thin HTTP endpoint wrappers and auth checks; `epg.py` handles all content generation. No behavior change.
- **Channel list with nested streams loads faster.** `GET /api/channels/channels/?include_streams=true` (Channels UI and single-channel fetch) now builds nested stream payloads from the prefetched `channelstream_set` instead of issuing one extra streams M2M query per channel.
- Dependency updates:
- `Django` 6.0.5 → 6.0.6 (security patch; see Security section)
- `requests` 2.33.1 → 2.34.2
- `gevent` 26.4.0 → 26.5.0
- `torch` 2.11.0+cpu → 2.12.1+cpu
- `sentence-transformers` 5.4.1 → 5.6.0
- `lxml` 6.1.0 → 6.1.1
### Fixed
- **Channel Initialization Grace Period is honoured during live stream startup.** Preview and playback no longer abort after a hardcoded 10s while the channel is still connecting with an empty buffer; the TS generator init-wait stall check and upstream health monitor now use the configured `channel_init_grace_period` (same as the server cleanup watchdog) instead of `CONNECTION_TIMEOUT`. (Fixes #1380)
- **Channels are marked `active` as soon as the buffer threshold is met and a client is streaming.** Once the initial buffer fills, state is set to `active` immediately when viewers are attached, or `waiting_for_clients` when the buffer is ready but no client is connected yet (e.g. proxy API warmup). The cleanup watchdog no longer waits an extra grace period before promoting to `active`. Proxy settings copy now describes `channel_init_grace_period` as an initialization buffer timeout.
- **DVR recording playback auth is complete for native video, HLS segments, and redirects.** Completed recordings use `/file/` with native `<video src>`, which cannot send `Authorization` headers. Playback accepts `?token=` query-param JWT (matching VOD streams), and the floating video player appends the token when assigning native URLs. The explicit HLS URL route (`/api/channels/recordings/.../hls/...`) now registers the same authenticators as the ViewSet `@action` handlers—previously only header JWT applied on that path, so `?token=` failed for native HLS clients. When the request used `?token=`, rewritten playlist segment URLs and `/file/`↔`/hls/` redirects preserve the token; hls.js clients that authenticate via `Authorization` are unchanged. `QueryParamJWTAuthentication` reads `request.query_params` on DRF requests.
- **In-progress DVR playback no longer jumps to the live edge.** The floating video player still opens in-progress recordings at the start of the seekable range, but hls.js was configured with `liveMaxLatencyDurationCount: 10`, which forced the playhead forward to "now" shortly after playback began. Live-edge sync is now disabled for recording HLS so users can watch, pause, and scrub from the beginning while the recording continues. (Fixes #1329)
- **`refresh_single_m3u_account` no longer re-raises after setting account ERROR.** Matches `refresh_epg_data`: the account status and UI already reflect the failure; Celery no longer marks the task FAILED after the terminal error state is persisted.
- **M3U refresh recovers DB connections and account status after failures.** Celery workers now call `close_old_connections()` before and after every task; `refresh_single_m3u_account` also resets its DB connection at task start and retries account loads once after a connection reset (avoids opaque `list index out of range` errors from poisoned worker connections). Accounts leave `fetching`/`parsing` for a terminal `error` or `success` state when refresh aborts. Error-path status updates use `QuerySet.update()` on a clean connection instead of `save()` on a connection that may already be INTRANS after `refresh_m3u_groups` or batch ORM failures. Failed group downloads now set `error` instead of returning silently. Refresh start replaces stale `last_message` text. `refresh_account_profiles` releases DB connections after each profile failure. (Fixes #1338)
- **EPG refresh recovers DB connections and source status after failures.** `refresh_epg_data` now mirrors the M3U hardening: connection reset at task start/end, retried source load after a poisoned-worker connection reset, `QuerySet.update()` for error status on a clean connection, and a `finally` guard that moves sources stuck in `fetching`/`parsing` to `error`. Programme index builds are skipped when programme parsing fails. `parse_channels_only` now persists the error status when a missing file has no URL to refetch.
- **M3U `custom_properties` JSON normalization.** Legacy rows and some API writes could store a JSON-encoded string instead of an object in `custom_properties`, causing refresh, profile sync, and auto-sync to fail with `'str' object has no attribute 'get'`. M3U account, profile, and group-relation models normalize on `save()`; group settings bulk writes and read/merge paths use shared helpers in `core.utils` without re-parsing dict values.
- **`POST /api/epg/import/` no longer loads the full EPG source row.** The dummy-source guard uses a narrow existence query instead of `objects.get()`, keeping the import trigger lightweight. Request/response shape is unchanged for the UI, plugins, and external importers.
- **XC live playback URL building now normalizes account server URLs.** On-demand live URLs (introduced for Server Group profile rotation in 0.27.0) rebuild `/live/{user}/{pass}/{stream_id}.ts` from current credentials instead of reusing the synced `stream.url`. That path now runs `normalize_server_url()` so pasted API URLs (e.g. `/player_api.php` with query params) are stripped while sub-paths like `/server1` are preserved. `get_transformed_credentials()` normalizes the base URL at the source; VOD movie and episode relations use the same helper instead of constructing a throwaway `XCClient` (which also avoided per-request HTTP session setup). (Fixes #1363)
- **Live proxy now releases geventpool DB connections on more paths.** `stream_ts()` calls `close_old_connections()` before `StreamingHttpResponse` so clients waiting on channel init do not keep a pool slot while the generator polls Redis. `generate_stream_url()`, `get_stream_info_for_switch()`, `get_alternate_streams()`, and `get_connections_left()` release in `finally` blocks after ORM work on stream-manager failover paths that run outside Django's request cycle. TS client disconnect cleanup matches fMP4, and `ChannelService` metadata helpers release after their ORM lookups.
- **OpenAPI schema paths for nested M3U profiles and filters no longer contain escaped slashes.** Router regex prefixes used unnecessary `\/`, which drf-spectacular serialized literally into the schema and broke client generators (e.g. oapi-codegen). Runtime URL matching is unchanged. (Fixes #1384)
- **Auto-sync range conflict warning no longer flags a group's own channels when using a channel-group override.** The M3U group settings "Range conflict" check compared occupant channel groups against the source group being configured. Auto-sync with `group_override` creates channels in the override target group, so those channels were misclassified as conflicts. The frontend now resolves the effective target group (override target when set, otherwise the source group) for classification. Genuine conflicts—manual channels, other accounts, different groups, or user-pinned numbers—still surface. (Fixes #1331) — Thanks [@CodeBormen](https://github.com/CodeBormen)
## [0.27.0] - 2026-06-16
### Added
- **Manual Server Groups for shared M3U connection limits.** The existing `ServerGroup` model and account FK are now wired into live and VOD playback. Accounts assigned to the same group share a credential-scoped Redis counter when their provider logins match (hashed fingerprint); unrelated logins in the same group keep separate counters. Enforcement uses each profile's `max_streams` - not a group-wide cap - and profiles with `max_streams=0` skip credential pooling for that profile while still rotating on their own per-profile counter. New `apps/m3u/connection_pool.py` centralizes reserve/release, profile rotation, and credential moves on profile switch. (Closes #1137) — Thanks [@Goldenfreddy0703](https://github.com/Goldenfreddy0703)
- **Server Groups manager UI** on the M3U Accounts page: create, rename, and delete groups with account counts and delete confirmation. Groups load on login via a new Zustand store and REST helpers (`/api/m3u/server-groups/`).
- **M3U account form** adds a Server Group picker (including inline “add group”), reorganized into three columns (source/auth, connection limits, sync/content), and a Manage server groups shortcut.
- **VOD profile selection** uses `pool_has_capacity_for_profile()` so grouped accounts respect shared credential limits before a profile is chosen (non-grouped accounts behave as before).
- **Live profile switches** move the shared credential counter when the new profile uses a different provider login; same-login switches leave the credential counter unchanged.
- **Credential release keys** stored at reserve time allow counters to be released even if the M3U profile row is deleted afterward.
- **PostgreSQL `application_name` tagging by process role.** Pool connections now set `application_name` at connect time (e.g. `Dispatcharr-uwsgi-{pid}`, `Dispatcharr-celery-worker-{pid}`, `Dispatcharr-celery-dvr-{pid}`) so `pg_stat_activity` shows which Dispatcharr process owns each backend instead of a generic client label.
### Changed
- **Plugin repo manifests support split download and metadata base URLs.** `download_base_url` and `metadata_base_url` are optional alternatives to `root_url` in the plugin repository manifest, so repo authors can serve metadata (manifests, icons) and release zips from different origins without absolute URLs everywhere. Manifest and icon URLs resolve via `metadata_base_url` then `root_url`; latest/download URLs resolve via `download_base_url` then `root_url`. When `metadata_base_url` is set and a plugin entry has `manifest_url` but no `icon_url`, the icon defaults to `logo.png` in the same directory as the per-plugin manifest. Manifests using only `root_url` behave identically to before. — Thanks [@sethwv](https://github.com/sethwv)
- **Live and VOD connection slot logic now routes through `connection_pool`.** `Channel.get_stream()`, direct `Stream.get_stream()`, VOD profile selection, and the multi-worker VOD connection manager all call `reserve_profile_slot()` / `release_profile_slot()` instead of inline Redis INCR/DECR. VOD profile selection also checks `pool_has_capacity_for_profile()` before choosing a profile.
- **Live XC upstream URLs use current credentials.** `_resolve_live_stream_url()` builds `/live/{user}/{pass}/{stream_id}.ts` from transformed account credentials and the stream's provider `stream_id`, so playback stays aligned with the active login after credential or profile changes instead of reusing a stale `stream.url` from sync.
- **Channel stream switches check pooled capacity.** `get_stream_info_for_switch()` uses `profile_available_for_channel_switch()` when targeting a specific stream, and releases a reserved slot if URL assembly fails after `get_stream()` allocated one.
- **Live proxy init reads Redis assignment once.** After `generate_stream_url()` reserves a slot, the stream handler reads `channel_stream` / `stream_profile` from Redis instead of calling `get_stream()` again, avoiding double INCR under concurrent release.
- **Centralized Dispatcharr User-Agent construction in `core.utils`.** Outbound HTTP calls (Schedules Direct API, update checks, logo/VOD fallbacks, DVR recording clients) now use `dispatcharr_user_agent()`, `dispatcharr_dvr_user_agent()`, and `dispatcharr_http_headers()` instead of ad-hoc `Dispatcharr/{version}` strings and stale `Dispatcharr/1.0` fallbacks.
- **EPG auto-match overhaul** — matching logic moved to `apps/channels/epg_matching.py`; Celery tasks in `tasks.py` are thin wrappers.
- Single-channel auto-match is now asynchronous: the API returns `202 Accepted` and pushes the result over WebSocket (`single_channel_epg_match`), so large EPG libraries no longer hit the previous 30-second HTTP timeout.
- Progress, bulk completion, and single-channel results use `send_websocket_update` instead of `async_to_sync(channel_layer.group_send)`, so notifications work reliably under gevent-patched uWSGI and Celery workers.
- Single-channel and selected-channel auto-match always run, even when the channel already has EPG assigned; match-all (no channel IDs) still only processes channels without EPG.
- Rematching to the same EPG no longer re-saves the channel or queues program-parse tasks; only assignments that actually change are written and refreshed.
- **Easier EPG search when editing a channel.** The filter in the EPG picker now works more like a normal search box. You can type several words at once (e.g. `sky uk`) and it finds channels where every word appears somewhere in the name or TVG-ID, the order doesn't matter, and a word in the name can pair with one in the ID. Accents are ignored too, so `decale` matches `Décalé` and the other way around. — Thanks [@FiveBoroughs](https://github.com/FiveBoroughs)
### Performance
- **XC live refresh releases bulk catalog data sooner during batch processing.** After filtering the single `get_all_live_streams` response, the full provider catalog list is dropped before batch DB work. `process_m3u_batch_direct` (the path XC refreshes use) now runs `gc.collect()` after each batch and clears batch slice references as thread futures complete. Large structures are still `del`'d in the refresh `finally` block before Celery's `task_postrun` runs `cleanup_memory()`.
- **VOD movie/series batch matching no longer scans the full no-ID catalog.** `process_movie_batch` and `process_series_batch` previously loaded every `Movie`/`Series` row without TMDB or IMDB IDs on each 1000-item chunk to resolve name+year duplicates. Lookup is now scoped to the names in the current batch via `lookup_by_name_year()`, which reduces memory and DB time per chunk. `refresh_vod_content` and `batch_refresh_series_episodes` are registered for Celery post-task memory cleanup (one GC pass at task end, not per chunk).
- **EPG programme parse now streams through PostgreSQL staging instead of holding the full catalogue in Python.** `parse_programs_for_source` writes parsed rows into a session-scoped temp table in batches (`_EPG_PARSE_BATCH_SIZE=2500`), then atomically swaps them into `ProgramData` with batched `DELETE ... RETURNING` + `INSERT` (`_EPG_SWAP_BATCH_SIZE=5000`) so Postgres never materializes the entire guide in one statement. Peak Celery memory during large XMLTV refreshes drops sharply compared with building a monolithic in-memory list before bulk insert. The byte-offset programme index build is deferred until after the swap completes so index construction no longer competes with parse for memory and I/O. `refresh_epg_data` closes its DB connection in `finally`, and `build_programme_index_task` is registered as a memory-intensive Celery task. SQLite/dev installs keep an in-memory fallback path.
- **Pooled PostgreSQL connections now rotate on a bounded lifetime.** psycopg3 client-side cache grows on handles kept open indefinitely by `django-db-geventpool`; recycling uWSGI workers would interrupt live streams. A thin custom backend (`dispatcharr.db.backends.postgresql_psycopg3`) closes and replaces pool connections after `DATABASE_POOL_CONN_MAX_LIFETIME` seconds (default 600, env-overridable; set `0` to disable) on checkout and return while preserving warm-pool reuse within the window. (Fixes #1343)
- **Schedules Direct refresh only fetches guide data for mapped channels.** Schedule MD5 checks and schedule downloads now target mapped lineup stations instead of the entire lineup, and schedule MD5 cache for unmapped stations is pruned each refresh. Unmapped lineup entries no longer trigger wasted schedule API calls when their MD5 changes.
- **EPG auto-match memory and throughput improvements.**
- Single-channel matching streams active EPG rows and keeps only the best match plus the top 20 candidates in memory; ML validates at most 21 names per channel instead of embedding the full catalog.
- Strong fuzzy matches (≥75% single channel, ≥80% bulk) skip ML entirely, avoiding a ~500MB PyTorch load when the fuzzy result is already reliable.
@ -40,12 +168,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **Live proxy channels could remain running with no clients after disconnect.** `stop_channel` called `log_system_event('channel_stop')` synchronously before local cleanup and Redis key deletion; Connect integrations or DB event writes on that path could block teardown indefinitely while the cleanup thread kept refreshing metadata TTL for a stale `stream_buffers` entry. Teardown now signals shutdown (`buffer.stopping`), runs model `release_stream()` and Redis key deletion, releases ownership, stops ffmpeg/output managers and local buffers, and only then logs the stop event asynchronously. Metadata TTL refresh skips channels mid-shutdown and non-owned channels; a stuck-stop watchdog forces cleanup if `stop_channel` does not return within ~10s (or 2× `channel_shutdown_delay`, whichever is greater). Stream buffers ignore late `add_chunk()` writes once shutdown begins (`buffer.stopping` set at teardown start).
- **Client reconnect during channel teardown could wedge the channel across uWSGI workers.** The disconnect path called `proxy_server.stop_channel()` directly, so other workers were not notified and reconnecting clients could attach while ownership was released on the owner worker. The cleanup thread then re-acquired expired ownership and looped on orphaned clients while the upstream connection stayed open. All lifecycle stops now go through coordinated Redis teardown (`channel_stopping` flag, metadata `stopping` state, and `CHANNEL_STOP` pubsub). New stream requests are rejected with `503 Retry-After` only during active teardown (not during the post-disconnect shutdown delay grace period); `initialize_channel()` and ownership re-acquisition refuse to proceed in those states; non-owner workers clean local resources only on pubsub; and orphaned-client sweeps force cleanup when teardown is already active. (Fixes #1342)
- **Orphaned upstream threads kept writing Redis after teardown.** Partial cleanup could remove a channel from `stream_managers` while the `stream-{uuid}` OS thread kept running, so orphan metadata sweeps only deleted Redis keys and chunks/metadata were immediately recreated (bare `total_bytes` hash with TTL -1). Stream managers now stop upstream when ownership is lost, orphan cleanup stops local ffmpeg/stream threads even when registry entries are gone (`_live_stream_managers` tracks managers until the thread exits), and forced recovery stops processes before Redis deletion instead of dropping dict entries without calling `stop()`.
- **Rapid channel switching could leave bare `buffer:index` keys in Redis.** `buffer.stop()` flushed a final partial chunk via `INCR` during teardown, creating a persistent index key (TTL -1) after chunk keys expired; under load overlapping disconnect and cleanup-thread stops could race on `_stop_local_stream_activity`, blocking on ffmpeg stderr join before `_clean_redis_keys` ever ran. Teardown no longer writes to Redis from `buffer.stop()`, disconnect uses coordinated stop only (no duplicate upstream stop), model `release_stream()` runs before Redis keys are scanned/deleted (so `profile_connections` counters are not left stuck), ownership is released after Redis cleanup but before the blocking local ffmpeg stop, and a `finally` block guarantees Redis cleanup if local teardown raises.
- **Live proxy could leak geventpool DB checkouts outside HTTP requests.** With `django-db-geventpool`, `connection.close()` returns handles to the per-worker pool (`MAX_CONNS=8`); without it, greenlets and OS threads keep connections checked out until the pool blocks on `pool.get()`. Proxy paths that touch the ORM outside Django's request cycle now call `close_old_connections()` after work completes: `_clean_redis_keys()`, profile lookup in `_establish_transcode_connection()` before spawning ffmpeg, and fMP4 client disconnect cleanup when releasing streams (TS disconnect relies on `log_system_event()`). (Fixes #1345)
- **System events could block callers on Connect and plugin handlers.** `log_system_event()` ran Connect subscriptions and plugin `"events"` hooks synchronously after the DB write, so slow integrations could stall live-proxy and streaming paths even when inserting the event row was fast. Connect/plugin dispatch now runs on a separate gevent when the hub is available (synchronously in Celery workers without a hub). `log_system_event()` and `dispatch_event_system()` also call `close_old_connections()` in `finally` blocks so integration work does not leave pool slots checked out on the caller or dispatch greenlet.
- **Plugins could leak geventpool DB checkouts after UI or Connect event runs.** Third-party plugins run inline on uWSGI greenlets (manual actions and Connect `"events"` hooks) with no guaranteed connection cleanup at the plugin boundary. `PluginManager.run_action()` and `stop_plugin()` now call `close_old_connections()` in a `finally` block so each action returns its pool slot whether the plugin succeeds or raises.
- **Connect events repeatedly queried the full plugin catalog during streaming.** `trigger_event()` called `list_plugins()` on every `client_connect` / `client_disconnect`, loading all `PluginConfig` rows and sometimes hitting plugin-repo work even when no plugin subscribed to the event. Dispatch now walks the in-memory registry via `iter_actions_for_event()`, returns immediately when no handlers exist, and runs a single batched `enabled` lookup for matching plugins only. Failed plugin actions are logged per handler instead of aborting the rest.
- **Plugin discovery left idle Postgres backends after worker boot.** `discover_plugins()` runs outside Django's request cycle during uWSGI and Celery startup; it now calls `close_old_connections()` in a `finally` block so bootstrap `PluginConfig` queries return their pool slot instead of appearing as long-lived idle connections in `pg_stat_activity`.
- **VOD proxy could leak geventpool DB checkouts during playback and stats updates.** `stream_vod()` ran ORM lookups for content and M3U profiles, then returned a long-lived `StreamingHttpResponse` without releasing the checkout, so each movie/episode stream could hold a pool slot for its full duration. Background VOD stats refresh (`build_vod_stats_data()`, triggered on start/stop and by the admin stats API) also queried movie, episode, and profile rows from daemon threads with no cleanup. `stream_vod()` now calls `close_old_connections()` before handing off to the streaming generator, and `build_vod_stats_data()` releases its checkout in a `finally` block.
- **Channel shutdown delay did not reset after a reconnect within the grace period.** `handle_client_disconnect()` used a fixed `gevent.sleep(shutdown_delay)` from the first last-client disconnect. If a client reconnected and disconnected again during the delay, an earlier disconnect handler could still stop the channel on the original timer instead of waiting the full delay from the latest disconnect. Shutdown now polls Redis (`last_client_disconnect` timestamp and client count) so concurrent disconnect handlers and multi-worker reconnects always honour the latest disconnect time.
- **Zombie ffmpeg could survive owner-lock expiry and orphan Redis sweeps.** When the 30s owner lock lapsed under single-worker load, disconnect handling treated the worker as non-owner so coordinated stop never ran, while ffmpeg kept writing and the orphan sweeper only deleted Redis keys (recreating them immediately). Disconnect now re-acquires ownership when local upstream is still active, stops locally when the last client leaves without a lock, orphan cleanup stops local ffmpeg before Redis deletion via `_has_local_upstream_activity`, re-init stops lingering upstream before starting a duplicate thread, and `is_channel_teardown_active` includes channels mid-`stop_channel` on this worker so rapid reconnect gets 503 during teardown.
- **Stale `channel_stream` Redis keys after a channel stopped could skip connection accounting on retune.** On `dev`, `get_stream()` reused any existing `channel_stream` / `stream_profile` assignment without reserving a new slot. If those keys were left behind after a stop, the next tune-in could reach the provider without incrementing Redis counters. `get_stream()` now releases stale assignments when proxy metadata shows the channel is inactive, then reserves fresh slots.
- **EPG auto-match reliability fixes.**
- Memory could spike to multiple GB on large EPG sources when building a full in-memory catalog before fuzzy matching; single-channel matching now streams rows and bounds ML work to a small candidate set.
- Wrong channel assignments from global ML similarity; ML validation now checks the fuzzy best match (or top fuzzy candidates as a last resort) instead of scoring the entire catalog.
- Channel form auto-match spinner could stick after errors or early task exits; all single-channel outcomes now push a WebSocket result, and the UI clears loading state after a 3-minute timeout.
- Bulk auto-match completion no longer calls `batch-set-epg` from the WebSocket handler, which had been re-applying every match and queueing redundant `parse_programs_for_tvg_id` tasks even when assignments were unchanged.
- **Schedules Direct lineup search country dropdown.** The country list was fetched directly from the SD API in the browser, which failed due to CORS and silently fell back to a 14-country hardcoded list. Countries are now included in the `GET sd-lineups` response (server-side fetch with proper User-Agent) so the full SD country list populates correctly. — Thanks [@sethwv](https://github.com/sethwv)
- **Schedules Direct program poster proxy omitted User-Agent on image requests.** The poster endpoint authenticated with SD correctly but fetched images with only the `token` header, which violates SD's API requirements (error 1003). Image requests now include the standard `Dispatcharr/{version}` User-Agent via `dispatcharr_http_headers()`.
- **Schedules Direct guide data missing after mapping a channel.** Lineup refreshes cached schedule MD5s for all stations, but `ProgramData` was only written for mapped channels. Mapping a channel later could leave MD5s looking unchanged so schedules were skipped and the channel had no guide until dates rolled outside the cached window. Refreshes now backfill fetch-window dates that lack `ProgramData` (including newly mapped stations with stale cache), fetch program metadata when no local `ProgramData` exists for a `programID`, and clean up orphaned `ProgramData` on unmapped `EPGData` entries. — Thanks [@sethwv](https://github.com/sethwv)
- **Schedules Direct guide fetch on channel map.** Mapping a channel (or bulk-assigning EPG) now triggers a targeted guide fetch for that `EPGData` entry—mirroring XMLTV's `parse_programs_for_tvg_id` flow—without waiting for the next full source refresh. Skips the fetch when `ProgramData` already exists so additional channels sharing the same `tvg-id` do not trigger redundant API calls. Bulk assignment of three or more SD stations without guide data on the same source queues one batched mapped-station fetch instead of separate per-station API sessions. Concurrent batch and single-EPG fetches coordinate via source-level locks with deferred retries so mappings are not dropped and overlapping SD API sessions are avoided when possible.
- **Auto-sync numbering modes now read only the fields each mode's UI exposes.** After the auto-sync overhaul, switching between Provider, Next Available, and Fixed modes left stale `auto_sync_channel_start` / `auto_sync_channel_end` values in the database while each mode's UI only edits a subset of those fields. Sync treated the hidden values as authoritative in every mode, which discarded valid provider numbers (floored at an auto-computed start), capped Next Available at a stale End, and ran range-enforcement deletes against provider- and next-available-numbered channels. Provider mode now honors `stream_chno` verbatim when free (Start/End bound only the fallback for numberless streams); Next Available ignores End; overflow-delete runs in Fixed mode only. Duplicate or already-taken provider numbers fall back to a free slot instead of being dropped or overwriting an existing channel. Provider mode's Start # field now drops End when it would invert the fallback range (matching Fixed mode). (Closes #1273) — Thanks [@CodeBormen](https://github.com/CodeBormen)
## [0.26.0] - 2026-06-07

14
CONTRIBUTING.md
View file

@ -144,7 +144,19 @@ Untested code is significantly less likely to be merged.
- Use Django's `TestCase` for unit/integration tests.
- Test files live at `apps/<app>/tests/`.
- Run the test suite with: `uv run python manage.py test`
- Run the backend test suite with:
```bash
python manage.py test
```
`manage.py` automatically uses `dispatcharr.settings_test`, which creates an empty PostgreSQL database `test_<dbname>` (same engine as production), runs migrations, and rolls back each test in a transaction. Your live VOD/channels data is not used.
Optional: `TEST_USE_SQLITE=1` for machines without Postgres (some PostgreSQL-only tests skip automatically).
Tests that exercise Celery task bodies should use `@override_settings(CELERY_TASK_ALWAYS_EAGER=True)` locally. Global eager mode is off because `post_save` signals on M3U/EPG models call `.delay()` and would break `TestCase` transaction isolation.
- Do **not** override with `--settings=dispatcharr.settings` on a live instance.
### Frontend

70
Plugin_repo.md
View file

@ -120,12 +120,14 @@ If the name contains any of these, the repo will be rejected on add and skipped
### Top-Level Metadata
| Field | Required | Description |
| --------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `registry_name` | **Yes** | Display name for the repo. Must not contain words like "official" or "dispatcharr" that could be mistaken for an official repo (see [Name Restrictions](#name-restrictions)). |
| `registry_url` | No | URL to the repo's home page (e.g. GitHub). Used as a fallback for generating icon URLs. |
| `root_url` | No | Base URL for resolving relative URLs in plugin entries. Trailing slashes are stripped. |
| `plugins` | **Yes** | Array of plugin entry objects. |
| Field | Required | Description |
| -------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `registry_name` | **Yes** | Display name for the repo. Must not contain words like "official" or "dispatcharr" that could be mistaken for an official repo (see [Name Restrictions](#name-restrictions)). |
| `registry_url` | No | URL to the repo's home page (e.g. GitHub). Used as a fallback for generating icon URLs. |
| `root_url` | No | Generic base URL for resolving all relative URLs in plugin entries. Trailing slashes are stripped. Used as the fallback when neither `download_base_url` nor `metadata_base_url` is set. |
| `download_base_url` | No | Base URL for resolving relative download URLs (`latest_url` in plugin entries; `url` and `latest_url` inside per-plugin manifest `versions`/`latest`). Overrides `root_url` for download assets when set. |
| `metadata_base_url` | No | Base URL for resolving relative metadata URLs (`manifest_url` and `icon_url` in plugin entries). Overrides `root_url` for metadata assets when set. |
| `plugins` | **Yes** | Array of plugin entry objects. |
### Plugin Entry Fields
@ -155,13 +157,18 @@ Extra fields in a plugin entry are passed through to the frontend as-is, so you
### URL Resolution
If `root_url` is set and a URL field (`manifest_url`, `latest_url`, `icon_url`) does not start with `http://` or `https://`, it is treated as relative and resolved as:
Relative URL fields are resolved against a base URL. Dispatcharr uses two separate base URLs (one for metadata assets and one for download assets) so you can serve them from different origins (e.g., manifests and icons on GitHub Pages, release zips on a CDN).
```
{root_url}/{field_value}
```
**Resolution priority:**
This lets you keep plugin entries compact:
| Field(s) | Priority |
| --------- | -------- |
| `manifest_url`, `icon_url` | `metadata_base_url` → `root_url` |
| `latest_url` (plugin entries); `url`, `latest_url` (per-plugin manifest versions/latest) | `download_base_url` → `root_url` |
A field value is treated as relative if it does not start with `http://` or `https://`. Relative values are resolved as `{base_url}/{field_value}`. All base URL fields are optional; if none are set, URL fields must be absolute.
**Single base URL (simplest):** use `root_url` for everything:
```json
{
@ -177,12 +184,45 @@ This lets you keep plugin entries compact:
}
```
**Icon fallback:** If `icon_url` is missing and `registry_url` is set, Dispatcharr generates a fallback URL by converting the GitHub URL to a raw content URL:
**Split base URLs:** use `metadata_base_url` and `download_base_url` when assets are served from different origins:
```json
{
"metadata_base_url": "https://raw.githubusercontent.com/myorg/my-plugins/main",
"download_base_url": "https://cdn.example.com/releases",
"plugins": [
{
"slug": "my_plugin",
"manifest_url": "plugins/my_plugin/manifest.json",
"icon_url": "plugins/my_plugin/logo.png",
"latest_url": "my_plugin/my_plugin-1.0.0.zip"
}
]
}
```
{registry_url => raw.githubusercontent.com}/refs/heads/main/plugins/{slug}/logo.png
You can also combine `root_url` with one specific field. The specific field overrides for its consumers, and `root_url` covers the rest:
```json
{
"root_url": "https://raw.githubusercontent.com/myorg/my-plugins/main",
"download_base_url": "https://cdn.example.com/releases"
}
```
**Icon fallback:** If `icon_url` is missing, Dispatcharr tries two fallbacks in order:
1. **Manifest-directory fallback**: if a base URL is set (`root_url`, `metadata_base_url`, etc.) and `manifest_url` is present, the logo is assumed to live in the same directory as the per-plugin manifest:
```
{directory of resolved manifest_url}/logo.png
```
For example, if `manifest_url` resolves to `https://example.com/plugins/my_plugin/manifest.json`, the fallback icon URL is `https://example.com/plugins/my_plugin/logo.png`.
2. **GitHub fallback**: if `registry_url` is a GitHub URL, Dispatcharr converts it to a raw content URL:
```
{registry_url => raw.githubusercontent.com}/refs/heads/main/plugins/{slug}/logo.png
```
---
## Per-Plugin Manifest (Optional)
@ -563,7 +603,9 @@ You can host release zips as GitHub Release assets and reference them with absol
"manifest": {
"registry_name": "string (required)",
"registry_url": "string (optional)",
"root_url": "string (optional)",
"root_url": "string (optional, generic base URL fallback)",
"download_base_url": "string (optional, overrides root_url for zip download URLs)",
"metadata_base_url": "string (optional, overrides root_url for manifest_url and icon_url)",
"plugins": [
{
"slug": "string (required)",

12
Plugins.md
View file

@ -307,6 +307,18 @@ Plugins are server-side Python code running within the Django application. You c
Prefer Celery tasks (`.delay()`) to keep `run` fast and non-blocking.
### Database connections
Dispatcharr uses `django-db-geventpool` with a bounded per-uWSGI-worker pool (`MAX_CONNS=8`). Each greenlet or OS thread that runs ORM code checks out a connection until Django closes it.
`PluginManager.run_action()` and `stop_plugin()` always call `close_old_connections()` in a `finally` block after your plugin returns (success or error). That returns the current greenlet's checkout to the pool. **You do not need to call `close_old_connections()` yourself for normal inline ORM inside `run()` or `stop()`.**
Still follow these rules:
- **Heavy or long work:** dispatch a Celery task (`.delay()`) and return quickly from `run()`. Celery workers close connections after each task; blocking the uWSGI gevent hub with `time.sleep`, sync HTTP, or large CPU work can freeze the whole worker regardless of DB cleanup.
- **Background threads or greenlets you spawn:** each thread/greenlet that uses the ORM must call `close_old_connections()` (or `connection.close()`) in its own `finally` block when done. The wrapper only covers the thread/greenlet that called `run_action()`.
- **Connect event hooks:** actions with an `"events"` list are dispatched from `log_system_event()` on a separate gevent when uWSGI has an active hub (otherwise synchronously, e.g. Celery). Keep handlers short or defer heavy work to Celery.
### Important: Don’t Ask Users for URL/User/Password
Dispatcharr plugins run **inside** the Dispatcharr backend process. That means they already have direct access to the app’s models, tasks, and internal utilities.
Plugins **should not** ask users for “Dispatcharr URL”, “Admin Username”, or “Admin Password” just to call the API. That is unnecessary and unsafe because:

3
apps/accounts/authentication.py
View file

@ -93,7 +93,8 @@ class QueryParamJWTAuthentication(JWTAuthentication):
where the browser cannot set Authorization headers (e.g. <video src>)."""
def authenticate(self, request):
raw_token = request.GET.get('token')
params = getattr(request, "query_params", request.GET)
raw_token = params.get("token")
if not raw_token:
return None
try:

6
apps/channels/api_urls.py
View file

@ -13,6 +13,7 @@ from .api_views import (
UpdateChannelMembershipAPIView,
BulkUpdateChannelMembershipAPIView,
RecordingViewSet,
RECORDING_PLAYBACK_AUTHENTICATORS,
RecurringRecordingRuleViewSet,
GetChannelStreamsAPIView,
GetChannelStreamStatsAPIView,
@ -53,7 +54,10 @@ urlpatterns = [
path('recordings/bulk-delete-upcoming/', BulkDeleteUpcomingRecordingsAPIView.as_view(), name='bulk_delete_upcoming_recordings'),
path(
'recordings/<int:pk>/hls/<path:seg_path>',
RecordingViewSet.as_view({'get': 'hls'}),
RecordingViewSet.as_view(
{'get': 'hls'},
authentication_classes=RECORDING_PLAYBACK_AUTHENTICATORS,
),
name='recording-hls',
),
path('dvr/comskip-config/', ComskipConfigAPIView.as_view(), name='comskip_config'),

117
apps/channels/api_views.py
View file

@ -3,6 +3,8 @@ from rest_framework.response import Response
from rest_framework.views import APIView
from rest_framework.permissions import AllowAny
from rest_framework.decorators import action
from rest_framework_simplejwt.authentication import JWTAuthentication
from apps.accounts.authentication import ApiKeyAuthentication, QueryParamJWTAuthentication
from rest_framework.parsers import MultiPartParser, FormParser, JSONParser
from drf_spectacular.utils import extend_schema, OpenApiParameter, inline_serializer
from drf_spectacular.types import OpenApiTypes
@ -12,6 +14,7 @@ from django.db import connection, transaction
from django.db.models import Count, F, Prefetch
from django.db.models import Q
import os, json, requests, logging, mimetypes, threading, time
from urllib.parse import urlencode
from datetime import timedelta
from django.utils.http import http_date
from apps.accounts.permissions import (
@ -24,6 +27,7 @@ from apps.accounts.permissions import (
from core.models import UserAgent, CoreSettings
from core.utils import RedisClient, safe_upload_path
from apps.m3u.utils import convert_js_numbered_backreferences
from .models import (
Stream,
@ -365,6 +369,17 @@ class StreamViewSet(viewsets.ModelViewSet):
except re.error as e:
exclude_error = str(e)
# The replace field accepts JS-style $1 backreferences, but the regex
# engine honors \1. Convert once so the preview's "after" matches the
# name the live rename produces (apps/m3u/tasks.py sync_auto_channels
# applies the same conversion on the same engine).
replace_repl = convert_js_numbered_backreferences(replace_pat)
# The live rename caps the result at Channel.name's column length
# before bulk_create; mirror that cap so the preview never shows a
# name the sync would truncate.
name_max_len = Channel._meta.get_field("name").max_length
# Capped at SCAN_CAP to bound memory on huge groups; the
# separate COUNT lets the client surface scan_limit_hit when
# the preview covers only a sample.
@ -387,11 +402,12 @@ class StreamViewSet(viewsets.ModelViewSet):
total_scanned += 1
if find_re is not None:
try:
new_name = find_re.sub(replace_pat, name, timeout=REGEX_TIMEOUT)
new_name = find_re.sub(replace_repl, name, timeout=REGEX_TIMEOUT)
except (TimeoutError, re.error) as e:
find_error = find_error or f"Pattern timed out: {e}"
find_re = None
continue
new_name = new_name[:name_max_len]
if new_name != name:
find_match_count += 1
if len(find_matches) < limit:
@ -2163,22 +2179,14 @@ class ChannelViewSet(viewsets.ModelViewSet):
epg_data = EPGData.objects.get(pk=epg_data_id)
# Set the EPG data and save
# Set the EPG data and save. refresh_epg_programs (post_save) queues
# parse_programs_for_tvg_id for non-dummy sources — no second dispatch here.
channel.epg_data = epg_data
channel.save(update_fields=["epg_data"])
# Only trigger program refresh for non-dummy EPG sources
status_message = None
if epg_data.epg_source.source_type != 'dummy':
# Explicitly trigger program refresh for this EPG
from apps.epg.tasks import parse_programs_for_tvg_id
task_result = parse_programs_for_tvg_id.delay(epg_data.id)
# Prepare response with task status info
status_message = "EPG refresh queued"
if task_result.result == "Task already running":
status_message = "EPG refresh already in progress"
# Build response message
message = f"EPG data set to {epg_data.tvg_id} for channel {channel.name}"
@ -2356,27 +2364,9 @@ class ChannelViewSet(viewsets.ModelViewSet):
channels_updated = len(channels_to_update)
# Trigger program refresh only for EPG ids newly assigned (skip dummy/SD)
from apps.epg.tasks import parse_programs_for_tvg_id
from apps.epg.models import EPGData
from apps.epg.tasks import dispatch_program_refresh_for_epg_ids
# Batch fetch EPG data (single query)
epg_data_dict = {
epg.id: epg
for epg in EPGData.objects.filter(id__in=changed_epg_ids).select_related('epg_source')
}
programs_refreshed = 0
for epg_id in changed_epg_ids:
epg_data = epg_data_dict.get(epg_id)
if not epg_data:
logger.error(f"EPGData with ID {epg_id} not found")
continue
source_type = epg_data.epg_source.source_type if epg_data.epg_source else None
if source_type not in ('dummy', 'schedules_direct'):
parse_programs_for_tvg_id.delay(epg_id)
programs_refreshed += 1
programs_refreshed = dispatch_program_refresh_for_epg_ids(changed_epg_ids)
return Response(
{
@ -2800,8 +2790,9 @@ class LogoViewSet(viewsets.ModelViewSet):
user_agent_obj = UserAgent.objects.get(id=int(default_user_agent_id))
user_agent = user_agent_obj.user_agent
except (CoreSettings.DoesNotExist, UserAgent.DoesNotExist, ValueError):
# Fallback to hardcoded if default not found
user_agent = 'Dispatcharr/1.0'
# Fallback if default not found
from core.utils import dispatcharr_user_agent
user_agent = dispatcharr_user_agent()
# Hard total timeout (connect + full download) prevents a slow
# server dripping bytes from holding a greenlet indefinitely.
@ -3261,12 +3252,41 @@ def _stop_dvr_clients(channel_uuid, recording_id=None):
return stopped
# QueryParamJWTAuthentication supports native <video src> clients that cannot
# send Authorization headers. Authorization still requires an authenticated
# user via _user_can_play_recording; these classes only populate request.user.
RECORDING_PLAYBACK_AUTHENTICATORS = [
JWTAuthentication,
ApiKeyAuthentication,
QueryParamJWTAuthentication,
]
def _recording_auth_query_suffix(request):
"""Suffix for rewritten recording URLs when auth used ?token= (native <video>).
hls.js clients authenticate via Authorization on each XHR and do not need
tokens embedded in playlist segment lines.
"""
from rest_framework.request import Request as DRFRequest
if isinstance(request, DRFRequest):
params = request.query_params
else:
params = request.GET
token = params.get("token")
if not token:
return ""
return "?" + urlencode({"token": token})
class RecordingViewSet(viewsets.ModelViewSet):
queryset = Recording.objects.all()
serializer_class = RecordingSerializer
def get_permissions(self):
# Allow unauthenticated playback of recording files (like other streaming endpoints)
# file/hls use AllowAny so DRF does not reject requests before auth
# classes run; _user_can_play_recording enforces authenticated access.
if self.action in ('file', 'hls'):
return [AllowAny()]
try:
@ -3330,7 +3350,12 @@ class RecordingViewSet(viewsets.ModelViewSet):
except Exception as e:
return Response({"success": False, "error": str(e)}, status=400)
@action(detail=True, methods=["get"], url_path="file")
@action(
detail=True,
methods=["get"],
url_path="file",
authentication_classes=RECORDING_PLAYBACK_AUTHENTICATORS,
)
def file(self, request, pk=None):
"""Stream a completed recording file with HTTP Range support for seeking.
@ -3354,7 +3379,7 @@ class RecordingViewSet(viewsets.ModelViewSet):
if hls_dir and os.path.isdir(hls_dir):
hls_url = request.build_absolute_uri(
f"/api/channels/recordings/{pk}/hls/index.m3u8"
)
) + _recording_auth_query_suffix(request)
return HttpResponseRedirect(hls_url)
if not file_path or not os.path.exists(file_path):
raise Http404("Recording file not found")
@ -3418,7 +3443,12 @@ class RecordingViewSet(viewsets.ModelViewSet):
response["Content-Disposition"] = f"inline; filename=\"{file_name}\""
return response
@action(detail=True, methods=["get"], url_path="hls/(?P<seg_path>.+)")
@action(
detail=True,
methods=["get"],
url_path="hls/(?P<seg_path>.+)",
authentication_classes=RECORDING_PLAYBACK_AUTHENTICATORS,
)
def hls(self, request, pk=None, seg_path=None):
"""Serve HLS playlist and segment files for an in-progress (or completed) recording.
@ -3442,9 +3472,10 @@ class RecordingViewSet(viewsets.ModelViewSet):
cp = recording.custom_properties or {}
file_path = cp.get("file_path")
if seg_path.endswith(".m3u8") and file_path and os.path.exists(file_path) and os.path.getsize(file_path) > 0:
return HttpResponseRedirect(
request.build_absolute_uri(f"/api/channels/recordings/{pk}/file/")
)
file_url = request.build_absolute_uri(
f"/api/channels/recordings/{pk}/file/"
) + _recording_auth_query_suffix(request)
return HttpResponseRedirect(file_url)
raise Http404("HLS content not available for this recording")
# Security: prevent path traversal outside the HLS directory
@ -3457,16 +3488,18 @@ class RecordingViewSet(viewsets.ModelViewSet):
raise Http404(f"HLS file not found: {seg_path}")
if seg_path.endswith(".m3u8"):
# Rewrite relative segment lines to absolute URLs through this API
# Rewrite relative segment lines to absolute URLs through this API.
# Propagate ?token= only for native <video> clients (see helper).
base_url = request.build_absolute_uri(
f"/api/channels/recordings/{pk}/hls/"
)
auth_suffix = _recording_auth_query_suffix(request)
lines = []
with open(requested) as _f:
for line in _f:
stripped = line.strip()
if stripped and not stripped.startswith("#"):
lines.append(f"{base_url}{stripped}\n")
lines.append(f"{base_url}{stripped}{auth_suffix}\n")
else:
lines.append(line)
return HttpResponse("".join(lines), content_type="application/x-mpegURL")

11
apps/channels/compact_numbering.py
View file

@ -25,7 +25,8 @@ import logging
from django.db import transaction
from .models import Channel, ChannelOverride, ChannelGroupM3UAccount
from apps.m3u.tasks import _custom_properties_as_dict, _next_available_number
from apps.m3u.tasks import _next_available_number
from core.utils import ensure_custom_properties_dict
from core.utils import (
acquire_task_lock,
natural_sort_key,
@ -37,7 +38,7 @@ logger = logging.getLogger(__name__)
def is_compact_group(group_relation):
"""Return True if the given ChannelGroupM3UAccount is in compact mode."""
cp = _custom_properties_as_dict(group_relation.custom_properties)
cp = ensure_custom_properties_dict(group_relation.custom_properties)
return bool(cp.get("compact_numbering"))
@ -72,7 +73,7 @@ def get_group_relation_for_channel(channel):
for rel in ChannelGroupM3UAccount.objects.filter(
m3u_account_id=channel.auto_created_by_id
):
cp = _custom_properties_as_dict(rel.custom_properties)
cp = ensure_custom_properties_dict(rel.custom_properties)
if str(cp.get("group_override", "")) == target:
return rel
return None
@ -179,7 +180,7 @@ def assign_compact_numbers_for_channels(channel_ids):
for rel in ChannelGroupM3UAccount.objects.filter(
m3u_account_id__in={k[0] for k in unresolved}
):
cp = _custom_properties_as_dict(rel.custom_properties)
cp = ensure_custom_properties_dict(rel.custom_properties)
target = cp.get("group_override")
if not target:
continue
@ -295,7 +296,7 @@ def _repack_inner(group_relation):
account_id = group_relation.m3u_account_id
group_id = group_relation.channel_group_id
cp = _custom_properties_as_dict(group_relation.custom_properties)
cp = ensure_custom_properties_dict(group_relation.custom_properties)
sort_order = cp.get("channel_sort_order") or ""
sort_reverse = bool(cp.get("channel_sort_reverse"))

16
apps/channels/epg_matching.py
View file

@ -306,7 +306,7 @@ def build_epg_tvg_id_index(epg_data):
def _dispatch_program_parse_for_epg_assignments(changed_associations):
"""
Queue parse_programs once per unique EPG id newly assigned to a channel.
Queue guide/program refresh for newly assigned EPG ids.
bulk_update bypasses post_save, so callers must invoke this when epg_data
actually changes (mirrors the M3U sync path).
@ -314,24 +314,14 @@ def _dispatch_program_parse_for_epg_assignments(changed_associations):
if not changed_associations:
return 0
from apps.epg.tasks import parse_programs_for_tvg_id
from apps.epg.tasks import dispatch_program_refresh_for_epg_ids
epg_ids = {
assoc["epg_data_id"]
for assoc in changed_associations
if assoc.get("epg_data_id")
}
if not epg_ids:
return 0
dispatched = 0
for epg in EPGData.objects.filter(id__in=epg_ids).select_related("epg_source"):
source_type = epg.epg_source.source_type if epg.epg_source else None
if source_type in ("dummy", "schedules_direct"):
continue
parse_programs_for_tvg_id.delay(epg.id)
dispatched += 1
return dispatched
return dispatch_program_refresh_for_epg_ids(epg_ids)
def _log_unchanged_epg_assignment(chan, epg_id, epg_name, epg_tvg_id, match_method):

101
apps/channels/migrations/0038_add_catchup_fields.py Normal file
View file

@ -0,0 +1,101 @@
"""Add denormalized catch-up fields to Stream and Channel."""
from django.db import migrations, models
def backfill_stream_catchup(apps, schema_editor):
"""Derive is_catchup/catchup_days from Stream.custom_properties JSON."""
with schema_editor.connection.cursor() as cursor:
cursor.execute("""
UPDATE dispatcharr_channels_stream
SET is_catchup = TRUE,
catchup_days = COALESCE(
CASE WHEN (custom_properties->>'tv_archive_duration') ~ '^\\d+$'
THEN (custom_properties->>'tv_archive_duration')::int
ELSE NULL
END, 7
)
WHERE custom_properties IS NOT NULL
AND custom_properties != 'null'::jsonb
AND (
custom_properties->>'tv_archive' = '1'
-- JSON booleans extract as lowercase 'true' via ->>; the
-- 'True' spelling covers Python-str values stored by
-- older import code.
OR custom_properties->>'tv_archive' = 'true'
OR custom_properties->>'tv_archive' = 'True'
)
""")
def backfill_channel_catchup(apps, schema_editor):
"""Roll up catch-up fields from streams to channels."""
with schema_editor.connection.cursor() as cursor:
cursor.execute("""
UPDATE dispatcharr_channels_channel c SET
is_catchup = EXISTS (
SELECT 1 FROM dispatcharr_channels_channelstream cs
JOIN dispatcharr_channels_stream s ON s.id = cs.stream_id
WHERE cs.channel_id = c.id AND s.is_catchup = TRUE
),
catchup_days = COALESCE((
SELECT MAX(s.catchup_days) FROM dispatcharr_channels_channelstream cs
JOIN dispatcharr_channels_stream s ON s.id = cs.stream_id
WHERE cs.channel_id = c.id AND s.is_catchup = TRUE
), 0)
""")
class Migration(migrations.Migration):
dependencies = [
("dispatcharr_channels", "0037_auto_sync_overhaul"),
]
operations = [
# Stream fields
migrations.AddField(
model_name="stream",
name="is_catchup",
field=models.BooleanField(
default=False,
db_index=True,
help_text="Whether this stream supports catch-up/timeshift (tv_archive=1)",
),
),
migrations.AddField(
model_name="stream",
name="catchup_days",
field=models.PositiveIntegerField(
default=0,
help_text="Number of days of catch-up archive available (tv_archive_duration)",
),
),
# Channel fields
migrations.AddField(
model_name="channel",
name="is_catchup",
field=models.BooleanField(
default=False,
db_index=True,
help_text="Whether any stream on this channel supports catch-up (tv_archive=1)",
),
),
migrations.AddField(
model_name="channel",
name="catchup_days",
field=models.PositiveIntegerField(
default=0,
help_text="Max catch-up archive days across all streams on this channel",
),
),
# Backfill existing data
migrations.RunPython(
backfill_stream_catchup,
reverse_code=migrations.RunPython.noop,
),
migrations.RunPython(
backfill_channel_catchup,
reverse_code=migrations.RunPython.noop,
),
]

59
apps/channels/models.py
View file

@ -2,7 +2,7 @@ from django.db import models
from django.core.exceptions import ValidationError
from django.conf import settings
from core.models import StreamProfile, CoreSettings
from core.utils import RedisClient
from core.utils import RedisClient, custom_properties_as_dict
from apps.proxy.live_proxy.redis_keys import RedisKeys
from apps.proxy.live_proxy.constants import ChannelMetadataField, ChannelState
import logging
@ -135,6 +135,17 @@ class Stream(models.Model):
db_index=True
)
# Populated at import from tv_archive / tv_archive_duration.
is_catchup = models.BooleanField(
default=False,
db_index=True,
help_text="Whether this stream supports catch-up/timeshift (tv_archive=1)",
)
catchup_days = models.PositiveIntegerField(
default=0,
help_text="Number of days of catch-up archive available (tv_archive_duration)",
)
class Meta:
# If you use m3u_account, you might do unique_together = ('name','url','m3u_account')
verbose_name = "Stream"
@ -364,6 +375,17 @@ class Channel(models.Model):
help_text="The M3U account that auto-created this channel"
)
# Populated at import; rolled up via ChannelStream signal / m3u refresh.
is_catchup = models.BooleanField(
default=False,
db_index=True,
help_text="Whether any stream on this channel supports catch-up (tv_archive=1)",
)
catchup_days = models.PositiveIntegerField(
default=0,
help_text="Max catch-up archive days across all streams on this channel",
)
# Hidden channels are excluded from HDHR, M3U, EPG, and XC output queries.
# Auto-sync still recognizes them so they are not recreated when their
# underlying provider stream persists; this is an output-layer concern, not
@ -626,11 +648,11 @@ class Channel(models.Model):
def _release_stale_stream_assignment(self, redis_client, stream_id: int) -> None:
"""Release pool counters and remove stale channel/stream assignment keys."""
profile_id = None
profile_id_bytes = redis_client.get(f"stream_profile:{stream_id}")
if profile_id_bytes:
try:
profile_id = int(profile_id_bytes)
release_profile_slot(profile_id, redis_client)
except (ValueError, TypeError):
logger.debug(
"Invalid profile ID for stale assignment on stream %s: %s",
@ -638,6 +660,32 @@ class Channel(models.Model):
profile_id_bytes,
)
if profile_id is None:
metadata_key = RedisKeys.channel_metadata(str(self.uuid))
meta_profile_id = redis_client.hget(
metadata_key, ChannelMetadataField.M3U_PROFILE
)
if meta_profile_id:
try:
profile_id = int(meta_profile_id)
except (ValueError, TypeError):
logger.debug(
"Invalid profile ID in metadata for stale assignment on "
"channel %s: %s",
self.uuid,
meta_profile_id,
)
if profile_id is not None:
release_profile_slot(profile_id, redis_client)
else:
logger.warning(
"Channel %s: releasing stale stream %s assignment without profile "
"info - profile_connections may leak",
self.uuid,
stream_id,
)
redis_client.delete(f"channel_stream:{self.id}")
redis_client.delete(f"stream_profile:{stream_id}")
@ -1093,6 +1141,13 @@ class ChannelGroupM3UAccount(models.Model):
def __str__(self):
return f"{self.channel_group.name} - {self.m3u_account.name} (Enabled: {self.enabled})"
def save(self, *args, **kwargs):
if self.custom_properties is not None and not isinstance(
self.custom_properties, dict
):
self.custom_properties = custom_properties_as_dict(self.custom_properties)
super().save(*args, **kwargs)
class Logo(models.Model):
name = models.CharField(max_length=255)

23
apps/channels/serializers.py
View file

@ -221,17 +221,6 @@ class ChannelGroupM3UAccountSerializer(serializers.ModelSerializer):
return data
def to_internal_value(self, data):
# Accept both dict and JSON string for custom_properties (for backward compatibility)
val = data.get("custom_properties")
if isinstance(val, str):
try:
data["custom_properties"] = json.loads(val)
except Exception:
pass
return super().to_internal_value(data)
def validate(self, attrs):
# Partial PATCHes only carry submitted fields; fill missing
# start/end from the instance so the validator catches a PATCH
@ -459,6 +448,7 @@ class ChannelSerializer(serializers.ModelSerializer):
"logo_id",
"user_level",
"is_adult",
"is_catchup",
"hidden_from_output",
"auto_created",
"auto_created_by",
@ -562,10 +552,13 @@ class ChannelSerializer(serializers.ModelSerializer):
return LogoSerializer(obj.logo).data
def get_streams(self, obj):
"""Retrieve ordered stream IDs for GET requests."""
return StreamSerializer(
obj.streams.all().order_by("channelstream__order"), many=True
).data
"""Retrieve ordered streams for GET requests using prefetched channelstream_set."""
ordered_streams = [
cs.stream
for cs in obj.channelstream_set.all()
if cs.stream_id is not None
]
return StreamSerializer(ordered_streams, many=True).data
def create(self, validated_data):
streams = validated_data.pop("streams", [])

28
apps/channels/signals.py
View file

@ -5,7 +5,7 @@ from django.dispatch import receiver
from django.utils.timezone import now, is_aware, make_aware
from celery.result import AsyncResult
from django_celery_beat.models import ClockedSchedule, PeriodicTask
from .models import Channel, Stream, ChannelProfile, ChannelProfileMembership, Recording
from .models import Channel, Stream, ChannelStream, ChannelProfile, ChannelProfileMembership, Recording
from apps.m3u.models import M3UAccount
from apps.epg.tasks import parse_programs_for_tvg_id
import json
@ -201,18 +201,13 @@ def refresh_epg_programs(sender, instance, created, **kwargs):
if not created and kwargs.get('update_fields') and 'epg_data' in kwargs['update_fields']:
logger.info(f"Channel {instance.id} ({instance.name}) EPG data updated, refreshing program data")
if instance.epg_data:
# Skip XMLTV program parser for SD sources — program data is handled
# by fetch_schedules_direct() directly.
if instance.epg_data.epg_source and instance.epg_data.epg_source.source_type == 'schedules_direct':
logger.info(f"Skipping XMLTV parse for SD source {instance.epg_data.tvg_id}")
if instance.epg_data.epg_source and instance.epg_data.epg_source.source_type == 'dummy':
return
logger.info(f"Triggering EPG program refresh for {instance.epg_data.tvg_id}")
parse_programs_for_tvg_id.delay(instance.epg_data.id)
# For new channels with EPG data, also refresh
elif created and instance.epg_data:
# Skip XMLTV program parser for SD sources
if instance.epg_data.epg_source and instance.epg_data.epg_source.source_type == 'schedules_direct':
logger.info(f"Skipping XMLTV parse for SD source {instance.epg_data.tvg_id}")
if instance.epg_data.epg_source and instance.epg_data.epg_source.source_type == 'dummy':
return
logger.info(f"New channel {instance.id} ({instance.name}) created with EPG data, refreshing program data")
parse_programs_for_tvg_id.delay(instance.epg_data.id)
@ -374,3 +369,20 @@ def schedule_task_on_save(sender, instance, created, **kwargs):
@receiver(post_delete, sender=Recording)
def revoke_task_on_delete(sender, instance, **kwargs):
revoke_task(instance.task_id)
@receiver([post_save, post_delete], sender=ChannelStream)
def update_channel_catchup_fields(sender, instance, **kwargs):
"""Roll up catch-up flags from active streams (UI path; import uses SQL rollup)."""
from django.db.models import Max
channel = instance.channel
catchup_qs = channel.streams.filter(
is_catchup=True,
m3u_account__is_active=True,
)
max_days = catchup_qs.aggregate(max_days=Max("catchup_days"))["max_days"]
Channel.objects.filter(pk=channel.pk).update(
is_catchup=catchup_qs.exists(),
catchup_days=max_days or 0,
)

3
apps/channels/tasks.py
View file

@ -1181,12 +1181,13 @@ def _dvr_ffmpeg_retry_backoff_seconds(retry_index):
def _dvr_build_ffmpeg_cmd(stream_url, recording_id, hls_m3u8, hls_seg_pattern, hls_start_number):
"""Build the FFmpeg command for DVR HLS segment recording."""
from core.utils import dispatcharr_dvr_user_agent
return [
"ffmpeg", "-y",
"-reconnect", "1",
"-reconnect_streamed", "1",
"-reconnect_delay_max", "5",
"-user_agent", f"Dispatcharr-DVR/recording-{recording_id}",
"-user_agent", dispatcharr_dvr_user_agent(recording_id),
# Regenerate monotonic PTS to handle erratic/discontinuous timestamps
# from IPTV sources.
"-fflags", "+genpts",

100
apps/channels/tests/test_catchup_utils.py Normal file
View file

@ -0,0 +1,100 @@
from unittest.mock import patch
from django.test import RequestFactory, SimpleTestCase, TestCase
from apps.accounts.models import User
from apps.channels.models import Channel, ChannelStream, Stream
from apps.channels.utils import resolve_xc_epg_prev_days
from apps.m3u.models import M3UAccount
class ResolveXcEpgPrevDaysTests(SimpleTestCase):
def setUp(self):
self.factory = RequestFactory()
self.user = User(username="xc-prev", custom_properties={})
def test_url_prev_days_zero_is_explicit(self):
request = self.factory.get("/xmltv.php?prev_days=0")
self.assertEqual(resolve_xc_epg_prev_days(request, self.user), 0)
def test_user_epg_prev_days_used_when_url_omitted(self):
self.user.custom_properties = {"epg_prev_days": 5}
request = self.factory.get("/xmltv.php")
self.assertEqual(resolve_xc_epg_prev_days(request, self.user), 5)
@patch("apps.channels.utils.compute_provider_archive_days_capped", return_value=14)
def test_auto_detect_only_when_no_url_or_user_default(self, mock_compute):
request = self.factory.get("/xmltv.php")
self.assertEqual(resolve_xc_epg_prev_days(request, self.user), 14)
mock_compute.assert_called_once()
@patch("apps.channels.utils.compute_provider_archive_days_capped", return_value=14)
def test_per_channel_epg_skips_global_auto_detect(self, mock_compute):
request = self.factory.get("/player_api.php")
self.assertEqual(
resolve_xc_epg_prev_days(request, self.user, auto_detect_fallback=False),
0,
)
mock_compute.assert_not_called()
@patch("apps.channels.utils.compute_provider_archive_days_capped", return_value=14)
def test_user_default_prevents_auto_detect(self, mock_compute):
self.user.custom_properties = {"epg_prev_days": 3}
request = self.factory.get("/xmltv.php")
self.assertEqual(resolve_xc_epg_prev_days(request, self.user), 3)
mock_compute.assert_not_called()
@patch("apps.channels.utils.compute_provider_archive_days_capped", return_value=14)
@patch("core.models.CoreSettings.get_xmltv_prev_days_override", return_value=7)
def test_epg_settings_override_prevents_auto_detect(self, _override, mock_compute):
request = self.factory.get("/xmltv.php")
self.assertEqual(resolve_xc_epg_prev_days(request, self.user), 7)
mock_compute.assert_not_called()
class CatchupRollupActiveAccountTests(TestCase):
"""Denormalized catch-up flags ignore disabled M3U accounts."""
@classmethod
def setUpTestData(cls):
cls.inactive = M3UAccount.objects.create(
name="catchup-inactive",
server_url="http://example.test",
account_type="XC",
is_active=False,
)
def test_channelstream_signal_ignores_inactive_catchup_stream(self):
channel = Channel.objects.create(name="inactive-only")
stream = Stream.objects.create(
name="inactive-catchup",
url="http://example.test/inactive",
m3u_account=self.inactive,
is_catchup=True,
catchup_days=9,
)
ChannelStream.objects.create(channel=channel, stream=stream, order=0)
channel.refresh_from_db()
self.assertFalse(channel.is_catchup)
self.assertEqual(channel.catchup_days, 0)
def test_rollup_ignores_inactive_catchup_stream(self):
from apps.m3u.tasks import rollup_channel_catchup_fields
channel = Channel.objects.create(name="rollup-inactive-only")
stream = Stream.objects.create(
name="rollup-inactive-catchup",
url="http://example.test/rollup-inactive",
m3u_account=self.inactive,
is_catchup=True,
catchup_days=9,
)
ChannelStream.objects.create(channel=channel, stream=stream, order=0)
Channel.objects.filter(pk=channel.pk).update(is_catchup=True, catchup_days=9)
rollup_channel_catchup_fields(self.inactive.id)
channel.refresh_from_db()
self.assertFalse(channel.is_catchup)
self.assertEqual(channel.catchup_days, 0)

69
apps/channels/tests/test_channel_api.py
View file

@ -504,3 +504,72 @@ class SeriesRuleAPITests(TestCase):
def test_bulk_remove_requires_tvg_id_or_title(self):
resp = self.client.post(self.bulk_remove_url, {}, format="json")
self.assertEqual(resp.status_code, status.HTTP_400_BAD_REQUEST)
class ChannelListIncludeStreamsQueryTests(TestCase):
"""include_streams=true must not issue one stream query per channel."""
def setUp(self):
from apps.channels.models import ChannelStream, Stream
from apps.m3u.models import M3UAccount
self.user = User.objects.create_user(username="list_admin", password="x")
self.user.user_level = 10
self.user.save()
self.client = APIClient()
self.client.force_authenticate(user=self.user)
self.account = M3UAccount.objects.create(
name="list-test-account",
account_type="XC",
username="user",
password="pass",
)
self.group = ChannelGroup.objects.create(name=f"List Group {self.id}")
def _add_channel_with_stream(self, number):
from apps.channels.models import ChannelStream, Stream
channel = Channel.objects.create(
channel_number=float(number),
name=f"Channel {number}",
channel_group=self.group,
)
stream = Stream.objects.create(
name=f"Stream {number}",
url=f"http://example.com/{number}.ts",
m3u_account=self.account,
)
ChannelStream.objects.create(channel=channel, stream=stream, order=0)
return channel
def _query_count_for_list(self):
from django.db import connection
from django.test.utils import CaptureQueriesContext
with CaptureQueriesContext(connection) as ctx:
response = self.client.get(
"/api/channels/channels/",
{"page": 1, "page_size": 50, "include_streams": "true"},
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
return len(ctx.captured_queries)
def test_include_streams_query_count_stable_as_channels_grow(self):
self._add_channel_with_stream(1)
self._add_channel_with_stream(2)
self._add_channel_with_stream(3)
q_small = self._query_count_for_list()
self._add_channel_with_stream(4)
self._add_channel_with_stream(5)
self._add_channel_with_stream(6)
self._add_channel_with_stream(7)
q_large = self._query_count_for_list()
self.assertEqual(
q_small,
q_large,
"include_streams list should use prefetched channelstream_set, "
"not one streams M2M query per channel",
)

193
apps/channels/tests/test_recording_playback_auth.py Normal file
View file

@ -0,0 +1,193 @@
"""Tests for DVR recording playback authentication (file/hls endpoints)."""
import os
import shutil
import tempfile
from types import SimpleNamespace
from unittest.mock import patch
from django.test import TestCase, override_settings
from django.utils import timezone
from rest_framework.test import APIClient
from rest_framework_simplejwt.tokens import RefreshToken
from apps.channels.api_views import _recording_auth_query_suffix
from apps.channels.models import Channel, Recording
def _make_admin():
from django.contrib.auth import get_user_model
User = get_user_model()
user, _ = User.objects.get_or_create(
username="recording_playback_admin",
defaults={"user_level": User.UserLevel.ADMIN},
)
user.user_level = User.UserLevel.ADMIN
user.set_password("pass")
user.save()
return user
@override_settings(ALLOWED_HOSTS=["testserver"])
@patch("apps.channels.api_views.network_access_allowed", return_value=True)
class RecordingPlaybackAuthTests(TestCase):
def setUp(self):
self.channel = Channel.objects.create(channel_number=42, name="Playback Auth Channel")
self.user = _make_admin()
self.client = APIClient()
self.tmp = tempfile.NamedTemporaryFile(suffix=".mkv", delete=False)
self.tmp.write(b"\x00" * 1024)
self.tmp.close()
self.hls_dir = tempfile.mkdtemp(prefix="dvr_playback_auth_hls_")
with open(os.path.join(self.hls_dir, "index.m3u8"), "w", encoding="utf-8") as playlist:
playlist.write("#EXTM3U\n#EXTINF:4.0,\nseg_00001.ts\n")
with open(os.path.join(self.hls_dir, "seg_00001.ts"), "wb") as segment:
segment.write(b"\x00" * 188)
now = timezone.now()
self.recording = Recording.objects.create(
channel=self.channel,
start_time=now,
end_time=now,
custom_properties={
"status": "completed",
"file_path": self.tmp.name,
"file_name": "test.mkv",
},
)
def tearDown(self):
if os.path.exists(self.tmp.name):
os.unlink(self.tmp.name)
if os.path.isdir(self.hls_dir):
shutil.rmtree(self.hls_dir, ignore_errors=True)
@staticmethod
def _jwt_for(user):
return str(RefreshToken.for_user(user).access_token)
def test_file_requires_authentication(self, _mock_network):
response = self.client.get(
f"/api/channels/recordings/{self.recording.id}/file/"
)
self.assertEqual(response.status_code, 403)
def test_file_accepts_jwt_query_param(self, _mock_network):
token = self._jwt_for(self.user)
response = self.client.get(
f"/api/channels/recordings/{self.recording.id}/file/",
{"token": token},
)
self.assertEqual(response.status_code, 200)
def test_file_redirect_to_hls_preserves_token(self, _mock_network):
pending = os.path.join(self.hls_dir, "pending.mkv")
now = timezone.now()
in_progress = Recording.objects.create(
channel=self.channel,
start_time=now,
end_time=now,
custom_properties={
"status": "recording",
"_hls_dir": self.hls_dir,
"file_path": pending,
},
)
token = self._jwt_for(self.user)
response = self.client.get(
f"/api/channels/recordings/{in_progress.id}/file/",
{"token": token},
)
self.assertEqual(response.status_code, 302)
self.assertIn("token=", response["Location"])
self.assertIn("/hls/index.m3u8", response["Location"])
def test_hls_playlist_rewrites_segments_with_token_when_present(self, _mock_network):
now = timezone.now()
hls_rec = Recording.objects.create(
channel=self.channel,
start_time=now,
end_time=now,
custom_properties={
"status": "recording",
"_hls_dir": self.hls_dir,
},
)
token = self._jwt_for(self.user)
response = self.client.get(
f"/api/channels/recordings/{hls_rec.id}/hls/index.m3u8",
{"token": token},
)
self.assertEqual(response.status_code, 200)
body = response.content.decode("utf-8")
self.assertIn("token=", body)
self.assertIn("seg_00001.ts", body)
def test_hls_playlist_omits_token_when_not_in_request(self, _mock_network):
now = timezone.now()
hls_rec = Recording.objects.create(
channel=self.channel,
start_time=now,
end_time=now,
custom_properties={
"status": "recording",
"_hls_dir": self.hls_dir,
},
)
token = self._jwt_for(self.user)
response = self.client.get(
f"/api/channels/recordings/{hls_rec.id}/hls/index.m3u8",
HTTP_AUTHORIZATION=f"Bearer {token}",
)
self.assertEqual(response.status_code, 200)
body = response.content.decode("utf-8")
self.assertIn("seg_00001.ts", body)
self.assertNotIn("token=", body)
def test_hls_segment_accepts_jwt_query_param(self, _mock_network):
now = timezone.now()
hls_rec = Recording.objects.create(
channel=self.channel,
start_time=now,
end_time=now,
custom_properties={
"status": "recording",
"_hls_dir": self.hls_dir,
},
)
token = self._jwt_for(self.user)
response = self.client.get(
f"/api/channels/recordings/{hls_rec.id}/hls/seg_00001.ts",
{"token": token},
)
self.assertEqual(response.status_code, 200)
def test_hls_redirect_to_file_preserves_token(self, _mock_network):
now = timezone.now()
hls_rec = Recording.objects.create(
channel=self.channel,
start_time=now,
end_time=now,
custom_properties={
"status": "completed",
"file_path": self.tmp.name,
"file_name": "test.mkv",
},
)
token = self._jwt_for(self.user)
response = self.client.get(
f"/api/channels/recordings/{hls_rec.id}/hls/index.m3u8",
{"token": token},
)
self.assertEqual(response.status_code, 302)
self.assertIn("token=", response["Location"])
self.assertIn("/file/", response["Location"])
class RecordingAuthQuerySuffixTests(TestCase):
def test_empty_when_no_token(self):
request = SimpleNamespace(GET={})
self.assertEqual(_recording_auth_query_suffix(request), "")
def test_includes_token_when_present(self):
request = SimpleNamespace(GET={"token": "abc123"})
self.assertEqual(_recording_auth_query_suffix(request), "?token=abc123")

8
apps/channels/tests/test_ts_proxy_ghost_clients.py
View file

@ -247,6 +247,14 @@ class BasicStatsGhostClientTests(TestCase):
redis.scard.return_value = len(client_ids)
redis.smembers.return_value = client_ids
redis.hget.return_value = None # individual field lookups
redis.hmget.return_value = [
b'VLC/3.0',
b'127.0.0.1',
b'1773500000.0',
None,
b'mpegts',
None,
]
# Pipeline for remove_ghost_clients
pipe = MagicMock()

12
apps/channels/tests/test_ts_proxy_keepalive.py
View file

@ -218,7 +218,7 @@ class DoStatsUpdateTests(TestCase):
mock_redis.scan.return_value = (0, [])
with patch("apps.proxy.live_proxy.client_manager.send_websocket_update") as mock_ws, \
patch("redis.Redis.from_url", return_value=mock_redis):
patch("core.utils.RedisClient.get_client", return_value=mock_redis):
cm._do_stats_update()
mock_ws.assert_called_once()
@ -231,25 +231,25 @@ class DoStatsUpdateTests(TestCase):
"""Redis failure must be swallowed (logged), not propagated."""
cm = self._make_client_manager()
with patch("redis.Redis.from_url", side_effect=Exception("Redis down")):
with patch("core.utils.RedisClient.get_client", side_effect=Exception("Redis down")):
try:
cm._do_stats_update()
except Exception as e:
self.fail(f"_do_stats_update raised an exception: {e}")
def test_do_stats_update_scans_channel_client_keys(self):
"""Must scan for live:channel:*:clients pattern."""
def test_do_stats_update_scans_channel_metadata_keys(self):
"""Must scan for live:channel:*:metadata pattern."""
cm = self._make_client_manager()
mock_redis = MagicMock()
mock_redis.scan.return_value = (0, [])
with patch("apps.proxy.live_proxy.client_manager.send_websocket_update"), \
patch("redis.Redis.from_url", return_value=mock_redis):
patch("core.utils.RedisClient.get_client", return_value=mock_redis):
cm._do_stats_update()
scan_call = mock_redis.scan.call_args
self.assertIn("live:channel:*:clients", str(scan_call))
self.assertIn("live:channel:*:metadata", str(scan_call))
# ---------------------------------------------------------------------------

984
apps/channels/tests/test_ts_proxy_teardown.py Normal file
View file

@ -0,0 +1,984 @@
"""Tests for multi-worker channel teardown coordination."""
import time
from unittest.mock import ANY, MagicMock, patch
from django.test import TestCase
from apps.proxy.live_proxy.constants import ChannelMetadataField, ChannelState
from apps.proxy.live_proxy.input.buffer import StreamBuffer
from apps.proxy.live_proxy.input.manager import StreamManager
from apps.proxy.live_proxy.redis_keys import RedisKeys
from apps.proxy.live_proxy.server import ProxyServer
from apps.proxy.live_proxy.services.channel_service import ChannelService
CHANNEL_ID = "00000000-0000-0000-0000-000000000099"
def _configure_ownership_pipeline(
redis,
*,
stop_exists=0,
metadata_exists=1,
client_count=0,
owner=None,
disconnect=None,
state=None,
):
pipe = MagicMock()
redis.pipeline.return_value = pipe
pipe.execute.return_value = (
stop_exists,
metadata_exists,
client_count,
owner,
disconnect,
state,
)
return pipe
def _mock_proxy_server(redis_client=None):
server = MagicMock()
server.redis_client = redis_client or MagicMock()
server._stopping_channels = set()
return server
class ChannelTeardownAvailabilityTests(TestCase):
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_teardown_active_when_stopping_key_exists(self, mock_get_instance):
redis = MagicMock()
redis.exists.side_effect = lambda key: key == RedisKeys.channel_stopping(CHANNEL_ID)
mock_get_instance.return_value = _mock_proxy_server(redis)
self.assertTrue(ChannelService.is_channel_teardown_active(CHANNEL_ID))
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_teardown_active_when_metadata_state_is_stopping(self, mock_get_instance):
redis = MagicMock()
redis.exists.return_value = False
redis.hget.return_value = ChannelState.STOPPING.encode()
mock_get_instance.return_value = _mock_proxy_server(redis)
self.assertTrue(ChannelService.is_channel_teardown_active(CHANNEL_ID))
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.channel_shutdown_delay")
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_shutdown_pending_within_delay_window(self, mock_get_instance, mock_delay):
mock_delay.return_value = 5
redis = MagicMock()
redis.exists.return_value = False
redis.get.return_value = str(time.time() - 2).encode()
mock_get_instance.return_value = _mock_proxy_server(redis)
self.assertTrue(ChannelService.is_shutdown_pending(CHANNEL_ID))
self.assertFalse(ChannelService.is_channel_unavailable_for_new_clients(CHANNEL_ID))
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.channel_shutdown_delay")
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_cancel_pending_shutdown_clears_disconnect_key(self, mock_get_instance, mock_delay):
mock_delay.return_value = 30
redis = MagicMock()
redis.exists.side_effect = lambda key: "last_client_disconnect" in key
redis.get.return_value = None
redis.hget.return_value = ChannelState.ACTIVE.encode()
mock_get_instance.return_value = _mock_proxy_server(redis)
self.assertTrue(ChannelService.cancel_pending_shutdown(CHANNEL_ID))
redis.delete.assert_any_call(RedisKeys.last_client_disconnect(CHANNEL_ID))
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.channel_shutdown_delay")
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_cancel_pending_shutdown_skips_during_active_stop(self, mock_get_instance, mock_delay):
mock_delay.return_value = 30
redis = MagicMock()
redis.exists.return_value = True
server = _mock_proxy_server(redis)
server._stopping_channels = {CHANNEL_ID}
mock_get_instance.return_value = server
self.assertFalse(ChannelService.cancel_pending_shutdown(CHANNEL_ID))
redis.delete.assert_not_called()
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.channel_shutdown_delay")
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_cancel_pending_shutdown_skips_real_teardown_without_grace(self, mock_get_instance, mock_delay):
mock_delay.return_value = 30
redis = MagicMock()
redis.exists.side_effect = lambda key: "stopping" in key
redis.get.return_value = None
redis.hget.return_value = ChannelState.STOPPING.encode()
mock_get_instance.return_value = _mock_proxy_server(redis)
self.assertFalse(ChannelService.cancel_pending_shutdown(CHANNEL_ID))
redis.hset.assert_not_called()
redis.delete.assert_not_called()
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.channel_shutdown_delay")
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_shutdown_pending_expired_after_delay(self, mock_get_instance, mock_delay):
mock_delay.return_value = 5
redis = MagicMock()
redis.exists.return_value = False
redis.get.return_value = str(time.time() - 10).encode()
mock_get_instance.return_value = _mock_proxy_server(redis)
self.assertFalse(ChannelService.is_shutdown_pending(CHANNEL_ID))
class ClientManagerAddClientTests(TestCase):
@patch("apps.proxy.live_proxy.services.channel_service.ChannelService.cancel_pending_shutdown", return_value=False)
@patch("apps.proxy.live_proxy.client_manager.send_websocket_update")
def test_add_client_stores_ip_and_user_agent_in_redis(self, _mock_ws, _mock_cancel):
from apps.proxy.live_proxy.client_manager import ClientManager
redis = MagicMock()
cm = ClientManager(CHANNEL_ID, redis_client=redis, worker_id="worker-1")
cm.proxy_server = MagicMock()
result = cm.add_client(
"client-1",
"10.0.2.163",
user_agent="VLC/3.0.21",
)
self.assertEqual(result, 1)
mapping = redis.hset.call_args[1]["mapping"]
self.assertEqual(mapping["ip_address"], "10.0.2.163")
self.assertEqual(mapping["user_agent"], "VLC/3.0.21")
class LocalStreamActivityTests(TestCase):
def _make_server(self):
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
server = ProxyServer()
server.worker_id = "testhost:1"
server.stream_managers = {}
server.stream_buffers = {}
server.client_managers = {}
server.profile_managers = {}
server.profile_buffers = {}
server._live_stream_managers = {}
server._stopping_channels = set()
server.redis_client = MagicMock()
server.stop_all_output_formats = MagicMock()
server.stop_all_output_profiles = MagicMock()
return server
@patch.object(ProxyServer, "_join_stream_thread")
def test_stop_local_stream_activity_stops_live_manager(self, mock_join):
server = self._make_server()
manager = MagicMock()
server._live_stream_managers[CHANNEL_ID] = manager
server._stop_local_stream_activity(CHANNEL_ID)
manager.stop.assert_called_once()
mock_join.assert_called_once_with(CHANNEL_ID)
self.assertNotIn(CHANNEL_ID, server._live_stream_managers)
class OrphanMetadataCleanupTests(TestCase):
def _make_server(self):
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
server = ProxyServer()
server.worker_id = "testhost:1"
server.stream_managers = {}
server.stream_buffers = {}
server.client_managers = {}
server.profile_managers = {}
server.profile_buffers = {}
server._live_stream_managers = {}
server._stopping_channels = set()
server.redis_client = MagicMock()
return server
@patch.object(ProxyServer, "_clean_redis_keys")
@patch.object(ProxyServer, "_stop_local_stream_activity")
@patch.object(ProxyServer, "_has_local_upstream_activity", return_value=True)
def test_orphan_metadata_stops_local_processes_before_redis(
self, mock_has_upstream, mock_stop_local, mock_clean_redis
):
server = self._make_server()
metadata_key = RedisKeys.channel_metadata(CHANNEL_ID)
server.redis_client.keys.return_value = [metadata_key.encode()]
server.redis_client.hgetall.return_value = {
b"owner": b"",
b"state": b"unknown",
}
server.redis_client.exists.return_value = False
server.redis_client.scard.return_value = 0
server._check_orphaned_metadata()
mock_has_upstream.assert_called_with(CHANNEL_ID)
mock_stop_local.assert_called_once_with(CHANNEL_ID)
mock_clean_redis.assert_called_once_with(CHANNEL_ID)
@patch.object(ProxyServer, "_clean_redis_keys")
@patch.object(ProxyServer, "_broadcast_upstream_stop")
@patch.object(ProxyServer, "_stop_local_stream_activity")
@patch.object(ProxyServer, "_has_local_upstream_activity", return_value=False)
def test_orphan_metadata_remote_channel_broadcasts_stop(
self, mock_has_upstream, mock_stop_local, mock_broadcast, mock_clean_redis
):
server = self._make_server()
metadata_key = RedisKeys.channel_metadata(CHANNEL_ID)
server.redis_client.keys.return_value = [metadata_key.encode()]
server.redis_client.hgetall.return_value = {b"owner": b"", b"state": b"unknown"}
server.redis_client.exists.return_value = False
server.redis_client.scard.return_value = 0
server._check_orphaned_metadata()
mock_broadcast.assert_called_once_with(CHANNEL_ID)
mock_stop_local.assert_not_called()
mock_clean_redis.assert_called_once_with(CHANNEL_ID)
class OrphanChannelCleanupTests(TestCase):
def _make_server(self):
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
server = ProxyServer()
server.worker_id = "testhost:1"
server.stream_managers = {}
server.stream_buffers = {}
server.client_managers = {}
server.profile_managers = {}
server.profile_buffers = {}
server._live_stream_managers = {}
server._stopping_channels = set()
server.redis_client = MagicMock()
server.get_channel_owner = MagicMock(return_value=None)
return server
@patch.object(ProxyServer, "_clean_redis_keys")
@patch.object(ProxyServer, "_stop_local_stream_activity")
@patch.object(ProxyServer, "_has_local_upstream_activity", return_value=True)
def test_orphan_channel_stops_local_before_redis(
self, mock_has_upstream, mock_stop_local, mock_clean_redis
):
server = self._make_server()
metadata_key = RedisKeys.channel_metadata(CHANNEL_ID)
server.redis_client.keys.return_value = [metadata_key.encode()]
server.redis_client.scard.return_value = 0
server._check_orphaned_channels()
mock_stop_local.assert_called_once_with(CHANNEL_ID)
mock_clean_redis.assert_called_once_with(CHANNEL_ID)
class StreamManagerOwnershipTests(TestCase):
def test_still_owner_false_when_different_worker(self):
buffer = MagicMock()
_configure_ownership_pipeline(
buffer.redis_client,
owner=b"worker-b",
)
manager = StreamManager(
CHANNEL_ID, "http://example/stream", buffer, worker_id="worker-a"
)
self.assertFalse(manager._still_owner())
def test_still_owner_true_when_owner_lock_expired_but_not_stopping(self):
buffer = MagicMock()
_configure_ownership_pipeline(
buffer.redis_client,
client_count=0,
owner=None,
state=ChannelState.CONNECTING.encode(),
)
manager = StreamManager(
CHANNEL_ID, "http://example/stream", buffer, worker_id="worker-a"
)
self.assertTrue(manager._still_owner())
def test_still_owner_false_when_channel_stopping_key_set(self):
buffer = MagicMock()
_configure_ownership_pipeline(
buffer.redis_client,
stop_exists=1,
)
manager = StreamManager(
CHANNEL_ID, "http://example/stream", buffer, worker_id="worker-a"
)
self.assertFalse(manager._still_owner())
def test_update_bytes_skipped_after_ownership_lost(self):
buffer = MagicMock()
_configure_ownership_pipeline(
buffer.redis_client,
owner=b"other-worker",
)
manager = StreamManager(
CHANNEL_ID, "http://example/stream", buffer, worker_id="worker-a"
)
manager.bytes_processed = 1000
manager._update_bytes_processed(500)
buffer.redis_client.hincrby.assert_not_called()
class StreamBufferStopTests(TestCase):
def test_stop_discards_local_data_without_redis_writes(self):
redis = MagicMock()
buffer = StreamBuffer(channel_id=CHANNEL_ID, redis_client=redis)
buffer._write_buffer = bytearray(b"x" * 376)
buffer.stop()
self.assertTrue(buffer.stopping)
self.assertEqual(len(buffer._write_buffer), 0)
redis.incr.assert_not_called()
redis.setex.assert_not_called()
redis.delete.assert_not_called()
class StopChannelTeardownTests(TestCase):
def _make_server(self):
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
server = ProxyServer()
server.worker_id = "testhost:1"
server.stream_managers = {}
server.stream_buffers = {}
server.client_managers = {}
server.profile_managers = {}
server.profile_buffers = {}
server._live_stream_managers = {}
server._stopping_channels = set()
server._stopping_since = {}
server.redis_client = MagicMock()
server.redis_client.exists.return_value = False
server.am_i_owner = MagicMock(return_value=False)
server._collect_channel_stop_event_data = MagicMock(return_value=None)
server.release_ownership = MagicMock()
return server
@patch.object(ProxyServer, "_spawn_channel_stop_event")
@patch.object(ProxyServer, "_clean_redis_keys")
@patch.object(ProxyServer, "_stop_local_stream_activity")
def test_stop_channel_cleans_redis_before_blocking_local_stop(
self, mock_stop_local, mock_clean_redis, mock_spawn_event
):
server = self._make_server()
call_order = []
def stop_local(channel_id):
call_order.append("local")
def clean_redis(channel_id):
call_order.append("redis")
mock_stop_local.side_effect = stop_local
mock_clean_redis.side_effect = clean_redis
server.stop_channel(CHANNEL_ID)
self.assertEqual(call_order, ["redis", "local"])
mock_spawn_event.assert_called_once_with(None)
@patch.object(ProxyServer, "_spawn_channel_stop_event")
@patch.object(ProxyServer, "_clean_redis_keys")
@patch.object(ProxyServer, "_stop_local_stream_activity", side_effect=RuntimeError("boom"))
def test_stop_channel_cleans_redis_in_finally_when_local_stop_fails(
self, mock_stop_local, mock_clean_redis, mock_spawn_event
):
server = self._make_server()
result = server.stop_channel(CHANNEL_ID)
self.assertFalse(result)
mock_clean_redis.assert_called_once_with(CHANNEL_ID)
mock_spawn_event.assert_not_called()
self.assertNotIn(CHANNEL_ID, server._stopping_channels)
@patch.object(ProxyServer, "_spawn_channel_stop_event")
@patch.object(ProxyServer, "_clean_redis_keys")
@patch.object(ProxyServer, "_stop_local_stream_activity")
def test_stop_channel_owner_releases_after_redis_cleanup(
self, mock_stop_local, mock_clean_redis, mock_spawn_event
):
server = self._make_server()
server.am_i_owner.return_value = True
stop_data = {"channel_id": CHANNEL_ID}
server._collect_channel_stop_event_data.return_value = stop_data
call_order = []
def stop_local(channel_id):
call_order.append("local")
def release(channel_id):
call_order.append("release")
def clean_redis(channel_id):
call_order.append("redis")
mock_stop_local.side_effect = stop_local
server.release_ownership.side_effect = release
mock_clean_redis.side_effect = clean_redis
server.stop_channel(CHANNEL_ID)
self.assertEqual(call_order, ["redis", "release", "local"])
server._collect_channel_stop_event_data.assert_called_once_with(CHANNEL_ID)
mock_spawn_event.assert_called_once_with(stop_data)
class CleanRedisKeysOrderTests(TestCase):
@patch("apps.proxy.live_proxy.server.Stream.objects.get")
@patch("apps.proxy.live_proxy.server.Channel.objects.get")
def test_clean_redis_keys_releases_profile_slot_before_live_keys_deleted(
self, mock_channel_get, mock_stream_get
):
from apps.channels.models import Channel, Stream
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
server = ProxyServer()
server.redis_client = MagicMock()
call_order = []
channel = MagicMock()
channel.release_stream.return_value = True
def channel_get(uuid):
call_order.append("release")
return channel
mock_channel_get.side_effect = channel_get
mock_stream_get.side_effect = Stream.DoesNotExist
channel_key = f"live:channel:{CHANNEL_ID}:input:buffer:index".encode()
def scan(cursor, match=None, count=100):
call_order.append("redis")
if match == f"live:channel:{CHANNEL_ID}:*":
return (0, [channel_key])
return (0, [])
server.redis_client.scan.side_effect = scan
server._clean_redis_keys(CHANNEL_ID)
self.assertEqual(call_order, ["release", "redis", "redis"])
channel.release_stream.assert_called_once()
server.redis_client.delete.assert_called_once_with(channel_key)
class LocalUpstreamActivityTests(TestCase):
def _make_server(self):
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
server = ProxyServer()
server.worker_id = "testhost:1"
server.stream_managers = {}
server.stream_buffers = {}
server.client_managers = {}
server._live_stream_managers = {}
return server
def test_upstream_activity_excludes_reader_only_client_manager(self):
server = self._make_server()
server.client_managers[CHANNEL_ID] = MagicMock()
server.stream_buffers[CHANNEL_ID] = MagicMock()
self.assertFalse(server._has_local_upstream_activity(CHANNEL_ID))
def test_upstream_activity_from_live_registry(self):
server = self._make_server()
server._live_stream_managers[CHANNEL_ID] = MagicMock()
self.assertTrue(server._has_local_upstream_activity(CHANNEL_ID))
class ClientDisconnectOwnershipTests(TestCase):
def test_last_client_triggers_stop_when_upstream_active_without_owner_lock(self):
from apps.proxy.live_proxy.client_manager import ClientManager
proxy_server = MagicMock()
proxy_server.worker_id = "testhost:1"
proxy_server.am_i_owner.return_value = False
proxy_server._has_local_upstream_activity.return_value = True
proxy_server.extend_ownership.return_value = False
redis = MagicMock()
redis.hget.return_value = b"viewer"
redis.scard.return_value = 0
manager = ClientManager(
channel_id=CHANNEL_ID,
redis_client=redis,
worker_id="testhost:1",
)
manager.proxy_server = proxy_server
manager.clients = {"client-1"}
manager.client_set_key = RedisKeys.clients(CHANNEL_ID)
manager._notify_owner_of_activity = MagicMock()
manager._trigger_stats_update = MagicMock()
manager.get_total_client_count = MagicMock(return_value=0)
proxy_server._spawn_on_hub = MagicMock()
manager.remove_client("client-1")
proxy_server._spawn_on_hub.assert_called_once_with(
proxy_server.handle_client_disconnect, CHANNEL_ID
)
class TeardownActiveLocalStopTests(TestCase):
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_teardown_active_when_local_stop_in_progress(self, mock_get_instance):
server = MagicMock()
server._stopping_channels = {CHANNEL_ID}
server.redis_client = MagicMock()
server.redis_client.exists.return_value = False
mock_get_instance.return_value = server
self.assertTrue(ChannelService.is_channel_teardown_active(CHANNEL_ID))
class HandleClientDisconnectUpstreamFirstTests(TestCase):
def _make_server(self):
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
server = ProxyServer()
server.worker_id = "testhost:1"
server.client_managers = {CHANNEL_ID: MagicMock()}
server.stream_managers = {CHANNEL_ID: MagicMock()}
server._live_stream_managers = {}
server.profile_managers = {}
server.output_managers = {}
server.redis_client = MagicMock()
server.redis_client.scard.return_value = 0
server._stopping_channels = set()
return server
@patch.object(ProxyServer, "_coordinated_stop_channel")
@patch.object(ProxyServer, "_stop_upstream_before_redis_cleanup")
@patch.object(ProxyServer, "_has_local_upstream_activity", return_value=True)
def test_last_client_uses_coordinated_stop_only(
self, _mock_upstream, mock_stop_upstream, mock_coordinated
):
server = self._make_server()
with patch(
"apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay",
return_value=0,
):
server.handle_client_disconnect(CHANNEL_ID)
mock_stop_upstream.assert_not_called()
mock_coordinated.assert_called_once_with(CHANNEL_ID)
class ShutdownDelayWaitTests(TestCase):
def _make_server(self):
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
with patch.object(ProxyServer, "_start_cleanup_thread"):
server = ProxyServer()
server.redis_client = MagicMock()
server.redis_client.scard.return_value = 0
return server
@patch("apps.proxy.live_proxy.server.gevent.sleep")
@patch("apps.proxy.live_proxy.server.time.time", return_value=1000.0)
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
def test_aborts_when_disconnect_key_deleted(self, _mock_delay, _mock_time, mock_sleep):
server = self._make_server()
server.redis_client.get.side_effect = [b"1000.0", None]
result = server._wait_for_shutdown_delay(CHANNEL_ID)
self.assertFalse(result)
self.assertGreaterEqual(mock_sleep.call_count, 1)
@patch("apps.proxy.live_proxy.server.gevent.sleep")
@patch("apps.proxy.live_proxy.server.time.time", return_value=1000.0)
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
def test_aborts_when_clients_reconnect(self, _mock_delay, _mock_time, _mock_sleep):
server = self._make_server()
server.redis_client.get.return_value = b"1000.0"
server.redis_client.scard.side_effect = [0, 1]
result = server._wait_for_shutdown_delay(CHANNEL_ID)
self.assertFalse(result)
server.redis_client.delete.assert_called_with(
RedisKeys.last_client_disconnect(CHANNEL_ID)
)
@patch("apps.proxy.live_proxy.server.gevent.sleep")
@patch("apps.proxy.live_proxy.server.time.time")
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
def test_timer_resets_when_disconnect_timestamp_updated(
self, _mock_delay, mock_time, mock_sleep
):
server = self._make_server()
disconnect_key = RedisKeys.last_client_disconnect(CHANNEL_ID)
current_time = [1000.0]
disconnect_timestamp = [1000.0]
poll_count = [0]
mock_time.side_effect = lambda: current_time[0]
def get_side_effect(key):
poll_count[0] += 1
if poll_count[0] >= 3:
disconnect_timestamp[0] = 1020.0
if key == disconnect_key:
return str(disconnect_timestamp[0]).encode()
return None
server.redis_client.get.side_effect = get_side_effect
def advance_sleep(duration):
current_time[0] += duration
mock_sleep.side_effect = advance_sleep
result = server._wait_for_shutdown_delay(CHANNEL_ID)
self.assertTrue(result)
self.assertGreaterEqual(current_time[0], 1050.0)
@patch.object(ProxyServer, "_coordinated_stop_channel")
@patch.object(ProxyServer, "_wait_for_shutdown_delay", return_value=True)
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
def test_handle_client_disconnect_uses_polling_wait(
self, _mock_delay, mock_wait, mock_coordinated
):
server = HandleClientDisconnectUpstreamFirstTests()._make_server()
server.redis_client.get.return_value = b"1700000000.0"
server.handle_client_disconnect(CHANNEL_ID)
mock_wait.assert_called_once_with(CHANNEL_ID)
mock_coordinated.assert_called_once_with(CHANNEL_ID)
@patch.object(ProxyServer, "_coordinated_stop_channel")
@patch.object(ProxyServer, "_wait_for_shutdown_delay", return_value=False)
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
def test_handle_client_disconnect_skips_stop_when_wait_aborted(
self, _mock_delay, _mock_wait, mock_coordinated
):
server = HandleClientDisconnectUpstreamFirstTests()._make_server()
server.redis_client.get.return_value = b"1700000000.0"
server.handle_client_disconnect(CHANNEL_ID)
mock_coordinated.assert_not_called()
class InitWaitAbortTests(TestCase):
def _make_generator(self):
from apps.proxy.live_proxy.output.ts.generator import StreamGenerator
return StreamGenerator(
CHANNEL_ID,
"client-1",
"127.0.0.1",
"test-agent",
channel_initializing=True,
)
def test_abort_when_client_removed_locally(self):
generator = self._make_generator()
server = MagicMock()
client_manager = MagicMock()
client_manager.clients = set()
server.client_managers = {CHANNEL_ID: client_manager}
server.redis_client = MagicMock()
self.assertEqual(generator._init_wait_abort_reason(server, time.time()), "client_gone")
@patch("apps.proxy.live_proxy.output.ts.generator.ConfigHelper.channel_init_grace_period", return_value=10)
def test_abort_when_connect_stalled_without_buffer(self, _mock_grace):
generator = self._make_generator()
server = MagicMock()
client_manager = MagicMock()
client_manager.clients = {"client-1"}
server.client_managers = {CHANNEL_ID: client_manager}
buffer = MagicMock()
buffer.index = 0
server.stream_buffers = {CHANNEL_ID: buffer}
server.redis_client = MagicMock()
server.redis_client.hget.return_value = b"connecting"
server.redis_client.get.return_value = None
started = time.time() - 11
self.assertEqual(generator._init_wait_abort_reason(server, started), "stalled")
@patch("apps.proxy.live_proxy.output.ts.generator.ConfigHelper.channel_init_grace_period", return_value=30)
def test_no_stall_abort_within_init_grace_period(self, _mock_grace):
generator = self._make_generator()
server = MagicMock()
client_manager = MagicMock()
client_manager.clients = {"client-1"}
server.client_managers = {CHANNEL_ID: client_manager}
buffer = MagicMock()
buffer.index = 0
server.stream_buffers = {CHANNEL_ID: buffer}
server.redis_client = MagicMock()
server.redis_client.hget.return_value = b"connecting"
server.redis_client.get.return_value = None
started = time.time() - 15
self.assertIsNone(generator._init_wait_abort_reason(server, started))
@patch("apps.proxy.live_proxy.output.ts.generator.ConfigHelper.channel_init_grace_period", return_value=30)
def test_stall_abort_after_init_grace_period(self, _mock_grace):
generator = self._make_generator()
server = MagicMock()
client_manager = MagicMock()
client_manager.clients = {"client-1"}
server.client_managers = {CHANNEL_ID: client_manager}
buffer = MagicMock()
buffer.index = 0
server.stream_buffers = {CHANNEL_ID: buffer}
server.redis_client = MagicMock()
server.redis_client.hget.return_value = b"connecting"
server.redis_client.get.return_value = None
started = time.time() - 31
self.assertEqual(generator._init_wait_abort_reason(server, started), "stalled")
class PromoteChannelWhenBufferReadyTests(TestCase):
def _mock_proxy(self, redis_client):
proxy_server = MagicMock()
proxy_server.redis_client = redis_client
return patch(
"apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance",
return_value=proxy_server,
), proxy_server
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.initial_behind_chunks", return_value=4)
def test_buffer_ready_with_clients_becomes_active(self, _mock_chunks):
redis = MagicMock()
redis.hget.return_value = ChannelState.CONNECTING.encode()
redis.get.return_value = b"4"
redis.scard.return_value = 2
ctx, proxy_server = self._mock_proxy(redis)
with ctx:
result = ChannelService.promote_channel_when_buffer_ready(CHANNEL_ID)
self.assertEqual(result, ChannelState.ACTIVE)
proxy_server.update_channel_state.assert_called_once()
args = proxy_server.update_channel_state.call_args[0]
self.assertEqual(args[1], ChannelState.ACTIVE)
self.assertEqual(args[2]["clients_at_activation"], "2")
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.initial_behind_chunks", return_value=4)
def test_buffer_ready_without_clients_becomes_waiting(self, _mock_chunks):
redis = MagicMock()
redis.hget.return_value = ChannelState.CONNECTING.encode()
redis.get.return_value = b"5"
redis.scard.return_value = 0
ctx, proxy_server = self._mock_proxy(redis)
with ctx:
result = ChannelService.promote_channel_when_buffer_ready(CHANNEL_ID)
self.assertEqual(result, ChannelState.WAITING_FOR_CLIENTS)
proxy_server.update_channel_state.assert_called_once_with(
CHANNEL_ID,
ChannelState.WAITING_FOR_CLIENTS,
{
ChannelMetadataField.CONNECTION_READY_TIME: ANY,
ChannelMetadataField.BUFFER_CHUNKS: "5",
},
)
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.initial_behind_chunks", return_value=4)
def test_buffer_not_ready_does_not_promote(self, _mock_chunks):
redis = MagicMock()
redis.hget.return_value = ChannelState.CONNECTING.encode()
redis.get.return_value = b"2"
ctx, proxy_server = self._mock_proxy(redis)
with ctx:
result = ChannelService.promote_channel_when_buffer_ready(CHANNEL_ID)
self.assertIsNone(result)
proxy_server.update_channel_state.assert_not_called()
def test_waiting_for_clients_with_clients_becomes_active(self):
redis = MagicMock()
def hget_side_effect(key, field):
if field == ChannelMetadataField.STATE:
return ChannelState.WAITING_FOR_CLIENTS.encode()
if field == ChannelMetadataField.CONNECTION_READY_TIME:
return b"1700000000.0"
return None
redis.hget.side_effect = hget_side_effect
redis.scard.return_value = 1
ctx, proxy_server = self._mock_proxy(redis)
with ctx:
result = ChannelService.promote_channel_when_buffer_ready(CHANNEL_ID)
self.assertEqual(result, ChannelState.ACTIVE)
proxy_server.update_channel_state.assert_called_once_with(
CHANNEL_ID,
ChannelState.ACTIVE,
{"clients_at_activation": "1"},
)
class UpstreamStopBroadcastTests(TestCase):
def _make_server(self):
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
server = ProxyServer()
server.worker_id = "testhost:1"
server.stream_managers = {}
server._live_stream_managers = {}
server.redis_client = MagicMock()
return server
@patch.object(ProxyServer, "_stop_local_stream_activity")
@patch.object(ProxyServer, "_broadcast_upstream_stop")
@patch.object(ProxyServer, "_has_local_upstream_activity", return_value=True)
def test_local_upstream_stops_locally_without_broadcast(
self, _mock_has, mock_broadcast, mock_stop_local
):
server = self._make_server()
server._stop_upstream_before_redis_cleanup(CHANNEL_ID)
mock_stop_local.assert_called_once_with(CHANNEL_ID)
mock_broadcast.assert_not_called()
@patch.object(ProxyServer, "_stop_local_stream_activity")
@patch.object(ProxyServer, "_broadcast_upstream_stop")
@patch.object(ProxyServer, "_has_local_upstream_activity", return_value=False)
def test_orphan_cleanup_broadcasts_when_no_local_upstream(
self, _mock_has, mock_broadcast, mock_stop_local
):
server = self._make_server()
server._stop_upstream_before_redis_cleanup(CHANNEL_ID)
mock_broadcast.assert_called_once_with(CHANNEL_ID)
mock_stop_local.assert_not_called()
class StreamManagerStillOwnerTests(TestCase):
def _make_manager(self, redis_client):
buffer = MagicMock()
buffer.redis_client = redis_client
buffer.channel_id = CHANNEL_ID
manager = StreamManager(
CHANNEL_ID,
"http://example/stream.ts",
buffer,
worker_id="testhost:1",
)
return manager
def test_stops_when_metadata_removed(self):
redis = MagicMock()
_configure_ownership_pipeline(redis, metadata_exists=0)
manager = self._make_manager(redis)
self.assertFalse(manager._still_owner())
def test_keeps_running_during_connecting_before_client_registered(self):
redis = MagicMock()
_configure_ownership_pipeline(
redis,
client_count=0,
owner=b"testhost:1",
state=ChannelState.CONNECTING.encode(),
)
manager = self._make_manager(redis)
self.assertTrue(manager._still_owner())
def test_stops_after_disconnect_when_shutdown_delay_is_zero(self):
redis = MagicMock()
_configure_ownership_pipeline(
redis,
client_count=0,
owner=b"testhost:1",
disconnect=b"1700000000.0",
state=ChannelState.ACTIVE.encode(),
)
manager = self._make_manager(redis)
with patch(
"apps.proxy.live_proxy.input.manager.ConfigHelper.channel_shutdown_delay",
return_value=0,
):
self.assertFalse(manager._still_owner())
def test_keeps_running_during_shutdown_delay(self):
redis = MagicMock()
_configure_ownership_pipeline(
redis,
client_count=0,
owner=b"testhost:1",
disconnect=str(time.time()).encode(),
state=ChannelState.ACTIVE.encode(),
)
manager = self._make_manager(redis)
with patch(
"apps.proxy.live_proxy.input.manager.ConfigHelper.channel_shutdown_delay",
return_value=5,
):
self.assertTrue(manager._still_owner())
class PreActiveNoClientsTimeoutTests(TestCase):
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_client_wait_period", return_value=5)
def test_buffer_ready_uses_client_wait_period(self, _mock_client_wait):
should_stop, timeout, reason = ProxyServer._pre_active_no_clients_should_stop(
connection_ready_time=1000.0,
start_time=900.0,
now=1006.0,
)
self.assertTrue(should_stop)
self.assertEqual(timeout, 5)
self.assertEqual(reason, "client_wait")
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_client_wait_period", return_value=5)
def test_buffer_ready_within_client_wait_period(self, _mock_client_wait):
should_stop, timeout, reason = ProxyServer._pre_active_no_clients_should_stop(
connection_ready_time=1000.0,
start_time=900.0,
now=1003.0,
)
self.assertFalse(should_stop)
self.assertEqual(timeout, 5)
self.assertEqual(reason, "client_wait")
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_init_grace_period", return_value=60)
def test_startup_uses_init_grace_period(self, _mock_init_grace):
should_stop, timeout, reason = ProxyServer._pre_active_no_clients_should_stop(
connection_ready_time=None,
start_time=1000.0,
now=1070.0,
)
self.assertTrue(should_stop)
self.assertEqual(timeout, 60)
self.assertEqual(reason, "startup")
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_init_grace_period", return_value=60)
def test_startup_within_init_grace_period(self, _mock_init_grace):
should_stop, timeout, reason = ProxyServer._pre_active_no_clients_should_stop(
connection_ready_time=None,
start_time=1000.0,
now=1030.0,
)
self.assertFalse(should_stop)
self.assertEqual(timeout, 60)
self.assertEqual(reason, "startup")
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_client_wait_period", return_value=5)
def test_buffer_ready_does_not_use_shutdown_delay(self, mock_client_wait, mock_shutdown_delay):
should_stop, _, reason = ProxyServer._pre_active_no_clients_should_stop(
connection_ready_time=1000.0,
start_time=900.0,
now=1006.0,
)
self.assertTrue(should_stop)
self.assertEqual(reason, "client_wait")
mock_client_wait.assert_called_once()
mock_shutdown_delay.assert_not_called()

90
apps/channels/utils.py
View file

@ -1,8 +1,13 @@
import logging
import threading
from django.core.cache import cache
logger = logging.getLogger(__name__)
PROVIDER_ARCHIVE_CACHE_TTL_SECONDS = 300
MAX_AUTO_PREV_DAYS = 30
# Bound memory/DB work per chunk for large libraries (20k+ channels).
EPG_LOGO_APPLY_BATCH_SIZE = 500
EPG_LOGO_APPLY_MAX_ERRORS = 100
@ -23,6 +28,91 @@ def format_channel_number(value, empty=""):
return int(value)
return value
def compute_provider_archive_days_capped():
"""Max ``catchup_days`` across active XC catch-up streams (capped, cached).
Cached briefly so XC XMLTV exports without an explicit ``prev_days`` do not
repeat the aggregate query on every request.
"""
def _scan():
from django.db.models import Max
from apps.channels.models import Stream
result = Stream.objects.filter(
m3u_account__account_type="XC",
m3u_account__is_active=True,
is_catchup=True,
).aggregate(max_days=Max("catchup_days"))
return min(result["max_days"] or 0, MAX_AUTO_PREV_DAYS)
return cache.get_or_set(
"channels:provider_archive_days_capped",
_scan,
PROVIDER_ARCHIVE_CACHE_TTL_SECONDS,
)
def resolve_xc_epg_prev_days(request, user, *, auto_detect_fallback=True):
"""Resolve ``prev_days`` for XC XMLTV and player_api EPG.
Args:
request: HTTP request (reads ``?prev_days=``).
user: Authenticated user (reads ``custom_properties.epg_prev_days``).
auto_detect_fallback: When True (XC XMLTV), fall back to the largest
provider archive depth. When False (per-channel EPG), return 0 so
``xc_get_epg`` can expand to each channel's ``catchup_days``.
Resolution order:
1. URL ``?prev_days=`` (explicit; 0 means no past programmes)
2. ``user.custom_properties.epg_prev_days``
3. ``CoreSettings.epg_settings.xmltv_prev_days_override`` when > 0
4. Auto-detect (only when *auto_detect_fallback* is True)
"""
user_custom = (user.custom_properties or {}) if user else {}
url_prev = request.GET.get("prev_days")
user_prev = user_custom.get("epg_prev_days") if user_custom else None
if url_prev is not None:
try:
return max(0, min(int(url_prev), MAX_AUTO_PREV_DAYS))
except (ValueError, TypeError):
return 0
if user_prev not in (None, ""):
try:
return max(0, min(int(user_prev), MAX_AUTO_PREV_DAYS))
except (ValueError, TypeError):
return 0
from core.models import CoreSettings
try:
override = int(CoreSettings.get_xmltv_prev_days_override() or 0)
except (TypeError, ValueError):
override = 0
if override > 0:
return max(0, min(override, MAX_AUTO_PREV_DAYS))
if auto_detect_fallback:
return compute_provider_archive_days_capped()
return 0
def get_channel_catchup_streams(channel):
"""Active catch-up streams for a channel, in ``channelstream`` order.
Inactive M3U accounts are excluded, matching live dispatch.
"""
if not getattr(channel, "is_catchup", False):
return []
return list(
channel.streams.filter(is_catchup=True, m3u_account__is_active=True)
.order_by("channelstream__order")
.select_related("m3u_account")
)
def increment_stream_count(account):
with lock:
current_usage = active_streams_map.get(account.id, 0)

129
apps/connect/tests/test_trigger_event.py
View file

@ -35,43 +35,56 @@ def _empty_subscription_chain():
class TriggerEventDispatchTests(SimpleTestCase):
def _run_trigger_event(self, plugins, event_name, payload):
def _run_trigger_event(self, handlers, event_name, payload, enabled_keys=None):
pm = MagicMock()
pm.list_plugins.return_value = plugins
pm.iter_actions_for_event.return_value = handlers
if enabled_keys is None:
enabled_keys = [key for key, _ in handlers]
enabled_qs = MagicMock()
enabled_qs.values_list.return_value = enabled_keys
with patch(
"apps.connect.utils.PluginManager.get", return_value=pm
), patch(
"apps.connect.utils.EventSubscription.objects.filter",
return_value=_empty_subscription_chain(),
):
), patch(
"apps.plugins.models.PluginConfig"
) as mock_cfg:
mock_cfg.objects.filter.return_value = enabled_qs
from apps.connect.utils import trigger_event
trigger_event(event_name, payload)
return pm
def test_disabled_plugin_does_not_abort_dispatch_for_later_enabled_plugin(self):
"""The original bug: AttributeError on a disabled plugin's debug log
aborted the loop, so any plugin iterated AFTER a disabled one never
received events."""
plugins = [
{
"key": "disabled-plugin",
"name": "Disabled Plugin",
"enabled": False,
"actions": [],
},
{
"key": "enabled-plugin",
"name": "Enabled Plugin",
"enabled": True,
"actions": [
{"id": "on_event", "events": ["channel_start"]},
],
},
"""Enabled handlers still run when other plugins are disabled in DB."""
handlers = [
("enabled-plugin", "on_event"),
]
pm = self._run_trigger_event(
plugins, "channel_start", {"channel_name": "TEST"}
handlers, "channel_start", {"channel_name": "TEST"}
)
pm.run_action.assert_called_once_with(
"enabled-plugin",
"on_event",
{"event": "channel_start", "payload": {"channel_name": "TEST"}},
)
def test_skips_handlers_for_disabled_plugins(self):
handlers = [
("disabled-plugin", "on_event"),
("enabled-plugin", "on_event"),
]
pm = self._run_trigger_event(
handlers,
"channel_start",
{"channel_name": "TEST"},
enabled_keys=["enabled-plugin"],
)
pm.run_action.assert_called_once_with(
@ -81,22 +94,64 @@ class TriggerEventDispatchTests(SimpleTestCase):
)
def test_action_without_matching_event_is_not_dispatched(self):
"""Sanity check: actions whose `events` list doesn't include the
fired event, or which have no `events` key at all, are skipped."""
plugins = [
{
"key": "plugin",
"name": "Plugin",
"enabled": True,
"actions": [
{"id": "on_event", "events": ["channel_stop"]},
{"id": "manual_button"}, # no events key
],
},
]
"""When no handlers are registered for the event, run_action is not called."""
pm = self._run_trigger_event(
plugins, "channel_start", {"channel_name": "TEST"}
[], "channel_start", {"channel_name": "TEST"}
)
pm.run_action.assert_not_called()
def test_no_plugin_config_query_when_no_handlers(self):
pm = MagicMock()
pm.iter_actions_for_event.return_value = []
with patch(
"apps.connect.utils.PluginManager.get", return_value=pm
), patch(
"apps.connect.utils.EventSubscription.objects.filter",
return_value=_empty_subscription_chain(),
), patch(
"apps.plugins.models.PluginConfig"
) as mock_cfg:
from apps.connect.utils import trigger_event
trigger_event("channel_start", {"channel_name": "TEST"})
mock_cfg.objects.filter.assert_not_called()
def test_plugin_action_failure_does_not_block_sibling_handlers(self):
handlers = [
("failing-plugin", "on_event"),
("working-plugin", "on_event"),
]
pm = MagicMock()
pm.iter_actions_for_event.return_value = handlers
pm.run_action.side_effect = [RuntimeError("boom"), {"status": "ok"}]
enabled_qs = MagicMock()
enabled_qs.values_list.return_value = ["failing-plugin", "working-plugin"]
with patch(
"apps.connect.utils.PluginManager.get", return_value=pm
), patch(
"apps.connect.utils.EventSubscription.objects.filter",
return_value=_empty_subscription_chain(),
), patch(
"apps.plugins.models.PluginConfig"
) as mock_cfg:
mock_cfg.objects.filter.return_value = enabled_qs
from apps.connect.utils import trigger_event
trigger_event("channel_start", {"channel_name": "TEST"})
self.assertEqual(pm.run_action.call_count, 2)
pm.run_action.assert_any_call(
"failing-plugin",
"on_event",
{"event": "channel_start", "payload": {"channel_name": "TEST"}},
)
pm.run_action.assert_any_call(
"working-plugin",
"on_event",
{"event": "channel_start", "payload": {"channel_name": "TEST"}},
)

59
apps/connect/utils.py
View file

@ -1,5 +1,5 @@
# connect/utils.py
import logging, json
import logging
from django.template import Template, Context
from .models import EventSubscription, DeliveryLog, SUPPORTED_EVENTS
from .handlers.webhook import WebhookHandler
@ -94,23 +94,44 @@ def trigger_event(event_name, payload):
pm = PluginManager.get()
pm.discover_plugins(sync_db=False, use_cache=True)
plugins = pm.list_plugins()
handlers = list(pm.iter_actions_for_event(event_name))
if not handlers:
return
logger.debug(f"Checking {len(plugins)} plugins for event '{event_name}'")
for plugin in plugins:
if not plugin["enabled"]:
logger.debug(f"Skipping disabled plugin id={plugin['key']} name={plugin['name']}")
from apps.plugins.models import PluginConfig
handler_keys = {key for key, _ in handlers}
enabled_keys = set(
PluginConfig.objects.filter(enabled=True, key__in=handler_keys).values_list(
"key", flat=True
)
)
logger.debug(
"Dispatching event '%s' to %d plugin action(s) (%d enabled)",
event_name,
len(handlers),
len(enabled_keys),
)
params = {"event": event_name, "payload": payload}
for key, action_id in handlers:
if key not in enabled_keys:
logger.debug(
"Skipping disabled plugin id=%s for event '%s'", key, event_name
)
continue
logger.debug(json.dumps(plugin))
for action in plugin["actions"]:
if "events" in action and event_name in action["events"]:
key = plugin["key"]
params = {"event": event_name, "payload": payload}
action_name = action.get("label") or action.get("id")
action_id = action.get("id")
logger.debug(
f"Triggering plugin action for event '{event_name}' on plugin id={key} action={action_name}"
)
if action_id:
pm.run_action(key, action_id, params)
logger.debug(
"Triggering plugin action for event '%s' on plugin id=%s action=%s",
event_name,
key,
action_id,
)
try:
pm.run_action(key, action_id, params)
except Exception:
logger.exception(
"Plugin action failed for event '%s' on plugin id=%s action=%s",
event_name,
key,
action_id,
)

99
apps/epg/api_views.py
View file

@ -64,8 +64,6 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
from apps.channels.models import Channel
return EPGSource.objects.select_related(
"refresh_task__crontab", "refresh_task__interval"
).defer(
'programme_index'
).annotate(
has_channels=Exists(
Channel.objects.filter(epg_data__epg_source_id=OuterRef('pk'))
@ -128,7 +126,7 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
import hashlib
import requests as http_requests
from apps.epg.tasks import SD_BASE_URL
from version import __version__ as dispatcharr_version
from core.utils import dispatcharr_http_headers
username = (source.username or '').strip()
password = (source.password or '').strip()
@ -143,7 +141,7 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
auth_response = http_requests.post(
f"{SD_BASE_URL}/token",
json={'username': username, 'password': sha1_password},
headers={'Content-Type': 'application/json', 'User-Agent': f'Dispatcharr/{dispatcharr_version}'},
headers=dispatcharr_http_headers(),
timeout=15,
)
auth_response.raise_for_status()
@ -227,6 +225,24 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
f"Lockout set until {reset_at.isoformat()}."
)
def _fetch_sd_countries(self):
"""Fetch the SD country list (token not required; User-Agent is)."""
import requests as http_requests
from apps.epg.tasks import SD_BASE_URL
from core.utils import dispatcharr_http_headers
try:
resp = http_requests.get(
f"{SD_BASE_URL}/available/countries",
headers=dispatcharr_http_headers(content_type=None),
timeout=15,
)
resp.raise_for_status()
return resp.json()
except http_requests.exceptions.RequestException as e:
logger.warning(f"Failed to fetch SD countries: {e}")
return None
@action(detail=True, methods=["get", "post", "delete"], url_path="sd-lineups")
def sd_lineups(self, request, pk=None):
"""
@ -236,7 +252,7 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
"""
import requests as http_requests
from apps.epg.tasks import SD_BASE_URL
from version import __version__ as dispatcharr_version
from core.utils import dispatcharr_http_headers
source = self.get_object()
if source.source_type != 'schedules_direct':
@ -249,13 +265,10 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
if error:
return error
headers = {
'Content-Type': 'application/json',
'User-Agent': f'Dispatcharr/{dispatcharr_version}',
'token': token,
}
headers = dispatcharr_http_headers(token=token)
if request.method == "GET":
countries = self._fetch_sd_countries()
try:
resp = http_requests.get(
f"{SD_BASE_URL}/lineups",
@ -272,6 +285,7 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
"changes_remaining": self._get_sd_changes_remaining(source),
"changes_reset_at": self._get_sd_reset_at(source),
"notice": "No lineups are currently configured on this Schedules Direct account. Use the search below to add one.",
"countries": countries,
})
resp.raise_for_status()
data = resp.json()
@ -281,6 +295,7 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
"max_lineups": 4,
"changes_remaining": self._get_sd_changes_remaining(source),
"changes_reset_at": self._get_sd_reset_at(source),
"countries": countries,
})
except http_requests.exceptions.RequestException as e:
return Response(
@ -399,7 +414,7 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
"""
import requests as http_requests
from apps.epg.tasks import SD_BASE_URL
from version import __version__ as dispatcharr_version
from core.utils import dispatcharr_http_headers
source = self.get_object()
if source.source_type != 'schedules_direct':
@ -420,11 +435,7 @@ class EPGSourceViewSet(viewsets.ModelViewSet):
if error:
return error
headers = {
'Content-Type': 'application/json',
'User-Agent': f'Dispatcharr/{dispatcharr_version}',
'token': token,
}
headers = dispatcharr_http_headers(token=token)
try:
resp = http_requests.get(
@ -493,6 +504,7 @@ class ProgramViewSet(viewsets.ModelViewSet):
"""Proxy endpoint for SD program poster images. Nginx caches the response."""
import requests as http_requests
from apps.epg.tasks import SD_BASE_URL
from core.utils import dispatcharr_http_headers
program = self.get_object()
poster_sd_url = (program.custom_properties or {}).get('sd_icon')
@ -516,14 +528,10 @@ class ProgramViewSet(viewsets.ModelViewSet):
if not token:
sha1_password = hashlib.sha1(source.password.encode('utf-8')).hexdigest()
try:
from version import __version__ as dispatcharr_version
auth_resp = http_requests.post(
f"{SD_BASE_URL}/token",
json={'username': source.username, 'password': sha1_password},
headers={
'Content-Type': 'application/json',
'User-Agent': f'Dispatcharr/{dispatcharr_version}',
},
headers=dispatcharr_http_headers(),
timeout=10,
)
auth_data = auth_resp.json()
@ -549,7 +557,7 @@ class ProgramViewSet(viewsets.ModelViewSet):
try:
img_resp = http_requests.get(
poster_sd_url,
headers={'token': token},
headers=dispatcharr_http_headers(token=token, content_type=None),
timeout=15,
allow_redirects=True,
)
@ -976,7 +984,9 @@ class EPGGridAPIView(APIView):
channel_name=name_to_parse,
num_days=1,
program_length_hours=4,
epg_source=epg_source
epg_source=epg_source,
export_lookback=one_hour_ago,
export_cutoff=twenty_four_hours_later,
)
# Custom dummy should always return data (either from patterns or fallback)
@ -1072,13 +1082,19 @@ class EPGGridAPIView(APIView):
f"Error creating standard dummy programs for channel {channel.name} (ID: {channel.id}): {str(e)}"
)
# Combine regular and dummy programs
all_programs = list(serialized_programs) + dummy_programs
# Combine regular and dummy programs in place to avoid copying the large list
serialized_programs.extend(dummy_programs)
logger.debug(
f"EPGGridAPIView: Returning {len(all_programs)} total programs (including {len(dummy_programs)} dummy programs)."
f"EPGGridAPIView: Returning {len(serialized_programs)} total programs (including {len(dummy_programs)} dummy programs)."
)
return Response({"data": all_programs}, status=status.HTTP_200_OK)
# The grid materializes tens of thousands of program dicts plus the
# rendered JSON; trim once the response is sent so worker RSS does not
# ratchet up per request.
from core.utils import spawn_memory_trim
response = Response({"data": serialized_programs}, status=status.HTTP_200_OK)
response._resource_closers.append(spawn_memory_trim)
return response
# ─────────────────────────────
@ -1109,18 +1125,24 @@ class EPGImportAPIView(APIView):
epg_id = request.data.get("id", None)
force = bool(request.data.get("force", False))
# Check if this is a dummy EPG source
try:
# Reject dummy sources with a narrow existence query, no full row load.
if epg_id is not None:
from .models import EPGSource
epg_source = EPGSource.objects.get(id=epg_id)
if epg_source.source_type == 'dummy':
logger.info(f"EPGImportAPIView: Skipping refresh for dummy EPG source {epg_id}")
if EPGSource.objects.filter(
id=epg_id, source_type="dummy"
).exists():
logger.info(
"EPGImportAPIView: Skipping refresh for dummy EPG source %s",
epg_id,
)
return Response(
{"success": False, "message": "Dummy EPG sources do not require refreshing."},
{
"success": False,
"message": "Dummy EPG sources do not require refreshing.",
},
status=status.HTTP_400_BAD_REQUEST,
)
except EPGSource.DoesNotExist:
pass # Let the task handle the missing source
refresh_epg_data.delay(epg_id, force=force) # Trigger Celery task
logger.info("EPGImportAPIView: Task dispatched to refresh EPG data.")
@ -1220,10 +1242,9 @@ class CurrentProgramsAPIView(APIView):
# Limit to 50 IDs per request
epg_data_ids = epg_data_ids[:50]
# Defer the multi-MB programme_index the JOIN would pull once per row. The lookup reads it via a targeted refresh_from_db
epg_data_entries = EPGData.objects.select_related('epg_source').defer(
'epg_source__programme_index'
).filter(id__in=epg_data_ids)
epg_data_entries = EPGData.objects.select_related('epg_source').filter(
id__in=epg_data_ids
)
# Batch-fetch current programs for all requested EPG entries in one query
db_programs = ProgramData.objects.filter(

40
apps/epg/migrations/0025_programdata_epg_id_index.py Normal file
View file

@ -0,0 +1,40 @@
from django.contrib.postgres.operations import AddIndexConcurrently
from django.db import migrations, models
class AddIndexConcurrentlyIfPostgres(AddIndexConcurrently):
"""Create the index CONCURRENTLY on PostgreSQL (no table lock on large
tables), falling back to a normal blocking AddIndex on other backends
such as the sqlite dev/test fallback."""
def database_forwards(self, app_label, schema_editor, from_state, to_state):
if schema_editor.connection.vendor == 'postgresql':
super().database_forwards(app_label, schema_editor, from_state, to_state)
else:
migrations.AddIndex.database_forwards(
self, app_label, schema_editor, from_state, to_state
)
def database_backwards(self, app_label, schema_editor, from_state, to_state):
if schema_editor.connection.vendor == 'postgresql':
super().database_backwards(app_label, schema_editor, from_state, to_state)
else:
migrations.AddIndex.database_backwards(
self, app_label, schema_editor, from_state, to_state
)
class Migration(migrations.Migration):
# CREATE INDEX CONCURRENTLY cannot run inside a transaction.
atomic = False
dependencies = [
('epg', '0024_remove_epgsource_api_key_epgsource_password_and_more'),
]
operations = [
AddIndexConcurrentlyIfPostgres(
model_name='programdata',
index=models.Index(fields=['epg', 'id'], name='epg_prog_epg_id_idx'),
),
]

52
apps/epg/migrations/0026_epgsourceindex.py Normal file
View file

@ -0,0 +1,52 @@
import django.db.models.deletion
from django.db import migrations, models
def copy_index_forward(apps, schema_editor):
EPGSource = apps.get_model('epg', 'EPGSource')
EPGSourceIndex = apps.get_model('epg', 'EPGSourceIndex')
rows = list(
EPGSource.objects.exclude(programme_index__isnull=True).values_list(
'id', 'programme_index'
)
)
for source_id, data in rows:
EPGSourceIndex.objects.update_or_create(
source_id=source_id, defaults={'data': data}
)
def copy_index_backward(apps, schema_editor):
EPGSource = apps.get_model('epg', 'EPGSource')
EPGSourceIndex = apps.get_model('epg', 'EPGSourceIndex')
for source_id, data in EPGSourceIndex.objects.values_list('source_id', 'data'):
EPGSource.objects.filter(id=source_id).update(programme_index=data)
class Migration(migrations.Migration):
dependencies = [
('epg', '0025_programdata_epg_id_index'),
]
operations = [
migrations.CreateModel(
name='EPGSourceIndex',
fields=[
('source', models.OneToOneField(
on_delete=django.db.models.deletion.CASCADE,
primary_key=True,
related_name='index_record',
serialize=False,
to='epg.epgsource',
)),
('data', models.JSONField(blank=True, default=None, null=True)),
('updated_at', models.DateTimeField(auto_now=True)),
],
),
migrations.RunPython(copy_index_forward, copy_index_backward),
migrations.RemoveField(
model_name='epgsource',
name='programme_index',
),
]

42
apps/epg/models.py
View file

@ -62,12 +62,6 @@ class EPGSource(models.Model):
blank=True,
help_text="Last status message, including success results or error information"
)
programme_index = models.JSONField(
null=True,
blank=True,
default=None,
help_text="Byte-offset index mapping tvg_id to file positions, built after each EPG refresh"
)
created_at = models.DateTimeField(
auto_now_add=True,
help_text="Time when this source was created"
@ -124,6 +118,37 @@ class EPGSource(models.Model):
kwargs['update_fields'].remove('updated_at')
super().save(*args, **kwargs)
@property
def programme_index(self):
"""Byte-offset index for this source, read on demand from the separate
EPGSourceIndex table so the multi-MB blob is never pulled into EPGSource
queries or select_related JOINs. Returns the stored dict or None."""
return (
EPGSourceIndex.objects.filter(source_id=self.pk)
.values_list('data', flat=True)
.first()
)
class EPGSourceIndex(models.Model):
"""Byte-offset programme index for an EPGSource, stored in its own table.
Kept out of EPGSource so the multi-MB JSON blob is only loaded when read
explicitly, never when querying or joining EPGSource rows.
"""
source = models.OneToOneField(
EPGSource,
on_delete=models.CASCADE,
related_name='index_record',
primary_key=True,
)
data = models.JSONField(null=True, blank=True, default=None)
updated_at = models.DateTimeField(auto_now=True)
def __str__(self):
return f"Programme index for source {self.source_id}"
class EPGData(models.Model):
tvg_id = models.CharField(max_length=255, null=True, blank=True, db_index=True)
name = models.CharField(max_length=512)
@ -153,6 +178,11 @@ class ProgramData(models.Model):
program_id = models.CharField(max_length=64, null=True, blank=True, help_text='Schedules Direct programID (e.g. EP123456789). Null for XMLTV sources.')
custom_properties = models.JSONField(default=dict, blank=True, null=True)
class Meta:
indexes = [
models.Index(fields=['epg', 'id'], name='epg_prog_epg_id_idx'),
]
def __str__(self):
return f"{self.title} ({self.start_time} - {self.end_time})"

2506
apps/epg/tasks.py
View file

File diff suppressed because it is too large Load diff

98
apps/epg/tests/test_epg_import_api.py Normal file
View file

@ -0,0 +1,98 @@
from unittest.mock import patch
from django.contrib.auth import get_user_model
from django.db import connection
from django.test.utils import CaptureQueriesContext
from django.test import TestCase
from rest_framework import status
from rest_framework.test import APIClient
from apps.epg.models import EPGSource, EPGSourceIndex
User = get_user_model()
IMPORT_URL = "/api/epg/import/"
class EPGImportAPITests(TestCase):
def setUp(self):
self.user = User.objects.create_user(
username="epg_import_admin", password="testpass123"
)
self.user.user_level = 10
self.user.save()
self.client = APIClient()
self.client.force_authenticate(user=self.user)
@patch("apps.epg.api_views.refresh_epg_data.delay")
def test_import_dummy_source_rejected_without_dispatch(self, mock_delay):
source = EPGSource.objects.create(
name="Dummy EPG",
source_type="dummy",
)
response = self.client.post(
IMPORT_URL, {"id": source.id}, format="json"
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertFalse(response.data["success"])
mock_delay.assert_not_called()
@patch("apps.epg.api_views.refresh_epg_data.delay")
def test_import_xmltv_dispatches_without_loading_programme_index(
self, mock_delay
):
source = EPGSource.objects.create(
name="Large Index XMLTV",
source_type="xmltv",
url="http://example.com/epg.xml",
)
EPGSourceIndex.objects.create(
source=source,
data={
"channels": {f"ch.{i}": {"offsets": [0, 100]} for i in range(200)},
"interleaved_channels": [],
},
)
mock_delay.reset_mock()
with CaptureQueriesContext(connection) as ctx:
response = self.client.post(
IMPORT_URL, {"id": source.id}, format="json"
)
self.assertEqual(response.status_code, status.HTTP_202_ACCEPTED)
self.assertTrue(response.data["success"])
mock_delay.assert_called_once_with(source.id, force=False)
for query in ctx.captured_queries:
self.assertNotIn(
"programme_index",
query["sql"].lower(),
"import trigger should not read programme_index",
)
@patch("apps.epg.api_views.refresh_epg_data.delay")
def test_import_missing_source_still_dispatches(self, mock_delay):
response = self.client.post(IMPORT_URL, {"id": 99999}, format="json")
self.assertEqual(response.status_code, status.HTTP_202_ACCEPTED)
mock_delay.assert_called_once_with(99999, force=False)
@patch("apps.epg.api_views.refresh_epg_data.delay")
def test_import_honours_force_flag(self, mock_delay):
source = EPGSource.objects.create(
name="Force XMLTV",
source_type="xmltv",
url="http://example.com/epg.xml",
)
mock_delay.reset_mock()
response = self.client.post(
IMPORT_URL,
{"id": source.id, "force": True},
format="json",
)
self.assertEqual(response.status_code, status.HTTP_202_ACCEPTED)
mock_delay.assert_called_once_with(source.id, force=True)

95
apps/epg/tests/test_epg_source_file_lock.py Normal file
View file

@ -0,0 +1,95 @@
from unittest.mock import patch
from django.test import SimpleTestCase, TestCase
from apps.epg.models import EPGSource, EPGData
from apps.epg.tasks import (
_EPG_PARSE_DEFER_MAX,
_EPG_REFRESH_DEFER_SECONDS,
_defer_parse_programs_for_tvg_id,
_refresh_epg_data_impl,
parse_programs_for_tvg_id,
)
class DeferParseProgramsTests(SimpleTestCase):
@patch("apps.epg.tasks.parse_programs_for_tvg_id.apply_async")
def test_defer_schedules_retry_for_refresh(self, mock_apply_async):
result = _defer_parse_programs_for_tvg_id(5, False, 0, "source refresh in progress")
self.assertEqual(result, "Deferred")
mock_apply_async.assert_called_once_with(
args=[5],
kwargs={"force": False, "_defer_retry": 1},
countdown=_EPG_REFRESH_DEFER_SECONDS,
)
@patch("apps.epg.tasks.parse_programs_for_tvg_id.apply_async")
def test_defer_gives_up_after_max_retries(self, mock_apply_async):
result = _defer_parse_programs_for_tvg_id(
5, False, _EPG_PARSE_DEFER_MAX, "source refresh in progress"
)
self.assertIn("Deferred too many times", result)
mock_apply_async.assert_not_called()
class ParseProgramsForTvgIdLockTests(TestCase):
def setUp(self):
self.source = EPGSource.objects.create(
name="Lock Test",
source_type="xmltv",
file_path="/tmp/unused.xml",
)
self.epg = EPGData.objects.create(
tvg_id="test.channel",
name="Test",
epg_source=self.source,
)
@patch("apps.epg.tasks.parse_programs_for_tvg_id.apply_async")
@patch("apps.epg.tasks.is_task_lock_held", return_value=True)
def test_defers_while_source_refresh_running(self, mock_held, mock_apply_async):
result = parse_programs_for_tvg_id(self.epg.id)
self.assertEqual(result, "Deferred")
mock_held.assert_called_once_with("refresh_epg_data", self.source.id)
mock_apply_async.assert_called_once()
@patch("apps.epg.tasks._decr_source_tvg_parse_count")
@patch("apps.epg.tasks._incr_source_tvg_parse_count")
@patch("apps.epg.tasks.acquire_task_lock", return_value=False)
@patch("apps.epg.tasks.is_task_lock_held", return_value=False)
def test_skips_when_duplicate_epg_parse_running(
self, mock_held, mock_acquire, mock_incr, mock_decr
):
result = parse_programs_for_tvg_id(self.epg.id)
self.assertEqual(result, "Task already running")
mock_acquire.assert_called_once_with("parse_epg_programs", self.epg.id)
mock_incr.assert_called_once_with(self.source.id)
mock_decr.assert_called_once_with(self.source.id)
class RefreshEpgDataDeferTests(TestCase):
def setUp(self):
self.source = EPGSource.objects.create(
name="Refresh Defer Test",
source_type="xmltv",
file_path="/tmp/unused.xml",
)
@patch("apps.epg.tasks.refresh_epg_data.apply_async")
@patch("apps.epg.tasks.fetch_xmltv")
@patch("apps.epg.tasks._source_tvg_parse_count", return_value=1)
def test_refresh_defers_while_per_channel_parses_running(
self, mock_count, mock_fetch, mock_apply_async
):
_refresh_epg_data_impl(self.source.id)
mock_fetch.assert_not_called()
mock_apply_async.assert_called_once_with(
args=[self.source.id],
kwargs={"force": False, "_file_defer_retry": 1},
countdown=_EPG_REFRESH_DEFER_SECONDS,
)

274
apps/epg/tests/test_parse_programs_for_source.py Normal file
View file

@ -0,0 +1,274 @@
import os
import tempfile
from datetime import timedelta
from unittest.mock import patch
from django.db import connection, transaction
from django.test import TestCase
from django.utils import timezone
from apps.channels.models import Channel
from apps.epg.models import EPGSource, EPGData, ProgramData
from apps.epg.tasks import (
parse_programs_for_source,
_flush_epg_program_staging_batch,
_swap_staged_epg_programs,
_delete_orphaned_epg_programs,
_dispatch_late_mapped_epg_parses,
_EPG_PARSE_BATCH_SIZE,
)
def _programme_xml(channel_id, title, start, stop):
return (
f' <programme start="{start}" stop="{stop}" channel="{channel_id}">\n'
f' <title>{title}</title>\n'
f' </programme>\n'
)
def _xmltv_file(programmes):
body = (
'<?xml version="1.0" encoding="UTF-8"?>\n'
'<tv generator-info-name="test">\n'
f'{programmes}'
'</tv>\n'
)
handle = tempfile.NamedTemporaryFile(
mode='w',
suffix='.xml',
delete=False,
encoding='utf-8',
)
handle.write(body)
handle.close()
return handle.name
class ParseProgramsForSourceTests(TestCase):
def setUp(self):
self.source = EPGSource.objects.create(
name='XMLTV Parse Test',
source_type='xmltv',
)
self.mapped_epg = EPGData.objects.create(
epg_source=self.source,
tvg_id='mapped.channel',
name='Mapped Channel',
)
self.unmapped_epg = EPGData.objects.create(
epg_source=self.source,
tvg_id='unmapped.channel',
name='Unmapped Channel',
)
Channel.objects.create(
channel_number=1,
name='Mapped Channel',
epg_data=self.mapped_epg,
)
self.base_time = timezone.now().replace(minute=0, second=0, microsecond=0)
self.start = self.base_time.strftime('%Y%m%d%H%M%S +0000')
self.stop = (self.base_time + timedelta(hours=1)).strftime('%Y%m%d%H%M%S +0000')
def tearDown(self):
if getattr(self, 'xml_path', None) and os.path.exists(self.xml_path):
os.unlink(self.xml_path)
def _configure_source_file(self, programmes):
self.xml_path = _xmltv_file(programmes)
self.source.file_path = self.xml_path
self.source.save(update_fields=['file_path'])
@patch('apps.epg.tasks.log_system_event')
@patch('apps.epg.tasks.send_epg_update')
def test_replaces_programs_for_mapped_channels(self, _send_update, _log_event):
old_start = self.base_time - timedelta(days=1)
ProgramData.objects.create(
epg=self.mapped_epg,
start_time=old_start,
end_time=old_start + timedelta(hours=1),
title='Old Programme',
tvg_id=self.mapped_epg.tvg_id,
)
orphan_start = self.base_time - timedelta(days=1)
ProgramData.objects.create(
epg=self.unmapped_epg,
start_time=orphan_start,
end_time=orphan_start + timedelta(hours=1),
title='Orphan Programme',
tvg_id=self.unmapped_epg.tvg_id,
)
programmes = (
_programme_xml('mapped.channel', 'New Show', self.start, self.stop)
+ _programme_xml('unmapped.channel', 'Skipped Show', self.start, self.stop)
)
self._configure_source_file(programmes)
result = parse_programs_for_source(self.source)
self.assertTrue(result)
mapped_programs = ProgramData.objects.filter(epg=self.mapped_epg)
self.assertEqual(mapped_programs.count(), 1)
self.assertEqual(mapped_programs.get().title, 'New Show')
self.assertFalse(ProgramData.objects.filter(epg=self.unmapped_epg).exists())
@patch('apps.epg.tasks.log_system_event')
@patch('apps.epg.tasks.send_epg_update')
def test_atomic_failure_rolls_back_and_preserves_existing_programs(self, _send_update, _log_event):
old_start = self.base_time - timedelta(days=1)
ProgramData.objects.create(
epg=self.mapped_epg,
start_time=old_start,
end_time=old_start + timedelta(hours=1),
title='Keep Me',
tvg_id=self.mapped_epg.tvg_id,
)
self._configure_source_file(
_programme_xml('mapped.channel', 'Replacement', self.start, self.stop)
)
swap_path = (
'apps.epg.tasks._swap_staged_epg_programs'
if connection.vendor == 'postgresql'
else 'apps.epg.tasks._swap_parsed_epg_programs'
)
with patch(swap_path, side_effect=RuntimeError('simulated insert failure')):
result = parse_programs_for_source(self.source)
self.assertFalse(result)
self.assertEqual(ProgramData.objects.filter(epg=self.mapped_epg).count(), 1)
self.assertEqual(
ProgramData.objects.get(epg=self.mapped_epg).title,
'Keep Me',
)
@patch('apps.epg.tasks.log_system_event')
@patch('apps.epg.tasks.send_epg_update')
def test_streams_batches_without_holding_full_program_list(self, _send_update, _log_event):
if connection.vendor != 'postgresql':
self.skipTest('PostgreSQL staging batches are required for this assertion')
programme_count = _EPG_PARSE_BATCH_SIZE * 2
programmes = ''.join(
_programme_xml(
'mapped.channel',
f'Show {idx}',
self.start,
self.stop,
)
for idx in range(programme_count)
)
self._configure_source_file(programmes)
flush_sizes = []
original_flush = _flush_epg_program_staging_batch
def tracking_flush(batch):
flush_sizes.append(len(batch))
return original_flush(batch)
with patch('apps.epg.tasks._flush_epg_program_staging_batch', side_effect=tracking_flush):
result = parse_programs_for_source(self.source)
self.assertTrue(result)
self.assertEqual(ProgramData.objects.filter(epg=self.mapped_epg).count(), programme_count)
self.assertEqual(sum(flush_sizes), programme_count)
self.assertTrue(all(size <= _EPG_PARSE_BATCH_SIZE for size in flush_sizes))
self.assertGreater(len(flush_sizes), 1)
@patch('apps.epg.tasks.log_system_event')
@patch('apps.epg.tasks.send_epg_update')
def test_live_programs_remain_until_swap_commits(self, _send_update, _log_event):
if connection.vendor != 'postgresql':
self.skipTest('PostgreSQL staging swap is required for this assertion')
old_start = self.base_time - timedelta(days=1)
ProgramData.objects.create(
epg=self.mapped_epg,
start_time=old_start,
end_time=old_start + timedelta(hours=1),
title='Old Programme',
tvg_id=self.mapped_epg.tvg_id,
)
self._configure_source_file(
_programme_xml('mapped.channel', 'New Show', self.start, self.stop)
)
observed_titles_at_swap = []
def swap_with_visibility_check(mapped_epg_ids, epg_source, *args, **kwargs):
observed_titles_at_swap.append(
ProgramData.objects.get(epg=self.mapped_epg).title
)
return _swap_staged_epg_programs(mapped_epg_ids, epg_source, *args, **kwargs)
with patch('apps.epg.tasks._swap_staged_epg_programs', side_effect=swap_with_visibility_check):
result = parse_programs_for_source(self.source)
self.assertTrue(result)
self.assertEqual(observed_titles_at_swap, ['Old Programme'])
self.assertEqual(ProgramData.objects.get(epg=self.mapped_epg).title, 'New Show')
@patch('apps.epg.tasks.log_system_event')
@patch('apps.epg.tasks.send_epg_update')
def test_swap_delete_is_rolled_back_when_insert_fails(self, _send_update, _log_event):
if connection.vendor != 'postgresql':
self.skipTest('PostgreSQL staging swap is required for this assertion')
old_start = self.base_time - timedelta(days=1)
ProgramData.objects.create(
epg=self.mapped_epg,
start_time=old_start,
end_time=old_start + timedelta(hours=1),
title='Keep Me',
tvg_id=self.mapped_epg.tvg_id,
)
self._configure_source_file(
_programme_xml('mapped.channel', 'Replacement', self.start, self.stop)
)
def failing_swap(mapped_epg_ids, epg_source, *args, **kwargs):
with transaction.atomic():
ProgramData.objects.filter(epg_id__in=mapped_epg_ids).delete()
raise RuntimeError('simulated insert failure')
with patch('apps.epg.tasks._swap_staged_epg_programs', side_effect=failing_swap):
result = parse_programs_for_source(self.source)
self.assertFalse(result)
self.assertEqual(ProgramData.objects.get(epg=self.mapped_epg).title, 'Keep Me')
def test_orphan_cleanup_respects_channels_mapped_during_bulk_parse(self):
late_start = self.base_time - timedelta(days=1)
ProgramData.objects.create(
epg=self.unmapped_epg,
start_time=late_start,
end_time=late_start + timedelta(hours=1),
title='Late Match Programme',
tvg_id=self.unmapped_epg.tvg_id,
)
Channel.objects.create(
channel_number=2,
name='Late Mapped Channel',
epg_data=self.unmapped_epg,
)
deleted = _delete_orphaned_epg_programs(self.source)
self.assertEqual(deleted, 0)
self.assertEqual(ProgramData.objects.filter(epg=self.unmapped_epg).count(), 1)
@patch('apps.epg.tasks.dispatch_program_refresh_for_epg_ids', return_value=1)
def test_late_mapped_dispatches_per_channel_parse(self, mock_dispatch):
Channel.objects.create(
channel_number=2,
name='Late Mapped Channel',
epg_data=self.unmapped_epg,
)
bulk_snapshot = {self.mapped_epg.id}
dispatched = _dispatch_late_mapped_epg_parses(self.source, bulk_snapshot)
self.assertEqual(dispatched, 1)
mock_dispatch.assert_called_once_with({self.unmapped_epg.id})

128
apps/epg/tests/test_parse_programs_for_tvg_id.py Normal file
View file

@ -0,0 +1,128 @@
import os
import tempfile
from datetime import timedelta
from unittest.mock import patch
from django.test import TestCase
from django.utils import timezone
from apps.channels.models import Channel
from apps.epg.models import EPGSource, EPGData, ProgramData
from apps.epg.tasks import parse_programs_for_tvg_id
def _programme_xml(channel_id, title, start, stop):
return (
f' <programme start="{start}" stop="{stop}" channel="{channel_id}">\n'
f' <title>{title}</title>\n'
f' </programme>\n'
)
def _xmltv_file(programmes):
body = (
'<?xml version="1.0" encoding="UTF-8"?>\n'
'<tv generator-info-name="test">\n'
f'{programmes}'
'</tv>\n'
)
handle = tempfile.NamedTemporaryFile(
mode='w',
suffix='.xml',
delete=False,
encoding='utf-8',
)
handle.write(body)
handle.close()
return handle.name
class ParseProgramsForTvgIdSwapTests(TestCase):
def setUp(self):
self.source = EPGSource.objects.create(
name='Per-Channel Parse Test',
source_type='xmltv',
)
self.epg = EPGData.objects.create(
epg_source=self.source,
tvg_id='test.channel',
name='Test Channel',
)
self.channel = Channel.objects.create(
channel_number=1,
name='Test Channel',
epg_data=self.epg,
)
self.base_time = timezone.now().replace(minute=0, second=0, microsecond=0)
self.start = self.base_time.strftime('%Y%m%d%H%M%S +0000')
self.stop = (self.base_time + timedelta(hours=1)).strftime('%Y%m%d%H%M%S +0000')
def tearDown(self):
if getattr(self, 'xml_path', None) and os.path.exists(self.xml_path):
os.unlink(self.xml_path)
def _configure_source_file(self, programmes):
self.xml_path = _xmltv_file(programmes)
self.source.file_path = self.xml_path
self.source.save(update_fields=['file_path'])
def test_replaces_programs_for_channel(self):
old_start = self.base_time - timedelta(days=1)
ProgramData.objects.create(
epg=self.epg,
start_time=old_start,
end_time=old_start + timedelta(hours=1),
title='Old Programme',
tvg_id=self.epg.tvg_id,
)
self._configure_source_file(
_programme_xml('test.channel', 'New Show', self.start, self.stop)
)
parse_programs_for_tvg_id(self.epg.id)
programs = ProgramData.objects.filter(epg=self.epg)
self.assertEqual(programs.count(), 1)
self.assertEqual(programs.get().title, 'New Show')
def test_failed_insert_preserves_existing_programs(self):
"""A failed atomic swap must not leave the channel with no guide data."""
old_start = self.base_time - timedelta(days=1)
ProgramData.objects.create(
epg=self.epg,
start_time=old_start,
end_time=old_start + timedelta(hours=1),
title='Keep Me',
tvg_id=self.epg.tvg_id,
)
self._configure_source_file(
_programme_xml('test.channel', 'Replacement', self.start, self.stop)
)
with patch(
'apps.epg.tasks.ProgramData.objects.bulk_create',
side_effect=RuntimeError('simulated insert failure'),
):
with self.assertRaises(RuntimeError):
parse_programs_for_tvg_id(self.epg.id)
remaining = ProgramData.objects.filter(epg=self.epg)
self.assertEqual(remaining.count(), 1)
self.assertEqual(remaining.get().title, 'Keep Me')
def test_does_not_refetch_epg_data_mid_task(self):
"""The task must reuse the EPGData row loaded at task start."""
self._configure_source_file(
_programme_xml('test.channel', 'New Show', self.start, self.stop)
)
with patch(
'apps.epg.tasks.EPGData.objects.get',
side_effect=AssertionError('should not re-fetch EPGData mid-task'),
) as mock_get:
parse_programs_for_tvg_id(self.epg.id)
mock_get.assert_not_called()
self.assertEqual(
ProgramData.objects.filter(epg=self.epg).get().title, 'New Show'
)

44
apps/epg/tests/test_parsing_channels_progress.py Normal file
View file

@ -0,0 +1,44 @@
from django.test import SimpleTestCase
from apps.epg.tasks import (
_CHANNEL_PARSE_PROGRESS_CAP,
_CHANNEL_PARSE_PROGRESS_START,
_channel_parse_progress,
)
class ChannelParseProgressTests(SimpleTestCase):
def test_exceeding_estimate_recalculates_instead_of_going_over_cap(self):
"""101/100 used to yield ~90%+; now bumps estimate to 101 so ratio stays <= 1."""
progress, estimate = _channel_parse_progress(100, 100, had_db_baseline=True)
self.assertEqual(estimate, 100)
self.assertLessEqual(progress, _CHANNEL_PARSE_PROGRESS_CAP)
progress, estimate = _channel_parse_progress(101, 100, had_db_baseline=True)
self.assertEqual(estimate, 101)
self.assertLessEqual(progress, _CHANNEL_PARSE_PROGRESS_CAP)
self.assertGreater(progress, 90)
def test_large_xml_growth_never_exceeds_cap(self):
"""Previously 425 channels vs DB estimate of 100 could show ~300%."""
estimate = 100
for processed in (100, 200, 300, 425):
progress, estimate = _channel_parse_progress(
processed, estimate, had_db_baseline=True
)
self.assertLessEqual(progress, _CHANNEL_PARSE_PROGRESS_CAP)
def test_first_import_crawls_instead_of_flat_ninety(self):
progress, estimate = _channel_parse_progress(100, 0, had_db_baseline=False)
self.assertEqual(estimate, 0)
self.assertEqual(progress, _CHANNEL_PARSE_PROGRESS_START + 1)
self.assertLess(progress, 90)
progress, _ = _channel_parse_progress(5000, 0, had_db_baseline=False)
self.assertLess(progress, _CHANNEL_PARSE_PROGRESS_CAP)
def test_partial_progress_when_below_estimate(self):
progress, estimate = _channel_parse_progress(50, 100, had_db_baseline=True)
self.assertEqual(estimate, 100)
self.assertGreater(progress, _CHANNEL_PARSE_PROGRESS_START)
self.assertLess(progress, _CHANNEL_PARSE_PROGRESS_CAP)

50
apps/epg/tests/test_programme_index.py
View file

@ -1,7 +1,7 @@
import os
import gzip
import tempfile
from unittest.mock import patch, MagicMock
from unittest.mock import patch
from django.test import TestCase
from django.utils import timezone
@ -11,6 +11,7 @@ from apps.epg.tasks import (
find_current_program_for_tvg_id,
build_programme_index,
build_programme_index_task,
_EPG_TVG_PARSE_DEFER_SECONDS,
)
FIXTURE_DIR = os.path.join(os.path.dirname(__file__), "fixtures")
@ -519,7 +520,6 @@ class FindCurrentProgramTests(TestCase):
name="No Index",
source_type="xmltv",
file_path=FIXTURE_XML,
programme_index=None,
)
epg = EPGData.objects.create(
tvg_id="channel.current",
@ -664,38 +664,40 @@ class BuildProgrammeIndexTests(TestCase):
build_programme_index(99999)
@patch("apps.epg.tasks.build_programme_index")
def test_task_builds_and_releases_lock_when_free(self, mock_build):
mock_redis = MagicMock()
mock_redis.set.return_value = True # lock acquired
with patch("core.utils.RedisClient.get_client", return_value=mock_redis):
build_programme_index_task(42)
@patch("apps.epg.tasks.release_task_lock")
@patch("apps.epg.tasks.acquire_task_lock", return_value=True)
def test_task_builds_and_releases_lock_when_free(
self, mock_acquire, mock_release, mock_build
):
build_programme_index_task(42)
mock_build.assert_called_once_with(42)
mock_redis.set.assert_called_once()
self.assertEqual(
mock_redis.set.call_args.args[0], "building_programme_index_42"
)
mock_redis.delete.assert_called_once_with("building_programme_index_42")
mock_acquire.assert_called_once_with("epg_source_file", 42)
mock_release.assert_called_once_with("epg_source_file", 42)
@patch("apps.epg.tasks.build_programme_index")
def test_task_skips_when_lock_held(self, mock_build):
mock_redis = MagicMock()
mock_redis.set.return_value = False # another build in flight
with patch("core.utils.RedisClient.get_client", return_value=mock_redis):
@patch("apps.epg.tasks.acquire_task_lock", return_value=False)
def test_task_defers_when_lock_held(self, mock_acquire, mock_build):
with patch("apps.epg.tasks.build_programme_index_task.apply_async") as mock_apply:
build_programme_index_task(42)
mock_build.assert_not_called()
mock_redis.delete.assert_not_called()
mock_apply.assert_called_once_with(
args=[42],
kwargs={'_defer_retry': 1},
countdown=_EPG_TVG_PARSE_DEFER_SECONDS,
)
@patch("apps.epg.tasks.build_programme_index", side_effect=RuntimeError("boom"))
def test_task_releases_lock_on_failure(self, mock_build):
mock_redis = MagicMock()
mock_redis.set.return_value = True
with patch("core.utils.RedisClient.get_client", return_value=mock_redis):
with self.assertRaises(RuntimeError):
build_programme_index_task(42)
@patch("apps.epg.tasks.release_task_lock")
@patch("apps.epg.tasks.acquire_task_lock", return_value=True)
def test_task_releases_lock_on_failure(
self, mock_acquire, mock_release, mock_build
):
with self.assertRaises(RuntimeError):
build_programme_index_task(42)
mock_redis.delete.assert_called_once_with("building_programme_index_42")
mock_release.assert_called_once_with("epg_source_file", 42)
def test_per_channel_interleaved_marking(self):
xml = (

106
apps/epg/tests/test_refresh_db_recovery.py Normal file
View file

@ -0,0 +1,106 @@
from unittest.mock import MagicMock, patch
from django.test import SimpleTestCase
from apps.epg.tasks import (
_db_query_with_retry,
_ensure_epg_refresh_terminal_status,
_get_epg_source,
_release_task_db_connection,
refresh_epg_data,
)
class DbQueryWithRetryTests(SimpleTestCase):
def test_retries_after_index_error_from_poisoned_connection(self):
fn = MagicMock(side_effect=[IndexError("list index out of range"), "ok"])
with patch(
"apps.epg.tasks._release_task_db_connection"
) as mock_release:
result = _db_query_with_retry(fn, label="test query")
self.assertEqual(result, "ok")
self.assertEqual(fn.call_count, 2)
mock_release.assert_called_once()
def test_raises_after_exhausting_retries(self):
fn = MagicMock(side_effect=IndexError("list index out of range"))
with patch("apps.epg.tasks._release_task_db_connection"):
with self.assertRaises(IndexError):
_db_query_with_retry(fn, label="test query", max_retries=2)
self.assertEqual(fn.call_count, 2)
class RefreshTaskDbStartupTests(SimpleTestCase):
@patch("apps.epg.tasks._ensure_epg_refresh_terminal_status")
@patch("apps.epg.tasks._refresh_epg_data_impl")
@patch("apps.epg.tasks.release_task_lock")
@patch("apps.epg.tasks.acquire_task_lock", return_value=True)
@patch("apps.epg.tasks.TaskLockRenewer")
@patch("apps.epg.tasks._release_task_db_connection")
def test_refresh_releases_db_connection_before_impl(
self,
mock_release,
_mock_renewer,
_mock_acquire,
_mock_release_lock,
mock_impl,
_mock_ensure_terminal,
):
call_order = []
def track_release():
call_order.append("release")
mock_release.side_effect = track_release
mock_impl.side_effect = lambda *_a, **_k: call_order.append("impl") or "done"
result = refresh_epg_data(42)
self.assertEqual(result, "done")
self.assertEqual(call_order[:2], ["release", "impl"])
@patch("apps.epg.tasks.EPGSource")
def test_get_epg_source_uses_retry_helper(self, mock_model):
mock_source = MagicMock(id=42)
mock_model.objects.get.return_value = mock_source
with patch("apps.epg.tasks._db_query_with_retry") as mock_retry:
mock_retry.side_effect = lambda fn, **_: fn()
source = _get_epg_source(42)
mock_retry.assert_called_once()
mock_model.objects.get.assert_called_once_with(id=42)
self.assertIs(source, mock_source)
class EnsureEpgTerminalStatusTests(SimpleTestCase):
@patch("apps.epg.tasks.send_epg_update")
@patch("apps.epg.tasks._release_task_db_connection")
def test_marks_stuck_fetching_as_error(self, _mock_release, mock_ws):
with patch("apps.epg.tasks.EPGSource") as mock_model:
mock_model.STATUS_ERROR = "error"
qs = MagicMock()
mock_model.objects.filter.return_value = qs
qs.values_list.return_value.first.return_value = "fetching"
_ensure_epg_refresh_terminal_status(7)
qs.update.assert_called_once()
mock_ws.assert_called_once()
@patch("apps.epg.tasks.send_epg_update")
@patch("apps.epg.tasks._release_task_db_connection")
def test_leaves_success_unchanged(self, _mock_release, mock_ws):
with patch("apps.epg.tasks.EPGSource") as mock_model:
qs = MagicMock()
mock_model.objects.filter.return_value = qs
qs.values_list.return_value.first.return_value = "success"
_ensure_epg_refresh_terminal_status(7)
qs.update.assert_not_called()
mock_ws.assert_not_called()

870
apps/epg/tests/test_schedules_direct.py
View file

@ -7,19 +7,27 @@ Covers:
- fetch_schedules_direct: credential validation
- fetch_schedules_direct: SHA1 password hashing and token exchange
- fetch_schedules_direct: graceful error handling on auth failure
- fetch_schedules_direct: schedule MD5 delta, backfill, and cache invalidation
- parse_schedules_direct_time: correct UTC parsing
- EPG signals: SD sources skip the XMLTV program parser
- fetch_sd_guide_for_epg: per-channel guide fetch on map
- EPG signals: SD sources queue guide fetch when a channel is mapped
"""
import hashlib
from datetime import datetime
from datetime import date, datetime, timedelta, timezone as dt_timezone
from unittest.mock import MagicMock, patch
from django.test import TestCase
from django.utils import timezone
from apps.epg.models import EPGSource, EPGData
from apps.channels.models import Channel
from apps.epg.models import EPGSource, EPGData, ProgramData, SDScheduleMD5
from apps.epg.serializers import EPGSourceSerializer
from apps.epg.tasks import (
_sd_backfill_schedule_dates_without_data,
_sd_compute_schedule_changes_from_md5,
_sd_programs_needing_metadata,
)
# ---------------------------------------------------------------------------
@ -270,6 +278,469 @@ class FetchSchedulesDirectStationsOnlyTests(TestCase):
self.assertEqual(complete_call[1]['channels_count'], 1)
# ---------------------------------------------------------------------------
# Schedule MD5 delta, backfill, and cache tests
# ---------------------------------------------------------------------------
class SDScheduleMd5DeltaTests(TestCase):
"""Pure-function tests for schedule MD5 comparison."""
DATE_LIST = ['2026-06-11', '2026-06-12', '2026-06-13']
def test_detects_changed_md5(self):
server_md5s = {
('10001', '2026-06-11'): {'md5': 'abc', 'last_modified': ''},
('10001', '2026-06-12'): {'md5': 'def', 'last_modified': ''},
}
cached_md5s = {
('10001', '2026-06-11'): 'abc',
('10001', '2026-06-12'): 'old',
}
changed = _sd_compute_schedule_changes_from_md5(
server_md5s, cached_md5s, self.DATE_LIST,
)
self.assertEqual(changed['10001'], ['2026-06-12'])
def test_missing_cache_treated_as_changed(self):
server_md5s = {
('10001', '2026-06-11'): {'md5': 'abc', 'last_modified': ''},
}
changed = _sd_compute_schedule_changes_from_md5(server_md5s, {}, self.DATE_LIST)
self.assertEqual(changed['10001'], ['2026-06-11'])
def test_ignores_dates_outside_fetch_window(self):
server_md5s = {
('10001', '2026-06-01'): {'md5': 'abc', 'last_modified': ''},
}
changed = _sd_compute_schedule_changes_from_md5(server_md5s, {}, self.DATE_LIST)
self.assertEqual(changed, {})
class SDScheduleBackfillTests(TestCase):
"""Backfill must fix stale-cache gaps without re-fetching empty cached days."""
DATE_LIST = ['2026-06-11', '2026-06-12', '2026-06-13']
EPG_ID = 42
EPG_ID_MAP = {'10001': 42}
def _server_md5s(self):
return {
(sid, ds): {'md5': 'hash', 'last_modified': ''}
for sid in ('10001', '10002')
for ds in self.DATE_LIST
}
def test_backfills_when_no_cache_and_no_program_data(self):
changed = {}
count = _sd_backfill_schedule_dates_without_data(
changed,
self._server_md5s(),
self.DATE_LIST,
['10001'],
self.EPG_ID_MAP,
set(),
{},
{'10001'},
)
self.assertEqual(count, 3)
self.assertEqual(len(changed['10001']), 3)
def test_stale_cache_ignored_when_station_has_zero_program_data(self):
"""Newly mapped channel: stale MD5 cache must not block backfill."""
changed = {}
cached_md5s = {
(sid, ds): 'hash'
for sid in ('10001',)
for ds in self.DATE_LIST
}
count = _sd_backfill_schedule_dates_without_data(
changed,
self._server_md5s(),
self.DATE_LIST,
['10001'],
self.EPG_ID_MAP,
set(),
cached_md5s,
{'10001'},
)
self.assertEqual(count, 3)
self.assertEqual(len(changed['10001']), 3)
def test_skips_cached_empty_day_when_station_has_other_program_data(self):
"""Legitimately empty schedule day must not be re-fetched every refresh."""
changed = {}
cached_md5s = {('10001', '2026-06-12'): 'hash'}
dates_with_data = {(self.EPG_ID, date(2026, 6, 11))}
count = _sd_backfill_schedule_dates_without_data(
changed,
self._server_md5s(),
self.DATE_LIST,
['10001'],
self.EPG_ID_MAP,
dates_with_data,
cached_md5s,
set(),
)
self.assertEqual(count, 1)
self.assertEqual(changed['10001'], ['2026-06-13'])
def test_does_not_duplicate_dates_already_marked_changed(self):
changed = {'10001': ['2026-06-11']}
count = _sd_backfill_schedule_dates_without_data(
changed,
self._server_md5s(),
self.DATE_LIST,
['10001'],
self.EPG_ID_MAP,
set(),
{},
{'10001'},
)
self.assertEqual(count, 2)
self.assertEqual(sorted(changed['10001']), self.DATE_LIST)
class SDProgramMetadataDeltaTests(TestCase):
def test_fetches_when_md5_changed(self):
needed = _sd_programs_needing_metadata(
{'EP0001'},
{'EP0001': 'new'},
{'EP0001': 'old'},
{'EP0001'},
)
self.assertEqual(needed, {'EP0001'})
def test_fetches_when_no_local_program_data(self):
needed = _sd_programs_needing_metadata(
{'EP0001', 'EP0002'},
{'EP0001': 'same', 'EP0002': 'same'},
{'EP0001': 'same', 'EP0002': 'same'},
{'EP0001'},
)
self.assertEqual(needed, {'EP0002'})
def test_skips_when_md5_matches_and_program_data_exists(self):
needed = _sd_programs_needing_metadata(
{'EP0001'},
{'EP0001': 'same'},
{'EP0001': 'same'},
{'EP0001'},
)
self.assertEqual(needed, set())
class SDScheduleDeltaIntegrationTests(TestCase):
"""DB-backed tests for cache pruning and mapped-only MD5 API calls."""
MAPPED_STATION = '10001'
UNMAPPED_STATION = '10002'
def _make_sd_source(self):
return EPGSource.objects.create(
name='SD Integration',
source_type='schedules_direct',
username='sduser',
password='sdpass',
)
def _lineup_get_side_effect(self, url, **kwargs):
if url.endswith('/status'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={'systemStatus': [{'status': 'Online'}]}),
)
if url.endswith('/lineups'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={'lineups': [{'lineupID': 'USA-TEST-X'}]}),
)
if '/lineups/USA-TEST-X' in url:
return MagicMock(
status_code=200,
json=MagicMock(return_value={
'stations': [
{
'stationID': self.MAPPED_STATION,
'name': 'Mapped Station',
'callsign': 'MAP',
},
{
'stationID': self.UNMAPPED_STATION,
'name': 'Unmapped Station',
'callsign': 'UNM',
},
],
}),
)
raise AssertionError(f'Unexpected GET URL: {url}')
def _build_date_list(self, days=3):
today = date.today()
return [(today + timedelta(days=i)).strftime('%Y-%m-%d') for i in range(days)]
def _seed_full_window_program_data(self, epg, days=3):
today = date.today()
for i in range(days):
day = today + timedelta(days=i)
start = datetime(day.year, day.month, day.day, 12, 0, tzinfo=dt_timezone.utc)
ProgramData.objects.create(
epg=epg,
start_time=start,
end_time=start + timedelta(hours=1),
title='Show',
tvg_id=epg.tvg_id,
)
@patch('apps.epg.tasks.SD_DAYS_TO_FETCH', 3)
@patch('apps.epg.tasks.send_epg_update')
@patch('apps.epg.tasks.requests.get')
@patch('apps.epg.tasks.requests.post')
def test_md5_api_only_requests_mapped_stations(
self, mock_post, mock_get, mock_send_epg_update,
):
from apps.epg.tasks import fetch_schedules_direct
source = self._make_sd_source()
mapped_epg = EPGData.objects.create(
tvg_id=self.MAPPED_STATION,
name='Mapped',
epg_source=source,
)
Channel.objects.create(name='Mapped Ch', epg_data=mapped_epg)
date_list = self._build_date_list(3)
self._seed_full_window_program_data(mapped_epg, days=3)
today = date.today()
for i, ds in enumerate(date_list):
SDScheduleMD5.objects.create(
epg_source=source,
station_id=self.MAPPED_STATION,
date=today + timedelta(days=i),
md5=f'md5-{ds}',
last_modified=timezone.now(),
)
SDScheduleMD5.objects.create(
epg_source=source,
station_id=self.UNMAPPED_STATION,
date=today,
md5='unmapped-stale',
last_modified=timezone.now(),
)
mock_get.side_effect = self._lineup_get_side_effect
md5_response_payload = {
self.MAPPED_STATION: {
ds: {'code': 0, 'md5': f'md5-{ds}', 'lastModified': '2026-06-11T00:00:00Z'}
for ds in date_list
},
}
def post_side_effect(url, **kwargs):
if url.endswith('/token'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={'code': 0, 'token': 'tok'}),
)
if url.endswith('/schedules/md5'):
return MagicMock(
status_code=200,
json=MagicMock(return_value=md5_response_payload),
)
raise AssertionError(f'Unexpected POST URL: {url}')
mock_post.side_effect = post_side_effect
fetch_schedules_direct(source, force=True)
md5_calls = [c for c in mock_post.call_args_list if c[0][0].endswith('/schedules/md5')]
self.assertEqual(len(md5_calls), 1)
request_body = md5_calls[0][1]['json']
station_ids_in_request = {entry['stationID'] for entry in request_body}
self.assertEqual(station_ids_in_request, {self.MAPPED_STATION})
self.assertFalse(
SDScheduleMD5.objects.filter(
epg_source=source,
station_id=self.UNMAPPED_STATION,
).exists()
)
schedule_calls = [c for c in mock_post.call_args_list if c[0][0].endswith('/schedules')]
self.assertEqual(len(schedule_calls), 0)
source.refresh_from_db()
self.assertEqual(source.status, EPGSource.STATUS_SUCCESS)
@patch('apps.epg.tasks.SD_DAYS_TO_FETCH', 3)
@patch('apps.epg.tasks.send_epg_update')
@patch('apps.epg.tasks.requests.get')
@patch('apps.epg.tasks.requests.post')
def test_newly_mapped_station_fetches_despite_stale_cache(
self, mock_post, mock_get, mock_send_epg_update,
):
from apps.epg.tasks import fetch_schedules_direct
source = self._make_sd_source()
mapped_epg = EPGData.objects.create(
tvg_id=self.MAPPED_STATION,
name='Mapped',
epg_source=source,
)
Channel.objects.create(name='Mapped Ch', epg_data=mapped_epg)
date_list = self._build_date_list(3)
today = date.today()
for i, ds in enumerate(date_list):
SDScheduleMD5.objects.create(
epg_source=source,
station_id=self.MAPPED_STATION,
date=today + timedelta(days=i),
md5=f'md5-{ds}',
last_modified=timezone.now(),
)
mock_get.side_effect = self._lineup_get_side_effect
schedule_payload = [{
'stationID': self.MAPPED_STATION,
'metadata': {'startDate': date_list[0], 'md5': 'md5-' + date_list[0], 'modified': '2026-06-11T00:00:00Z'},
'programs': [{
'programID': 'EP000000000001',
'airDateTime': f'{date_list[0]}T12:00:00Z',
'duration': 3600,
'md5': 'prog-md5-1',
}],
}]
program_payload = [{
'programID': 'EP000000000001',
'titles': [{'title120': 'Test Show'}],
}]
def post_side_effect(url, **kwargs):
if url.endswith('/token'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={'code': 0, 'token': 'tok'}),
)
if url.endswith('/schedules/md5'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={
self.MAPPED_STATION: {
ds: {'code': 0, 'md5': f'md5-{ds}', 'lastModified': '2026-06-11T00:00:00Z'}
for ds in date_list
},
}),
)
if url.endswith('/schedules'):
return MagicMock(status_code=200, json=MagicMock(return_value=schedule_payload))
if url.endswith('/programs'):
return MagicMock(status_code=200, json=MagicMock(return_value=program_payload))
raise AssertionError(f'Unexpected POST URL: {url}')
mock_post.side_effect = post_side_effect
fetch_schedules_direct(source, force=True)
schedule_calls = [c for c in mock_post.call_args_list if c[0][0].endswith('/schedules')]
self.assertGreaterEqual(len(schedule_calls), 1)
self.assertEqual(
ProgramData.objects.filter(epg=mapped_epg).count(),
1,
)
@patch('apps.epg.tasks.SD_DAYS_TO_FETCH', 3)
@patch('apps.epg.tasks.send_epg_update')
@patch('apps.epg.tasks.requests.get')
@patch('apps.epg.tasks.requests.post')
def test_orphan_program_data_removed_on_post_refresh(
self, mock_post, mock_get, mock_send_epg_update,
):
from apps.epg.tasks import fetch_schedules_direct
source = self._make_sd_source()
mapped_epg = EPGData.objects.create(
tvg_id=self.MAPPED_STATION,
name='Mapped',
epg_source=source,
)
orphan_epg = EPGData.objects.create(
tvg_id=self.UNMAPPED_STATION,
name='Orphan',
epg_source=source,
)
Channel.objects.create(name='Mapped Ch', epg_data=mapped_epg)
date_list = self._build_date_list(3)
self._seed_full_window_program_data(mapped_epg, days=3)
start = timezone.now()
ProgramData.objects.create(
epg=orphan_epg,
start_time=start,
end_time=start + timedelta(hours=1),
title='Orphan Show',
tvg_id=orphan_epg.tvg_id,
)
today = date.today()
for i, ds in enumerate(date_list):
SDScheduleMD5.objects.create(
epg_source=source,
station_id=self.MAPPED_STATION,
date=today + timedelta(days=i),
md5=f'md5-{ds}',
last_modified=timezone.now(),
)
mock_get.side_effect = self._lineup_get_side_effect
def post_side_effect(url, **kwargs):
if url.endswith('/token'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={'code': 0, 'token': 'tok'}),
)
if url.endswith('/schedules/md5'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={
self.MAPPED_STATION: {
ds: {'code': 0, 'md5': f'md5-{ds}', 'lastModified': '2026-06-11T00:00:00Z'}
for ds in date_list
},
}),
)
raise AssertionError(f'Unexpected POST URL: {url}')
mock_post.side_effect = post_side_effect
fetch_schedules_direct(source, force=True)
self.assertFalse(ProgramData.objects.filter(epg=orphan_epg).exists())
def test_stale_program_md5_fetched_when_no_program_data(self):
programs_with_data = set()
needed = _sd_programs_needing_metadata(
{'EP0001'},
{'EP0001': 'cached-md5'},
{'EP0001': 'cached-md5'},
programs_with_data,
)
self.assertEqual(needed, {'EP0001'})
def test_shared_program_skips_metadata_when_cached(self):
needed = _sd_programs_needing_metadata(
{'EP0001'},
{'EP0001': 'cached-md5'},
{'EP0001': 'cached-md5'},
{'EP0001'},
)
self.assertEqual(needed, set())
# ---------------------------------------------------------------------------
# parse_schedules_direct_time tests
# ---------------------------------------------------------------------------
@ -298,17 +769,400 @@ class ParseSchedulesDirectTimeTests(TestCase):
parse_schedules_direct_time('not-a-timestamp')
# ---------------------------------------------------------------------------
# dispatch_program_refresh_for_epg_ids tests
# ---------------------------------------------------------------------------
class SDDispatchProgramRefreshTests(TestCase):
"""Bulk SD assignment should batch guide fetches above the threshold."""
STATION = '10001'
def _make_sd_source(self):
return EPGSource.objects.create(
name='SD Dispatch Test',
source_type='schedules_direct',
username='sduser',
password='sdpass',
)
def _make_xml_source(self):
return EPGSource.objects.create(
name='XML Dispatch Test',
source_type='xmltv',
url='http://example.com/epg.xml',
)
@patch('apps.epg.tasks.fetch_sd_mapped_guide_batch.delay')
@patch('apps.epg.tasks.parse_programs_for_tvg_id.delay')
def test_xmltv_still_uses_parse_programs_per_id(
self, mock_parse_delay, mock_batch_delay,
):
from apps.epg.tasks import dispatch_program_refresh_for_epg_ids
xml_source = self._make_xml_source()
epg = EPGData.objects.create(
tvg_id='xml-1',
name='XML Channel',
epg_source=xml_source,
)
count = dispatch_program_refresh_for_epg_ids({epg.id})
self.assertEqual(count, 1)
mock_parse_delay.assert_called_once_with(epg.id)
mock_batch_delay.assert_not_called()
@patch('apps.epg.tasks.fetch_sd_mapped_guide_batch.delay')
@patch('apps.epg.tasks.parse_programs_for_tvg_id.delay')
def test_sd_below_threshold_uses_per_epg_tasks(
self, mock_parse_delay, mock_batch_delay,
):
from apps.epg.tasks import dispatch_program_refresh_for_epg_ids
source = self._make_sd_source()
epgs = [
EPGData.objects.create(
tvg_id=f'{self.STATION}{i}',
name=f'Station {i}',
epg_source=source,
)
for i in range(2)
]
count = dispatch_program_refresh_for_epg_ids({e.id for e in epgs})
self.assertEqual(count, 2)
self.assertEqual(mock_parse_delay.call_count, 2)
mock_batch_delay.assert_not_called()
@patch('apps.epg.tasks.fetch_sd_mapped_guide_batch.delay')
@patch('apps.epg.tasks.parse_programs_for_tvg_id.delay')
def test_sd_at_threshold_uses_batched_fetch(
self, mock_parse_delay, mock_batch_delay,
):
from apps.epg.tasks import dispatch_program_refresh_for_epg_ids
source = self._make_sd_source()
epgs = [
EPGData.objects.create(
tvg_id=f'{self.STATION}{i}',
name=f'Station {i}',
epg_source=source,
)
for i in range(3)
]
count = dispatch_program_refresh_for_epg_ids({e.id for e in epgs})
self.assertEqual(count, 1)
mock_batch_delay.assert_called_once_with(source.id)
mock_parse_delay.assert_not_called()
@patch('apps.epg.tasks.fetch_sd_mapped_guide_batch.delay')
@patch('apps.epg.tasks.parse_programs_for_tvg_id.delay')
def test_sd_skips_when_program_data_exists(
self, mock_parse_delay, mock_batch_delay,
):
from apps.epg.tasks import dispatch_program_refresh_for_epg_ids
source = self._make_sd_source()
epg = EPGData.objects.create(
tvg_id=self.STATION,
name='Has Data',
epg_source=source,
)
start = timezone.now()
ProgramData.objects.create(
epg=epg,
start_time=start,
end_time=start + timedelta(hours=1),
title='Show',
tvg_id=epg.tvg_id,
)
count = dispatch_program_refresh_for_epg_ids({epg.id})
self.assertEqual(count, 0)
mock_parse_delay.assert_not_called()
mock_batch_delay.assert_not_called()
class SDGuideFetchCoordinationTests(TestCase):
"""Batch and single-EPG SD fetches coordinate via locks and deferred retries."""
STATION = '10001'
def _make_sd_source(self):
return EPGSource.objects.create(
name='SD Coordination',
source_type='schedules_direct',
username='sduser',
password='sdpass',
)
@patch('apps.epg.tasks.fetch_schedules_direct')
@patch('apps.epg.tasks.acquire_task_lock', return_value=False)
@patch('apps.epg.tasks.fetch_sd_mapped_guide_batch.apply_async')
def test_batch_fetch_defers_when_lock_held(
self, mock_apply_async, mock_acquire, mock_fetch,
):
from apps.epg.tasks import (
fetch_sd_mapped_guide_batch,
SD_MAPPED_GUIDE_BATCH_DEFER_SECONDS,
)
source = self._make_sd_source()
result = fetch_sd_mapped_guide_batch(source.id)
self.assertEqual(result, 'Deferred - batch already in progress')
mock_apply_async.assert_called_once_with(
args=[source.id],
kwargs={'force': False, '_defer_retry': 1},
countdown=SD_MAPPED_GUIDE_BATCH_DEFER_SECONDS,
)
mock_fetch.assert_not_called()
@patch('apps.epg.tasks.fetch_schedules_direct')
@patch('apps.epg.tasks.acquire_task_lock', return_value=False)
@patch('apps.epg.tasks.fetch_sd_mapped_guide_batch.apply_async')
def test_batch_fetch_stops_after_max_defer_retries(
self, mock_apply_async, mock_acquire, mock_fetch,
):
from apps.epg.tasks import fetch_sd_mapped_guide_batch
source = self._make_sd_source()
result = fetch_sd_mapped_guide_batch(source.id, _defer_retry=2)
self.assertEqual(result, 'Task already running')
mock_apply_async.assert_not_called()
mock_fetch.assert_not_called()
@patch('apps.epg.tasks.fetch_schedules_direct')
@patch('apps.epg.tasks.acquire_task_lock', return_value=True)
@patch('apps.epg.tasks.release_task_lock')
@patch('apps.epg.tasks.TaskLockRenewer')
@patch('apps.epg.tasks.is_task_lock_held', return_value=True)
@patch('apps.epg.tasks.fetch_sd_guide_for_epg.apply_async')
def test_single_epg_defers_while_batch_running(
self, mock_apply_async, mock_batch_held, mock_renewer,
mock_release, mock_acquire, mock_fetch,
):
from apps.epg.tasks import (
fetch_sd_guide_for_epg,
SD_MAPPED_GUIDE_BATCH_DEFER_SECONDS,
)
source = self._make_sd_source()
epg = EPGData.objects.create(
tvg_id=self.STATION,
name='Deferred Station',
epg_source=source,
)
result = fetch_sd_guide_for_epg(epg.id)
self.assertEqual(result, 'Deferred - mapped batch in progress')
mock_apply_async.assert_called_once_with(
args=[epg.id],
kwargs={'force': False, '_defer_retry': 1},
countdown=SD_MAPPED_GUIDE_BATCH_DEFER_SECONDS,
)
mock_fetch.assert_not_called()
mock_acquire.assert_not_called()
@patch('apps.epg.tasks.fetch_schedules_direct')
@patch('apps.epg.tasks.acquire_task_lock', return_value=True)
@patch('apps.epg.tasks.release_task_lock')
@patch('apps.epg.tasks.TaskLockRenewer')
@patch('apps.epg.tasks.is_task_lock_held', return_value=True)
@patch('apps.epg.tasks.fetch_sd_guide_for_epg.apply_async')
def test_single_epg_proceeds_after_max_batch_deferrals(
self, mock_apply_async, mock_batch_held, mock_renewer,
mock_release, mock_acquire, mock_fetch,
):
from apps.epg.tasks import fetch_sd_guide_for_epg
source = self._make_sd_source()
epg = EPGData.objects.create(
tvg_id=self.STATION,
name='Fallback Station',
epg_source=source,
)
result = fetch_sd_guide_for_epg(epg.id, _defer_retry=2)
self.assertEqual(result, 'SD guide fetch complete')
mock_apply_async.assert_not_called()
mock_fetch.assert_called_once()
mock_acquire.assert_called_once_with('parse_epg_programs', epg.id)
# ---------------------------------------------------------------------------
# Signal tests
# ---------------------------------------------------------------------------
class SDSingleEpgFetchTests(TestCase):
"""Per-channel SD guide fetch on map (epg_id_only path)."""
MAPPED_STATION = '10001'
def _make_sd_source(self, updated_at=None):
source = EPGSource.objects.create(
name='SD Single EPG',
source_type='schedules_direct',
username='sduser',
password='sdpass',
)
if updated_at is not None:
EPGSource.objects.filter(id=source.id).update(updated_at=updated_at)
source.refresh_from_db()
return source
def _lineup_get_side_effect(self, url, **kwargs):
if url.endswith('/status'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={'systemStatus': [{'status': 'Online'}]}),
)
if url.endswith('/lineups'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={'lineups': [{'lineupID': 'USA-TEST-X'}]}),
)
raise AssertionError(f'Unexpected GET URL: {url}')
@patch('apps.epg.tasks.acquire_task_lock', return_value=True)
@patch('apps.epg.tasks.release_task_lock')
@patch('apps.epg.tasks.TaskLockRenewer')
def test_fetch_sd_guide_skips_when_program_data_exists(
self, mock_renewer, mock_release, mock_acquire,
):
from apps.epg.tasks import fetch_sd_guide_for_epg
source = self._make_sd_source()
epg = EPGData.objects.create(
tvg_id=self.MAPPED_STATION,
name='Mapped',
epg_source=source,
)
start = timezone.now()
ProgramData.objects.create(
epg=epg,
start_time=start,
end_time=start + timedelta(hours=1),
title='Existing',
tvg_id=epg.tvg_id,
)
with patch('apps.epg.tasks.fetch_schedules_direct') as mock_fetch:
result = fetch_sd_guide_for_epg(epg.id)
self.assertEqual(result, 'Guide data already present')
mock_fetch.assert_not_called()
@patch('apps.epg.tasks.acquire_task_lock', return_value=True)
@patch('apps.epg.tasks.release_task_lock')
@patch('apps.epg.tasks.TaskLockRenewer')
def test_parse_programs_for_tvg_id_delegates_to_sd_fetch(
self, mock_renewer, mock_release, mock_acquire,
):
from apps.epg.tasks import parse_programs_for_tvg_id
source = self._make_sd_source()
epg = EPGData.objects.create(
tvg_id=self.MAPPED_STATION,
name='Mapped',
epg_source=source,
)
with patch('apps.epg.tasks.fetch_sd_guide_for_epg', return_value='SD guide fetch complete') as mock_sd:
result = parse_programs_for_tvg_id(epg.id)
mock_sd.assert_called_once_with(epg.id, force=False)
self.assertEqual(result, 'SD guide fetch complete')
@patch('apps.epg.tasks.SD_DAYS_TO_FETCH', 3)
@patch('apps.epg.tasks.send_epg_update')
@patch('apps.epg.tasks.requests.get')
@patch('apps.epg.tasks.requests.post')
def test_single_epg_fetch_skips_lineup_sync_and_updated_at(
self, mock_post, mock_get, mock_send_epg_update,
):
from apps.epg.tasks import fetch_schedules_direct
prior_updated = timezone.now() - timedelta(hours=1)
source = self._make_sd_source(updated_at=prior_updated)
epg = EPGData.objects.create(
tvg_id=self.MAPPED_STATION,
name='Mapped',
epg_source=source,
)
Channel.objects.create(name='Mapped Ch', epg_data=epg)
date_list = [
(date.today() + timedelta(days=i)).strftime('%Y-%m-%d')
for i in range(3)
]
mock_get.side_effect = self._lineup_get_side_effect
schedule_payload = [{
'stationID': self.MAPPED_STATION,
'metadata': {'startDate': date_list[0], 'md5': 'md5-new', 'modified': '2026-06-11T00:00:00Z'},
'programs': [{
'programID': 'EP000000000001',
'airDateTime': f'{date_list[0]}T12:00:00Z',
'duration': 3600,
'md5': 'prog-md5-1',
}],
}]
program_payload = [{
'programID': 'EP000000000001',
'titles': [{'title120': 'Test Show'}],
}]
def post_side_effect(url, **kwargs):
if url.endswith('/token'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={'code': 0, 'token': 'tok'}),
)
if url.endswith('/schedules/md5'):
return MagicMock(
status_code=200,
json=MagicMock(return_value={
self.MAPPED_STATION: {
ds: {'code': 0, 'md5': f'md5-{ds}', 'lastModified': '2026-06-11T00:00:00Z'}
for ds in date_list
},
}),
)
if url.endswith('/schedules'):
return MagicMock(status_code=200, json=MagicMock(return_value=schedule_payload))
if url.endswith('/programs'):
return MagicMock(status_code=200, json=MagicMock(return_value=program_payload))
raise AssertionError(f'Unexpected POST URL: {url}')
mock_post.side_effect = post_side_effect
fetch_schedules_direct(source, epg_id_only=epg.id)
lineup_detail_calls = [
c for c in mock_get.call_args_list
if '/lineups/' in c[0][0] and not c[0][0].endswith('/lineups')
]
self.assertEqual(lineup_detail_calls, [])
source.refresh_from_db()
self.assertEqual(source.updated_at, prior_updated)
self.assertEqual(ProgramData.objects.filter(epg=epg).count(), 1)
class SDSourceSignalTests(TestCase):
"""SD EPG sources must skip the XMLTV program parser signal."""
"""SD EPG sources queue per-EPG guide fetch when a channel is mapped."""
@patch('apps.channels.signals.parse_programs_for_tvg_id')
def test_sd_source_skips_xmltv_parse_on_channel_create(self, mock_parse):
"""Creating a channel linked to an SD EPG source must not trigger
the XMLTV program parser — SD data is handled by fetch_schedules_direct."""
def test_sd_source_queues_guide_fetch_on_channel_create(self, mock_parse):
from apps.epg.models import EPGData
from apps.channels.models import Channel
@ -329,7 +1183,7 @@ class SDSourceSignalTests(TestCase):
epg_data=epg_data,
)
mock_parse.delay.assert_not_called()
mock_parse.delay.assert_called_once_with(epg_data.id)
# ---------------------------------------------------------------------------

4
apps/m3u/api_urls.py
View file

@ -16,12 +16,12 @@ app_name = "m3u"
router = DefaultRouter()
router.register(r"accounts", M3UAccountViewSet, basename="m3u-account")
router.register(
r"accounts\/(?P<account_id>\d+)\/profiles",
r"accounts/(?P<account_id>\d+)/profiles",
M3UAccountProfileViewSet,
basename="m3u-account-profiles",
)
router.register(
r"accounts\/(?P<account_id>\d+)\/filters",
r"accounts/(?P<account_id>\d+)/filters",
M3UFilterViewSet,
basename="m3u-filters",
)

10
apps/m3u/api_views.py
View file

@ -23,7 +23,7 @@ logger = logging.getLogger(__name__)
from .models import M3UAccount, M3UFilter, ServerGroup, M3UAccountProfile
from core.models import UserAgent
from core.utils import safe_upload_path
from core.utils import safe_upload_path, ensure_custom_properties_dict
from apps.channels.models import ChannelGroupM3UAccount
from core.serializers import UserAgentSerializer
from apps.vod.models import M3UVODCategoryRelation
@ -536,7 +536,9 @@ class M3UAccountViewSet(viewsets.ModelViewSet):
auto_channel_sync=setting.get("auto_channel_sync", False),
auto_sync_channel_start=setting.get("auto_sync_channel_start"),
auto_sync_channel_end=setting.get("auto_sync_channel_end"),
custom_properties=setting.get("custom_properties", {}),
custom_properties=ensure_custom_properties_dict(
setting.get("custom_properties")
),
)
for setting in group_settings
if setting.get("channel_group")
@ -561,7 +563,9 @@ class M3UAccountViewSet(viewsets.ModelViewSet):
category_id=setting["id"],
m3u_account=account,
enabled=setting.get("enabled", True),
custom_properties=setting.get("custom_properties", {}),
custom_properties=ensure_custom_properties_dict(
setting.get("custom_properties")
),
)
for setting in category_settings
if setting.get("id")

8
apps/m3u/forms.py
View file

@ -1,6 +1,7 @@
# apps/m3u/forms.py
from django import forms
from .models import M3UAccount, M3UFilter
from core.utils import ensure_custom_properties_dict
import re
class M3UAccountForm(forms.ModelForm):
@ -28,7 +29,9 @@ class M3UAccountForm(forms.ModelForm):
# Set initial value for enable_vod from custom_properties
if self.instance and self.instance.custom_properties:
custom_props = self.instance.custom_properties or {}
custom_props = self.instance.custom_properties
if not isinstance(custom_props, dict):
custom_props = ensure_custom_properties_dict(custom_props)
self.fields['enable_vod'].initial = custom_props.get('enable_vod', False)
def save(self, commit=True):
@ -37,8 +40,9 @@ class M3UAccountForm(forms.ModelForm):
# Handle enable_vod field
enable_vod = self.cleaned_data.get('enable_vod', False)
# Parse existing custom_properties
custom_props = instance.custom_properties or {}
if not isinstance(custom_props, dict):
custom_props = ensure_custom_properties_dict(custom_props)
# Update VOD preference
custom_props['enable_vod'] = enable_vod

11
apps/m3u/models.py
View file

@ -7,6 +7,7 @@ from django.dispatch import receiver
from apps.channels.models import StreamProfile
from django_celery_beat.models import PeriodicTask
from core.models import CoreSettings, UserAgent
from core.utils import custom_properties_as_dict
CUSTOM_M3U_ACCOUNT_NAME = "custom"
@ -124,6 +125,11 @@ class M3UAccount(models.Model):
return user_agent
def save(self, *args, **kwargs):
if self.custom_properties is not None and not isinstance(
self.custom_properties, dict
):
self.custom_properties = custom_properties_as_dict(self.custom_properties)
# Prevent auto_now behavior by handling updated_at manually
if "update_fields" in kwargs and "updated_at" not in kwargs["update_fields"]:
# Don't modify updated_at for regular updates
@ -263,6 +269,11 @@ class M3UAccountProfile(models.Model):
def save(self, *args, **kwargs):
"""Auto-sync exp_date from custom_properties for XC accounts on every save.
For non-XC accounts, exp_date is set directly and left untouched here."""
if self.custom_properties is not None and not isinstance(
self.custom_properties, dict
):
self.custom_properties = custom_properties_as_dict(self.custom_properties)
parsed = self._parse_exp_date_from_custom_properties()
if parsed is not None:
# XC account with exp_date in custom_properties — always sync

20
apps/m3u/serializers.py
View file

@ -1,4 +1,4 @@
from core.utils import validate_flexible_url
from core.utils import validate_flexible_url, ensure_custom_properties_dict
from rest_framework import serializers, status
from rest_framework.response import Response
from .models import M3UAccount, M3UFilter, ServerGroup, M3UAccountProfile
@ -270,11 +270,15 @@ class M3UAccountSerializer(serializers.ModelSerializer):
# overwrite their corresponding keys; clients should set those via
# the typed top-level fields rather than the custom_properties
# payload.
incoming_custom = validated_data.get("custom_properties") or {}
custom_props = {
**(instance.custom_properties or {}),
**incoming_custom,
}
incoming_custom = {}
if "custom_properties" in validated_data:
incoming_custom = validated_data["custom_properties"] or {}
if not isinstance(incoming_custom, dict):
incoming_custom = ensure_custom_properties_dict(incoming_custom)
existing_custom = instance.custom_properties or {}
if not isinstance(existing_custom, dict):
existing_custom = ensure_custom_properties_dict(existing_custom)
custom_props = {**existing_custom, **incoming_custom}
if enable_vod is not None:
custom_props["enable_vod"] = enable_vod
@ -346,7 +350,9 @@ class M3UAccountSerializer(serializers.ModelSerializer):
auto_enable_new_groups_series = validated_data.pop("auto_enable_new_groups_series", True)
# Parse existing custom_properties or create new
custom_props = validated_data.get("custom_properties", {})
custom_props = validated_data.get("custom_properties") or {}
if not isinstance(custom_props, dict):
custom_props = ensure_custom_properties_dict(custom_props)
# Set preferences (default to True for auto_enable_new_groups)
custom_props["enable_vod"] = enable_vod

1028
apps/m3u/tasks.py
View file

File diff suppressed because it is too large Load diff

40
apps/m3u/tests/test_memory_cleanup.py
View file

@ -30,9 +30,47 @@ class ProcessM3UBatchCleanupTests(SimpleTestCase):
mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123")
with patch("django.db.connections") as mock_connections:
process_m3u_batch_direct(1, [], {}, ["name", "url"])
process_m3u_batch_direct(1, [], {}, ["name", "url"], compiled_filters=[])
mock_connections.close_all.assert_called()
@patch("apps.m3u.tasks.Stream")
@patch("apps.m3u.tasks.M3UAccount")
def test_batch_calls_gc_collect(self, mock_account_cls, mock_stream_cls):
"""gc.collect() must run after each batch so XC refresh threads release promptly."""
from apps.m3u.tasks import process_m3u_batch_direct
mock_account = MagicMock()
mock_account.filters.order_by.return_value = []
mock_account_cls.objects.get.return_value = mock_account
mock_stream_cls.objects.filter.return_value.select_related.return_value.only.return_value = (
[]
)
mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123")
with patch("gc.collect") as mock_gc, patch("django.db.connections"):
process_m3u_batch_direct(1, [], {}, ["name", "url"], compiled_filters=[])
mock_gc.assert_called()
@patch("apps.m3u.tasks.Stream")
@patch("apps.m3u.tasks.M3UAccount")
def test_precompiled_empty_filters_skip_db_lookup(
self, mock_account_cls, mock_stream_cls,
):
"""When filters are precompiled as empty, batch workers must not query filters."""
from apps.m3u.tasks import process_m3u_batch_direct
mock_account = MagicMock()
mock_account_cls.objects.get.return_value = mock_account
mock_stream_cls.objects.filter.return_value.select_related.return_value.only.return_value = (
[]
)
mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123")
with patch("django.db.connections"):
process_m3u_batch_direct(1, [], {}, ["name", "url"], compiled_filters=[])
mock_account.filters.order_by.assert_not_called()
class LockReleaseTests(SimpleTestCase):
"""Verify task lock is released on all exit paths."""

74
apps/m3u/tests/test_refresh_db_recovery.py Normal file
View file

@ -0,0 +1,74 @@
from unittest.mock import MagicMock, patch
from django.test import SimpleTestCase
from apps.m3u.tasks import (
_db_query_with_retry,
_get_active_m3u_account,
_release_task_db_connection,
refresh_single_m3u_account,
)
class DbQueryWithRetryTests(SimpleTestCase):
def test_retries_after_index_error_from_poisoned_connection(self):
fn = MagicMock(side_effect=[IndexError("list index out of range"), "ok"])
with patch(
"apps.m3u.tasks._release_task_db_connection"
) as mock_release:
result = _db_query_with_retry(fn, label="test query")
self.assertEqual(result, "ok")
self.assertEqual(fn.call_count, 2)
mock_release.assert_called_once()
def test_raises_after_exhausting_retries(self):
fn = MagicMock(side_effect=IndexError("list index out of range"))
with patch("apps.m3u.tasks._release_task_db_connection"):
with self.assertRaises(IndexError):
_db_query_with_retry(fn, label="test query", max_retries=2)
self.assertEqual(fn.call_count, 2)
class RefreshTaskDbStartupTests(SimpleTestCase):
@patch("apps.m3u.tasks._ensure_m3u_refresh_terminal_status")
@patch("apps.m3u.tasks._refresh_single_m3u_account_impl")
@patch("apps.m3u.tasks.release_task_lock")
@patch("apps.m3u.tasks.acquire_task_lock", return_value=True)
@patch("apps.m3u.tasks.TaskLockRenewer")
@patch("apps.m3u.tasks._release_task_db_connection")
def test_refresh_releases_db_connection_before_impl(
self,
mock_release,
_mock_renewer,
_mock_acquire,
_mock_release_lock,
mock_impl,
_mock_ensure_terminal,
):
call_order = []
def track_release():
call_order.append("release")
mock_release.side_effect = track_release
mock_impl.side_effect = lambda *_a, **_k: call_order.append("impl") or "done"
result = refresh_single_m3u_account(140)
self.assertEqual(result, "done")
self.assertEqual(call_order[:2], ["release", "impl"])
@patch("apps.m3u.tasks.M3UAccount")
def test_get_active_m3u_account_uses_retry_helper(self, mock_model):
mock_model.objects.get.return_value = MagicMock(is_active=True)
with patch("apps.m3u.tasks._db_query_with_retry") as mock_retry:
mock_retry.side_effect = lambda fn, **_: fn()
account = _get_active_m3u_account(140)
mock_retry.assert_called_once()
self.assertTrue(account.is_active)

212
apps/m3u/tests/test_rename_preview_parity.py Normal file
View file

@ -0,0 +1,212 @@
"""
Differential parity tests: the auto-sync rename preview must predict the exact
name the live rename produces, across a broad matrix of regex strategies.
Both the preview and the live rename compile with the third-party `regex`
module (for its JS-aligned syntax and per-call timeout). They can only be
trusted together if, for every find/replace a user might author, the preview's
predicted `after` equals the channel name the sync actually writes.
Each case is run end-to-end: real streams, the real sync_auto_channels rename,
and the real regex-preview endpoint, compared per original stream name.
"""
from django.test import TestCase
from django.utils import timezone
from apps.channels.models import Channel, ChannelGroup, ChannelStream, Stream
from apps.m3u.models import M3UAccount
from apps.m3u.tasks import sync_auto_channels
# Diverse names: distinct word-prefixes (collision-resistant), with quality
# tags, brackets, pipes, ampersands, extra whitespace, underscores, dots, CJK,
# and emoji, to exercise anchors, classes, boundaries, and Unicode handling.
NAMES = [
"Alpha Channel 11",
"Bravo Sports HD",
"Charlie News FHD",
"Delta Movie (2024)",
"Echo [UK] 4K",
"Foxtrot spaced",
"Golf|Pipe|Name",
"Hotel & Inn <x>",
"India 日本語 77",
"Juliet 📺 88",
"Kilo_under_9",
"Lima.dot.name",
]
# (find, replace) pairs spanning common user strategies and edge cases.
STRATEGIES = [
# --- capture groups ---
(r"(.+) Channel (\d+)", r"$1 #$2"),
(r"(\w+) (\w+)", r"$2 $1"),
(r"(.+)", r"$1 - $1"),
(r"(.+)", r"[$1]"),
(r"(.+) (\d+)$", r"$2 $1"),
# --- strip / delete ---
(r" (HD|FHD|4K|SD)\b", r""),
(r"\s+", r" "),
(r"[\[\(].*?[\]\)]", r""),
(r"\d+", r""),
(r"[_.]", r" "),
# --- anchors / inserts ---
(r"^", r"NEW "),
(r"$", r" LIVE"),
(r"^(\w+)", r"<$1>"),
# --- char classes / Unicode (divergence hunters) ---
(r"\w+", r"W"),
(r"\b\w", r"_"),
(r"[A-Z]", r"*"),
(r"\s", r"_"),
(r"[^\x00-\x7F]+", r"?"),
# --- non-capturing / lookaround / in-pattern backref ---
(r"(?:Channel|Movie|News) ", r""),
(r"(\w)\1", r"$1"),
(r"\w+(?= )", r"X"),
(r"(?<=\d)\d", r"#"),
# --- literal $ and odd replacements ---
(r" ", r" $ "),
(r"o", r"0"),
# --- invalid group references (rejected by both engines) ---
(r"(.+)", r"$2"),
(r"(.+)", r"$10"),
# --- rename that expands past Channel.name's column length ---
(r"(.+)", r"$1" * 40),
# --- regex-module syntax: quantified anchor, JS-style and duplicate
# named groups (these transform; both paths use the regex module) ---
(r"^*", r"$"),
(r"(?<season>\d+)", r"S$1"),
(r"(?P<n>x)(?P<n>y)", r"z"),
]
class RenamePreviewParityTests(TestCase):
@classmethod
def setUpClass(cls):
super().setUpClass()
from rest_framework.test import APIClient
from apps.accounts.models import User
admin = User.objects.create_superuser(
username="admin_rename_parity", password="pw", user_level=10
)
cls.client_api = APIClient()
cls.client_api.force_authenticate(user=admin)
cls.account = M3UAccount.objects.create(
name="Rename Parity Provider",
server_url="http://example.com/test.m3u",
)
def _sync(self):
return sync_auto_channels(
self.account.id,
scan_start_time=(
timezone.now() - timezone.timedelta(minutes=1)
).isoformat(),
)
def _run_case(self, group_name, find, replace):
"""Returns a list of human-readable mismatch strings (empty == parity)."""
group = ChannelGroup.objects.create(name=group_name)
from apps.channels.models import ChannelGroupM3UAccount
ChannelGroupM3UAccount.objects.create(
m3u_account=self.account,
channel_group=group,
enabled=True,
auto_channel_sync=True,
auto_sync_channel_start=1000,
custom_properties={
"name_regex_pattern": find,
"name_replace_pattern": replace,
},
)
for i, name in enumerate(NAMES):
Stream.objects.create(
name=name,
url=f"http://example.com/{group_name}_{i}.m3u8",
m3u_account=self.account,
channel_group=group,
tvg_id=f"{group_name}-{i}",
last_seen=timezone.now(),
)
# --- live rename via sync ---
result = self._sync()
if result.get("status") != "ok":
return [f"[{find!r} -> {replace!r}] sync status={result.get('status')}"]
channels = Channel.objects.filter(
auto_created_by=self.account, channel_group=group
)
cs_rows = ChannelStream.objects.filter(
channel__in=channels
).select_related("channel", "stream")
sync_map = {row.stream.name: row.channel.name for row in cs_rows}
# --- preview endpoint ---
response = self.client_api.get(
"/api/channels/streams/regex-preview/",
{
"channel_group": group_name,
"find": find,
"replace": replace,
"limit": 50,
},
)
if response.status_code != 200:
return [f"[{find!r} -> {replace!r}] preview HTTP {response.status_code}"]
data = response.data
# When the preview reports find_error it predicts no rename at all.
preview_map = {name: name for name in NAMES}
if "find_error" not in data:
for m in data.get("find_matches", []):
preview_map[m["before"]] = m["after"]
mismatches = []
for name in NAMES:
sync_after = sync_map.get(name)
if sync_after is None:
mismatches.append(
f"[{find!r} -> {replace!r}] {name!r}: no channel created"
)
continue
preview_after = preview_map[name]
if preview_after != sync_after:
mismatches.append(
f"[{find!r} -> {replace!r}] {name!r}: "
f"preview={preview_after!r} sync={sync_after!r} "
f"(find_error={data.get('find_error')!r})"
)
return mismatches
def test_preview_predicts_rename_across_strategies(self):
all_mismatches = []
for idx, (find, replace) in enumerate(STRATEGIES):
all_mismatches.extend(
self._run_case(f"ParityG{idx}", find, replace)
)
self.assertEqual(
all_mismatches,
[],
"Preview diverged from the live rename:\n"
+ "\n".join(all_mismatches),
)
def test_overlong_rename_is_bounded_not_aborting_sync(self):
# A rename that expands past Channel.name's column length must not
# abort the bulk_create sync. Both sync and preview cap at the column
# length so the channel is created (truncated) and the preview shows
# the same bounded name.
max_len = Channel._meta.get_field("name").max_length
mismatches = self._run_case("OverlongG", r"(.+)", "$1" * 40)
self.assertEqual(mismatches, [], "\n".join(mismatches))
group = ChannelGroup.objects.get(name="OverlongG")
channels = Channel.objects.filter(
auto_created_by=self.account, channel_group=group
)
self.assertEqual(channels.count(), len(NAMES))
self.assertTrue(all(len(c.name) <= max_len for c in channels))
self.assertTrue(any(len(c.name) == max_len for c in channels))

156
apps/m3u/tests/test_stream_filters.py Normal file
View file

@ -0,0 +1,156 @@
"""Tests for M3U stream filter compilation and batch application."""
from unittest.mock import MagicMock, patch
from django.test import SimpleTestCase
from apps.m3u.tasks import (
_compile_m3u_stream_filters,
_stream_passes_m3u_filters,
process_m3u_batch_direct,
)
class CompileM3UStreamFiltersTests(SimpleTestCase):
def test_compiles_case_insensitive_when_configured(self):
filter_obj = MagicMock()
filter_obj.regex_pattern = "news"
filter_obj.custom_properties = {"case_sensitive": False}
compiled = _compile_m3u_stream_filters([filter_obj])
self.assertEqual(len(compiled), 1)
pattern, _ = compiled[0]
self.assertTrue(pattern.search("NEWS"))
def test_compiles_case_sensitive_by_default(self):
filter_obj = MagicMock()
filter_obj.regex_pattern = "news"
filter_obj.custom_properties = {}
compiled = _compile_m3u_stream_filters([filter_obj])
pattern, _ = compiled[0]
self.assertIsNone(pattern.search("NEWS"))
self.assertTrue(pattern.search("news"))
class StreamPassesM3UFiltersTests(SimpleTestCase):
def _compiled(self, *, filter_type="name", exclude=False, pattern="Adult"):
filter_obj = MagicMock()
filter_obj.filter_type = filter_type
filter_obj.exclude = exclude
filter_obj.regex_pattern = pattern
filter_obj.custom_properties = {}
return _compile_m3u_stream_filters([filter_obj])
def test_include_filter_passes_matching_stream(self):
compiled = self._compiled(exclude=False)
self.assertTrue(
_stream_passes_m3u_filters("Adult Channel", "http://x", "News", compiled)
)
def test_include_filter_passes_non_matching_stream(self):
"""Non-matching streams still pass unless a matching exclude filter hits."""
compiled = self._compiled(exclude=False, pattern="news")
self.assertTrue(
_stream_passes_m3u_filters("Sports", "http://x", "Sports", compiled)
)
def test_exclude_filter_rejects_matching_stream(self):
compiled = self._compiled(exclude=True, pattern="Adult")
self.assertFalse(
_stream_passes_m3u_filters("Adult Channel", "http://x", "News", compiled)
)
def test_url_filter_type_targets_url(self):
compiled = self._compiled(filter_type="url", exclude=True, pattern="blocked")
self.assertFalse(
_stream_passes_m3u_filters("OK", "http://blocked.example/live", "News", compiled)
)
self.assertTrue(
_stream_passes_m3u_filters("blocked name", "http://ok.example/live", "News", compiled)
)
def test_group_filter_type_targets_group(self):
compiled = self._compiled(filter_type="group", exclude=True, pattern="Hidden")
self.assertFalse(
_stream_passes_m3u_filters("Channel", "http://x", "Hidden Group", compiled)
)
class ProcessM3UBatchFilterTests(SimpleTestCase):
def _mock_stream_meta(self, mock_stream_cls, max_length=255):
mock_field = MagicMock()
mock_field.max_length = max_length
mock_stream_cls._meta.get_field.return_value = mock_field
@patch("apps.m3u.tasks._bulk_update_stream_refresh_batches")
@patch("apps.m3u.tasks.Stream")
@patch("apps.m3u.tasks.M3UAccount")
def test_exclude_filter_skips_stream_import(
self, mock_account_cls, mock_stream_cls, mock_bulk_update,
):
self._mock_stream_meta(mock_stream_cls)
mock_account = MagicMock()
mock_account.account_type = "STD"
mock_account_cls.objects.get.return_value = mock_account
mock_stream_cls.objects.filter.return_value.select_related.return_value.only.return_value = (
[]
)
mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123")
filter_obj = MagicMock()
filter_obj.regex_pattern = "skip-me"
filter_obj.filter_type = "name"
filter_obj.exclude = True
filter_obj.custom_properties = {}
compiled = _compile_m3u_stream_filters([filter_obj])
batch = [{
"name": "skip-me channel",
"url": "http://example/live",
"attributes": {"group-title": "News"},
"vlc_opts": {},
}]
with patch("django.db.connections"):
result = process_m3u_batch_direct(
1, batch, {"News": 1}, ["name", "url"], compiled_filters=compiled,
)
self.assertIn("0 created", result)
mock_stream_cls.objects.bulk_create.assert_not_called()
mock_bulk_update.assert_called_once_with([], [], batch_size=200)
@patch("apps.m3u.tasks._bulk_update_stream_refresh_batches")
@patch("apps.m3u.tasks.Stream")
@patch("apps.m3u.tasks.M3UAccount")
def test_no_filters_imports_matching_stream(
self, mock_account_cls, mock_stream_cls, mock_bulk_update,
):
self._mock_stream_meta(mock_stream_cls)
mock_account = MagicMock()
mock_account.account_type = "STD"
mock_account_cls.objects.get.return_value = mock_account
mock_stream_cls.objects.filter.return_value.select_related.return_value.only.return_value = (
[]
)
mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123")
mock_stream_cls.objects.bulk_create.return_value = []
batch = [{
"name": "News One",
"url": "http://example/live",
"attributes": {"group-title": "News"},
"vlc_opts": {},
}]
with patch("django.db.connections"), patch(
"apps.m3u.tasks.transaction.atomic",
):
result = process_m3u_batch_direct(
1, batch, {"News": 1}, ["name", "url"], compiled_filters=[],
)
self.assertIn("1 created", result)
mock_stream_cls.objects.bulk_create.assert_called_once()

483
apps/m3u/tests/test_sync_correctness.py
View file

@ -46,13 +46,14 @@ def _attach_group_to_account(account, group, custom_properties=None):
)
def _make_stream(account, group, name="ESPN", tvg_id="espn"):
def _make_stream(account, group, name="ESPN", tvg_id="espn", stream_chno=None):
return Stream.objects.create(
name=name,
url=f"http://example.com/{name.lower()}.m3u8",
m3u_account=account,
channel_group=group,
tvg_id=tvg_id,
stream_chno=stream_chno,
last_seen=timezone.now(),
)
@ -413,21 +414,21 @@ class RangeEnforcementTests(TestCase):
"new auto-channel duplicates A's effective channel number.",
)
def test_provider_mode_fallback_respects_range_start(self):
# Provider-mode streams without a usable stream_chno fall back to
# the next-available picker. The fallback must honor the group's
# configured `auto_sync_channel_start` so freshly-created channels
# never land below the user's chosen range. Without this, a group
# configured for [100, 200] silently spawned channels at #1 when
# the provider omitted channel-number metadata.
def test_provider_mode_numberless_fallback_uses_visible_start(self):
# In provider mode the visible "Start #" is channel_numbering_fallback,
# so a numberless stream's fallback walks from there, not from the
# hidden auto_sync_channel_start (set far above the range here to prove
# it is ignored).
# Fail signature: 0 channels created, or a channel below 100 = fallback
# seeded from the wrong field.
account = _make_account()
group = _make_group(name="Sports")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_start = 100
rel.auto_sync_channel_start = 5000 # hidden; must be ignored
rel.auto_sync_channel_end = 200
rel.custom_properties = {
"channel_numbering_mode": "provider",
"channel_numbering_fallback": 1,
"channel_numbering_fallback": 100, # the visible "Start #"
}
rel.save()
@ -643,6 +644,34 @@ class NumbersInRangeLookupTests(TestCase):
)
self.assertFalse(occupant["has_channel_number_override"])
def test_group_override_channel_reports_target_group(self):
# When auto-sync routes channels into a different group via
# group_override, the occupant's channel_group_id is the override
# target, not the source group being configured. The frontend relies
# on this to recognize override-routed channels as the config's own
# output (effectiveSyncGroupId), so the warning does not flag them.
account = _make_account()
source = _make_group(name="SourceGrp")
target = _make_group(name="TargetGrp")
Channel.objects.create(
name="Routed",
channel_number=3210,
channel_group=target,
auto_created=True,
auto_created_by=account,
)
client = self._client()
response = client.get(
"/api/channels/channels/numbers-in-range/?start=3210&end=3210"
)
occupant = response.data["occupants"][0]
self.assertEqual(occupant["channel_group_id"], target.id)
self.assertNotEqual(occupant["channel_group_id"], source.id)
self.assertTrue(occupant["auto_created"])
self.assertEqual(occupant["auto_created_by_account_id"], account.id)
def test_manual_channel_exposed_with_auto_created_false(self):
# Manual channels are always a real collision worth surfacing.
# The response must flag them with auto_created=False and a null
@ -734,6 +763,128 @@ class RegexPreviewTests(TestCase):
self.assertEqual(response.data["total_scanned"], 3)
self.assertFalse(response.data["scan_limit_hit"])
def test_find_replace_applies_numbered_capture_group(self):
# The replace field accepts JS-style $1 backreferences, but the regex
# engine expects \1. Without the conversion the preview echoes the
# literal "$1", so the previewed "after" disagrees with the name the
# live rename produces.
account = self._make_account()
group = _make_group(name="Sports")
Stream.objects.create(
name="High Limit Racing at Eagle @ Jun 9 7:00 PM",
url="http://example.com/hlr.m3u8",
m3u_account=account,
channel_group=group,
last_seen=timezone.now(),
)
client = self._client()
response = client.get(
"/api/channels/streams/regex-preview/",
{"channel_group": "Sports", "find": r"(.+) @.*", "replace": "$1"},
)
self.assertEqual(response.status_code, 200)
self.assertEqual(response.data["find_match_count"], 1)
after = response.data["find_matches"][0]["after"]
self.assertEqual(after, "High Limit Racing at Eagle")
self.assertNotIn("$1", after)
def test_preview_after_matches_live_sync_rename(self):
# Guards the defect class: the preview and the live rename are
# separate code paths that must convert the replacement identically,
# so the preview can never promise an output the sync would not yield.
name = "High Limit Racing at Eagle @ Jun 9 7:00 PM"
account = self._make_account()
group = _make_group(name="Racing")
_attach_group_to_account(
account,
group,
custom_properties={
"name_regex_pattern": r"(.+) @.*",
"name_replace_pattern": "$1",
},
)
_make_stream(account, group, name=name, tvg_id="hlr")
result = _sync(account)
self.assertEqual(result.get("status"), "ok")
channel = Channel.objects.get(auto_created=True, auto_created_by=account)
live_name = channel.name
client = self._client()
response = client.get(
"/api/channels/streams/regex-preview/",
{"channel_group": "Racing", "find": r"(.+) @.*", "replace": "$1"},
)
self.assertEqual(response.status_code, 200)
preview_after = response.data["find_matches"][0]["after"]
self.assertEqual(preview_after, live_name)
self.assertEqual(preview_after, "High Limit Racing at Eagle")
def test_regex_engine_pattern_transforms_in_preview(self):
# Both the preview and the live rename use the regex module, which is
# more permissive than stdlib re and matches the JS-style syntax the UI
# authors. A quantified anchor like "^*" (which stdlib re rejects)
# compiles and transforms rather than reporting an error.
account = self._make_account()
group = _make_group(name="Sports")
Stream.objects.create(
name="Doc95",
url="http://example.com/doc95.m3u8",
m3u_account=account,
channel_group=group,
last_seen=timezone.now(),
)
client = self._client()
response = client.get(
"/api/channels/streams/regex-preview/",
{"channel_group": "Sports", "find": "^*", "replace": "$"},
)
self.assertEqual(response.status_code, 200)
self.assertNotIn("find_error", response.data)
self.assertEqual(response.data["find_match_count"], 1)
# ^* matches the empty string at every position, so the literal $
# replacement is inserted between characters.
self.assertEqual(
response.data["find_matches"][0]["after"], "$D$o$c$9$5$"
)
def test_preview_and_sync_agree_on_regex_only_pattern(self):
# Parity guard for the engine alignment: a pattern valid in regex but
# not stdlib re must transform identically in the sync and the preview,
# rather than diverging (the sync no longer silently keeps the
# original name for these patterns).
name = "Doc95"
account = self._make_account()
group = _make_group(name="Docs")
_attach_group_to_account(
account,
group,
custom_properties={
"name_regex_pattern": "^*",
"name_replace_pattern": "$",
},
)
_make_stream(account, group, name=name, tvg_id="doc95")
result = _sync(account)
self.assertEqual(result.get("status"), "ok")
channel = Channel.objects.get(auto_created=True, auto_created_by=account)
live_name = channel.name
self.assertNotEqual(live_name, name)
client = self._client()
response = client.get(
"/api/channels/streams/regex-preview/",
{"channel_group": "Docs", "find": "^*", "replace": "$"},
)
self.assertEqual(response.status_code, 200)
self.assertEqual(response.data["find_matches"][0]["after"], live_name)
def test_filter_returns_matched_names_with_count(self):
account = self._make_account()
group = _make_group(name="Sports")
@ -2300,3 +2451,315 @@ class CompactNumberingIdempotencyTests(TransactionTestCase):
"pass with no hide or override change. _repack_inner is following "
"physical row order instead of id order.",
)
class ProviderNumberingHonorsProviderNumberTests(TestCase):
"""
Provider numbering uses a stream's provider number (stream_chno) verbatim.
The group start is auto-populated by the UI and is not editable in provider
mode (the UI binds "Start #" to channel_numbering_fallback), so treating it
as a lower bound silently discarded valid provider numbers: on a lineup
topping out near 5000, provider numbers 100-150 landed at ~5000.
The start and end bound only the fallback for numberless streams.
"""
def test_provider_number_below_high_auto_start_is_honored(self):
# Provider numbers 100-104 with an auto-set start of 5000 must land at
# their provider numbers.
# Fail signature: channels at 5000-5004 = start used as a hard floor.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_start = 5000
rel.auto_sync_channel_end = None
rel.custom_properties = {
"channel_numbering_mode": "provider",
"channel_numbering_fallback": 1,
}
rel.save()
for i in range(5):
_make_stream(
account, group, name=f"PPV {i}", tvg_id=f"ppv{i}",
stream_chno=100 + i,
)
result = _sync(account)
self.assertEqual(result["status"], "ok")
numbers = sorted(
Channel.objects.filter(
auto_created=True, auto_created_by=account
).values_list("channel_number", flat=True)
)
self.assertEqual(numbers, [100.0, 101.0, 102.0, 103.0, 104.0])
def test_provider_number_honored_when_start_unset(self):
# start blank -> defaults to 1.0; provider numbers still honored.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_start = None
rel.auto_sync_channel_end = None
rel.custom_properties = {"channel_numbering_mode": "provider"}
rel.save()
_make_stream(account, group, name="PPV", tvg_id="ppv", stream_chno=100)
result = _sync(account)
self.assertEqual(result["status"], "ok")
created = Channel.objects.get(auto_created=True, auto_created_by=account)
self.assertEqual(created.channel_number, 100.0)
def test_numberless_stream_uses_fallback_not_hidden_start(self):
# In provider mode without a range, a stream lacking a provider
# number falls back to channel_numbering_fallback (the visible
# "Start #"), not the hidden auto_sync_channel_start.
# Fail signature: channel at 5000 = fallback bumped to hidden start.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_start = 5000
rel.auto_sync_channel_end = None
rel.custom_properties = {
"channel_numbering_mode": "provider",
"channel_numbering_fallback": 300,
}
rel.save()
_make_stream(account, group, name="NoChno", tvg_id="nc")
result = _sync(account)
self.assertEqual(result["status"], "ok")
created = Channel.objects.get(auto_created=True, auto_created_by=account)
self.assertEqual(created.channel_number, 300.0)
def test_provider_number_below_range_is_honored_verbatim(self):
# A provider number below the group's Start/End is used as-is, not
# coerced into the range.
# Fail signature: channel pulled to >= 100 = range coercing a provider
# number.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_end = 200
rel.custom_properties = {
"channel_numbering_mode": "provider",
"channel_numbering_fallback": 100,
}
rel.save()
_make_stream(account, group, name="Low", tvg_id="low", stream_chno=50)
result = _sync(account)
self.assertEqual(result["status"], "ok")
created = Channel.objects.get(auto_created=True, auto_created_by=account)
self.assertEqual(created.channel_number, 50.0)
def test_provider_number_above_end_is_honored_verbatim(self):
# Provider numbers above the configured End are also honored as-is;
# the End caps only the fallback for numberless streams.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_end = 200
rel.custom_properties = {
"channel_numbering_mode": "provider",
"channel_numbering_fallback": 1,
}
rel.save()
_make_stream(account, group, name="High", tvg_id="high", stream_chno=5000)
result = _sync(account)
self.assertEqual(result["status"], "ok")
created = Channel.objects.get(auto_created=True, auto_created_by=account)
self.assertEqual(created.channel_number, 5000.0)
def test_provider_number_within_range_is_honored(self):
# An in-range provider number is used as-is.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_end = 200
rel.custom_properties = {
"channel_numbering_mode": "provider",
"channel_numbering_fallback": 1,
}
rel.save()
_make_stream(account, group, name="Mid", tvg_id="mid", stream_chno=150)
result = _sync(account)
self.assertEqual(result["status"], "ok")
created = Channel.objects.get(auto_created=True, auto_created_by=account)
self.assertEqual(created.channel_number, 150.0)
def test_duplicate_provider_numbers_keep_one_and_fall_back_the_other(self):
# Two streams claim the same provider number (common in messy event
# feeds). One keeps it; the colliding one falls back to a different free
# number rather than being dropped or overwriting the first.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.custom_properties = {
"channel_numbering_mode": "provider",
"channel_numbering_fallback": 1,
}
rel.save()
_make_stream(account, group, name="A", tvg_id="a", stream_chno=77250)
_make_stream(account, group, name="B", tvg_id="b", stream_chno=77250)
result = _sync(account)
self.assertEqual(result["status"], "ok")
numbers = sorted(
Channel.objects.filter(
auto_created=True, auto_created_by=account
).values_list("channel_number", flat=True)
)
self.assertEqual(len(numbers), 2)
self.assertIn(77250.0, numbers)
self.assertNotEqual(numbers[0], numbers[1])
def test_provider_number_colliding_with_manual_channel_falls_back(self):
# A provider number matching an existing channel must not overwrite it;
# the auto-created channel falls back to a different free number.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.custom_properties = {
"channel_numbering_mode": "provider",
"channel_numbering_fallback": 1,
}
rel.save()
manual = Channel.objects.create(name="Manual", channel_number=88250)
_make_stream(account, group, name="P", tvg_id="p", stream_chno=88250)
result = _sync(account)
self.assertEqual(result["status"], "ok")
created = Channel.objects.get(auto_created=True, auto_created_by=account)
self.assertNotEqual(created.channel_number, 88250.0)
manual.refresh_from_db()
self.assertEqual(manual.channel_number, 88250.0)
def test_provider_fallback_exhaustion_reports_visible_start(self):
# RANGE_EXHAUSTED must cite the fallback range (channel_numbering_fallback
# to End), not the hidden auto_sync_channel_start left from another mode.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_start = 5000
rel.auto_sync_channel_end = 102
rel.custom_properties = {
"channel_numbering_mode": "provider",
"channel_numbering_fallback": 100,
}
rel.save()
for i in range(4):
_make_stream(account, group, name=f"S{i}", tvg_id=f"s{i}")
result = _sync(account)
self.assertEqual(result["channels_created"], 3)
self.assertEqual(result["channels_failed"], 1)
error = result["failed_stream_details"][0]["error"]
self.assertIn("100-102", error)
self.assertNotIn("5000", error)
class CrossModeNumberingFieldTests(TestCase):
"""
Each numbering mode's UI exposes only a subset of the persisted fields,
and switching modes does not reset the others. The backend must therefore
read only the fields a mode actually owns, so a stale/hidden value left by
another mode cannot silently change numbering. These guard the two
remaining facets of that family (the provider-floor facet is covered by
ProviderNumberingHonorsProviderNumberTests).
"""
def _restamp(self, account):
Stream.objects.filter(m3u_account=account).update(
last_seen=timezone.now()
)
def test_next_available_ignores_configured_end(self):
# next_available exposes no Start/End in its UI, so a stale End left
# over from a prior mode must not cap it. Every stream gets the lowest
# free number from 1 regardless of the End.
# Fail signature: streams beyond the End fail = next_available honoring
# a hidden cap.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_start = 1
rel.auto_sync_channel_end = 3 # stale cap from a prior mode
rel.custom_properties = {"channel_numbering_mode": "next_available"}
rel.save()
for i in range(5):
_make_stream(account, group, name=f"S{i}", tvg_id=f"s{i}")
result = _sync(account)
self.assertEqual(result["status"], "ok")
self.assertEqual(result["channels_created"], 5)
self.assertEqual(result["channels_failed"], 0)
def test_provider_channels_outside_range_are_not_deleted(self):
# Range enforcement (the overflow-delete) is fixed-mode only. A provider
# channel whose number is outside [start, end] is authoritative and must
# survive sync, not be deleted and churned into a new row.
# Fail signature: channels_deleted > 0 on the second sync = overflow
# delete firing in provider mode.
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_end = 200
rel.custom_properties = {
"channel_numbering_mode": "provider",
"channel_numbering_fallback": 1,
}
rel.save()
_make_stream(account, group, name="High", tvg_id="high", stream_chno=5000)
first = _sync(account)
self.assertEqual(first["channels_created"], 1)
original = Channel.objects.get(auto_created=True, auto_created_by=account)
self.assertEqual(original.channel_number, 5000.0)
self._restamp(account)
second = _sync(account)
self.assertEqual(second["channels_deleted"], 0)
survivor = Channel.objects.get(auto_created=True, auto_created_by=account)
self.assertEqual(survivor.id, original.id)
self.assertEqual(survivor.channel_number, 5000.0)
def test_next_available_channels_outside_stale_range_not_deleted(self):
# Same gate for next_available: tightening a stale End must not delete
# already-assigned channels (range enforcement is fixed-mode only).
account = _make_account()
group = _make_group(name="PPV")
rel = _attach_group_to_account(account, group)
rel.auto_sync_channel_start = 1
rel.auto_sync_channel_end = None
rel.custom_properties = {"channel_numbering_mode": "next_available"}
rel.save()
for i in range(5):
_make_stream(account, group, name=f"S{i}", tvg_id=f"s{i}")
self.assertEqual(_sync(account)["channels_created"], 5)
rel.auto_sync_channel_end = 3 # stale cap appears
rel.save()
self._restamp(account)
second = _sync(account)
self.assertEqual(second["channels_deleted"], 0)
self.assertEqual(
Channel.objects.filter(
auto_created=True, auto_created_by=account
).count(),
5,
)

131
apps/m3u/tests/test_xc_empty_fetch_guard.py Normal file
View file

@ -0,0 +1,131 @@
"""
Regression test for the Xtream Codes empty-fetch channel wipe.
Bug: when an XC provider returns no live streams on a routine refresh (a
transient upstream failure, a fetch exception, or no enabled category
matching), ``collect_xc_streams`` returns ``[]`` and the refresh used to fall
through to stale-marking and ``sync_auto_channels``. With nothing "seen" this
refresh, auto-sync deletes the account's entire auto-created channel lineup.
Fix: ``_refresh_single_m3u_account_impl`` aborts the XC branch when
``collect_xc_streams`` returns empty, setting the account to ERROR before any
stale-marking or auto-sync runs, mirroring the standard-path empty guards.
"""
from unittest.mock import MagicMock, patch
from django.test import TransactionTestCase
from django.utils import timezone
from apps.channels.models import (
Channel,
ChannelGroup,
ChannelGroupM3UAccount,
Stream,
)
from apps.m3u.models import M3UAccount
from apps.m3u.tasks import _refresh_single_m3u_account_impl
class XCEmptyFetchGuardTests(TransactionTestCase):
def _setup_xc_account_with_auto_channel(self):
account = M3UAccount.objects.create(
name="Test XC Provider",
server_url="http://example.com",
username="user",
password="pass",
account_type=M3UAccount.Types.XC,
is_active=True,
)
group = ChannelGroup.objects.create(name="Sports")
ChannelGroupM3UAccount.objects.create(
m3u_account=account,
channel_group=group,
enabled=True,
auto_channel_sync=True,
auto_sync_channel_start=100,
custom_properties={"xc_id": "123"},
)
# A pre-existing stream and the auto-created channel built from it on a
# prior healthy refresh -- this is exactly what the bug deletes.
stream = Stream.objects.create(
name="ESPN",
url="http://example.com/espn.m3u8",
m3u_account=account,
channel_group=group,
last_seen=timezone.now(),
is_stale=False,
)
channel = Channel.objects.create(
channel_number=100,
name="ESPN",
channel_group=group,
auto_created=True,
auto_created_by=account,
)
return account, group, stream, channel
@patch("apps.m3u.tasks.sync_auto_channels")
@patch("apps.m3u.tasks.collect_xc_streams", return_value=[])
@patch("apps.m3u.tasks.refresh_m3u_groups")
def test_empty_xc_fetch_aborts_before_sync_and_preserves_channels(
self, mock_refresh_groups, _mock_collect, mock_sync
):
account, group, stream, channel = self._setup_xc_account_with_auto_channel()
# XC refresh: empty extinf_data is normal, groups must be present.
mock_refresh_groups.return_value = ([], {"Sports": group.id})
result = _refresh_single_m3u_account_impl(account.id)
# The refresh aborts, so auto channel sync never runs.
mock_sync.assert_not_called()
# The auto-created channel survives the empty fetch.
self.assertTrue(Channel.objects.filter(pk=channel.pk).exists())
# The stream is not marked stale (stale-marking is skipped on abort).
stream.refresh_from_db()
self.assertFalse(stream.is_stale)
# The account is surfaced as errored, not silently "successful".
account.refresh_from_db()
self.assertEqual(account.status, M3UAccount.Status.ERROR)
self.assertIn("no streams returned from provider", result)
@patch("apps.m3u.tasks.log_system_event")
@patch("apps.m3u.tasks.send_m3u_update")
@patch("apps.m3u.tasks.cleanup_stale_group_relationships")
@patch("apps.m3u.tasks.cleanup_streams", return_value=0)
@patch("apps.m3u.tasks.process_m3u_batch_direct", return_value="1 created, 0 updated")
@patch("apps.m3u.tasks.sync_auto_channels")
@patch("apps.m3u.tasks.refresh_m3u_groups")
def test_non_empty_xc_fetch_still_runs_sync(
self,
mock_refresh_groups,
mock_sync,
_mock_process,
_mock_cleanup_streams,
_mock_cleanup_groups,
_mock_ws,
_mock_log,
):
# The guard must not fire on a healthy refresh: a non-empty fetch
# proceeds to auto channel sync as before.
account, group, _stream, _channel = self._setup_xc_account_with_auto_channel()
mock_refresh_groups.return_value = ([], {"Sports": group.id})
mock_sync.return_value = {
"status": "ok",
"channels_created": 1,
"channels_updated": 0,
"channels_deleted": 0,
"channels_failed": 0,
"failed_stream_details": [],
}
xc_stream = {
"name": "ESPN",
"url": "http://example.com/espn.m3u8",
"attributes": {"group-title": "Sports", "stream_id": "1"},
}
with patch("apps.m3u.tasks.collect_xc_streams", return_value=[xc_stream]):
_refresh_single_m3u_account_impl(account.id)
mock_sync.assert_called_once()
account.refresh_from_db()
self.assertEqual(account.status, M3UAccount.Status.SUCCESS)

139
apps/m3u/tests/test_xc_live_url.py Normal file
View file

@ -0,0 +1,139 @@
"""Tests for XC stream URL normalization and on-demand URL building."""
from django.test import TestCase
from apps.channels.models import Stream
from apps.m3u.models import M3UAccount, M3UAccountProfile
from apps.m3u.tasks import get_transformed_credentials
from apps.proxy.live_proxy.url_utils import _resolve_live_stream_url
from apps.vod.models import Episode, M3UEpisodeRelation, M3UMovieRelation, Movie, Series
from core.xtream_codes import normalize_server_url
class NormalizeServerUrlTests(TestCase):
def test_preserves_sub_path(self):
url = "https://myserver.fun/server1"
self.assertEqual(normalize_server_url(url), "https://myserver.fun/server1")
def test_strips_player_api_php_and_query_params(self):
url = "https://myserver.fun/server1/player_api.php?username=foo&password=bar"
self.assertEqual(normalize_server_url(url), "https://myserver.fun/server1")
def test_strips_trailing_slash(self):
url = "https://myserver.fun/server1/"
self.assertEqual(normalize_server_url(url), "https://myserver.fun/server1")
def test_nested_sub_path_with_php_endpoint(self):
url = "http://server/Pluto/gb/player_api.php"
self.assertEqual(normalize_server_url(url), "http://server/Pluto/gb")
class GetTransformedCredentialsTests(TestCase):
def test_returns_normalized_server_url(self):
account = M3UAccount.objects.create(
name="Sub-path XC",
account_type="XC",
server_url="https://myserver.fun/server1/player_api.php?username=foo",
username="alice",
password="secret",
)
profile = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
server_url, username, password = get_transformed_credentials(account, profile)
self.assertEqual(server_url, "https://myserver.fun/server1")
self.assertEqual(username, "alice")
self.assertEqual(password, "secret")
class ResolveLiveStreamUrlTests(TestCase):
def test_builds_url_from_normalized_base_not_raw_account_url(self):
account = M3UAccount.objects.create(
name="Live sub-path",
account_type="XC",
server_url="https://myserver.fun/server1/player_api.php?username=foo",
username="alice",
password="secret",
)
profile = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
stream = Stream.objects.create(
name="Test Channel",
m3u_account=account,
stream_id="12345",
url="https://myserver.fun/server1/live/olduser/oldpass/12345.ts",
)
url = _resolve_live_stream_url(stream, account, profile)
self.assertEqual(
url,
"https://myserver.fun/server1/live/alice/secret/12345.ts",
)
def test_std_account_uses_stored_stream_url(self):
account = M3UAccount.objects.create(
name="STD account",
account_type="STD",
server_url="https://example.com/list.m3u",
username="alice",
password="secret",
)
profile = M3UAccountProfile.objects.get(m3u_account=account, is_default=True)
stream = Stream.objects.create(
name="STD Stream",
m3u_account=account,
url="https://provider.example/stream/abc123",
)
url = _resolve_live_stream_url(stream, account, profile)
self.assertEqual(url, "https://provider.example/stream/abc123")
class VodStreamUrlTests(TestCase):
def setUp(self):
self.account = M3UAccount.objects.create(
name="VOD sub-path",
account_type="XC",
server_url="https://myserver.fun/server1/player_api.php?username=foo",
username="alice",
password="secret",
)
def test_movie_relation_builds_normalized_url(self):
movie = Movie.objects.create(name="Test Movie")
relation = M3UMovieRelation.objects.create(
m3u_account=self.account,
movie=movie,
stream_id="999",
container_extension="mkv",
)
url = relation.get_stream_url()
self.assertEqual(
url,
"https://myserver.fun/server1/movie/alice/secret/999.mkv",
)
def test_episode_relation_builds_normalized_url(self):
series = Series.objects.create(name="Test Series")
episode = Episode.objects.create(
series=series,
name="Pilot",
season_number=1,
episode_number=1,
)
relation = M3UEpisodeRelation.objects.create(
m3u_account=self.account,
episode=episode,
stream_id="888",
container_extension="mp4",
)
url = relation.get_stream_url()
self.assertEqual(
url,
"https://myserver.fun/server1/series/alice/secret/888.mp4",
)

13
apps/m3u/utils.py
View file

@ -1,4 +1,5 @@
# apps/m3u/utils.py
import regex
import threading
import logging
from django.db import models
@ -9,6 +10,18 @@ active_streams_map = {}
logger = logging.getLogger(__name__)
def convert_js_numbered_backreferences(replacement):
"""Translate JS-style ``$1``/``$2`` backreferences to Python ``\\1``/``\\2``.
Auto-sync replace patterns are authored in JS regex syntax, but Python's
regex engines honor backslash backreferences, not ``$1``. The live rename
and the UI preview must convert identically, so both call this single
helper and cannot drift apart (otherwise the preview promises an output
the sync would never produce).
"""
return regex.sub(r"\$(\d+)", r"\\\1", replacement)
def normalize_stream_url(url):
"""
Normalize stream URLs for compatibility with FFmpeg.

1732
apps/output/epg.py Normal file
View file

File diff suppressed because it is too large Load diff

239
apps/output/streaming_chunk_cache.py Normal file
View file

@ -0,0 +1,239 @@
"""Single-flight Redis chunk cache for large streaming HTTP responses."""
import logging
import time
from django.http import StreamingHttpResponse
logger = logging.getLogger(__name__)
STATUS_BUILDING = "building"
STATUS_READY = "ready"
STATUS_ERROR = "error"
DEFAULT_CACHE_TTL = 300
DEFAULT_LOCK_TTL = 120
DEFAULT_POLL_INTERVAL = 0.05
DEFAULT_MAX_FOLLOWER_WAIT = 600
def _chunks_key(base_key):
return f"{base_key}:chunks"
def _ready_key(base_key):
return f"{base_key}:ready"
def _status_key(base_key):
return f"{base_key}:status"
def _lock_key(base_key):
return f"{base_key}:lock"
def _decode_chunk(chunk):
if chunk is None:
return None
if isinstance(chunk, bytes):
return chunk.decode("utf-8")
return chunk
def _encode_chunk(chunk):
if isinstance(chunk, bytes):
return chunk
return chunk.encode("utf-8")
def _poll_wait(interval):
try:
from core.utils import _is_gevent_monkey_patched
if _is_gevent_monkey_patched():
import gevent
gevent.sleep(interval)
return
except ImportError:
pass
time.sleep(interval)
def _get_redis():
from django_redis import get_redis_connection
return get_redis_connection("default")
def _get_status(redis, base_key):
raw = redis.get(_status_key(base_key))
if raw is None:
return None
return _decode_chunk(raw)
def _clear_build_keys(redis, base_key):
redis.delete(
_chunks_key(base_key),
_status_key(base_key),
_ready_key(base_key),
_lock_key(base_key),
)
def _try_acquire_lock(redis, base_key, lock_ttl):
return bool(redis.set(_lock_key(base_key), "1", nx=True, ex=lock_ttl))
def _refresh_build_ttl(redis, base_key, lock_ttl):
redis.expire(_lock_key(base_key), lock_ttl)
redis.expire(_status_key(base_key), lock_ttl)
redis.expire(_chunks_key(base_key), lock_ttl)
def _stream_ready(redis, base_key):
offset = 0
chunks_key = _chunks_key(base_key)
while True:
chunk = redis.lindex(chunks_key, offset)
if chunk is None:
break
yield _decode_chunk(chunk)
offset += 1
def _stream_build(redis, base_key, source, cache_ttl, lock_ttl):
"""Leader: stream to client and append each chunk to Redis."""
chunks_key = _chunks_key(base_key)
status_key = _status_key(base_key)
try:
from django.core.cache import cache as django_cache
django_cache.delete(base_key) # clear any non-chunked entry under this key
redis.delete(chunks_key, _ready_key(base_key))
redis.set(status_key, STATUS_BUILDING, ex=lock_ttl)
refresh_interval = max(1, lock_ttl // 4)
last_refresh = 0.0
for chunk in source():
redis.rpush(chunks_key, _encode_chunk(chunk))
now = time.monotonic()
if now - last_refresh >= refresh_interval:
_refresh_build_ttl(redis, base_key, lock_ttl)
last_refresh = now
yield chunk
redis.set(status_key, STATUS_READY)
redis.set(_ready_key(base_key), "1")
redis.expire(chunks_key, cache_ttl)
redis.expire(status_key, cache_ttl)
redis.expire(_ready_key(base_key), cache_ttl)
logger.debug("Cached response in %s chunks", redis.llen(chunks_key))
except Exception:
logger.exception("Chunk cache build failed for %s", base_key)
redis.delete(chunks_key)
redis.set(status_key, STATUS_ERROR, ex=60)
raise
finally:
redis.delete(_lock_key(base_key))
def _stream_follow(redis, base_key, source, cache_ttl, lock_ttl, poll_interval, max_follower_wait):
"""Follower: read chunks as the leader writes them."""
offset = 0
deadline = time.monotonic() + max_follower_wait
idle_polls = 0
chunks_key = _chunks_key(base_key)
lock_key = _lock_key(base_key)
while True:
chunk = redis.lindex(chunks_key, offset)
if chunk is not None:
idle_polls = 0
yield _decode_chunk(chunk)
offset += 1
continue
status = _get_status(redis, base_key)
if status == STATUS_READY:
break
if status == STATUS_ERROR:
_clear_build_keys(redis, base_key)
if offset == 0 and _try_acquire_lock(redis, base_key, lock_ttl):
yield from _stream_build(redis, base_key, source, cache_ttl, lock_ttl)
return
raise RuntimeError("Chunk cache build failed")
if time.monotonic() >= deadline:
if offset == 0 and _try_acquire_lock(redis, base_key, lock_ttl):
logger.warning("Chunk cache follower timed out; rebuilding %s", base_key)
yield from _stream_build(redis, base_key, source, cache_ttl, lock_ttl)
return
logger.warning("Chunk cache follower timed out after partial read for %s", base_key)
break
lock_active = bool(redis.exists(lock_key))
if status != STATUS_BUILDING and not lock_active:
idle_polls += 1
if offset == 0 and idle_polls >= max(1, int(1.0 / poll_interval)):
if _try_acquire_lock(redis, base_key, lock_ttl):
logger.warning("Chunk cache leader lost; rebuilding %s", base_key)
yield from _stream_build(redis, base_key, source, cache_ttl, lock_ttl)
return
else:
idle_polls = 0
_poll_wait(poll_interval)
def stream_cached_response(
cache_key,
source,
*,
content_type="application/xml",
filename=None,
cache_ttl=DEFAULT_CACHE_TTL,
lock_ttl=DEFAULT_LOCK_TTL,
poll_interval=DEFAULT_POLL_INTERVAL,
max_follower_wait=DEFAULT_MAX_FOLLOWER_WAIT,
redis=None,
):
"""
Stream a large response with single-flight Redis chunk caching.
``source`` must be a callable returning a chunk iterator. Only the leader
invokes it; concurrent followers replay chunks already written to Redis, so
the expensive ``source`` runs at most once per ``cache_key``.
"""
if redis is None:
redis = _get_redis()
if redis.get(_ready_key(cache_key)):
logger.debug("Serving response from chunk cache")
stream = _stream_ready(redis, cache_key)
else:
status = _get_status(redis, cache_key)
if status == STATUS_ERROR:
_clear_build_keys(redis, cache_key)
if _try_acquire_lock(redis, cache_key, lock_ttl):
logger.debug("Building response (cache leader)")
stream = _stream_build(redis, cache_key, source, cache_ttl, lock_ttl)
else:
logger.debug("Following in-flight cache build")
stream = _stream_follow(
redis,
cache_key,
source,
cache_ttl,
lock_ttl,
poll_interval,
max_follower_wait,
)
response = StreamingHttpResponse(stream, content_type=content_type)
if filename:
response["Content-Disposition"] = f'attachment; filename="{filename}"'
response["Cache-Control"] = "no-cache"
return response

187
apps/output/test_streaming_chunk_cache.py Normal file
View file

@ -0,0 +1,187 @@
import threading
import time
from unittest import TestCase
from apps.output.streaming_chunk_cache import (
STATUS_BUILDING,
STATUS_READY,
_chunks_key,
_lock_key,
_ready_key,
_status_key,
stream_cached_response,
)
class FakeRedis:
"""Minimal Redis stand-in for chunk-cache unit tests."""
def __init__(self):
self._strings = {}
self._lists = {}
self._expires_at = {}
def _purge_expired(self):
now = time.monotonic()
expired = [key for key, deadline in self._expires_at.items() if deadline <= now]
for key in expired:
self._strings.pop(key, None)
self._lists.pop(key, None)
self._expires_at.pop(key, None)
def get(self, key):
self._purge_expired()
return self._strings.get(key)
def set(self, key, value, nx=False, ex=None):
self._purge_expired()
if nx and key in self._strings:
return None
self._strings[key] = value
if ex is not None:
self._expires_at[key] = time.monotonic() + ex
return True
def delete(self, *keys):
for key in keys:
self._strings.pop(key, None)
self._lists.pop(key, None)
self._expires_at.pop(key, None)
def exists(self, key):
self._purge_expired()
return key in self._strings or key in self._lists
def expire(self, key, ttl):
if key in self._strings or key in self._lists:
self._expires_at[key] = time.monotonic() + ttl
return True
def rpush(self, key, value):
self._lists.setdefault(key, []).append(value)
def lindex(self, key, offset):
items = self._lists.get(key, [])
if offset < len(items):
return items[offset]
return None
def llen(self, key):
return len(self._lists.get(key, []))
def _consume(response):
return b"".join(response.streaming_content).decode("utf-8")
class StreamingChunkCacheTests(TestCase):
def test_leader_caches_chunks_and_sets_ready(self):
redis = FakeRedis()
calls = []
def source():
calls.append(1)
yield "<tv>"
yield "</tv>"
body = _consume(stream_cached_response("cache:test", source, redis=redis))
self.assertEqual(body, "<tv></tv>")
self.assertEqual(calls, [1])
self.assertEqual(redis.get(_ready_key("cache:test")), "1")
self.assertEqual(redis.get(_status_key("cache:test")), STATUS_READY)
self.assertEqual(redis.llen(_chunks_key("cache:test")), 2)
self.assertFalse(redis.exists(_lock_key("cache:test")))
def test_cache_hit_skips_source(self):
redis = FakeRedis()
calls = []
def source():
calls.append(1)
yield "<tv>"
yield "</tv>"
_consume(stream_cached_response("cache:test", source, redis=redis))
calls.clear()
body = _consume(stream_cached_response("cache:test", source, redis=redis))
self.assertEqual(body, "<tv></tv>")
self.assertEqual(calls, [])
def test_follower_reads_leader_chunks_without_rebuilding(self):
redis = FakeRedis()
base = "cache:follow"
leader_started = threading.Event()
rebuild_calls = []
def slow_source():
rebuild_calls.append(1)
leader_started.set()
yield "a"
time.sleep(0.05)
yield "b"
def forbidden_source():
rebuild_calls.append(2)
yield "SHOULD_NOT_RUN"
def leader():
_consume(
stream_cached_response(
base,
slow_source,
redis=redis,
poll_interval=0.01,
)
)
leader_thread = threading.Thread(target=leader)
leader_thread.start()
leader_started.wait(timeout=5)
follower_body = _consume(
stream_cached_response(
base,
forbidden_source,
redis=redis,
poll_interval=0.01,
)
)
leader_thread.join(timeout=5)
self.assertEqual(follower_body, "ab")
self.assertEqual(rebuild_calls, [1])
def test_only_one_leader_when_two_clients_start_together(self):
redis = FakeRedis()
build_calls = []
barrier = threading.Barrier(2)
results = {}
def source():
build_calls.append(threading.current_thread().name)
yield "x"
def worker():
barrier.wait()
results[threading.current_thread().name] = _consume(
stream_cached_response(
"cache:race",
source,
redis=redis,
poll_interval=0.01,
)
)
threads = [
threading.Thread(target=worker, name="t1"),
threading.Thread(target=worker, name="t2"),
]
for thread in threads:
thread.start()
for thread in threads:
thread.join(timeout=10)
self.assertEqual(results["t1"], "x")
self.assertEqual(results["t2"], "x")
self.assertEqual(len(build_calls), 1)

854
apps/output/tests.py
View file

@ -1,31 +1,131 @@
from django.test import TestCase, Client
from django.test import TestCase, Client, SimpleTestCase, RequestFactory
from django.urls import reverse
from apps.channels.models import Channel, ChannelGroup
from unittest import skipUnless
from unittest.mock import patch
from uuid import uuid4
from django.db import connection
from django.test.utils import CaptureQueriesContext
from apps.channels.models import Channel, ChannelGroup, ChannelProfile, ChannelProfileMembership
from apps.epg.models import EPGData, EPGSource
from apps.accounts.models import User
from apps.m3u.models import M3UAccount
from apps.output.views import xc_get_series, xc_get_vod_streams
from apps.vod.models import (
M3UMovieRelation,
M3USeriesRelation,
Movie,
Series,
VODCategory,
VODLogo,
)
import xml.etree.ElementTree as ET
from datetime import timedelta
def _response_text(response):
"""Read body from HttpResponse or StreamingHttpResponse."""
if getattr(response, "streaming", False):
return b"".join(response.streaming_content).decode()
return response.content.decode()
def _epg_response_without_redis(cache_key, source, **kwargs):
"""Test helper: stream EPG directly without Redis chunk caching."""
from django.http import StreamingHttpResponse
response = StreamingHttpResponse(source(), content_type="application/xml")
response["Content-Disposition"] = 'attachment; filename="Dispatcharr.xml"'
response["Cache-Control"] = "no-cache"
return response
class OutputEndpointTestMixin:
"""Isolate HTTP endpoint tests from network ACL, logging, DB teardown, and Redis."""
class OutputM3UTest(TestCase):
def setUp(self):
super().setUp()
self._network_patch = patch(
"apps.output.views.network_access_allowed",
return_value=True,
)
self._epg_teardown_patch = patch("apps.output.epg._epg_export_teardown")
self._log_event_patch = patch("apps.output.views.log_system_event")
self._epg_log_event_patch = patch("apps.output.epg.log_system_event")
self._close_db_patch = patch("django.db.close_old_connections")
self._epg_cache_patch = patch(
"apps.output.epg.stream_cached_response",
side_effect=_epg_response_without_redis,
)
self._network_patch.start()
self._epg_teardown_patch.start()
self._log_event_patch.start()
self._epg_log_event_patch.start()
self._close_db_patch.start()
self._epg_cache_patch.start()
def tearDown(self):
from django.core.cache import cache
cache.clear()
self._epg_cache_patch.stop()
self._close_db_patch.stop()
self._epg_log_event_patch.stop()
self._log_event_patch.stop()
self._epg_teardown_patch.stop()
self._network_patch.stop()
super().tearDown()
def _create_isolated_profile(self, prefix):
"""New profiles auto-include every channel via signal; clear that for tests."""
profile = ChannelProfile.objects.create(name=f"{prefix}-{uuid4().hex[:8]}")
ChannelProfileMembership.objects.filter(channel_profile=profile).delete()
return profile
def _add_channel_to_profile(self, profile, group, **kwargs):
channel = Channel.objects.create(channel_group=group, **kwargs)
ChannelProfileMembership.objects.create(
channel_profile=profile,
channel=channel,
enabled=True,
)
return channel
class OutputM3UTest(OutputEndpointTestMixin, TestCase):
def setUp(self):
super().setUp()
self.client = Client()
self.group = ChannelGroup.objects.create(name=f"M3U Group {uuid4().hex[:8]}")
self.profile = self._create_isolated_profile("m3u")
self._add_channel_to_profile(
self.profile,
self.group,
channel_number=1.0,
name="Test M3U Channel",
)
def _m3u_url(self):
return reverse("output:m3u_endpoint", kwargs={"profile_name": self.profile.name})
def test_generate_m3u_response(self):
"""
Test that the M3U endpoint returns a valid M3U file.
"""
url = reverse('output:generate_m3u')
response = self.client.get(url)
response = self.client.get(self._m3u_url())
self.assertEqual(response.status_code, 200)
content = response.content.decode()
content = _response_text(response)
self.assertIn("#EXTM3U", content)
def test_generate_m3u_response_post_empty_body(self):
"""
Test that a POST request with an empty body returns 200 OK.
"""
url = reverse('output:generate_m3u')
response = self.client.post(url, data=None, content_type='application/x-www-form-urlencoded')
content = response.content.decode()
response = self.client.post(
self._m3u_url(),
data=None,
content_type="application/x-www-form-urlencoded",
)
content = _response_text(response)
self.assertEqual(response.status_code, 200, "POST with empty body should return 200 OK")
self.assertIn("#EXTM3U", content)
@ -34,35 +134,40 @@ class OutputM3UTest(TestCase):
"""
Test that a POST request with a non-empty body returns 403 Forbidden.
"""
url = reverse('output:generate_m3u')
response = self.client.post(url, data={'evilstring': 'muhahaha'})
response = self.client.post(self._m3u_url(), data={"evilstring": "muhahaha"})
self.assertEqual(response.status_code, 403, "POST with body should return 403 Forbidden")
self.assertIn("POST requests with body are not allowed, body is:", response.content.decode())
self.assertIn("POST requests with body are not allowed", _response_text(response))
class OutputEPGXMLEscapingTest(TestCase):
class OutputEPGXMLEscapingTest(OutputEndpointTestMixin, TestCase):
"""Test XML escaping of channel_id attributes in EPG generation"""
def setUp(self):
super().setUp()
self.client = Client()
self.group = ChannelGroup.objects.create(name="Test Group")
self.group = ChannelGroup.objects.create(name=f"Test Group {uuid4().hex[:8]}")
self.profile = self._create_isolated_profile("epg-xml")
def _add_channel(self, **kwargs):
return self._add_channel_to_profile(self.profile, self.group, **kwargs)
def _epg_url(self, query="tvg_id_source=tvg_id&days=0&prev_days=0"):
base = reverse("output:epg_endpoint", kwargs={"profile_name": self.profile.name})
return f"{base}?{query}"
def test_channel_id_with_ampersand(self):
"""Test channel ID with ampersand is properly escaped"""
channel = Channel.objects.create(
self._add_channel(
channel_number=1.0,
name="Test Channel",
tvg_id="News & Sports",
channel_group=self.group
)
url = reverse('output:generate_epg') + '?tvg_id_source=tvg_id'
response = self.client.get(url)
response = self.client.get(self._epg_url())
self.assertEqual(response.status_code, 200)
content = response.content.decode()
content = _response_text(response)
# Should contain escaped ampersand
self.assertIn('id="News &amp; Sports"', content)
@ -76,17 +181,15 @@ class OutputEPGXMLEscapingTest(TestCase):
def test_channel_id_with_angle_brackets(self):
"""Test channel ID with < and > characters"""
channel = Channel.objects.create(
self._add_channel(
channel_number=2.0,
name="HD Channel",
tvg_id="Channel <HD>",
channel_group=self.group
)
url = reverse('output:generate_epg') + '?tvg_id_source=tvg_id'
response = self.client.get(url)
response = self.client.get(self._epg_url())
content = response.content.decode()
content = _response_text(response)
self.assertIn('id="Channel &lt;HD&gt;"', content)
try:
@ -96,23 +199,28 @@ class OutputEPGXMLEscapingTest(TestCase):
def test_channel_id_with_all_special_chars(self):
"""Test channel ID with all XML special characters"""
channel = Channel.objects.create(
expected_id = 'Test & "Special" <Chars>'
self._add_channel(
channel_number=3.0,
name="Complex Channel",
tvg_id='Test & "Special" <Chars>',
channel_group=self.group
tvg_id=expected_id,
)
url = reverse('output:generate_epg') + '?tvg_id_source=tvg_id'
response = self.client.get(url)
response = self.client.get(self._epg_url())
content = response.content.decode()
content = _response_text(response)
self.assertIn('id="Test &amp; &quot;Special&quot; &lt;Chars&gt;"', content)
try:
tree = ET.fromstring(content)
# Verify we can find the channel with correct ID in parsed tree
channel_elem = tree.find('.//channel[@id="Test & \\"Special\\" <Chars>"]')
channel_elem = next(
(
elem
for elem in tree.findall(".//channel")
if elem.get("id") == expected_id
),
None,
)
self.assertIsNotNone(channel_elem)
except ET.ParseError as e:
self.fail(f"Generated EPG with all special chars is not valid XML: {e}")
@ -121,25 +229,689 @@ class OutputEPGXMLEscapingTest(TestCase):
"""Test that programme elements also have escaped channel attributes"""
epg_source = EPGSource.objects.create(name="Test EPG", source_type="dummy")
epg_data = EPGData.objects.create(name="Test EPG Data", epg_source=epg_source)
channel = Channel.objects.create(
self._add_channel(
channel_number=4.0,
name="Program Test",
tvg_id="News & Sports",
epg_data=epg_data,
channel_group=self.group
)
url = reverse('output:generate_epg') + '?tvg_id_source=tvg_id'
response = self.client.get(url)
response = self.client.get(self._epg_url())
content = response.content.decode()
content = _response_text(response)
# Check programme elements have escaped channel attributes
self.assertIn('channel="News &amp; Sports"', content)
try:
tree = ET.fromstring(content)
programmes = tree.findall('.//programme[@channel="News & Sports"]')
programmes = [
programme
for programme in tree.findall(".//programme")
if programme.get("channel") == "News & Sports"
]
self.assertGreater(len(programmes), 0)
except ET.ParseError as e:
self.fail(f"Generated EPG with programme elements is not valid XML: {e}")
def test_programmes_emitted_in_start_time_order(self):
"""Programmes for a channel are emitted in start_time order, not insert order."""
from django.utils import timezone
from apps.epg.models import ProgramData
epg_source = EPGSource.objects.create(name="Real EPG", source_type="xmltv")
epg_data = EPGData.objects.create(name="Station", epg_source=epg_source, tvg_id="station1")
self._add_channel(
channel_number=149.0,
name="Food Network",
tvg_id="station1",
epg_data=epg_data,
)
now = timezone.now()
# Insert out of chronological order so id order != start_time order.
ProgramData.objects.create(
epg=epg_data,
start_time=now + timedelta(days=3),
end_time=now + timedelta(days=3, hours=1),
title="Third",
tvg_id="station1",
)
ProgramData.objects.create(
epg=epg_data,
start_time=now + timedelta(days=1),
end_time=now + timedelta(days=1, hours=1),
title="First",
tvg_id="station1",
)
ProgramData.objects.create(
epg=epg_data,
start_time=now + timedelta(days=2),
end_time=now + timedelta(days=2, hours=1),
title="Second",
tvg_id="station1",
)
content = _response_text(self.client.get(self._epg_url("tvg_id_source=tvg_id&days=7")))
self.assertLess(content.find('<title>First</title>'), content.find('<title>Second</title>'))
self.assertLess(content.find('<title>Second</title>'), content.find('<title>Third</title>'))
class OutputEPGCustomDummyTest(TestCase):
"""Custom dummy EPG must not fall back to default when pattern matched but event is outside window."""
def setUp(self):
self.group = ChannelGroup.objects.create(name="Sports Group")
def test_custom_dummy_outside_window_fills_with_ended_programmes(self):
from django.utils import timezone
from apps.output.views import generate_dummy_programs
epg_source = EPGSource.objects.create(
name="NHL Dummy",
source_type="dummy",
custom_properties={
"title_pattern": r"(?<league>.*)\s\d+:\s(?<team1>.*?)(?:\s+vs\s+)(?<team2>.*?)\s*@.*",
"time_pattern": r"(?<hour>\d{1,2}):(?<minute>\d{2})\s*(?<ampm>AM|PM)",
"date_pattern": r"@ (?<month>[A-Za-z]+)\s+(?<day>\d{1,2})",
"timezone": "US/Eastern",
"program_duration": 180,
},
)
channel_name = (
"NHL 01: Washington Capitals vs Philadelphia Flyers @ April 16 07:30 PM ET"
)
now = timezone.now()
lookback = now - timedelta(days=7)
programs = generate_dummy_programs(
channel_id="nhl01",
channel_name=channel_name,
num_days=7,
epg_source=epg_source,
export_lookback=lookback,
export_cutoff=now + timedelta(days=7),
)
self.assertGreater(len(programs), 0)
self.assertTrue(
all(p['end_time'] >= lookback for p in programs),
"All programmes should fall inside the export window",
)
self.assertTrue(
any('Ended' in p['description'] for p in programs),
"Past events outside the window should still show ended filler",
)
for program in programs:
start = program['start_time']
self.assertEqual(start.second, 0)
self.assertEqual(start.microsecond, 0)
self.assertIn(
start.minute, (0, 30),
"Filler programmes should start on half-hour boundaries",
)
self.assertGreaterEqual(programs[0]['start_time'], lookback)
def test_custom_dummy_future_event_fills_grid_window_with_upcoming(self):
"""Grid-style window: future event should show upcoming filler, not empty."""
from django.utils import timezone
from apps.output.epg import _programme_overlaps_export_window, generate_dummy_programs
epg_source = EPGSource.objects.create(
name="NHL Dummy Future",
source_type="dummy",
custom_properties={
"title_pattern": r"(?<league>.*)\s\d+:\s(?<team1>.*?)(?:\s+vs\s+)(?<team2>.*?)\s*@.*",
"time_pattern": r"(?<hour>\d{1,2}):(?<minute>\d{2})\s*(?<ampm>AM|PM)",
"date_pattern": r"@ (?<month>[A-Za-z]+)\s+(?<day>\d{1,2})",
"timezone": "US/Eastern",
"program_duration": 180,
},
)
now = timezone.now()
grid_start = now - timedelta(hours=1)
grid_end = now + timedelta(hours=24)
future = now + timedelta(days=3)
channel_name = (
f"NHL 01: Washington Capitals vs Philadelphia Flyers @ "
f"{future.strftime('%B')} {future.day} 07:30 PM ET"
)
programs = generate_dummy_programs(
channel_id="nhl01",
channel_name=channel_name,
num_days=1,
epg_source=epg_source,
export_lookback=grid_start,
export_cutoff=grid_end,
)
self.assertGreater(len(programs), 0)
self.assertTrue(
all(
_programme_overlaps_export_window(
p["start_time"], p["end_time"], grid_start, grid_end
)
for p in programs
),
"All programmes should overlap the grid query window",
)
self.assertTrue(
any("Upcoming" in p.get("description", "") for p in programs),
"Future events outside the window should show upcoming filler",
)
class OutputEPGHelperTest(SimpleTestCase):
def test_ceil_to_half_hour_on_boundary(self):
from django.utils import timezone
from apps.output.epg import _ceil_to_half_hour
dt = timezone.now().replace(minute=30, second=0, microsecond=0)
self.assertEqual(_ceil_to_half_hour(dt), dt)
def test_ceil_to_half_hour_rounds_up(self):
from django.utils import timezone
from apps.output.epg import _ceil_to_half_hour
dt = timezone.now().replace(minute=17, second=42, microsecond=123456)
aligned = _ceil_to_half_hour(dt)
self.assertEqual(aligned.minute, 30)
self.assertEqual(aligned.second, 0)
self.assertGreaterEqual(aligned, dt.replace(microsecond=0))
def test_ceil_to_half_hour_past_boundary_second(self):
from django.utils import timezone
from apps.output.epg import _ceil_to_half_hour
dt = timezone.now().replace(minute=0, second=52, microsecond=123456)
aligned = _ceil_to_half_hour(dt)
self.assertEqual(aligned.minute, 30)
self.assertEqual(aligned.second, 0)
self.assertGreaterEqual(aligned, dt.replace(microsecond=0))
class XcVodSeriesDistinctTests(TestCase):
def setUp(self):
self.factory = RequestFactory()
self.user = User.objects.create_user(
username=f"xc-{uuid4().hex[:8]}",
password="pass",
custom_properties={"xc_password": "xcpass"},
)
self.request = self.factory.get("/player_api.php")
def _account(self, name, *, priority=0, is_active=True):
return M3UAccount.objects.create(
name=name,
server_url="http://example.com",
priority=priority,
is_active=is_active,
)
def test_vod_streams_picks_highest_priority_relation(self):
low = self._account(f"low-{uuid4().hex[:6]}", priority=1)
high = self._account(f"high-{uuid4().hex[:6]}", priority=10)
movie = Movie.objects.create(name="Shared Movie", year=2020)
M3UMovieRelation.objects.create(
m3u_account=low,
movie=movie,
stream_id="low-stream",
container_extension="mkv",
)
M3UMovieRelation.objects.create(
m3u_account=high,
movie=movie,
stream_id="high-stream",
container_extension="mp4",
)
streams = xc_get_vod_streams(self.request, self.user)
self.assertEqual(len(streams), 1)
self.assertEqual(streams[0]["name"], "Shared Movie")
self.assertEqual(streams[0]["container_extension"], "mp4")
def test_vod_streams_excludes_inactive_accounts(self):
active = self._account(f"active-{uuid4().hex[:6]}", priority=1)
inactive = self._account(
f"inactive-{uuid4().hex[:6]}", priority=99, is_active=False
)
active_movie = Movie.objects.create(name="Active Movie")
inactive_movie = Movie.objects.create(name="Inactive Only Movie")
M3UMovieRelation.objects.create(
m3u_account=active,
movie=active_movie,
stream_id="active-1",
)
M3UMovieRelation.objects.create(
m3u_account=inactive,
movie=inactive_movie,
stream_id="inactive-1",
)
streams = xc_get_vod_streams(self.request, self.user)
names = {s["name"] for s in streams}
self.assertEqual(names, {"Active Movie"})
def test_vod_streams_category_filter(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
action = VODCategory.objects.create(name="Action", category_type="movie")
comedy = VODCategory.objects.create(name="Comedy", category_type="movie")
action_movie = Movie.objects.create(name="Action Movie")
comedy_movie = Movie.objects.create(name="Comedy Movie")
M3UMovieRelation.objects.create(
m3u_account=account,
movie=action_movie,
category=action,
stream_id="action-1",
)
M3UMovieRelation.objects.create(
m3u_account=account,
movie=comedy_movie,
category=comedy,
stream_id="comedy-1",
)
streams = xc_get_vod_streams(self.request, self.user, category_id=action.id)
self.assertEqual(len(streams), 1)
self.assertEqual(streams[0]["name"], "Action Movie")
self.assertEqual(streams[0]["category_id"], str(action.id))
def test_vod_streams_sorted_alphabetically_by_name(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
zebra = Movie.objects.create(name="Zebra Film")
apple = Movie.objects.create(name="Apple Film")
M3UMovieRelation.objects.create(
m3u_account=account, movie=zebra, stream_id="z-1"
)
M3UMovieRelation.objects.create(
m3u_account=account, movie=apple, stream_id="a-1"
)
streams = xc_get_vod_streams(self.request, self.user)
self.assertEqual([s["name"] for s in streams], ["Apple Film", "Zebra Film"])
def test_vod_streams_includes_metadata_fields(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
movie = Movie.objects.create(
name="Rich Movie",
description="A plot",
genre="Drama",
year=2021,
rating="8",
custom_properties={
"director": "Dir",
"actors": "Cast",
"release_date": "2021-01-01",
"youtube_trailer": "yt123",
},
)
M3UMovieRelation.objects.create(
m3u_account=account,
movie=movie,
stream_id="rich-1",
container_extension="avi",
)
stream = xc_get_vod_streams(self.request, self.user)[0]
self.assertEqual(stream["plot"], "A plot")
self.assertEqual(stream["genre"], "Drama")
self.assertEqual(stream["year"], 2021)
self.assertEqual(stream["director"], "Dir")
self.assertEqual(stream["cast"], "Cast")
self.assertEqual(stream["release_date"], "2021-01-01")
self.assertEqual(stream["trailer"], "yt123")
self.assertEqual(stream["container_extension"], "avi")
def test_vod_streams_stream_icon_uses_logo_id_without_logo_join(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
logo = VODLogo.objects.create(name="Poster", url="http://example.com/poster.png")
movie = Movie.objects.create(name="Logo Movie", logo=logo)
M3UMovieRelation.objects.create(
m3u_account=account,
movie=movie,
stream_id="logo-1",
)
stream = xc_get_vod_streams(self.request, self.user)[0]
self.assertIn(f"/{logo.id}/", stream["stream_icon"])
def test_series_picks_highest_priority_relation(self):
low = self._account(f"low-{uuid4().hex[:6]}", priority=1)
high = self._account(f"high-{uuid4().hex[:6]}", priority=10)
series = Series.objects.create(name="Shared Series", year=2019)
M3USeriesRelation.objects.create(
m3u_account=low,
series=series,
external_series_id="low-series",
)
high_rel = M3USeriesRelation.objects.create(
m3u_account=high,
series=series,
external_series_id="high-series",
)
results = xc_get_series(self.request, self.user)
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["name"], "Shared Series")
self.assertEqual(results[0]["series_id"], high_rel.id)
def test_series_excludes_inactive_accounts(self):
active = self._account(f"active-{uuid4().hex[:6]}")
inactive = self._account(f"inactive-{uuid4().hex[:6]}", is_active=False)
active_series = Series.objects.create(name="Active Series")
inactive_series = Series.objects.create(name="Inactive Only Series")
M3USeriesRelation.objects.create(
m3u_account=active,
series=active_series,
external_series_id="active-s",
)
M3USeriesRelation.objects.create(
m3u_account=inactive,
series=inactive_series,
external_series_id="inactive-s",
)
results = xc_get_series(self.request, self.user)
self.assertEqual({r["name"] for r in results}, {"Active Series"})
def test_series_sorted_alphabetically_by_name(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
z = Series.objects.create(name="Zulu Show")
a = Series.objects.create(name="Alpha Show")
M3USeriesRelation.objects.create(
m3u_account=account, series=z, external_series_id="z"
)
M3USeriesRelation.objects.create(
m3u_account=account, series=a, external_series_id="a"
)
results = xc_get_series(self.request, self.user)
self.assertEqual([r["name"] for r in results], ["Alpha Show", "Zulu Show"])
@skipUnless(connection.vendor == "postgresql", "PostgreSQL-specific query shape")
def test_vod_streams_dedupe_query_avoids_movie_join(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
movie = Movie.objects.create(name="Query Shape Movie")
M3UMovieRelation.objects.create(
m3u_account=account, movie=movie, stream_id="qs-1"
)
with CaptureQueriesContext(connection) as ctx:
xc_get_vod_streams(self.request, self.user)
distinct_queries = [q for q in ctx.captured_queries if "DISTINCT" in q["sql"]]
self.assertEqual(len(distinct_queries), 1)
self.assertNotIn('"vod_movie"', distinct_queries[0]["sql"])
self.assertNotIn('"vod_vodlogo"', distinct_queries[0]["sql"])
fetch_queries = [
q
for q in ctx.captured_queries
if '"vod_movie"' in q["sql"] and "DISTINCT" not in q["sql"]
]
self.assertGreaterEqual(len(fetch_queries), 1)
fetch_sql = fetch_queries[0]["sql"]
self.assertNotIn('"vod_vodlogo"', fetch_sql)
self.assertNotIn('"vod_vodcategory"', fetch_sql)
@skipUnless(connection.vendor == "postgresql", "PostgreSQL-specific query shape")
def test_series_dedupe_query_avoids_series_join(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
series = Series.objects.create(name="Query Shape Series")
M3USeriesRelation.objects.create(
m3u_account=account, series=series, external_series_id="qs-s"
)
with CaptureQueriesContext(connection) as ctx:
xc_get_series(self.request, self.user)
distinct_queries = [q for q in ctx.captured_queries if "DISTINCT" in q["sql"]]
self.assertEqual(len(distinct_queries), 1)
self.assertNotIn('"vod_series"', distinct_queries[0]["sql"])
fetch_queries = [
q
for q in ctx.captured_queries
if '"vod_series"' in q["sql"] and "DISTINCT" not in q["sql"]
]
self.assertGreaterEqual(len(fetch_queries), 1)
fetch_sql = fetch_queries[0]["sql"]
self.assertNotIn('"vod_vodlogo"', fetch_sql)
self.assertNotIn('"vod_vodcategory"', fetch_sql)
XC_VOD_STREAM_KEYS = frozenset({
"num", "name", "stream_type", "stream_id", "stream_icon", "rating",
"rating_5based", "added", "is_adult", "tmdb_id", "imdb_id", "trailer",
"plot", "genre", "year", "director", "cast", "release_date", "category_id",
"category_ids", "container_extension", "custom_sid", "direct_source",
})
XC_SERIES_KEYS = frozenset({
"num", "name", "series_id", "cover", "plot", "cast", "director", "genre",
"release_date", "releaseDate", "last_modified", "rating", "rating_5based",
"backdrop_path", "youtube_trailer", "episode_run_time", "category_id",
"category_ids", "tmdb_id", "imdb_id",
})
class XcVodSeriesRegressionTests(TestCase):
"""Full output-shape and edge-case regressions for XC list endpoints."""
def setUp(self):
self.factory = RequestFactory()
self.user = User.objects.create_user(
username=f"xc-reg-{uuid4().hex[:8]}",
password="pass",
custom_properties={"xc_password": "xcpass"},
)
self.request = self.factory.get("/player_api.php")
def _account(self, name, *, priority=0):
return M3UAccount.objects.create(
name=name,
server_url="http://example.com",
priority=priority,
)
def test_vod_streams_empty_library(self):
self.assertEqual(xc_get_vod_streams(self.request, self.user), [])
def test_series_empty_library(self):
self.assertEqual(xc_get_series(self.request, self.user), [])
def test_vod_streams_response_keys(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
movie = Movie.objects.create(name="Schema Movie", rating="10")
M3UMovieRelation.objects.create(
m3u_account=account, movie=movie, stream_id="schema-1"
)
stream = xc_get_vod_streams(self.request, self.user)[0]
self.assertEqual(set(stream.keys()), XC_VOD_STREAM_KEYS)
self.assertEqual(stream["stream_type"], "movie")
self.assertEqual(stream["stream_id"], movie.id)
self.assertEqual(stream["rating_5based"], 5.0)
self.assertEqual(stream["custom_sid"], None)
self.assertEqual(stream["direct_source"], "")
def test_vod_streams_null_optional_fields(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
movie = Movie.objects.create(name="Sparse Movie")
M3UMovieRelation.objects.create(
m3u_account=account,
movie=movie,
stream_id="sparse-1",
container_extension=None,
)
stream = xc_get_vod_streams(self.request, self.user)[0]
self.assertIsNone(stream["stream_icon"])
self.assertEqual(stream["category_id"], "0")
self.assertEqual(stream["category_ids"], [])
self.assertEqual(stream["container_extension"], "mp4")
self.assertEqual(stream["plot"], "")
self.assertEqual(stream["trailer"], "")
self.assertEqual(stream["tmdb_id"], "")
self.assertEqual(stream["imdb_id"], "")
def test_vod_streams_category_from_winning_relation(self):
"""Category must come from the highest-priority relation, not any relation."""
low = self._account(f"low-{uuid4().hex[:6]}", priority=1)
high = self._account(f"high-{uuid4().hex[:6]}", priority=10)
action = VODCategory.objects.create(name="Action", category_type="movie")
comedy = VODCategory.objects.create(name="Comedy", category_type="movie")
movie = Movie.objects.create(name="Dual Category Movie")
M3UMovieRelation.objects.create(
m3u_account=low,
movie=movie,
category=action,
stream_id="low-cat",
)
M3UMovieRelation.objects.create(
m3u_account=high,
movie=movie,
category=comedy,
stream_id="high-cat",
)
stream = xc_get_vod_streams(self.request, self.user)[0]
self.assertEqual(stream["category_id"], str(comedy.id))
self.assertEqual(stream["category_ids"], [comedy.id])
def test_series_response_keys_and_metadata(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
logo = VODLogo.objects.create(name="Cover", url="http://example.com/cover.png")
category = VODCategory.objects.create(name="Drama", category_type="series")
series = Series.objects.create(
name="Schema Series",
description="Series plot",
genre="Sci-Fi",
year=2022,
rating="8",
tmdb_id="tm123",
imdb_id="tt123",
logo=logo,
custom_properties={
"cast": "Actor A",
"director": "Director B",
"release_date": "2022-06-01",
"backdrop_path": ["/img1.jpg"],
"youtube_trailer": "yt-series",
"episode_run_time": "45",
},
)
relation = M3USeriesRelation.objects.create(
m3u_account=account,
series=series,
category=category,
external_series_id="schema-s",
)
row = xc_get_series(self.request, self.user)[0]
self.assertEqual(set(row.keys()), XC_SERIES_KEYS)
self.assertEqual(row["series_id"], relation.id)
self.assertIn(f"/{logo.id}/", row["cover"])
self.assertEqual(row["plot"], "Series plot")
self.assertEqual(row["cast"], "Actor A")
self.assertEqual(row["director"], "Director B")
self.assertEqual(row["genre"], "Sci-Fi")
self.assertEqual(row["release_date"], "2022-06-01")
self.assertEqual(row["releaseDate"], "2022-06-01")
self.assertEqual(row["backdrop_path"], ["/img1.jpg"])
self.assertEqual(row["youtube_trailer"], "yt-series")
self.assertEqual(row["episode_run_time"], "45")
self.assertEqual(row["tmdb_id"], "tm123")
self.assertEqual(row["imdb_id"], "tt123")
self.assertEqual(row["category_id"], str(category.id))
self.assertEqual(row["category_ids"], [category.id])
self.assertEqual(row["last_modified"], str(int(relation.updated_at.timestamp())))
def test_series_null_optional_fields(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
series = Series.objects.create(name="Sparse Series")
M3USeriesRelation.objects.create(
m3u_account=account,
series=series,
external_series_id="sparse-s",
)
row = xc_get_series(self.request, self.user)[0]
self.assertIsNone(row["cover"])
self.assertEqual(row["category_id"], "0")
self.assertEqual(row["category_ids"], [])
self.assertEqual(row["release_date"], "")
self.assertEqual(row["releaseDate"], "")
self.assertEqual(row["backdrop_path"], [])
self.assertEqual(row["youtube_trailer"], "")
self.assertEqual(row["episode_run_time"], "")
def test_series_release_date_falls_back_to_year(self):
account = self._account(f"acct-{uuid4().hex[:6]}")
series = Series.objects.create(name="Year Only", year=2018)
M3USeriesRelation.objects.create(
m3u_account=account,
series=series,
external_series_id="year-s",
)
row = xc_get_series(self.request, self.user)[0]
self.assertEqual(row["release_date"], "2018")
self.assertEqual(row["releaseDate"], "2018")
def test_priority_tiebreaker_uses_lower_relation_id(self):
"""Same priority: DISTINCT ON tie-breaks on relation id ascending."""
a1 = self._account(f"a1-{uuid4().hex[:6]}", priority=5)
a2 = self._account(f"a2-{uuid4().hex[:6]}", priority=5)
movie = Movie.objects.create(name="Tie Movie")
first = M3UMovieRelation.objects.create(
m3u_account=a1,
movie=movie,
stream_id="first",
container_extension="mkv",
)
M3UMovieRelation.objects.create(
m3u_account=a2,
movie=movie,
stream_id="second",
container_extension="mp4",
)
stream = xc_get_vod_streams(self.request, self.user)[0]
self.assertEqual(stream["container_extension"], first.container_extension)
class GenerateEpgPrevDaysTests(SimpleTestCase):
"""Profile EPG keeps legacy prev_days=0 unless URL or user setting says otherwise."""
def setUp(self):
self.factory = RequestFactory()
@patch("apps.output.epg.stream_cached_response")
@patch("apps.output.epg.Channel.objects")
def test_non_xc_epg_defaults_prev_days_to_zero(self, _channels, mock_cache):
from apps.output.epg import generate_epg
mock_cache.side_effect = lambda cache_key, _source, **_kwargs: cache_key
request = self.factory.get("/epg/")
cache_key = generate_epg(request, profile_name="test", user=None)
self.assertIn(":p=0:", cache_key)

1921
apps/output/views.py
View file

File diff suppressed because it is too large Load diff

48
apps/plugins/api_views.py
View file

@ -335,16 +335,28 @@ def _save_fetched_manifest_to_repo(repo, data, verified):
return None
def _resolve_manifest_base_urls(manifest: dict) -> tuple[str, str]:
"""Return (download_base_url, metadata_base_url) from a manifest dict.
Both fields fall back to root_url when not set. All base URL fields are
optional; callers must guard against empty strings before building URLs.
"""
root_url = manifest.get("root_url", "").rstrip("/")
download_base_url = manifest.get("download_base_url", "").rstrip("/") or root_url
metadata_base_url = manifest.get("metadata_base_url", "").rstrip("/") or root_url
return download_base_url, metadata_base_url
def _invalidate_plugin_detail_cache(repo_id, manifest_data):
manifest = manifest_data.get("manifest", manifest_data)
root_url = manifest.get("root_url", "").rstrip("/")
_, metadata_base_url = _resolve_manifest_base_urls(manifest)
keys = []
for p in manifest.get("plugins", []):
url = p.get("manifest_url", "")
if not url:
continue
if root_url and not url.startswith(("http://", "https://")):
url = f"{root_url}/{url}"
if metadata_base_url and not url.startswith(("http://", "https://")):
url = f"{metadata_base_url}/{url}"
keys.append(f"plugin_detail:{repo_id}:{hashlib.md5(url.encode()).hexdigest()}")
if keys:
cache.delete_many(keys)
@ -1045,18 +1057,28 @@ class AvailablePluginsAPIView(PluginAuthMixin, APIView):
for repo in repos:
manifest_data = repo.cached_manifest or {}
manifest = manifest_data.get("manifest", manifest_data)
root_url = manifest.get("root_url", "").rstrip("/")
download_base_url, metadata_base_url = _resolve_manifest_base_urls(manifest)
registry_url = manifest.get("registry_url", "").rstrip("/")
repo_plugins = manifest.get("plugins", [])
for p in repo_plugins:
slug = p.get("slug", "")
plugin_data = {**p}
# Resolve relative URLs against root_url; absolute URLs pass through
if root_url:
for url_field in ("manifest_url", "latest_url", "icon_url"):
# Resolve relative URLs; metadata and download assets use separate bases
if metadata_base_url:
for url_field in ("manifest_url", "icon_url"):
val = plugin_data.get(url_field, "")
if val and not val.startswith(("http://", "https://")):
plugin_data[url_field] = f"{root_url}/{val}"
plugin_data[url_field] = f"{metadata_base_url}/{val}"
if download_base_url:
val = plugin_data.get("latest_url", "")
if val and not val.startswith(("http://", "https://")):
plugin_data["latest_url"] = f"{download_base_url}/{val}"
# Fallback icon_url: if metadata_base_url is explicitly set and manifest_url
# is known, assume logo.png lives in the same directory as the per-plugin
# manifest. Guard against root_url-only manifests to preserve the GitHub fallback.
if not plugin_data.get("icon_url") and manifest.get("metadata_base_url") and plugin_data.get("manifest_url"):
manifest_dir = plugin_data["manifest_url"].rsplit("/", 1)[0]
plugin_data["icon_url"] = f"{manifest_dir}/logo.png"
# Fallback icon_url from main branch when not provided
if not plugin_data.get("icon_url") and registry_url:
# registry_url is e.g. https://github.com/Dispatcharr/Plugins
@ -1161,18 +1183,18 @@ class PluginDetailManifestAPIView(PluginAuthMixin, APIView):
# Resolve relative URLs in versions
repo_manifest = repo.cached_manifest or {}
inner = repo_manifest.get("manifest", repo_manifest)
root_url = inner.get("root_url", "").rstrip("/")
download_base_url, _ = _resolve_manifest_base_urls(inner)
if root_url and isinstance(manifest_obj.get("versions"), list):
if download_base_url and isinstance(manifest_obj.get("versions"), list):
for v in manifest_obj["versions"]:
url_val = v.get("url", "")
if url_val and not url_val.startswith(("http://", "https://")):
v["url"] = f"{root_url}/{url_val}"
if root_url and isinstance(manifest_obj.get("latest"), dict):
v["url"] = f"{download_base_url}/{url_val}"
if download_base_url and isinstance(manifest_obj.get("latest"), dict):
for url_field in ("url", "latest_url"):
url_val = manifest_obj["latest"].get(url_field, "")
if url_val and not url_val.startswith(("http://", "https://")):
manifest_obj["latest"][url_field] = f"{root_url}/{url_val}"
manifest_obj["latest"][url_field] = f"{download_base_url}/{url_val}"
result = {
"manifest": manifest_obj,

162
apps/plugins/loader.py
View file

@ -8,9 +8,9 @@ import sys
import threading
import types
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
from typing import Any, Dict, Iterator, List, Optional, Tuple
from django.db import transaction
from django.db import close_old_connections, transaction
from .models import PluginConfig
@ -92,6 +92,29 @@ class PluginManager:
key: lp.path for key, lp in self._registry.items() if lp and lp.path
}
try:
return self._discover_plugins_impl(
sync_db=sync_db,
force_reload=force_reload,
previous_packages=previous_packages,
previous_aliases=previous_aliases,
previous_paths=previous_paths,
token=token,
)
finally:
# Discovery runs outside Django's request/task cycle (boot, worker_ready).
close_old_connections()
def _discover_plugins_impl(
self,
*,
sync_db: bool,
force_reload: bool,
previous_packages: Dict[str, str],
previous_aliases: Dict[str, str],
previous_paths: Dict[str, str],
token: int,
) -> Dict[str, LoadedPlugin]:
try:
configs: Optional[Dict[str, PluginConfig]] = None
try:
@ -247,6 +270,23 @@ class PluginManager:
logger.exception("Deferring plugin DB sync; database not ready yet")
return self._registry
def iter_actions_for_event(self, event_name: str) -> Iterator[Tuple[str, str]]:
"""Yield (plugin_key, action_id) pairs from the in-memory registry."""
with self._lock:
registry = list(self._registry.items())
for key, lp in registry:
for action in lp.actions or []:
if not isinstance(action, dict):
continue
action_id = action.get("id")
events = action.get("events")
if (
action_id
and isinstance(events, (list, tuple))
and event_name in events
):
yield key, action_id
def _load_plugin(
self,
key: str,
@ -492,79 +532,85 @@ class PluginManager:
return cfg.settings
def run_action(self, key: str, action_id: str, params: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
lp = self.get_plugin(key)
if not lp or not lp.instance:
# Attempt a lightweight re-discovery in case the registry was rebuilt
self.discover_plugins(sync_db=False, force_reload=False, use_cache=False)
try:
lp = self.get_plugin(key)
if not lp or not lp.instance:
raise ValueError(f"Plugin '{key}' not found")
# Attempt a lightweight re-discovery in case the registry was rebuilt
self.discover_plugins(sync_db=False, force_reload=False, use_cache=False)
lp = self.get_plugin(key)
if not lp or not lp.instance:
raise ValueError(f"Plugin '{key}' not found")
cfg = PluginConfig.objects.get(key=key)
if not cfg.enabled:
raise PermissionError(f"Plugin '{key}' is disabled")
params = params or {}
cfg = PluginConfig.objects.get(key=key)
if not cfg.enabled:
raise PermissionError(f"Plugin '{key}' is disabled")
params = params or {}
context = self._build_context(lp, cfg)
context = self._build_context(lp, cfg)
# Run either via Celery if plugin provides a delayed method, or inline
run_method = getattr(lp.instance, "run", None)
if not callable(run_method):
raise ValueError(f"Plugin '{key}' has no runnable 'run' method")
run_method = getattr(lp.instance, "run", None)
if not callable(run_method):
raise ValueError(f"Plugin '{key}' has no runnable 'run' method")
try:
result = run_method(action_id, params, context)
except Exception:
logger.exception(f"Plugin '{key}' action '{action_id}' failed")
raise
try:
result = run_method(action_id, params, context)
except Exception:
logger.exception(f"Plugin '{key}' action '{action_id}' failed")
raise
# Normalize return
if isinstance(result, dict):
return result
return {"status": "ok", "result": result}
if isinstance(result, dict):
return result
return {"status": "ok", "result": result}
finally:
# Return geventpool checkouts for this greenlet/thread after every action,
# including Connect event hooks and manual UI runs.
close_old_connections()
def stop_plugin(self, key: str, reason: Optional[str] = None) -> bool:
lp = self.get_plugin(key)
if not lp or not lp.instance:
return False
try:
cfg = PluginConfig.objects.get(key=key)
except PluginConfig.DoesNotExist:
return False
if not cfg.enabled:
return False
context = self._build_context(lp, cfg)
if reason:
context["reason"] = reason
stop_method = getattr(lp.instance, "stop", None)
if callable(stop_method):
lp = self.get_plugin(key)
if not lp or not lp.instance:
return False
try:
stop_method(context)
return True
except TypeError:
cfg = PluginConfig.objects.get(key=key)
except PluginConfig.DoesNotExist:
return False
if not cfg.enabled:
return False
context = self._build_context(lp, cfg)
if reason:
context["reason"] = reason
stop_method = getattr(lp.instance, "stop", None)
if callable(stop_method):
try:
stop_method()
stop_method(context)
return True
except TypeError:
try:
stop_method()
return True
except Exception:
logger.exception("Plugin '%s' stop() failed", key)
return False
except Exception:
logger.exception("Plugin '%s' stop() failed", key)
return False
except Exception:
logger.exception("Plugin '%s' stop() failed", key)
return False
run_method = getattr(lp.instance, "run", None)
if callable(run_method):
actions = {a.get("id") for a in (lp.actions or []) if isinstance(a, dict)}
if "stop" in actions:
try:
run_method("stop", {}, context)
return True
except Exception:
logger.exception("Plugin '%s' stop action failed", key)
return False
return False
run_method = getattr(lp.instance, "run", None)
if callable(run_method):
actions = {a.get("id") for a in (lp.actions or []) if isinstance(a, dict)}
if "stop" in actions:
try:
run_method("stop", {}, context)
return True
except Exception:
logger.exception("Plugin '%s' stop action failed", key)
return False
return False
finally:
close_old_connections()
def stop_all_plugins(self, reason: Optional[str] = None) -> int:
stopped = 0

0
apps/plugins/tests/__init__.py Normal file
View file

51
apps/plugins/tests/test_iter_actions_for_event.py Normal file
View file

@ -0,0 +1,51 @@
from django.test import SimpleTestCase
from apps.plugins.loader import LoadedPlugin, PluginManager
class IterActionsForEventTests(SimpleTestCase):
def test_yields_matching_handlers(self):
pm = PluginManager()
pm._registry = {
"alpha": LoadedPlugin(
key="alpha",
name="Alpha",
actions=[
{"id": "on_start", "events": ["channel_start"]},
{"id": "manual"},
],
),
"beta": LoadedPlugin(
key="beta",
name="Beta",
actions=[
{"id": "on_both", "events": ["channel_start", "channel_stop"]},
],
),
}
self.assertEqual(
list(pm.iter_actions_for_event("channel_start")),
[("alpha", "on_start"), ("beta", "on_both")],
)
self.assertEqual(list(pm.iter_actions_for_event("channel_stop")), [("beta", "on_both")])
def test_ignores_string_events_value(self):
pm = PluginManager()
pm._registry = {
"bad": LoadedPlugin(
key="bad",
name="Bad",
actions=[{"id": "hook", "events": "client_connect"}],
),
"good": LoadedPlugin(
key="good",
name="Good",
actions=[{"id": "hook", "events": ["client_connect"]}],
),
}
self.assertEqual(
list(pm.iter_actions_for_event("client_connect")),
[("good", "hook")],
)

81
apps/plugins/tests/test_run_action_db_cleanup.py Normal file
View file

@ -0,0 +1,81 @@
"""PluginManager must release geventpool checkouts after every run/stop."""
from contextlib import contextmanager
from unittest.mock import MagicMock, patch
from django.test import SimpleTestCase
from apps.plugins.loader import LoadedPlugin, PluginManager
class PluginRunActionDbCleanupTests(SimpleTestCase):
@contextmanager
def _manager_with_plugin(self, run_impl):
instance = MagicMock()
instance.run = run_impl
lp = LoadedPlugin(
key="test_plugin",
name="Test Plugin",
instance=instance,
actions=[{"id": "do_work"}],
)
pm = PluginManager()
cfg = MagicMock(enabled=True, settings={})
with patch.object(pm, "get_plugin", return_value=lp), patch(
"apps.plugins.loader.PluginConfig.objects.get", return_value=cfg
):
yield pm
@patch("apps.plugins.loader.close_old_connections")
def test_run_action_closes_connections_on_success(self, mock_close):
with self._manager_with_plugin(lambda *_a, **_k: {"status": "ok"}) as pm:
result = pm.run_action("test_plugin", "do_work")
self.assertEqual(result, {"status": "ok"})
mock_close.assert_called_once()
@patch("apps.plugins.loader.close_old_connections")
def test_run_action_closes_connections_on_plugin_error(self, mock_close):
def _boom(*_a, **_k):
raise RuntimeError("plugin failed")
with self._manager_with_plugin(_boom) as pm:
with self.assertRaises(RuntimeError):
pm.run_action("test_plugin", "do_work")
mock_close.assert_called_once()
@patch("apps.plugins.loader.close_old_connections")
def test_stop_plugin_closes_connections(self, mock_close):
instance = MagicMock()
instance.stop = MagicMock()
lp = LoadedPlugin(
key="test_plugin",
name="Test Plugin",
instance=instance,
)
pm = PluginManager()
cfg = MagicMock(enabled=True, settings={})
with patch.object(pm, "get_plugin", return_value=lp), patch(
"apps.plugins.loader.PluginConfig.objects.get", return_value=cfg
):
self.assertTrue(pm.stop_plugin("test_plugin", reason="shutdown"))
mock_close.assert_called_once()
class PluginDiscoverDbCleanupTests(SimpleTestCase):
@patch("apps.plugins.loader.close_old_connections")
@patch.object(PluginManager, "_discover_plugins_impl", return_value={})
def test_discover_plugins_closes_connections(self, _mock_impl, mock_close):
pm = PluginManager()
pm.discover_plugins(sync_db=False)
mock_close.assert_called_once()
@patch("apps.plugins.loader.close_old_connections")
def test_discover_plugins_cache_hit_skips_close(self, mock_close):
pm = PluginManager()
pm._discovery_completed = True
pm._last_reload_token = pm._get_reload_token()
pm.discover_plugins(sync_db=False, use_cache=True)
mock_close.assert_not_called()

18
apps/proxy/config.py
View file

@ -42,7 +42,8 @@ class BaseConfig:
"buffering_speed": 1.0,
"redis_chunk_ttl": 60,
"channel_shutdown_delay": 0,
"channel_init_grace_period": 5,
"channel_init_grace_period": 60,
"channel_client_wait_period": 5,
"new_client_behind_seconds": 5,
}
@ -133,9 +134,15 @@ class TSConfig(BaseConfig):
@classmethod
def get_channel_init_grace_period(cls):
"""Get channel init grace period from database or default"""
"""Max seconds to wait for initial buffer fill during channel startup."""
settings = cls.get_proxy_settings()
return settings.get("channel_init_grace_period", 5)
return settings.get("channel_init_grace_period", 60)
@classmethod
def get_channel_client_wait_period(cls):
"""Seconds to keep a ready channel alive waiting for the first client to connect."""
settings = cls.get_proxy_settings()
return settings.get("channel_client_wait_period", 5)
# Dynamic property access for these settings
@property
@ -154,5 +161,6 @@ class TSConfig(BaseConfig):
def CHANNEL_INIT_GRACE_PERIOD(self):
return self.get_channel_init_grace_period()
@property
def CHANNEL_CLIENT_WAIT_PERIOD(self):
return self.get_channel_client_wait_period()

14
apps/proxy/live_proxy/channel_status.py
View file

@ -407,6 +407,20 @@ class ChannelStatus:
if channel_name:
info['channel_name'] = channel_name
info['is_timeshift'] = bool(metadata.get(ChannelMetadataField.IS_TIMESHIFT))
for key, field in (
('logo_id', ChannelMetadataField.LOGO_ID),
('m3u_profile_id', ChannelMetadataField.M3U_PROFILE),
):
raw = metadata.get(field)
if not raw:
continue
try:
info[key] = int(raw)
except (TypeError, ValueError):
pass
stream_id_bytes = metadata.get(ChannelMetadataField.STREAM_ID)
if stream_id_bytes:
try:

148
apps/proxy/live_proxy/client_manager.py
View file

@ -102,6 +102,7 @@ class ClientManager:
break
# Send heartbeat for all local clients
clients_to_remove = set()
with self.lock:
# Skip this cycle if we have no local clients
if not self.clients:
@ -109,7 +110,6 @@ class ClientManager:
# IMPROVED GHOST DETECTION: Check for stale clients before sending heartbeats
current_time = time.time()
clients_to_remove = set()
# First identify clients that should be removed
for client_id in self.clients:
@ -132,12 +132,11 @@ class ClientManager:
logger.debug(f"Client {client_id} inactive for {current_time - last_active_time:.1f}s, removing as ghost")
clients_to_remove.add(client_id)
# Remove ghost clients in a separate step
for client_id in clients_to_remove:
self.remove_client(client_id)
if clients_to_remove:
logger.info(f"Removed {len(clients_to_remove)} ghost clients from channel {self.channel_id}")
logger.info(
f"Identified {len(clients_to_remove)} ghost clients "
f"for channel {self.channel_id}"
)
# Now send heartbeats only for remaining clients
pipe = self.redis_client.pipeline()
@ -172,6 +171,9 @@ class ClientManager:
if self.clients and not all(c in clients_to_remove for c in self.clients):
self._notify_owner_of_activity()
for client_id in clients_to_remove:
self.remove_client(client_id)
except Exception as e:
logger.error(f"Error in client heartbeat thread: {e}")
@ -254,6 +256,14 @@ class ClientManager:
# Store in Redis
if self.redis_client:
from .services.channel_service import ChannelService
if ChannelService.cancel_pending_shutdown(self.channel_id):
logger.info(
f"Cancelled pending shutdown for channel {self.channel_id} "
f"(client {client_id} reconnected)"
)
self.redis_client.hset(client_key, mapping=client_data)
self.redis_client.expire(client_key, self.client_ttl)
@ -291,6 +301,8 @@ class ClientManager:
# Trigger channel stats update via WebSocket
self._trigger_stats_update()
ChannelService.promote_channel_when_buffer_ready(self.channel_id)
# Get total clients across all workers
total_clients = self.get_total_client_count()
logger.info(f"New client connected: {client_id} (local: {len(self.clients)}, total: {total_clients})")
@ -300,11 +312,17 @@ class ClientManager:
return len(self.clients)
except Exception as e:
logger.error(f"Error adding client {client_id}: {e}")
logger.error(f"Error adding client {client_id}: {e}", exc_info=True)
with self.lock:
self.clients.discard(client_id)
self._registered_clients.discard(client_id)
return False
def remove_client(self, client_id):
"""Remove a client from this channel and Redis"""
client_username = "unknown"
remaining = None
with self.lock:
if client_id in self.clients:
self.clients.remove(client_id)
@ -315,61 +333,81 @@ class ClientManager:
self.last_active_time = time.time()
if self.redis_client:
# Get client data before removing the data
client_key = f"live:channel:{self.channel_id}:clients:{client_id}"
client_username = self.redis_client.hget(client_key, "username") or "unknown"
if isinstance(client_username, bytes):
client_username = client_username.decode("utf-8")
# Remove from channel's client set
self.redis_client.srem(self.client_set_key, client_id)
client_key = f"live:channel:{self.channel_id}:clients:{client_id}"
self.redis_client.delete(client_key)
# Check if this was the last client
remaining = self.redis_client.scard(self.client_set_key) or 0
if remaining == 0:
logger.warning(f"Last client removed: {client_id} - channel may shut down soon")
# Trigger disconnect time tracking even if we're not the owner
disconnect_key = RedisKeys.last_client_disconnect(self.channel_id)
self.redis_client.setex(disconnect_key, 60, str(time.time()))
local_count = len(self.clients)
self._notify_owner_of_activity()
if not self.redis_client:
return local_count
# Check if we're the owner - if so, handle locally; if not, publish event
am_i_owner = self.proxy_server and self.proxy_server.am_i_owner(self.channel_id)
if remaining == 0:
logger.warning(f"Last client removed: {client_id} - channel may shut down soon")
disconnect_key = RedisKeys.last_client_disconnect(self.channel_id)
ttl = max(int(ConfigHelper.channel_shutdown_delay() * 2), 60)
self.redis_client.setex(disconnect_key, ttl, str(time.time()))
if am_i_owner:
# We're the owner - handle the disconnect directly
logger.debug(f"Owner handling CLIENT_DISCONNECTED for client {client_id} locally (not publishing)")
if (remaining == 0
or self.proxy_server.output_managers.get(self.channel_id)
or self.proxy_server.profile_managers.get(self.channel_id)):
import gevent
gevent.spawn(self.proxy_server.handle_client_disconnect, self.channel_id)
else:
# We're not the owner - publish event so owner can handle it
logger.debug(f"Non-owner publishing CLIENT_DISCONNECTED event for client {client_id} on channel {self.channel_id} from worker {self.worker_id}")
event_data = json.dumps({
"event": EventType.CLIENT_DISCONNECTED,
"channel_id": self.channel_id,
"client_id": client_id,
"worker_id": self.worker_id or "unknown",
"timestamp": time.time(),
"remaining_clients": remaining,
"username": client_username
})
self.redis_client.publish(RedisKeys.events_channel(self.channel_id), event_data)
self._notify_owner_of_activity()
# Trigger channel stats update via WebSocket
self._trigger_stats_update()
am_i_owner = self.proxy_server and self.proxy_server.am_i_owner(self.channel_id)
total_clients = self.get_total_client_count()
logger.info(f"Client disconnected: {client_id} (local: {len(self.clients)}, total: {total_clients})")
# Owner lock TTL can expire while local ffmpeg is still running on this worker.
if (not am_i_owner and self.proxy_server
and self.proxy_server._has_local_upstream_activity(self.channel_id)):
if self.proxy_server.extend_ownership(self.channel_id):
am_i_owner = True
return len(self.clients)
schedule_disconnect = False
if am_i_owner:
logger.debug(
f"Owner handling CLIENT_DISCONNECTED for client {client_id} locally (not publishing)"
)
if (remaining == 0
or self.proxy_server.output_managers.get(self.channel_id)
or self.proxy_server.profile_managers.get(self.channel_id)):
schedule_disconnect = True
elif (remaining == 0 and self.proxy_server
and self.proxy_server._has_local_upstream_activity(self.channel_id)):
logger.warning(
f"Last client removed while local upstream still active for "
f"{self.channel_id} without owner lock - stopping channel"
)
schedule_disconnect = True
else:
logger.debug(
f"Non-owner publishing CLIENT_DISCONNECTED event for client {client_id} "
f"on channel {self.channel_id} from worker {self.worker_id}"
)
event_data = json.dumps({
"event": EventType.CLIENT_DISCONNECTED,
"channel_id": self.channel_id,
"client_id": client_id,
"worker_id": self.worker_id or "unknown",
"timestamp": time.time(),
"remaining_clients": remaining,
"username": client_username
})
self.redis_client.publish(RedisKeys.events_channel(self.channel_id), event_data)
if schedule_disconnect and self.proxy_server:
self.proxy_server._spawn_on_hub(
self.proxy_server.handle_client_disconnect, self.channel_id
)
self._trigger_stats_update()
total_clients = self.get_total_client_count()
logger.info(
f"Client disconnected: {client_id} (local: {local_count}, total: {total_clients})"
)
return local_count
def get_client_count(self):
"""Get local client count"""
@ -442,3 +480,19 @@ class ClientManager:
)
return stale_ids
@staticmethod
def clear_all_clients(redis_client, channel_id):
"""Remove every client entry from Redis for a channel."""
client_set_key = RedisKeys.clients(channel_id)
client_ids = redis_client.smembers(client_set_key)
if not client_ids:
return 0
pipe = redis_client.pipeline(transaction=False)
for cid in client_ids:
cid_str = cid.decode() if isinstance(cid, bytes) else cid
pipe.delete(RedisKeys.client_metadata(channel_id, cid_str))
pipe.delete(client_set_key)
pipe.execute()
return len(client_ids)

7
apps/proxy/live_proxy/config_helper.py
View file

@ -107,9 +107,14 @@ class ConfigHelper:
@staticmethod
def channel_init_grace_period():
"""Get channel initialization grace period in seconds"""
"""Max seconds to wait for initial buffer fill during channel startup."""
return Config.get_channel_init_grace_period()
@staticmethod
def channel_client_wait_period():
"""Seconds to keep a ready channel alive waiting for the first client to connect."""
return Config.get_channel_client_wait_period()
@staticmethod
def chunk_timeout():
"""

2
apps/proxy/live_proxy/constants.py
View file

@ -54,6 +54,8 @@ class ChannelMetadataField:
STREAM_ID = "stream_id"
CHANNEL_NAME = "channel_name"
STREAM_NAME = "stream_name"
IS_TIMESHIFT = "is_timeshift"
LOGO_ID = "logo_id"
# Profile fields
STREAM_PROFILE = "stream_profile"

44
apps/proxy/live_proxy/input/buffer.py
View file

@ -64,7 +64,7 @@ class StreamBuffer:
def add_chunk(self, chunk):
"""Add data with optimized Redis storage and TS packet alignment"""
if not chunk:
if not chunk or self.stopping:
return False
try:
@ -309,38 +309,16 @@ class StreamBuffer:
self.fill_timers.clear()
try:
# Flush any remaining data in the write buffer
if hasattr(self, '_write_buffer') and len(self._write_buffer) > 0:
# Ensure remaining data is aligned to TS packets
complete_size = (len(self._write_buffer) // 188) * 188
if complete_size > 0:
final_chunk = self._write_buffer[:complete_size]
# Write final chunk to Redis
with self.lock:
if self.redis_client:
try:
chunk_index = self.redis_client.incr(self.buffer_index_key)
chunk_key = f"{self.buffer_prefix}{chunk_index}"
self.redis_client.setex(chunk_key, self.chunk_ttl, bytes(final_chunk))
self.index = chunk_index
logger.info(f"Flushed final chunk of {len(final_chunk)} bytes to Redis")
except Exception as e:
logger.error(f"Error flushing final chunk: {e}")
# Clear buffers
self._write_buffer = bytearray()
if hasattr(self, '_partial_packet'):
self._partial_packet = bytearray()
# Clean up the chunk timestamps sorted set
if self.redis_client and self.chunk_timestamps_key:
try:
self.redis_client.delete(self.chunk_timestamps_key)
except Exception as e:
logger.error(f"Error deleting chunk timestamps key: {e}")
with self.lock:
if hasattr(self, '_write_buffer') and len(self._write_buffer) > 0:
discarded = len(self._write_buffer)
self._write_buffer = bytearray()
if hasattr(self, '_partial_packet'):
self._partial_packet = bytearray()
logger.debug(
f"Discarded {discarded} bytes from local write buffer "
f"for channel {self.channel_id}"
)
except Exception as e:
logger.error(f"Error during buffer stop: {e}")

290
apps/proxy/live_proxy/input/manager.py
View file

@ -67,6 +67,7 @@ class StreamManager:
# Add to your __init__ method
self._buffer_check_timers = []
self.stopping = False
self.stop_requested = False
# Add tracking for tried streams and current stream
self.current_stream_id = stream_id
@ -112,6 +113,11 @@ class StreamManager:
# Add tracking for data throughput
self.bytes_processed = 0
self.last_bytes_update = time.time()
# Cached result of the pipelined Redis ownership audit (hot read path).
self._ownership_cache_valid_until = 0.0
self._ownership_cached = True
self._OWNERSHIP_CHECK_INTERVAL = 1.0
self.bytes_update_interval = 5 # Update Redis every 5 seconds
# Add stderr reader thread property
@ -182,6 +188,153 @@ class StreamManager:
logger.warning(f"Timeout waiting for existing processes to close for channel {self.channel_id} after {timeout}s")
return False
def _invalidate_ownership_cache(self):
self._ownership_cache_valid_until = 0.0
@staticmethod
def _decode_redis_value(value):
if value is None:
return None
if isinstance(value, bytes):
return value.decode()
return value
def _disconnect_shutdown_ready(self, disconnect_value):
"""True when last-client disconnect has passed the configured shutdown delay."""
if not disconnect_value:
return False
shutdown_delay = ConfigHelper.channel_shutdown_delay()
if shutdown_delay <= 0:
return True
disconnect_value = self._decode_redis_value(disconnect_value)
try:
disconnect_time = float(disconnect_value)
except (ValueError, TypeError):
return False
return (time.time() - disconnect_time) >= shutdown_delay
def _evaluate_ownership_from_redis(self, redis_client):
"""Single pipelined Redis round-trip for the full ownership audit."""
stop_key = RedisKeys.channel_stopping(self.channel_id)
metadata_key = RedisKeys.channel_metadata(self.channel_id)
clients_key = RedisKeys.clients(self.channel_id)
owner_key = RedisKeys.channel_owner(self.channel_id)
disconnect_key = RedisKeys.last_client_disconnect(self.channel_id)
pipe = redis_client.pipeline(transaction=False)
pipe.exists(stop_key)
pipe.exists(metadata_key)
pipe.scard(clients_key)
pipe.get(owner_key)
pipe.get(disconnect_key)
pipe.hget(metadata_key, ChannelMetadataField.STATE)
(
stop_exists,
metadata_exists,
client_count,
current_owner,
disconnect_value,
state_raw,
) = pipe.execute()
if stop_exists:
return False
if not metadata_exists:
logger.warning(
f"Channel {self.channel_id} metadata removed from Redis - stopping upstream"
)
return False
client_count = client_count or 0
state = self._decode_redis_value(state_raw)
current_owner = self._decode_redis_value(current_owner)
if current_owner and current_owner != self.worker_id:
return False
if not current_owner:
if client_count == 0 and state not in ChannelState.PRE_ACTIVE:
logger.warning(
f"Channel {self.channel_id} has no owner and no clients - stopping upstream"
)
return False
return True
if client_count == 0 and self._disconnect_shutdown_ready(disconnect_value):
logger.info(
f"Channel {self.channel_id} disconnect shutdown ready - stopping upstream"
)
return False
return True
def _still_owner(self, *, force=False):
"""Return True while this worker should keep the upstream connection open."""
if self.stopping or self.stop_requested:
return False
if not self.worker_id:
return True
redis_client = getattr(self.buffer, 'redis_client', None)
if not redis_client:
return True
now = time.time()
if not force and now < self._ownership_cache_valid_until:
return self._ownership_cached
try:
result = self._evaluate_ownership_from_redis(redis_client)
self._ownership_cached = result
self._ownership_cache_valid_until = now + self._OWNERSHIP_CHECK_INTERVAL
return result
except Exception as e:
logger.debug(f"Ownership check failed for channel {self.channel_id}: {e}")
return True
def _upstream_may_continue(self):
"""
Per-chunk gate for the hot read path.
Local stop flags are checked every chunk. The coordinated-teardown Redis
flag is checked every chunk (one EXISTS). The full ownership audit is
pipelined and cached for ~1s — enough for loop boundaries while avoiding
6+ Redis round-trips per chunk during steady streaming.
"""
if self.stopping or self.stop_requested or not self.running:
return False
if self.buffer is not None and self.buffer.stopping:
return False
redis_client = getattr(self.buffer, 'redis_client', None)
if redis_client and self.worker_id:
try:
if redis_client.exists(RedisKeys.channel_stopping(self.channel_id)):
self._ownership_cached = False
self._ownership_cache_valid_until = 0.0
return False
except Exception as e:
logger.debug(
f"Channel stopping check failed for {self.channel_id}: {e}"
)
return self._still_owner()
def _ensure_owner_or_stop(self):
if self._still_owner(force=True):
return True
logger.warning(
f"Stream manager for channel {self.channel_id} lost ownership "
f"(worker {self.worker_id}) - stopping upstream"
)
self.stop()
return False
def run(self):
"""Main execution loop using HTTP streaming with improved connection handling and stream switching"""
# Add a stop flag to the class properties
@ -203,6 +356,8 @@ class StreamManager:
# Main stream switching loop - we'll try different streams if needed
while self.running and stream_switch_attempts <= max_stream_switches:
close_old_connections()
if not self._ensure_owner_or_stop():
break
# Check for stuck switching state
if self.url_switching and time.time() - self.url_switch_start_time > self.url_switch_timeout:
logger.warning(f"URL switching state appears stuck for channel {self.channel_id} "
@ -255,6 +410,8 @@ class StreamManager:
continue
# Connection retry loop for current URL
while self.running and self.retry_count < self.max_retries and not url_failed and not self.needs_stream_switch:
if not self._ensure_owner_or_stop():
break
logger.info(f"Connection attempt {self.retry_count + 1}/{self.max_retries} for URL: {self.url} for channel {self.channel_id}")
@ -382,6 +539,12 @@ class StreamManager:
except Exception as e:
logger.error(f"Stream error: {e}", exc_info=True)
finally:
try:
from ..server import ProxyServer
ProxyServer.get_instance()._live_stream_managers.pop(self.channel_id, None)
except Exception:
pass
# Enhanced cleanup in the finally block
self.connected = False
@ -410,7 +573,9 @@ class StreamManager:
try:
metadata_key = RedisKeys.channel_metadata(self.channel_id)
owner_key = RedisKeys.channel_owner(self.channel_id)
current_owner = self.buffer.redis_client.get(owner_key)
current_owner = self._decode_redis_value(
self.buffer.redis_client.get(owner_key)
)
is_owner = (
current_owner
@ -421,12 +586,10 @@ class StreamManager:
should_update = is_owner
if not should_update and no_owner:
current_state_bytes = self.buffer.redis_client.hget(
metadata_key, ChannelMetadataField.STATE
)
current_state = (
current_state_bytes
if current_state_bytes else None
current_state = self._decode_redis_value(
self.buffer.redis_client.hget(
metadata_key, ChannelMetadataField.STATE
)
)
should_update = current_state in ChannelState.PRE_ACTIVE
if not should_update and current_state:
@ -488,44 +651,48 @@ class StreamManager:
logger.debug(f"Closing existing HTTP connections before establishing transcode connection for channel {self.channel_id}")
self._close_connection()
channel = get_stream_object(self.channel_id)
try:
channel = get_stream_object(self.channel_id)
# Use FFmpeg specifically for HLS streams
if hasattr(self, 'force_ffmpeg') and self.force_ffmpeg:
from core.models import StreamProfile
try:
stream_profile = StreamProfile.objects.get(name='ffmpeg', locked=True)
logger.info("Using FFmpeg stream profile for unsupported proxy content (HLS/RTSP/UDP)")
except StreamProfile.DoesNotExist:
# Fall back to channel's profile if FFmpeg not found
# Use FFmpeg specifically for HLS streams
if hasattr(self, 'force_ffmpeg') and self.force_ffmpeg:
from core.models import StreamProfile
try:
stream_profile = StreamProfile.objects.get(name='ffmpeg', locked=True)
logger.info("Using FFmpeg stream profile for unsupported proxy content (HLS/RTSP/UDP)")
except StreamProfile.DoesNotExist:
# Fall back to channel's profile if FFmpeg not found
stream_profile = channel.get_stream_profile()
logger.warning(f"FFmpeg profile not found, using channel default profile for channel: {self.channel_id}")
else:
stream_profile = channel.get_stream_profile()
logger.warning(f"FFmpeg profile not found, using channel default profile for channel: {self.channel_id}")
else:
stream_profile = channel.get_stream_profile()
# Build and start transcode command
self.transcode_cmd = stream_profile.build_command(self.url, self.user_agent)
# Build and start transcode command
self.transcode_cmd = stream_profile.build_command(self.url, self.user_agent)
# Store stream command for efficient log parser routing
self.stream_command = stream_profile.command
# Map actual commands to parser types for direct routing
command_to_parser = {
'ffmpeg': 'ffmpeg',
'cvlc': 'vlc',
'vlc': 'vlc',
'streamlink': 'streamlink'
}
self.parser_type = command_to_parser.get(self.stream_command.lower())
if self.parser_type:
logger.debug(f"Using {self.parser_type} parser for log parsing (command: {self.stream_command})")
else:
logger.debug(f"Unknown stream command '{self.stream_command}', will use auto-detection for log parsing")
# Store stream command for efficient log parser routing
self.stream_command = stream_profile.command
# Map actual commands to parser types for direct routing
command_to_parser = {
'ffmpeg': 'ffmpeg',
'cvlc': 'vlc',
'vlc': 'vlc',
'streamlink': 'streamlink'
}
self.parser_type = command_to_parser.get(self.stream_command.lower())
if self.parser_type:
logger.debug(f"Using {self.parser_type} parser for log parsing (command: {self.stream_command})")
else:
logger.debug(f"Unknown stream command '{self.stream_command}', will use auto-detection for log parsing")
# For UDP streams, remove any user_agent parameters from the command
if hasattr(self, 'stream_type') and self.stream_type == StreamType.UDP:
# Filter out any arguments that contain the user_agent value or related headers
self.transcode_cmd = [arg for arg in self.transcode_cmd if self.user_agent not in arg and 'user-agent' not in arg.lower() and 'user_agent' not in arg.lower()]
logger.debug(f"Removed user_agent parameters from UDP stream command for channel: {self.channel_id}")
# For UDP streams, remove any user_agent parameters from the command
if hasattr(self, 'stream_type') and self.stream_type == StreamType.UDP:
# Filter out any arguments that contain the user_agent value or related headers
self.transcode_cmd = [arg for arg in self.transcode_cmd if self.user_agent not in arg and 'user-agent' not in arg.lower() and 'user_agent' not in arg.lower()]
logger.debug(f"Removed user_agent parameters from UDP stream command for channel: {self.channel_id}")
finally:
# Release the pool slot before posix_spawn or before returning on profile errors.
close_old_connections()
logger.debug(f"Starting transcode process: {self.transcode_cmd} for channel: {self.channel_id}")
@ -1020,6 +1187,9 @@ class StreamManager:
def _update_bytes_processed(self, chunk_size):
"""Update the total bytes processed in Redis metadata"""
if not self._upstream_may_continue():
return
try:
# Update local counter
self.bytes_processed += chunk_size
@ -1092,14 +1262,10 @@ class StreamManager:
"""Stop the stream manager and cancel all timers"""
logger.info(f"Stopping stream manager for channel {self.channel_id}")
# Add at the beginning of your stop method
self.stopping = True
# Release stream resources if we're the owner
if self.current_stream_id and hasattr(self, 'worker_id') and self.worker_id:
if hasattr(self.buffer, 'redis_client') and self.buffer.redis_client:
owner_key = RedisKeys.channel_owner(self.channel_id)
current_owner = self.buffer.redis_client.get(owner_key)
self._invalidate_ownership_cache()
if self.buffer is not None:
self.buffer.stopping = True
# Cancel all buffer check timers
for timer in list(self._buffer_check_timers):
@ -1254,6 +1420,12 @@ class StreamManager:
"""Check if connection retry is allowed"""
return self.retry_count < self.max_retries
def _health_inactivity_threshold(self):
"""How long without data before marking the stream unhealthy."""
if self.connected and getattr(self.buffer, 'index', 0) == 0:
return ConfigHelper.channel_init_grace_period()
return getattr(Config, 'CONNECTION_TIMEOUT', 10)
def _monitor_health(self):
"""Monitor stream health and set flags for the main loop to handle recovery"""
consecutive_unhealthy_checks = 0
@ -1269,7 +1441,7 @@ class StreamManager:
try:
now = time.time()
inactivity_duration = now - self.last_data_time
timeout_threshold = getattr(Config, 'CONNECTION_TIMEOUT', 10)
timeout_threshold = self._health_inactivity_threshold()
if inactivity_duration > timeout_threshold and self.connected:
if self.healthy:
@ -1489,7 +1661,8 @@ class StreamManager:
if hasattr(self, 'stderr_reader_thread') and self.stderr_reader_thread and self.stderr_reader_thread.is_alive():
try:
logger.debug(f"Waiting for stderr reader thread to terminate for channel {self.channel_id}")
self.stderr_reader_thread.join(timeout=2.0)
stderr_join_timeout = 0.25 if self.stopping else 2.0
self.stderr_reader_thread.join(timeout=stderr_join_timeout)
if self.stderr_reader_thread.is_alive():
logger.warning(f"Stderr reader thread did not terminate within timeout for channel {self.channel_id}")
except Exception as e:
@ -1582,6 +1755,10 @@ class StreamManager:
self.connected = False
return False
if not self._upstream_may_continue():
self.stop()
return False
# Track chunk size before adding to buffer
chunk_size = len(chunk)
self._update_bytes_processed(chunk_size)
@ -1589,7 +1766,6 @@ class StreamManager:
# Add directly to buffer without TS-specific processing
success = self.buffer.add_chunk(chunk)
# Update last data timestamp in Redis if successful
if success and hasattr(self.buffer, 'redis_client') and self.buffer.redis_client:
last_data_key = RedisKeys.last_data(self.buffer.channel_id)
self.buffer.redis_client.set(last_data_key, str(time.time()), ex=60)
@ -1660,19 +1836,9 @@ class StreamManager:
timer.start()
return False
# We have enough buffer, proceed with state change
update_data = {
ChannelMetadataField.STATE: ChannelState.WAITING_FOR_CLIENTS,
ChannelMetadataField.CONNECTION_READY_TIME: current_time,
ChannelMetadataField.STATE_CHANGED_AT: current_time,
ChannelMetadataField.BUFFER_CHUNKS: str(current_buffer_index)
}
redis_client.hset(metadata_key, mapping=update_data)
from ..services.channel_service import ChannelService
# Get configured grace period or default
grace_period = ConfigHelper.channel_init_grace_period()
logger.info(f"STREAM MANAGER: Updated channel {channel_id} state: {current_state or 'None'} -> {ChannelState.WAITING_FOR_CLIENTS} with {current_buffer_index} buffer chunks")
logger.info(f"Started initial connection grace period ({grace_period}s) for channel {channel_id}")
ChannelService.promote_channel_when_buffer_ready(channel_id)
else:
logger.debug(f"Not changing state: channel {channel_id} already in {current_state} state")
except Exception as e:

86
apps/proxy/live_proxy/output/fmp4/generator.py
View file

@ -11,6 +11,7 @@ import time
import gevent
from apps.channels.models import Channel, Stream
from core.utils import log_system_event
from django.db import close_old_connections
from ...server import ProxyServer
from ...redis_keys import RedisKeys
from ...constants import ChannelMetadataField
@ -331,46 +332,53 @@ class FMP4StreamGenerator:
# ------------------------------------------------------------------
def _cleanup(self):
elapsed = time.time() - self.stream_start_time
proxy_server = ProxyServer.get_instance()
try:
elapsed = time.time() - self.stream_start_time
proxy_server = ProxyServer.get_instance()
# Release stream allocation if last client (mirrors StreamGenerator)
if proxy_server.redis_client:
try:
meta_key = RedisKeys.channel_metadata(self.channel_id)
stream_id_bytes = proxy_server.redis_client.hget(
meta_key, ChannelMetadataField.STREAM_ID
)
if stream_id_bytes:
if self.channel_id in proxy_server.client_managers:
client_count = proxy_server.client_managers[
self.channel_id
].get_total_client_count()
if client_count <= 1 and proxy_server.am_i_owner(self.channel_id):
try:
# Release stream allocation if last client (mirrors StreamGenerator)
if proxy_server.redis_client:
try:
meta_key = RedisKeys.channel_metadata(self.channel_id)
stream_id_bytes = proxy_server.redis_client.hget(
meta_key, ChannelMetadataField.STREAM_ID
)
if stream_id_bytes:
if self.channel_id in proxy_server.client_managers:
client_count = proxy_server.client_managers[
self.channel_id
].get_total_client_count()
if (
client_count <= 1
and proxy_server.am_i_owner(self.channel_id)
and ConfigHelper.channel_shutdown_delay() <= 0
):
try:
obj = Channel.objects.get(uuid=self.channel_id)
except (Channel.DoesNotExist, Exception):
obj = Stream.objects.get(stream_hash=self.channel_id)
obj.release_stream()
except Exception as e:
logger.error(
f"[{self.client_id}] Error releasing stream: {e}"
)
except Exception as e:
logger.error(f"[{self.client_id}] Error in stream release check: {e}")
try:
obj = Channel.objects.get(uuid=self.channel_id)
except (Channel.DoesNotExist, Exception):
obj = Stream.objects.get(stream_hash=self.channel_id)
obj.release_stream()
except Exception as e:
logger.error(
f"[{self.client_id}] Error releasing stream: {e}"
)
except Exception as e:
logger.error(f"[{self.client_id}] Error in stream release check: {e}")
# Remove from MAIN ClientManager - this is what triggers handle_client_disconnect
# and the zero-clients → stop_channel path, same as TS clients.
local_clients = 0
total_clients = 0
if self.channel_id in proxy_server.client_managers:
client_manager = proxy_server.client_managers[self.channel_id]
local_clients = client_manager.remove_client(self.client_id)
total_clients = client_manager.get_total_client_count()
# Remove from MAIN ClientManager - this is what triggers handle_client_disconnect
# and the zero-clients → stop_channel path, same as TS clients.
local_clients = 0
total_clients = 0
if self.channel_id in proxy_server.client_managers:
client_manager = proxy_server.client_managers[self.channel_id]
local_clients = client_manager.remove_client(self.client_id)
total_clients = client_manager.get_total_client_count()
logger.info(
f"[{self.client_id}] fMP4 client disconnected after {elapsed:.2f}s "
f"({self.bytes_sent / 1024:.1f} KB sent, "
f"local: {local_clients}, total: {total_clients})"
)
logger.info(
f"[{self.client_id}] fMP4 client disconnected after {elapsed:.2f}s "
f"({self.bytes_sent / 1024:.1f} KB sent, "
f"local: {local_clients}, total: {total_clients})"
)
finally:
close_old_connections()

170
apps/proxy/live_proxy/output/ts/generator.py
View file

@ -7,6 +7,7 @@ import time
import gevent
from apps.proxy.config import TSConfig as Config
from apps.channels.models import Channel, Stream
from django.db import close_old_connections
from core.utils import log_system_event
from ...server import ProxyServer
from ...utils import create_ts_packet, get_logger
@ -139,12 +140,75 @@ class StreamGenerator:
finally:
self._cleanup()
def _init_wait_abort_reason(self, proxy_server, initialization_start):
"""Return a reason string when init wait should end early, else None."""
from ...services.channel_service import ChannelService
if ChannelService.is_channel_teardown_active(self.channel_id):
return "stopping"
client_mgr = proxy_server.client_managers.get(self.channel_id)
if client_mgr is not None:
if self.client_id not in client_mgr.clients:
return "client_gone"
elif proxy_server.redis_client:
client_key = RedisKeys.client_metadata(self.channel_id, self.client_id)
if not proxy_server.redis_client.exists(client_key):
total = proxy_server.redis_client.scard(RedisKeys.clients(self.channel_id)) or 0
if total == 0:
return "client_gone"
elapsed = time.time() - initialization_start
init_grace_period = ConfigHelper.channel_init_grace_period()
if elapsed < init_grace_period or not proxy_server.redis_client:
return None
metadata_key = RedisKeys.channel_metadata(self.channel_id)
state_raw = proxy_server.redis_client.hget(metadata_key, 'state')
if not state_raw:
return None
state = state_raw.decode() if isinstance(state_raw, bytes) else state_raw
if state not in ('connecting', 'initializing', 'buffering'):
return None
buffer = proxy_server.stream_buffers.get(self.channel_id)
if buffer is not None and buffer.index > 0:
return None
if proxy_server.redis_client.get(RedisKeys.last_data(self.channel_id)):
return None
return "stalled"
def _wait_for_initialization(self):
initialization_start = time.time()
max_init_wait = ConfigHelper.client_wait_timeout()
proxy_server = ProxyServer.get_instance()
while time.time() - initialization_start < max_init_wait:
abort_reason = self._init_wait_abort_reason(proxy_server, initialization_start)
if abort_reason == "stopping":
logger.error(
f"[{self.client_id}] Channel {self.channel_id} teardown active during initialization"
)
yield create_ts_packet('error', "Error: Channel is stopping")
return False
if abort_reason == "client_gone":
logger.info(
f"[{self.client_id}] Client disconnected during initialization wait, aborting"
)
yield create_ts_packet('error', "Error: Client disconnected")
return False
if abort_reason == "stalled":
logger.warning(
f"[{self.client_id}] Channel {self.channel_id} stalled in connecting state "
f"with no buffer data after "
f"{ConfigHelper.channel_init_grace_period()}s, aborting init wait"
)
yield create_ts_packet('error', "Error: Connection stalled")
return False
if proxy_server.redis_client:
metadata_key = RedisKeys.channel_metadata(self.channel_id)
metadata = proxy_server.redis_client.hgetall(metadata_key)
@ -527,63 +591,63 @@ class StreamGenerator:
def _cleanup(self):
"""Clean up resources and report final statistics."""
# Client cleanup
elapsed = time.time() - self.stream_start_time
local_clients = 0
total_clients = 0
proxy_server = ProxyServer.get_instance()
try:
# Client cleanup
elapsed = time.time() - self.stream_start_time
local_clients = 0
total_clients = 0
proxy_server = ProxyServer.get_instance()
# Release M3U profile stream allocation if this is the last client
stream_released = False
if proxy_server.redis_client:
try:
metadata_key = RedisKeys.channel_metadata(self.channel_id)
metadata = proxy_server.redis_client.hgetall(metadata_key)
if metadata:
stream_id_bytes = proxy_server.redis_client.hget(metadata_key, ChannelMetadataField.STREAM_ID)
if stream_id_bytes:
# Check if we're the last client
if self.channel_id in proxy_server.client_managers:
client_count = proxy_server.client_managers[self.channel_id].get_total_client_count()
# Only the last client or owner should release the stream
if client_count <= 1 and proxy_server.am_i_owner(self.channel_id):
# Release M3U profile stream allocation if this is the last client
stream_released = False
if proxy_server.redis_client:
try:
if self.channel_id in proxy_server.client_managers:
client_count = proxy_server.client_managers[self.channel_id].get_total_client_count()
# Pool slots are global; the last client on any worker must release.
# During shutdown_delay, keep the slot until coordinated stop runs.
if client_count <= 1 and ConfigHelper.channel_shutdown_delay() <= 0:
try:
try:
# Try Channel first (normal flow), fall back to Stream (preview flow)
try:
obj = Channel.objects.get(uuid=self.channel_id)
except (Channel.DoesNotExist, Exception):
obj = Stream.objects.get(stream_hash=self.channel_id)
stream_released = obj.release_stream()
if stream_released:
logger.debug(f"[{self.client_id}] Released stream for channel {self.channel_id}")
else:
logger.warning(f"[{self.client_id}] release_stream found no keys for channel {self.channel_id}")
except Exception as e:
logger.error(f"[{self.client_id}] Error releasing stream for channel {self.channel_id}: {e}")
except Exception as e:
logger.error(f"[{self.client_id}] Error checking stream data for release: {e}")
obj = Channel.objects.get(uuid=self.channel_id)
except (Channel.DoesNotExist, Exception):
obj = Stream.objects.get(stream_hash=self.channel_id)
stream_released = obj.release_stream()
if stream_released:
logger.debug(f"[{self.client_id}] Released stream for channel {self.channel_id}")
else:
logger.warning(f"[{self.client_id}] release_stream found no keys for channel {self.channel_id}")
except Exception as e:
logger.error(f"[{self.client_id}] Error releasing stream for channel {self.channel_id}: {e}")
except Exception as e:
logger.error(f"[{self.client_id}] Error checking stream data for release: {e}")
if self.channel_id in proxy_server.client_managers:
client_manager = proxy_server.client_managers[self.channel_id]
local_clients = client_manager.remove_client(self.client_id)
total_clients = client_manager.get_total_client_count()
logger.info(f"[{self.client_id}] Disconnected after {elapsed:.2f}s (local: {local_clients}, total: {total_clients})")
if self.channel_id in proxy_server.client_managers:
client_manager = proxy_server.client_managers[self.channel_id]
if self.client_id in client_manager.clients:
local_clients = client_manager.remove_client(self.client_id)
else:
local_clients = client_manager.get_client_count()
total_clients = client_manager.get_total_client_count()
logger.info(f"[{self.client_id}] Disconnected after {elapsed:.2f}s (local: {local_clients}, total: {total_clients})")
# Log client disconnect event
try:
log_system_event(
'client_disconnect',
channel_id=self.channel_id,
channel_name=self.channel_name,
client_ip=self.client_ip,
client_id=self.client_id,
user_agent=self.client_user_agent[:100] if self.client_user_agent else None,
duration=round(elapsed, 2),
bytes_sent=self.bytes_sent,
username=self.user.username if self.user else None
)
except Exception as e:
logger.error(f"Could not log client disconnect event: {e}")
# Log client disconnect event
try:
log_system_event(
'client_disconnect',
channel_id=self.channel_id,
channel_name=self.channel_name,
client_ip=self.client_ip,
client_id=self.client_id,
user_agent=self.client_user_agent[:100] if self.client_user_agent else None,
duration=round(elapsed, 2),
bytes_sent=self.bytes_sent,
username=self.user.username if self.user else None
)
except Exception as e:
logger.error(f"Could not log client disconnect event: {e}")
finally:
close_old_connections()
def create_stream_generator(channel_id, client_id, client_ip, client_user_agent, channel_initializing=False, user=None, buffer=None):

886
apps/proxy/live_proxy/server.py
View file

File diff suppressed because it is too large Load diff

433
apps/proxy/live_proxy/services/channel_service.py
View file

@ -10,6 +10,7 @@ from apps.channels.models import Channel, Stream
from ..server import ProxyServer
from ..redis_keys import RedisKeys
from ..constants import EventType, ChannelState, ChannelMetadataField
from ..config_helper import ConfigHelper
from ..url_utils import get_stream_info_for_switch
from core.utils import log_system_event
from .log_parsers import LogParserFactory
@ -19,6 +20,250 @@ logger = logging.getLogger("live_proxy")
class ChannelService:
"""Service class for channel operations"""
@staticmethod
def mark_channel_stopping(channel_id, broadcast=False):
"""Mark a channel as stopping in Redis so all uWSGI workers converge on teardown."""
proxy_server = ProxyServer.get_instance()
if not proxy_server.redis_client:
return False
try:
metadata_key = RedisKeys.channel_metadata(channel_id)
if proxy_server.redis_client.exists(metadata_key):
proxy_server.redis_client.hset(metadata_key, mapping={
ChannelMetadataField.STATE: ChannelState.STOPPING,
ChannelMetadataField.STATE_CHANGED_AT: str(time.time()),
})
stop_key = RedisKeys.channel_stopping(channel_id)
proxy_server.redis_client.setex(stop_key, 60, "true")
if broadcast:
ChannelService._publish_channel_stop_event(channel_id)
return True
except Exception as e:
logger.error(f"Error marking channel {channel_id} as stopping: {e}")
return False
@staticmethod
def is_shutdown_pending(channel_id):
"""True while the post-disconnect shutdown delay has started but stop has not run."""
proxy_server = ProxyServer.get_instance()
if not proxy_server.redis_client:
return False
delay = ConfigHelper.channel_shutdown_delay()
if delay <= 0:
return False
disconnect_value = proxy_server.redis_client.get(
RedisKeys.last_client_disconnect(channel_id)
)
if not disconnect_value:
return False
try:
disconnect_time = float(disconnect_value)
except (ValueError, TypeError):
return False
return (time.time() - disconnect_time) < delay
@staticmethod
def is_channel_teardown_active(channel_id):
"""True when a coordinated channel stop is in progress (visible to all workers)."""
proxy_server = ProxyServer.get_instance()
if channel_id in proxy_server._stopping_channels:
return True
if not proxy_server.redis_client:
return False
if proxy_server.redis_client.exists(RedisKeys.channel_stopping(channel_id)):
return True
metadata_key = RedisKeys.channel_metadata(channel_id)
state = proxy_server.redis_client.hget(metadata_key, ChannelMetadataField.STATE)
if state:
state_str = state.decode() if isinstance(state, bytes) else state
if state_str == ChannelState.STOPPING:
return True
return False
@staticmethod
def is_channel_unavailable_for_new_clients(channel_id):
"""Reject new stream requests only while coordinated teardown is active."""
return ChannelService.is_channel_teardown_active(channel_id)
@staticmethod
def cancel_pending_shutdown(channel_id):
"""
Abort the post-disconnect grace timer when a client reconnects.
Clears the disconnect timestamp and any leaked stopping markers. When
upstream is still active but the last client released its profile slot
during the grace window, re-reserve the slot from Redis metadata.
Does not run during coordinated stop_channel() — clearing teardown
markers mid-stop would leave clients attached to upstream that is
about to be torn down.
"""
from django.db import close_old_connections
proxy_server = ProxyServer.get_instance()
if not proxy_server.redis_client:
return False
if channel_id in proxy_server._stopping_channels:
return False
disconnect_key = RedisKeys.last_client_disconnect(channel_id)
had_pending = bool(proxy_server.redis_client.exists(disconnect_key))
in_grace = had_pending or ChannelService.is_shutdown_pending(channel_id)
if not in_grace:
return False
try:
proxy_server.redis_client.delete(disconnect_key)
metadata_key = RedisKeys.channel_metadata(channel_id)
state = proxy_server.redis_client.hget(
metadata_key, ChannelMetadataField.STATE
)
if state:
state_str = state.decode() if isinstance(state, bytes) else state
if state_str == ChannelState.STOPPING:
proxy_server.redis_client.hset(metadata_key, mapping={
ChannelMetadataField.STATE: ChannelState.ACTIVE,
ChannelMetadataField.STATE_CHANGED_AT: str(time.time()),
})
stop_key = RedisKeys.channel_stopping(channel_id)
if proxy_server.redis_client.exists(stop_key):
proxy_server.redis_client.delete(stop_key)
if ChannelService._channel_proxy_is_active(
proxy_server.redis_client, channel_id
):
from apps.channels.models import Channel
channel = Channel.objects.filter(uuid=channel_id).first()
if channel and not proxy_server.redis_client.get(
f"channel_stream:{channel.id}"
):
sid, pid, error, slot_reserved = channel.get_stream()
if error:
logger.warning(
f"Could not re-reserve stream for {channel_id} "
f"after shutdown cancel: {error}"
)
elif slot_reserved and sid and pid:
proxy_server.redis_client.hset(metadata_key, mapping={
ChannelMetadataField.STREAM_ID: str(sid),
ChannelMetadataField.M3U_PROFILE: str(pid),
})
logger.info(
f"Re-reserved profile slot for {channel_id} "
f"(stream={sid}, profile={pid})"
)
finally:
close_old_connections()
return True
@staticmethod
def _channel_proxy_is_active(redis_client, channel_id):
"""True when live proxy metadata shows this channel is still running."""
metadata_key = RedisKeys.channel_metadata(channel_id)
if not redis_client.exists(metadata_key):
return False
state = redis_client.hget(metadata_key, ChannelMetadataField.STATE)
if state is None:
return False
if isinstance(state, bytes):
state = state.decode()
return state in (
ChannelState.ACTIVE,
ChannelState.WAITING_FOR_CLIENTS,
ChannelState.BUFFERING,
ChannelState.INITIALIZING,
ChannelState.CONNECTING,
)
@staticmethod
def promote_channel_when_buffer_ready(channel_id):
"""
Promote channel state once the initial buffer threshold is met.
- connecting/initializing + buffer ready + clients -> active
- connecting/initializing + buffer ready + no clients -> waiting_for_clients
- waiting_for_clients + clients -> active
Returns the resulting state, or None when no promotion applies.
"""
proxy_server = ProxyServer.get_instance()
redis_client = proxy_server.redis_client
if not redis_client:
return None
metadata_key = RedisKeys.channel_metadata(channel_id)
state_raw = redis_client.hget(metadata_key, ChannelMetadataField.STATE)
if not state_raw:
return None
state = state_raw.decode() if isinstance(state_raw, bytes) else state_raw
if state == ChannelState.ACTIVE:
return ChannelState.ACTIVE
if state == ChannelState.WAITING_FOR_CLIENTS:
ready_raw = redis_client.hget(
metadata_key, ChannelMetadataField.CONNECTION_READY_TIME
)
if not ready_raw:
return None
client_count = redis_client.scard(RedisKeys.clients(channel_id)) or 0
if client_count <= 0:
return ChannelState.WAITING_FOR_CLIENTS
proxy_server.update_channel_state(
channel_id,
ChannelState.ACTIVE,
{"clients_at_activation": str(client_count)},
)
return ChannelState.ACTIVE
if state not in (ChannelState.INITIALIZING, ChannelState.CONNECTING):
return None
try:
buffer_index = int(redis_client.get(RedisKeys.buffer_index(channel_id)) or 0)
except (TypeError, ValueError):
buffer_index = 0
chunks_needed = ConfigHelper.initial_behind_chunks()
if buffer_index < chunks_needed:
return None
client_count = redis_client.scard(RedisKeys.clients(channel_id)) or 0
new_state = (
ChannelState.ACTIVE if client_count > 0 else ChannelState.WAITING_FOR_CLIENTS
)
current_time = str(time.time())
extra = {
ChannelMetadataField.CONNECTION_READY_TIME: current_time,
ChannelMetadataField.BUFFER_CHUNKS: str(buffer_index),
}
if new_state == ChannelState.ACTIVE:
extra["clients_at_activation"] = str(client_count)
proxy_server.update_channel_state(channel_id, new_state, extra)
logger.info(
f"Channel {channel_id} buffer ready ({buffer_index}/{chunks_needed} chunks) "
f"-> {new_state} (clients={client_count})"
)
return new_state
@staticmethod
def initialize_channel(channel_id, stream_url, user_agent, transcode=False, stream_profile_value=None, stream_id=None, m3u_profile_id=None, channel_name=None, stream_name=None):
"""
@ -39,6 +284,7 @@ class ChannelService:
bool: Success status
"""
proxy_server = ProxyServer.get_instance()
if stream_id and proxy_server.redis_client:
metadata_key = RedisKeys.channel_metadata(channel_id)
# Check if metadata already exists
@ -67,34 +313,38 @@ class ChannelService:
# Store additional metadata if initialization was successful
if success and proxy_server.redis_client:
metadata_key = RedisKeys.channel_metadata(channel_id)
update_data = {}
if stream_profile_value:
update_data[ChannelMetadataField.STREAM_PROFILE] = stream_profile_value
if stream_id:
update_data[ChannelMetadataField.STREAM_ID] = str(stream_id)
if m3u_profile_id:
update_data[ChannelMetadataField.M3U_PROFILE] = str(m3u_profile_id)
# Store channel name and stream name so stats workers don't need DB calls
try:
if not channel_name:
from apps.channels.models import Channel
channel_name = Channel.objects.filter(uuid=channel_id).values_list('name', flat=True).first()
if channel_name:
update_data[ChannelMetadataField.CHANNEL_NAME] = channel_name
else:
# No channel name means stream preview mode, use stream name as display fallback
if stream_id and not stream_name:
from apps.channels.models import Stream
stream_name = Stream.objects.filter(id=stream_id).values_list('name', flat=True).first()
if stream_name:
update_data[ChannelMetadataField.STREAM_NAME] = stream_name
except Exception as e:
logger.warning(f"Failed to store channel/stream names in Redis for {channel_id}: {e}")
metadata_key = RedisKeys.channel_metadata(channel_id)
update_data = {}
if stream_profile_value:
update_data[ChannelMetadataField.STREAM_PROFILE] = stream_profile_value
if stream_id:
update_data[ChannelMetadataField.STREAM_ID] = str(stream_id)
if m3u_profile_id:
update_data[ChannelMetadataField.M3U_PROFILE] = str(m3u_profile_id)
if update_data:
proxy_server.redis_client.hset(metadata_key, mapping=update_data)
# Store channel name and stream name so stats workers don't need DB calls
try:
if not channel_name:
from apps.channels.models import Channel
channel_name = Channel.objects.filter(uuid=channel_id).values_list('name', flat=True).first()
if channel_name:
update_data[ChannelMetadataField.CHANNEL_NAME] = channel_name
else:
# No channel name means stream preview mode, use stream name as display fallback
if stream_id and not stream_name:
from apps.channels.models import Stream
stream_name = Stream.objects.filter(id=stream_id).values_list('name', flat=True).first()
if stream_name:
update_data[ChannelMetadataField.STREAM_NAME] = stream_name
except Exception as e:
logger.warning(f"Failed to store channel/stream names in Redis for {channel_id}: {e}")
if update_data:
proxy_server.redis_client.hset(metadata_key, mapping=update_data)
finally:
from django.db import close_old_connections
close_old_connections()
return success
@ -263,58 +513,23 @@ class ChannelService:
try:
metadata = proxy_server.redis_client.hgetall(metadata_key)
if metadata and 'state' in metadata:
state = metadata['state']
channel_info = {"state": state}
# Immediately mark as stopping in metadata so clients detect it faster
proxy_server.redis_client.hset(metadata_key, ChannelMetadataField.STATE, ChannelState.STOPPING)
proxy_server.redis_client.hset(metadata_key, ChannelMetadataField.STATE_CHANGED_AT, str(time.time()))
channel_info = {"state": metadata['state']}
except Exception as e:
logger.error(f"Error fetching channel state: {e}")
# Set stopping flag with higher TTL to ensure it persists
# Mark stopping in Redis and notify all workers before local teardown.
# stop_channel() releases profile slots via _clean_redis_keys() before Redis deletion.
if proxy_server.redis_client:
stop_key = RedisKeys.channel_stopping(channel_id)
proxy_server.redis_client.setex(stop_key, 60, "true") # Higher TTL of 60 seconds
logger.info(f"Set channel stopping flag with 60s TTL for channel {channel_id}")
# Broadcast stop event to all workers via PubSub
if proxy_server.redis_client:
ChannelService._publish_channel_stop_event(channel_id)
# Also stop locally to ensure this worker cleans up right away
local_result = proxy_server.stop_channel(channel_id)
else:
# No Redis, just stop locally
local_result = proxy_server.stop_channel(channel_id)
# Release the channel in the channel model if applicable
try:
channel = Channel.objects.get(uuid=channel_id)
model_released = channel.release_stream()
if model_released:
logger.info(f"Released channel {channel_id} stream allocation")
else:
logger.warning(f"Channel {channel_id}: release_stream found no keys to clean")
except (Channel.DoesNotExist, Exception):
logger.warning(f"Could not find Channel model for UUID {channel_id}, attempting stream hash")
try:
stream = Stream.objects.get(stream_hash=channel_id)
model_released = stream.release_stream()
if model_released:
logger.info(f"Released stream {channel_id} stream allocation")
else:
logger.warning(f"Stream {channel_id}: release_stream found no keys to clean")
except (Stream.DoesNotExist, Exception) as e:
logger.error(f"No Channel or Stream found for {channel_id}: {e}")
model_released = False
ChannelService.mark_channel_stopping(channel_id, broadcast=True)
logger.info(f"Marked channel {channel_id} stopping and broadcast stop to all workers")
local_result = proxy_server.stop_channel(channel_id)
return {
'status': 'success',
'message': 'Channel stop request sent',
'channel_id': channel_id,
'previous_state': channel_info,
'model_released': model_released,
'model_released': bool(local_result),
'local_stop_result': local_result
}
@ -597,53 +812,57 @@ class ChannelService:
@staticmethod
def _update_channel_metadata(channel_id, url, user_agent=None, stream_id=None, m3u_profile_id=None, stream_name=None):
"""Update channel metadata in Redis"""
proxy_server = ProxyServer.get_instance()
try:
proxy_server = ProxyServer.get_instance()
if not proxy_server.redis_client:
return False
if not proxy_server.redis_client:
return False
metadata_key = RedisKeys.channel_metadata(channel_id)
metadata_key = RedisKeys.channel_metadata(channel_id)
# First check if the key exists and what type it is
key_type = proxy_server.redis_client.type(metadata_key)
logger.debug(f"Redis key {metadata_key} is of type: {key_type}")
# First check if the key exists and what type it is
key_type = proxy_server.redis_client.type(metadata_key)
logger.debug(f"Redis key {metadata_key} is of type: {key_type}")
# Build metadata update dict
metadata = {ChannelMetadataField.URL: url}
if user_agent:
metadata[ChannelMetadataField.USER_AGENT] = user_agent
if stream_id:
metadata[ChannelMetadataField.STREAM_ID] = str(stream_id)
if not stream_name:
try:
from apps.channels.models import Stream
stream_name = Stream.objects.filter(id=stream_id).values_list('name', flat=True).first()
except Exception as e:
logger.warning(f"Failed to update stream name in Redis for stream {stream_id}: {e}")
if stream_name:
metadata[ChannelMetadataField.STREAM_NAME] = stream_name
if m3u_profile_id:
metadata[ChannelMetadataField.M3U_PROFILE] = str(m3u_profile_id)
# Build metadata update dict
metadata = {ChannelMetadataField.URL: url}
if user_agent:
metadata[ChannelMetadataField.USER_AGENT] = user_agent
if stream_id:
metadata[ChannelMetadataField.STREAM_ID] = str(stream_id)
if not stream_name:
try:
from apps.channels.models import Stream
stream_name = Stream.objects.filter(id=stream_id).values_list('name', flat=True).first()
except Exception as e:
logger.warning(f"Failed to update stream name in Redis for stream {stream_id}: {e}")
if stream_name:
metadata[ChannelMetadataField.STREAM_NAME] = stream_name
if m3u_profile_id:
metadata[ChannelMetadataField.M3U_PROFILE] = str(m3u_profile_id)
# Also update the stream switch time field
metadata[ChannelMetadataField.STREAM_SWITCH_TIME] = str(time.time())
# Also update the stream switch time field
metadata[ChannelMetadataField.STREAM_SWITCH_TIME] = str(time.time())
# Use the appropriate method based on the key type
if key_type == 'hash':
proxy_server.redis_client.hset(metadata_key, mapping=metadata)
elif key_type == 'none': # Key doesn't exist yet
proxy_server.redis_client.hset(metadata_key, mapping=metadata)
else:
# If key exists with wrong type, delete it and recreate
proxy_server.redis_client.delete(metadata_key)
proxy_server.redis_client.hset(metadata_key, mapping=metadata)
# Use the appropriate method based on the key type
if key_type == 'hash':
proxy_server.redis_client.hset(metadata_key, mapping=metadata)
elif key_type == 'none': # Key doesn't exist yet
proxy_server.redis_client.hset(metadata_key, mapping=metadata)
else:
# If key exists with wrong type, delete it and recreate
proxy_server.redis_client.delete(metadata_key)
proxy_server.redis_client.hset(metadata_key, mapping=metadata)
# Set switch request flag to ensure all workers see it
switch_key = RedisKeys.switch_request(channel_id)
proxy_server.redis_client.setex(switch_key, 30, url) # 30 second TTL
# Set switch request flag to ensure all workers see it
switch_key = RedisKeys.switch_request(channel_id)
proxy_server.redis_client.setex(switch_key, 30, url) # 30 second TTL
logger.debug(f"Updated metadata for channel {channel_id} in Redis")
return True
logger.debug(f"Updated metadata for channel {channel_id} in Redis")
return True
finally:
from django.db import close_old_connections
close_old_connections()
@staticmethod
def _publish_stream_switch_event(channel_id, new_url, user_agent=None, stream_id=None, m3u_profile_id=None):

142
apps/proxy/live_proxy/tests/test_live_db_cleanup.py Normal file
View file

@ -0,0 +1,142 @@
"""Live proxy must release geventpool checkouts after ORM on stream and URL paths."""
from unittest.mock import MagicMock, patch
from django.http import StreamingHttpResponse
from django.test import RequestFactory, SimpleTestCase
class StreamTsDbCleanupTests(SimpleTestCase):
def setUp(self):
self.factory = RequestFactory()
@patch("apps.proxy.live_proxy.views.close_old_connections")
@patch("apps.proxy.live_proxy.views.create_stream_generator")
@patch("apps.proxy.live_proxy.views._resolve_output_format", return_value="mpegts")
@patch("apps.proxy.live_proxy.views._resolve_output_profile", return_value=None)
@patch("apps.proxy.live_proxy.views.ChannelService.is_channel_unavailable_for_new_clients", return_value=False)
@patch("apps.proxy.live_proxy.views.get_stream_object")
@patch("apps.proxy.live_proxy.views.network_access_allowed", return_value=True)
@patch("apps.proxy.live_proxy.views.ProxyServer")
def test_stream_ts_closes_db_before_streaming_response(
self,
mock_proxy_cls,
_network_ok,
mock_get_stream_object,
_unavailable,
_output_profile,
_output_format,
mock_create_generator,
mock_close,
):
channel = MagicMock()
channel.id = 1
channel.uuid = "channel-uuid"
channel.name = "Test Channel"
mock_get_stream_object.return_value = channel
client_manager = MagicMock()
proxy_server = MagicMock()
proxy_server.redis_client = MagicMock()
proxy_server.redis_client.exists.return_value = True
proxy_server.redis_client.hgetall.return_value = {"state": "active"}
proxy_server.stream_buffers = {"channel-uuid": MagicMock()}
proxy_server.client_managers = {"channel-uuid": client_manager}
proxy_server.check_if_channel_exists.return_value = True
proxy_server.get_buffer.return_value = MagicMock()
mock_proxy_cls.get_instance.return_value = proxy_server
def _generate():
yield b"chunk"
mock_create_generator.return_value = lambda: _generate()
request = self.factory.get("/proxy/live/channel-uuid/")
request.user = MagicMock(is_authenticated=False)
from apps.proxy.live_proxy.views import stream_ts
response = stream_ts(request, "channel-uuid")
self.assertIsInstance(response, StreamingHttpResponse)
mock_close.assert_called_once()
class UrlUtilsDbCleanupTests(SimpleTestCase):
@patch("apps.proxy.live_proxy.url_utils.close_old_connections")
@patch("apps.proxy.live_proxy.url_utils.get_stream_object")
def test_generate_stream_url_closes_db(self, mock_get_object, mock_close):
channel = MagicMock()
channel.get_stream.return_value = (None, None, "no streams", False)
mock_get_object.return_value = channel
from apps.proxy.live_proxy.url_utils import generate_stream_url
result = generate_stream_url("channel-uuid")
self.assertIsNone(result[0])
mock_close.assert_called_once()
@patch("apps.proxy.live_proxy.url_utils.close_old_connections")
@patch("apps.proxy.live_proxy.url_utils.get_stream_object")
def test_get_alternate_streams_closes_db(self, mock_get_object, mock_close):
channel = MagicMock()
channel.streams.all.return_value.order_by.return_value.exists.return_value = False
mock_get_object.return_value = channel
from apps.proxy.live_proxy.url_utils import get_alternate_streams
result = get_alternate_streams("channel-uuid", current_stream_id=1)
self.assertEqual(result, [])
mock_close.assert_called_once()
@patch("apps.proxy.live_proxy.url_utils.close_old_connections")
@patch("apps.proxy.live_proxy.url_utils.get_object_or_404")
def test_get_stream_info_for_switch_closes_db_on_error(self, mock_get_404, mock_close):
mock_get_404.side_effect = RuntimeError("db error")
from apps.proxy.live_proxy.url_utils import get_stream_info_for_switch
result = get_stream_info_for_switch("channel-uuid", target_stream_id=99)
self.assertIn("error", result)
mock_close.assert_called_once()
@patch("apps.proxy.live_proxy.url_utils.close_old_connections")
@patch("apps.proxy.live_proxy.url_utils.M3UAccountProfile.objects.get")
def test_get_connections_left_closes_db(self, mock_get, mock_close):
mock_get.side_effect = Exception("not found")
from apps.proxy.live_proxy.url_utils import get_connections_left
result = get_connections_left(999)
self.assertEqual(result, 0)
mock_close.assert_called_once()
class TsGeneratorDbCleanupTests(SimpleTestCase):
@patch("apps.proxy.live_proxy.output.ts.generator.close_old_connections")
@patch("apps.proxy.live_proxy.output.ts.generator.ProxyServer.get_instance")
def test_ts_cleanup_closes_db(self, mock_proxy_cls, mock_close):
proxy_server = MagicMock()
proxy_server.redis_client = None
proxy_server.client_managers = {}
mock_proxy_cls.return_value = proxy_server
from apps.proxy.live_proxy.output.ts.generator import StreamGenerator
gen = StreamGenerator.__new__(StreamGenerator)
gen.channel_id = "channel-uuid"
gen.client_id = "client-1"
gen.stream_start_time = 0
gen.channel_name = "Test"
gen.client_ip = "127.0.0.1"
gen.client_user_agent = "agent"
gen.bytes_sent = 0
gen.user = None
gen._cleanup()
mock_close.assert_called_once()

143
apps/proxy/live_proxy/tests/test_proxy_settings.py Normal file
View file

@ -0,0 +1,143 @@
"""Tests for proxy settings defaults, serializer validation, and migration 0026."""
from importlib import import_module
from unittest.mock import patch
from django.apps import apps
from django.test import SimpleTestCase, TestCase
from apps.proxy.config import TSConfig
from core.models import CoreSettings
from core.serializers import ProxySettingsSerializer
MIGRATION_0026 = import_module("core.migrations.0026_add_channel_client_wait_period")
class TSConfigProxySettingsDefaultsTests(SimpleTestCase):
@patch.object(TSConfig, "get_proxy_settings", return_value={})
def test_channel_init_grace_period_default(self, _mock_settings):
self.assertEqual(TSConfig.get_channel_init_grace_period(), 60)
@patch.object(TSConfig, "get_proxy_settings", return_value={})
def test_channel_client_wait_period_default(self, _mock_settings):
self.assertEqual(TSConfig.get_channel_client_wait_period(), 5)
@patch.object(
TSConfig,
"get_proxy_settings",
return_value={
"channel_init_grace_period": 120,
"channel_client_wait_period": 15,
},
)
def test_settings_override_db_values(self, _mock_settings):
self.assertEqual(TSConfig.get_channel_init_grace_period(), 120)
self.assertEqual(TSConfig.get_channel_client_wait_period(), 15)
class ProxySettingsSerializerTests(SimpleTestCase):
def _valid_payload(self, **overrides):
payload = {
"buffering_timeout": 15,
"buffering_speed": 1.0,
"redis_chunk_ttl": 60,
"channel_shutdown_delay": 0,
"channel_init_grace_period": 60,
"channel_client_wait_period": 5,
"new_client_behind_seconds": 5,
}
payload.update(overrides)
return payload
def test_accepts_new_client_wait_period(self):
serializer = ProxySettingsSerializer(data=self._valid_payload())
self.assertTrue(serializer.is_valid(), serializer.errors)
self.assertEqual(serializer.validated_data["channel_client_wait_period"], 5)
def test_init_grace_period_allows_up_to_300(self):
serializer = ProxySettingsSerializer(
data=self._valid_payload(channel_init_grace_period=300)
)
self.assertTrue(serializer.is_valid(), serializer.errors)
def test_init_grace_period_rejects_above_300(self):
serializer = ProxySettingsSerializer(
data=self._valid_payload(channel_init_grace_period=301)
)
self.assertFalse(serializer.is_valid())
self.assertIn("channel_init_grace_period", serializer.errors)
class CoreSettingsProxyDefaultsTests(TestCase):
def test_get_proxy_settings_defaults_when_missing(self):
CoreSettings.objects.filter(key="proxy_settings").delete()
defaults = CoreSettings.get_proxy_settings()
self.assertEqual(defaults["channel_init_grace_period"], 60)
self.assertEqual(defaults["channel_client_wait_period"], 5)
class Migration0026ProxySettingsTests(TestCase):
def _run_migration_forward(self):
MIGRATION_0026.add_channel_client_wait_period(apps, None)
def _set_proxy_settings(self, value):
settings_obj, _ = CoreSettings.objects.get_or_create(
key="proxy_settings",
defaults={"name": "Proxy Settings", "value": value},
)
settings_obj.value = value
settings_obj.save(update_fields=["value"])
return settings_obj
def test_bumps_legacy_init_grace_and_adds_client_wait(self):
settings_obj = self._set_proxy_settings(
{
"buffering_timeout": 15,
"buffering_speed": 1.0,
"redis_chunk_ttl": 60,
"channel_shutdown_delay": 0,
"channel_init_grace_period": 5,
"new_client_behind_seconds": 5,
}
)
self._run_migration_forward()
settings_obj.refresh_from_db()
self.assertEqual(settings_obj.value["channel_init_grace_period"], 60)
self.assertEqual(settings_obj.value["channel_client_wait_period"], 5)
def test_bumps_init_grace_below_new_default(self):
settings_obj = self._set_proxy_settings(
{
"buffering_timeout": 15,
"buffering_speed": 1.0,
"redis_chunk_ttl": 60,
"channel_shutdown_delay": 0,
"channel_init_grace_period": 45,
"new_client_behind_seconds": 5,
}
)
self._run_migration_forward()
settings_obj.refresh_from_db()
self.assertEqual(settings_obj.value["channel_init_grace_period"], 60)
def test_preserves_init_grace_at_or_above_new_default(self):
settings_obj = self._set_proxy_settings(
{
"buffering_timeout": 15,
"buffering_speed": 1.0,
"redis_chunk_ttl": 60,
"channel_shutdown_delay": 0,
"channel_init_grace_period": 90,
"new_client_behind_seconds": 5,
}
)
self._run_migration_forward()
settings_obj.refresh_from_db()
self.assertEqual(settings_obj.value["channel_init_grace_period"], 90)
self.assertEqual(settings_obj.value["channel_client_wait_period"], 5)

9
apps/proxy/live_proxy/url_utils.py
View file

@ -5,6 +5,7 @@ Utilities for handling stream URLs and transformations.
import logging
import regex
from typing import Optional, Tuple, List
from django.db import close_old_connections
from django.shortcuts import get_object_or_404
from apps.channels.models import Channel, Stream
from apps.m3u.models import M3UAccount, M3UAccountProfile
@ -156,6 +157,8 @@ def generate_stream_url(
except Exception as e:
logger.error(f"Error generating stream URL: {e}")
return None, None, False, None, False, str(e)
finally:
close_old_connections()
def transform_url(input_url: str, search_pattern: str, replace_pattern: str) -> str:
"""
@ -300,6 +303,8 @@ def get_stream_info_for_switch(channel_id: str, target_stream_id: Optional[int]
channel.release_stream()
logger.error(f"Error getting stream info for switch: {e}", exc_info=True)
return {'error': f'Error: {str(e)}'}
finally:
close_old_connections()
def get_alternate_streams(channel_id: str, current_stream_id: Optional[int] = None) -> List[dict]:
"""
@ -422,6 +427,8 @@ def get_alternate_streams(channel_id: str, current_stream_id: Optional[int] = No
except Exception as e:
logger.error(f"Error getting alternate streams for channel {channel_id}: {e}", exc_info=True)
return []
finally:
close_old_connections()
def validate_stream_url(url, user_agent=None, timeout=(5, 5)):
"""
@ -592,3 +599,5 @@ def get_connections_left(m3u_profile_id: int) -> int:
except Exception as e:
logger.error(f"Error getting connections left for M3U profile {m3u_profile_id}: {e}")
return 0
finally:
close_old_connections()

52
apps/proxy/live_proxy/views.py
View file

@ -3,6 +3,7 @@ import time
import random
import re
import pathlib
from django.db import close_old_connections
from django.http import StreamingHttpResponse, JsonResponse, HttpResponseRedirect, HttpResponse
from django.views.decorators.csrf import csrf_exempt
from django.shortcuts import get_object_or_404
@ -43,6 +44,15 @@ from apps.proxy.utils import check_user_stream_limits
logger = get_logger()
def _channel_stopping_response():
response = JsonResponse(
{"error": "Channel is stopping, retry shortly"},
status=503,
)
response["Retry-After"] = "1"
return response
def _resolve_output_format(user, force=None, request=None):
"""Return the output format string to use for this client."""
_FORMAT_ALIASES = {
@ -123,6 +133,12 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
status=429
)
if ChannelService.is_channel_unavailable_for_new_clients(channel_id):
logger.info(
f"[{client_id}] Channel {channel_id} unavailable. Teardown or pending shutdown"
)
return _channel_stopping_response()
# Check if we need to reinitialize the channel
needs_initialization = True
channel_state = None
@ -144,7 +160,6 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
ChannelState.BUFFERING,
ChannelState.INITIALIZING,
ChannelState.CONNECTING,
ChannelState.STOPPING,
]:
needs_initialization = False
logger.debug(
@ -160,6 +175,11 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
logger.debug(
f"[{client_id}] Channel {channel_id} is still initializing, client will wait"
)
elif channel_state == ChannelState.STOPPING:
logger.info(
f"[{client_id}] Channel {channel_id} is stopping, rejecting request"
)
return _channel_stopping_response()
# Terminal states - channel needs cleanup before reinitialization
elif channel_state in [
ChannelState.ERROR,
@ -192,6 +212,12 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
# Start initialization if needed
if needs_initialization or not proxy_server.check_if_channel_exists(channel_id):
if ChannelService.is_channel_unavailable_for_new_clients(channel_id):
logger.info(
f"[{client_id}] Channel {channel_id} became unavailable before init, rejecting"
)
return _channel_stopping_response()
logger.info(f"[{client_id}] Starting channel {channel_id} initialization")
# Force cleanup of any previous instance if in terminal state
if channel_state in [
@ -414,6 +440,16 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
) # 502 Bad Gateway
# Initialize channel with the stream's user agent (not the client's)
if ChannelService.is_channel_unavailable_for_new_clients(channel_id):
if connection_allocated:
if not channel.release_stream():
logger.warning(f"[{client_id}] Failed to release stream before teardown reject")
connection_allocated = False
logger.info(
f"[{client_id}] Channel {channel_id} unavailable before init call, rejecting"
)
return _channel_stopping_response()
success = ChannelService.initialize_channel(
channel_id,
stream_url,
@ -522,6 +558,16 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
f"[{client_id}] Successfully initialized channel {channel_id} locally"
)
if ChannelService.is_channel_unavailable_for_new_clients(channel_id):
if _client_pre_registered:
mgr = proxy_server.client_managers.get(channel_id)
if mgr:
mgr.remove_client(client_id)
logger.info(
f"[{client_id}] Channel {channel_id} became unavailable during setup, rejecting"
)
return _channel_stopping_response()
# Register client
output_profile = _resolve_output_profile(request, user)
if output_profile:
@ -572,7 +618,9 @@ def stream_ts(request, channel_id, user=None, force_output_format=None):
)
content_type = "video/mp2t"
# Return the StreamingHttpResponse from the main function
# Release ORM checkout before returning a long-lived StreamingHttpResponse.
close_old_connections()
response = StreamingHttpResponse(
streaming_content=generate(), content_type=content_type
)

56
apps/proxy/utils.py
View file

@ -1,6 +1,7 @@
import logging
from core.utils import RedisClient
from apps.proxy.vod_proxy.multi_worker_connection_manager import MultiWorkerVODConnectionManager, get_vod_client_stop_key
from apps.proxy.live_proxy.redis_keys import RedisKeys
from core.models import CoreSettings
from apps.proxy.live_proxy.services.channel_service import ChannelService
@ -61,6 +62,15 @@ def attempt_stream_termination(user_id, requesting_client_id, active_connections
result = ChannelService.stop_client(t['media_id'], t['client_id'])
if result.get("status") == "error":
logger.warning(f"[stream limits][{requesting_client_id}] Failed to stop client {t['client_id']} on channel {t['media_id']}")
elif t['type'] == 'timeshift':
# Same Redis stop key as live; timeshift generator polls it every 5s.
redis_client = RedisClient.get_client()
if not redis_client:
# Deny the new stream if we cannot stop the old one.
return False
stop_key = RedisKeys.client_stop(t['media_id'], t['client_id'])
redis_client.setex(stop_key, 60, "true")
logger.info(f"[stream limits][{requesting_client_id}] Set stop key for timeshift client {t['client_id']}")
else:
connection_manager = MultiWorkerVODConnectionManager.get_instance()
redis_client = connection_manager.redis_client
@ -106,13 +116,14 @@ def get_user_active_connections(user_id):
if user_id is None or (client_user_id and int(client_user_id) == user_id):
try:
logger.debug(f"[stream limits] Found LIVE connection for user {user_id} on channel {channel_id} with client ID {client_id}")
conn_type = 'timeshift' if channel_id.startswith('timeshift_') else 'live'
logger.debug(f"[stream limits] Found {conn_type.upper()} connection for user {user_id} on channel {channel_id} with client ID {client_id}")
connected_at = float(connected_at) if connected_at else 0
connections.append({
'media_id': channel_id,
'client_id': client_id,
'connected_at': connected_at,
'type': 'live',
'type': conn_type,
})
except (ValueError, TypeError):
pass
@ -172,6 +183,22 @@ def check_user_stream_limits(user, client_id, media_id=None):
logger.debug(f"[stream limits][{client_id}] Same-channel reconnect for {media_id} allowed (ignore_same_channel=True)")
return True
# Timeshift sibling range/probe requests share one provider slot per
# session_id. Each distinct client/session still consumes its own slot.
if ignore_same_channel and media_id:
media_id_str = str(media_id)
for conn in active_connections:
if conn.get('type') != 'timeshift':
continue
if conn.get('client_id') != client_id:
continue
conn_media_id = str(conn.get('media_id') or '')
if conn_media_id == media_id_str or conn_media_id.startswith(f"{media_id_str}_"):
logger.debug(
f"[stream limits][{client_id}] Same timeshift session probe for {media_id} allowed (ignore_same_channel=True)"
)
return True
if user_stream_count >= user.stream_limit:
if user_limit_settings.get("terminate_on_limit_exceeded", True) == False:
return False
@ -186,3 +213,28 @@ def check_user_stream_limits(user, client_id, media_id=None):
return False
return True
_TS_PACKET_SIZE = 188
_TS_SYNC_BYTE = 0x47
def find_ts_sync(buf):
"""Return byte offset of the first valid MPEG-TS sync chain in *buf*, or -1.
Args:
buf: Raw bytes from an upstream HTTP response (typically the first 1 KB).
Returns:
Offset of the first 0x47 byte that starts three consecutive 188-byte
packets, or -1. Used to strip PHP/HTML preamble before streaming.
"""
end = len(buf) - 2 * _TS_PACKET_SIZE
for i in range(0, end):
if (
buf[i] == _TS_SYNC_BYTE
and buf[i + _TS_PACKET_SIZE] == _TS_SYNC_BYTE
and buf[i + 2 * _TS_PACKET_SIZE] == _TS_SYNC_BYTE
):
return i
return -1

55
apps/proxy/vod_proxy/tests/test_streamid_fallback.py
View file

@ -23,10 +23,16 @@ unchanged for the happy case. Both branches (movie + episode) exercise:
* UUID miss + no stream_id → Http404 (no fallback attempt)
"""
import logging
from unittest.mock import MagicMock, patch
from django.test import SimpleTestCase
from django.http import Http404
def _wire_m3u_relations(content_mock, relations):
"""Wire content_obj.m3u_relations to the materialised candidate list."""
qs = MagicMock()
qs.select_related.return_value.order_by.return_value = list(relations)
content_mock.m3u_relations.filter.return_value = qs
return qs
# ---------- Movie branch --------------------------------------------------
@ -56,16 +62,17 @@ class TestStreamIdFallbackMovie(SimpleTestCase):
# stream_id so we exit cleanly.
live_relation = MagicMock(stream_id='S1')
live_relation.m3u_account.name = 'AcmeProvider'
live_movie.m3u_relations.filter.return_value.filter.return_value.first.return_value = live_relation
_wire_m3u_relations(live_movie, [live_relation])
with patch('apps.proxy.vod_proxy.views.Movie') as MovieMock, \
patch('apps.proxy.vod_proxy.views.M3UMovieRelation') as RelMock:
MovieMock.objects.filter.return_value.first.return_value = live_movie
content, relation = self._call(
content, relation, candidates = self._call(
content_type='movie', content_id='live-uuid', preferred_stream_id='S1',
)
self.assertIs(content, live_movie)
self.assertIs(relation, live_relation)
self.assertEqual(candidates, [live_relation])
# Fallback path must not have queried the relation table directly
# — happy path is unchanged.
RelMock.objects.filter.assert_not_called()
@ -75,11 +82,11 @@ class TestStreamIdFallbackMovie(SimpleTestCase):
recovered movie is returned and the fallback line is logged."""
recovered_movie = MagicMock(name='Movie', uuid='new-uuid', id=99)
recovered_movie.name = 'Recovered Movie'
fallback_rel = MagicMock(movie=recovered_movie)
fallback_rel = MagicMock(movie=recovered_movie, stream_id='S1')
fallback_rel.m3u_account.name = 'AcmeProvider'
# The fallback only sets content_obj; the existing relation-selection
# logic then re-discovers the same relation via the reverse FK.
recovered_movie.m3u_relations.filter.return_value.filter.return_value.first.return_value = fallback_rel
# The fallback only sets content_obj; relation selection then re-discovers
# the same relation from the materialised candidate list.
_wire_m3u_relations(recovered_movie, [fallback_rel])
with patch('apps.proxy.vod_proxy.views.Movie') as MovieMock, \
patch('apps.proxy.vod_proxy.views.M3UMovieRelation') as RelMock, \
@ -87,10 +94,11 @@ class TestStreamIdFallbackMovie(SimpleTestCase):
MovieMock.objects.filter.return_value.first.return_value = None
# The non-account-scoped fallback chain returns our rel.
RelMock.objects.filter.return_value.select_related.return_value.order_by.return_value.first.return_value = fallback_rel
content, _ = self._call(
content, relation, _ = self._call(
content_type='movie', content_id='dead-uuid', preferred_stream_id='S1',
)
self.assertIs(content, recovered_movie)
self.assertIs(relation, fallback_rel)
self.assertTrue(
any('[STREAMID-FALLBACK]' in m for m in logs.output),
f"expected [STREAMID-FALLBACK] in warnings, got: {logs.output}",
@ -102,9 +110,9 @@ class TestStreamIdFallbackMovie(SimpleTestCase):
fallback. This is the strictest-match-first contract."""
preferred_movie = MagicMock(name='PreferredMovie', uuid='preferred-uuid', id=1)
preferred_movie.name = 'Preferred'
preferred_rel = MagicMock(movie=preferred_movie)
preferred_rel = MagicMock(movie=preferred_movie, stream_id='S1', m3u_account_id=7)
preferred_rel.m3u_account.name = 'Preferred'
preferred_movie.m3u_relations.filter.return_value.filter.return_value.first.return_value = preferred_rel
_wire_m3u_relations(preferred_movie, [preferred_rel])
with patch('apps.proxy.vod_proxy.views.Movie') as MovieMock, \
patch('apps.proxy.vod_proxy.views.M3UMovieRelation') as RelMock:
@ -130,7 +138,7 @@ class TestStreamIdFallbackMovie(SimpleTestCase):
return chain
RelMock.objects.filter.side_effect = filter_router
content, _ = self._call(
content, relation, _ = self._call(
content_type='movie',
content_id='dead-uuid',
preferred_stream_id='S1',
@ -139,19 +147,21 @@ class TestStreamIdFallbackMovie(SimpleTestCase):
# The account-scoped relation wins; the unrestricted-ordered one
# is never consulted because the strict match succeeded.
self.assertIs(content, preferred_movie)
self.assertIs(relation, preferred_rel)
def test_uuid_miss_with_no_stream_id_raises_404(self):
with patch('apps.proxy.vod_proxy.views.Movie') as MovieMock, \
patch('apps.proxy.vod_proxy.views.M3UMovieRelation') as RelMock:
MovieMock.objects.filter.return_value.first.return_value = None
content, relation = self._call(
content, relation, candidates = self._call(
content_type='movie', content_id='dead-uuid', preferred_stream_id=None,
)
# _get_content_and_relation swallows exceptions and returns
# (None, None) for any error including Http404 — caller checks for
# that. Verify the fallback was NEVER attempted.
# (None, None, []) for any error including Http404 — caller checks
# for that. Verify the fallback was NEVER attempted.
self.assertIsNone(content)
self.assertIsNone(relation)
self.assertEqual(candidates, [])
RelMock.objects.filter.assert_not_called()
def test_uuid_miss_with_no_matching_relation_raises_404(self):
@ -161,13 +171,14 @@ class TestStreamIdFallbackMovie(SimpleTestCase):
# Both the account-scoped and unrestricted chains return None.
RelMock.objects.filter.return_value.select_related.return_value.order_by.return_value.first.return_value = None
RelMock.objects.filter.return_value.select_related.return_value.first.return_value = None
content, relation = self._call(
content, relation, candidates = self._call(
content_type='movie',
content_id='dead-uuid',
preferred_stream_id='ghost-stream',
)
self.assertIsNone(content)
self.assertIsNone(relation)
self.assertEqual(candidates, [])
# ---------- Episode branch ------------------------------------------------
@ -192,17 +203,18 @@ class TestStreamIdFallbackEpisode(SimpleTestCase):
recovered_episode = MagicMock(uuid='new-uuid', id=77)
recovered_episode.name = 'Recovered S01E01'
recovered_episode.series.name = 'Recovered Show'
fallback_rel = MagicMock(episode=recovered_episode)
fallback_rel = MagicMock(episode=recovered_episode, stream_id='S99')
fallback_rel.m3u_account.name = 'AcmeProvider'
recovered_episode.m3u_relations.filter.return_value.filter.return_value.first.return_value = fallback_rel
_wire_m3u_relations(recovered_episode, [fallback_rel])
with patch('apps.proxy.vod_proxy.views.Episode') as EpisodeMock, \
patch('apps.proxy.vod_proxy.views.M3UEpisodeRelation') as RelMock, \
self.assertLogs('apps.proxy.vod_proxy.views', level='WARNING') as logs:
EpisodeMock.objects.filter.return_value.first.return_value = None
RelMock.objects.filter.return_value.select_related.return_value.order_by.return_value.first.return_value = fallback_rel
content, _ = self._call(content_id='dead-uuid', preferred_stream_id='S99')
content, relation, _ = self._call(content_id='dead-uuid', preferred_stream_id='S99')
self.assertIs(content, recovered_episode)
self.assertIs(relation, fallback_rel)
self.assertTrue(
any('[STREAMID-FALLBACK]' in m and 'Episode' in m for m in logs.output),
f"expected episode-flavoured [STREAMID-FALLBACK] warning, got: {logs.output}",
@ -214,12 +226,13 @@ class TestStreamIdFallbackEpisode(SimpleTestCase):
live_episode.series.name = 'Live Show'
live_relation = MagicMock(stream_id='S2')
live_relation.m3u_account.name = 'AcmeProvider'
live_episode.m3u_relations.filter.return_value.filter.return_value.first.return_value = live_relation
_wire_m3u_relations(live_episode, [live_relation])
with patch('apps.proxy.vod_proxy.views.Episode') as EpisodeMock, \
patch('apps.proxy.vod_proxy.views.M3UEpisodeRelation') as RelMock:
EpisodeMock.objects.filter.return_value.first.return_value = live_episode
content, relation = self._call(content_id='live-uuid', preferred_stream_id='S2')
content, relation, candidates = self._call(content_id='live-uuid', preferred_stream_id='S2')
self.assertIs(content, live_episode)
self.assertIs(relation, live_relation)
self.assertEqual(candidates, [live_relation])
RelMock.objects.filter.assert_not_called()

146
apps/proxy/vod_proxy/tests/test_vod_db_cleanup.py Normal file
View file

@ -0,0 +1,146 @@
"""VOD proxy must release geventpool checkouts after ORM on stream and stats paths."""
from unittest.mock import MagicMock, patch
from django.http import StreamingHttpResponse
from django.test import RequestFactory, SimpleTestCase
class StreamVodDbCleanupTests(SimpleTestCase):
def setUp(self):
self.factory = RequestFactory()
@patch("apps.proxy.vod_proxy.views.close_old_connections")
@patch("apps.proxy.vod_proxy.views.MultiWorkerVODConnectionManager")
@patch("apps.proxy.vod_proxy.views._transform_url", return_value="http://example.com/movie.mp4")
@patch("apps.proxy.vod_proxy.views._get_m3u_profile")
@patch("apps.proxy.vod_proxy.views._get_stream_url_from_relation", return_value="http://upstream/movie.mp4")
@patch("apps.proxy.vod_proxy.views._get_content_and_relation")
@patch("apps.proxy.vod_proxy.views.network_access_allowed", return_value=True)
def test_stream_vod_closes_db_before_streaming_response(
self,
_network_ok,
mock_content,
_stream_url,
mock_profile,
_transform,
mock_manager_cls,
mock_close,
):
movie = MagicMock()
movie.name = "Test Movie"
relation = MagicMock()
relation.m3u_account.name = "Provider"
mock_content.return_value = (movie, relation, [relation])
profile = MagicMock()
profile.id = 1
profile.max_streams = 5
mock_profile.return_value = (profile, 0)
mock_manager = MagicMock()
mock_manager.stream_content_with_session.return_value = StreamingHttpResponse(
streaming_content=iter([b"data"]),
content_type="video/mp4",
)
mock_manager_cls.get_instance.return_value = mock_manager
request = self.factory.get(
"/proxy/vod/movie/uuid/session123/",
HTTP_USER_AGENT="test-agent",
)
request.user = MagicMock(is_authenticated=False)
from apps.proxy.vod_proxy.views import stream_vod
response = stream_vod(
request,
content_type="movie",
content_id="uuid",
session_id="session123",
)
self.assertIsInstance(response, StreamingHttpResponse)
mock_close.assert_called_once()
mock_manager.stream_content_with_session.assert_called_once()
class BuildVodStatsDbCleanupTests(SimpleTestCase):
@patch("apps.proxy.vod_proxy.views.close_old_connections")
@patch("apps.proxy.vod_proxy.views.Movie")
def test_build_vod_stats_data_closes_db(self, mock_movie, mock_close):
redis_client = MagicMock()
redis_client.scan.side_effect = [
(0, ["vod_persistent_connection:s1"]),
]
redis_client.hgetall.return_value = {
"content_obj_type": "movie",
"content_uuid": "movie-uuid",
"content_name": "Test Movie",
"m3u_profile_id": "1",
"client_ip": "127.0.0.1",
"client_user_agent": "agent",
"connected_at": "1000.0",
"last_activity": "1001.0",
"active_streams": "1",
}
movie_obj = MagicMock(
name="Test Movie",
logo=None,
year=2020,
rating=7.5,
genre="Action",
description="Desc",
tmdb_id="1",
imdb_id="tt1",
)
mock_movie.objects.select_related.return_value.get.return_value = movie_obj
with patch("apps.m3u.models.M3UAccountProfile") as mock_profile_model:
mock_profile_model.objects.select_related.return_value.get.return_value = MagicMock(
name="Profile 1",
m3u_account=MagicMock(name="Account", id=1),
)
from apps.proxy.vod_proxy.views import build_vod_stats_data
stats = build_vod_stats_data(redis_client)
self.assertEqual(stats["total_connections"], 1)
mock_close.assert_called_once()
@patch("apps.proxy.vod_proxy.views.close_old_connections")
def test_build_vod_stats_data_closes_db_on_error(self, mock_close):
redis_client = MagicMock()
redis_client.scan.side_effect = RuntimeError("redis down")
from apps.proxy.vod_proxy.views import build_vod_stats_data
stats = build_vod_stats_data(redis_client)
self.assertEqual(stats["total_connections"], 0)
mock_close.assert_called_once()
class VodStatsUpdateDbCleanupTests(SimpleTestCase):
@patch("core.utils.send_websocket_update")
@patch("apps.proxy.vod_proxy.views.build_vod_stats_data")
def test_do_vod_stats_update_uses_build_vod_stats_data(self, mock_build, mock_ws):
mock_build.return_value = {
"vod_connections": [],
"total_connections": 0,
"timestamp": 0,
}
from apps.proxy.vod_proxy.multi_worker_connection_manager import (
MultiWorkerVODConnectionManager,
)
manager = MultiWorkerVODConnectionManager.__new__(MultiWorkerVODConnectionManager)
manager.redis_client = MagicMock()
manager._do_vod_stats_update()
mock_build.assert_called_once_with(manager.redis_client)
mock_ws.assert_called_once()

90
apps/proxy/vod_proxy/tests/test_vod_failover.py Normal file
View file

@ -0,0 +1,90 @@
"""
Tests for VOD provider failover (PR: "Add VOD failover logic for M3U relations").
The VOD proxy previously selected a single highest-priority relation and
returned 503 if that account was at capacity, without trying other accounts
that carry the same title.
`_get_content_and_relation()` now materialises the active, priority-ordered
relations once (single DB query) and returns that list. `_order_candidates()`
is a pure in-memory helper that moves the preferred relation to the front and
removes duplicates, so the initial connection path hits the database exactly
once. stream_vod()/head_vod() then walk the ordered list and use the first
account with spare capacity.
These tests cover the in-memory ordering helper: preferred-first placement,
de-duplication, empty-input fallbacks, and the guarantee that it performs no
database access.
"""
from unittest.mock import MagicMock
from django.test import TestCase
from apps.proxy.vod_proxy.views import _order_candidates
def _rel(rel_id, priority):
rel = MagicMock()
rel.id = rel_id
rel.m3u_account = MagicMock()
rel.m3u_account.priority = priority
return rel
class TestOrderCandidates(TestCase):
def test_preferred_relation_is_placed_first(self):
preferred = _rel(rel_id=2, priority=0)
candidates = [_rel(rel_id=1, priority=5), _rel(rel_id=3, priority=2), preferred]
result = _order_candidates(candidates, preferred_relation=preferred)
self.assertEqual(result[0].id, 2, "Preferred relation must be first")
self.assertEqual({r.id for r in result}, {1, 2, 3})
def test_preferred_relation_not_duplicated(self):
preferred = _rel(rel_id=2, priority=0)
candidates = [preferred, _rel(rel_id=1, priority=5)]
result = _order_candidates(candidates, preferred_relation=preferred)
ids = [r.id for r in result]
self.assertEqual(ids.count(2), 1, "Preferred relation must not be duplicated")
self.assertEqual(len(result), 2)
def test_no_preferred_keeps_order(self):
candidates = [_rel(rel_id=1, priority=0), _rel(rel_id=2, priority=5)]
result = _order_candidates(candidates, preferred_relation=None)
self.assertEqual([r.id for r in result], [1, 2])
def test_empty_with_preferred_returns_preferred(self):
preferred = _rel(rel_id=7, priority=0)
result = _order_candidates([], preferred_relation=preferred)
self.assertEqual(result, [preferred])
def test_empty_without_preferred_returns_empty(self):
result = _order_candidates([], preferred_relation=None)
self.assertEqual(result, [])
def test_no_database_access(self):
"""The helper must be pure in-memory: it must never touch the ORM."""
class Boom:
def __init__(self, rel_id, priority):
self.id = rel_id
self.m3u_account = MagicMock()
self.m3u_account.priority = priority
def __getattr__(self, name):
if name in ('filter', 'objects', 'all', 'select_related', 'order_by'):
raise AssertionError(f"ORM access attempted via .{name}()")
raise AttributeError(name)
candidates = [Boom(1, 0), Boom(2, 5)]
result = _order_candidates(candidates, preferred_relation=None)
self.assertEqual([r.id for r in result], [1, 2])

191
apps/proxy/vod_proxy/views.py
View file

@ -8,6 +8,7 @@ import random
import logging
import requests
from urllib.parse import urlencode
from django.db import close_old_connections
from django.http import JsonResponse, Http404, HttpResponse
from django.shortcuts import get_object_or_404
from django.views.decorators.csrf import csrf_exempt
@ -23,6 +24,7 @@ from rest_framework_simplejwt.authentication import JWTAuthentication
from apps.accounts.authentication import ApiKeyAuthentication, QueryParamJWTAuthentication
from apps.proxy.utils import check_user_stream_limits
from dispatcharr.utils import network_access_allowed
from core.utils import dispatcharr_user_agent
logger = logging.getLogger(__name__)
@ -84,32 +86,39 @@ def _get_content_and_relation(content_type, content_id, preferred_m3u_account_id
)
logger.info(f"[CONTENT-FOUND] Movie: {content_obj.name} (ID: {content_obj.id})")
# Filter by preferred stream ID first (most specific)
relations_query = content_obj.m3u_relations.filter(m3u_account__is_active=True)
# Materialise the active relations once (single DB hit), ordered by
# priority. Selection below is done in memory, and the full ordered
# list is returned so the caller can fail over without re-querying.
candidates = list(
content_obj.m3u_relations
.filter(m3u_account__is_active=True)
.select_related('m3u_account')
.order_by('-m3u_account__priority', 'id')
)
if preferred_stream_id:
specific_relation = relations_query.filter(stream_id=preferred_stream_id).first()
specific_relation = next(
(r for r in candidates if str(r.stream_id) == str(preferred_stream_id)), None)
if specific_relation:
logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}")
return content_obj, specific_relation
return content_obj, specific_relation, candidates
else:
logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection")
# Filter by preferred M3U account if specified
if preferred_m3u_account_id:
specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first()
specific_relation = next(
(r for r in candidates if r.m3u_account_id == preferred_m3u_account_id), None)
if specific_relation:
logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}")
return content_obj, specific_relation
return content_obj, specific_relation, candidates
else:
logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority")
# Get the highest priority active relation (fallback or default)
relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first()
relation = candidates[0] if candidates else None
if relation:
logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})")
return content_obj, relation
return content_obj, relation, candidates
elif content_type == 'episode':
content_obj = Episode.objects.filter(uuid=content_id).first()
@ -152,32 +161,39 @@ def _get_content_and_relation(content_type, content_id, preferred_m3u_account_id
)
logger.info(f"[CONTENT-FOUND] Episode: {content_obj.name} (ID: {content_obj.id}, Series: {content_obj.series.name})")
# Filter by preferred stream ID first (most specific)
relations_query = content_obj.m3u_relations.filter(m3u_account__is_active=True)
# Materialise the active relations once (single DB hit), ordered by
# priority. Selection below is done in memory, and the full ordered
# list is returned so the caller can fail over without re-querying.
candidates = list(
content_obj.m3u_relations
.filter(m3u_account__is_active=True)
.select_related('m3u_account')
.order_by('-m3u_account__priority', 'id')
)
if preferred_stream_id:
specific_relation = relations_query.filter(stream_id=preferred_stream_id).first()
specific_relation = next(
(r for r in candidates if str(r.stream_id) == str(preferred_stream_id)), None)
if specific_relation:
logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}")
return content_obj, specific_relation
return content_obj, specific_relation, candidates
else:
logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection")
# Filter by preferred M3U account if specified
if preferred_m3u_account_id:
specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first()
specific_relation = next(
(r for r in candidates if r.m3u_account_id == preferred_m3u_account_id), None)
if specific_relation:
logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}")
return content_obj, specific_relation
return content_obj, specific_relation, candidates
else:
logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority")
# Get the highest priority active relation (fallback or default)
relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first()
relation = candidates[0] if candidates else None
if relation:
logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})")
return content_obj, relation
return content_obj, relation, candidates
elif content_type == 'series':
# For series, get the first episode
@ -186,43 +202,64 @@ def _get_content_and_relation(content_type, content_id, preferred_m3u_account_id
episode = series.episodes.first()
if not episode:
logger.error(f"[CONTENT-ERROR] No episodes found for series {series.name}")
return None, None
return None, None, []
logger.info(f"[CONTENT-FOUND] First episode: {episode.name} (ID: {episode.id})")
# Filter by preferred stream ID first (most specific)
relations_query = episode.m3u_relations.filter(m3u_account__is_active=True)
# Materialise once (single DB hit), ordered by priority; select in memory.
candidates = list(
episode.m3u_relations
.filter(m3u_account__is_active=True)
.select_related('m3u_account')
.order_by('-m3u_account__priority', 'id')
)
if preferred_stream_id:
specific_relation = relations_query.filter(stream_id=preferred_stream_id).first()
specific_relation = next(
(r for r in candidates if str(r.stream_id) == str(preferred_stream_id)), None)
if specific_relation:
logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}")
return episode, specific_relation
return episode, specific_relation, candidates
else:
logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection")
# Filter by preferred M3U account if specified
if preferred_m3u_account_id:
specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first()
specific_relation = next(
(r for r in candidates if r.m3u_account_id == preferred_m3u_account_id), None)
if specific_relation:
logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}")
return episode, specific_relation
return episode, specific_relation, candidates
else:
logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority")
# Get the highest priority active relation (fallback or default)
relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first()
relation = candidates[0] if candidates else None
if relation:
logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})")
return episode, relation
return episode, relation, candidates
else:
logger.error(f"[CONTENT-ERROR] Invalid content type: {content_type}")
return None, None
return None, None, []
except Exception as e:
logger.error(f"Error getting content object: {e}")
return None, None
return None, None, []
def _order_candidates(candidates, preferred_relation=None):
"""In-memory ordering helper (no DB access).
`candidates` is the already-materialised, priority-ordered list of active
relations produced by _get_content_and_relation(). This helper only moves
the preferred relation to the front and removes duplicates, so the initial
connection path hits the database exactly once.
"""
if not candidates:
return [preferred_relation] if preferred_relation else []
if preferred_relation is not None:
return [preferred_relation] + [
r for r in candidates if r.id != preferred_relation.id
]
return list(candidates)
def _get_stream_url_from_relation(relation):
"""Get stream URL from the M3U relation"""
@ -558,32 +595,44 @@ def stream_vod(request, content_type, content_id, session_id=None, profile_id=No
logger.info(f"[VOD-PARAM] Preferred stream ID: {preferred_stream_id}")
# Get the content object and its relation
content_obj, relation = _get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id)
content_obj, relation, candidates = _get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id)
if not content_obj or not relation:
logger.error(f"[VOD-ERROR] Content or relation not found: {content_type} {content_id}")
raise Http404(f"Content not found: {content_type} {content_id}")
logger.info(f"[VOD-CONTENT] Found content: {getattr(content_obj, 'name', 'Unknown')}")
# Get M3U account from relation
m3u_account = relation.m3u_account
logger.info(f"[VOD-ACCOUNT] Using M3U account: {m3u_account.name}")
# Provider failover: walk the already-materialised candidate list
# (preferred relation first, then by account priority) and use the
# first account whose profile pool has spare capacity. Only return 503
# when every account carrying the title is genuinely full. No extra DB
# query — candidates come from _get_content_and_relation().
ordered = _order_candidates(candidates, relation)
m3u_account = stream_url = profile_result = None
for cand in ordered:
cand_account = cand.m3u_account
cand_url = _get_stream_url_from_relation(cand)
if not cand_url:
logger.warning(f"[VOD-FAILOVER] No URL for relation on account {cand_account.name}, skipping")
continue
cand_profile = _get_m3u_profile(cand_account, profile_id, session_id)
if cand_profile and cand_profile[0]:
relation = cand
m3u_account = cand_account
stream_url = cand_url
profile_result = cand_profile
logger.info(f"[VOD-FAILOVER] Selected account {cand_account.name} (priority {cand_account.priority})")
break
else:
logger.warning(f"[VOD-FAILOVER] Account {cand_account.name} at capacity, trying next provider")
# Get stream URL from relation
stream_url = _get_stream_url_from_relation(relation)
logger.info(f"[VOD-CONTENT] Content URL: {stream_url or 'No URL found'}")
if not stream_url:
logger.error(f"[VOD-ERROR] No stream URL available for {content_type} {content_id}")
return HttpResponse("No stream URL available", status=503)
# Get M3U profile (returns profile and current connection count)
profile_result = _get_m3u_profile(m3u_account, profile_id, session_id)
if not profile_result or not profile_result[0]:
logger.error(f"[VOD-ERROR] No suitable M3U profile found for {content_type} {content_id}")
if not m3u_account or not stream_url or not profile_result or not profile_result[0]:
logger.error(f"[VOD-ERROR] No available provider with capacity for {content_type} {content_id}")
return HttpResponse("No available stream", status=503)
logger.info(f"[VOD-ACCOUNT] Using M3U account: {m3u_account.name}")
logger.info(f"[VOD-CONTENT] Content URL: {stream_url or 'No URL found'}")
m3u_profile, current_connections = profile_result
logger.info(f"[VOD-PROFILE] Using M3U profile: {m3u_profile.id} (max_streams: {m3u_profile.max_streams}, current: {current_connections})")
@ -600,6 +649,9 @@ def stream_vod(request, content_type, content_id, session_id=None, profile_id=No
# Get connection manager (Redis-backed for multi-worker support)
connection_manager = MultiWorkerVODConnectionManager.get_instance()
# Release ORM checkout before returning a long-lived StreamingHttpResponse.
close_old_connections()
# Stream the content with session-based connection reuse
logger.info("[VOD-STREAM] Calling connection manager to stream content")
response = connection_manager.stream_content_with_session(
@ -676,22 +728,29 @@ def head_vod(request, content_type, content_id, session_id=None, profile_id=None
logger.info(f"[VOD-HEAD] Preferred stream ID: {preferred_stream_id}")
# Get content and relation (same as GET)
content_obj, relation = _get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id)
content_obj, relation, candidates = _get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id)
if not content_obj or not relation:
logger.error(f"[VOD-HEAD] Content or relation not found: {content_type} {content_id}")
return HttpResponse("Content not found", status=404)
# Get M3U account and stream URL
m3u_account = relation.m3u_account
stream_url = _get_stream_url_from_relation(relation)
if not stream_url:
logger.error(f"[VOD-HEAD] No stream URL available for {content_type} {content_id}")
return HttpResponse("No stream URL available", status=503)
# Provider failover (same logic as GET), reusing the materialised list.
ordered = _order_candidates(candidates, relation)
m3u_account = stream_url = profile_result = None
for cand in ordered:
cand_account = cand.m3u_account
cand_url = _get_stream_url_from_relation(cand)
if not cand_url:
continue
cand_profile = _get_m3u_profile(cand_account, profile_id, session_id)
if cand_profile and cand_profile[0]:
relation = cand
m3u_account = cand_account
stream_url = cand_url
profile_result = cand_profile
break
# Get M3U profile (returns profile and current connection count)
profile_result = _get_m3u_profile(m3u_account, profile_id, session_id)
if not profile_result or not profile_result[0]:
logger.error(f"[VOD-HEAD] No M3U profile found or all profiles at capacity")
if not m3u_account or not stream_url or not profile_result or not profile_result[0]:
logger.error(f"[VOD-HEAD] No available provider with capacity for {content_type} {content_id}")
return HttpResponse("No available stream", status=503)
m3u_profile, current_connections = profile_result
@ -704,7 +763,7 @@ def head_vod(request, content_type, content_id, session_id=None, profile_id=None
# Use M3U account's user agent as primary, client user agent as fallback
m3u_user_agent = m3u_account.get_user_agent().user_agent if m3u_account.get_user_agent() else None
headers = {
'User-Agent': m3u_user_agent or client_user_agent or 'Dispatcharr/1.0',
'User-Agent': m3u_user_agent or client_user_agent or dispatcharr_user_agent(),
'Accept': '*/*',
'Range': 'bytes=0-1' # Request only first 2 bytes
}
@ -1062,6 +1121,8 @@ def build_vod_stats_data(redis_client):
except Exception as e:
logger.error(f"Error building VOD stats: {e}")
return {'vod_connections': [], 'total_connections': 0, 'timestamp': time.time()}
finally:
close_old_connections()
@api_view(["GET"])

0
apps/timeshift/__init__.py Normal file
View file

7
apps/timeshift/apps.py Normal file
View file

@ -0,0 +1,7 @@
from django.apps import AppConfig
class TimeshiftConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "apps.timeshift"
verbose_name = "Timeshift"

252
apps/timeshift/helpers.py Normal file
View file

@ -0,0 +1,252 @@
"""URL builders and timestamp helpers for XC catch-up."""
import logging
import re
from collections import namedtuple
from datetime import datetime, timezone
from urllib.parse import quote
from zoneinfo import ZoneInfo
logger = logging.getLogger(__name__)
# Credentials for the profile whose pool slot was reserved (not raw account fields).
TimeshiftCredentials = namedtuple(
"TimeshiftCredentials", ("server_url", "username", "password")
)
DEFAULT_DURATION_MINUTES = 120
DURATION_BUFFER_MINUTES = 5
MAX_DURATION_MINUTES = 480
# Wall-clock shapes seen from XC / iPlayTV / TiviMate clients. Compiled once.
_CATCHUP_WALL_CLOCK_RE = re.compile(
r"^"
r"(?P<date>\d{4}-\d{2}-\d{2})"
r"(?P<dtsep>[:_]| )"
r"(?P<hour>\d{2})"
r"(?P<hmsep>[-:])"
r"(?P<minute>\d{2})"
r"(?:"
r":"
r"(?P<second>\d{2})"
r")?"
r"$"
)
def normalize_catchup_timestamp_input(timestamp_str):
"""Map a client catch-up timestamp to an ISO-8601 string for ``fromisoformat``.
Supported inputs:
- ``YYYY-MM-DD:HH-MM`` (iPlayTV/TiviMate colon-dash)
- ``YYYY-MM-DD_HH-MM`` (XC underscore)
- ``YYYY-MM-DD:HH:MM[:SS]`` (XC colon time in catch-up URLs)
- ``YYYY-MM-DD HH:MM[:SS]`` (EPG / SQL datetime)
- Unix epoch seconds (10 digits) or milliseconds (13 digits)
Returns:
An ISO-8601 date-time string (``YYYY-MM-DDTHH:MM:SS``), or None if
the value does not match a known catch-up shape.
"""
if timestamp_str is None:
return None
if not isinstance(timestamp_str, str):
timestamp_str = str(timestamp_str)
value = timestamp_str.strip()
if not value:
return None
if value.isdigit():
length = len(value)
if length == 10:
dt = datetime.fromtimestamp(int(value), tz=timezone.utc)
return dt.replace(tzinfo=None).isoformat(timespec="seconds")
if length == 13:
dt = datetime.fromtimestamp(int(value) / 1000, tz=timezone.utc)
return dt.replace(tzinfo=None).isoformat(timespec="seconds")
return None
match = _CATCHUP_WALL_CLOCK_RE.match(value)
if not match:
return None
parts = match.groupdict()
second = parts["second"] or "00"
return f"{parts['date']}T{parts['hour']}:{parts['minute']}:{second}"
def parse_catchup_timestamp(timestamp_str):
"""Parse a catch-up timestamp string into a naive UTC wall-clock datetime.
See ``normalize_catchup_timestamp_input`` for supported input shapes.
Returns:
A naive datetime on success, or None.
"""
iso_value = normalize_catchup_timestamp_input(timestamp_str)
if iso_value is None:
if timestamp_str is not None and str(timestamp_str).strip():
logger.debug(
"Timeshift: unrecognised catch-up timestamp: %r", timestamp_str
)
return None
try:
return datetime.fromisoformat(iso_value)
except ValueError:
logger.debug(
"Timeshift: invalid catch-up timestamp after normalize: %r -> %r",
timestamp_str,
iso_value,
)
return None
def _reshape_timestamp(timestamp, strftime_fmt, label):
dt = parse_catchup_timestamp(timestamp)
if dt is None:
logger.error(
"Timeshift %s reshape failed for %r: unrecognised format", label, timestamp
)
return timestamp
return dt.strftime(strftime_fmt)
def convert_timestamp_to_provider_tz(timestamp_str, provider_tz_name):
"""Convert a UTC catch-up timestamp to the provider's local zone.
Args:
timestamp_str: UTC wall-clock in ``YYYY-MM-DD:HH-MM`` or underscore form.
provider_tz_name: IANA zone from the provider's ``server_info.timezone``
(e.g. ``Europe/Brussels``). Falsy, ``UTC``, or unknown: no conversion.
Returns:
``YYYY-MM-DD:HH-MM`` in the provider zone, or the input unchanged on skip/failure.
"""
if not provider_tz_name or provider_tz_name == "UTC":
return timestamp_str
dt = parse_catchup_timestamp(timestamp_str)
if dt is None:
return timestamp_str
try:
target = ZoneInfo(provider_tz_name)
except Exception:
logger.warning(
"Timeshift: unknown provider timezone %r, no conversion applied",
provider_tz_name,
)
return timestamp_str
# timezone.utc, not ZoneInfo("UTC"): avoids mis-set Docker /etc/timezone.
local_dt = dt.replace(tzinfo=timezone.utc).astimezone(target)
return local_dt.strftime("%Y-%m-%d:%H-%M")
def get_programme_duration(channel, timestamp_str):
"""Look up catch-up duration in minutes from EPG.
Args:
channel: Channel with optional ``epg_data`` relation loaded.
timestamp_str: Programme start in UTC (same shape as the client URL).
Returns:
Programme length plus a small buffer, capped at ``MAX_DURATION_MINUTES``,
or ``DEFAULT_DURATION_MINUTES`` when EPG lookup fails.
"""
try:
dt = parse_catchup_timestamp(timestamp_str)
if dt is None:
return DEFAULT_DURATION_MINUTES
# EPG times are timezone-aware; parsed value must be too.
dt = dt.replace(tzinfo=timezone.utc)
if not channel.epg_data:
return DEFAULT_DURATION_MINUTES
programme = channel.epg_data.programs.filter(
start_time__lte=dt, end_time__gt=dt
).first()
if not programme:
return DEFAULT_DURATION_MINUTES
duration_seconds = (programme.end_time - programme.start_time).total_seconds()
duration_minutes = int(duration_seconds / 60) + DURATION_BUFFER_MINUTES
return min(duration_minutes, MAX_DURATION_MINUTES)
except Exception:
return DEFAULT_DURATION_MINUTES
def build_timeshift_url_format_a(creds, stream_id, timestamp, duration_minutes):
"""QUERY layout: ``/streaming/timeshift.php?username=...&start=...``"""
return (
f"{creds.server_url.rstrip('/')}/streaming/timeshift.php"
f"?username={quote(str(creds.username), safe='')}"
f"&password={quote(str(creds.password), safe='')}"
f"&stream={stream_id}"
f"&start={timestamp}"
f"&duration={duration_minutes}"
)
def build_timeshift_url_format_b(creds, stream_id, timestamp, duration_minutes):
"""PATH layout: ``/timeshift/{user}/{pass}/{dur}/{start}/{id}.ts``"""
return (
f"{creds.server_url.rstrip('/')}/timeshift"
f"/{quote(str(creds.username), safe='')}"
f"/{quote(str(creds.password), safe='')}"
f"/{duration_minutes}"
f"/{timestamp}"
f"/{stream_id}.ts"
)
def build_timeshift_candidate_urls(creds, stream_id, timestamp, duration_minutes):
"""Build ordered upstream URL candidates (PATH forms first, QUERY last).
Args:
creds: ``TimeshiftCredentials`` for the reserved profile.
stream_id: Provider stream id from the catch-up stream's custom properties.
timestamp: Already converted to the serving provider's local zone.
duration_minutes: Archive window length passed to the provider.
Returns:
List of URL strings to try in order. QUERY forms are last because some
providers return live TV even when ``start`` is set.
"""
dt = parse_catchup_timestamp(timestamp)
if dt is None:
colon_dash_ts = timestamp
underscore_ts = timestamp
colon_seconds_ts = timestamp
sql_ts = timestamp
else:
colon_dash_ts = dt.strftime("%Y-%m-%d:%H-%M")
underscore_ts = dt.strftime("%Y-%m-%d_%H-%M")
colon_seconds_ts = dt.strftime("%Y-%m-%d:%H:%M:%S")
sql_ts = dt.strftime("%Y-%m-%d %H:%M:%S")
return [
build_timeshift_url_format_b(creds, stream_id, colon_dash_ts, duration_minutes),
build_timeshift_url_format_b(creds, stream_id, underscore_ts, duration_minutes),
build_timeshift_url_format_b(creds, stream_id, colon_seconds_ts, duration_minutes),
build_timeshift_url_format_a(creds, stream_id, underscore_ts, duration_minutes),
build_timeshift_url_format_a(creds, stream_id, sql_ts, duration_minutes),
build_timeshift_url_format_a(creds, stream_id, colon_dash_ts, duration_minutes),
build_timeshift_url_format_a(creds, stream_id, colon_seconds_ts, duration_minutes),
]
def format_timestamp_as_colon_dash(timestamp):
"""Reshape to ``YYYY-MM-DD:HH-MM`` without timezone conversion."""
return _reshape_timestamp(timestamp, "%Y-%m-%d:%H-%M", "colon-dash")
def format_timestamp_as_colon_seconds(timestamp):
"""Reshape to ``YYYY-MM-DD:HH:MM:SS`` without timezone conversion."""
return _reshape_timestamp(timestamp, "%Y-%m-%d:%H:%M:%S", "colon-seconds")
def format_timestamp_as_underscore(timestamp):
"""Reshape to ``YYYY-MM-DD_HH-MM`` without timezone conversion."""
return _reshape_timestamp(timestamp, "%Y-%m-%d_%H-%M", "underscore")
def format_timestamp_as_sql_datetime(timestamp):
"""Reshape to ``YYYY-MM-DD HH:MM:SS`` without timezone conversion."""
return _reshape_timestamp(timestamp, "%Y-%m-%d %H:%M:%S", "SQL")

0
apps/timeshift/tests/__init__.py Normal file
View file

325
apps/timeshift/tests/test_helpers.py Normal file
View file

@ -0,0 +1,325 @@
"""Tests for `apps.timeshift.helpers`: timestamp shape conversion and URL build."""
from datetime import datetime, timezone
from django.test import TestCase
from apps.timeshift.helpers import (
TimeshiftCredentials,
build_timeshift_candidate_urls,
build_timeshift_url_format_a,
build_timeshift_url_format_b,
convert_timestamp_to_provider_tz,
format_timestamp_as_colon_dash,
format_timestamp_as_colon_seconds,
format_timestamp_as_sql_datetime,
format_timestamp_as_underscore,
normalize_catchup_timestamp_input,
parse_catchup_timestamp,
)
def _make_creds():
# The builders consume resolved per-profile credentials, never an account
# object — get_transformed_credentials() produces these in the view.
return TimeshiftCredentials("http://example.test", "user", "pass")
class TimestampFormatTests(TestCase):
"""Timestamp reshape functions change format only; no timezone conversion."""
def test_normalize_colon_dash_shape(self):
self.assertEqual(
normalize_catchup_timestamp_input("2026-05-21:12-55"),
"2026-05-21T12:55:00",
)
def test_normalize_colon_seconds_xc_format(self):
self.assertEqual(
normalize_catchup_timestamp_input("2026-06-23:04:00:00"),
"2026-06-23T04:00:00",
)
def test_normalize_epg_sql_format(self):
self.assertEqual(
normalize_catchup_timestamp_input("2026-06-23 04:00:00"),
"2026-06-23T04:00:00",
)
def test_normalize_unix_epoch_seconds(self):
epoch = str(int(datetime(2026, 6, 23, 4, 0, 0, tzinfo=timezone.utc).timestamp()))
self.assertEqual(
normalize_catchup_timestamp_input(epoch),
"2026-06-23T04:00:00",
)
def test_normalize_unix_epoch_milliseconds(self):
epoch_ms = str(
int(datetime(2026, 6, 23, 4, 0, 0, tzinfo=timezone.utc).timestamp() * 1000)
)
self.assertEqual(
normalize_catchup_timestamp_input(epoch_ms),
"2026-06-23T04:00:00",
)
def test_normalize_rejects_garbage(self):
self.assertIsNone(normalize_catchup_timestamp_input("garbage"))
self.assertIsNone(normalize_catchup_timestamp_input(""))
self.assertIsNone(normalize_catchup_timestamp_input("12345"))
def test_parse_rejects_invalid_calendar_date(self):
self.assertIsNone(parse_catchup_timestamp("2026-13-45:04-00"))
def test_parse_colon_dash_format(self):
dt = parse_catchup_timestamp("2026-05-21:12-55")
self.assertEqual(dt, datetime(2026, 5, 21, 12, 55, 0))
def test_parse_underscore_format(self):
dt = parse_catchup_timestamp("2026-05-21_12-55")
self.assertEqual(dt, datetime(2026, 5, 21, 12, 55, 0))
def test_parse_colon_minutes_without_seconds(self):
dt = parse_catchup_timestamp("2026-06-23:04:00")
self.assertEqual(dt, datetime(2026, 6, 23, 4, 0, 0))
def test_parse_colon_seconds_xc_format(self):
dt = parse_catchup_timestamp("2026-06-23:04:00:00")
self.assertEqual(dt, datetime(2026, 6, 23, 4, 0, 0))
def test_parse_epg_sql_format(self):
dt = parse_catchup_timestamp("2026-06-23 04:00:00")
self.assertEqual(dt, datetime(2026, 6, 23, 4, 0, 0))
def test_format_colon_dash_from_colon_seconds(self):
self.assertEqual(
format_timestamp_as_colon_dash("2026-06-23:04:00:00"),
"2026-06-23:04-00",
)
def test_format_colon_seconds_from_colon_dash(self):
self.assertEqual(
format_timestamp_as_colon_seconds("2026-06-23:04-00"),
"2026-06-23:04:00:00",
)
def test_format_colon_seconds_from_unix_epoch(self):
epoch = str(int(datetime(2026, 6, 23, 4, 0, 0, tzinfo=timezone.utc).timestamp()))
self.assertEqual(
format_timestamp_as_colon_dash(epoch),
"2026-06-23:04-00",
)
def test_format_sql_reshapes_without_tz_conversion(self):
self.assertEqual(
format_timestamp_as_sql_datetime("2026-05-12:17-00"),
"2026-05-12 17:00:00",
)
def test_format_sql_accepts_underscore_input(self):
self.assertEqual(
format_timestamp_as_sql_datetime("2026-05-12_17-00"),
"2026-05-12 17:00:00",
)
def test_format_sql_invalid_falls_back(self):
self.assertEqual(format_timestamp_as_sql_datetime("garbage"), "garbage")
def test_format_underscore_from_colon_dash(self):
self.assertEqual(
format_timestamp_as_underscore("2026-05-21:12-55"),
"2026-05-21_12-55",
)
def test_format_underscore_idempotent(self):
# Underscore input → underscore output (no change)
self.assertEqual(
format_timestamp_as_underscore("2026-05-21_12-55"),
"2026-05-21_12-55",
)
def test_format_underscore_invalid_falls_back(self):
self.assertEqual(format_timestamp_as_underscore("garbage"), "garbage")
class BuildTimeshiftUrlTests(TestCase):
def setUp(self):
self.creds = _make_creds()
def test_format_a_passes_dash_shape_unchanged(self):
url = build_timeshift_url_format_a(
self.creds, "22372", "2026-05-12:19-00", 40
)
self.assertIn("start=2026-05-12:19-00", url)
self.assertIn("stream=22372", url)
self.assertIn("duration=40", url)
def test_format_a_passes_sql_shape_unchanged(self):
url = build_timeshift_url_format_a(
self.creds, "22372", "2026-05-12 19:00:00", 40
)
self.assertIn("start=2026-05-12 19:00:00", url)
def test_format_b_path_with_dash_shape(self):
url = build_timeshift_url_format_b(
self.creds, "22372", "2026-05-12:19-00", 40
)
self.assertIn("/40/2026-05-12:19-00/22372.ts", url)
class CandidateOrderingTests(TestCase):
"""`build_timeshift_candidate_urls` must try the PATH form (which seeks the
archive) before the QUERY form (which returns LIVE on some providers,
silently ignoring the requested timestamp). Regression guard for the
"catch-up plays the live stream instead of the requested programme" bug."""
def setUp(self):
self.creds = _make_creds()
def _is_path_form(self, url):
return "/timeshift/" in url and url.endswith(".ts") and "timeshift.php" not in url
def _is_query_form(self, url):
return "timeshift.php?" in url
def test_every_path_candidate_precedes_every_query_candidate(self):
urls = build_timeshift_candidate_urls(self.creds, "22372", "2026-05-12:19-00", 40)
path_indices = [i for i, u in enumerate(urls) if self._is_path_form(u)]
query_indices = [i for i, u in enumerate(urls) if self._is_query_form(u)]
# Each URL is classified as exactly one form.
self.assertEqual(len(path_indices) + len(query_indices), len(urls))
self.assertTrue(path_indices and query_indices)
# The last PATH candidate still comes before the first QUERY candidate.
self.assertLess(max(path_indices), min(query_indices))
def test_first_candidate_is_path_form_with_canonical_dash_timestamp(self):
urls = build_timeshift_candidate_urls(self.creds, "22372", "2026-05-12:19-00", 40)
self.assertTrue(self._is_path_form(urls[0]))
# Canonical colon-dash timestamp, passed through unchanged.
self.assertIn("/40/2026-05-12:19-00/22372.ts", urls[0])
def test_accepts_colon_seconds_input_timestamp(self):
urls = build_timeshift_candidate_urls(
self.creds, "22372", "2026-06-23:04:00:00", 40
)
self.assertTrue(self._is_path_form(urls[0]))
self.assertIn("/40/2026-06-23:04-00/22372.ts", urls[0])
self.assertIn("/40/2026-06-23:04:00:00/22372.ts", urls[2])
def test_accepts_underscore_input_timestamp(self):
# Client may send the underscore shape; PATH form still leads.
urls = build_timeshift_candidate_urls(self.creds, "22372", "2026-05-12_19-00", 40)
self.assertTrue(self._is_path_form(urls[0]))
class ConvertTimestampToProviderTzTests(TestCase):
"""`convert_timestamp_to_provider_tz` shifts a UTC catch-up timestamp into the
serving provider's local zone (XC providers index archives in their own zone),
DST-correct, and is a no-op when the zone is UTC/unknown/missing."""
def test_utc_to_brussels_summer_is_plus_two(self):
# June → CEST (+02:00): 17:00 UTC == 19:00 Brussels (the 19h JT case).
self.assertEqual(
convert_timestamp_to_provider_tz("2026-06-08:17-00", "Europe/Brussels"),
"2026-06-08:19-00",
)
def test_utc_to_brussels_winter_is_plus_one(self):
# January → CET (+01:00): 17:00 UTC == 18:00 Brussels (DST handled).
self.assertEqual(
convert_timestamp_to_provider_tz("2026-01-08:17-00", "Europe/Brussels"),
"2026-01-08:18-00",
)
def test_day_rollover(self):
# 23:30 UTC + 2h (CEST) crosses midnight into the next day.
self.assertEqual(
convert_timestamp_to_provider_tz("2026-06-08:23-30", "Europe/Brussels"),
"2026-06-09:01-30",
)
def test_underscore_input_returns_colon_dash(self):
self.assertEqual(
convert_timestamp_to_provider_tz("2026-06-08_17-00", "Europe/Brussels"),
"2026-06-08:19-00",
)
def test_utc_zone_is_noop(self):
self.assertEqual(
convert_timestamp_to_provider_tz("2026-06-08:17-00", "UTC"),
"2026-06-08:17-00",
)
def test_none_zone_is_noop(self):
self.assertEqual(
convert_timestamp_to_provider_tz("2026-06-08:17-00", None),
"2026-06-08:17-00",
)
def test_unknown_zone_is_noop(self):
self.assertEqual(
convert_timestamp_to_provider_tz("2026-06-08:17-00", "Mars/Phobos"),
"2026-06-08:17-00",
)
def test_utc_to_brussels_from_unix_epoch(self):
epoch = str(int(datetime(2026, 6, 8, 17, 0, 0, tzinfo=timezone.utc).timestamp()))
self.assertEqual(
convert_timestamp_to_provider_tz(epoch, "Europe/Brussels"),
"2026-06-08:19-00",
)
def test_garbage_timestamp_passthrough(self):
self.assertEqual(
convert_timestamp_to_provider_tz("garbage", "Europe/Brussels"),
"garbage",
)
class GetProgrammeDurationTests(TestCase):
"""Duration window resolution: programme length + buffer, capped, with a
safe default whenever the EPG lookup cannot resolve."""
def _channel_with_programme(self, minutes):
from datetime import datetime, timedelta, timezone as dt_timezone
from unittest.mock import MagicMock
start = datetime(2026, 6, 8, 17, 0, tzinfo=dt_timezone.utc)
programme = MagicMock(
start_time=start, end_time=start + timedelta(minutes=minutes)
)
channel = MagicMock()
channel.epg_data.programs.filter.return_value.first.return_value = programme
return channel
def test_duration_is_programme_length_plus_buffer(self):
from apps.timeshift.helpers import get_programme_duration
# 40-minute programme + 5-minute buffer.
self.assertEqual(
get_programme_duration(self._channel_with_programme(40), "2026-06-08:17-00"),
45,
)
def test_duration_capped_at_max(self):
from apps.timeshift.helpers import get_programme_duration
self.assertEqual(
get_programme_duration(self._channel_with_programme(1000), "2026-06-08:17-00"),
480,
)
def test_no_epg_data_falls_back_to_default(self):
from unittest.mock import MagicMock
from apps.timeshift.helpers import get_programme_duration
channel = MagicMock(epg_data=None)
self.assertEqual(get_programme_duration(channel, "2026-06-08:17-00"), 120)
def test_no_matching_programme_falls_back_to_default(self):
from unittest.mock import MagicMock
from apps.timeshift.helpers import get_programme_duration
channel = MagicMock()
channel.epg_data.programs.filter.return_value.first.return_value = None
self.assertEqual(get_programme_duration(channel, "2026-06-08:17-00"), 120)
def test_garbage_timestamp_falls_back_to_default(self):
from unittest.mock import MagicMock
from apps.timeshift.helpers import get_programme_duration
self.assertEqual(get_programme_duration(MagicMock(), "garbage"), 120)

2050
apps/timeshift/tests/test_views.py Normal file
View file

File diff suppressed because it is too large Load diff

1355
apps/timeshift/views.py Normal file
View file

File diff suppressed because it is too large Load diff

21
apps/vod/models.py
View file

@ -243,12 +243,12 @@ class M3UMovieRelation(models.Model):
def get_stream_url(self):
"""Get the full stream URL for this movie from this provider"""
# Build URL dynamically for XtreamCodes accounts
if self.m3u_account.account_type == 'XC':
from core.xtream_codes import Client as XCClient
# Use XC client's URL normalization to handle malformed URLs
# (e.g., URLs with /player_api.php or query parameters)
normalized_url = XCClient(self.m3u_account.server_url, '', '')._normalize_url(self.m3u_account.server_url)
from core.xtream_codes import normalize_server_url
normalized_url = normalize_server_url(self.m3u_account.server_url)
if not normalized_url:
return None
username = self.m3u_account.username
password = self.m3u_account.password
return f"{normalized_url}/movie/{username}/{password}/{self.stream_id}.{self.container_extension or 'mp4'}"
@ -292,13 +292,12 @@ class M3UEpisodeRelation(models.Model):
def get_stream_url(self):
"""Get the full stream URL for this episode from this provider"""
from core.xtream_codes import Client as XtreamCodesClient
if self.m3u_account.account_type == 'XC':
# For XtreamCodes accounts, build the URL dynamically
# Use XC client's URL normalization to handle malformed URLs
# (e.g., URLs with /player_api.php or query parameters)
normalized_url = XtreamCodesClient(self.m3u_account.server_url, '', '')._normalize_url(self.m3u_account.server_url)
from core.xtream_codes import normalize_server_url
normalized_url = normalize_server_url(self.m3u_account.server_url)
if not normalized_url:
return None
username = self.m3u_account.username
password = self.m3u_account.password
return f"{normalized_url}/series/{username}/{password}/{self.stream_id}.{self.container_extension or 'mp4'}"

57
apps/vod/tasks.py
View file

@ -16,6 +16,27 @@ import re
logger = logging.getLogger(__name__)
def lookup_by_name_year(model, name_year_pairs):
"""Return {(name, year): row} for rows without TMDB/IMDB IDs.
Scoped to the names in this batch instead of scanning the full table.
"""
if not name_year_pairs:
return {}
wanted = set(name_year_pairs)
names = {name for name, _ in wanted}
found = {}
for row in model.objects.filter(
tmdb_id__isnull=True,
imdb_id__isnull=True,
name__in=names,
):
key = (row.name, row.year)
if key in wanted:
found[key] = row
return found
@shared_task
def refresh_vod_content(account_id):
"""Refresh VOD content for an M3U account with batch processing for improved performance"""
@ -176,6 +197,7 @@ def refresh_movies(client, account, categories_by_provider, relations, scan_star
logger.info(f"Processing movie chunk {chunk_num}/{total_chunks} ({len(chunk)} movies)")
process_movie_batch(account, chunk, categories_by_provider, relations, scan_start_time)
del all_movies_data
logger.info(f"Completed processing all {total_movies} movies in {total_chunks} chunks")
@ -230,6 +252,7 @@ def refresh_series(client, account, categories_by_provider, relations, scan_star
logger.info(f"Processing series chunk {chunk_num}/{total_chunks} ({len(chunk)} series)")
process_series_batch(account, chunk, categories_by_provider, relations, scan_start_time)
del all_series_data
logger.info(f"Completed processing all {total_series} series in {total_chunks} chunks")
@ -539,10 +562,12 @@ def process_movie_batch(account, batch, categories, relations, scan_start_time=N
# Query by name+year for movies without external IDs
name_year_keys = [k for k in movie_keys.keys() if k.startswith('name_')]
if name_year_keys:
for movie in Movie.objects.filter(tmdb_id__isnull=True, imdb_id__isnull=True):
key = f"name_{movie.name}_{movie.year or 'None'}"
if key in name_year_keys:
existing_movies[key] = movie
name_year_pairs = [
(movie_keys[k]['props']['name'], movie_keys[k]['props'].get('year'))
for k in name_year_keys
]
for key_tuple, movie in lookup_by_name_year(Movie, name_year_pairs).items():
existing_movies[f"name_{key_tuple[0]}_{key_tuple[1] or 'None'}"] = movie
# Get existing relations
stream_ids = [data['stream_id'] for data in movie_keys.values()]
@ -659,12 +684,7 @@ def process_movie_batch(account, batch, categories, relations, scan_start_time=N
existing_by_tmdb = {m.tmdb_id: m for m in Movie.objects.filter(tmdb_id__in=tmdb_ids)} if tmdb_ids else {}
existing_by_imdb = {m.imdb_id: m for m in Movie.objects.filter(imdb_id__in=imdb_ids)} if imdb_ids else {}
existing_by_name_year = {}
if name_year_pairs:
for movie in Movie.objects.filter(tmdb_id__isnull=True, imdb_id__isnull=True):
key = (movie.name, movie.year)
if key in name_year_pairs:
existing_by_name_year[key] = movie
existing_by_name_year = lookup_by_name_year(Movie, name_year_pairs)
# Check each movie against the bulk query results
movies_actually_created = []
@ -911,10 +931,12 @@ def process_series_batch(account, batch, categories, relations, scan_start_time=
# Query by name+year for series without external IDs
name_year_keys = [k for k in series_keys.keys() if k.startswith('name_')]
if name_year_keys:
for series in Series.objects.filter(tmdb_id__isnull=True, imdb_id__isnull=True):
key = f"name_{series.name}_{series.year or 'None'}"
if key in name_year_keys:
existing_series[key] = series
name_year_pairs = [
(series_keys[k]['props']['name'], series_keys[k]['props'].get('year'))
for k in name_year_keys
]
for key_tuple, series in lookup_by_name_year(Series, name_year_pairs).items():
existing_series[f"name_{key_tuple[0]}_{key_tuple[1] or 'None'}"] = series
# Get existing relations
series_ids = [data['series_id'] for data in series_keys.values()]
@ -1023,12 +1045,7 @@ def process_series_batch(account, batch, categories, relations, scan_start_time=
existing_by_tmdb = {s.tmdb_id: s for s in Series.objects.filter(tmdb_id__in=tmdb_ids)} if tmdb_ids else {}
existing_by_imdb = {s.imdb_id: s for s in Series.objects.filter(imdb_id__in=imdb_ids)} if imdb_ids else {}
existing_by_name_year = {}
if name_year_pairs:
for series in Series.objects.filter(tmdb_id__isnull=True, imdb_id__isnull=True):
key = (series.name, series.year)
if key in name_year_pairs:
existing_by_name_year[key] = series
existing_by_name_year = lookup_by_name_year(Series, name_year_pairs)
# Check each series against the bulk query results
series_actually_created = []

3
core/api_views.py
View file

@ -225,7 +225,8 @@ class ProxySettingsViewSet(viewsets.ViewSet):
"buffering_speed": 1.0,
"redis_chunk_ttl": 60,
"channel_shutdown_delay": 0,
"channel_init_grace_period": 5,
"channel_init_grace_period": 60,
"channel_client_wait_period": 5,
"new_client_behind_seconds": 5,
}
settings_obj, created = CoreSettings.objects.get_or_create(

51
core/migrations/0026_add_channel_client_wait_period.py Normal file
View file

@ -0,0 +1,51 @@
from django.db import migrations
PROXY_SETTINGS_KEY = "proxy_settings"
def add_channel_client_wait_period(apps, schema_editor):
CoreSettings = apps.get_model("core", "CoreSettings")
try:
obj = CoreSettings.objects.get(key=PROXY_SETTINGS_KEY)
except CoreSettings.DoesNotExist:
return
value = obj.value if isinstance(obj.value, dict) else {}
# Add the new client-connect grace period default.
value.setdefault("channel_client_wait_period", 5)
# channel_init_grace_period was repurposed in 0.27.1 as the channel startup
# timeout (replacing a hardcoded 10s). Values below the new 60s default are
# too short when a channel has many failover streams to cycle through.
current_init = value.get("channel_init_grace_period", 5)
if current_init < 60:
value["channel_init_grace_period"] = 60
obj.value = value
obj.save()
def remove_channel_client_wait_period(apps, schema_editor):
CoreSettings = apps.get_model("core", "CoreSettings")
try:
obj = CoreSettings.objects.get(key=PROXY_SETTINGS_KEY)
except CoreSettings.DoesNotExist:
return
value = obj.value if isinstance(obj.value, dict) else {}
value.pop("channel_client_wait_period", None)
obj.value = value
obj.save()
class Migration(migrations.Migration):
dependencies = [
("core", "0025_move_preferred_region_and_auto_import_to_system_settings"),
]
operations = [
migrations.RunPython(add_channel_client_wait_period, remove_channel_client_wait_period),
]

13
core/models.py
View file

@ -280,8 +280,18 @@ class CoreSettings(models.Model):
"epg_match_ignore_prefixes": [],
"epg_match_ignore_suffixes": [],
"epg_match_ignore_custom": [],
# XC catch-up: forced XMLTV lookback (0 = auto-detect, capped at 30).
"xmltv_prev_days_override": 0,
})
@classmethod
def get_xmltv_prev_days_override(cls):
"""Global XC XMLTV prev_days default (0 = auto-detect from provider archives)."""
try:
return int(cls.get_epg_settings().get("xmltv_prev_days_override", 0) or 0)
except (TypeError, ValueError):
return 0
@classmethod
def _safe_string_list(cls, value):
"""Return a list of strings, filtering out non-list or non-string values."""
@ -394,7 +404,8 @@ class CoreSettings(models.Model):
"buffering_speed": 1.0,
"redis_chunk_ttl": 60,
"channel_shutdown_delay": 0,
"channel_init_grace_period": 5,
"channel_init_grace_period": 60,
"channel_client_wait_period": 5,
"new_client_behind_seconds": 5,
})

16
core/serializers.py
View file

@ -96,7 +96,8 @@ class ProxySettingsSerializer(serializers.Serializer):
buffering_speed = serializers.FloatField(min_value=0.1, max_value=10.0)
redis_chunk_ttl = serializers.IntegerField(min_value=10, max_value=3600)
channel_shutdown_delay = serializers.IntegerField(min_value=0, max_value=300)
channel_init_grace_period = serializers.IntegerField(min_value=0, max_value=60)
channel_init_grace_period = serializers.IntegerField(min_value=0, max_value=300)
channel_client_wait_period = serializers.IntegerField(min_value=0, max_value=300, required=False, default=5)
new_client_behind_seconds = serializers.IntegerField(min_value=0, max_value=120, required=False, default=5)
def validate_buffering_timeout(self, value):
@ -120,8 +121,17 @@ class ProxySettingsSerializer(serializers.Serializer):
return value
def validate_channel_init_grace_period(self, value):
if value < 0 or value > 60:
raise serializers.ValidationError("Channel init grace period must be between 0 and 60 seconds")
if value < 0 or value > 300:
raise serializers.ValidationError(
"Channel initialization timeout must be between 0 and 300 seconds"
)
return value
def validate_channel_client_wait_period(self, value):
if value < 0 or value > 300:
raise serializers.ValidationError(
"Client connect grace period must be between 0 and 300 seconds"
)
return value
def validate_new_client_behind_seconds(self, value):

12
core/tasks.py
View file

@ -398,12 +398,16 @@ def scan_and_process_files():
def _rebuild_programme_indices():
"""Queue index builds for active EPG sources that are missing their DB index."""
try:
from django.db.models import Q
from apps.epg.tasks import build_programme_index_task
sources = EPGSource.objects.filter(
is_active=True,
programme_index__isnull=True,
).exclude(source_type__in=('dummy', 'schedules_direct'))
).exclude(
source_type__in=('dummy', 'schedules_direct')
).filter(
Q(index_record__isnull=True) | Q(index_record__data__isnull=True)
)
count = 0
for source in sources:
@ -796,11 +800,11 @@ def check_for_version_update():
from packaging import version as pkg_version
from version import __version__, __timestamp__
from core.models import SystemNotification
from core.utils import send_websocket_notification
from core.utils import dispatcharr_http_headers, send_websocket_notification
try:
is_dev_build = __timestamp__ is not None
DISPATCHARR_HEADERS = {'User-Agent': f'Dispatcharr/{__version__}'}
DISPATCHARR_HEADERS = dispatcharr_http_headers(content_type=None)
if is_dev_build:
# Check Docker Hub for newer dev builds

48
core/tests.py
View file

@ -1,21 +1,46 @@
from unittest.mock import patch, MagicMock
from django.test import TestCase
from django.test import TestCase, SimpleTestCase
from apps.epg.models import EPGSource
from apps.epg.models import EPGSource, EPGSourceIndex
from core.models import CoreSettings, DVR_SETTINGS_KEY, EPG_SETTINGS_KEY
class DispatcharrUserAgentTests(TestCase):
@patch('version.__version__', '1.2.3')
def test_dispatcharr_user_agent(self):
from core.utils import dispatcharr_user_agent
self.assertEqual(dispatcharr_user_agent(), 'Dispatcharr/1.2.3')
def test_dispatcharr_dvr_user_agent(self):
from core.utils import dispatcharr_dvr_user_agent
self.assertEqual(dispatcharr_dvr_user_agent(42), 'Dispatcharr-DVR/recording-42')
@patch('version.__version__', '1.2.3')
def test_dispatcharr_http_headers_with_token(self):
from core.utils import dispatcharr_http_headers
headers = dispatcharr_http_headers(token='tok123')
self.assertEqual(headers, {
'User-Agent': 'Dispatcharr/1.2.3',
'Content-Type': 'application/json',
'token': 'tok123',
})
@patch('version.__version__', '1.2.3')
def test_dispatcharr_http_headers_without_content_type(self):
from core.utils import dispatcharr_http_headers
self.assertEqual(
dispatcharr_http_headers(content_type=None),
{'User-Agent': 'Dispatcharr/1.2.3'},
)
class ProgrammeIndexRebuildTests(TestCase):
def test_startup_rebuild_does_not_lock_out_queued_build_task(self):
EPGSource.objects.update(
programme_index={"channels": {}, "interleaved_channels": []}
)
source = EPGSource.objects.create(
name="Missing Index",
source_type="xmltv",
is_active=True,
programme_index=None,
)
class FakeRedis:
@ -272,3 +297,14 @@ class DropDBCommandTlsTest(TestCase):
host='localhost', port=5432,
autocommit=True,
)
class MallocTrimTests(SimpleTestCase):
def test_trim_is_noop_when_libc_has_no_malloc_trim(self):
from core.utils import trim_c_allocator_heap
fake_libc = MagicMock(spec=[])
with patch('ctypes.util.find_library', return_value='libc.so.6'), patch(
'ctypes.CDLL', return_value=fake_libc
):
self.assertFalse(trim_c_allocator_heap())

183
core/utils.py
View file

@ -1,3 +1,4 @@
import json
import redis
import logging
import time
@ -7,7 +8,6 @@ from pathlib import Path
import re
from django.conf import settings
from redis.exceptions import ConnectionError, TimeoutError
from django.core.cache import cache
from asgiref.sync import async_to_sync
from channels.layers import get_channel_layer
from django.core.validators import URLValidator
@ -21,6 +21,33 @@ logger = logging.getLogger(__name__)
# Import the command detector
from .command_utils import is_management_command
def dispatcharr_user_agent():
"""Return the standard Dispatcharr User-Agent string (Dispatcharr/{version})."""
from version import __version__
return f'Dispatcharr/{__version__}'
def dispatcharr_dvr_user_agent(recording_id):
"""Return the User-Agent string used by DVR FFmpeg clients for a recording."""
return f'Dispatcharr-DVR/recording-{recording_id}'
def dispatcharr_http_headers(*, token=None, content_type='application/json'):
"""
Build HTTP headers for outbound Dispatcharr requests.
content_type=None omits Content-Type (e.g. simple GET proxies).
token is included when authenticating with Schedules Direct.
"""
headers = {'User-Agent': dispatcharr_user_agent()}
if content_type:
headers['Content-Type'] = content_type
if token:
headers['token'] = token
return headers
def natural_sort_key(text):
"""
Convert a string into a list of string and number chunks for natural sorting.
@ -44,6 +71,46 @@ def natural_sort_key(text):
return [convert(c) for c in re.split('([0-9]+)', text)]
def custom_properties_as_dict(value):
"""
Normalize a JSONField-backed custom_properties value into a dict.
Historical rows (TextField era and early JSONField migration) may store a
JSON-encoded string instead of an object. API clients can also submit a
string value because JSONField accepts any JSON type. Call this before
reading or merging custom_properties.
"""
if isinstance(value, dict):
return value
if isinstance(value, str):
try:
parsed = json.loads(value)
except (ValueError, TypeError):
logger.warning(
"custom_properties stored as non-JSON string; ignoring: %r",
value[:100],
)
return {}
return parsed if isinstance(parsed, dict) else {}
if value is None:
return {}
return {}
def ensure_custom_properties_dict(value):
"""
Return a dict for read/merge/bulk-write paths. Dict values pass through
without re-parsing. Use model ``save()`` (not this) as the canonical
normalizer for ORM writes that go through ``save()``.
"""
if isinstance(value, dict):
return value
if value is None:
return {}
return custom_properties_as_dict(value)
class RedisClient:
_client = None
_buffer = None
@ -244,6 +311,15 @@ def release_task_lock(task_name, id):
redis_client.delete(lock_id)
def is_task_lock_held(task_name, id):
"""Return True when another worker holds the task lock (read-only check)."""
redis_client = RedisClient.get_client()
if redis_client is None:
return False
lock_id = f"task_lock_{task_name}_{id}"
return bool(redis_client.exists(lock_id))
class TaskLockRenewer:
"""Periodically renews a Redis task lock to prevent expiry during long-running tasks.
@ -470,13 +546,34 @@ def monitor_memory_usage(func):
return result
return wrapper
def cleanup_memory(log_usage=False, force_collection=True):
def trim_c_allocator_heap():
"""Return unused C heap pages to the OS where supported (glibc malloc_trim)."""
try:
import ctypes
import ctypes.util
libc_name = ctypes.util.find_library("c")
if not libc_name:
return False
libc = ctypes.CDLL(libc_name)
if not hasattr(libc, "malloc_trim"):
return False
libc.malloc_trim(0)
return True
except Exception:
logger.debug("malloc_trim unavailable or failed", exc_info=True)
return False
def cleanup_memory(log_usage=False, force_collection=True, trim_heap=False):
"""
Comprehensive memory cleanup function to reduce memory footprint
Args:
log_usage: Whether to log memory usage before and after cleanup
force_collection: Whether to force garbage collection
trim_heap: Return freed C heap pages to the OS. Only use after DB
connections are closed (e.g. Celery task_postrun).
"""
logger.trace("Starting memory cleanup django memory cleanup")
# Skip logging if log level is not set to debug or more verbose (like trace)
@ -511,8 +608,33 @@ def cleanup_memory(log_usage=False, force_collection=True):
logger.debug(f"Memory after cleanup: {after_mem:.2f} MB (change: {after_mem-before_mem:.2f} MB)")
except (ImportError, Exception):
pass
if trim_heap:
trim_c_allocator_heap()
logger.trace("Memory cleanup complete for django")
def spawn_memory_trim(close_connections=False):
"""Reclaim a request's heap pages: GC, then return freed C pages to the OS.
On gevent uWSGI workers the trim runs in a spawned greenlet so it never
blocks the caller; Celery prefork workers (no gevent hub) run it inline.
Set close_connections=True when called from a streaming generator's teardown
so the pooled DB connection is released first.
"""
def _run():
cleanup_memory(force_collection=True, trim_heap=True)
if close_connections:
from django.db import close_old_connections
close_old_connections()
if _is_gevent_monkey_patched():
import gevent
gevent.spawn(_run)
else:
_run()
def safe_upload_path(filename: str, base_dir) -> str:
"""Return a safe absolute path for an uploaded file within base_dir.
@ -587,6 +709,8 @@ def validate_flexible_url(value):
raise ValidationError("Enter a valid URL.")
def dispatch_event_system(event_type, channel_id=None, channel_name=None, **details):
from django.db import close_old_connections
try:
from apps.connect.utils import trigger_event
from apps.channels.models import Channel, Stream
@ -660,9 +784,48 @@ def dispatch_event_system(event_type, channel_id=None, channel_name=None, **deta
trigger_event(event_type, payload)
except Exception as e:
except Exception:
# Don't fail main path if connect dispatch fails
pass
finally:
close_old_connections()
def _dispatch_system_event_integrations(
event_type, channel_id=None, channel_name=None, **details
):
"""
Run Connect subscriptions and plugin event hooks without blocking the caller.
On gevent uWSGI workers, dispatch runs in a spawned greenlet so slow webhooks,
scripts, or plugin handlers cannot stall live-proxy teardown or streaming paths.
Celery prefork workers (gevent patched but no hub) run synchronously instead.
"""
def _run():
try:
dispatch_event_system(
event_type,
channel_id=channel_id,
channel_name=channel_name,
**details,
)
except Exception as e:
logger.error(
"Failed to dispatch Connect/plugin handlers for event %s: %s",
event_type,
e,
)
if _should_use_sync_websocket_send():
_run()
elif _is_gevent_monkey_patched():
import gevent
gevent.spawn(_run)
else:
_run()
def log_system_event(event_type, channel_id=None, channel_name=None, **details):
"""
@ -679,6 +842,7 @@ def log_system_event(event_type, channel_id=None, channel_name=None, **details):
stream_url='http://...', user='admin')
"""
from core.models import SystemEvent, CoreSettings
from django.db import close_old_connections
try:
# Create the event
@ -689,8 +853,13 @@ def log_system_event(event_type, channel_id=None, channel_name=None, **details):
details=details
)
# Trigger connect integrations for specific events
dispatch_event_system(event_type, channel_id=channel_id, channel_name=channel_name, **details)
# Connect integrations and plugin event hooks (non-blocking on gevent uWSGI)
_dispatch_system_event_integrations(
event_type,
channel_id=channel_id,
channel_name=channel_name,
**details,
)
# Get max events from settings (default 100)
try:
@ -714,6 +883,10 @@ def log_system_event(event_type, channel_id=None, channel_name=None, **details):
except Exception as e:
# Don't let event logging break the main application
logger.error(f"Failed to log system event {event_type}: {e}")
finally:
# geventpool keeps checked-out connections until close(); release promptly
# when logging from proxy greenlets/threads outside a normal request cycle.
close_old_connections()
def _send_async(channel_layer, group, message):

38
core/xtream_codes.py
View file

@ -5,6 +5,26 @@ import json
logger = logging.getLogger(__name__)
def normalize_server_url(url):
"""Normalize server URL: strip XC API endpoints and query params, preserve base path."""
if not url:
return url
from urllib.parse import urlparse, urlunparse
parsed = urlparse(url.strip())
path = parsed.path.rstrip('/')
# XC API endpoints are always .php files; legitimate base paths never are.
# Stripping the trailing segment when it ends in .php handles any pasted API URL.
last_segment = path.rsplit('/', 1)[-1]
if last_segment.endswith('.php'):
path = path[:-(len(last_segment) + 1)] if '/' in path else ''
return urlunparse((parsed.scheme, parsed.netloc, path, '', '', ''))
class Client:
"""Xtream Codes API Client with robust error handling"""
@ -43,22 +63,10 @@ class Client:
self.server_info = None
def _normalize_url(self, url):
"""Normalize server URL: strip XC API endpoints and query params, preserve base path."""
if not url:
normalized = normalize_server_url(url)
if not normalized:
raise ValueError("Server URL cannot be empty")
from urllib.parse import urlparse, urlunparse
parsed = urlparse(url.strip())
path = parsed.path.rstrip('/')
# XC API endpoints are always .php files; legitimate base paths never are.
# Stripping the trailing segment when it ends in .php handles any pasted API URL.
last_segment = path.rsplit('/', 1)[-1]
if last_segment.endswith('.php'):
path = path[:-(len(last_segment) + 1)] if '/' in path else ''
return urlunparse((parsed.scheme, parsed.netloc, path, '', '', ''))
return normalized
def _make_request(self, endpoint, params=None):
"""Make request with detailed error handling"""

47
dispatcharr/celery.py
View file

@ -2,7 +2,7 @@
import os
from celery import Celery
import logging
from celery.signals import task_postrun, worker_ready
from celery.signals import task_postrun, task_prerun, worker_process_init, worker_ready
logger = logging.getLogger(__name__)
@ -42,13 +42,26 @@ app.autodiscover_tasks()
# Plugins live outside INSTALLED_APPS, so autodiscover_tasks() never imports
# them. Without an eager import, workers reject plugin @shared_tasks with
# "Received unregistered task" until a lazy event import warms the module.
@worker_ready.connect(weak=False)
def discover_plugins_on_worker_ready(**_kwargs):
# Discovery runs in worker_process_init (each prefork child / thread worker)
# rather than worker_ready (the long-lived arbiter) so the parent process
# never opens DB connections that autoscale children would inherit via fork().
@worker_process_init.connect(weak=False)
def init_worker_process(**_kwargs):
from django.db import connections
# Standard Celery + Django guidance for prefork pools: discard any
# connection state inherited from the parent across fork().
try:
connections.close_all()
except Exception:
logger.warning("Failed to close inherited DB connections after fork", exc_info=True)
try:
from apps.plugins.loader import PluginManager
PluginManager.get().discover_plugins(sync_db=False)
except Exception:
logger.exception("plugin discovery on worker_ready failed")
logger.exception("plugin discovery on worker_process_init failed")
# Use environment variable for log level with fallback to INFO
CELERY_LOG_LEVEL = os.environ.get('DISPATCHARR_LOG_LEVEL', 'INFO').upper()
@ -68,18 +81,30 @@ app.conf.task_routes = {
'apps.channels.tasks.run_recording': {'queue': 'dvr'},
}
@task_prerun.connect
def reset_db_connection_before_task(**kwargs):
"""Discard stale DB connections before each task (Celery workers are long-lived)."""
from django.db import close_old_connections
try:
close_old_connections()
except Exception:
pass
# Add memory cleanup after task completion
@task_postrun.connect # Use the imported signal
def cleanup_task_memory(**kwargs):
"""Clean up memory and database connections after each task completes"""
from django.db import connection
from django.db import close_old_connections
# Get task name from kwargs
task_name = kwargs.get('task').name if kwargs.get('task') else ''
# Close database connection for this Celery worker process
# Return all DB connections to the pool in a clean state
try:
connection.close()
close_old_connections()
except Exception:
pass
@ -87,6 +112,7 @@ def cleanup_task_memory(**kwargs):
memory_intensive_tasks = [
'apps.m3u.tasks.refresh_single_m3u_account',
'apps.m3u.tasks.refresh_m3u_accounts',
'apps.m3u.tasks.refresh_m3u_groups',
'apps.m3u.tasks.process_m3u_batch',
'apps.m3u.tasks.process_xc_category',
'apps.m3u.tasks.sync_auto_channels',
@ -94,10 +120,13 @@ def cleanup_task_memory(**kwargs):
'apps.epg.tasks.refresh_all_epg_data',
'apps.epg.tasks.parse_programs_for_source',
'apps.epg.tasks.parse_programs_for_tvg_id',
'apps.epg.tasks.build_programme_index_task',
'apps.channels.tasks.match_epg_channels',
'apps.channels.tasks.match_selected_channels_epg',
'apps.channels.tasks.match_single_channel_epg',
'core.tasks.rehash_streams'
'core.tasks.rehash_streams',
'apps.vod.tasks.refresh_vod_content',
'apps.vod.tasks.batch_refresh_series_episodes',
]
# Check if this is a memory-intensive task
@ -106,7 +135,7 @@ def cleanup_task_memory(**kwargs):
from core.utils import cleanup_memory
# Use the comprehensive cleanup function
cleanup_memory(log_usage=True, force_collection=True)
cleanup_memory(log_usage=True, force_collection=True, trim_heap=True)
# Log memory usage if psutil is installed
try:

0
dispatcharr/db/__init__.py Normal file
View file

0
dispatcharr/db/backends/__init__.py Normal file
View file

Some files were not shown because too many files have changed in this diff Show more