diff --git a/CHANGELOG.md b/CHANGELOG.md index 71b8b8c1..3a1f493f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,172 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.23.0] - 2026-04-17 + +### Security + +- Set `DEFAULT_PERMISSION_CLASSES` to `IsAdmin` in the DRF configuration. All viewsets and function-based views that require non-admin or unauthenticated access were explicitly annotated: proxy streaming endpoints (`stream_ts`, `stream_xc`, `stream_vod`, `head_vod`, `stream_xc_movie`, `stream_xc_episode`) use `@permission_classes([AllowAny])` (access is controlled by the per-stream-type network allow-list inside the view body); the `UserAgentViewSet`, `StreamProfileViewSet`, `CoreSettingsViewSet`, and `ProxySettingsViewSet` gained `get_permissions()` methods mapping read actions to `IsStandardUser` and write actions to `IsAdmin`; and `AuthViewSet.logout` was updated to return `[Authenticated()]`. +- Fixed missing `network_access_allowed` checks in the VOD proxy. `stream_vod`, `head_vod`, `stream_xc_movie`, and `stream_xc_episode` were not checking the `STREAMS` network policy, unlike the equivalent TS proxy endpoints. +- Explicitly marked the HDHomeRun discovery endpoints (`DiscoverAPIView`, `LineupAPIView`, `LineupStatusAPIView`, `HDHRDeviceXMLAPIView`) and the version endpoint with `permission_classes = [AllowAny]` to document their intentionally public access now that the global default is `IsAdmin`. +- Fixed path traversal vulnerability in file uploads. The M3U account upload (`apps/m3u/api_views.py`), logo upload (`apps/channels/api_views.py`), and backup upload (`apps/backups/api_views.py`) all used the uploaded filename directly without sanitization. `os.path.join()` discards all preceding components when it encounters an absolute path segment, and `pathlib`'s `/` operator behaves identically; a relative `../` sequence also escapes via OS path resolution at `open()` time. All three upload paths now strip directory components via `Path(name).name` and validate the resolved path remains within the intended upload directory. Exploiting any of these required admin credentials. +- Prevented users from setting `xc_password` (and other admin-managed keys) on their own account via the `PATCH /api/accounts/users/me/` endpoint. +- Hardened the HLS proxy `change_stream` endpoint by converting it from a plain Django view to a DRF `@api_view` with `@permission_classes([IsAdmin])`, ensuring the endpoint actually enforces admin-only access. The previous decorator arrangement (`@csrf_exempt` + `@permission_classes`) had no effect on a plain Django view. +- Added rate limiting to the login endpoint (`POST /api/accounts/token/`) using DRF's built-in throttling. A `LoginRateThrottle` (3 requests/minute per IP, sliding window) is applied to the `TokenObtainPairView`. Repeated failed attempts from the same IP receive `429 Too Many Requests`. +- Extended rate limiting to the session-auth login alias (`POST /api/accounts/auth/login/`). It now delegates entirely to `TokenObtainPairView`, inheriting its throttle, network access check, and audit logging, and returns JWT tokens instead of a session cookie (the session-based response was unusable since `SessionAuthentication` is not in `DEFAULT_AUTHENTICATION_CLASSES`). Both endpoints share the same `"login"` throttle scope, so attempts across either path count against the same per-IP limit. +- Removed `CORS_ALLOW_CREDENTIALS = True` from CORS configuration. Dispatcharr authenticates via JWT `Authorization` headers and API keys — not cookies — so credentials are never sent cross-origin by browsers. The setting was also redundant: browsers reject `Access-Control-Allow-Credentials: true` when `Access-Control-Allow-Origin` is a wildcard (`*`), so it had no effect in practice. +- Updated frontend npm dependencies to resolve 6 audit vulnerabilities (6 high): + - Updated `@xmldom/xmldom` 0.8.11 → 0.8.12, resolving **high** XML injection via unsafe CDATA serialization allowing attacker-controlled markup insertion ([GHSA-wh4c-j3r5-mjhp](https://github.com/advisories/GHSA-wh4c-j3r5-mjhp)) + - Updated `lodash` 4.17.23 → 4.18.1, resolving **high** Code Injection via `_.template` imports key names ([GHSA-r5fr-rjxr-66jc](https://github.com/advisories/GHSA-r5fr-rjxr-66jc)) and **high** Prototype Pollution via array path bypass in `_.unset` and `_.omit` ([GHSA-f23m-r3pf-42rh](https://github.com/advisories/GHSA-f23m-r3pf-42rh)) + - Updated `vite` 7.3.1 → 7.3.2, resolving **high** Path Traversal in optimized deps `.map` handling ([GHSA-4w7w-66w2-5vf9](https://github.com/advisories/GHSA-4w7w-66w2-5vf9)), **high** `server.fs.deny` bypass with queries ([GHSA-v2wj-q39q-566r](https://github.com/advisories/GHSA-v2wj-q39q-566r)), and **high** Arbitrary File Read via dev server WebSocket ([GHSA-p9ff-h696-f583](https://github.com/advisories/GHSA-p9ff-h696-f583)) +- Updated `Django` 6.0.3 → 6.0.4, resolving the following CVEs: + - **CVE-2026-33033**: Potential DoS via `MultiPartParser` through crafted multipart uploads. + - **CVE-2026-33034**: SGI requests with a missing or understated `Content-Length` header could bypass the `DATA_UPLOAD_MAX_MEMORY_SIZE` limit. + - **CVE-2026-4292**: Privilege abuse in `ModelAdmin.list_editable`. + - **CVE-2026-3902**: ASGI header spoofing via underscore/hyphen conflation. + - **CVE-2026-4277**: Privilege abuse in `GenericInlineModelAdmin`. + +### Added + +- **EPG historical data window**: the EPG XML output and XC EPG API now support a `prev_days` URL parameter (e.g. `&prev_days=3`) to include past programs in the EPG response. This allows third-party players that request historical program schedules to receive the data they need. The EPG URL builder in the Channels page exposes "Days forward" and "Days back" controls. Per-user defaults for both values (`epg_days` / `epg_prev_days`) can be configured in the User settings modal and are applied automatically when no URL parameter is present. (Closes #1154) +- **Plugin Hub**: administrators can now browse, install, and update plugins directly from remote repositories via a new Plugin Hub page in Settings. (Closes #393) — Thanks [@sethwv](https://github.com/sethwv) + - Install plugins directly from the hub: the release zip is downloaded, SHA256 integrity is verified, and the plugin is installed atomically. + - Update managed plugins when a newer version is available from their source repo. Version compatibility constraints (`min_dispatcharr_version` / `max_dispatcharr_version`) are enforced at install time. + - Browse available plugins from all enabled repos with name, description, version, author, and icon. + - Plugins installed from a repo are tracked as "managed": source repo, slug, installed version, prerelease flag, and deprecated status are all persisted and surfaced in the UI. + - Add plugin repositories by manifest URL. The official Dispatcharr Plugins repository is pre-configured; third-party repos are supported by supplying an optional GPG public key. + - Manifest signatures are verified via GPG; the official repo uses a bundled public key. Signature status is displayed per-repo. + - Preview a repository URL before adding it - validates the manifest and reports plugin count and signature status without saving anything. + - Configurable automatic manifest refresh interval (in hours; 0 to disable) runs as a Celery background task. + +### Removed + +- Removed dead `VODConnectionManager` class (`apps/proxy/vod_proxy/connection_manager.py`) and its associated helpers, which had been superseded by `MultiWorkerVODConnectionManager`. All active code already used the multi-worker implementation. Removed the unused `VODConnectionManager` import from `vod_proxy/views.py`, the unscheduled `cleanup_vod_connections` task from `apps/proxy/tasks.py`, and the unscheduled `cleanup_vod_persistent_connections` task from `core/tasks.py`. +- Removed dead VOD URL routes: `VODPlaylistView` (playlist generation), `VODPositionView` (position tracking), and the class-based `VODStatsView` (replaced by the existing function-based `vod_stats` view). +- Removed dead `updateVODPosition()` API method from `frontend/src/api.js`, which called the now-removed position tracking endpoint. + +### Fixed + +- Fixed TV Guide "Record One" always scheduling the recording on the first channel that matched the program's `tvg_id`, rather than the channel the user actually selected. When multiple channels share the same EPG source, the intended channel was silently ignored. The selected channel object is now passed explicitly through the click handler chain to `recordOne`, bypassing the `findChannelByTvgId` fallback lookup entirely. (Fixes #1140) — Thanks [@fezster](https://github.com/fezster) +- Graceful container shutdown: `docker stop` no longer results in exit 137 (SIGKILL). The entrypoint now explicitly stops all child processes — including uWSGI workers, Celery, Daphne, and Redis, which are spawned as uWSGI `attach-daemon` children and were previously invisible to the signal handler. A polling loop replaces the old fixed `sleep`, exiting as soon as all processes have stopped (up to an 8-second ceiling before force-stopping). PostgreSQL is stopped using `pg_ctl stop -m immediate` as a fallback rather than SIGKILL to avoid data corruption. Process names are now recorded at startup and displayed correctly in crash diagnostics. The unexpected-exit diagnostic block is now suppressed on normal `docker stop` shutdowns. — Thanks [@Shokkstokk](https://github.com/Shokkstokk) for the initial fix! +- Fixed two race conditions in the VOD proxy that caused the `profile_connections` counter to go permanently negative, allowing connections beyond the configured profile limit. (1) `_decrement_profile_connections()` used a GET-before-DECR guard: two concurrent decrements could both read the same positive value, both pass the guard, and both fire, driving the counter below zero. Replaced with an unconditional `DECR` followed by a clamp-to-zero if the result is negative. (2) The `stream_generator` decremented `active_streams` and then checked `has_active_streams()` in two separate Redis round-trips without locking. A concurrent generator on another worker could read `active_streams=0` in the window between those two calls and also decrement the profile counter, producing a double-decrement. A new `decrement_active_streams_and_check()` method performs both operations under a single distributed lock, and a `profile_decremented` flag guards all four call sites in the generator so the profile counter is only ever decremented once per stream. (Closes #1125) — Thanks [@firestaerter3](https://github.com/firestaerter3) +- Fixed a provider TCP connection leak in the VOD proxy `stream_generator`. When a stream ended via an unhandled exception path that reached the `finally` block without any of the three exception handlers having run (e.g. an error raised before the first `yield`), the `finally` block decremented counters but never called `redis_connection.cleanup()`. The upstream `requests.Response` and `requests.Session` were left open until garbage collection. The `finally` block now starts a `delayed_cleanup` daemon thread (matching the 1-second delay used by the normal-completion and `GeneratorExit` paths) so that seeking clients have time to reconnect and increment `active_streams` before `cleanup()` checks whether it is safe to close the connection. +- Fixed manual stream selection from the Stats page not enforcing M3U profile connection limits in multi-worker deployments. When a non-owning worker handled the `change_stream` request it correctly packaged `stream_id` and `m3u_profile_id` into the Redis pubsub message, but the owning worker's pubsub handler only consumed `url` and `user_agent` silently dropping both IDs before calling `stream_manager.update_url()`. Because `update_url` only calls `update_stream_profile()` when a `stream_id` is provided, the `profile_connections` counter was never updated after the switch, causing subsequent capacity checks to see incorrect counts and bypass the full-profile guard. The handler now extracts `stream_id` and `m3u_profile_id` from the event and forwards them to `update_url()`. The bug did not affect single-worker / dev-mode deployments because the owning worker handles those requests directly without pubsub. +- Fixed the `next_stream` rotation endpoint applying the same class of bug: `get_stream_info_for_switch()` was called and returned `m3u_profile_id`, but the result was dropped when forwarding to `ChannelService.change_stream_url()`, so `update_stream_profile()` was never called and `profile_connections` counters were not updated after an automatic stream rotation. +- Fixed stream switch metadata (`url`, `user_agent`, `stream_id`, `m3u_profile`) being written to Redis before the switch was confirmed to succeed. If the switch failed, URL unchanged or exception during teardown, Redis described a URL not actually in use. Metadata is now written only after `update_url()` returns `True`; on failure the owner writes `stream_manager.url` back as the ground truth. The non-owner no longer pre-writes metadata at all, all needed info is carried in the pubsub payload and written by the owner after confirmation. +- Fixed the Stats page "Active Stream" dropdown not updating when a stream switch occurs. The card was matching the active stream by comparing the URL stored in Redis against stream URLs from the database, which failed silently when the stored URL was a transformed/rewritten value that didn't substring-match the original. The dropdown now matches by `stream_id` (the authoritative value already present in the stats payload) and re-runs only when `stream_id` changes, so the normal polling interval drives updates with no extra renders. +- Fixed the XC Password field in the User modal being editable by standard users despite the backend (`PATCH /api/accounts/users/me/`) stripping `xc_password` from `custom_properties` for non-admin users, causing the change to silently revert on save. The field and its generate button are now disabled with an explanatory description when the current user is not an administrator. +- Fixed live stream hiccups caused by nginx buffering TS proxy data to disk. The `/proxy/` location block used `proxy_buffering off` and `proxy_read/send_timeout` directives, which are silently ignored when the upstream is `uwsgi_pass` (a different directive family). nginx was therefore defaulting to `uwsgi_buffering on`, spooling stream data through temp files on disk. Replaced with the correct `uwsgi_buffering off`, `uwsgi_read_timeout 300s`, and `uwsgi_send_timeout 300s` directives so stream data flows directly from uWSGI to the client socket without intermediate disk I/O. +- Fixed the logo cache endpoint (`/api/channels/logos/{id}/cache/`) holding a uWSGI greenlet indefinitely when fetching from a slow or dripping remote server. The previous implementation used `StreamingHttpResponse(iter_content())` with only a per-chunk read timeout; a server that drips data just fast enough to reset the per-read timer could hold the greenlet open forever. Replaced with an eager read loop enforcing a hard total-download deadline (10 s) and a size cap (5 MB). Also fixed a race condition in the existing negative-cache logic: the failure entry for a URL was cleared immediately upon receiving HTTP 200, before the body was read. A concurrent greenlet seeing no failure entry during a slow download that ultimately timed out would also attempt the fetch, defeating the cache. The entry is now cleared only after the full body has been successfully received. +- Fixed uploading a local M3U file with no expiration date set sending the string `"null"` as the `exp_date` field in the `FormData` request, causing a 400 validation error from the API. Null/undefined values are now skipped when building the `FormData` body, matching the behaviour already present in the update path. +- Fixed `PATCH /api/channels/channels/edit/bulk/` returning a 500 error when the request body included a `streams` list. The bulk edit handler was iterating `validated_data` directly and calling `setattr(channel, "streams", value)`, which Django prohibits on ManyToMany fields. Also added an `@extend_schema` decorator so the Swagger UI correctly documents the endpoint as accepting a JSON array and shows the `streams` field. (Fixes #883) +- Fixed several incorrect or incomplete OpenAPI (`@extend_schema`) schemas across the API: + - `POST /api/epg/import/` — request body was undocumented; now correctly shows the `id` field. Description updated from "import" to "refresh" to match frontend and backend terminology. + - `DELETE /api/channels/logos/bulk-delete/` — `delete_files` boolean was missing from the documented request body. + - `POST /api/channels/channels/batch-set-epg/` — `epg_data_id` inside each association object was not marked `allow_null`/`required=False`, even though passing `null` is the correct way to remove an EPG link. + - `PUT /api/connect/integrations/{id}/subscriptions/set/` — endpoint had no `@extend_schema` at all; now documents that the request body is a JSON array of subscription objects. + +### Changed + +- **Output bitrate DB persistence**: the `ffmpeg_output_bitrate` stat is no longer written to the database on every FFmpeg stats tick (~2/second). Instead, a local exponential moving average (EMA, α=0.1) accumulates readings continuously. The first 10 samples (~5 seconds) are discarded as warmup to avoid polluting the average with FFmpeg's unstable ramp-up values. After warmup, the smoothed value is flushed to the database at most once every 30 seconds, and a final flush occurs when the stream stops but only if the EMA has been seeded (i.e. the stream ran past warmup). Streams that stop during warmup leave the existing database value untouched, preserving previously accurate measurements when channel-hopping. +- Performance: `generate_m3u`, `generate_epg`, and `xc_get_live_streams` now use `select_related('channel_group', 'logo')` (or `select_related('logo')` for EPG) on every Channel queryset in `apps/output/views.py`. Previously each channel in the loop triggered a separate database query for its `logo` and `channel_group` foreign keys; with the JOIN-based prefetch this is reduced to a single query per request. On deployments with ~2 000 channels, `xc_get_live_streams` response time drops from ~2.5–4 s to ~250–450 ms. (Closes #1127) — Thanks [@xBOBxSAGETx](https://github.com/xBOBxSAGETx) +- Performance: `generate_epg` now uses `select_related('epg_data__epg_source')` on all EPG channel querysets, eliminating N+1 database queries for `EPGSource` traversal per channel (~15 s improvement on ~2000-channel deployments; total EPG generation time dropped from ~87 s to ~72 s in benchmarks). +- Performance: `xc_get_epg` now uses `select_related('epg_data__epg_source')` on all three channel fetch paths. Previously each request triggered 2 additional queries to resolve `channel.epg_data` and `channel.epg_data.epg_source`. +- Performance: `generate_m3u` now uses `prefetch_related` for streams when `?direct=true` is requested, eliminating N+1 stream queries (one per channel) on that code path. +- Performance: `EPGGridAPIView` (`apps/epg/api_views.py`) now uses `select_related('epg_data__epg_source')` on the `channels_with_custom_dummy` queryset, eliminating 2 extra queries per channel (for `epg_data` and `epg_source`) in the dummy EPG generation loop. +- Performance: `generate_epg` now issues a single cross-channel `ProgramData` bulk query. `.values()` returns plain dicts, bypassing per-row Django model instantiation. Results are consumed in independent 5000-row keyset-paginated chunks. Combined with the `select_related` improvements above, EPG generation time on large deployments is significantly reduced. +- Performance: `xc_get_live_streams` no longer calls `ChannelGroup.objects.get_or_create(name="Default Group")` once per null-group channel; replaced with a lazy-initialised closure that executes at most one query regardless of how many ungrouped channels are present. +- AIO containers now connect to the internal PostgreSQL instance via a Unix domain socket instead of TCP loopback. Users who have `POSTGRES_HOST` explicitly set to `localhost` or `127.0.0.1` in their compose file are automatically migrated to the socket path; any other explicit value (external host/IP) is left untouched. — Thanks [@JCBird1012](https://github.com/JCBird1012) +- Improved the EPG response cache key. Previously it was based on the raw query string and username, meaning a user default of `epg_days=7` and an explicit `&days=7` URL parameter produced different cache entries for identical output. The key is now built from all resolved effective parameter values (`days`, `prev_days`, `cachedlogos`, `tvg_id_source`) so semantically equivalent requests always share the same cache entry. +- Improved the HDHR, M3U, and EPG URL builder popovers in the Channels table: each popover now opens with a brief intro sentence describing its purpose. Toggle switches were refactored to use Mantine's native `label` and `description` props (replacing the previous manual `Group`/`Stack`/`Text` layout), giving each switch a properly styled description line beneath its label. Switch alignment was also corrected. Toggles now appear on the left with the label and description stacked to the right, consistent with standard Mantine form layout. +- Redesigned the User settings modal with a tabbed layout: **Account** (username, email, name, password), **Permissions** (user level, stream limit, channel profiles, mature content filter - admin only), **EPG Defaults** (days forward/back), and **API & XC** (XC password, API key management). Fields are now logically grouped rather than split across two ad-hoc columns. +- EPG channel scanning now automatically removes stale `EPGData` entries. tvg-ids that were present in a previous scan but are no longer found in the upstream source, provided they are not mapped to any channel. This prevents unbounded database bloat over time. Entries mapped to at least one channel are always preserved. +- Rewrote the M3U line parser as an `iter_m3u_entries` generator that owns the full per-entry state machine. Intermediate directive lines between `#EXTINF` and the stream URL are now handled correctly rather than corrupting the pending entry or being silently misassigned. A `#EXTINF` with no following URL is discarded with a warning instead of carrying over a `url`-less entry into batch processing. Attribute keys are normalised to lowercase during parsing (provider attribute names remain case-insensitive end-to-end). The `#EXTINF` attribute regex is pre-compiled at module load, and attribute lookups use O(1) `dict.get()` instead of linear scans — approximately 10% faster parsing on large M3U files. +- Added support for the `#EXTGRP` directive in M3U files. When a `group-title` attribute is absent from the `#EXTINF` line, the value from a following `#EXTGRP:` line is used as the group. An explicit `group-title` attribute always takes priority. (Closes #1088) +- Added accumulation of `#EXTVLCOPT` directives per entry. Options are stored as a list under `vlc_opts` inside the stream's `custom_properties`, available for downstream use (e.g. passing VLC-specific options to the player). This is for a planned future enhancement and can also be utlized with the API. +- M3U stream name parsing now uses the comma text (the canonical display title per the base `#EXTINF` spec) as the primary stream name, falling back to `tvc-guide-title`, then `tvg-name`, rather than preferring `tvg-name` first. Providers that use `tvg-name` as an EPG key and put the human-readable title after the comma will now display the correct name. Providers that duplicate the same value in both fields are unaffected. (Fixes #1081) +- FloatingVideo player: the native video controls (timeline, play/pause, volume) are now hidden by default when a live stream starts and only appear when the user hovers over the player. +- Enhanced Swagger UI authorization dialog: registered a custom `OpenApiAuthenticationExtension` for `ApiKeyAuthentication` so drf-spectacular now generates an `ApiKeyAuth (apiKey)` entry alongside `jwtAuth`. Both entries include descriptive text linking to the relevant endpoints (`/api/accounts/token/`, `/api/accounts/api-keys/generate/`, `/api/accounts/api-keys/revoke/`). +- Refactored frontend form components (`AccountInfoModal`, `AssignChannelNumbers`, `Channel`, `ChannelBatch`, `ChannelGroup`, `Connection`, `CronBuilder`, `DummyEPG`, and `EPG`) to extract business logic into dedicated utility modules under `src/utils/forms/`. Each extracted module is covered by unit tests. Mantine compound component references (`Table.Tbody`, `Popover.Target`, `Accordion.Item`, etc.) have been updated to use flat named imports. — Thanks [@nick4810](https://github.com/nick4810) +- Improved the EPG BOM fix from v0.22.1: replaced the `lstrip(b'\xef\xbb\xbf')` / `startswith` approach with `start.find(b'...)`) without any conversion, eliminating the root cause of patterns that matched in the frontend live preview but failed on the backend with a `re.error`. As a further improvement, `regex` also supports variable-length lookbehind assertions (e.g. `(?<=a+)`), which `re` rejects with an error; patterns using these will now work correctly on the backend as well. Replace-pattern JS tokens are still normalised before calling `regex.sub`: `$` → `\g` and `$1`/`$2`/… → `\1`/`\2`/… (Python replacement syntax). Also fixed a bug in the WebSocket preview handler where a pattern error was incorrectly returning the search pattern string as the preview output instead of the original URL. (Fixes #1005) +- Web UI stream preview (`FloatingVideo`) was calling `mpegts.createPlayer()` with all `Config` options (e.g. `enableWorker`, `liveSync`, `headers`) merged into the first `MediaDataSource` argument. mpegts.js only reads `Config` from the optional second argument; unrecognised fields in the first are silently ignored. As a result all player configuration was effectively the library defaults — worker offloading was disabled, latency management had no effect, and the `Authorization: Bearer` header (required for user identification) was never sent. Fixed by splitting into the correct two-argument call. Both `liveBufferLatencyChasing` and `liveSync` have been disabled, eliminating playback-rate fluctuations that caused audible stuttering on live streams. SourceBuffer cleanup thresholds were also relaxed from 10s/5s to 120s/60s to prevent frequent SourceBuffer pauses. +- HTML named entities in XMLTV EPG files are now correctly preserved during lxml parsing. Some EPG providers (particularly French and other European sources) use HTML named entities like `é`, `î`, `ü` in channel names, program titles, and metadata. These are not valid XML entities — lxml 6.0.2 with `recover=True` silently drops them, causing characters to go missing (e.g., "Chaîne Télé" becomes "Chane Tl"). This is now fixed by injecting an XML `` internal subset declaring all 252 HTML 4 named entities directly into the byte stream that lxml reads, using a lightweight in-memory wrapper (`_PrependStream`) with zero disk I/O. libxml2 resolves the entities during its normal C-level parse pass — no Python-level preprocessing or temporary files are involved. The DOCTYPE block (~8 KB) is built once at module load from Python's stdlib `html.entities.name2codepoint` and reused for every parse. Files that already declare their own `` are passed through unchanged. (Closes #1095) — Thanks [@CodeBormen](https://github.com/CodeBormen) for helping with this! +- Duplicate recordings created when EPG sources refresh and re-evaluate series rules (Fixes #940) — Thanks [@CodeBormen](https://github.com/CodeBormen): + - **Program ID instability**: `parse_programs_for_source()` deletes and recreates all `ProgramData` rows with new auto-increment IDs on every EPG refresh. The dedup set used these IDs, so it never matched after a refresh. Deduplication now uses a stable `(tvg_id, start_time, end_time)` composite key sourced from `Recording.custom_properties.program`. + - **Secondary guard using wrong times**: The DB guard compared unadjusted program times against offset-adjusted `Recording.start_time`/`end_time`, so it never matched when any DVR pre/post offset was configured. It now queries `custom_properties__program__start_time/end_time` (the original, unadjusted program times stored at recording creation). + - **No concurrency guard**: Each EPG source refresh fired `evaluate_series_rules.delay()` independently. Concurrent tasks loaded the dedup set before others committed, allowing races. Evaluation is now serialized with `acquire_task_lock` (reusing the existing EPG task pattern). Gracefully degrades if Redis is unavailable — the primary and secondary dedup guards still protect. +- EPG refresh tasks (`refresh_epg_data`) were being killed mid-transaction on large EPG sources. The `soft_time_limit=1700s` introduced in v0.21.0 raised `SoftTimeLimitExceeded`, a subclass of `Exception`, which was swallowed by the existing `except Exception` handler in `parse_programs_for_source`, leaving the database in a partial state with no logged error. `soft_time_limit` has been removed from `refresh_epg_data` and `time_limit` raised to 14400s (4 hours) as a true last-resort ceiling; the existing `TaskLockRenewer` daemon thread continues to renew the Redis lock every 120s for legitimately long-running tasks. + ## [0.21.1] - 2026-03-18 ### Fixed diff --git a/Plugin_repo.md b/Plugin_repo.md new file mode 100644 index 00000000..d846526d --- /dev/null +++ b/Plugin_repo.md @@ -0,0 +1,586 @@ +# Dispatcharr Plugin Repository Specification + +How to create and host a plugin repository that Dispatcharr can consume. + +For writing plugins themselves, see [Plugins.md](Plugins.md). + +--- + +## Overview + +Dispatcharr discovers plugins from remote repositories using a two-level manifest system: + +1. **Repo manifest** - a JSON file listing all plugins in the repo with basic metadata. +2. **Per-plugin manifest** (optional) - a JSON file per plugin with full version history, checksums, and compatibility info. + +Users add a repo by its manifest URL. Dispatcharr fetches and caches the repo manifest periodically (default: every 6 hours, configurable). The UI displays all plugins from enabled repos in a browsable store. + +--- + +## Repo Manifest + +The repo manifest is the entry point. Dispatcharr fetches this URL and caches the response. + +### Minimal Example (no signing) + +```json +{ + "registry_name": "My Plugin Repo", + "plugins": [ + { + "slug": "my_plugin", + "name": "My Plugin", + "description": "Does something useful", + "author": "Your Name", + "latest_version": "1.0.0", + "latest_url": "https://example.com/releases/my_plugin-1.0.0.zip" + } + ] +} +``` + +This is the simplest valid repo manifest - one plugin with enough info to show in the store and install. + +### Full Example (with signing) + +```json +{ + "manifest": { + "registry_name": "My Plugin Repo", + "registry_url": "https://github.com/myorg/my-plugins", + "root_url": "https://raw.githubusercontent.com/myorg/my-plugins/releases", + "plugins": [ + { + "slug": "weather_display", + "name": "Weather Display", + "description": "Shows weather info on the dashboard", + "author": "Acme Labs", + "license": "MIT", + "latest_version": "1.2.5", + "last_updated": "2025-01-20T15:30:00Z", + "manifest_url": "plugins/weather_display/manifest.json", + "latest_url": "plugins/weather_display/releases/weather_display-1.2.5.zip", + "latest_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "icon_url": "plugins/weather_display/logo.png", + "min_dispatcharr_version": "2.5.0", + "max_dispatcharr_version": null + } + ] + }, + "signature": "-----BEGIN PGP SIGNATURE-----\n..." +} +``` + +### Accepted Formats + +Dispatcharr accepts two top-level shapes: + +**Wrapped (supports signing):** +```json +{ + "manifest": { "plugins": [...], ... }, + "signature": "..." +} +``` + +**Flat (no signing):** +```json +{ + "plugins": [...], + "registry_name": "...", + "root_url": "..." +} +``` + +The wrapped format is required for signing. If you don't need signing, the flat format works and is simpler. + +### Name Restrictions + +`registry_name` is required. Dispatcharr rejects repos that are missing it. + +Third-party repos must not use names that could be confused with an official Dispatcharr repo. The following words are blocked in `registry_name` (case-insensitive): + +- "official" +- "dispatcharr plugins" +- "dispatcharr repo" +- "dispatcharr official" + +If the name contains any of these, the repo will be rejected on add and skipped during refresh. + +--- + +## Repo Manifest Fields + +### Top-Level Metadata + +| Field | Required | Description | +|-------|----------|-------------| +| `registry_name` | **Yes** | Display name for the repo. Must not contain words like "official" or "dispatcharr" that could be mistaken for an official repo (see [Name Restrictions](#name-restrictions)). | +| `registry_url` | No | URL to the repo's home page (e.g. GitHub). Used as a fallback for generating icon URLs. | +| `root_url` | No | Base URL for resolving relative URLs in plugin entries. Trailing slashes are stripped. | +| `plugins` | **Yes** | Array of plugin entry objects. | + +### Plugin Entry Fields + +| Field | Required | Description | +|-------|----------|-------------| +| `slug` | **Yes** | Unique identifier. Alphanumeric, dashes, and underscores. Used as the install directory name (lowercased, dashes converted to underscores). | +| `name` | **Yes** | Human-readable display name. | +| `description` | No | Short description shown on the plugin card. | +| `author` | No | Author or organization name. | +| `license` | No | SPDX license identifier (e.g. `MIT`, `GPL-3.0`). Displayed as a link to the SPDX license page. | +| `latest_version` | No | Current latest version string (semver: `1.2.3` or `v1.2.3`). Drives update detection. | +| `last_updated` | No | ISO 8601 timestamp of the latest release. Shown as "Built" date in the detail view. | +| `manifest_url` | No | URL (or relative path) to the per-plugin manifest with full version history. See [Per-Plugin Manifest](#per-plugin-manifest). | +| `latest_url` | No | Direct download URL (or relative path) to the latest release zip. | +| `latest_sha256` | No | SHA256 checksum of the latest release zip (lowercase hex, 64 chars). | +| `latest_md5` | No | MD5 checksum of the latest release zip. Informational only - not validated by Dispatcharr. | +| `icon_url` | No | URL (or relative path) to a logo image (PNG recommended). | +| `min_dispatcharr_version` | No | Minimum Dispatcharr version required. Install is blocked if the running version is older. | +| `max_dispatcharr_version` | No | Maximum Dispatcharr version supported. Install is blocked if the running version is newer. | + +Extra fields in a plugin entry are passed through to the frontend as-is, so you can include custom metadata (e.g. `homepage`, `tags`) without breaking anything. + +### URL Resolution + +If `root_url` is set and a URL field (`manifest_url`, `latest_url`, `icon_url`) does not start with `http://` or `https://`, it is treated as relative and resolved as: + +``` +{root_url}/{field_value} +``` + +This lets you keep plugin entries compact: +```json +{ + "root_url": "https://raw.githubusercontent.com/myorg/my-plugins/releases", + "plugins": [ + { + "slug": "my_plugin", + "latest_url": "plugins/my_plugin/my_plugin-1.0.0.zip", + "icon_url": "plugins/my_plugin/logo.png", + "manifest_url": "plugins/my_plugin/manifest.json" + } + ] +} +``` + +**Icon fallback:** If `icon_url` is missing and `registry_url` is set, Dispatcharr generates a fallback URL by converting the GitHub URL to a raw content URL: +``` +{registry_url => raw.githubusercontent.com}/refs/heads/main/plugins/{slug}/logo.png +``` + +--- + +## Per-Plugin Manifest (Optional) + +The per-plugin manifest provides full version history. It is fetched on-demand when a user clicks "More Info" on a plugin card. It is **not required** - if `manifest_url` is absent, the UI builds a detail view from the repo-level fields instead. + +Include a per-plugin manifest if you want to: +- Offer multiple downloadable versions +- Show per-version compatibility ranges +- Display build timestamps and commit links for each version +- Provide detailed author/license info beyond what's in the repo manifest + +### Accepted Formats + +Same as the root manifest - both flat and wrapped formats are accepted: + +**Flat (no signing):** +```json +{ + "slug": "...", + "versions": [...] +} +``` + +**Wrapped (supports signing):** +```json +{ + "manifest": { + "slug": "...", + "versions": [...] + }, + "signature": "-----BEGIN PGP SIGNATURE-----\n..." +} +``` + +Use the wrapped format if you want to GPG-sign the per-plugin manifest. + +### Example + +```json +{ + "slug": "weather_display", + "name": "Weather Display", + "description": "Shows weather information on the Dispatcharr dashboard", + "author": "Acme Labs", + "license": "MIT", + "latest_version": "1.2.5", + "versions": [ + { + "version": "1.2.5", + "url": "releases/weather_display-1.2.5.zip", + "checksum_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "build_timestamp": "2025-01-20T15:30:00Z", + "commit_sha": "4e8f1b108c1e84f60520710d13e54eb2fb519648", + "commit_sha_short": "4e8f1b1", + "min_dispatcharr_version": "2.5.0", + "max_dispatcharr_version": null + }, + { + "version": "1.2.5-rc.1", + "url": "releases/weather_display-1.2.5-rc.1.zip", + "checksum_sha256": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2", + "prerelease": true, + "build_timestamp": "2025-01-18T09:00:00Z", + "min_dispatcharr_version": "2.5.0" + }, + { + "version": "1.2.4", + "url": "releases/weather_display-1.2.4.zip", + "checksum_sha256": "d4d967a67a4947e55183308cece206b30dda3e1b4fe00aae60f45a49c83b7ed6", + "build_timestamp": "2025-01-15T10:00:00Z", + "min_dispatcharr_version": "2.4.0" + } + ], + "latest": { + "version": "1.2.5", + "url": "releases/weather_display-1.2.5.zip", + "checksum_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "build_timestamp": "2025-01-20T15:30:00Z", + "min_dispatcharr_version": "2.5.0" + } +} +``` + +### Per-Plugin Manifest Fields + +| Field | Required | Description | +|-------|----------|-------------| +| `slug` | No | Plugin identifier (should match the repo entry). | +| `name` | No | Display name. | +| `description` | No | Full description shown in the detail modal. | +| `author` | No | Author/org name shown in the detail modal. | +| `license` | No | SPDX license identifier. | +| `latest_version` | No | Latest version string. | +| `versions` | No | Array of version objects (newest first recommended). | +| `latest` | No | Object mirroring the latest version entry for quick access. | + +### Version Object Fields + +| Field | Required | Description | +|-------|----------|-------------| +| `version` | **Yes** | Version string (`1.2.3` or `v1.2.3`). | +| `url` | **Yes** | Download URL for the zip. Relative URLs are resolved against the repo's `root_url`. | +| `checksum_sha256` | No | SHA256 hex checksum. **Strongly recommended.** Validated on install - mismatch blocks the install. | +| `prerelease` | No | Boolean. When `true`, marks this version as a pre-release (alpha, beta, RC, etc.). If the installed version is a prerelease, Dispatcharr will not suggest updating to the latest stable version - the user must install a new version manually. The latest version in the root manifest is always assumed to be stable, so this field only needs to appear in the per-plugin manifest. Omit or set to `false` for stable releases. | +| `build_timestamp` | No | ISO 8601 build timestamp. Shown as "Built" in the version detail. | +| `commit_sha` | No | Full Git commit SHA. Used to build a commit link if `registry_url` is set. | +| `commit_sha_short` | No | Abbreviated commit SHA. Displayed in the version detail table as a clickable link. | +| `min_dispatcharr_version` | No | Minimum compatible Dispatcharr version. | +| `max_dispatcharr_version` | No | Maximum compatible Dispatcharr version. | + +Relative `url` values in versions are resolved the same way as repo-level URLs: `{root_url}/{url}`. + +--- + +## Without a Per-Plugin Manifest + +If you omit `manifest_url` from a plugin entry, the store still works. When a user clicks "More Info", the UI builds a detail view from the repo-level fields: + +- `description`, `author`, `license` from the plugin entry +- A single version entry built from `latest_version`, `latest_url`, `latest_sha256`, `min_dispatcharr_version`, `max_dispatcharr_version`, and `last_updated` + +This is the simplest path for third-party repos that only publish one version at a time. You lose version history and per-version release dates, but install, update detection, and everything else works the same. + +--- + +## Signing + +Signing your repo manifest lets Dispatcharr verify it hasn't been tampered with. Signing is **optional** - unsigned repos work fine but show an "unverified" badge in the UI. + +### How It Works + +1. You generate a GPG keypair. +2. You sign the manifest JSON and include the detached signature in the response. +3. When adding the repo in Dispatcharr, the user pastes your public key. +4. Dispatcharr verifies the signature on every manifest fetch. + +### Key Format + +Standard PGP/GPG armored keys: + +``` +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQINBG... +... +-----END PGP PUBLIC KEY BLOCK----- +``` + +### Signing Convention + +The signature is computed over the **canonical JSON** representation of the `manifest` object (not the entire response), plus a trailing newline: + +```bash +# Canonical format: compact JSON (no spaces) + trailing newline +jq -c '.manifest' manifest.json | gpg --armor --detach-sign +``` + +In code terms: +```python +import json +canonical = json.dumps(manifest_obj, separators=(",", ":")) + "\n" +``` + +> **Important:** The signing input must be `json.dumps(obj, separators=(",", ":")) + "\n"` - compact JSON with no whitespace, followed by exactly one newline. Any difference (pretty-printing, trailing spaces, key ordering changes) will cause verification to fail. + +### Manifest Structure for Signing + +Use the wrapped format so the signature sits alongside the manifest: + +```json +{ + "manifest": { + "registry_name": "...", + "plugins": [...] + }, + "signature": "-----BEGIN PGP SIGNATURE-----\n...\n-----END PGP SIGNATURE-----" +} +``` + +### Verification Results + +| Result | Meaning | UI Badge | +|--------|---------|----------| +| `true` | Valid signature | Green checkmark | +| `false` | Invalid signature or verification error | Red X | +| `null` | Not attempted (no signature, no key, or `python-gnupg` not installed) | Gray/neutral | + +### Signing Workflow Example + +```bash +# Generate a keypair (one-time) +gpg --gen-key + +# Export your public key (give this to repo users) +gpg --armor --export "your@email.com" > my-repo.pub + +# Build your manifest +cat > manifest.json << 'EOF' +{ + "manifest": { + "registry_name": "My Repo", + "root_url": "https://example.com/releases", + "plugins": [ + { + "slug": "my_plugin", + "name": "My Plugin", + "latest_version": "1.0.0", + "latest_url": "plugins/my_plugin/my_plugin-1.0.0.zip" + } + ] + } +} +EOF + +# Sign the manifest object (canonical JSON + newline) +jq -c '.manifest' manifest.json | gpg --armor --detach-sign > manifest.sig + +# Combine into final output +jq --arg sig "$(cat manifest.sig)" '.signature = $sig' manifest.json > signed_manifest.json +``` + +### Third-Party Key Management + +When a user adds your repo URL, they can paste your public key. Dispatcharr stores the key per-repo and uses it for verification. Users can update the key at any time from the repo management UI. + +If you don't provide a key and the repo is not the official Dispatcharr repo, signature verification is skipped (result: `null`). + +--- + +## Release Zip Format + +Each plugin release is a `.zip` archive. + +### Requirements + +- Must contain a `plugin.py` with a `Plugin` class, **or** a Python package with `__init__.py` exporting a `Plugin` class. +- Files can be at the top level of the zip or inside a single subdirectory. +- Optionally include `plugin.json` for metadata discovery without code execution. +- Optionally include `logo.png` for the plugin icon. + +### Size Limits + +- Maximum 2000 files per archive. +- Maximum total size: 200 MB (configurable via `MAX_PLUGIN_IMPORT_BYTES` setting). + +### Recommended Structure + +``` +my_plugin-1.0.0.zip + plugin.py + plugin.json + logo.png + (any other files your plugin needs) +``` + +Or with a subdirectory: +``` +my_plugin-1.0.0.zip + my_plugin/ + plugin.py + plugin.json + logo.png + utils.py +``` + +--- + +## Install Flow + +When a user installs a plugin from the store: + +1. **Version compatibility check** - if `min_dispatcharr_version` or `max_dispatcharr_version` is set, the running Dispatcharr version is compared. Install is blocked if out of range. +2. **Download** - the zip is streamed from `download_url` (max 200 MB). +3. **SHA256 integrity check** - if `sha256` was provided, the download is hashed and compared. Mismatch blocks the install. +4. **Extraction** - the zip is extracted to a temp directory, validated, then moved to `/data/plugins/{plugin_key}/`. If the plugin already exists, the old version is backed up and restored on failure (atomic rollback). +5. **Registration** - a `PluginConfig` record is created or updated, linking the plugin to its source repo and slug. +6. **Discovery reload** - the plugin loader re-scans all plugin directories. + +The plugin is installed **disabled** by default. The user can enable it from the post-install dialog or the My Plugins page. + +--- + +## Update Detection + +Dispatcharr detects updates by comparing `installed_version` (stored in the database) against `latest_version` from the repo manifest. This uses repo-level fields only - per-plugin manifests are not needed for update detection. + +A plugin shows "Update Available" when: +- It is managed (installed from a repo) +- Its `installed_version` differs from `latest_version` +- It was installed from the same repo + +--- + +## Hosting Options + +A plugin repo manifest is just a JSON file served over HTTPS. Some options: + +### GitHub Pages / Raw Content +Host your manifest and release zips in a GitHub repo. Use raw.githubusercontent.com URLs: +``` +https://raw.githubusercontent.com/myorg/my-plugins/main/manifest.json +``` + +Use `root_url` pointing to your releases branch/path so version URLs stay relative. + +### Static File Server +Any web server that serves JSON works. Dispatcharr fetches manifests server-side, so CORS is not needed. + +### GitHub Releases +You can host release zips as GitHub Release assets and reference them with absolute URLs in your manifest. The manifest itself can live in the repo's default branch. + +--- + +## Refresh Behavior + +- Manifests are refreshed automatically at a configurable interval (default: 6 hours, setting: `refresh_interval_hours`, 0 = disabled). +- Users can force a refresh from the repo management UI. +- A new repo is refreshed immediately when added. +- On refresh, if a plugin's `slug` disappears from the manifest, its `PluginConfig` is unlinked from the repo (becomes "unmanaged") but the installed files are not deleted. + +--- + +## Checklist: Publishing a Plugin Repo + +### Minimum Viable Repo + +- [ ] Host a JSON file at a stable, public URL +- [ ] Set `registry_name` (required, must not sound official) +- [ ] Include at least one plugin entry with `slug`, `name`, and `latest_version` +- [ ] Host a downloadable `.zip` for each plugin and set `latest_url` +- [ ] Share the manifest URL with users + +### Recommended + +- [ ] Set `root_url` so plugin URLs can be relative +- [ ] Include `description`, `author`, and `icon_url` per plugin +- [ ] Include `latest_sha256` for integrity verification +- [ ] Include `license` (SPDX identifier) +- [ ] Include `last_updated` timestamps +- [ ] Add a per-plugin `manifest_url` with version history +- [ ] Include `sha256` in every version object +- [ ] Include `min_dispatcharr_version` where applicable +- [ ] Include `plugin.json` in each release zip + +### Optional + +- [ ] Sign your manifest with GPG and publish your public key +- [ ] Set `registry_url` to enable automatic icon fallback +- [ ] Set `max_dispatcharr_version` if a plugin is incompatible with newer releases + +--- + +## Quick Reference: Repo Manifest Schema + +```json +{ + "manifest": { + "registry_name": "string (required)", + "registry_url": "string (optional)", + "root_url": "string (optional)", + "plugins": [ + { + "slug": "string (required)", + "name": "string (required)", + "description": "string", + "author": "string", + "license": "string (SPDX)", + "latest_version": "string (semver)", + "last_updated": "string (ISO 8601)", + "manifest_url": "string (URL or relative path)", + "latest_url": "string (URL or relative path)", + "latest_sha256": "string (64-char hex)", + "icon_url": "string (URL or relative path)", + "min_dispatcharr_version": "string (semver)", + "max_dispatcharr_version": "string (semver) or null" + } + ] + }, + "signature": "string (armored PGP signature, optional)" +} +``` + +## Quick Reference: Per-Plugin Manifest Schema + +```json +{ + "slug": "string", + "name": "string", + "description": "string", + "author": "string", + "license": "string (SPDX)", + "latest_version": "string (semver)", + "versions": [ + { + "version": "string (required)", + "url": "string (required, URL or relative path)", + "checksum_sha256": "string (64-char hex)", + "build_timestamp": "string (ISO 8601)", + "commit_sha": "string", + "commit_sha_short": "string", + "min_dispatcharr_version": "string (semver)", + "max_dispatcharr_version": "string (semver) or null" + } + ], + "latest": { + "version": "string", + "url": "string", + "checksum_sha256": "string", + "build_timestamp": "string", + "min_dispatcharr_version": "string", + "max_dispatcharr_version": "string or null" + } +} +``` diff --git a/apps/accounts/api_views.py b/apps/accounts/api_views.py index 76892929..2ac410f2 100644 --- a/apps/accounts/api_views.py +++ b/apps/accounts/api_views.py @@ -6,6 +6,7 @@ from django.views.decorators.csrf import csrf_exempt from rest_framework.decorators import api_view, permission_classes, action from rest_framework.response import Response from rest_framework import viewsets, status, serializers +from rest_framework.throttling import AnonRateThrottle from drf_spectacular.utils import extend_schema, OpenApiParameter, inline_serializer from drf_spectacular.types import OpenApiTypes import json @@ -20,9 +21,14 @@ from rest_framework_simplejwt.views import TokenObtainPairView, TokenRefreshView logger = logging.getLogger(__name__) +class LoginRateThrottle(AnonRateThrottle): + scope = "login" + + class TokenObtainPairView(TokenObtainPairView): + throttle_classes = [LoginRateThrottle] + def post(self, request, *args, **kwargs): - # Custom logic here if not network_access_allowed(request, "UI"): # Log blocked login attempt due to network restrictions from core.utils import log_system_event @@ -153,12 +159,11 @@ class AuthViewSet(viewsets.ViewSet): Login doesn't require auth, but logout does """ if self.action == 'logout': - from rest_framework.permissions import IsAuthenticated - return [IsAuthenticated()] + return [Authenticated()] return [] @extend_schema( - description="Authenticate and log in a user", + description="Alias for POST /api/accounts/token/ — returns JWT access and refresh tokens.", request=inline_serializer( name="LoginRequest", fields={ @@ -168,55 +173,10 @@ class AuthViewSet(viewsets.ViewSet): ), ) def login(self, request): - """Logs in a user and returns user details""" - username = request.data.get("username") - password = request.data.get("password") - user = authenticate(request, username=username, password=password) - - # Get client info for logging - from core.utils import log_system_event - client_ip = request.META.get('REMOTE_ADDR', 'unknown') - user_agent = request.META.get('HTTP_USER_AGENT', 'unknown') - logger.debug(f"Login attempt via session: user={username} ip={client_ip}") - - if user: - login(request, user) - # Update last_login timestamp - from django.utils import timezone - user.last_login = timezone.now() - user.save(update_fields=['last_login']) - - # Log successful login - log_system_event( - event_type='login_success', - user=username, - client_ip=client_ip, - user_agent=user_agent, - ) - logger.info(f"Login success via session: user={username} ip={client_ip}") - - return Response( - { - "message": "Login successful", - "user": { - "id": user.id, - "username": user.username, - "email": user.email, - "groups": list(user.groups.values_list("name", flat=True)), - }, - } - ) - - # Log failed login attempt - log_system_event( - event_type='login_failed', - user=username or 'unknown', - client_ip=client_ip, - user_agent=user_agent, - reason='Invalid credentials', - ) - logger.info(f"Login failed via session: user={username} ip={client_ip}") - return Response({"error": "Invalid credentials"}, status=400) + """Delegates to TokenObtainPairView (JWT login). Throttling, logging, and + network access checks are handled there.""" + view = TokenObtainPairView.as_view() + return view(request._request) @extend_schema( description="Log out the current user", @@ -287,11 +247,18 @@ class UserViewSet(viewsets.ModelViewSet): if request.method == "PATCH": ALLOWED_FIELDS = {"custom_properties", "first_name", "last_name", "email", "password"} disallowed = set(request.data.keys()) - ALLOWED_FIELDS - if disallowed: - return Response( - {"detail": f"Fields not allowed for self-update: {', '.join(disallowed)}"}, - status=400, - ) + + for key in disallowed: + request.data.pop(key, None) + + # Strip admin-managed keys from custom_properties so users cannot + # set their own XC credentials via this endpoint. + ADMIN_ONLY_PROPS = {"xc_password"} + cp = request.data.get("custom_properties") + if isinstance(cp, dict): + for key in ADMIN_ONLY_PROPS: + cp.pop(key, None) + serializer = UserSerializer(user, data=request.data, partial=True) serializer.is_valid(raise_exception=True) serializer.save() diff --git a/apps/accounts/authentication.py b/apps/accounts/authentication.py index 5380af49..f5169e7f 100644 --- a/apps/accounts/authentication.py +++ b/apps/accounts/authentication.py @@ -1,9 +1,46 @@ from rest_framework import authentication from rest_framework import exceptions from django.conf import settings +from drf_spectacular.extensions import OpenApiAuthenticationExtension from .models import User +class JWTAuthenticationScheme(OpenApiAuthenticationExtension): + target_class = "rest_framework_simplejwt.authentication.JWTAuthentication" + name = "jwtAuth" + + def get_security_definition(self, auto_schema): + return { + "type": "http", + "scheme": "bearer", + "bearerFormat": "JWT", + "description": ( + "JWT Bearer authentication.\n\n" + "Obtain a token pair via `POST /api/accounts/token/` using your username and password, " + "then paste the **access token** here — Swagger adds the `Bearer ` prefix automatically.\n\n" + "Access tokens expire after 30 minutes. Refresh using `POST /api/accounts/token/refresh/`." + ), + } + + +class ApiKeyAuthenticationScheme(OpenApiAuthenticationExtension): + target_class = "apps.accounts.authentication.ApiKeyAuthentication" + name = "ApiKeyAuth" + + def get_security_definition(self, auto_schema): + return { + "type": "apiKey", + "in": "header", + "name": "X-API-Key", + "description": ( + "API key authentication.\n\n" + "Pass your personal API key in the `X-API-Key` request header. " + "Keys can be generated via `POST /api/accounts/api-keys/generate/` " + "and revoked via `POST /api/accounts/api-keys/revoke/`." + ), + } + + class ApiKeyAuthentication(authentication.BaseAuthentication): """ Accepts header `Authorization: ApiKey ` or `X-API-Key: `. diff --git a/apps/accounts/migrations/0006_user_stream_limit.py b/apps/accounts/migrations/0006_user_stream_limit.py new file mode 100644 index 00000000..5714e1ef --- /dev/null +++ b/apps/accounts/migrations/0006_user_stream_limit.py @@ -0,0 +1,18 @@ +# Generated by Django 5.2.11 on 2026-03-19 13:46 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('accounts', '0005_alter_user_managers'), + ] + + operations = [ + migrations.AddField( + model_name='user', + name='stream_limit', + field=models.IntegerField(default=0), + ), + ] diff --git a/apps/accounts/models.py b/apps/accounts/models.py index e04d66ed..2a32f95a 100644 --- a/apps/accounts/models.py +++ b/apps/accounts/models.py @@ -30,6 +30,7 @@ class User(AbstractUser): user_level = models.IntegerField(default=UserLevel.STREAMER) custom_properties = models.JSONField(default=dict, blank=True, null=True) api_key = models.CharField(max_length=200, blank=True, null=True, db_index=True) + stream_limit = models.IntegerField(default=0) def __str__(self): return self.username diff --git a/apps/accounts/serializers.py b/apps/accounts/serializers.py index c90ffef4..8a9f609e 100644 --- a/apps/accounts/serializers.py +++ b/apps/accounts/serializers.py @@ -65,6 +65,7 @@ class UserSerializer(serializers.ModelSerializer): "channel_profiles", "custom_properties", "avatar_config", + "stream_limit", "is_staff", "is_superuser", "last_login", diff --git a/apps/backups/api_views.py b/apps/backups/api_views.py index a5af495e..36334d06 100644 --- a/apps/backups/api_views.py +++ b/apps/backups/api_views.py @@ -13,6 +13,7 @@ from rest_framework.permissions import AllowAny from apps.accounts.permissions import IsAdmin from rest_framework.parsers import MultiPartParser, FormParser from rest_framework.response import Response +from core.utils import safe_upload_path from . import services from .tasks import create_backup_task, restore_backup_task @@ -267,10 +268,18 @@ def upload_backup(request): try: backup_dir = services.get_backup_dir() - filename = uploaded.name or "uploaded-backup.zip" + # Sanitize filename: strip directory components to prevent path traversal + filename = Path(uploaded.name or "uploaded-backup.zip").name + if not filename: + filename = "uploaded-backup.zip" + + try: + safe_upload_path(filename, str(backup_dir)) + except ValueError: + return Response({"detail": "Invalid filename."}, status=status.HTTP_400_BAD_REQUEST) # Ensure unique filename - backup_file = backup_dir / filename + backup_file = (backup_dir / filename).resolve() counter = 1 while backup_file.exists(): name_parts = filename.rsplit(".", 1) diff --git a/apps/backups/services.py b/apps/backups/services.py index b638e701..c70160e1 100644 --- a/apps/backups/services.py +++ b/apps/backups/services.py @@ -28,10 +28,36 @@ def _is_postgresql() -> bool: def _get_pg_env() -> dict: - """Get environment variables for PostgreSQL commands.""" + """Get environment variables for PostgreSQL commands. + + Includes PGPASSWORD for password auth and PGSSL* variables for TLS. + Reads TLS config from DATABASES['default']['OPTIONS'], which is + populated by settings.py when POSTGRES_SSL=true. + """ db_config = settings.DATABASES["default"] env = os.environ.copy() - env["PGPASSWORD"] = db_config.get("PASSWORD", "") + + password = db_config.get("PASSWORD", "") + if password: + env["PGPASSWORD"] = password + else: + env.pop("PGPASSWORD", None) + + # Propagate TLS configuration from Django OPTIONS to libpq env vars. + options = db_config.get("OPTIONS", {}) + _ssl_env_map = { + "sslmode": "PGSSLMODE", + "sslrootcert": "PGSSLROOTCERT", + "sslcert": "PGSSLCERT", + "sslkey": "PGSSLKEY", + } + # Always strip inherited PGSSL* vars first, then set only what is explicitly configured + for opt_key, env_key in _ssl_env_map.items(): + env.pop(env_key, None) + value = options.get(opt_key) + if value: + env[env_key] = value + return env diff --git a/apps/backups/tests.py b/apps/backups/tests.py index e5743623..bdea2be5 100644 --- a/apps/backups/tests.py +++ b/apps/backups/tests.py @@ -15,6 +15,96 @@ from . import services User = get_user_model() +class PgEnvTlsTestCase(TestCase): + """Test that _get_pg_env includes TLS and password env vars correctly.""" + databases = [] + + @patch('apps.backups.services.settings') + def test_pg_env_includes_ssl_vars_when_tls_enabled(self, mock_settings): + mock_settings.DATABASES = { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": "testdb", + "USER": "testuser", + "PASSWORD": "testpass", + "HOST": "localhost", + "PORT": 5432, + "OPTIONS": { + "sslmode": "verify-full", + "sslrootcert": "/certs/ca.crt", + "sslcert": "/certs/client.crt", + "sslkey": "/certs/client.key", + }, + } + } + env = services._get_pg_env() + self.assertEqual(env["PGSSLMODE"], "verify-full") + self.assertEqual(env["PGSSLROOTCERT"], "/certs/ca.crt") + self.assertEqual(env["PGSSLCERT"], "/certs/client.crt") + self.assertEqual(env["PGSSLKEY"], "/certs/client.key") + self.assertEqual(env["PGPASSWORD"], "testpass") + + @patch('apps.backups.services.settings') + def test_pg_env_no_ssl_vars_when_tls_disabled(self, mock_settings): + mock_settings.DATABASES = { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": "testdb", + "USER": "testuser", + "PASSWORD": "testpass", + "HOST": "localhost", + "PORT": 5432, + } + } + env = services._get_pg_env() + self.assertNotIn("PGSSLMODE", env) + self.assertNotIn("PGSSLROOTCERT", env) + self.assertNotIn("PGSSLCERT", env) + self.assertNotIn("PGSSLKEY", env) + self.assertEqual(env["PGPASSWORD"], "testpass") + + @patch('apps.backups.services.settings') + def test_pg_env_no_password_when_empty(self, mock_settings): + """Cert-only auth: PGPASSWORD must not be set when password is empty.""" + mock_settings.DATABASES = { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": "testdb", + "USER": "testuser", + "PASSWORD": "", + "HOST": "localhost", + "PORT": 5432, + "OPTIONS": {"sslmode": "verify-full"}, + } + } + env = services._get_pg_env() + self.assertNotIn("PGPASSWORD", env) + self.assertEqual(env["PGSSLMODE"], "verify-full") + + @patch('apps.backups.services.settings') + def test_pg_env_partial_ssl_options(self, mock_settings): + """Server-only TLS: only sslmode and CA cert, no client cert/key.""" + mock_settings.DATABASES = { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": "testdb", + "USER": "testuser", + "PASSWORD": "pass", + "HOST": "localhost", + "PORT": 5432, + "OPTIONS": { + "sslmode": "verify-ca", + "sslrootcert": "/certs/ca.crt", + }, + } + } + env = services._get_pg_env() + self.assertEqual(env["PGSSLMODE"], "verify-ca") + self.assertEqual(env["PGSSLROOTCERT"], "/certs/ca.crt") + self.assertNotIn("PGSSLCERT", env) + self.assertNotIn("PGSSLKEY", env) + + class BackupServicesTestCase(TestCase): """Test cases for backup services""" diff --git a/apps/channels/api_views.py b/apps/channels/api_views.py index 08283e73..10d855db 100644 --- a/apps/channels/api_views.py +++ b/apps/channels/api_views.py @@ -24,12 +24,13 @@ from apps.accounts.permissions import ( ) from core.models import UserAgent, CoreSettings -from core.utils import RedisClient +from core.utils import RedisClient, safe_upload_path from .models import ( Stream, Channel, ChannelGroup, + ChannelStream, Logo, ChannelProfile, ChannelProfileMembership, @@ -61,7 +62,7 @@ from rest_framework.filters import SearchFilter, OrderingFilter from apps.epg.models import EPGData from apps.vod.models import Movie, Series from django.db.models import Q -from django.http import StreamingHttpResponse, FileResponse, Http404 +from django.http import HttpResponse, StreamingHttpResponse, FileResponse, Http404 from django.utils import timezone import mimetypes from django.conf import settings @@ -630,6 +631,53 @@ class ChannelViewSet(viewsets.ModelViewSet): context["include_streams"] = include_streams return context + @extend_schema( + methods=["PATCH"], + description=( + "Bulk edit multiple channels in a single request. " + "Accepts a JSON array of channel update objects. Each object must include `id` (the channel's primary key). " + "All other fields are optional and support partial updates. " + "The `streams` field accepts a list of stream IDs and will replace the channel's current stream assignments. " + "All updates are validated before any changes are applied and executed in a single database transaction." + ), + request=inline_serializer( + name="ChannelBulkEditRequest", + fields={ + "id": serializers.IntegerField(help_text="ID of the channel to update (required)."), + "name": serializers.CharField(required=False), + "channel_number": serializers.FloatField(required=False), + "channel_group_id": serializers.IntegerField(required=False, allow_null=True), + "streams": serializers.ListField( + child=serializers.IntegerField(), + required=False, + help_text="List of stream IDs to assign to this channel (replaces existing assignments).", + ), + "stream_profile_id": serializers.IntegerField(required=False, allow_null=True), + "logo_id": serializers.IntegerField(required=False, allow_null=True), + "tvg_id": serializers.CharField(required=False, allow_blank=True), + "tvc_guide_stationid": serializers.CharField(required=False, allow_blank=True), + "epg_data_id": serializers.IntegerField(required=False, allow_null=True), + "user_level": serializers.IntegerField(required=False), + "is_adult": serializers.BooleanField(required=False), + }, + many=True, + ), + responses={ + 200: inline_serializer( + name="ChannelBulkEditResponse", + fields={ + "message": serializers.CharField(), + "channels": ChannelSerializer(many=True), + }, + ), + 400: inline_serializer( + name="ChannelBulkEditErrorResponse", + fields={ + "errors": serializers.ListField(child=serializers.DictField()), + }, + ), + }, + ) @action(detail=False, methods=["patch"], url_path="edit/bulk") def edit_bulk(self, request): """ @@ -709,19 +757,24 @@ class ChannelViewSet(viewsets.ModelViewSet): # Apply all updates in a transaction with transaction.atomic(): + streams_updates = [] for channel, validated_data in validated_updates: + # Pop streams before setattr loop — M2M fields can't be set via setattr + streams = validated_data.pop("streams", None) + if streams is not None: + streams_updates.append((channel, streams)) for key, value in validated_data.items(): setattr(channel, key, value) # Single bulk_update query instead of individual saves channels_to_update = [channel for channel, _ in validated_updates] if channels_to_update: - # Collect all unique field names from all updates + # Collect all unique field names from all updates (streams already popped) all_fields = set() for _, validated_data in validated_updates: all_fields.update(validated_data.keys()) - # Only call bulk_update if there are fields to update + # Only call bulk_update if there are non-M2M fields to update if all_fields: Channel.objects.bulk_update( channels_to_update, @@ -729,6 +782,32 @@ class ChannelViewSet(viewsets.ModelViewSet): batch_size=100 ) + # Handle streams M2M updates separately + for channel, streams in streams_updates: + normalized_ids = [ + stream.id if hasattr(stream, "id") else stream for stream in streams + ] + current_links = { + cs.stream_id: cs for cs in channel.channelstream_set.all() + } + existing_ids = set(current_links.keys()) + new_ids = set(normalized_ids) + + to_remove = existing_ids - new_ids + if to_remove: + channel.channelstream_set.filter(stream_id__in=to_remove).delete() + + for order, stream_id in enumerate(normalized_ids): + if stream_id in current_links: + cs = current_links[stream_id] + if cs.order != order: + cs.order = order + cs.save(update_fields=["order"]) + else: + ChannelStream.objects.create( + channel=channel, stream_id=stream_id, order=order + ) + # Return the updated objects (already in memory) serialized_channels = ChannelSerializer( [channel for channel, _ in validated_updates], @@ -1486,7 +1565,11 @@ class ChannelViewSet(viewsets.ModelViewSet): name="EpgAssociation", fields={ "channel_id": serializers.IntegerField(), - "epg_data_id": serializers.IntegerField(), + "epg_data_id": serializers.IntegerField( + required=False, + allow_null=True, + help_text="EPG data ID to link. Pass null to remove EPG linkage.", + ), }, ), ) @@ -1662,7 +1745,12 @@ class BulkDeleteLogosAPIView(APIView): "logo_ids": serializers.ListField( child=serializers.IntegerField(), help_text="Logo IDs to delete", - ) + ), + "delete_files": serializers.BooleanField( + required=False, + default=False, + help_text="Whether to also delete local logo files from disk.", + ), }, ), ) @@ -1901,10 +1989,13 @@ class LogoViewSet(viewsets.ModelViewSet): {"error": str(e)}, status=status.HTTP_400_BAD_REQUEST ) - file_name = file.name - file_path = os.path.join("/data/logos", file_name) + # Sanitize filename: strip directory components to prevent path traversal + try: + file_path = safe_upload_path(file.name, "/data/logos") + except ValueError: + return Response({"error": "Invalid filename."}, status=status.HTTP_400_BAD_REQUEST) - os.makedirs(os.path.dirname(file_path), exist_ok=True) + os.makedirs("/data/logos", exist_ok=True) with open(file_path, "wb+") as destination: for chunk in file.chunks(): destination.write(chunk) @@ -1924,7 +2015,7 @@ class LogoViewSet(viewsets.ModelViewSet): # Get custom name from request data, fallback to filename custom_name = request.data.get('name', '').strip() - logo_name = custom_name if custom_name else file_name + logo_name = custom_name if custom_name else os.path.basename(file_path) logo, _ = Logo.objects.get_or_create( url=file_path, @@ -1966,7 +2057,7 @@ class LogoViewSet(viewsets.ModelViewSet): return response else: # Remote image - # Skip URLs that recently failed to avoid blocking Daphne workers + # Skip URLs that recently failed to avoid blocking workers # on unreachable hosts (e.g., dead CDNs referenced by old recordings). fail_expiry = _logo_fetch_failures.get(logo_url) if fail_expiry and time.monotonic() < fail_expiry: @@ -1982,15 +2073,37 @@ class LogoViewSet(viewsets.ModelViewSet): # Fallback to hardcoded if default not found user_agent = 'Dispatcharr/1.0' - # Add proper timeouts to prevent hanging + # Hard total timeout (connect + full download) prevents a slow + # server dripping bytes from holding a greenlet indefinitely. + _LOGO_TOTAL_TIMEOUT = 10 # seconds + _LOGO_MAX_BYTES = 5 * 1024 * 1024 # 5 MB + remote_response = requests.get( logo_url, stream=True, - timeout=(3, 5), # (connect_timeout, read_timeout) + timeout=(3, 5), # (connect_timeout, read_timeout per chunk) headers={'User-Agent': user_agent} ) if remote_response.status_code == 200: - # Success — clear any previous failure entry + # Eagerly read the full image with a total time + size cap + # so the greenlet is released quickly. + chunks = [] + total = 0 + deadline = time.monotonic() + _LOGO_TOTAL_TIMEOUT + for chunk in remote_response.iter_content(chunk_size=8192): + total += len(chunk) + if total > _LOGO_MAX_BYTES: + remote_response.close() + raise Http404("Remote image too large") + if time.monotonic() > deadline: + remote_response.close() + now = time.monotonic() + _logo_fetch_failures[logo_url] = now + _LOGO_FAIL_TTL + raise Http404("Remote image fetch timed out") + chunks.append(chunk) + body = b"".join(chunks) + + # Full read succeeded, clear any previous failure entry _logo_fetch_failures.pop(logo_url, None) # Try to get content type from response headers first @@ -2004,13 +2117,14 @@ class LogoViewSet(viewsets.ModelViewSet): if not content_type: content_type = "image/jpeg" - response = StreamingHttpResponse( - remote_response.iter_content(chunk_size=8192), + response = HttpResponse( + body, content_type=content_type, ) - if(remote_response.headers.get("Cache-Control")): + response["Content-Length"] = str(len(body)) + if remote_response.headers.get("Cache-Control"): response["Cache-Control"] = remote_response.headers.get("Cache-Control") - if(remote_response.headers.get("Last-Modified")): + if remote_response.headers.get("Last-Modified"): response["Last-Modified"] = remote_response.headers.get("Last-Modified") response["Content-Disposition"] = 'inline; filename="{}"'.format( os.path.basename(logo_url) @@ -2674,7 +2788,49 @@ class RecordingViewSet(viewsets.ModelViewSet): recording_id = instance.pk channel_name = instance.channel.name - # Capture state before the DB row is deleted + # Attempt to close the DVR client connection for this channel if active + try: + channel_uuid = str(instance.channel.uuid) + # Lazy imports to avoid module overhead if proxy isn't used + from core.utils import RedisClient + from apps.proxy.ts_proxy.redis_keys import RedisKeys + from apps.proxy.ts_proxy.services.channel_service import ChannelService + + r = RedisClient.get_client() + if r: + client_set_key = RedisKeys.clients(channel_uuid) + client_ids = r.smembers(client_set_key) or [] + stopped = 0 + for cid in client_ids: + try: + meta_key = RedisKeys.client_metadata(channel_uuid, cid) + ua = r.hget(meta_key, "user_agent") + # Identify DVR recording client by its user agent + if ua and "Dispatcharr-DVR" in ua: + try: + ChannelService.stop_client(channel_uuid, cid) + stopped += 1 + except Exception as inner_e: + logger.debug(f"Failed to stop DVR client {cid} for channel {channel_uuid}: {inner_e}") + except Exception as inner: + logger.debug(f"Error while checking client metadata: {inner}") + if stopped: + logger.info(f"Stopped {stopped} DVR client(s) for channel {channel_uuid} due to recording cancellation") + # If no clients remain after stopping DVR clients, proactively stop the channel + try: + remaining = r.scard(client_set_key) or 0 + except Exception: + remaining = 0 + if remaining == 0: + try: + ChannelService.stop_channel(channel_uuid) + logger.info(f"Stopped channel {channel_uuid} (no clients remain)") + except Exception as sc_e: + logger.debug(f"Unable to stop channel {channel_uuid}: {sc_e}") + except Exception as e: + logger.debug(f"Unable to stop DVR clients for cancelled recording: {e}") + + # Capture paths before deletion cp = instance.custom_properties or {} rec_status = cp.get("status", "") file_path = cp.get("file_path") diff --git a/apps/channels/models.py b/apps/channels/models.py index 3bebd224..51629db3 100644 --- a/apps/channels/models.py +++ b/apps/channels/models.py @@ -204,7 +204,7 @@ class Stream(models.Model): return stream_profile - def get_stream(self): + def get_stream(self, requester=None): """ Finds an available profile for this stream and reserves a connection slot. @@ -400,6 +400,144 @@ class Channel(models.Model): return stream_profile + def _pick_channel_to_preempt( + self, + profile_id, + requester_level, + redis_client, + exclude_channel_ids=None, + cooldown_seconds=30, + ): + """ + Pick the lowest-impact channel to terminate on the given profile. + Returns: Optional[int] channel_id to preempt + """ + exclude_channel_ids = set(exclude_channel_ids or []) + candidates = [] + + # 1) Try to get active channel IDs for this profile from an index set if available + ch_set_key = f"ts_proxy:profile:{profile_id}:channels" + try: + ch_ids = { (int(x) if not isinstance(x, int) else x) for x in (redis_client.smembers(ch_set_key) or set()) } + except Exception: + ch_ids = set() + + logger.debug("Candidate channels for preemption:") + logger.debug(ch_ids) + + # 2) Fallback: scan metadata keys and filter by m3u_profile == profile_id + if not ch_ids: + cursor = 0 + pattern = "ts_proxy:channel:*:metadata" + while True: + cursor, keys = redis_client.scan(cursor=cursor, match=pattern, count=500) + if keys: + # Prefer HGET m3u_profile if metadata is a hash + pipe = redis_client.pipeline() + for k in keys: + pipe.hget(k, "m3u_profile") + prof_vals = pipe.execute() + for k, prof_val in zip(keys, prof_vals): + try: + pid = int(prof_val) if prof_val is not None else None + except Exception: + pid = None + + if pid == profile_id: + parts = k.split(":") # ts_proxy:channel:{id}:metadata + if len(parts) >= 4: + try: + ch_ids.add(int(parts[2])) + except Exception: + pass + if cursor == 0: + break + + logger.debug("Candidate channels for preemption:") + logger.debug(ch_ids) + + if not ch_ids: + return None + + # 3) Score candidates + for ch_id in ch_ids: + if ch_id in exclude_channel_ids: + continue + + # Skip if recently preempted + last_preempt_key = f"ts_proxy:channel:{ch_id}:last_preempt" + try: + last_preempt = float(redis_client.get(last_preempt_key) or 0.0) + except Exception: + last_preempt = 0.0 + if last_preempt and (time.time() - last_preempt) < cooldown_seconds: + continue + + # Clients and their levels + clients_key = f"ts_proxy:channel:{ch_id}:clients" + member_ids = list(redis_client.smembers(clients_key) or []) + viewer_count = len(member_ids) + max_viewer_level = 0 + if viewer_count: + pipe = redis_client.pipeline() + for cid in member_ids: + pipe.hget(f"ts_proxy:channel:{ch_id}:clients:{cid}", "user_level") + levels_raw = pipe.execute() + levels = [] + for lv in levels_raw: + try: + levels.append(int(lv or 0)) + except Exception: + levels.append(0) + max_viewer_level = max(levels or [0]) + + # Only preempt if requester strictly outranks this channel's viewers + if requester_level <= max_viewer_level: + continue + + # Metadata (protected/recording/started_at_ts) + meta_key = f"ts_proxy:channel:{ch_id}:metadata" + try: + protected, recording, started_at_ts = redis_client.hmget( + meta_key, "protected", "recording", "started_at_ts" + ) + except Exception: + protected = recording = started_at_ts = None + + protected = str(protected or "0") in ("1", "true", "True") + recording = str(recording or "0") in ("1", "true", "True") + if protected or recording: + continue + + try: + started_at_ts = float(started_at_ts) if started_at_ts is not None else None + except Exception: + started_at_ts = None + if started_at_ts is None: + started_at_ts = time.time() # treat unknown as newest + + # Score: lower is safer to terminate + has_viewers = 1 if viewer_count > 0 else 0 + score = (has_viewers, max_viewer_level, viewer_count, started_at_ts) + candidates.append((score, ch_id)) + + logger.debug("Candidate channels after scoring:") + logger.debug(candidates) + + if not candidates: + return None + + candidates.sort(key=lambda x: x[0]) + victim_id = candidates[0][1] + + # Mark preempt timestamp to avoid thrashing + try: + redis_client.set(f"ts_proxy:channel:{victim_id}:last_preempt", str(time.time()), ex=3600) + except Exception: + pass + + return victim_id + def _check_and_reserve_profile_slot(self, profile, redis_client): """ Atomically check and reserve a connection slot for the given profile. @@ -432,7 +570,7 @@ class Channel(models.Model): redis_client.decr(profile_connections_key) return (False, new_count - 1) - def get_stream(self): + def get_stream(self, requester=None): """ Finds an available stream for the requested channel and returns the selected stream and profile. @@ -513,6 +651,17 @@ class Channel(models.Model): None, ) # Return newly assigned stream and matched profile else: + # At capacity: try to preempt a lower-impact channel on this profile + victim_channel_id = self._pick_channel_to_preempt( + profile_id=profile.id, + requester_level=requester.user_level if requester else 100, + redis_client=redis_client, + exclude_channel_ids=None, + ) + if victim_channel_id: + logger.info(f"Preempting channel {victim_channel_id} for new stream on profile {profile.id}") + # return self.id, profile.id, victim_channel_id + # This profile is at max connections has_streams_but_maxed_out = True logger.debug( diff --git a/apps/channels/tasks.py b/apps/channels/tasks.py index b5969cd1..59122d9f 100755 --- a/apps/channels/tasks.py +++ b/apps/channels/tasks.py @@ -19,6 +19,7 @@ from rapidfuzz import fuzz from apps.channels.models import Channel from apps.epg.models import EPGData from core.models import CoreSettings +from core.utils import acquire_task_lock, release_task_lock from django.db import OperationalError, close_old_connections from channels.layers import get_channel_layer @@ -1070,12 +1071,39 @@ def match_single_channel_epg(channel_id): def evaluate_series_rules_impl(tvg_id: str | None = None): """Synchronous implementation of series rule evaluation; returns details for debugging.""" + result = {"scheduled": 0, "details": []} + + # Serialize all invocations to prevent concurrent evaluations from + # racing to create duplicate recordings (e.g. multiple EPG sources + # refreshing simultaneously each firing evaluate_series_rules.delay()). + # If Redis is unavailable, proceed without lock — the primary and + # secondary dedup guards still prevent duplicates. + lock_acquired = False + try: + lock_acquired = acquire_task_lock('evaluate_series_rules', 'all') + if not lock_acquired: + result["details"].append({"status": "skipped", "reason": "concurrent evaluation in progress"}) + return result + except (ConnectionError, OSError, AttributeError): + logger.warning("Could not acquire series rule evaluation lock (Redis unavailable), proceeding without lock") + + try: + return _evaluate_series_rules_locked(tvg_id, result) + finally: + if lock_acquired: + try: + release_task_lock('evaluate_series_rules', 'all') + except (ConnectionError, OSError, AttributeError): + logger.warning("Could not release series rule evaluation lock") + + +def _evaluate_series_rules_locked(tvg_id, result): + """Inner implementation of series rule evaluation, called under lock.""" from django.utils import timezone from apps.channels.models import Recording, Channel from apps.epg.models import EPGData, ProgramData rules = CoreSettings.get_dvr_series_rules() - result = {"scheduled": 0, "details": []} if not isinstance(rules, list) or not rules: return result @@ -1089,14 +1117,23 @@ def evaluate_series_rules_impl(tvg_id: str | None = None): now = timezone.now() horizon = now + timedelta(days=7) - # Preload existing recordings' program ids to avoid duplicates - existing_program_ids = set() - for rec in Recording.objects.all().only("custom_properties"): + # Preload existing recordings keyed by stable program attributes that + # survive EPG refreshes (tvg_id + original start/end times stored in + # custom_properties). ProgramData.id changes on every EPG refresh so + # it cannot be used for deduplication. Only load future recordings + # to bound the set size — past recordings cannot collide with newly + # scheduled future programs. + existing_program_keys = set() + for cp in Recording.objects.filter( + end_time__gte=now, + ).values_list("custom_properties", flat=True): try: - pid = rec.custom_properties.get("program", {}).get("id") if rec.custom_properties else None - if pid is not None: - # Normalize to string for consistent comparisons - existing_program_ids.add(str(pid)) + prog_data = (cp or {}).get("program", {}) + tvg_id_val = prog_data.get("tvg_id") + st = prog_data.get("start_time") + et = prog_data.get("end_time") + if tvg_id_val and st and et: + existing_program_keys.add((str(tvg_id_val), str(st), str(et))) except Exception: continue @@ -1191,17 +1228,21 @@ def evaluate_series_rules_impl(tvg_id: str | None = None): created_here = 0 for prog in unique_programs: try: - # Skip if already scheduled by program id - if str(prog.id) in existing_program_ids: + # Skip if a recording already exists for this exact airing + # (keyed by tvg_id + original program times, which are stable + # across EPG refreshes unlike ProgramData.id). + prog_key = (str(prog.tvg_id), prog.start_time.isoformat(), prog.end_time.isoformat()) + if prog_key in existing_program_keys: continue - # Extra guard: skip if a recording exists for the same channel + timeslot + # Extra guard: DB query using the same stable attributes + # stored in custom_properties (unadjusted program times, + # not offset-adjusted Recording.start_time/end_time). try: - from django.db.models import Q if Recording.objects.filter( - channel=channel, - start_time=prog.start_time, - end_time=prog.end_time, - ).filter(Q(custom_properties__program__id=prog.id) | Q(custom_properties__program__title=prog.title)).exists(): + custom_properties__program__tvg_id=prog.tvg_id, + custom_properties__program__start_time=prog.start_time.isoformat(), + custom_properties__program__end_time=prog.end_time.isoformat(), + ).exists(): continue except Exception: continue # already scheduled/recorded @@ -1245,7 +1286,7 @@ def evaluate_series_rules_impl(tvg_id: str | None = None): } }, ) - existing_program_ids.add(str(prog.id)) + existing_program_keys.add(prog_key) created_here += 1 try: prefetch_recording_artwork.apply_async(args=[rec.id], countdown=1) @@ -2452,15 +2493,12 @@ def run_recording(recording_id, channel_id, start_time_str, end_time_str): metadata_key = RedisKeys.channel_metadata(str(channel.uuid)) md = r.hgetall(metadata_key) if md: - def _gv(bkey): - return md.get(bkey.encode('utf-8')) - def _d(bkey, cast=str): - v = _gv(bkey) + v = md.get(bkey) try: if v is None: return None - s = v.decode('utf-8') + s = v return cast(s) if cast is not str else s except Exception: return None diff --git a/apps/channels/tests/test_series_rule_dedup.py b/apps/channels/tests/test_series_rule_dedup.py new file mode 100644 index 00000000..33aff695 --- /dev/null +++ b/apps/channels/tests/test_series_rule_dedup.py @@ -0,0 +1,718 @@ +"""Tests for series rule evaluation deduplication. + +Unit tests verify the dedup logic in evaluate_series_rules_impl. +Integration tests exercise the full path: EPG refresh → series rule +evaluation → Recording creation → post_save signal chain. +""" +from datetime import timedelta +from unittest.mock import patch, MagicMock + +from django.test import TestCase +from django.utils import timezone + +from apps.channels.models import Channel, Recording +from apps.epg.models import EPGSource, EPGData, ProgramData +from core.models import CoreSettings + + +def _set_series_rules(rules): + """Helper to store series rules in CoreSettings.""" + CoreSettings.set_dvr_series_rules(rules) + + +def _set_dvr_offsets(pre_min=0, post_min=0): + """Helper to store DVR pre/post offsets.""" + CoreSettings._update_group("dvr_settings", "DVR Settings", { + "pre_offset_minutes": pre_min, + "post_offset_minutes": post_min, + }) + + +class SeriesRuleDedupBaseTestCase(TestCase): + """Shared setup for series rule dedup tests.""" + + def setUp(self): + self.now = timezone.now() + self.epg_source = EPGSource.objects.create( + name="Test EPG", source_type="xmltv" + ) + self.epg = EPGData.objects.create( + tvg_id="test.channel.1", + name="Test Channel EPG", + epg_source=self.epg_source, + ) + self.channel = Channel.objects.create( + channel_number=1, name="Test Channel", epg_data=self.epg + ) + + _set_series_rules([{ + "tvg_id": "test.channel.1", + "mode": "all", + "title": "Test Show", + }]) + _set_dvr_offsets(pre_min=0, post_min=0) + + def _create_program(self, hours_from_now=1, title="Test Show", + sub_title="Episode 1", tvg_id="test.channel.1"): + """Create a ProgramData at the given offset.""" + start = self.now + timedelta(hours=hours_from_now) + end = start + timedelta(hours=1) + return ProgramData.objects.create( + epg=self.epg, + tvg_id=tvg_id, + start_time=start, + end_time=end, + title=title, + sub_title=sub_title, + ) + + def _simulate_epg_refresh(self, programs_data): + """Delete all ProgramData and recreate with new IDs (simulates EPG refresh).""" + ProgramData.objects.filter(epg=self.epg).delete() + new_programs = [] + for data in programs_data: + prog = ProgramData.objects.create(epg=self.epg, **data) + new_programs.append(prog) + return new_programs + + def _program_data_for_refresh(self, prog): + """Build the dict needed by _simulate_epg_refresh from a ProgramData.""" + return { + "tvg_id": prog.tvg_id, + "start_time": prog.start_time, + "end_time": prog.end_time, + "title": prog.title, + "sub_title": prog.sub_title, + } + + +# --------------------------------------------------------------------------- +# Unit tests: dedup logic in evaluate_series_rules_impl +# --------------------------------------------------------------------------- + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class ProgramIdStabilityTests(SeriesRuleDedupBaseTestCase): + """Verify dedup works after EPG refresh changes ProgramData IDs.""" + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_no_duplicate_after_epg_refresh(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Same program should not be recorded twice after EPG refresh.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2) + old_id = prog.id + result1 = evaluate_series_rules_impl() + self.assertEqual(result1["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 1) + + new_programs = self._simulate_epg_refresh( + [self._program_data_for_refresh(prog)] + ) + self.assertNotEqual(old_id, new_programs[0].id) + + result2 = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + self.assertEqual(result2["scheduled"], 0) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_no_duplicate_with_offsets_after_refresh(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Dedup works when DVR offsets shift Recording times away from program times.""" + from apps.channels.tasks import evaluate_series_rules_impl + + _set_dvr_offsets(pre_min=5, post_min=5) + prog = self._create_program(hours_from_now=2) + result1 = evaluate_series_rules_impl() + self.assertEqual(result1["scheduled"], 1) + + rec = Recording.objects.first() + self.assertEqual(rec.start_time, prog.start_time - timedelta(minutes=5)) + self.assertEqual(rec.end_time, prog.end_time + timedelta(minutes=5)) + + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + result2 = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_different_episodes_still_recorded(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Different episodes on the same channel should each get a recording.""" + from apps.channels.tasks import evaluate_series_rules_impl + + self._create_program(hours_from_now=2, sub_title="Episode 1") + self._create_program(hours_from_now=4, sub_title="Episode 2") + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 2) + self.assertEqual(Recording.objects.count(), 2) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_new_episode_after_refresh_is_recorded(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """A genuinely new episode appearing after EPG refresh should be recorded.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2, sub_title="Episode 1") + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + self._simulate_epg_refresh([ + self._program_data_for_refresh(prog), + { + "tvg_id": "test.channel.1", + "start_time": prog.end_time, + "end_time": prog.end_time + timedelta(hours=1), + "title": "Test Show", + "sub_title": "Episode 2", + }, + ]) + + result2 = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 2) + self.assertEqual(result2["scheduled"], 1) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_multiple_epg_refreshes_no_duplicates(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Multiple consecutive EPG refreshes should not accumulate duplicates.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + for _ in range(5): + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + evaluate_series_rules_impl() + + self.assertEqual(Recording.objects.count(), 1) + + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class ConcurrencyGuardTests(SeriesRuleDedupBaseTestCase): + """Verify the task lock prevents concurrent evaluation.""" + + def test_lock_acquired_and_released(self, mock_schedule, mock_artwork): + """evaluate_series_rules_impl acquires and releases the task lock.""" + from apps.channels.tasks import evaluate_series_rules_impl + + self._create_program(hours_from_now=2) + + with patch("apps.channels.tasks.acquire_task_lock", return_value=True) as mock_lock, \ + patch("apps.channels.tasks.release_task_lock") as mock_release: + evaluate_series_rules_impl() + mock_lock.assert_called_once_with('evaluate_series_rules', 'all') + mock_release.assert_called_once_with('evaluate_series_rules', 'all') + + def test_skips_when_lock_held(self, mock_schedule, mock_artwork): + """Returns early with skip reason when lock is already held.""" + from apps.channels.tasks import evaluate_series_rules_impl + + self._create_program(hours_from_now=2) + + with patch("apps.channels.tasks.acquire_task_lock", return_value=False): + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 0) + self.assertTrue( + any(d.get("reason") == "concurrent evaluation in progress" + for d in result["details"]), + ) + self.assertEqual(Recording.objects.count(), 0) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_lock_released_on_exception(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Lock is released even if the inner implementation raises.""" + from apps.channels.tasks import evaluate_series_rules_impl + + with patch("apps.channels.tasks._evaluate_series_rules_locked", + side_effect=RuntimeError("test error")): + with self.assertRaises(RuntimeError): + evaluate_series_rules_impl() + mock_release.assert_called_once_with('evaluate_series_rules', 'all') + + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class SecondaryGuardTests(SeriesRuleDedupBaseTestCase): + """Verify the secondary DB guard uses stable program attributes.""" + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_secondary_guard_catches_duplicate_with_offsets(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Secondary guard works with stale program IDs and DVR offsets.""" + from apps.channels.tasks import evaluate_series_rules_impl + + _set_dvr_offsets(pre_min=10, post_min=10) + prog = self._create_program(hours_from_now=2) + + # Pre-existing recording with a stale program ID (from previous EPG refresh) + Recording.objects.create( + channel=self.channel, + start_time=prog.start_time - timedelta(minutes=10), + end_time=prog.end_time + timedelta(minutes=10), + custom_properties={ + "program": { + "id": 99999, + "tvg_id": prog.tvg_id, + "title": prog.title, + "start_time": prog.start_time.isoformat(), + "end_time": prog.end_time.isoformat(), + } + }, + ) + + result = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + self.assertEqual(result["scheduled"], 0) + + +# --------------------------------------------------------------------------- +# Integration tests: full path from EPG refresh through recording creation +# --------------------------------------------------------------------------- + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class IntegrationEPGRefreshTests(SeriesRuleDedupBaseTestCase): + """End-to-end tests simulating the EPG refresh → evaluate → record flow. + + These exercise the full signal chain: evaluate_series_rules_impl creates + a Recording, the post_save signal fires schedule_recording_task, and + subsequent evaluations (after EPG refresh) must not create duplicates. + """ + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_single_episode_no_duplicates(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Simulate: create rule → evaluate → EPG refresh → re-evaluate. + + The full recording lifecycle must result in exactly 1 recording. + """ + from apps.channels.tasks import evaluate_series_rules_impl + + # Initial EPG data + prog = self._create_program(hours_from_now=2, sub_title="Pilot") + + # First evaluation creates the recording + result1 = evaluate_series_rules_impl() + self.assertEqual(result1["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 1) + + # Verify the recording was created with correct program metadata + rec = Recording.objects.first() + self.assertEqual(rec.custom_properties["program"]["tvg_id"], "test.channel.1") + self.assertEqual(rec.custom_properties["program"]["title"], "Test Show") + self.assertEqual( + rec.custom_properties["program"]["start_time"], + prog.start_time.isoformat() + ) + + # Verify the post_save signal scheduled a task + mock_schedule.assert_called() + initial_schedule_count = mock_schedule.call_count + + # Simulate EPG refresh (programs get new DB IDs) + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + + # Re-evaluate after refresh (this is what EPG refresh triggers) + result2 = evaluate_series_rules_impl() + self.assertEqual(result2["scheduled"], 0) + self.assertEqual(Recording.objects.count(), 1) + + # No additional task scheduling should have occurred + self.assertEqual(mock_schedule.call_count, initial_schedule_count) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_with_offsets_no_duplicates(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Full flow with DVR offsets: recording times differ from program times.""" + from apps.channels.tasks import evaluate_series_rules_impl + + _set_dvr_offsets(pre_min=5, post_min=10) + prog = self._create_program(hours_from_now=3, sub_title="Episode 1") + + result1 = evaluate_series_rules_impl() + self.assertEqual(result1["scheduled"], 1) + + rec = Recording.objects.first() + # Verify offset-adjusted recording times + self.assertEqual(rec.start_time, prog.start_time - timedelta(minutes=5)) + self.assertEqual(rec.end_time, prog.end_time + timedelta(minutes=10)) + # Verify original (unadjusted) program times in custom_properties + self.assertEqual( + rec.custom_properties["program"]["start_time"], + prog.start_time.isoformat() + ) + self.assertEqual( + rec.custom_properties["program"]["end_time"], + prog.end_time.isoformat() + ) + + # EPG refresh + re-evaluate + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + result2 = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + self.assertEqual(result2["scheduled"], 0) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_multiple_episodes_across_refreshes(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """New episodes appear across multiple EPG refreshes; each recorded once.""" + from apps.channels.tasks import evaluate_series_rules_impl + + ep1 = self._create_program(hours_from_now=2, sub_title="Episode 1") + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + # EPG refresh adds episode 2 alongside episode 1 + ep1_data = self._program_data_for_refresh(ep1) + ep2_start = ep1.end_time + ep2_data = { + "tvg_id": "test.channel.1", + "start_time": ep2_start, + "end_time": ep2_start + timedelta(hours=1), + "title": "Test Show", + "sub_title": "Episode 2", + } + self._simulate_epg_refresh([ep1_data, ep2_data]) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 2) + + # Another EPG refresh adds episode 3 + ep3_start = ep2_start + timedelta(hours=1) + ep3_data = { + "tvg_id": "test.channel.1", + "start_time": ep3_start, + "end_time": ep3_start + timedelta(hours=1), + "title": "Test Show", + "sub_title": "Episode 3", + } + self._simulate_epg_refresh([ep1_data, ep2_data, ep3_data]) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 3) + + # Final EPG refresh with no new episodes — count must stay at 3 + self._simulate_epg_refresh([ep1_data, ep2_data, ep3_data]) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 3) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_multiple_series_rules(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Multiple series rules on different channels, each evaluated correctly.""" + from apps.channels.tasks import evaluate_series_rules_impl + + # Second channel with its own EPG + epg2 = EPGData.objects.create( + tvg_id="test.channel.2", + name="Channel 2 EPG", + epg_source=self.epg_source, + ) + channel2 = Channel.objects.create( + channel_number=2, name="Test Channel 2", epg_data=epg2 + ) + + _set_series_rules([ + {"tvg_id": "test.channel.1", "mode": "all", "title": "Show A"}, + {"tvg_id": "test.channel.2", "mode": "all", "title": "Show B"}, + ]) + + # Programs on both channels + start1 = self.now + timedelta(hours=2) + prog1 = ProgramData.objects.create( + epg=self.epg, tvg_id="test.channel.1", + start_time=start1, end_time=start1 + timedelta(hours=1), + title="Show A", sub_title="Episode 1", + ) + start2 = self.now + timedelta(hours=3) + prog2 = ProgramData.objects.create( + epg=epg2, tvg_id="test.channel.2", + start_time=start2, end_time=start2 + timedelta(hours=1), + title="Show B", sub_title="Episode 1", + ) + + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 2) + self.assertEqual(Recording.objects.filter(channel=self.channel).count(), 1) + self.assertEqual(Recording.objects.filter(channel=channel2).count(), 1) + + # EPG refresh for both channels + ProgramData.objects.filter(epg=self.epg).delete() + ProgramData.objects.filter(epg=epg2).delete() + ProgramData.objects.create( + epg=self.epg, tvg_id="test.channel.1", + start_time=start1, end_time=start1 + timedelta(hours=1), + title="Show A", sub_title="Episode 1", + ) + ProgramData.objects.create( + epg=epg2, tvg_id="test.channel.2", + start_time=start2, end_time=start2 + timedelta(hours=1), + title="Show B", sub_title="Episode 1", + ) + + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 2, + "No duplicates across multiple series rules after EPG refresh") + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_rapid_epg_refreshes_simulate_user_report( + self, mock_release, mock_lock, mock_schedule, mock_artwork + ): + """Reproduce the user-reported scenario: series rule + multiple EPG refreshes + causing count to balloon from 6 to 25 and 5 simultaneous recordings. + + Simulates 6 episodes with 5 EPG refreshes (each assigning new ProgramData IDs). + """ + from apps.channels.tasks import evaluate_series_rules_impl + + # Create 6 episodes (the user had "next of 6") + episodes = [] + for i in range(6): + start = self.now + timedelta(hours=2 + i * 2) + episodes.append({ + "tvg_id": "test.channel.1", + "start_time": start, + "end_time": start + timedelta(hours=1), + "title": "Test Show", + "sub_title": f"Episode {i + 1}", + }) + + # Create initial ProgramData + for ep in episodes: + ProgramData.objects.create(epg=self.epg, **ep) + + # First evaluation: should create exactly 6 recordings + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 6) + + # Simulate 5 EPG refreshes (the user saw count balloon to 25) + for refresh_num in range(5): + self._simulate_epg_refresh(episodes) + result = evaluate_series_rules_impl() + self.assertEqual( + Recording.objects.count(), 6, + f"After EPG refresh #{refresh_num + 1}, expected 6 recordings " + f"but got {Recording.objects.count()}" + ) + self.assertEqual(result["scheduled"], 0) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_recording_survives_program_removal_and_readd( + self, mock_release, mock_lock, mock_schedule, mock_artwork + ): + """Program temporarily disappears from EPG then reappears — no duplicate.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2, sub_title="Episode 1") + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + # EPG refresh removes the program entirely + self._simulate_epg_refresh([]) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1, + "Existing recording preserved when program disappears from EPG") + + # EPG refresh adds the program back (new ID) + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1, + "No duplicate when program reappears with new ID") + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_celery_task_wrapper_calls_impl(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """The @shared_task evaluate_series_rules delegates to _impl correctly.""" + from apps.channels.tasks import evaluate_series_rules + + self._create_program(hours_from_now=2) + result = evaluate_series_rules() + self.assertEqual(result["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 1) + + # Call again (simulating a second EPG refresh trigger) + result2 = evaluate_series_rules() + self.assertEqual(result2["scheduled"], 0) + self.assertEqual(Recording.objects.count(), 1) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_tvg_id_scoped_evaluation(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Scoped evaluation (tvg_id parameter) still prevents duplicates.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2) + result1 = evaluate_series_rules_impl(tvg_id="test.channel.1") + self.assertEqual(result1["scheduled"], 1) + + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + result2 = evaluate_series_rules_impl(tvg_id="test.channel.1") + self.assertEqual(result2["scheduled"], 0) + self.assertEqual(Recording.objects.count(), 1) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_offset_change_between_refreshes(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Changing DVR offsets between EPG refreshes doesn't create duplicates. + + Even though Recording.start_time/end_time change when offsets change, + the dedup key uses the original program times from custom_properties. + """ + from apps.channels.tasks import evaluate_series_rules_impl + + _set_dvr_offsets(pre_min=5, post_min=5) + prog = self._create_program(hours_from_now=2) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + rec = Recording.objects.first() + original_start = rec.start_time + original_end = rec.end_time + + # Change offsets + _set_dvr_offsets(pre_min=10, post_min=15) + + # EPG refresh + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + result = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1, + "Changing offsets between refreshes should not create duplicates") + self.assertEqual(result["scheduled"], 0) + + +# --------------------------------------------------------------------------- +# Edge case tests: Redis unavailability, non-series recordings, robustness +# --------------------------------------------------------------------------- + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class RedisUnavailabilityTests(SeriesRuleDedupBaseTestCase): + """Verify evaluation works when Redis is unavailable (lock cannot be acquired).""" + + def test_proceeds_when_redis_down(self, mock_schedule, mock_artwork): + """Evaluation succeeds (with dedup guards) when Redis raises on lock acquire.""" + from apps.channels.tasks import evaluate_series_rules_impl + + self._create_program(hours_from_now=2) + + with patch("apps.channels.tasks.acquire_task_lock", + side_effect=ConnectionError("Redis unavailable")): + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 1) + + def test_dedup_still_works_without_lock(self, mock_schedule, mock_artwork): + """Dedup guards prevent duplicates even when the lock is unavailable.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2) + + # First call: Redis down, proceeds without lock + with patch("apps.channels.tasks.acquire_task_lock", + side_effect=ConnectionError("Redis unavailable")): + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + # EPG refresh + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + + # Second call: Redis still down + with patch("apps.channels.tasks.acquire_task_lock", + side_effect=ConnectionError("Redis unavailable")): + result = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1, + "Dedup guards prevent duplicates even without lock") + self.assertEqual(result["scheduled"], 0) + + def test_lock_not_released_when_not_acquired(self, mock_schedule, mock_artwork): + """release_task_lock is not called if acquire raised an exception.""" + from apps.channels.tasks import evaluate_series_rules_impl + + self._create_program(hours_from_now=2) + + with patch("apps.channels.tasks.acquire_task_lock", + side_effect=ConnectionError("Redis unavailable")), \ + patch("apps.channels.tasks.release_task_lock") as mock_release: + evaluate_series_rules_impl() + mock_release.assert_not_called() + + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class NonSeriesRecordingTests(SeriesRuleDedupBaseTestCase): + """Verify non-series recordings don't interfere with series rule dedup.""" + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_manual_recording_without_program_data_ignored(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Recordings without custom_properties.program are skipped by dedup key builder.""" + from apps.channels.tasks import evaluate_series_rules_impl + + # Manual recording with no program metadata + Recording.objects.create( + channel=self.channel, + start_time=self.now + timedelta(hours=2), + end_time=self.now + timedelta(hours=3), + custom_properties={}, + ) + + prog = self._create_program(hours_from_now=2) + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 2) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_recurring_rule_recording_does_not_interfere(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Recordings from recurring rules (custom_properties.rule) don't block series rules.""" + from apps.channels.tasks import evaluate_series_rules_impl + + Recording.objects.create( + channel=self.channel, + start_time=self.now + timedelta(hours=2), + end_time=self.now + timedelta(hours=3), + custom_properties={"rule": {"id": 1, "name": "Daily News"}}, + ) + + prog = self._create_program(hours_from_now=2) + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 2) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_recording_with_null_custom_properties_ignored(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Recordings with None custom_properties don't crash the dedup key builder.""" + from apps.channels.tasks import evaluate_series_rules_impl + + Recording.objects.create( + channel=self.channel, + start_time=self.now + timedelta(hours=2), + end_time=self.now + timedelta(hours=3), + custom_properties=None, + ) + + prog = self._create_program(hours_from_now=2) + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 1) diff --git a/apps/connect/api_views.py b/apps/connect/api_views.py index 739e8e32..fdb5d43a 100644 --- a/apps/connect/api_views.py +++ b/apps/connect/api_views.py @@ -1,9 +1,10 @@ -from rest_framework import viewsets, status +from rest_framework import viewsets, status, serializers from rest_framework.pagination import PageNumberPagination from django_filters.rest_framework import DjangoFilterBackend from rest_framework.response import Response from rest_framework.decorators import action from django.utils import timezone +from drf_spectacular.utils import extend_schema, inline_serializer from .models import Integration, EventSubscription, DeliveryLog from .serializers import ( IntegrationSerializer, @@ -37,6 +38,33 @@ class IntegrationViewSet(viewsets.ModelViewSet): serializer = EventSubscriptionSerializer(qs, many=True) return Response(serializer.data) + @extend_schema( + methods=["PUT"], + description=( + "Replace the integration's event subscriptions with the provided list. " + "Accepts a JSON array of subscription objects. " + "Existing subscriptions not in the list will be deleted. " + "The 'payload_template' field is only relevant for webhook integrations." + ), + request=inline_serializer( + name="SetSubscriptionsRequest", + fields={ + "event": serializers.CharField(help_text="Event name (e.g. 'channel_start')."), + "enabled": serializers.BooleanField(required=False, default=True), + "payload_template": serializers.CharField(required=False, allow_blank=True, allow_null=True, help_text="Custom payload template (webhook integrations only)."), + }, + many=True, + ), + responses={200: inline_serializer( + name="SetSubscriptionsResponse", + fields={ + "event": serializers.CharField(), + "enabled": serializers.BooleanField(), + "payload_template": serializers.CharField(allow_null=True), + }, + many=True, + )}, + ) @action(detail=True, methods=["put"], url_path=r"subscriptions/set") def set_subscriptions(self, request, pk=None): """ diff --git a/apps/epg/api_views.py b/apps/epg/api_views.py index 9832b5c1..33483e60 100644 --- a/apps/epg/api_views.py +++ b/apps/epg/api_views.py @@ -175,7 +175,7 @@ class EPGGridAPIView(APIView): # Get channels with custom dummy EPG sources (generate on-demand with patterns) channels_with_custom_dummy = Channel.objects.filter( epg_data__epg_source__source_type='dummy' - ).distinct() + ).select_related('epg_data__epg_source').distinct() # Log what we found without_count = channels_without_epg.count() @@ -427,7 +427,13 @@ class EPGImportAPIView(APIView): return [Authenticated()] @extend_schema( - description="Triggers an EPG data import", + description="Triggers an EPG data refresh for the given source.", + request=inline_serializer( + name="EPGImportRequest", + fields={ + "id": serializers.IntegerField(help_text="ID of the EPG source to refresh."), + }, + ), ) def post(self, request, format=None): logger.info("EPGImportAPIView: Received request to import EPG data.") @@ -449,7 +455,7 @@ class EPGImportAPIView(APIView): refresh_epg_data.delay(epg_id) # Trigger Celery task logger.info("EPGImportAPIView: Task dispatched to refresh EPG data.") return Response( - {"success": True, "message": "EPG data import initiated."}, + {"success": True, "message": "EPG data refresh initiated."}, status=status.HTTP_202_ACCEPTED, ) diff --git a/apps/epg/tasks.py b/apps/epg/tasks.py index f4c85256..7700ad1b 100644 --- a/apps/epg/tasks.py +++ b/apps/epg/tasks.py @@ -2,6 +2,7 @@ import logging import gzip +import html.entities import os import uuid import requests @@ -15,7 +16,7 @@ import zipfile from celery import shared_task from django.conf import settings -from django.db import transaction +from django.db import connection, transaction from django.utils import timezone from apps.channels.models import Channel from core.models import UserAgent, CoreSettings @@ -28,6 +29,103 @@ from core.utils import acquire_task_lock, release_task_lock, TaskLockRenewer, se logger = logging.getLogger(__name__) +# DOCTYPE internal subset for XMLTV files. Declares all 252 HTML 4 named +# entities so lxml/libxml2 can resolve references like é correctly +# instead of silently dropping them in recovery mode. +# The 5 XML-predefined entities (amp, lt, gt, quot, apos) are always +# recognised by the XML spec and must not be redeclared. +_XML_ENTITIES = frozenset({'amp', 'lt', 'gt', 'quot', 'apos'}) + + +def _build_html_entity_doctype() -> bytes: + """Build a DOCTYPE internal subset declaring all HTML 4 named entities.""" + lines = [b'\n'.encode('ascii')) + lines.append(b']>\n') + return b''.join(lines) + + +_HTML_ENTITY_DOCTYPE = _build_html_entity_doctype() + + +class _PrependStream: + """Wraps an open binary file and prepends a bytes prefix to its content. + + Used by _open_xmltv_file to inject a DOCTYPE entity block before the + file content reaches lxml's iterparse, with zero disk I/O. + """ + + __slots__ = ('_prefix', '_prefix_pos', '_file') + + def __init__(self, prefix: bytes, file_obj): + self._prefix = prefix + self._prefix_pos = 0 + self._file = file_obj + + def read(self, size=-1): + prefix_len = len(self._prefix) + if self._prefix_pos >= prefix_len: + return self._file.read(size) + remaining = prefix_len - self._prefix_pos + if size < 0: + chunk = self._prefix[self._prefix_pos:] + self._file.read() + self._prefix_pos = prefix_len + return chunk + if size <= remaining: + chunk = self._prefix[self._prefix_pos:self._prefix_pos + size] + self._prefix_pos += size + return chunk + chunk = self._prefix[self._prefix_pos:] + self._prefix_pos = prefix_len + return chunk + self._file.read(size - remaining) + + def close(self): + self._file.close() + + def __enter__(self): + return self + + def __exit__(self, *_): + self.close() + + +def _open_xmltv_file(file_path: str): + """Open an XMLTV file for lxml iterparse, injecting an HTML entity DOCTYPE. + + Prepends a block that declares all 252 HTML 4 named + entities so lxml/libxml2 resolves references like é correctly + instead of silently dropping them in recovery mode. This involves zero + disk I/O — the DOCTYPE is streamed in-memory before the file content. + + If the file already contains a declaration the file is returned + unchanged; a second DOCTYPE would be invalid XML. + + The caller is responsible for closing the returned object. + """ + f = open(file_path, 'rb') + start = f.read(512) + + # Do not inject if the file already declares a DOCTYPE. + if b'= 0: + decl_end = start.find(b'?>', xml_pos) + if decl_end >= 0: + xml_decl = start[:decl_end + 2] + f.seek(decl_end + 2) + return _PrependStream(xml_decl + b'\n' + _HTML_ENTITY_DOCTYPE, f) + + # No XML declaration — insert DOCTYPE at the very start of the file. + f.seek(0) + return _PrependStream(_HTML_ENTITY_DOCTYPE, f) + def validate_icon_url_fast(icon_url, max_length=None): """ @@ -146,7 +244,7 @@ def refresh_all_epg_data(): return "EPG data refreshed." -@shared_task(time_limit=1800, soft_time_limit=1700) +@shared_task(time_limit=14400) def refresh_epg_data(source_id): if not acquire_task_lock('refresh_epg_data', source_id): logger.debug(f"EPG refresh for {source_id} already running") @@ -397,42 +495,41 @@ def fetch_xmltv(source): # Download to temporary file with open(temp_download_path, 'wb') as f: - for chunk in response.iter_content(chunk_size=16384): # Increased chunk size for better performance - if chunk: - f.write(chunk) + for chunk in response.iter_content(chunk_size=16384): + f.write(chunk) - downloaded += len(chunk) - elapsed_time = time.time() - start_time + downloaded += len(chunk) + elapsed_time = time.time() - start_time - # Calculate download speed in KB/s - speed = downloaded / elapsed_time / 1024 if elapsed_time > 0 else 0 + # Calculate download speed in KB/s + speed = downloaded / elapsed_time / 1024 if elapsed_time > 0 else 0 - # Calculate progress percentage - if total_size and total_size > 0: - progress = min(100, int((downloaded / total_size) * 100)) - else: - # If no content length header, estimate progress - progress = min(95, int((downloaded / (10 * 1024 * 1024)) * 100)) # Assume 10MB if unknown + # Calculate progress percentage + if total_size and total_size > 0: + progress = min(100, int((downloaded / total_size) * 100)) + else: + # If no content length header, estimate progress + progress = min(95, int((downloaded / (10 * 1024 * 1024)) * 100)) # Assume 10MB if unknown - # Time remaining (in seconds) - time_remaining = (total_size - downloaded) / (speed * 1024) if speed > 0 and total_size > 0 else 0 + # Time remaining (in seconds) + time_remaining = (total_size - downloaded) / (speed * 1024) if speed > 0 and total_size > 0 else 0 - # Only send updates at specified intervals to avoid flooding - current_time = time.time() - if current_time - last_update_time >= update_interval and progress > 0: - last_update_time = current_time - send_epg_update( - source.id, - "downloading", - progress, - speed=round(speed, 2), - elapsed_time=round(elapsed_time, 1), - time_remaining=round(time_remaining, 1), - downloaded=f"{downloaded / (1024 * 1024):.2f} MB" - ) + # Only send updates at specified intervals to avoid flooding + current_time = time.time() + if current_time - last_update_time >= update_interval and progress > 0: + last_update_time = current_time + send_epg_update( + source.id, + "downloading", + progress, + speed=round(speed, 2), + elapsed_time=round(elapsed_time, 1), + time_remaining=round(time_remaining, 1), + downloaded=f"{downloaded / (1024 * 1024):.2f} MB" + ) - # Explicitly delete the chunk to free memory immediately - del chunk + # Explicitly delete the chunk to free memory immediately + del chunk # Send completion notification send_epg_update(source.id, "downloading", 100) @@ -526,6 +623,7 @@ def fetch_xmltv(source): source.save(update_fields=['status']) logger.info(f"Cached EPG file saved to {source.file_path}") + return True except requests.exceptions.HTTPError as e: @@ -840,7 +938,8 @@ def parse_channels_only(source): # Replace full dictionary load with more efficient lookup set existing_tvg_ids = set() - existing_epgs = {} # Initialize the dictionary that will lazily load objects + existing_epgs = {} + scanned_tvg_ids = set() # Track tvg_ids seen in the current scan for stale cleanup last_id = 0 chunk_size = 5000 @@ -888,7 +987,7 @@ def parse_channels_only(source): # Open the file - no need to check file type since it's always XML now logger.debug(f"Opening file for channel parsing: {file_path}") - source_file = open(file_path, 'rb') + source_file = _open_xmltv_file(file_path) if process: logger.debug(f"[parse_channels_only] Memory after opening file: {process.memory_info().rss / 1024 / 1024:.2f} MB") @@ -908,6 +1007,7 @@ def parse_channels_only(source): channel_count += 1 tvg_id = elem.get('id', '').strip() if tvg_id: + scanned_tvg_ids.add(tvg_id) display_name = None icon_url = None for child in elem: @@ -1055,6 +1155,16 @@ def parse_channels_only(source): if epgs_to_update: EPGData.objects.bulk_update(epgs_to_update, ["name", "icon_url"]) logger.debug(f"[parse_channels_only] Updated final batch of {len(epgs_to_update)} EPG entries") + + # Clean up stale EPGData: entries that existed before the scan but weren't seen, and aren't mapped to any channel. + # Use existing_tvg_ids - scanned_tvg_ids to avoid a full-table scan with a large EXCLUDE list. + potentially_stale = existing_tvg_ids - scanned_tvg_ids + if potentially_stale: + stale_qs = EPGData.objects.filter(epg_source=source, tvg_id__in=potentially_stale, channels__isnull=True) + deleted_count, _ = stale_qs.delete() + if deleted_count: + logger.info(f"[parse_channels_only] Cleaned up {deleted_count} stale EPG entries not in current scan and unmapped to any channel") + if process: logger.debug(f"[parse_channels_only] Memory after final batch creation: {process.memory_info().rss / 1024 / 1024:.2f} MB") @@ -1121,6 +1231,9 @@ def parse_channels_only(source): existing_epgs = None epgs_to_create = None epgs_to_update = None + if 'scanned_tvg_ids' in locals() and scanned_tvg_ids is not None: + scanned_tvg_ids.clear() + scanned_tvg_ids = None cleanup_memory(log_usage=should_log_memory, force_collection=True) except Exception as e: logger.warning(f"Cleanup error: {e}") @@ -1271,7 +1384,7 @@ def parse_programs_for_tvg_id(epg_id): try: # Open the file directly - no need to check compression logger.debug(f"Opening file for parsing: {file_path}") - source_file = open(file_path, 'rb') + source_file = _open_xmltv_file(file_path) # Stream parse the file using lxml's iterparse program_parser = etree.iterparse(source_file, events=('end',), tag='programme', remove_blank_text=True, recover=True) @@ -1544,7 +1657,7 @@ def parse_programs_for_source(epg_source, tvg_id=None): try: logger.debug(f"Opening file for single-pass parsing: {file_path}") - source_file = open(file_path, 'rb') + source_file = _open_xmltv_file(file_path) # Stream parse the file using lxml's iterparse program_parser = etree.iterparse(source_file, events=('end',), tag='programme', remove_blank_text=True, recover=True) @@ -1657,6 +1770,10 @@ def parse_programs_for_source(epg_source, tvg_id=None): batch_size = 1000 try: with transaction.atomic(): + # Kill any individual statement that hangs longer than 10 minutes. + # SET LOCAL automatically resets when this transaction ends (commit or rollback). + with connection.cursor() as cursor: + cursor.execute("SET LOCAL statement_timeout = '10min'") # Delete existing programs for mapped EPGs deleted_count = ProgramData.objects.filter(epg_id__in=mapped_epg_ids).delete()[0] logger.debug(f"Deleted {deleted_count} existing programs") diff --git a/apps/epg/tests/test_entity_resolution.py b/apps/epg/tests/test_entity_resolution.py new file mode 100644 index 00000000..353d9b51 --- /dev/null +++ b/apps/epg/tests/test_entity_resolution.py @@ -0,0 +1,195 @@ +import os +import tempfile + +from django.test import TestCase + +from apps.epg.tasks import ( + _NAMED_ENTITY_RE, + _detect_xml_encoding, + _replace_html_entity, + _resolve_html_entities, +) + + +class ReplaceHtmlEntityTests(TestCase): + """Tests for the regex callback that resolves individual HTML entities.""" + + def _sub(self, text): + return _NAMED_ENTITY_RE.sub(_replace_html_entity, text) + + def test_french_accented(self): + self.assertEqual(self._sub("Chaîne Télé"), "Chaîne Télé") + + def test_german_umlauts(self): + self.assertEqual(self._sub("München Übersicht ß"), "München Übersicht ß") + + def test_spanish(self): + self.assertEqual(self._sub("España ¿Qué?"), "España ¿Qué?") + + def test_portuguese(self): + self.assertEqual(self._sub("Comunicação"), "Comunicação") + + def test_scandinavian(self): + self.assertEqual(self._sub("Norsk ø å æ"), "Norsk ø å æ") + + def test_greek_letters(self): + self.assertEqual(self._sub("αβγ"), "αβγ") + + def test_currency_and_symbols(self): + self.assertEqual(self._sub("© € £ ¥"), "© € £ ¥") + + def test_preserves_xml_amp(self): + self.assertEqual(self._sub("A & B"), "A & B") + + def test_preserves_xml_lt_gt(self): + self.assertEqual(self._sub("<tag>"), "<tag>") + + def test_preserves_xml_quot_apos(self): + self.assertEqual(self._sub(""hello'"), ""hello'") + + def test_preserves_uppercase_xml_entities(self): + """&, <, >, " resolve to XML-special chars; must not be replaced.""" + self.assertEqual(self._sub("&"), "&") + self.assertEqual(self._sub("<"), "<") + self.assertEqual(self._sub(">"), ">") + self.assertEqual(self._sub("""), """) + + def test_partial_entity_match_preserved(self): + """html.unescape can partially match & inside &ersand; — must not corrupt.""" + self.assertEqual(self._sub("&ersand;"), "&ersand;") + + def test_mixed_html_and_xml_entities(self): + self.assertEqual( + self._sub("Résumé & Co <test>"), + "Résumé & Co <test>", + ) + + def test_plain_ascii_unchanged(self): + self.assertEqual(self._sub("Plain ASCII text"), "Plain ASCII text") + + def test_direct_utf8_unchanged(self): + self.assertEqual(self._sub("日本語テレビ"), "日本語テレビ") + + def test_unknown_entity_preserved(self): + self.assertEqual(self._sub("&zzfakeentity;"), "&zzfakeentity;") + + +class ResolveHtmlEntitiesFileTests(TestCase): + """Tests for the file-level preprocessing function.""" + + def _make_file(self, content): + fd, path = tempfile.mkstemp(suffix=".xml") + with os.fdopen(fd, "w", encoding="utf-8") as f: + f.write(content) + self.addCleanup(lambda: os.unlink(path) if os.path.exists(path) else None) + return path + + def test_resolves_entities_in_file(self): + path = self._make_file( + '\nTélé' + ) + _resolve_html_entities(path) + with open(path, "r", encoding="utf-8") as f: + content = f.read() + self.assertIn("Télé", content) + self.assertNotIn("é", content) + + def test_preserves_xml_entities_in_file(self): + path = self._make_file("A & B <C>") + _resolve_html_entities(path) + with open(path, "r", encoding="utf-8") as f: + content = f.read() + self.assertIn("&", content) + self.assertIn("<", content) + self.assertIn(">", content) + + def test_no_temp_file_left_on_success(self): + path = self._make_file("test") + _resolve_html_entities(path) + self.assertFalse(os.path.exists(path + ".entity_tmp")) + + def test_plain_file_unchanged(self): + original = '\nPlain' + path = self._make_file(original) + _resolve_html_entities(path) + with open(path, "r", encoding="utf-8") as f: + content = f.read() + self.assertEqual(content, original) + + def test_utf8_content_preserved(self): + original = "日本語テレビ" + path = self._make_file(original) + _resolve_html_entities(path) + with open(path, "r", encoding="utf-8") as f: + content = f.read() + self.assertIn("日本語テレビ", content) + + def test_iso_8859_1_encoding(self): + """Files declaring ISO-8859-1 should be read in that encoding.""" + xml = '\nChaîne' + fd, path = tempfile.mkstemp(suffix=".xml") + with os.fdopen(fd, "wb") as f: + f.write(xml.encode("iso-8859-1")) + self.addCleanup(lambda: os.unlink(path) if os.path.exists(path) else None) + + _resolve_html_entities(path) + with open(path, "r", encoding="iso-8859-1") as f: + content = f.read() + self.assertIn("Cha\u00eene", content) + self.assertNotIn("î", content) + + def test_detect_encoding_utf8_default(self): + """Headers without an encoding declaration default to UTF-8.""" + self.assertEqual(_detect_xml_encoding(b''), "utf-8") + + def test_detect_encoding_iso_8859_1(self): + """Encoding is read from the XML declaration.""" + self.assertEqual( + _detect_xml_encoding(b''), + "ISO-8859-1", + ) + + def test_detect_encoding_single_quotes(self): + """Encoding detection works with single-quoted attributes.""" + self.assertEqual( + _detect_xml_encoding(b""), + "windows-1252", + ) + + def test_detect_encoding_unknown_falls_back(self): + """Unrecognized encoding falls back to UTF-8.""" + self.assertEqual( + _detect_xml_encoding(b''), + "utf-8", + ) + + def test_iso_8859_1_with_entities_roundtrip(self): + """ISO-8859-1 file with entities: resolved without corrupting existing accented chars.""" + # Mix of direct ISO-8859-1 chars and HTML entities + xml_str = '\nD\xe9j\xe0 émission' + fd, path = tempfile.mkstemp(suffix=".xml") + with os.fdopen(fd, "wb") as f: + f.write(xml_str.encode("iso-8859-1")) + self.addCleanup(lambda: os.unlink(path) if os.path.exists(path) else None) + + _resolve_html_entities(path) + with open(path, "r", encoding="iso-8859-1") as f: + content = f.read() + self.assertIn("D\xe9j\xe0", content, "Existing accented chars should be preserved") + self.assertIn("\xe9mission", content, "Entity should be resolved") + self.assertNotIn("é", content) + + def test_mismatched_encoding_leaves_file_untouched(self): + """File declaring UTF-8 but containing Latin-1 bytes is left alone.""" + # \xe9 is valid ISO-8859-1 but invalid as a standalone UTF-8 byte + raw = b'\n\xe9' + fd, path = tempfile.mkstemp(suffix=".xml") + with os.fdopen(fd, "wb") as f: + f.write(raw) + self.addCleanup(lambda: os.unlink(path) if os.path.exists(path) else None) + + original_bytes = raw # save for comparison + _resolve_html_entities(path) + with open(path, "rb") as f: + result_bytes = f.read() + self.assertEqual(result_bytes, original_bytes, "File should be untouched on decode error") diff --git a/apps/hdhr/api_views.py b/apps/hdhr/api_views.py index 2227170e..539adffe 100644 --- a/apps/hdhr/api_views.py +++ b/apps/hdhr/api_views.py @@ -1,6 +1,7 @@ from rest_framework import viewsets, status from rest_framework.response import Response from rest_framework.views import APIView +from rest_framework.permissions import AllowAny from apps.accounts.permissions import Authenticated, permission_classes_by_action from django.http import JsonResponse, HttpResponseForbidden, HttpResponse import logging @@ -46,6 +47,7 @@ class HDHRDeviceViewSet(viewsets.ModelViewSet): # 🔹 2) Discover API class DiscoverAPIView(APIView): """Returns device discovery information""" + permission_classes = [AllowAny] @extend_schema( description="Retrieve HDHomeRun device discovery information", @@ -98,6 +100,7 @@ class DiscoverAPIView(APIView): # 🔹 3) Lineup API class LineupAPIView(APIView): """Returns available channel lineup""" + permission_classes = [AllowAny] @extend_schema( description="Retrieve the available channel lineup", @@ -138,6 +141,7 @@ class LineupAPIView(APIView): # 🔹 4) Lineup Status API class LineupStatusAPIView(APIView): """Returns the current status of the HDHR lineup""" + permission_classes = [AllowAny] @extend_schema( description="Retrieve the HDHomeRun lineup status", @@ -155,6 +159,7 @@ class LineupStatusAPIView(APIView): # 🔹 5) Device XML API class HDHRDeviceXMLAPIView(APIView): """Returns HDHomeRun device configuration in XML""" + permission_classes = [AllowAny] @extend_schema( description="Retrieve the HDHomeRun device XML configuration", diff --git a/apps/m3u/api_views.py b/apps/m3u/api_views.py index 3b856dc8..b8cb099a 100644 --- a/apps/m3u/api_views.py +++ b/apps/m3u/api_views.py @@ -20,6 +20,7 @@ import json from .models import M3UAccount, M3UFilter, ServerGroup, M3UAccountProfile from core.models import UserAgent +from core.utils import safe_upload_path from apps.channels.models import ChannelGroupM3UAccount from core.serializers import UserAgentSerializer from apps.vod.models import M3UVODCategoryRelation @@ -54,10 +55,12 @@ class M3UAccountViewSet(viewsets.ModelViewSet): file_path = None if "file" in request.FILES: file = request.FILES["file"] - file_name = file.name - file_path = os.path.join("/data/uploads/m3us", file_name) + try: + file_path = safe_upload_path(file.name, "/data/uploads/m3us") + except ValueError: + return Response({"detail": "Invalid filename."}, status=status.HTTP_400_BAD_REQUEST) - os.makedirs(os.path.dirname(file_path), exist_ok=True) + os.makedirs("/data/uploads/m3us", exist_ok=True) with open(file_path, "wb+") as destination: for chunk in file.chunks(): destination.write(chunk) @@ -117,10 +120,12 @@ class M3UAccountViewSet(viewsets.ModelViewSet): file_path = None if "file" in request.FILES: file = request.FILES["file"] - file_name = file.name - file_path = os.path.join("/data/uploads/m3us", file_name) + try: + file_path = safe_upload_path(file.name, "/data/uploads/m3us") + except ValueError: + return Response({"detail": "Invalid filename."}, status=status.HTTP_400_BAD_REQUEST) - os.makedirs(os.path.dirname(file_path), exist_ok=True) + os.makedirs("/data/uploads/m3us", exist_ok=True) with open(file_path, "wb+") as destination: for chunk in file.chunks(): destination.write(chunk) diff --git a/apps/m3u/tasks.py b/apps/m3u/tasks.py index 9c31fa75..8f446b6e 100644 --- a/apps/m3u/tasks.py +++ b/apps/m3u/tasks.py @@ -1,6 +1,7 @@ # apps/m3u/tasks.py import logging import re +import regex import requests import os import gc @@ -38,6 +39,8 @@ logger = logging.getLogger(__name__) BATCH_SIZE = 1500 # Optimized batch size for threading m3u_dir = os.path.join(settings.MEDIA_ROOT, "cached_m3u") +_EXTINF_ATTR_RE = re.compile(r'([^\s=]+)\s*=\s*(["\'])(.*?)\2') + def fetch_m3u_lines(account, use_cache=False): os.makedirs(m3u_dir, exist_ok=True) @@ -484,18 +487,13 @@ def parse_extinf_line(line: str) -> dict: return None content = line[len("#EXTINF:") :].strip() - # Single pass: extract all attributes AND track the last attribute position - # This regex matches both key="value" and key='value' patterns + # Single pass: extract all attributes AND track the last attribute position. + # Keys are normalised to lowercase so downstream code can use plain dict.get() attrs = {} last_attr_end = 0 - # Use a single regex that handles both quote types. - # Keys must stop at '=' so values like base64-padded URLs ending with '==' - # don't get folded into the preceding attribute name. - for match in re.finditer(r'([^\s=]+)\s*=\s*(["\'])(.*?)\2', content): - key = match.group(1) - value = match.group(3) - attrs[key] = value + for match in _EXTINF_ATTR_RE.finditer(content): + attrs[match.group(1).lower()] = match.group(3) last_attr_end = match.end() # Everything after the last attribute (skipping leading comma and whitespace) is the display name @@ -513,15 +511,80 @@ def parse_extinf_line(line: str) -> dict: else: display_name = content.strip() - # Use tvg-name attribute if available; otherwise try tvc-guide-title, then fall back to display name. - name = get_case_insensitive_attr(attrs, "tvg-name", None) - if not name: - name = get_case_insensitive_attr(attrs, "tvc-guide-title", None) - if not name: - name = display_name + # Per the base #EXTINF spec, the comma text is the canonical human-readable title. + # Fall back to tvc-guide-title, then tvg-name (which some providers use as an EPG key, + # not a display label), and finally the raw content if everything else is empty. + name = display_name or attrs.get("tvc-guide-title") or attrs.get("tvg-name") or content.strip() return {"attributes": attrs, "display_name": display_name, "name": name} +def iter_m3u_entries(lines): + """ + Generator that yields fully-assembled M3U stream entries from raw lines. + + Each yielded dict is guaranteed to contain a ``url`` key in addition to the + fields produced by :func:`parse_extinf_line` (``attributes``, ``display_name``, + ``name``). Recognised extended-tag lines that appear *between* an ``#EXTINF`` + and its URL are accumulated into the pending entry so they are available for + downstream processing: + + - ``#EXTGRP`` — sets ``attributes["group-title"]`` when no ``group-title`` + attribute was present on the ``#EXTINF`` line (explicit attribute wins). + - ``#EXTVLCOPT`` — stored as a list under the ``vlc_opts`` key. + + Unknown directives (``#KODIPROP``, etc.) and blank lines are + silently skipped while keeping the pending entry intact. A second ``#EXTINF`` + before a URL discards the first entry with a warning. A trailing ``#EXTINF`` + at end-of-file with no URL is also discarded. + + Adding support for a new directive requires only a new ``elif`` branch here; + no other code needs to change. + """ + pending = None + pending_line_num = None + for line_num, raw_line in enumerate(lines, 1): + line = raw_line.strip() + if not line: + continue + + if line.startswith("#EXTINF"): + if pending is not None: + logger.warning( + f"Line {pending_line_num}: #EXTINF had no URL (next #EXTINF at line {line_num}); " + f"discarding entry: {list(pending['attributes'].items())[:3]}" + ) + parsed = parse_extinf_line(line) + if parsed is None: + logger.warning(f"Line {line_num}: Failed to parse #EXTINF: {line[:200]}") + pending = parsed # None if malformed; URL branch guards on `pending is not None` + pending_line_num = line_num + + elif line.startswith("#EXTGRP:"): + # Only apply when group-title is absent — explicit attribute wins. + if pending is not None and "group-title" not in pending["attributes"]: + pending["attributes"]["group-title"] = line[len("#EXTGRP:"):].strip() + # else: #EXTGRP outside an entry, or group-title already set — silently skip + + elif line.startswith("#EXTVLCOPT:"): + if pending is not None: + pending.setdefault("vlc_opts", []).append(line[len("#EXTVLCOPT:"):]) + # else: #EXTVLCOPT outside an entry — silently skip + + elif pending is not None and line.startswith(("http", "rtsp", "rtp", "udp")): + pending["url"] = normalize_stream_url(line) if line.startswith("udp") else line + yield pending + pending = None + pending_line_num = None + + # else: unknown directive or bare content — skip, keeping pending intact + + if pending is not None: + logger.warning( + f"Line {pending_line_num}: #EXTINF at end of file had no URL; " + f"discarding entry: {list(pending['attributes'].items())[:3]}" + ) + + @shared_task def refresh_m3u_accounts(): """Queue background parse for all active M3UAccounts.""" @@ -1114,7 +1177,7 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys): "m3u_account": account, "channel_group_id": int(groups.get(group_title)), "stream_hash": stream_hash, - "custom_properties": stream_info["attributes"], + "custom_properties": {**stream_info["attributes"], "vlc_opts": stream_info["vlc_opts"]} if "vlc_opts" in stream_info else stream_info["attributes"], "is_adult": parse_is_adult(stream_info["attributes"].get("is_adult", 0)), "is_stale": False, "stream_id": provider_stream_id, @@ -1482,7 +1545,6 @@ def refresh_m3u_groups(account_id, use_cache=False, full_refresh=False, scan_sta release_task_lock("refresh_m3u_account_groups", account_id) return error_msg, None else: - # Here's the key change - use the success flag from fetch_m3u_lines lines, success = fetch_m3u_lines(account, use_cache) if not success: # If fetch failed, don't continue processing @@ -1493,71 +1555,20 @@ def refresh_m3u_groups(account_id, use_cache=False, full_refresh=False, scan_sta # Log basic file structure for debugging logger.debug(f"Processing {len(lines)} lines from M3U file") - line_count = 0 - extinf_count = 0 - url_count = 0 valid_stream_count = 0 - problematic_lines = [] - for line_index, line in enumerate(lines): - line_count += 1 - line = line.strip() + for entry in iter_m3u_entries(lines): + valid_stream_count += 1 + group_title_attr = get_case_insensitive_attr(entry["attributes"], "group-title", "") + if group_title_attr and group_title_attr not in groups: + logger.debug(f"Found new group for M3U account {account_id}: '{group_title_attr}'") + groups[group_title_attr] = {} + extinf_data.append(entry) - if line.startswith("#EXTINF"): - extinf_count += 1 - parsed = parse_extinf_line(line) - if parsed: - group_title_attr = get_case_insensitive_attr( - parsed["attributes"], "group-title", "" - ) - if group_title_attr: - group_name = group_title_attr - # Log new groups as they're discovered - if group_name not in groups: - logger.debug( - f"Found new group for M3U account {account_id}: '{group_name}'" - ) - groups[group_name] = {} + if valid_stream_count % 1000 == 0: + logger.debug(f"Processed {valid_stream_count} valid streams so far for M3U account: {account_id}") - extinf_data.append(parsed) - else: - # Log problematic EXTINF lines - logger.warning( - f"Failed to parse EXTINF at line {line_index+1}: {line[:200]}" - ) - problematic_lines.append((line_index + 1, line[:200])) - - elif extinf_data and (line.startswith("http") or line.startswith("rtsp") or line.startswith("rtp") or line.startswith("udp")): - url_count += 1 - # Normalize UDP URLs only (e.g., remove VLC-specific @ prefix) - normalized_url = normalize_stream_url(line) if line.startswith("udp") else line - # Associate URL with the last EXTINF line - extinf_data[-1]["url"] = normalized_url - valid_stream_count += 1 - - # Periodically log progress for large files - if valid_stream_count % 1000 == 0: - logger.debug( - f"Processed {valid_stream_count} valid streams so far for M3U account: {account_id}" - ) - - # Log summary statistics - logger.info( - f"M3U parsing complete - Lines: {line_count}, EXTINF: {extinf_count}, URLs: {url_count}, Valid streams: {valid_stream_count}" - ) - - if problematic_lines: - logger.warning( - f"Found {len(problematic_lines)} problematic lines during parsing" - ) - for i, (line_num, content) in enumerate( - problematic_lines[:10] - ): # Log max 10 examples - logger.warning(f"Problematic line #{i+1} at line {line_num}: {content}") - if len(problematic_lines) > 10: - logger.warning( - f"... and {len(problematic_lines) - 10} more problematic lines" - ) + logger.info(f"M3U parsing complete - Valid streams: {valid_stream_count}") # Log group statistics logger.info( @@ -2399,11 +2410,13 @@ def get_transformed_credentials(account, profile=None): # Apply profile-specific transformations if profile is provided if profile and profile.search_pattern and profile.replace_pattern: try: - # Handle backreferences in the replacement pattern - safe_replace_pattern = re.sub(r'\$(\d+)', r'\\\1', profile.replace_pattern) + # Handle backreferences: convert JS-style $ -> \g, $1 -> \1 + # regex module accepts JS-style (?...) named groups natively + safe_replace_pattern = regex.sub(r'\$<([^>]+)>', r'\\g<\1>', profile.replace_pattern) + safe_replace_pattern = regex.sub(r'\$(\d+)', r'\\\1', safe_replace_pattern) # Apply transformation to the complete URL - transformed_complete_url = re.sub(profile.search_pattern, safe_replace_pattern, complete_url) + transformed_complete_url = regex.sub(profile.search_pattern, safe_replace_pattern, complete_url) logger.info(f"Transformed complete URL: {complete_url} -> {transformed_complete_url}") # Extract components from the transformed URL diff --git a/apps/output/views.py b/apps/output/views.py index 84c5f29e..5b061f1f 100644 --- a/apps/output/views.py +++ b/apps/output/views.py @@ -1,8 +1,8 @@ -import ipaddress from django.http import HttpResponse, JsonResponse, Http404, HttpResponseForbidden, StreamingHttpResponse from rest_framework.response import Response from django.urls import reverse -from apps.channels.models import Channel, ChannelProfile, ChannelGroup +from apps.channels.models import Channel, ChannelProfile, ChannelGroup, Stream +from django.db.models import Prefetch from django.views.decorators.csrf import csrf_exempt from django.views.decorators.http import require_http_methods from apps.epg.models import ProgramData @@ -11,9 +11,8 @@ from dispatcharr.utils import network_access_allowed from django.utils import timezone as django_timezone from django.shortcuts import get_object_or_404 from datetime import datetime, timedelta -import html # Add this import for XML escaping -import json # Add this import for JSON parsing -import time # Add this import for keep-alive delays +import html +import time from tzlocal import get_localzone from urllib.parse import urlparse import base64 @@ -138,7 +137,7 @@ def generate_m3u(request, profile_name=None, user=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).order_by("channel_number") + channels = Channel.objects.filter(**filters).select_related('channel_group', 'logo').order_by("channel_number") else: # User has specific limited profiles assigned filters = { @@ -149,9 +148,9 @@ def generate_m3u(request, profile_name=None, user=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).distinct().order_by("channel_number") + channels = Channel.objects.filter(**filters).select_related('channel_group', 'logo').distinct().order_by("channel_number") else: - channels = Channel.objects.filter(user_level__lte=user.user_level).order_by( + channels = Channel.objects.filter(user_level__lte=user.user_level).select_related('channel_group', 'logo').order_by( "channel_number" ) @@ -165,9 +164,9 @@ def generate_m3u(request, profile_name=None, user=None): channels = Channel.objects.filter( channelprofilemembership__channel_profile=channel_profile, channelprofilemembership__enabled=True - ).order_by('channel_number') + ).select_related('channel_group', 'logo').order_by('channel_number') else: - channels = Channel.objects.order_by("channel_number") + channels = Channel.objects.select_related('channel_group', 'logo').order_by("channel_number") # Check if the request wants to use direct logo URLs instead of cache use_cached_logos = request.GET.get('cachedlogos', 'true').lower() != 'false' @@ -175,6 +174,12 @@ def generate_m3u(request, profile_name=None, user=None): # Check if direct stream URLs should be used instead of proxy use_direct_urls = request.GET.get('direct', 'false').lower() == 'true' + # Prefetch streams only when direct URLs are requested (avoids N+1 per channel) + if use_direct_urls: + channels = channels.prefetch_related( + Prefetch('streams', queryset=Stream.objects.order_by('channelstream__order')) + ) + # Get the source to use for tvg-id value # Options: 'channel_number' (default), 'tvg_id', 'gracenote' tvg_id_source = request.GET.get('tvg_id_source', 'channel_number').lower() @@ -194,7 +199,7 @@ def generate_m3u(request, profile_name=None, user=None): epg_base_url = build_absolute_uri_with_port(request, reverse('output:epg_endpoint', args=[profile_name]) if profile_name else reverse('output:epg_endpoint')) # Optionally preserve certain query parameters - preserved_params = ['tvg_id_source', 'cachedlogos', 'days'] + preserved_params = ['tvg_id_source', 'cachedlogos', 'days', 'prev_days'] query_params = {k: v for k, v in request.GET.items() if k in preserved_params} if query_params: from urllib.parse import urlencode @@ -262,7 +267,8 @@ def generate_m3u(request, profile_name=None, user=None): stream_url = f"{base_url}/live/{xc_username}/{xc_password}/{channel.id}" elif use_direct_urls: # Try to get the first stream's direct URL - first_stream = channel.streams.order_by('channelstream__order').first() + all_streams = channel.streams.all() + first_stream = all_streams[0] if all_streams else None if first_stream and first_stream.url: # Use the direct stream URL stream_url = first_stream.url @@ -1268,7 +1274,28 @@ def generate_epg(request, profile_name=None, user=None): """ # Check cache for recent identical request (helps with double-GET from browsers) from django.core.cache import cache - cache_params = f"{profile_name or 'all'}:{user.username if user else 'anonymous'}:{request.GET.urlencode()}" + # Resolve all effective parameter values once here so they are reused for both + # the cache key and inside epg_generator() via closure. + # The cache key is built from resolved values only — not from the raw query string — + # so equivalent requests (e.g. days=7 via URL param vs. user default of 7) share + # the same cache entry regardless of how the value was supplied. + user_custom = (user.custom_properties or {}) if user else {} + try: + num_days = int(request.GET.get('days', user_custom.get('epg_days', 0))) + num_days = max(0, min(num_days, 365)) + except (ValueError, TypeError): + num_days = 0 + try: + prev_days = int(request.GET.get('prev_days', user_custom.get('epg_prev_days', 0))) + prev_days = max(0, min(prev_days, 30)) + except (ValueError, TypeError): + prev_days = 0 + use_cached_logos = request.GET.get('cachedlogos', 'true').lower() != 'false' + tvg_id_source = request.GET.get('tvg_id_source', 'channel_number').lower() + cache_params = ( + f"{profile_name or 'all'}:{user.username if user else 'anonymous'}" + f":d={num_days}:p={prev_days}:logos={use_cached_logos}:tvgid={tvg_id_source}" + ) content_cache_key = f"epg_content:{cache_params}" cached_content = cache.get(content_cache_key) @@ -1280,8 +1307,7 @@ def generate_epg(request, profile_name=None, user=None): return response def epg_generator(): - """Generator function that yields EPG data with keep-alives during processing""" - # Send initial HTTP headers as comments (these will be ignored by XML parsers but keep connection alive) + """Generator function that yields EPG data with keep-alives during processing.""" xml_lines = [] xml_lines.append('') @@ -1301,7 +1327,7 @@ def generate_epg(request, profile_name=None, user=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).order_by("channel_number") + channels = Channel.objects.filter(**filters).select_related('logo', 'epg_data__epg_source').order_by("channel_number") else: # User has specific limited profiles assigned filters = { @@ -1312,9 +1338,9 @@ def generate_epg(request, profile_name=None, user=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).distinct().order_by("channel_number") + channels = Channel.objects.filter(**filters).select_related('logo', 'epg_data__epg_source').distinct().order_by("channel_number") else: - channels = Channel.objects.filter(user_level__lte=user.user_level).order_by( + channels = Channel.objects.filter(user_level__lte=user.user_level).select_related('logo', 'epg_data__epg_source').order_by( "channel_number" ) else: @@ -1327,33 +1353,18 @@ def generate_epg(request, profile_name=None, user=None): channels = Channel.objects.filter( channelprofilemembership__channel_profile=channel_profile, channelprofilemembership__enabled=True, - ).order_by("channel_number") + ).select_related('logo', 'epg_data__epg_source').order_by("channel_number") else: - channels = Channel.objects.all().order_by("channel_number") + channels = Channel.objects.all().select_related('logo', 'epg_data__epg_source').order_by("channel_number") - # Check if the request wants to use direct logo URLs instead of cache - use_cached_logos = request.GET.get('cachedlogos', 'true').lower() != 'false' - - # Get the source to use for tvg-id value - # Options: 'channel_number' (default), 'tvg_id', 'gracenote' - tvg_id_source = request.GET.get('tvg_id_source', 'channel_number').lower() - - # Get the number of days for EPG data - try: - # Default to 0 days (everything) for real EPG if not specified - days_param = request.GET.get('days', '0') - num_days = int(days_param) - # Set reasonable limits - num_days = max(0, min(num_days, 365)) # Between 0 and 365 days - except ValueError: - num_days = 0 # Default to all data if invalid value # For dummy EPG, use either the specified value or default to 3 days dummy_days = num_days if num_days > 0 else 3 - # Calculate cutoff date for EPG data filtering (only if days > 0) + # Calculate cutoff dates for EPG data filtering now = django_timezone.now() cutoff_date = now + timedelta(days=num_days) if num_days > 0 else None + lookback_cutoff = now - timedelta(days=prev_days) # Build collision-free channel number mapping for XC clients (if user is authenticated) # XC clients require integer channel numbers, so we need to ensure no conflicts @@ -1380,13 +1391,10 @@ def generate_epg(request, profile_name=None, user=None): # Process channels for the section for channel in channels: - # For XC clients (user is not None), use collision-free integer mapping - # For regular clients (user is None), use original formatting logic + # user is set only for XC clients, which require integer channel numbers if user is not None: - # XC client - use collision-free integer formatted_channel_number = channel_num_map[channel.id] else: - # Regular client - format channel number as integer if it has no decimal component if channel.channel_number is not None: if channel.channel_number == int(channel.channel_number): formatted_channel_number = int(channel.channel_number) @@ -1401,10 +1409,8 @@ def generate_epg(request, profile_name=None, user=None): elif tvg_id_source == 'gracenote' and channel.tvc_guide_stationid: channel_id = channel.tvc_guide_stationid else: - # Default to channel number (original behavior) channel_id = str(formatted_channel_number) if formatted_channel_number != "" else str(channel.id) - # Add channel logo if available tvg_logo = "" # Check if this is a custom dummy EPG with channel logo URL template @@ -1463,12 +1469,10 @@ def generate_epg(request, profile_name=None, user=None): # If no custom dummy logo, use regular logo logic if not tvg_logo and channel.logo: if use_cached_logos: - # Use cached logo as before tvg_logo = build_absolute_uri_with_port(request, reverse('api:channels:logo-cache', args=[channel.logo.id])) else: - # Try to find direct logo URL from channel's streams + # Use direct URL if available, otherwise fall back to cached version direct_logo = channel.logo.url if channel.logo.url.startswith(('http://', 'https://')) else None - # If direct logo found, use it; otherwise fall back to cached version if direct_logo: tvg_logo = direct_logo else: @@ -1484,22 +1488,21 @@ def generate_epg(request, profile_name=None, user=None): yield channel_xml xml_lines = [] # Clear to save memory - # Process programs for each channel - for channel in channels: + # Pre-pass: categorize channels into dummy and real EPG groups + dummy_program_list = [] # (channel_id, pattern_match_name, epg_source_or_None) + real_epg_map = {} # epg_data_id -> [channel_id, ...] + dummy_epg_checked = {} # epg_data_id -> bool (has stored programs) - # Use the same channel ID determination for program entries + for channel in channels: + # Determine channel_id (same logic as channel section) if tvg_id_source == 'tvg_id' and channel.tvg_id: channel_id = channel.tvg_id elif tvg_id_source == 'gracenote' and channel.tvc_guide_stationid: channel_id = channel.tvc_guide_stationid else: - # For XC clients (user is not None), use collision-free integer mapping - # For regular clients (user is None), use original formatting logic if user is not None: - # XC client - use collision-free integer from map formatted_channel_number = channel_num_map[channel.id] else: - # Regular client - format channel number as before if channel.channel_number is not None: if channel.channel_number == int(channel.channel_number): formatted_channel_number = int(channel.channel_number) @@ -1507,12 +1510,9 @@ def generate_epg(request, profile_name=None, user=None): formatted_channel_number = channel.channel_number else: formatted_channel_number = "" - # Default to channel number channel_id = str(formatted_channel_number) if formatted_channel_number != "" else str(channel.id) - # Use EPG data name for display, but channel name for pattern matching display_name = channel.epg_data.name if channel.epg_data else channel.name - # For dummy EPG pattern matching, determine which name to use pattern_match_name = channel.name # Check if we should use stream name instead of channel name @@ -1534,372 +1534,343 @@ def generate_epg(request, profile_name=None, user=None): logger.warning(f"Stream index {stream_index} not found for channel {channel.name}, falling back to channel name") if not channel.epg_data: - # Use the enhanced dummy EPG generation function with defaults - program_length_hours = 4 # Default to 4-hour program blocks - dummy_programs = generate_dummy_programs( - channel_id, pattern_match_name, - num_days=dummy_days, - program_length_hours=program_length_hours, - epg_source=None + dummy_program_list.append((channel_id, pattern_match_name, None)) + else: + if channel.epg_data.epg_source and channel.epg_data.epg_source.source_type == 'dummy': + epg_data_id = channel.epg_data_id + if epg_data_id not in dummy_epg_checked: + dummy_epg_checked[epg_data_id] = channel.epg_data.programs.exists() + if dummy_epg_checked[epg_data_id]: + real_epg_map.setdefault(epg_data_id, []).append(channel_id) + else: + dummy_program_list.append((channel_id, pattern_match_name, channel.epg_data.epg_source)) + continue + + real_epg_map.setdefault(channel.epg_data_id, []).append(channel_id) + + # Emit dummy programmes + for channel_id, pattern_match_name, epg_source in dummy_program_list: + program_length_hours = 4 + dummy_programs = generate_dummy_programs( + channel_id, pattern_match_name, + num_days=dummy_days, + program_length_hours=program_length_hours, + epg_source=epg_source + ) + for program in dummy_programs: + start_str = program['start_time'].strftime("%Y%m%d%H%M%S %z") + stop_str = program['end_time'].strftime("%Y%m%d%H%M%S %z") + yield f' \n' + yield f" {html.escape(program['title'])}\n" + if program.get('sub_title'): + yield f" {html.escape(program['sub_title'])}\n" + yield f" {html.escape(program['description'])}\n" + custom_data = program.get('custom_properties', {}) + if 'categories' in custom_data: + for cat in custom_data['categories']: + yield f" {html.escape(cat)}\n" + if 'date' in custom_data: + yield f" {html.escape(custom_data['date'])}\n" + if custom_data.get('live', False): + yield f" \n" + if custom_data.get('new', False): + yield f" \n" + if 'icon' in custom_data: + yield f' \n' + yield f" \n" + + # Emit real programmes: single bulk query, chunked to avoid server-side cursor issues. + all_epg_ids = list(real_epg_map.keys()) + if all_epg_ids: + if num_days > 0: + programs_qs = ProgramData.objects.filter( + epg_id__in=all_epg_ids, + end_time__gte=lookback_cutoff, + start_time__lt=cutoff_date, + ) + else: + programs_qs = ProgramData.objects.filter( + epg_id__in=all_epg_ids, + end_time__gte=lookback_cutoff, ) - for program in dummy_programs: - # Format times in XMLTV format - start_str = program['start_time'].strftime("%Y%m%d%H%M%S %z") - stop_str = program['end_time'].strftime("%Y%m%d%H%M%S %z") + programs_base_qs = programs_qs.order_by('epg_id', 'id').values( + 'id', 'epg_id', 'start_time', 'end_time', 'title', 'sub_title', + 'description', 'custom_properties', + ) - # Create program entry with escaped channel name - yield f' \n' - yield f" {html.escape(program['title'])}\n" + current_epg_id = None + channel_ids_for_epg = None + is_multi = False + multi_buffer = [] + program_batch = [] + batch_size = 1000 + chunk_size = 5000 + # Keyset pagination: track last (epg_id, id) instead of OFFSET + # to avoid skipping/duplicating rows if the table changes mid-stream. + last_epg_id = 0 + last_id = 0 - # Add subtitle if available - if program.get('sub_title'): - yield f" {html.escape(program['sub_title'])}\n" + while True: + program_chunk = list( + programs_base_qs.filter(epg_id__gte=last_epg_id) + .exclude(epg_id=last_epg_id, id__lte=last_id)[:chunk_size] + ) - yield f" {html.escape(program['description'])}\n" + if not program_chunk: + break - # Add custom_properties if present - custom_data = program.get('custom_properties', {}) + # Advance keyset cursor to last row in this chunk + last_row = program_chunk[-1] + last_epg_id = last_row['epg_id'] + last_id = last_row['id'] - # Categories - if 'categories' in custom_data: - for cat in custom_data['categories']: - yield f" {html.escape(cat)}\n" + for prog in program_chunk: + epg_id = prog['epg_id'] - # Date tag - if 'date' in custom_data: - yield f" {html.escape(custom_data['date'])}\n" + # When epg_id changes, flush multi-channel buffer for previous group + if epg_id != current_epg_id: + if is_multi and multi_buffer: + escaped_primary = html.escape(channel_ids_for_epg[0]) + for extra_cid in channel_ids_for_epg[1:]: + escaped_extra = html.escape(extra_cid) + for xml_text in multi_buffer: + program_batch.append(xml_text.replace( + f'channel="{escaped_primary}"', + f'channel="{escaped_extra}"', + 1, + )) + if len(program_batch) >= batch_size: + yield '\n'.join(program_batch) + '\n' + program_batch = [] + multi_buffer = [] - # Live tag - if custom_data.get('live', False): - yield f" \n" + current_epg_id = epg_id + channel_ids_for_epg = real_epg_map[epg_id] + is_multi = len(channel_ids_for_epg) > 1 - # New tag - if custom_data.get('new', False): - yield f" \n" + # Build programme XML for primary channel_id + primary_cid = channel_ids_for_epg[0] + start_str = prog['start_time'].strftime("%Y%m%d%H%M%S %z") + stop_str = prog['end_time'].strftime("%Y%m%d%H%M%S %z") - # Icon/poster URL - if 'icon' in custom_data: - yield f" \n" + program_xml = [f' '] + program_xml.append(f' {html.escape(prog["title"])}') - yield f" \n" + if prog['sub_title']: + program_xml.append(f" {html.escape(prog['sub_title'])}") - else: - # Check if this is a dummy EPG with no programs (generate on-demand) - if channel.epg_data.epg_source and channel.epg_data.epg_source.source_type == 'dummy': - # This is a custom dummy EPG - check if it has programs - if not channel.epg_data.programs.exists(): - # No programs stored, generate on-demand using custom patterns - # Use actual channel name for pattern matching - program_length_hours = 4 - dummy_programs = generate_dummy_programs( - channel_id, pattern_match_name, - num_days=dummy_days, - program_length_hours=program_length_hours, - epg_source=channel.epg_data.epg_source - ) + if prog['description']: + program_xml.append(f" {html.escape(prog['description'])}") - for program in dummy_programs: - start_str = program['start_time'].strftime("%Y%m%d%H%M%S %z") - stop_str = program['end_time'].strftime("%Y%m%d%H%M%S %z") + custom_data = prog['custom_properties'] or {} + if custom_data: - yield f' \n' - yield f" {html.escape(program['title'])}\n" + if "categories" in custom_data and custom_data["categories"]: + for category in custom_data["categories"]: + program_xml.append(f" {html.escape(category)}") - # Add subtitle if available - if program.get('sub_title'): - yield f" {html.escape(program['sub_title'])}\n" + if "keywords" in custom_data and custom_data["keywords"]: + for keyword in custom_data["keywords"]: + program_xml.append(f" {html.escape(keyword)}") - yield f" {html.escape(program['description'])}\n" + # onscreen_episode takes priority over episode for the onscreen system + if "onscreen_episode" in custom_data: + program_xml.append(f' {html.escape(custom_data["onscreen_episode"])}') + elif "episode" in custom_data: + program_xml.append(f' E{custom_data["episode"]}') - # Add custom_properties if present - custom_data = program.get('custom_properties', {}) + # Handle dd_progid format + if 'dd_progid' in custom_data: + program_xml.append(f' {html.escape(custom_data["dd_progid"])}') - # Categories - if 'categories' in custom_data: - for cat in custom_data['categories']: - yield f" {html.escape(cat)}\n" + # Handle external database IDs + for system in ['thetvdb.com', 'themoviedb.org', 'imdb.com']: + if f'{system}_id' in custom_data: + program_xml.append(f' {html.escape(custom_data[f"{system}_id"])}') - # Date tag - if 'date' in custom_data: - yield f" {html.escape(custom_data['date'])}\n" + # Add season and episode numbers in xmltv_ns format if available + if "season" in custom_data and "episode" in custom_data: + season = ( + int(custom_data["season"]) - 1 + if str(custom_data["season"]).isdigit() + else 0 + ) + episode = ( + int(custom_data["episode"]) - 1 + if str(custom_data["episode"]).isdigit() + else 0 + ) + program_xml.append(f' {season}.{episode}.') - # Live tag - if custom_data.get('live', False): - yield f" \n" + if "language" in custom_data: + program_xml.append(f' {html.escape(custom_data["language"])}') - # New tag - if custom_data.get('new', False): - yield f" \n" + if "original_language" in custom_data: + program_xml.append(f' {html.escape(custom_data["original_language"])}') - # Icon/poster URL - if 'icon' in custom_data: - yield f" \n" + if "length" in custom_data and isinstance(custom_data["length"], dict): + length_value = custom_data["length"].get("value", "") + length_units = custom_data["length"].get("units", "minutes") + program_xml.append(f' {html.escape(str(length_value))}') - yield f" \n" + if "video" in custom_data and isinstance(custom_data["video"], dict): + program_xml.append(" ") - continue # Skip to next channel + if "audio" in custom_data and isinstance(custom_data["audio"], dict): + program_xml.append(" ") - # For real EPG data - filter only if days parameter was specified - if num_days > 0: - programs_qs = channel.epg_data.programs.filter( - end_time__gte=now, - start_time__lt=cutoff_date - ).order_by('id') # Explicit ordering for consistent chunking - else: - # Return all non-expired programs if days=0 or not specified - programs_qs = channel.epg_data.programs.filter( - end_time__gte=now - ).order_by('id') + if "subtitles" in custom_data and isinstance(custom_data["subtitles"], list): + for subtitle in custom_data["subtitles"]: + if isinstance(subtitle, dict): + subtitle_type = subtitle.get("type", "") + type_attr = f' type="{html.escape(subtitle_type)}"' if subtitle_type else "" + program_xml.append(f" ") + if "language" in subtitle: + program_xml.append(f" {html.escape(subtitle['language'])}") + program_xml.append(" ") - # Process programs in chunks to avoid cursor timeout issues - program_batch = [] - batch_size = 250 - chunk_size = 1000 # Fetch 1000 programs at a time from DB + if "rating" in custom_data: + rating_system = custom_data.get("rating_system", "TV Parental Guidelines") + program_xml.append(f' ') + program_xml.append(f' {html.escape(custom_data["rating"])}') + program_xml.append(f" ") - # Fetch chunks until no more results (avoids count() query) - offset = 0 - while True: - # Fetch a chunk of programs - this closes the cursor after fetching - program_chunk = list(programs_qs[offset:offset + chunk_size]) + if "star_ratings" in custom_data and isinstance(custom_data["star_ratings"], list): + for star_rating in custom_data["star_ratings"]: + if isinstance(star_rating, dict) and "value" in star_rating: + system_attr = f' system="{html.escape(star_rating["system"])}"' if "system" in star_rating else "" + program_xml.append(f" ") + program_xml.append(f" {html.escape(star_rating['value'])}") + program_xml.append(" ") - # Break if no more programs - if not program_chunk: - break + if "reviews" in custom_data and isinstance(custom_data["reviews"], list): + for review in custom_data["reviews"]: + if isinstance(review, dict) and "content" in review: + review_type = review.get("type", "text") + attrs = [f'type="{html.escape(review_type)}"'] + if "source" in review: + attrs.append(f'source="{html.escape(review["source"])}"') + if "reviewer" in review: + attrs.append(f'reviewer="{html.escape(review["reviewer"])}"') + attr_str = " ".join(attrs) + program_xml.append(f' {html.escape(review["content"])}') - # Process each program in the chunk - for prog in program_chunk: - start_str = prog.start_time.strftime("%Y%m%d%H%M%S %z") - stop_str = prog.end_time.strftime("%Y%m%d%H%M%S %z") + if "images" in custom_data and isinstance(custom_data["images"], list): + for image in custom_data["images"]: + if isinstance(image, dict) and "url" in image: + attrs = [] + for attr in ['type', 'size', 'orient', 'system']: + if attr in image: + attrs.append(f'{attr}="{html.escape(image[attr])}"') + attr_str = " " + " ".join(attrs) if attrs else "" + program_xml.append(f' {html.escape(image["url"])}') - program_xml = [f' '] - program_xml.append(f' {html.escape(prog.title)}') + # Add enhanced credits handling + if "credits" in custom_data: + program_xml.append(" ") + credits = custom_data["credits"] - # Add subtitle if available - if prog.sub_title: - program_xml.append(f" {html.escape(prog.sub_title)}") - - # Add description if available - if prog.description: - program_xml.append(f" {html.escape(prog.description)}") - - # Process custom properties if available - if prog.custom_properties: - custom_data = prog.custom_properties or {} - - # Add categories if available - if "categories" in custom_data and custom_data["categories"]: - for category in custom_data["categories"]: - program_xml.append(f" {html.escape(category)}") - - # Add keywords if available - if "keywords" in custom_data and custom_data["keywords"]: - for keyword in custom_data["keywords"]: - program_xml.append(f" {html.escape(keyword)}") - - # Handle episode numbering - multiple formats supported - # Prioritize onscreen_episode over standalone episode for onscreen system - if "onscreen_episode" in custom_data: - program_xml.append(f' {html.escape(custom_data["onscreen_episode"])}') - elif "episode" in custom_data: - program_xml.append(f' E{custom_data["episode"]}') - - # Handle dd_progid format - if 'dd_progid' in custom_data: - program_xml.append(f' {html.escape(custom_data["dd_progid"])}') - - # Handle external database IDs - for system in ['thetvdb.com', 'themoviedb.org', 'imdb.com']: - if f'{system}_id' in custom_data: - program_xml.append(f' {html.escape(custom_data[f"{system}_id"])}') - - # Add season and episode numbers in xmltv_ns format if available - if "season" in custom_data and "episode" in custom_data: - season = ( - int(custom_data["season"]) - 1 - if str(custom_data["season"]).isdigit() - else 0 - ) - episode = ( - int(custom_data["episode"]) - 1 - if str(custom_data["episode"]).isdigit() - else 0 - ) - program_xml.append(f' {season}.{episode}.') - - # Add language information - if "language" in custom_data: - program_xml.append(f' {html.escape(custom_data["language"])}') - - if "original_language" in custom_data: - program_xml.append(f' {html.escape(custom_data["original_language"])}') - - # Add length information - if "length" in custom_data and isinstance(custom_data["length"], dict): - length_value = custom_data["length"].get("value", "") - length_units = custom_data["length"].get("units", "minutes") - program_xml.append(f' {html.escape(str(length_value))}') - - # Add video information - if "video" in custom_data and isinstance(custom_data["video"], dict): - program_xml.append(" ") - - # Add audio information - if "audio" in custom_data and isinstance(custom_data["audio"], dict): - program_xml.append(" ") - - # Add subtitles information - if "subtitles" in custom_data and isinstance(custom_data["subtitles"], list): - for subtitle in custom_data["subtitles"]: - if isinstance(subtitle, dict): - subtitle_type = subtitle.get("type", "") - type_attr = f' type="{html.escape(subtitle_type)}"' if subtitle_type else "" - program_xml.append(f" ") - if "language" in subtitle: - program_xml.append(f" {html.escape(subtitle['language'])}") - program_xml.append(" ") - - # Add rating if available - if "rating" in custom_data: - rating_system = custom_data.get("rating_system", "TV Parental Guidelines") - program_xml.append(f' ') - program_xml.append(f' {html.escape(custom_data["rating"])}') - program_xml.append(f" ") - - # Add star ratings - if "star_ratings" in custom_data and isinstance(custom_data["star_ratings"], list): - for star_rating in custom_data["star_ratings"]: - if isinstance(star_rating, dict) and "value" in star_rating: - system_attr = f' system="{html.escape(star_rating["system"])}"' if "system" in star_rating else "" - program_xml.append(f" ") - program_xml.append(f" {html.escape(star_rating['value'])}") - program_xml.append(" ") - - # Add reviews - if "reviews" in custom_data and isinstance(custom_data["reviews"], list): - for review in custom_data["reviews"]: - if isinstance(review, dict) and "content" in review: - review_type = review.get("type", "text") - attrs = [f'type="{html.escape(review_type)}"'] - if "source" in review: - attrs.append(f'source="{html.escape(review["source"])}"') - if "reviewer" in review: - attrs.append(f'reviewer="{html.escape(review["reviewer"])}"') - attr_str = " ".join(attrs) - program_xml.append(f' {html.escape(review["content"])}') - - # Add images - if "images" in custom_data and isinstance(custom_data["images"], list): - for image in custom_data["images"]: - if isinstance(image, dict) and "url" in image: - attrs = [] - for attr in ['type', 'size', 'orient', 'system']: - if attr in image: - attrs.append(f'{attr}="{html.escape(image[attr])}"') - attr_str = " " + " ".join(attrs) if attrs else "" - program_xml.append(f' {html.escape(image["url"])}') - - # Add enhanced credits handling - if "credits" in custom_data: - program_xml.append(" ") - credits = custom_data["credits"] - - # Handle different credit types - for role in ['director', 'writer', 'adapter', 'producer', 'composer', 'editor', 'presenter', 'commentator', 'guest']: - if role in credits: - people = credits[role] - if isinstance(people, list): - for person in people: - program_xml.append(f" <{role}>{html.escape(person)}") - else: - program_xml.append(f" <{role}>{html.escape(people)}") - - # Handle actors separately to include role and guest attributes - if "actor" in credits: - actors = credits["actor"] - if isinstance(actors, list): - for actor in actors: - if isinstance(actor, dict): - name = actor.get("name", "") - role_attr = f' role="{html.escape(actor["role"])}"' if "role" in actor else "" - guest_attr = ' guest="yes"' if actor.get("guest") else "" - program_xml.append(f" {html.escape(name)}") - else: - program_xml.append(f" {html.escape(actor)}") + for role in ['director', 'writer', 'adapter', 'producer', 'composer', 'editor', 'presenter', 'commentator', 'guest']: + if role in credits: + people = credits[role] + if isinstance(people, list): + for person in people: + program_xml.append(f" <{role}>{html.escape(person)}") else: - program_xml.append(f" {html.escape(actors)}") + program_xml.append(f" <{role}>{html.escape(people)}") - program_xml.append(" ") - - # Add program date if available (full date, not just year) - if "date" in custom_data: - program_xml.append(f' {html.escape(custom_data["date"])}') - - # Add country if available - if "country" in custom_data: - program_xml.append(f' {html.escape(custom_data["country"])}') - - # Add icon if available - if "icon" in custom_data: - program_xml.append(f' ') - - # Add special flags as proper tags with enhanced handling - if custom_data.get("previously_shown", False): - prev_shown_details = custom_data.get("previously_shown_details", {}) - attrs = [] - if "start" in prev_shown_details: - attrs.append(f'start="{html.escape(prev_shown_details["start"])}"') - if "channel" in prev_shown_details: - attrs.append(f'channel="{html.escape(prev_shown_details["channel"])}"') - attr_str = " " + " ".join(attrs) if attrs else "" - program_xml.append(f" ") - - if custom_data.get("premiere", False): - premiere_text = custom_data.get("premiere_text", "") - if premiere_text: - program_xml.append(f" {html.escape(premiere_text)}") + # Handle actors separately to include role and guest attributes + if "actor" in credits: + actors = credits["actor"] + if isinstance(actors, list): + for actor in actors: + if isinstance(actor, dict): + name = actor.get("name", "") + role_attr = f' role="{html.escape(actor["role"])}"' if "role" in actor else "" + guest_attr = ' guest="yes"' if actor.get("guest") else "" + program_xml.append(f" {html.escape(name)}") + else: + program_xml.append(f" {html.escape(actor)}") else: - program_xml.append(" ") + program_xml.append(f" {html.escape(actors)}") - if custom_data.get("last_chance", False): - last_chance_text = custom_data.get("last_chance_text", "") - if last_chance_text: - program_xml.append(f" {html.escape(last_chance_text)}") - else: - program_xml.append(" ") + program_xml.append(" ") - if custom_data.get("new", False): - program_xml.append(" ") + if "date" in custom_data: + program_xml.append(f' {html.escape(custom_data["date"])}') - if custom_data.get('live', False): - program_xml.append(' ') + if "country" in custom_data: + program_xml.append(f' {html.escape(custom_data["country"])}') - program_xml.append(" ") + if "icon" in custom_data: + program_xml.append(f' ') - # Add to batch - program_batch.extend(program_xml) + # Add special flags as proper tags with enhanced handling + if custom_data.get("previously_shown", False): + prev_shown_details = custom_data.get("previously_shown_details", {}) + attrs = [] + if "start" in prev_shown_details: + attrs.append(f'start="{html.escape(prev_shown_details["start"])}"') + if "channel" in prev_shown_details: + attrs.append(f'channel="{html.escape(prev_shown_details["channel"])}"') + attr_str = " " + " ".join(attrs) if attrs else "" + program_xml.append(f" ") - # Send batch when full or send keep-alive - if len(program_batch) >= batch_size: - batch_xml = '\n'.join(program_batch) + '\n' - yield batch_xml - program_batch = [] + if custom_data.get("premiere", False): + premiere_text = custom_data.get("premiere_text", "") + if premiere_text: + program_xml.append(f" {html.escape(premiere_text)}") + else: + program_xml.append(" ") - # Move to next chunk - offset += chunk_size + if custom_data.get("last_chance", False): + last_chance_text = custom_data.get("last_chance_text", "") + if last_chance_text: + program_xml.append(f" {html.escape(last_chance_text)}") + else: + program_xml.append(" ") - # Send remaining programs in batch - if program_batch: - batch_xml = '\n'.join(program_batch) + '\n' - yield batch_xml + if custom_data.get("new", False): + program_xml.append(" ") + + if custom_data.get('live', False): + program_xml.append(' ') + + program_xml.append(" ") + + xml_text = '\n'.join(program_xml) + program_batch.append(xml_text) + + if is_multi: + multi_buffer.append(xml_text) + + if len(program_batch) >= batch_size: + yield '\n'.join(program_batch) + '\n' + program_batch = [] + + # Final flush of multi-channel buffer + if is_multi and multi_buffer: + escaped_primary = html.escape(channel_ids_for_epg[0]) + for extra_cid in channel_ids_for_epg[1:]: + escaped_extra = html.escape(extra_cid) + for xml_text in multi_buffer: + program_batch.append(xml_text.replace( + f'channel="{escaped_primary}"', + f'channel="{escaped_extra}"', + 1, + )) + + if program_batch: + yield '\n'.join(program_batch) + '\n' # Send final closing tag and completion message yield "\n" @@ -1929,7 +1900,6 @@ def generate_epg(request, profile_name=None, user=None): cache.set(content_cache_key, full_content, 300) logger.debug("Cached EPG content (%d bytes)", len(full_content)) - # Return streaming response response = StreamingHttpResponse( streaming_content=caching_generator(), content_type="application/xml" @@ -2181,7 +2151,7 @@ def xc_get_live_streams(request, user, category_id=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).order_by("channel_number") + channels = Channel.objects.filter(**filters).select_related('channel_group', 'logo').order_by("channel_number") else: # User has specific limited profiles assigned filters = { @@ -2194,14 +2164,23 @@ def xc_get_live_streams(request, user, category_id=None): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channels = Channel.objects.filter(**filters).distinct().order_by("channel_number") + channels = Channel.objects.filter(**filters).select_related('channel_group', 'logo').distinct().order_by("channel_number") else: if not category_id: - channels = Channel.objects.filter(user_level__lte=user.user_level).order_by("channel_number") + channels = Channel.objects.filter(user_level__lte=user.user_level).select_related('channel_group', 'logo').order_by("channel_number") else: channels = Channel.objects.filter( channel_group__id=category_id, user_level__lte=user.user_level - ).order_by("channel_number") + ).select_related('channel_group', 'logo').order_by("channel_number") + + # Resolve the fallback group ID once to avoid a get_or_create query per null-group channel + _default_group_id = None + + def _get_default_group_id(): + nonlocal _default_group_id + if _default_group_id is None: + _default_group_id = ChannelGroup.objects.get_or_create(name="Default Group")[0].id + return _default_group_id # Build collision-free mapping for XC clients (which require integers) # This ensures channels with float numbers don't conflict with existing integers @@ -2249,8 +2228,8 @@ def xc_get_live_streams(request, user, category_id=None): "epg_channel_id": str(channel_num_int), "added": str(int(channel.created_at.timestamp())), "is_adult": int(channel.is_adult), - "category_id": str(channel.channel_group.id if channel.channel_group else ChannelGroup.objects.get_or_create(name="Default Group")[0].id), - "category_ids": [channel.channel_group.id if channel.channel_group else ChannelGroup.objects.get_or_create(name="Default Group")[0].id], + "category_id": str(channel.channel_group.id if channel.channel_group else _get_default_group_id()), + "category_ids": [channel.channel_group.id if channel.channel_group else _get_default_group_id()], "custom_sid": None, "tv_archive": 0, "direct_source": "", @@ -2280,7 +2259,7 @@ def xc_get_epg(request, user, short=False): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channel = Channel.objects.filter(**filters).first() + channel = Channel.objects.filter(**filters).select_related('epg_data__epg_source').first() else: # User has specific limited profiles assigned filters = { @@ -2292,12 +2271,12 @@ def xc_get_epg(request, user, short=False): # Hide adult content if user preference is set if (user.custom_properties or {}).get('hide_adult_content', False): filters["is_adult"] = False - channel = Channel.objects.filter(**filters).distinct().first() + channel = Channel.objects.filter(**filters).select_related('epg_data__epg_source').distinct().first() if not channel: raise Http404() else: - channel = get_object_or_404(Channel, id=channel_id) + channel = get_object_or_404(Channel.objects.select_related('epg_data__epg_source'), id=channel_id) if not channel: raise Http404() @@ -2332,6 +2311,20 @@ def xc_get_epg(request, user, short=False): channel_num_int = channel_num_map.get(channel.id, int(channel.channel_number)) limit = int(request.GET.get('limit', 4)) + user_custom = user.custom_properties or {} + try: + num_days = int(request.GET.get('days', user_custom.get('epg_days', 0))) + num_days = max(0, min(num_days, 365)) + except (ValueError, TypeError): + num_days = 0 + try: + prev_days = int(request.GET.get('prev_days', user_custom.get('epg_prev_days', 0))) + prev_days = max(0, min(prev_days, 30)) + except (ValueError, TypeError): + prev_days = 0 + now = django_timezone.now() + lookback_cutoff = now - timedelta(days=prev_days) + forward_cutoff = now + timedelta(days=num_days) if num_days > 0 else None if channel.epg_data: # Check if this is a dummy EPG that generates on-demand if channel.epg_data.epg_source and channel.epg_data.epg_source.source_type == 'dummy': @@ -2344,24 +2337,28 @@ def xc_get_epg(request, user, short=False): ) else: # Has stored programs, use them - if short == False: + if short: + # Short EPG: current and upcoming only (never historical), limited count programs = channel.epg_data.programs.filter( - end_time__gt=django_timezone.now() - ).order_by('start_time') - else: - programs = channel.epg_data.programs.filter( - end_time__gt=django_timezone.now() + end_time__gt=now ).order_by('start_time')[:limit] + else: + qs = channel.epg_data.programs.filter(end_time__gt=lookback_cutoff) + if forward_cutoff: + qs = qs.filter(start_time__lt=forward_cutoff) + programs = qs.order_by('start_time') else: # Regular EPG with stored programs - if short == False: + if short: + # Short EPG: current and upcoming only (never historical), limited count programs = channel.epg_data.programs.filter( - end_time__gt=django_timezone.now() - ).order_by('start_time') + end_time__gt=now + ).order_by('start_time')[:limit] else: - programs = channel.epg_data.programs.filter( - end_time__gt=django_timezone.now() - ).order_by('start_time')[:limit] + qs = channel.epg_data.programs.filter(end_time__gt=lookback_cutoff) + if forward_cutoff: + qs = qs.filter(start_time__lt=forward_cutoff) + programs = qs.order_by('start_time') else: # No EPG data assigned, generate default dummy programs = generate_dummy_programs(channel_id=channel_id, channel_name=channel.name, epg_source=None) diff --git a/apps/plugins/api_urls.py b/apps/plugins/api_urls.py index 5ba85be2..99355232 100644 --- a/apps/plugins/api_urls.py +++ b/apps/plugins/api_urls.py @@ -8,6 +8,14 @@ from .api_views import ( PluginImportAPIView, PluginDeleteAPIView, PluginLogoAPIView, + PluginRepoListCreateAPIView, + PluginRepoPreviewAPIView, + PluginRepoDetailAPIView, + PluginRepoRefreshAPIView, + AvailablePluginsAPIView, + PluginDetailManifestAPIView, + PluginInstallFromRepoAPIView, + PluginRepoSettingsAPIView, ) app_name = "plugins" @@ -21,4 +29,13 @@ urlpatterns = [ path("plugins//run/", PluginRunAPIView.as_view(), name="run"), path("plugins//enabled/", PluginEnabledAPIView.as_view(), name="enabled"), path("plugins//logo/", PluginLogoAPIView.as_view(), name="logo"), + # Plugin repos (hub / store) - static paths first, then parametric + path("repos/", PluginRepoListCreateAPIView.as_view(), name="repo-list"), + path("repos/available/", AvailablePluginsAPIView.as_view(), name="available-plugins"), + path("repos/plugin-detail/", PluginDetailManifestAPIView.as_view(), name="plugin-detail-manifest"), + path("repos/install/", PluginInstallFromRepoAPIView.as_view(), name="repo-install"), + path("repos/settings/", PluginRepoSettingsAPIView.as_view(), name="repo-settings"), + path("repos/preview/", PluginRepoPreviewAPIView.as_view(), name="repo-preview"), + path("repos//", PluginRepoDetailAPIView.as_view(), name="repo-detail"), + path("repos//refresh/", PluginRepoRefreshAPIView.as_view(), name="repo-refresh"), ] diff --git a/apps/plugins/api_views.py b/apps/plugins/api_views.py index 624dcc4d..8f7a2c71 100644 --- a/apps/plugins/api_views.py +++ b/apps/plugins/api_views.py @@ -1,15 +1,23 @@ +import hashlib +import ipaddress import logging +import io +import json import re +import socket from rest_framework.views import APIView from rest_framework.response import Response -from rest_framework import status +from rest_framework import status, serializers +from drf_spectacular.utils import extend_schema, inline_serializer from django.conf import settings from django.core.files.uploadedfile import UploadedFile from django.http import FileResponse +from django.utils import timezone import os import zipfile import shutil import tempfile +import requests as http_requests from urllib.parse import urlparse from apps.accounts.permissions import ( Authenticated, @@ -18,11 +26,35 @@ from apps.accounts.permissions import ( from dispatcharr.utils import network_access_allowed from .loader import PluginManager -from .models import PluginConfig +from .models import PluginConfig, PluginRepo +from .serializers import PluginRepoSerializer logger = logging.getLogger(__name__) +def _compare_versions(a, b): + """Compare two semver-like version strings. + Returns negative if a < b, 0 if equal, positive if a > b. + + If either version is a prerelease (any dot-segment contains non-digit + characters), numeric ordering is meaningless. Falls back to exact string + equality: 0 if identical, 1 otherwise. + """ + if not a or not b: + return 0 + na = a.lstrip("v") + nb = b.lstrip("v") + if any(not p.isdigit() for p in na.split(".")) or any(not p.isdigit() for p in nb.split(".")): + return 0 if na == nb else 1 + pa = [int(x) for x in na.split(".")] + pb = [int(x) for x in nb.split(".")] + for i in range(max(len(pa), len(pb))): + diff = (pa[i] if i < len(pa) else 0) - (pb[i] if i < len(pb) else 0) + if diff != 0: + return diff + return 0 + + MAX_PLUGIN_IMPORT_FILES = getattr(settings, "DISPATCHARR_PLUGIN_IMPORT_MAX_FILES", 2000) MAX_PLUGIN_IMPORT_BYTES = getattr(settings, "DISPATCHARR_PLUGIN_IMPORT_MAX_BYTES", 200 * 1024 * 1024) MAX_PLUGIN_IMPORT_FILE_BYTES = getattr(settings, "DISPATCHARR_PLUGIN_IMPORT_MAX_FILE_BYTES", 200 * 1024 * 1024) @@ -44,12 +76,43 @@ def _parse_bool(value): def _sanitize_plugin_key(value: str) -> str: base = os.path.basename(value or "") - base = base.replace(" ", "_").lower() - base = re.sub(r"[^a-z0-9_-]", "_", base) - base = base.strip("._-") + base = base.replace(" ", "_").replace("-", "_").lower() + base = re.sub(r"[^a-z0-9_]", "_", base) + base = base.strip("._ ") return base or "plugin" +def _validate_fetch_url(url): + """Raise ValueError if the URL must not be fetched (SSRF prevention). + + Only http and https schemes are allowed. Hostnames that resolve to + loopback, private, link-local, or otherwise non-routable addresses + are rejected. + """ + parsed = urlparse(url) + if parsed.scheme not in ("http", "https"): + raise ValueError( + f"URL scheme '{parsed.scheme}' is not allowed; only http and https are permitted." + ) + hostname = parsed.hostname + if not hostname: + raise ValueError("URL has no hostname.") + try: + infos = socket.getaddrinfo(hostname, None) + except socket.gaierror as exc: + raise ValueError(f"Could not resolve hostname '{hostname}': {exc}") from exc + for _family, _type, _proto, _canon, sockaddr in infos: + addr_str = sockaddr[0] + try: + ip = ipaddress.ip_address(addr_str) + except ValueError: + continue + if ip.is_loopback or ip.is_link_local or ip.is_private or ip.is_reserved or ip.is_unspecified: + raise ValueError( + f"URL resolves to a non-routable address ({addr_str}) and cannot be fetched." + ) + + def _absolutize_logo_url(request, url: str | None) -> str | None: if not url or not request: return url @@ -59,7 +122,10 @@ def _absolutize_logo_url(request, url: str | None) -> str | None: return request.build_absolute_uri(url) -class PluginsListAPIView(APIView): +class PluginAuthMixin: + """Mixin that routes permission resolution through permission_classes_by_method, + falling back to Authenticated() for any method not explicitly listed.""" + def get_permissions(self): try: return [ @@ -68,6 +134,8 @@ class PluginsListAPIView(APIView): except KeyError: return [Authenticated()] + +class PluginsListAPIView(PluginAuthMixin, APIView): def get(self, request): pm = PluginManager.get() # Prefer cached registry; reload explicitly via the reload endpoint @@ -78,15 +146,7 @@ class PluginsListAPIView(APIView): return Response({"plugins": plugins}) -class PluginReloadAPIView(APIView): - def get_permissions(self): - try: - return [ - perm() for perm in permission_classes_by_method[self.request.method] - ] - except KeyError: - return [Authenticated()] - +class PluginReloadAPIView(PluginAuthMixin, APIView): def post(self, request): pm = PluginManager.get() pm.stop_all_plugins(reason="reload") @@ -94,143 +154,236 @@ class PluginReloadAPIView(APIView): return Response({"success": True, "count": len(pm._registry)}) -class PluginImportAPIView(APIView): - def get_permissions(self): +def _install_plugin_from_zip(zip_file, plugins_dir, *, file_name="plugin.zip", allow_overwrite_key=None, allow_overwrite=False): + """Extract and install a plugin from a zip file-like object. + + Args: + zip_file: File-like object containing the zip. + plugins_dir: Path to the plugins directory. + file_name: Name hint for deriving plugin key when the zip has flat contents. + allow_overwrite_key: If set, allow overwriting this specific plugin directory. + allow_overwrite: If True, allow overwriting any existing plugin with the same key. + + Returns: + dict with "success" bool, and either "plugin_key" on success or "error" on failure. + """ + try: + zf = zipfile.ZipFile(zip_file) + except zipfile.BadZipFile: + return {"success": False, "error": "Invalid zip file"} + + tmp_root = tempfile.mkdtemp(prefix="plugin_import_") + try: + file_members = [m for m in zf.infolist() if not m.is_dir()] + if not file_members: + return {"success": False, "error": "Archive is empty"} + if len(file_members) > MAX_PLUGIN_IMPORT_FILES: + return {"success": False, "error": "Archive has too many files"} + + total_size = 0 + for member in file_members: + total_size += member.file_size + if member.file_size > MAX_PLUGIN_IMPORT_FILE_BYTES: + return {"success": False, "error": "Archive contains a file that is too large"} + if total_size > MAX_PLUGIN_IMPORT_BYTES: + return {"success": False, "error": "Archive is too large"} + + for member in file_members: + name = member.filename + if not name or name.endswith("/"): + continue + norm = os.path.normpath(name) + if norm.startswith("..") or os.path.isabs(norm): + return {"success": False, "error": "Unsafe path in archive"} + dest_path = os.path.join(tmp_root, norm) + os.makedirs(os.path.dirname(dest_path), exist_ok=True) + with zf.open(member, "r") as src, open(dest_path, "wb") as dst: + shutil.copyfileobj(src, dst) + + # Single walk: find candidate plugin dirs AND logo.png simultaneously + candidates = [] + logo_candidates_raw = [] + for dirpath, dirnames, filenames in os.walk(tmp_root): + depth = len(os.path.relpath(dirpath, tmp_root).split(os.sep)) + has_pluginpy = "plugin.py" in filenames + has_init = "__init__.py" in filenames + if has_pluginpy or has_init: + candidates.append((0 if has_pluginpy else 1, depth, dirpath)) + for filename in filenames: + if filename.lower() == "logo.png": + logo_candidates_raw.append((depth, os.path.join(dirpath, filename))) + if not candidates: + return {"success": False, "error": "Invalid plugin: missing plugin.py or package __init__.py"} + + candidates.sort() + chosen = candidates[0][2] + + # Determine plugin key + base_name = os.path.splitext(file_name)[0] + plugin_key = os.path.basename(chosen.rstrip(os.sep)) + if chosen.rstrip(os.sep) == tmp_root.rstrip(os.sep): + plugin_key = base_name + plugin_key = _sanitize_plugin_key(plugin_key) + if len(plugin_key) > 128: + plugin_key = plugin_key[:128] + + # Extract logo (prefer one inside the chosen plugin dir, then shallowest) + logo_bytes = None try: - return [ - perm() for perm in permission_classes_by_method[self.request.method] - ] - except KeyError: - return [Authenticated()] - - def post(self, request): - file: UploadedFile = request.FILES.get("file") - if not file: - return Response({"success": False, "error": "Missing 'file' upload"}, status=status.HTTP_400_BAD_REQUEST) - - pm = PluginManager.get() - plugins_dir = pm.plugins_dir - - try: - zf = zipfile.ZipFile(file) - except zipfile.BadZipFile: - return Response({"success": False, "error": "Invalid zip file"}, status=status.HTTP_400_BAD_REQUEST) - - # Extract to a temporary directory first to avoid server reload thrash - tmp_root = tempfile.mkdtemp(prefix="plugin_import_") - try: - file_members = [m for m in zf.infolist() if not m.is_dir()] - if not file_members: - shutil.rmtree(tmp_root, ignore_errors=True) - return Response({"success": False, "error": "Archive is empty"}, status=status.HTTP_400_BAD_REQUEST) - if len(file_members) > MAX_PLUGIN_IMPORT_FILES: - shutil.rmtree(tmp_root, ignore_errors=True) - return Response({"success": False, "error": "Archive has too many files"}, status=status.HTTP_400_BAD_REQUEST) - - total_size = 0 - for member in file_members: - total_size += member.file_size - if member.file_size > MAX_PLUGIN_IMPORT_FILE_BYTES: - shutil.rmtree(tmp_root, ignore_errors=True) - return Response({"success": False, "error": "Archive contains a file that is too large"}, status=status.HTTP_400_BAD_REQUEST) - if total_size > MAX_PLUGIN_IMPORT_BYTES: - shutil.rmtree(tmp_root, ignore_errors=True) - return Response({"success": False, "error": "Archive is too large"}, status=status.HTTP_400_BAD_REQUEST) - - for member in file_members: - name = member.filename - if not name or name.endswith("/"): - continue - # Normalize and prevent path traversal - norm = os.path.normpath(name) - if norm.startswith("..") or os.path.isabs(norm): - shutil.rmtree(tmp_root, ignore_errors=True) - return Response({"success": False, "error": "Unsafe path in archive"}, status=status.HTTP_400_BAD_REQUEST) - dest_path = os.path.join(tmp_root, norm) - os.makedirs(os.path.dirname(dest_path), exist_ok=True) - with zf.open(member, 'r') as src, open(dest_path, 'wb') as dst: - shutil.copyfileobj(src, dst) - - # Find candidate directory containing plugin.py or __init__.py - candidates = [] - for dirpath, dirnames, filenames in os.walk(tmp_root): - has_pluginpy = "plugin.py" in filenames - has_init = "__init__.py" in filenames - if has_pluginpy or has_init: - depth = len(os.path.relpath(dirpath, tmp_root).split(os.sep)) - candidates.append((0 if has_pluginpy else 1, depth, dirpath)) - if not candidates: - shutil.rmtree(tmp_root, ignore_errors=True) - return Response({"success": False, "error": "Invalid plugin: missing plugin.py or package __init__.py"}, status=status.HTTP_400_BAD_REQUEST) - - candidates.sort() - chosen = candidates[0][2] - # Determine plugin key: prefer chosen folder name; if chosen is tmp_root, use zip base name - base_name = os.path.splitext(getattr(file, "name", "plugin"))[0] - plugin_key = os.path.basename(chosen.rstrip(os.sep)) - if chosen.rstrip(os.sep) == tmp_root.rstrip(os.sep): - plugin_key = base_name - plugin_key = _sanitize_plugin_key(plugin_key) - if len(plugin_key) > 128: - plugin_key = plugin_key[:128] + chosen_abs = os.path.abspath(chosen) + logo_candidates = [] + for depth, full_path in logo_candidates_raw: + full_abs = os.path.abspath(full_path) + try: + in_chosen = os.path.commonpath([chosen_abs, full_abs]) == chosen_abs + except Exception: + in_chosen = False + logo_candidates.append((0 if in_chosen else 1, depth, full_path)) + if logo_candidates: + logo_candidates.sort() + with open(logo_candidates[0][2], "rb") as fh: + logo_bytes = fh.read() + except Exception: logo_bytes = None - try: - logo_candidates = [] - chosen_abs = os.path.abspath(chosen) - for dirpath, _, filenames in os.walk(tmp_root): - for filename in filenames: - if filename.lower() != "logo.png": - continue - full_path = os.path.join(dirpath, filename) - full_abs = os.path.abspath(full_path) - try: - in_chosen = os.path.commonpath([chosen_abs, full_abs]) == chosen_abs - except Exception: - in_chosen = False - depth = len(os.path.relpath(dirpath, tmp_root).split(os.sep)) - logo_candidates.append((0 if in_chosen else 1, depth, full_path)) - if logo_candidates: - logo_candidates.sort() - with open(logo_candidates[0][2], "rb") as fh: - logo_bytes = fh.read() - except Exception: - logo_bytes = None - final_dir = os.path.join(plugins_dir, plugin_key) - if os.path.exists(final_dir): - # If final dir exists but contains a valid plugin, refuse; otherwise clear it - if os.path.exists(os.path.join(final_dir, "plugin.py")) or os.path.exists(os.path.join(final_dir, "__init__.py")): - shutil.rmtree(tmp_root, ignore_errors=True) - return Response({"success": False, "error": f"Plugin '{plugin_key}' already exists"}, status=status.HTTP_400_BAD_REQUEST) + final_dir = os.path.join(plugins_dir, plugin_key) + should_overwrite = (allow_overwrite_key and plugin_key == allow_overwrite_key) or allow_overwrite + if os.path.exists(final_dir): + if should_overwrite: + # Atomic swap: rename old to backup, move new in, delete backup + backup_dir = final_dir + ".__backup__" + try: + if os.path.exists(backup_dir): + shutil.rmtree(backup_dir) + os.rename(final_dir, backup_dir) + except Exception as e: + return {"success": False, "error": f"Failed to back up existing plugin: {e}"} + try: + if chosen.rstrip(os.sep) == tmp_root.rstrip(os.sep): + os.makedirs(final_dir, exist_ok=True) + for item in os.listdir(tmp_root): + shutil.move(os.path.join(tmp_root, item), os.path.join(final_dir, item)) + else: + shutil.move(chosen, final_dir) + if logo_bytes: + try: + with open(os.path.join(final_dir, "logo.png"), "wb") as fh: + fh.write(logo_bytes) + except Exception: + pass + # Success - remove backup + shutil.rmtree(backup_dir, ignore_errors=True) + return {"success": True, "plugin_key": plugin_key} + except Exception as e: + # Rollback: restore backup + logger.exception("Failed to install updated plugin; rolling back") + shutil.rmtree(final_dir, ignore_errors=True) + try: + os.rename(backup_dir, final_dir) + except Exception: + logger.exception("Failed to rollback plugin backup") + return {"success": False, "error": f"Failed to install updated plugin: {e}"} + elif os.path.exists(os.path.join(final_dir, "plugin.py")) or os.path.exists(os.path.join(final_dir, "__init__.py")): + return {"success": False, "error": f"Plugin '{plugin_key}' already exists"} + else: try: shutil.rmtree(final_dir) except Exception: pass - # Move chosen directory into final location - if chosen.rstrip(os.sep) == tmp_root.rstrip(os.sep): - # Move all contents into final_dir - os.makedirs(final_dir, exist_ok=True) - for item in os.listdir(tmp_root): - shutil.move(os.path.join(tmp_root, item), os.path.join(final_dir, item)) - else: - shutil.move(chosen, final_dir) - if logo_bytes: - try: - with open(os.path.join(final_dir, "logo.png"), "wb") as fh: - fh.write(logo_bytes) - except Exception: - pass - # Cleanup temp - shutil.rmtree(tmp_root, ignore_errors=True) - target_dir = final_dir - finally: + # Move plugin files into final location + if chosen.rstrip(os.sep) == tmp_root.rstrip(os.sep): + os.makedirs(final_dir, exist_ok=True) + for item in os.listdir(tmp_root): + shutil.move(os.path.join(tmp_root, item), os.path.join(final_dir, item)) + else: + shutil.move(chosen, final_dir) + + if logo_bytes: try: - shutil.rmtree(tmp_root, ignore_errors=True) + with open(os.path.join(final_dir, "logo.png"), "wb") as fh: + fh.write(logo_bytes) except Exception: pass + return {"success": True, "plugin_key": plugin_key} + finally: + shutil.rmtree(tmp_root, ignore_errors=True) + + +def _save_fetched_manifest_to_repo(repo, data, verified): + """Validate and persist a freshly-fetched manifest onto a PluginRepo. + + Validates that 'registry_name' is present and not official-sounding (for + non-official repos). On success, updates repo fields and saves to DB. + + Returns an error string if validation fails, or None on success. + Does *not* call _unmanage_dropped_slugs — caller does that when needed. + """ + manifest_inner = data.get("manifest", data) + registry_name = (manifest_inner.get("registry_name") or "").strip() + if not registry_name: + return "Manifest is missing a 'registry_name'. The repo maintainer must set this field." + if not repo.is_official and _is_official_sounding(registry_name): + return f"The registry name '{registry_name}' is not allowed because it may be confused with an official repo." + repo.cached_manifest = data + repo.last_fetched = timezone.now() + repo.last_fetch_status = "200" + repo.name = registry_name + repo.signature_verified = verified + repo.save(update_fields=["name", "cached_manifest", "signature_verified", "last_fetched", "last_fetch_status", "updated_at"]) + return None + + +def _unmanage_dropped_slugs(repo, new_manifest_data): + """After a manifest refresh, clear source_repo on any installed plugins + whose slug is no longer listed in the repo's manifest. Also syncs the + 'deprecated' flag for all plugins that remain managed by this repo.""" + manifest = new_manifest_data.get("manifest", new_manifest_data) + plugin_entries = {p["slug"]: p for p in manifest.get("plugins", []) if p.get("slug")} + current_slugs = set(plugin_entries.keys()) + + dropped = PluginConfig.objects.filter(source_repo=repo).exclude(slug__in=current_slugs) + count = dropped.update(source_repo=None, slug="", deprecated=False) + if count: + logger.info( + "Unmanaged %d plugin(s) removed from repo '%s' manifest", + count, repo.name, + ) + + # Sync deprecated flag for retained managed plugins + for cfg in PluginConfig.objects.filter(source_repo=repo, slug__in=current_slugs): + new_deprecated = bool(plugin_entries.get(cfg.slug, {}).get("deprecated", False)) + if cfg.deprecated != new_deprecated: + cfg.deprecated = new_deprecated + cfg.save(update_fields=["deprecated", "updated_at"]) + + +class PluginImportAPIView(PluginAuthMixin, APIView): + def post(self, request): + file: UploadedFile = request.FILES.get("file") + if not file: + return Response({"success": False, "error": "Missing 'file' upload"}, status=status.HTTP_400_BAD_REQUEST) + + # Manual imports default to non-overwrite; require explicit flag to replace existing plugins + overwrite_flag = bool(request.data.get("overwrite")) + + pm = PluginManager.get() + result = _install_plugin_from_zip( + file, pm.plugins_dir, + file_name=getattr(file, "name", "plugin.zip"), + allow_overwrite=overwrite_flag, + ) + if not result["success"]: + return Response( + {"success": False, "error": result["error"]}, + status=status.HTTP_400_BAD_REQUEST, + ) + + plugin_key = result["plugin_key"] + # Ensure DB config exists (untrusted plugins are registered without loading) + was_managed = False try: cfg, _ = PluginConfig.objects.get_or_create( key=plugin_key, @@ -241,6 +394,13 @@ class PluginImportAPIView(APIView): "settings": {}, }, ) + # Manual install always breaks the managed relationship + if cfg and cfg.source_repo_id: + was_managed = True + cfg.source_repo = None + cfg.slug = "" + cfg.save(update_fields=["source_repo", "slug", "updated_at"]) + logger.info("Plugin '%s' manually replaced - cleared managed repo link", plugin_key) except Exception: cfg = None @@ -253,9 +413,9 @@ class PluginImportAPIView(APIView): plugin_entry = None if not plugin_entry: - logo_path = os.path.join(plugins_dir, plugin_key, "logo.png") + logo_path = os.path.join(pm.plugins_dir, plugin_key, "logo.png") logo_url = f"/api/plugins/plugins/{plugin_key}/logo/" if os.path.isfile(logo_path) else None - legacy = not os.path.isfile(os.path.join(plugins_dir, plugin_key, "plugin.json")) + legacy = not os.path.isfile(os.path.join(pm.plugins_dir, plugin_key, "plugin.json")) plugin_entry = { "key": plugin_key, "name": cfg.name if cfg else plugin_key, @@ -273,18 +433,10 @@ class PluginImportAPIView(APIView): } plugin_entry["logo_url"] = _absolutize_logo_url(request, plugin_entry.get("logo_url")) - return Response({"success": True, "plugin": plugin_entry}) + return Response({"success": True, "plugin": plugin_entry, "was_managed": was_managed}) -class PluginSettingsAPIView(APIView): - def get_permissions(self): - try: - return [ - perm() for perm in permission_classes_by_method[self.request.method] - ] - except KeyError: - return [Authenticated()] - +class PluginSettingsAPIView(PluginAuthMixin, APIView): def post(self, request, key): pm = PluginManager.get() data = request.data or {} @@ -296,15 +448,7 @@ class PluginSettingsAPIView(APIView): return Response({"success": False, "error": str(e)}, status=status.HTTP_400_BAD_REQUEST) -class PluginRunAPIView(APIView): - def get_permissions(self): - try: - return [ - perm() for perm in permission_classes_by_method[self.request.method] - ] - except KeyError: - return [Authenticated()] - +class PluginRunAPIView(PluginAuthMixin, APIView): def post(self, request, key): pm = PluginManager.get() action = request.data.get("action") @@ -330,15 +474,7 @@ class PluginRunAPIView(APIView): return Response({"success": False, "error": str(e)}, status=status.HTTP_500_INTERNAL_SERVER_ERROR) -class PluginEnabledAPIView(APIView): - def get_permissions(self): - try: - return [ - perm() for perm in permission_classes_by_method[self.request.method] - ] - except KeyError: - return [Authenticated()] - +class PluginEnabledAPIView(PluginAuthMixin, APIView): def post(self, request, key): enabled_raw = request.data.get("enabled") if enabled_raw is None: @@ -397,15 +533,7 @@ class PluginLogoAPIView(APIView): return FileResponse(open(logo_path, "rb"), content_type="image/png") -class PluginDeleteAPIView(APIView): - def get_permissions(self): - try: - return [ - perm() for perm in permission_classes_by_method[self.request.method] - ] - except KeyError: - return [Authenticated()] - +class PluginDeleteAPIView(PluginAuthMixin, APIView): def delete(self, request, key): pm = PluginManager.get() try: @@ -436,3 +564,739 @@ class PluginDeleteAPIView(APIView): # Reload registry pm.discover_plugins(force_reload=True) return Response({"success": True}) + + +# --------------------------------------------------------------------------- +# Plugin Repo (Hub / Store) views +# --------------------------------------------------------------------------- + +MANIFEST_FETCH_TIMEOUT = 15 + +OFFICIAL_KEY_PATH = os.path.join( + os.path.dirname(__file__), "keys", "dispatcharr-plugins.pub" +) + + +def _normalize_pgp_key(text): + """Ensure PGP public key text has armor header/footer.""" + if not text or not text.strip(): + return text + text = text.strip() + if "-----BEGIN PGP PUBLIC KEY BLOCK-----" not in text: + text = "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\n" + text + if "-----END PGP PUBLIC KEY BLOCK-----" not in text: + text = text + "\n-----END PGP PUBLIC KEY BLOCK-----" + return text + + +def _verify_manifest_signature(manifest_obj, signature_armored, public_key_text=None): + """Verify a detached GPG signature over the canonical manifest JSON. + + *signature_armored* is the PGP armored signature string from the manifest. + *public_key_text* is an armored PGP public-key string (for third-party + repos). When *None* the bundled official key is used instead. + + Returns True if valid, False if invalid/error, None if verification + could not be attempted (no signature, no key, gnupg missing, etc.). + """ + if not signature_armored: + return None + + # Determine which key material to use + key_text = None + if public_key_text: + key_text = _normalize_pgp_key(public_key_text) + elif os.path.isfile(OFFICIAL_KEY_PATH): + with open(OFFICIAL_KEY_PATH, "r") as fh: + key_text = fh.read() + + if not key_text: + logger.debug("No GPG public key available; skipping verification") + return None + + try: + import gnupg + except ImportError: + logger.debug("python-gnupg not installed; skipping signature verification") + return None + + tmp_home = tempfile.mkdtemp(prefix="gpg_verify_") + try: + gpg = gnupg.GPG(gnupghome=tmp_home) + import_result = gpg.import_keys(key_text) + if not import_result.fingerprints: + logger.warning("Failed to import GPG public key") + return None + + # Must match what the signing script produces: jq -c '.manifest' + # which uses compact separators, preserves key order, and appends \n. + manifest_bytes = ( + json.dumps(manifest_obj, separators=(",", ":")) + "\n" + ).encode("utf-8") + + # Write the PGP armored signature directly to file + sig_path = os.path.join(tmp_home, "manifest.sig") + with open(sig_path, "w") as sf: + sf.write(signature_armored) + + verified = gpg.verify_data(sig_path, manifest_bytes) + return bool(verified) + except Exception: + logger.exception("GPG signature verification error") + return False + finally: + shutil.rmtree(tmp_home, ignore_errors=True) + + +_OFFICIAL_NAME_PATTERNS = [ + "official", + "official repo", + "dispatcharr plugins", + "dispatcharr repo", + "dispatcharr official", +] + + +def _is_official_sounding(name): + """Return True if the name could be mistaken for an official repo.""" + lower = (name or "").lower().strip() + return any(pat in lower for pat in _OFFICIAL_NAME_PATTERNS) + + +def _fetch_manifest(url, public_key_text=None): + """Fetch a remote manifest JSON, validate structure, return (data, verified).""" + _validate_fetch_url(url) + with http_requests.get(url, timeout=MANIFEST_FETCH_TIMEOUT, stream=True) as resp: + resp.raise_for_status() + body = b"".join(resp.iter_content(8192)) + data = json.loads(body) + # Accept both top-level {manifest: {plugins: [...]}} and {plugins: [...]} + if "manifest" in data and "plugins" in data["manifest"]: + signature = data.get("signature") + verified = _verify_manifest_signature( + data["manifest"], signature, public_key_text + ) + return data, verified + if "plugins" in data: + return {"manifest": data}, None + raise ValueError("Manifest JSON missing 'manifest.plugins' list") + + +class PluginRepoListCreateAPIView(PluginAuthMixin, APIView): + @extend_schema( + description="List all plugin repositories.", + responses={200: PluginRepoSerializer(many=True)}, + ) + def get(self, request): + repos = PluginRepo.objects.all() + return Response(PluginRepoSerializer(repos, many=True).data) + + @extend_schema( + description="Add a new plugin repository by manifest URL. Fetches and validates the manifest immediately.", + request=PluginRepoSerializer, + responses={201: PluginRepoSerializer, 400: inline_serializer(name="RepoAddError", fields={"error": serializers.CharField()})}, + ) + def post(self, request): + serializer = PluginRepoSerializer(data=request.data) + serializer.is_valid(raise_exception=True) + repo = serializer.save(name="") + # Fetch manifest and validate + save + try: + key_text = repo.public_key if not repo.is_official else None + data, verified = _fetch_manifest(repo.url, public_key_text=key_text) + except Exception as e: + logger.warning("Initial manifest fetch failed for %s: %s", repo.url, e) + repo.delete() + return Response( + {"error": "Failed to fetch manifest. Check the URL and try again."}, + status=status.HTTP_400_BAD_REQUEST, + ) + err = _save_fetched_manifest_to_repo(repo, data, verified) + if err: + repo.delete() + return Response({"error": err}, status=status.HTTP_400_BAD_REQUEST) + return Response( + PluginRepoSerializer(repo).data, status=status.HTTP_201_CREATED + ) + + +class PluginRepoPreviewAPIView(PluginAuthMixin, APIView): + """Fetch and validate a manifest URL without saving anything.""" + + @extend_schema( + description="Preview a manifest URL: fetch and validate without saving. Returns validity, repo name, signature status, and plugin count.", + request=inline_serializer(name="RepoPreviewRequest", fields={ + "url": serializers.URLField(), + "public_key": serializers.CharField(required=False, allow_blank=True), + }), + responses={200: inline_serializer(name="RepoPreviewResponse", fields={ + "valid": serializers.BooleanField(), + "registry_name": serializers.CharField(), + "registry_url": serializers.CharField(), + "signature_verified": serializers.BooleanField(allow_null=True), + "plugin_count": serializers.IntegerField(), + "errors": serializers.ListField(child=serializers.CharField()), + })}, + ) + def post(self, request): + url = (request.data.get("url") or "").strip() + public_key = (request.data.get("public_key") or "").strip() + if not url: + return Response( + {"error": "url is required"}, + status=status.HTTP_400_BAD_REQUEST, + ) + try: + key_text = public_key or None + data, verified = _fetch_manifest(url, public_key_text=key_text) + manifest_inner = data.get("manifest", data) + registry_name = (manifest_inner.get("registry_name") or "").strip() + registry_url = (manifest_inner.get("registry_url") or "").strip() + plugin_count = len(manifest_inner.get("plugins", [])) + errors = [] + if not registry_name: + errors.append("Manifest is missing a 'registry_name'.") + elif _is_official_sounding(registry_name): + errors.append(f"The registry name '{registry_name}' is not allowed because it may be confused with an official repo.") + if PluginRepo.objects.filter(url=url).exists(): + errors.append("This manifest URL has already been added.") + return Response({ + "valid": len(errors) == 0, + "registry_name": registry_name, + "registry_url": registry_url, + "signature_verified": verified, + "plugin_count": plugin_count, + "errors": errors, + }) + except http_requests.exceptions.Timeout: + return Response( + {"valid": False, "errors": ["The request timed out. Check the URL and try again."]}, + status=status.HTTP_200_OK, + ) + except http_requests.exceptions.ConnectionError: + return Response( + {"valid": False, "errors": ["Could not connect to the server. Check the URL and your network connection."]}, + status=status.HTTP_200_OK, + ) + except http_requests.exceptions.HTTPError as e: + code = e.response.status_code if e.response is not None else None + if code == 404: + msg = "Manifest not found (404). Check that the URL points to a valid manifest file." + elif code == 403: + msg = "Access denied (403). The server refused the request." + elif code is not None: + msg = f"The server returned an error ({code}). Check the URL and try again." + else: + msg = "The server returned an unexpected error. Check the URL and try again." + return Response( + {"valid": False, "errors": [msg]}, + status=status.HTTP_200_OK, + ) + except (json.JSONDecodeError, ValueError) as e: + msg = str(e) + # Pass through messages from _validate_fetch_url and _fetch_manifest + # as-is; only substitute the generic JSON message for actual parse errors. + if "missing" in msg.lower() and "plugins" in msg.lower(): + friendly = msg + elif any(kw in msg.lower() for kw in ("non-routable", "scheme", "hostname", "resolve")): + friendly = msg + else: + friendly = "The URL did not return valid JSON. Make sure it points directly to a manifest .json file." + return Response( + {"valid": False, "errors": [friendly]}, + status=status.HTTP_200_OK, + ) + except Exception as e: + return Response( + {"valid": False, "errors": ["An unexpected error occurred. Check the URL and try again."]}, + status=status.HTTP_200_OK, + ) + + +class PluginRepoDetailAPIView(PluginAuthMixin, APIView): + @extend_schema( + description="Update a plugin repository (e.g. public key).", + request=PluginRepoSerializer, + responses={200: PluginRepoSerializer, 404: inline_serializer(name="RepoNotFound", fields={"error": serializers.CharField()})}, + ) + def put(self, request, pk): + try: + repo = PluginRepo.objects.get(pk=pk) + except PluginRepo.DoesNotExist: + return Response( + {"error": "Repo not found"}, status=status.HTTP_404_NOT_FOUND + ) + # Only public_key and enabled are mutable after creation. + # url, is_official, name, etc. must not be changed via the API. + ALLOWED_FIELDS = {"public_key", "enabled"} + update_data = {k: v for k, v in request.data.items() if k in ALLOWED_FIELDS} + serializer = PluginRepoSerializer(repo, data=update_data, partial=True) + serializer.is_valid(raise_exception=True) + serializer.save() + return Response(PluginRepoSerializer(repo).data) + + @extend_schema( + description="Remove a plugin repository.", + responses={200: inline_serializer(name="RepoDeleteSuccess", fields={"success": serializers.BooleanField()}), 404: inline_serializer(name="RepoDeleteNotFound", fields={"error": serializers.CharField()})}, + ) + def delete(self, request, pk): + try: + repo = PluginRepo.objects.get(pk=pk) + except PluginRepo.DoesNotExist: + return Response( + {"error": "Repo not found"}, status=status.HTTP_404_NOT_FOUND + ) + if repo.is_official: + return Response( + {"error": "Cannot delete the official repository"}, + status=status.HTTP_400_BAD_REQUEST, + ) + repo.delete() + return Response({"success": True}) + + +class PluginRepoRefreshAPIView(PluginAuthMixin, APIView): + @extend_schema( + description="Re-fetch and update the cached manifest for a plugin repository.", + request=None, + responses={200: PluginRepoSerializer, 404: inline_serializer(name="RepoRefreshNotFound", fields={"error": serializers.CharField()}), 502: inline_serializer(name="RepoRefreshError", fields={"error": serializers.CharField()})}, + ) + def post(self, request, pk): + try: + repo = PluginRepo.objects.get(pk=pk) + except PluginRepo.DoesNotExist: + return Response( + {"error": "Repo not found"}, status=status.HTTP_404_NOT_FOUND + ) + try: + key_text = repo.public_key if not repo.is_official else None + data, verified = _fetch_manifest(repo.url, public_key_text=key_text) + except Exception as e: + logger.exception("Manifest fetch failed for %s", repo.url) + return Response( + {"error": "Failed to fetch manifest. Check the URL and try again."}, + status=status.HTTP_502_BAD_GATEWAY, + ) + err = _save_fetched_manifest_to_repo(repo, data, verified) + if err: + return Response({"error": err}, status=status.HTTP_400_BAD_REQUEST) + _unmanage_dropped_slugs(repo, data) + return Response(PluginRepoSerializer(repo).data) + + +class AvailablePluginsAPIView(PluginAuthMixin, APIView): + """Aggregate plugins from all enabled repo manifests.""" + + @extend_schema( + description="Return the aggregated list of available plugins from all enabled repositories, annotated with installation status.", + responses={200: inline_serializer(name="AvailablePluginsResponse", fields={ + "plugins": serializers.ListField(child=serializers.DictField()), + })}, + ) + def get(self, request): + repos = PluginRepo.objects.filter(enabled=True) + configs = list(PluginConfig.objects.select_related("source_repo").all()) + # Build lookup: slug -> (version, repo_id, repo_name) for managed plugins, + # plus key -> version for all plugins (legacy matching) + installed_by_slug = {} + installed_by_key = {} + # Secondary dict keyed by dash-to-underscore-normalized key, for backward compat + # with existing DB entries that were saved before normalization was enforced. + installed_by_key_norm = {} + for cfg in configs: + installed_by_key[cfg.key] = cfg.version + installed_by_key_norm[cfg.key.replace("-", "_")] = cfg.key + if cfg.slug: + installed_by_slug[cfg.slug] = { + "version": cfg.version, + "source_repo_id": cfg.source_repo_id, + "source_repo_name": cfg.source_repo.name if cfg.source_repo else None, + "is_prerelease": cfg.installed_version_is_prerelease, + } + # Also index by normalized slug so a dash-variant in the manifest still matches + norm_slug = cfg.slug.replace("-", "_") + if norm_slug not in installed_by_slug: + installed_by_slug[norm_slug] = installed_by_slug[cfg.slug] + + plugins = [] + for repo in repos: + manifest_data = repo.cached_manifest or {} + manifest = manifest_data.get("manifest", manifest_data) + root_url = manifest.get("root_url", "").rstrip("/") + registry_url = manifest.get("registry_url", "").rstrip("/") + repo_plugins = manifest.get("plugins", []) + for p in repo_plugins: + slug = p.get("slug", "") + plugin_data = {**p} + # Resolve relative URLs against root_url; absolute URLs pass through + if root_url: + for url_field in ("manifest_url", "latest_url", "icon_url"): + val = plugin_data.get(url_field, "") + if val and not val.startswith(("http://", "https://")): + plugin_data[url_field] = f"{root_url}/{val}" + # Fallback icon_url from main branch when not provided + if not plugin_data.get("icon_url") and registry_url: + # registry_url is e.g. https://github.com/Dispatcharr/Plugins + # Convert to raw.githubusercontent.com URL for the main branch + raw_base = registry_url.replace( + "https://github.com/", "https://raw.githubusercontent.com/" + ) + plugin_data["icon_url"] = f"{raw_base}/refs/heads/main/plugins/{slug}/logo.png" + # Determine install status + managed = installed_by_slug.get(slug) or installed_by_slug.get(slug.replace("-", "_")) + sanitized_slug = _sanitize_plugin_key(slug) + key_match = sanitized_slug in installed_by_key or sanitized_slug in installed_by_key_norm + if managed: + is_installed = True + installed_version = managed["version"] + latest = plugin_data.get("latest_version") + if managed["source_repo_id"] == repo.id: + if installed_version and latest and installed_version != latest and not managed.get("is_prerelease"): + install_status = "update_available" + else: + install_status = "installed" + else: + install_status = "different_repo" + elif key_match: + is_installed = True + installed_version = installed_by_key.get(sanitized_slug) or installed_by_key.get( + installed_by_key_norm.get(sanitized_slug, sanitized_slug) + ) + install_status = "unmanaged" + else: + is_installed = False + installed_version = None + install_status = "not_installed" + entry = { + **plugin_data, + "repo_id": repo.id, + "repo_name": repo.name, + "is_official_repo": repo.is_official, + "signature_verified": repo.signature_verified, + "installed": is_installed, + "installed_version": installed_version, + "installed_version_is_prerelease": managed.get("is_prerelease", False) if managed else False, + "install_status": install_status, + "key": _sanitize_plugin_key(slug), + } + if install_status == "different_repo": + entry["installed_source_repo_name"] = managed["source_repo_name"] + plugins.append(entry) + return Response({"plugins": plugins}) + + +class PluginDetailManifestAPIView(PluginAuthMixin, APIView): + """Fetch and verify a per-plugin manifest given repo_id and manifest_url.""" + + @extend_schema( + description="Fetch and GPG-verify a per-plugin manifest from a repo, resolving relative URLs against the repo root.", + request=inline_serializer(name="PluginDetailManifestRequest", fields={ + "repo_id": serializers.IntegerField(), + "manifest_url": serializers.URLField(), + }), + responses={200: inline_serializer(name="PluginDetailManifestResponse", fields={ + "manifest": serializers.DictField(), + "signature_verified": serializers.BooleanField(allow_null=True), + }), 502: inline_serializer(name="PluginDetailManifestError", fields={"error": serializers.CharField()})}, + ) + def post(self, request): + repo_id = request.data.get("repo_id") + manifest_url = request.data.get("manifest_url") + if not repo_id or not manifest_url: + return Response( + {"error": "repo_id and manifest_url are required"}, + status=status.HTTP_400_BAD_REQUEST, + ) + try: + repo = PluginRepo.objects.get(pk=repo_id) + except PluginRepo.DoesNotExist: + return Response( + {"error": "Repo not found"}, status=status.HTTP_404_NOT_FOUND + ) + try: + _validate_fetch_url(manifest_url) + except ValueError as e: + return Response({"error": str(e)}, status=status.HTTP_400_BAD_REQUEST) + try: + resp = http_requests.get(manifest_url, timeout=MANIFEST_FETCH_TIMEOUT) + resp.raise_for_status() + data = resp.json() + + signature = data.get("signature") + manifest_obj = data.get("manifest", data) + verified = _verify_manifest_signature( + manifest_obj, signature, + repo.public_key if not repo.is_official else None + ) + + # Resolve relative URLs in versions + repo_manifest = repo.cached_manifest or {} + inner = repo_manifest.get("manifest", repo_manifest) + root_url = inner.get("root_url", "").rstrip("/") + + if root_url and isinstance(manifest_obj.get("versions"), list): + for v in manifest_obj["versions"]: + url_val = v.get("url", "") + if url_val and not url_val.startswith(("http://", "https://")): + v["url"] = f"{root_url}/{url_val}" + if root_url and isinstance(manifest_obj.get("latest"), dict): + for url_field in ("url", "latest_url"): + url_val = manifest_obj["latest"].get(url_field, "") + if url_val and not url_val.startswith(("http://", "https://")): + manifest_obj["latest"][url_field] = f"{root_url}/{url_val}" + + return Response({ + "manifest": manifest_obj, + "signature_verified": verified, + }) + except Exception as e: + logger.exception("Failed to fetch plugin manifest from %s", manifest_url) + return Response( + {"error": f"Failed to fetch plugin manifest: {e}"}, + status=status.HTTP_502_BAD_GATEWAY, + ) + + +class PluginInstallFromRepoAPIView(PluginAuthMixin, APIView): + """Install a plugin from a managed repo by downloading its release zip.""" + + @extend_schema( + description="Download and install a plugin release zip from a managed repository. Verifies SHA256 if provided.", + request=inline_serializer(name="PluginInstallFromRepoRequest", fields={ + "repo_id": serializers.IntegerField(), + "slug": serializers.CharField(), + "version": serializers.CharField(), + "download_url": serializers.URLField(), + "sha256": serializers.CharField(required=False, allow_blank=True), + "min_dispatcharr_version": serializers.CharField(required=False, allow_blank=True), + "max_dispatcharr_version": serializers.CharField(required=False, allow_blank=True), + }), + responses={ + 200: inline_serializer(name="PluginInstallFromRepoResponse", fields={"success": serializers.BooleanField(), "plugin": serializers.DictField()}), + 201: inline_serializer(name="PluginInstallFromRepoCreated", fields={"success": serializers.BooleanField(), "plugin": serializers.DictField()}), + 400: inline_serializer(name="PluginInstallFromRepoError", fields={"error": serializers.CharField()}), + }, + ) + def post(self, request): + repo_id = request.data.get("repo_id") + slug = request.data.get("slug") + version = request.data.get("version") + download_url = request.data.get("download_url") + + if not all([repo_id, slug, version, download_url]): + return Response( + {"error": "repo_id, slug, version, and download_url are required"}, + status=status.HTTP_400_BAD_REQUEST, + ) + + try: + repo = PluginRepo.objects.get(pk=repo_id) + except PluginRepo.DoesNotExist: + return Response( + {"error": "Repo not found"}, status=status.HTTP_404_NOT_FOUND + ) + + # Resolve the plugin key and look up any existing install + plugin_key = _sanitize_plugin_key(slug) + if len(plugin_key) > 128: + plugin_key = plugin_key[:128] + + existing_cfg = PluginConfig.objects.filter(key=plugin_key).first() + # Backward compat: if no match, also try with dashes (legacy entries saved before + # normalization was enforced) so overwrite is still allowed on update. + if not existing_cfg: + dash_key = plugin_key.replace("_", "-") + if dash_key != plugin_key: + existing_cfg = PluginConfig.objects.filter(key=dash_key).first() + + # Version compatibility check against the running Dispatcharr version + min_version = request.data.get("min_dispatcharr_version") + max_version = request.data.get("max_dispatcharr_version") + if min_version or max_version: + from version import __version__ as app_version + try: + if min_version and _compare_versions(app_version, min_version) < 0: + return Response( + {"error": f"This plugin version requires Dispatcharr {min_version} or newer (you have {app_version})"}, + status=status.HTTP_400_BAD_REQUEST, + ) + if max_version and _compare_versions(app_version, max_version) > 0: + return Response( + {"error": f"This plugin version requires Dispatcharr {max_version} or older (you have {app_version})"}, + status=status.HTTP_400_BAD_REQUEST, + ) + except (ValueError, TypeError): + logger.warning("Failed to parse version constraints: min=%s, max=%s", min_version, max_version) + + # Download the zip + try: + _validate_fetch_url(download_url) + except ValueError as e: + return Response({"error": str(e)}, status=status.HTTP_400_BAD_REQUEST) + try: + resp = http_requests.get(download_url, timeout=60, stream=True) + resp.raise_for_status() + except Exception as e: + logger.exception("Failed to download plugin from %s", download_url) + return Response( + {"error": "Failed to download plugin. Check the URL and try again."}, + status=status.HTTP_502_BAD_GATEWAY, + ) + + # SHA256 integrity check (streamed) + expected_sha256 = request.data.get("sha256", "").lower().strip() + hasher = hashlib.sha256() if expected_sha256 else None + + # Stream the response to a temporary file to avoid buffering in memory + with tempfile.NamedTemporaryFile(suffix=".zip") as tmp_file: + total = 0 + for chunk in resp.iter_content(chunk_size=8192): + if not chunk: + continue + total += len(chunk) + if total > MAX_PLUGIN_IMPORT_BYTES: + return Response( + {"error": "Download is too large"}, + status=status.HTTP_400_BAD_REQUEST, + ) + if hasher is not None: + hasher.update(chunk) + tmp_file.write(chunk) + + if expected_sha256: + actual_sha256 = hasher.hexdigest() + if actual_sha256 != expected_sha256: + logger.warning( + "SHA256 mismatch for plugin '%s' from %s: expected %s, got %s", + slug, download_url, expected_sha256, actual_sha256, + ) + return Response( + { + "error": "SHA256 integrity check failed - download discarded. The file may be corrupted or tampered with." + }, + status=status.HTTP_400_BAD_REQUEST, + ) + + # Delegate to shared install logic (allow overwrite for managed updates) + tmp_file.flush() + tmp_file.seek(0) + pm = PluginManager.get() + result = _install_plugin_from_zip( + tmp_file, + pm.plugins_dir, + file_name=f"{slug}.zip", + allow_overwrite_key=plugin_key if existing_cfg else None, + ) + if not result["success"]: + return Response( + {"success": False, "error": result["error"]}, + status=status.HTTP_400_BAD_REQUEST, + ) + + actual_key = result["plugin_key"] + + # Create/update PluginConfig with managed fields + # Use defaults for creation only; explicitly update fields on existing records + # to preserve settings, enabled, and ever_enabled + is_prerelease = bool(request.data.get("prerelease", False)) + + # Determine deprecated status from the repo's cached manifest + is_deprecated = False + manifest_data = repo.cached_manifest or {} + manifest_inner = manifest_data.get("manifest", manifest_data) + for rp in manifest_inner.get("plugins", []): + if rp.get("slug") == slug: + is_deprecated = bool(rp.get("deprecated", False)) + break + + cfg, created = PluginConfig.objects.get_or_create( + key=actual_key, + defaults={ + "name": slug, + "version": version, + "slug": slug, + "source_repo": repo, + "installed_version_is_prerelease": is_prerelease, + "deprecated": is_deprecated, + }, + ) + if not created: + cfg.version = version + cfg.slug = slug + cfg.source_repo = repo + cfg.installed_version_is_prerelease = is_prerelease + cfg.deprecated = is_deprecated + cfg.save(update_fields=["version", "slug", "source_repo", "installed_version_is_prerelease", "deprecated", "updated_at"]) + + # Reload discovery + pm.discover_plugins(force_reload=True) + plugin_entry = None + try: + plugin_entry = next( + (p for p in pm.list_plugins() if p.get("key") == actual_key), + None, + ) + except Exception: + plugin_entry = None + + return Response( + { + "success": True, + "plugin": plugin_entry or {"key": actual_key, "slug": slug, "version": version}, + }, + status=status.HTTP_201_CREATED if created else status.HTTP_200_OK, + ) + + +class PluginRepoSettingsAPIView(PluginAuthMixin, APIView): + """Get/update plugin repo refresh settings (interval in hours, 0=disabled).""" + + @extend_schema( + description="Get the plugin repository refresh interval setting.", + responses={200: inline_serializer(name="PluginRepoSettingsResponse", fields={"refresh_interval_hours": serializers.IntegerField()})}, + ) + def get(self, request): + from core.models import CoreSettings + try: + obj = CoreSettings.objects.get(key="plugin_repo_settings") + return Response(obj.value) + except CoreSettings.DoesNotExist: + return Response({"refresh_interval_hours": 6}) + + @extend_schema( + description="Update the plugin repository refresh interval (hours). Set to 0 to disable automatic refresh.", + request=inline_serializer(name="PluginRepoSettingsRequest", fields={"refresh_interval_hours": serializers.IntegerField()}), + responses={200: inline_serializer(name="PluginRepoSettingsUpdated", fields={"refresh_interval_hours": serializers.IntegerField()})}, + ) + def put(self, request): + from core.models import CoreSettings + from core.scheduling import create_or_update_periodic_task, delete_periodic_task + from .tasks import PLUGIN_REPO_REFRESH_TASK_NAME + + interval = request.data.get("refresh_interval_hours", 6) + try: + interval = int(interval) + if interval < 0: + interval = 0 + except (TypeError, ValueError): + interval = 6 + + obj, _ = CoreSettings.objects.update_or_create( + key="plugin_repo_settings", + defaults={ + "name": "Plugin Repo Settings", + "value": {"refresh_interval_hours": interval}, + }, + ) + + if interval == 0: + delete_periodic_task(PLUGIN_REPO_REFRESH_TASK_NAME) + else: + create_or_update_periodic_task( + task_name=PLUGIN_REPO_REFRESH_TASK_NAME, + celery_task_path="apps.plugins.tasks.refresh_plugin_repos", + interval_hours=interval, + enabled=True, + ) + + return Response(obj.value) diff --git a/apps/plugins/apps.py b/apps/plugins/apps.py index 3ab44cb1..c2c0bce6 100644 --- a/apps/plugins/apps.py +++ b/apps/plugins/apps.py @@ -52,3 +52,53 @@ class PluginsConfig(AppConfig): import logging logging.getLogger(__name__).exception("Plugin discovery wiring failed during app ready") + + # Register periodic task for refreshing plugin repo manifests + self._setup_repo_refresh_schedule() + + # Refresh repo manifests once at startup so the UI always has current data + self._enqueue_startup_refresh() + + def _enqueue_startup_refresh(self): + from dispatcharr.app_initialization import should_skip_initialization + if should_skip_initialization(): + return + try: + from .tasks import refresh_plugin_repos + refresh_plugin_repos.apply_async(countdown=10) + except Exception: + import logging + logging.getLogger(__name__).debug( + "Could not enqueue startup plugin repo refresh (Celery may not be ready yet)" + ) + + def _setup_repo_refresh_schedule(self): + from dispatcharr.app_initialization import should_skip_initialization + if should_skip_initialization(): + return + try: + from core.scheduling import create_or_update_periodic_task, delete_periodic_task + from core.models import CoreSettings + from .tasks import PLUGIN_REPO_REFRESH_TASK_NAME + + interval = 6 + try: + obj = CoreSettings.objects.get(key="plugin_repo_settings") + interval = obj.value.get("refresh_interval_hours", 6) + except CoreSettings.DoesNotExist: + pass + + if interval == 0: + delete_periodic_task(PLUGIN_REPO_REFRESH_TASK_NAME) + else: + create_or_update_periodic_task( + task_name=PLUGIN_REPO_REFRESH_TASK_NAME, + celery_task_path="apps.plugins.tasks.refresh_plugin_repos", + interval_hours=interval, + enabled=True, + ) + except Exception: + import logging + logging.getLogger(__name__).debug( + "Could not set up plugin repo refresh schedule (migrations may not have run yet)" + ) diff --git a/apps/plugins/keys/dispatcharr-plugins.pub b/apps/plugins/keys/dispatcharr-plugins.pub new file mode 100644 index 00000000..1f6ba60a --- /dev/null +++ b/apps/plugins/keys/dispatcharr-plugins.pub @@ -0,0 +1,11 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mDMEacgfABYJKwYBBAHaRw8BAQdAh1MuVNBxk+CExQPjOVDvAGvIk6BdGS2ce9/h +zB7lYtW0TERpc3BhdGNoYXJyIFBsdWdpbiBSZXBvIChkaXNwYXRjaGFyci1hdXRv +Z2VuZXJhdGVkKSA8cGx1Z2luc0BkaXNwYXRjaGFyci50dj6IrwQTFgoAVxYhBEap +MFaOD7nKg0zX+H7AOmtMIjTOBQJpyB8AGxSAAAAAAAQADm1hbnUyLDIuNSsxLjEy +LDAsMwIbAwULCQgHAgIiAgYVCgkICwIEFgIDAQIeBwIXgAAKCRB+wDprTCI0zvNZ +AP9r3TpMpiI8BCNo9B5M9lJ+QLRo9ihPWIcqBzJ9eFCoSQEAgguiZsNy6aJzKjIb +yDvGuoZi3I2/GNM/f2qVzFtgPQk= +=Zf/y +-----END PGP PUBLIC KEY BLOCK----- \ No newline at end of file diff --git a/apps/plugins/loader.py b/apps/plugins/loader.py index 4b53e08b..6084b28e 100644 --- a/apps/plugins/loader.py +++ b/apps/plugins/loader.py @@ -367,21 +367,35 @@ class PluginManager: obj.save() def list_plugins(self) -> List[Dict[str, Any]]: - from .models import PluginConfig + from .models import PluginConfig, PluginRepo plugins: List[Dict[str, Any]] = [] with self._lock: registry_snapshot = dict(self._registry) try: - configs = {c.key: c for c in PluginConfig.objects.all()} + configs = {c.key: c for c in PluginConfig.objects.select_related("source_repo").all()} except Exception as e: # Database might not be migrated yet; fall back to registry only logger.warning("PluginConfig table unavailable; listing registry only: %s", e) configs = {} + # Build repo latest-version lookup from cached manifests + repo_latest = {} # slug -> latest_version + try: + for repo in PluginRepo.objects.filter(enabled=True): + manifest_data = repo.cached_manifest or {} + manifest = manifest_data.get("manifest", manifest_data) + for rp in manifest.get("plugins", []): + s = rp.get("slug", "") + if s: + repo_latest[s] = rp.get("latest_version", "") + except Exception: + pass + # First, include all discovered plugins for key, lp in registry_snapshot.items(): conf = configs.get(key) + conf_slug = conf.slug if conf else "" trusted = bool(conf and (conf.ever_enabled or conf.enabled)) logo_url = self._get_logo_url(key, path=lp.path) plugins.append( @@ -393,7 +407,7 @@ class PluginManager: "author": getattr(lp, "author", "") or "", "help_url": getattr(lp, "help_url", "") or "", "enabled": conf.enabled if conf else False, - "ever_enabled": getattr(conf, "ever_enabled", False) if conf else False, + "ever_enabled": conf.ever_enabled if conf else False, "fields": lp.fields or [], "settings": (conf.settings if conf else {}), "actions": lp.actions or [], @@ -402,6 +416,22 @@ class PluginManager: "loaded": bool(lp.loaded), "legacy": bool(getattr(lp, "legacy", False)), "logo_url": logo_url, + "source_repo": conf.source_repo_id if conf else None, + "source_repo_name": conf.source_repo.name if conf and conf.source_repo else None, + "is_official_repo": bool(conf and conf.source_repo and conf.source_repo.is_official), + "slug": conf_slug, + "is_managed": bool(conf and conf.source_repo_id), + "installed_version_is_prerelease": bool( + conf and conf.installed_version_is_prerelease + ), + "update_available": bool( + conf_slug and conf and conf.source_repo_id + and not (conf and conf.installed_version_is_prerelease) + and repo_latest.get(conf_slug) + and lp.version != repo_latest.get(conf_slug) + ), + "latest_version": repo_latest.get(conf_slug, ""), + "deprecated": conf.deprecated if conf else False, } ) @@ -428,6 +458,22 @@ class PluginManager: "loaded": False, "legacy": False, "logo_url": self._get_logo_url(key), + "source_repo": conf.source_repo_id, + "source_repo_name": conf.source_repo.name if conf.source_repo else None, + "is_official_repo": bool(conf.source_repo and conf.source_repo.is_official), + "slug": conf.slug, + "is_managed": bool(conf.source_repo_id), + "installed_version_is_prerelease": bool( + conf.installed_version_is_prerelease + ), + "update_available": bool( + conf.slug and conf.source_repo_id + and not conf.installed_version_is_prerelease + and repo_latest.get(conf.slug) + and conf.version != repo_latest.get(conf.slug) + ), + "latest_version": repo_latest.get(conf.slug or "", ""), + "deprecated": conf.deprecated, } ) diff --git a/apps/plugins/migrations/0002_pluginrepo.py b/apps/plugins/migrations/0002_pluginrepo.py new file mode 100644 index 00000000..5c750fc9 --- /dev/null +++ b/apps/plugins/migrations/0002_pluginrepo.py @@ -0,0 +1,84 @@ +import django.db.models.deletion +from django.db import migrations, models + + +def seed_official_repo(apps, schema_editor): + PluginRepo = apps.get_model("plugins", "PluginRepo") + PluginRepo.objects.get_or_create( + url="https://raw.githubusercontent.com/Dispatcharr/Plugins/releases/manifest.json", + defaults={ + "name": "Dispatcharr Official", + "is_official": True, + "enabled": True, + }, + ) + + +def unseed_official_repo(apps, schema_editor): + PluginRepo = apps.get_model("plugins", "PluginRepo") + PluginRepo.objects.filter(is_official=True).delete() + + +class Migration(migrations.Migration): + + dependencies = [ + ("plugins", "0001_initial"), + ] + + operations = [ + migrations.CreateModel( + name="PluginRepo", + fields=[ + ( + "id", + models.BigAutoField( + auto_created=True, + primary_key=True, + serialize=False, + verbose_name="ID", + ), + ), + ("name", models.CharField(max_length=255)), + ("url", models.URLField(unique=True)), + ("is_official", models.BooleanField(default=False)), + ("enabled", models.BooleanField(default=True)), + ("cached_manifest", models.JSONField(blank=True, default=dict)), + ("last_fetched", models.DateTimeField(blank=True, null=True)), + ("public_key", models.TextField(blank=True, default="")), + ("signature_verified", models.BooleanField(blank=True, default=None, null=True)), + ("last_fetch_status", models.CharField(blank=True, default="", max_length=255)), + ("created_at", models.DateTimeField(auto_now_add=True)), + ("updated_at", models.DateTimeField(auto_now=True)), + ], + options={ + "ordering": ["-is_official", "name"], + }, + ), + migrations.RunPython(seed_official_repo, unseed_official_repo), + migrations.AddField( + model_name="pluginconfig", + name="source_repo", + field=models.ForeignKey( + blank=True, + null=True, + on_delete=django.db.models.deletion.SET_NULL, + related_name="installed_plugins", + to="plugins.pluginrepo", + ), + ), + migrations.AddField( + model_name="pluginconfig", + name="slug", + field=models.CharField(blank=True, default="", max_length=128), + ), + migrations.AddField( + model_name="pluginconfig", + name="installed_version_is_prerelease", + field=models.BooleanField(default=False), + ), + migrations.AddField( + model_name="pluginconfig", + name="deprecated", + field=models.BooleanField(default=False), + ), + ] diff --git a/apps/plugins/models.py b/apps/plugins/models.py index 8ae0b5be..f1960fd9 100644 --- a/apps/plugins/models.py +++ b/apps/plugins/models.py @@ -12,8 +12,52 @@ class PluginConfig(models.Model): # Tracks whether this plugin has ever been enabled at least once ever_enabled = models.BooleanField(default=False) settings = models.JSONField(default=dict, blank=True) + + # Managed plugin fields (populated when installed from a repo) + source_repo = models.ForeignKey( + "PluginRepo", + null=True, + blank=True, + on_delete=models.SET_NULL, + related_name="installed_plugins", + ) + slug = models.CharField(max_length=128, blank=True, default="") + installed_version_is_prerelease = models.BooleanField(default=False) + deprecated = models.BooleanField(default=False) + created_at = models.DateTimeField(auto_now_add=True) updated_at = models.DateTimeField(auto_now=True) + @property + def is_managed(self): + return bool(self.source_repo_id) + def __str__(self) -> str: return f"{self.name} ({self.key})" + + +OFFICIAL_REPO_URL = ( + "https://raw.githubusercontent.com/Dispatcharr/Plugins/releases/manifest.json" +) + + +class PluginRepo(models.Model): + """A remote plugin repository manifest URL.""" + + name = models.CharField(max_length=255) + url = models.URLField(unique=True) + is_official = models.BooleanField(default=False) + enabled = models.BooleanField(default=True) + cached_manifest = models.JSONField(default=dict, blank=True) + public_key = models.TextField(blank=True, default="") + signature_verified = models.BooleanField(null=True, blank=True, default=None) + last_fetched = models.DateTimeField(null=True, blank=True) + last_fetch_status = models.CharField(max_length=255, blank=True, default="") + created_at = models.DateTimeField(auto_now_add=True) + updated_at = models.DateTimeField(auto_now=True) + + class Meta: + ordering = ["-is_official", "name"] + + def __str__(self) -> str: + return self.name diff --git a/apps/plugins/serializers.py b/apps/plugins/serializers.py index 9f568054..ed2b57d7 100644 --- a/apps/plugins/serializers.py +++ b/apps/plugins/serializers.py @@ -1,4 +1,5 @@ from rest_framework import serializers +from .models import PluginRepo class PluginActionSerializer(serializers.Serializer): @@ -46,3 +47,40 @@ class PluginSerializer(serializers.Serializer): fields = PluginFieldSerializer(many=True) settings = serializers.JSONField() actions = PluginActionSerializer(many=True) + source_repo = serializers.IntegerField(required=False, allow_null=True) + slug = serializers.CharField(required=False, allow_blank=True) + is_managed = serializers.BooleanField(required=False) + deprecated = serializers.BooleanField(required=False) + + +class PluginRepoSerializer(serializers.ModelSerializer): + registry_url = serializers.SerializerMethodField() + plugin_count = serializers.SerializerMethodField() + + class Meta: + model = PluginRepo + fields = [ + "id", + "name", + "url", + "is_official", + "enabled", + "public_key", + "signature_verified", + "registry_url", + "plugin_count", + "last_fetched", + "last_fetch_status", + "created_at", + "updated_at", + ] + read_only_fields = ["id", "name", "is_official", "signature_verified", "registry_url", "plugin_count", "last_fetched", "last_fetch_status", "created_at", "updated_at"] + + def get_registry_url(self, obj): + manifest = (obj.cached_manifest or {}).get("manifest", obj.cached_manifest or {}) + return manifest.get("registry_url", "") or "" + + def get_plugin_count(self, obj): + manifest = (obj.cached_manifest or {}).get("manifest", obj.cached_manifest or {}) + plugins = manifest.get("plugins", []) + return len(plugins) if isinstance(plugins, list) else 0 diff --git a/apps/plugins/tasks.py b/apps/plugins/tasks.py new file mode 100644 index 00000000..6bd1544c --- /dev/null +++ b/apps/plugins/tasks.py @@ -0,0 +1,33 @@ +import logging +from celery import shared_task + +logger = logging.getLogger(__name__) + +PLUGIN_REPO_REFRESH_TASK_NAME = "plugin-repo-refresh-task" + + +@shared_task +def refresh_plugin_repos(): + """Refresh cached manifests for all enabled plugin repos.""" + from .models import PluginRepo + from .api_views import _fetch_manifest, _save_fetched_manifest_to_repo, _unmanage_dropped_slugs + from django.utils import timezone + + repos = PluginRepo.objects.filter(enabled=True) + for repo in repos: + try: + key_text = repo.public_key if not repo.is_official else None + data, verified = _fetch_manifest(repo.url, public_key_text=key_text) + err = _save_fetched_manifest_to_repo(repo, data, verified) + if err: + logger.warning("Skipping repo '%s': %s", repo.name, err) + continue + _unmanage_dropped_slugs(repo, data) + logger.info("Refreshed plugin repo '%s'", repo.name) + except Exception as e: + resp = getattr(e, 'response', None) + status_str = str(resp.status_code) if resp is not None and hasattr(resp, 'status_code') else type(e).__name__ + repo.last_fetch_status = status_str[:255] + repo.last_fetched = timezone.now() + repo.save(update_fields=["last_fetch_status", "last_fetched", "updated_at"]) + logger.warning("Failed to refresh plugin repo '%s': %s", repo.name, e) diff --git a/apps/proxy/hls_proxy/views.py b/apps/proxy/hls_proxy/views.py index eaf0f1cd..835b431e 100644 --- a/apps/proxy/hls_proxy/views.py +++ b/apps/proxy/hls_proxy/views.py @@ -4,6 +4,8 @@ import logging from django.http import StreamingHttpResponse, JsonResponse, HttpResponse from django.views.decorators.csrf import csrf_exempt from django.views.decorators.http import require_http_methods +from rest_framework.decorators import api_view, permission_classes +from apps.accounts.permissions import IsAdmin from .server import ProxyServer, Config logger = logging.getLogger(__name__) @@ -15,7 +17,7 @@ def stream_endpoint(request, channel_id): """Handle HLS manifest requests""" if channel_id not in proxy_server.stream_managers: return JsonResponse({'error': 'Channel not found'}, status=404) - + response = proxy_server.stream_endpoint(channel_id) return StreamingHttpResponse( response[0], @@ -30,10 +32,10 @@ def get_segment(request, segment_name): try: segment_num = int(segment_name.split('.')[0]) buffer = proxy_server.stream_buffers.get(segment_num) - + if not buffer: return JsonResponse({'error': 'Segment not found'}, status=404) - + return StreamingHttpResponse( buffer, content_type='video/MP2T' @@ -44,19 +46,19 @@ def get_segment(request, segment_name): logger.error(f"Error serving segment: {e}") return JsonResponse({'error': str(e)}, status=500) -@csrf_exempt -@require_http_methods(["POST"]) +@api_view(["POST"]) +@permission_classes([IsAdmin]) def change_stream(request, channel_id): """Change stream URL for existing channel""" try: if channel_id not in proxy_server.stream_managers: return JsonResponse({'error': 'Channel not found'}, status=404) - + data = json.loads(request.body) new_url = data.get('url') if not new_url: return JsonResponse({'error': 'No URL provided'}, status=400) - + manager = proxy_server.stream_managers[channel_id] if manager.update_url(new_url): return JsonResponse({ @@ -64,7 +66,7 @@ def change_stream(request, channel_id): 'channel': channel_id, 'url': new_url }) - + return JsonResponse({ 'message': 'URL unchanged', 'channel': channel_id, @@ -85,7 +87,7 @@ def initialize_stream(request, channel_id): url = data.get('url') if not url: return JsonResponse({'error': 'No URL provided'}, status=400) - + proxy_server.initialize_channel(url, channel_id) return JsonResponse({ 'message': 'Stream initialized', diff --git a/apps/proxy/tasks.py b/apps/proxy/tasks.py index 68843712..b376f99e 100644 --- a/apps/proxy/tasks.py +++ b/apps/proxy/tasks.py @@ -1,16 +1,11 @@ -# yourapp/tasks.py from celery import shared_task -from channels.layers import get_channel_layer -from asgiref.sync import async_to_sync -import redis import json import logging import re -import gc # Add import for garbage collection +import gc from core.utils import RedisClient from apps.proxy.ts_proxy.channel_status import ChannelStatus from core.utils import send_websocket_update -from apps.proxy.vod_proxy.connection_manager import get_connection_manager logger = logging.getLogger(__name__) @@ -31,7 +26,7 @@ def fetch_channel_stats(): while True: cursor, keys = redis_client.scan(cursor, match=channel_pattern) for key in keys: - channel_id_match = re.search(r"ts_proxy:channel:(.*):metadata", key.decode('utf-8')) + channel_id_match = re.search(r"ts_proxy:channel:(.*):metadata", key) if channel_id_match: ch_id = channel_id_match.group(1) channel_info = ChannelStatus.get_basic_channel_info(ch_id) @@ -61,12 +56,4 @@ def fetch_channel_stats(): all_channels = None gc.collect() -@shared_task -def cleanup_vod_connections(): - """Clean up stale VOD connections""" - try: - connection_manager = get_connection_manager() - connection_manager.cleanup_stale_connections(max_age_seconds=3600) # 1 hour - logger.info("VOD connection cleanup completed") - except Exception as e: - logger.error(f"Error in VOD connection cleanup: {e}", exc_info=True) + diff --git a/apps/proxy/ts_proxy/channel_status.py b/apps/proxy/ts_proxy/channel_status.py index b5012b77..c4a54cdc 100644 --- a/apps/proxy/ts_proxy/channel_status.py +++ b/apps/proxy/ts_proxy/channel_status.py @@ -39,19 +39,19 @@ class ChannelStatus: info = { 'channel_id': channel_id, - 'state': metadata.get(ChannelMetadataField.STATE.encode('utf-8'), b'unknown').decode('utf-8'), - 'url': metadata.get(ChannelMetadataField.URL.encode('utf-8'), b'').decode('utf-8'), - 'stream_profile': metadata.get(ChannelMetadataField.STREAM_PROFILE.encode('utf-8'), b'').decode('utf-8'), - 'started_at': metadata.get(ChannelMetadataField.INIT_TIME.encode('utf-8'), b'0').decode('utf-8'), - 'owner': metadata.get(ChannelMetadataField.OWNER.encode('utf-8'), b'unknown').decode('utf-8'), - 'buffer_index': int(buffer_index_value.decode('utf-8')) if buffer_index_value else 0, + 'state': metadata.get(ChannelMetadataField.STATE, 'unknown'), + 'url': metadata.get(ChannelMetadataField.URL, ''), + 'stream_profile': metadata.get(ChannelMetadataField.STREAM_PROFILE, ''), + 'started_at': metadata.get(ChannelMetadataField.INIT_TIME, '0'), + 'owner': metadata.get(ChannelMetadataField.OWNER, 'unknown'), + 'buffer_index': int(buffer_index_value) if buffer_index_value else 0, } # Add stream ID and name information - stream_id_bytes = metadata.get(ChannelMetadataField.STREAM_ID.encode('utf-8')) + stream_id_bytes = metadata.get(ChannelMetadataField.STREAM_ID) if stream_id_bytes: try: - stream_id = int(stream_id_bytes.decode('utf-8')) + stream_id = int(stream_id_bytes) info['stream_id'] = stream_id # Look up stream name from database @@ -66,10 +66,10 @@ class ChannelStatus: logger.warning(f"Invalid stream_id format in Redis: {stream_id_bytes}") # Add M3U profile information - m3u_profile_id_bytes = metadata.get(ChannelMetadataField.M3U_PROFILE.encode('utf-8')) + m3u_profile_id_bytes = metadata.get(ChannelMetadataField.M3U_PROFILE) if m3u_profile_id_bytes: try: - m3u_profile_id = int(m3u_profile_id_bytes.decode('utf-8')) + m3u_profile_id = int(m3u_profile_id_bytes) info['m3u_profile_id'] = m3u_profile_id # Look up M3U profile name from database @@ -84,22 +84,22 @@ class ChannelStatus: logger.warning(f"Invalid m3u_profile_id format in Redis: {m3u_profile_id_bytes}") # Add timing information - state_changed_field = ChannelMetadataField.STATE_CHANGED_AT.encode('utf-8') + state_changed_field = ChannelMetadataField.STATE_CHANGED_AT if state_changed_field in metadata: - state_changed_at = float(metadata[state_changed_field].decode('utf-8')) + state_changed_at = float(metadata[state_changed_field]) info['state_changed_at'] = state_changed_at info['state_duration'] = time.time() - state_changed_at - init_time_field = ChannelMetadataField.INIT_TIME.encode('utf-8') + init_time_field = ChannelMetadataField.INIT_TIME if init_time_field in metadata: - created_at = float(metadata[init_time_field].decode('utf-8')) + created_at = float(metadata[init_time_field]) info['started_at'] = created_at info['uptime'] = time.time() - created_at # Add data throughput information - total_bytes_field = ChannelMetadataField.TOTAL_BYTES.encode('utf-8') + total_bytes_field = ChannelMetadataField.TOTAL_BYTES if total_bytes_field in metadata: - total_bytes = int(metadata[total_bytes_field].decode('utf-8')) + total_bytes = int(metadata[total_bytes_field]) info['total_bytes'] = total_bytes # Format total bytes in human-readable form @@ -130,7 +130,7 @@ class ChannelStatus: stale_client_ids = [] for client_id in client_ids: - client_id_str = client_id.decode('utf-8') + client_id_str = client_id client_key = RedisKeys.client_metadata(channel_id, client_id_str) client_data = proxy_server.redis_client.hgetall(client_key) @@ -141,33 +141,35 @@ class ChannelStatus: client_info = { 'client_id': client_id_str, - 'user_agent': client_data.get(b'user_agent', b'unknown').decode('utf-8'), - 'worker_id': client_data.get(b'worker_id', b'unknown').decode('utf-8'), + 'user_agent': client_data.get('user_agent', 'unknown'), + 'worker_id': client_data.get('worker_id', 'unknown'), + 'ip_address': client_data.get('ip_address', 'unknown'), + 'user_id': client_data.get('user_id', '0'), } - if b'connected_at' in client_data: - connected_at = float(client_data[b'connected_at'].decode('utf-8')) + if 'connected_at' in client_data: + connected_at = float(client_data['connected_at']) client_info['connected_at'] = connected_at client_info['connection_duration'] = time.time() - connected_at - if b'last_active' in client_data: - last_active = float(client_data[b'last_active'].decode('utf-8')) + if 'last_active' in client_data: + last_active = float(client_data['last_active']) client_info['last_active'] = last_active client_info['last_active_ago'] = time.time() - last_active # Add transfer rate statistics - if b'bytes_sent' in client_data: - client_info['bytes_sent'] = int(client_data[b'bytes_sent'].decode('utf-8')) + if 'bytes_sent' in client_data: + client_info['bytes_sent'] = int(client_data['bytes_sent']) # Add average transfer rate - if b'avg_rate_KBps' in client_data: - client_info['avg_rate_KBps'] = float(client_data[b'avg_rate_KBps'].decode('utf-8')) - elif b'transfer_rate_KBps' in client_data: # For backward compatibility - client_info['avg_rate_KBps'] = float(client_data[b'transfer_rate_KBps'].decode('utf-8')) + if 'avg_rate_KBps' in client_data: + client_info['avg_rate_KBps'] = float(client_data['avg_rate_KBps']) + elif 'transfer_rate_KBps' in client_data: # For backward compatibility + client_info['avg_rate_KBps'] = float(client_data['transfer_rate_KBps']) # Add current transfer rate - if b'current_rate_KBps' in client_data: - client_info['current_rate_KBps'] = float(client_data[b'current_rate_KBps'].decode('utf-8')) + if 'current_rate_KBps' in client_data: + client_info['current_rate_KBps'] = float(client_data['current_rate_KBps']) clients.append(client_info) @@ -249,7 +251,7 @@ class ChannelStatus: while True: cursor, keys = proxy_server.redis_client.scan(cursor, match=buffer_key_pattern, count=100) if keys: - all_buffer_keys.extend([k.decode('utf-8') for k in keys]) + all_buffer_keys.extend([k for k in keys]) if cursor == 0 or len(all_buffer_keys) >= 20: # Limit to 20 keys break @@ -279,61 +281,64 @@ class ChannelStatus: } # Add FFmpeg stream information - video_codec = metadata.get(ChannelMetadataField.VIDEO_CODEC.encode('utf-8')) + video_codec = metadata.get(ChannelMetadataField.VIDEO_CODEC) if video_codec: - info['video_codec'] = video_codec.decode('utf-8') + info['video_codec'] = video_codec - resolution = metadata.get(ChannelMetadataField.RESOLUTION.encode('utf-8')) + resolution = metadata.get(ChannelMetadataField.RESOLUTION) if resolution: - info['resolution'] = resolution.decode('utf-8') + info['resolution'] = resolution - source_fps = metadata.get(ChannelMetadataField.SOURCE_FPS.encode('utf-8')) + source_fps = metadata.get(ChannelMetadataField.SOURCE_FPS) if source_fps: - info['source_fps'] = float(source_fps.decode('utf-8')) + info['source_fps'] = source_fps - pixel_format = metadata.get(ChannelMetadataField.PIXEL_FORMAT.encode('utf-8')) + pixel_format = metadata.get(ChannelMetadataField.PIXEL_FORMAT) if pixel_format: - info['pixel_format'] = pixel_format.decode('utf-8') + info['pixel_format'] = pixel_format - source_bitrate = metadata.get(ChannelMetadataField.SOURCE_BITRATE.encode('utf-8')) + source_bitrate = metadata.get(ChannelMetadataField.SOURCE_BITRATE) if source_bitrate: - info['source_bitrate'] = float(source_bitrate.decode('utf-8')) + info['source_bitrate'] = source_bitrate - audio_codec = metadata.get(ChannelMetadataField.AUDIO_CODEC.encode('utf-8')) + audio_codec = metadata.get(ChannelMetadataField.AUDIO_CODEC) if audio_codec: - info['audio_codec'] = audio_codec.decode('utf-8') + info['audio_codec'] = audio_codec - sample_rate = metadata.get(ChannelMetadataField.SAMPLE_RATE.encode('utf-8')) + sample_rate = metadata.get(ChannelMetadataField.SAMPLE_RATE) if sample_rate: - info['sample_rate'] = int(sample_rate.decode('utf-8')) + info['sample_rate'] = sample_rate - audio_channels = metadata.get(ChannelMetadataField.AUDIO_CHANNELS.encode('utf-8')) + audio_channels = metadata.get(ChannelMetadataField.AUDIO_CHANNELS) if audio_channels: - info['audio_channels'] = audio_channels.decode('utf-8') + info['audio_channels'] = audio_channels - audio_bitrate = metadata.get(ChannelMetadataField.AUDIO_BITRATE.encode('utf-8')) + audio_bitrate = metadata.get(ChannelMetadataField.AUDIO_BITRATE) if audio_bitrate: - info['audio_bitrate'] = float(audio_bitrate.decode('utf-8')) + info['audio_bitrate'] = audio_bitrate + # Add FFmpeg performance stats - ffmpeg_speed = metadata.get(ChannelMetadataField.FFMPEG_SPEED.encode('utf-8')) + ffmpeg_speed = metadata.get(ChannelMetadataField.FFMPEG_SPEED) if ffmpeg_speed: - info['ffmpeg_speed'] = float(ffmpeg_speed.decode('utf-8')) + info['ffmpeg_speed'] = ffmpeg_speed - ffmpeg_fps = metadata.get(ChannelMetadataField.FFMPEG_FPS.encode('utf-8')) + ffmpeg_fps = metadata.get(ChannelMetadataField.FFMPEG_FPS) if ffmpeg_fps: - info['ffmpeg_fps'] = float(ffmpeg_fps.decode('utf-8')) + info['ffmpeg_fps'] = ffmpeg_fps - actual_fps = metadata.get(ChannelMetadataField.ACTUAL_FPS.encode('utf-8')) + actual_fps = metadata.get(ChannelMetadataField.ACTUAL_FPS) if actual_fps: - info['actual_fps'] = float(actual_fps.decode('utf-8')) + info['actual_fps'] = actual_fps - ffmpeg_bitrate = metadata.get(ChannelMetadataField.FFMPEG_BITRATE.encode('utf-8')) + ffmpeg_bitrate = metadata.get(ChannelMetadataField.FFMPEG_BITRATE) if ffmpeg_bitrate: - info['ffmpeg_bitrate'] = float(ffmpeg_bitrate.decode('utf-8')) - stream_type = metadata.get(ChannelMetadataField.STREAM_TYPE.encode('utf-8')) + info['ffmpeg_bitrate'] = ffmpeg_bitrate + + stream_type = metadata.get(ChannelMetadataField.STREAM_TYPE) if stream_type: - info['stream_type'] = stream_type.decode('utf-8') + info['stream_type'] = stream_type + return info @@ -378,33 +383,27 @@ class ChannelStatus: client_count = proxy_server.redis_client.scard(client_set_key) or 0 # Calculate uptime - init_time_bytes = metadata.get(ChannelMetadataField.INIT_TIME.encode('utf-8'), b'0') - created_at = float(init_time_bytes.decode('utf-8')) + init_time_bytes = metadata.get(ChannelMetadataField.INIT_TIME, '0') + created_at = float(init_time_bytes) uptime = time.time() - created_at if created_at > 0 else 0 - # Safely decode bytes or use defaults - def safe_decode(bytes_value, default="unknown"): - if bytes_value is None: - return default - return bytes_value.decode('utf-8') - # Simplified info info = { 'channel_id': channel_id, - 'state': safe_decode(metadata.get(ChannelMetadataField.STATE.encode('utf-8'))), - 'url': safe_decode(metadata.get(ChannelMetadataField.URL.encode('utf-8')), ""), - 'stream_profile': safe_decode(metadata.get(ChannelMetadataField.STREAM_PROFILE.encode('utf-8')), ""), - 'owner': safe_decode(metadata.get(ChannelMetadataField.OWNER.encode('utf-8'))), - 'buffer_index': int(buffer_index_value.decode('utf-8')) if buffer_index_value else 0, + 'state': metadata.get(ChannelMetadataField.STATE), + 'url': metadata.get(ChannelMetadataField.URL, ""), + 'stream_profile': metadata.get(ChannelMetadataField.STREAM_PROFILE, ""), + 'owner': metadata.get(ChannelMetadataField.OWNER), + 'buffer_index': int(buffer_index_value) if buffer_index_value else 0, 'client_count': client_count, 'uptime': uptime } # Add stream ID and name information - stream_id_bytes = metadata.get(ChannelMetadataField.STREAM_ID.encode('utf-8')) + stream_id_bytes = metadata.get(ChannelMetadataField.STREAM_ID) if stream_id_bytes: try: - stream_id = int(stream_id_bytes.decode('utf-8')) + stream_id = int(stream_id_bytes) info['stream_id'] = stream_id # Look up stream name from database @@ -419,9 +418,9 @@ class ChannelStatus: logger.warning(f"Invalid stream_id format in Redis: {stream_id_bytes}") # Add data throughput information to basic info - total_bytes_bytes = proxy_server.redis_client.hget(metadata_key, ChannelMetadataField.TOTAL_BYTES.encode('utf-8')) + total_bytes_bytes = proxy_server.redis_client.hget(metadata_key, ChannelMetadataField.TOTAL_BYTES) if total_bytes_bytes: - total_bytes = int(total_bytes_bytes.decode('utf-8')) + total_bytes = int(total_bytes_bytes) info['total_bytes'] = total_bytes # Calculate and add bitrate @@ -458,25 +457,28 @@ class ChannelStatus: if client_id in stale_client_ids: continue - client_id_str = client_id.decode('utf-8') - client_key = RedisKeys.client_metadata(channel_id, client_id_str) + client_key = RedisKeys.client_metadata(channel_id, client_id) client_info = { - 'client_id': client_id_str, + 'client_id': client_id, } user_agent_bytes = proxy_server.redis_client.hget(client_key, 'user_agent') - client_info['user_agent'] = safe_decode(user_agent_bytes) + client_info['user_agent'] = user_agent_bytes ip_address_bytes = proxy_server.redis_client.hget(client_key, 'ip_address') if ip_address_bytes: - client_info['ip_address'] = safe_decode(ip_address_bytes) + client_info['ip_address'] = ip_address_bytes connected_at_bytes = proxy_server.redis_client.hget(client_key, 'connected_at') if connected_at_bytes: - connected_at = float(connected_at_bytes.decode('utf-8')) + connected_at = float(connected_at_bytes) client_info['connected_since'] = time.time() - connected_at + user_id_bytes = proxy_server.redis_client.hget(client_key, 'user_id') + if user_id_bytes: + client_info['user_id'] = user_id_bytes + clients.append(client_info) # Add clients to info @@ -484,10 +486,10 @@ class ChannelStatus: info['client_count'] = client_count # Add M3U profile information - m3u_profile_id_bytes = metadata.get(ChannelMetadataField.M3U_PROFILE.encode('utf-8')) - if m3u_profile_id_bytes: + m3u_profile_id = metadata.get(ChannelMetadataField.M3U_PROFILE) + if m3u_profile_id: try: - m3u_profile_id = int(m3u_profile_id_bytes.decode('utf-8')) + m3u_profile_id = int(m3u_profile_id) info['m3u_profile_id'] = m3u_profile_id # Look up M3U profile name from database @@ -499,32 +501,36 @@ class ChannelStatus: except (ImportError, DatabaseError) as e: logger.warning(f"Failed to get M3U profile name for ID {m3u_profile_id}: {e}") except ValueError: - logger.warning(f"Invalid m3u_profile_id format in Redis: {m3u_profile_id_bytes}") + logger.warning(f"Invalid m3u_profile_id format in Redis: {m3u_profile_id}") # Add stream info to basic info as well - video_codec = metadata.get(ChannelMetadataField.VIDEO_CODEC.encode('utf-8')) + video_codec = metadata.get(ChannelMetadataField.VIDEO_CODEC) if video_codec: - info['video_codec'] = video_codec.decode('utf-8') + info['video_codec'] = video_codec - resolution = metadata.get(ChannelMetadataField.RESOLUTION.encode('utf-8')) + resolution = metadata.get(ChannelMetadataField.RESOLUTION) if resolution: - info['resolution'] = resolution.decode('utf-8') + info['resolution'] = resolution - source_fps = metadata.get(ChannelMetadataField.SOURCE_FPS.encode('utf-8')) + source_fps = metadata.get(ChannelMetadataField.SOURCE_FPS) if source_fps: - info['source_fps'] = float(source_fps.decode('utf-8')) - ffmpeg_speed = metadata.get(ChannelMetadataField.FFMPEG_SPEED.encode('utf-8')) + info['source_fps'] = float(source_fps) + + ffmpeg_speed = metadata.get(ChannelMetadataField.FFMPEG_SPEED) if ffmpeg_speed: - info['ffmpeg_speed'] = float(ffmpeg_speed.decode('utf-8')) - audio_codec = metadata.get(ChannelMetadataField.AUDIO_CODEC.encode('utf-8')) + info['ffmpeg_speed'] = float(ffmpeg_speed) + + audio_codec = metadata.get(ChannelMetadataField.AUDIO_CODEC) if audio_codec: - info['audio_codec'] = audio_codec.decode('utf-8') - audio_channels = metadata.get(ChannelMetadataField.AUDIO_CHANNELS.encode('utf-8')) + info['audio_codec'] = audio_codec + + audio_channels = metadata.get(ChannelMetadataField.AUDIO_CHANNELS) if audio_channels: - info['audio_channels'] = audio_channels.decode('utf-8') - stream_type = metadata.get(ChannelMetadataField.STREAM_TYPE.encode('utf-8')) + info['audio_channels'] = audio_channels + + stream_type = metadata.get(ChannelMetadataField.STREAM_TYPE) if stream_type: - info['stream_type'] = stream_type.decode('utf-8') + info['stream_type'] = stream_type return info except Exception as e: diff --git a/apps/proxy/ts_proxy/client_manager.py b/apps/proxy/ts_proxy/client_manager.py index ea2aa5b0..af7eb7d3 100644 --- a/apps/proxy/ts_proxy/client_manager.py +++ b/apps/proxy/ts_proxy/client_manager.py @@ -1,10 +1,8 @@ """Client connection management for TS streams""" import threading -import logging import time import json -import gevent from typing import Set, Optional from apps.proxy.config import TSConfig as Config from redis.exceptions import ConnectionError, TimeoutError @@ -58,7 +56,8 @@ class ClientManager: from django.conf import settings redis_url = getattr(settings, 'REDIS_URL', 'redis://localhost:6379/0') - redis_client = redis.Redis.from_url(redis_url, decode_responses=True) + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) + redis_client = redis.Redis.from_url(redis_url, decode_responses=True, **ssl_params) all_channels = [] cursor = 0 @@ -129,7 +128,7 @@ class ClientManager: # Check for stale activity using last_active field last_active = self.redis_client.hget(client_key, "last_active") if last_active: - last_active_time = float(last_active.decode('utf-8')) + last_active_time = float(last_active) ghost_timeout = self.heartbeat_interval * getattr(Config, 'GHOST_CLIENT_MULTIPLIER', 5.0) if current_time - last_active_time > ghost_timeout: @@ -229,7 +228,7 @@ class ClientManager: except Exception as e: logger.error(f"Error notifying owner of client activity: {e}") - def add_client(self, client_id, client_ip, user_agent=None): + def add_client(self, client_id, client_ip, user_agent=None, user=None): """Add a client with duplicate prevention""" if client_id in self._registered_clients: logger.debug(f"Client {client_id} already registered, skipping") @@ -247,7 +246,9 @@ class ClientManager: "ip_address": client_ip, "connected_at": current_time, "last_active": current_time, - "worker_id": self.worker_id or "unknown" + "worker_id": self.worker_id or "unknown", + "user_id": str(user.id) if user is not None else "0", + # "user_level": user.user_level if user is not None else 100, # default to a high value since no user means the non-user specific M3U/HDHR } try: @@ -277,7 +278,8 @@ class ClientManager: "channel_id": self.channel_id, "client_id": client_id, "worker_id": self.worker_id or "unknown", - "timestamp": time.time() + "timestamp": time.time(), + "username": user.username if user is not None else "unknown" } if user_agent: @@ -308,8 +310,6 @@ class ClientManager: def remove_client(self, client_id): """Remove a client from this channel and Redis""" - client_ip = None - with self.lock: if client_id in self.clients: self.clients.remove(client_id) @@ -320,13 +320,11 @@ class ClientManager: self.last_active_time = time.time() if self.redis_client: - # Get client IP before removing the data + # Get client data before removing the data client_key = f"ts_proxy:channel:{self.channel_id}:clients:{client_id}" - client_data = self.redis_client.hgetall(client_key) - if client_data and b'ip_address' in client_data: - client_ip = client_data[b'ip_address'].decode('utf-8') - elif client_data and 'ip_address' in client_data: - client_ip = client_data['ip_address'] + client_username = self.redis_client.hget(client_key, "username") or "unknown" + if isinstance(client_username, bytes): + client_username = client_username.decode("utf-8") # Remove from channel's client set self.redis_client.srem(self.client_set_key, client_id) @@ -367,7 +365,8 @@ class ClientManager: "client_id": client_id, "worker_id": self.worker_id or "unknown", "timestamp": time.time(), - "remaining_clients": remaining + "remaining_clients": remaining, + "username": client_username }) self.redis_client.publish(RedisKeys.events_channel(self.channel_id), event_data) @@ -434,8 +433,7 @@ class ClientManager: client_id_list = list(client_ids) pipe = redis_client.pipeline() for cid in client_id_list: - cid_str = cid.decode('utf-8') - pipe.exists(RedisKeys.client_metadata(channel_id, cid_str)) + pipe.exists(RedisKeys.client_metadata(channel_id, cid)) results = pipe.execute() stale_ids = [ diff --git a/apps/proxy/ts_proxy/server.py b/apps/proxy/ts_proxy/server.py index b72b350a..b5820ca7 100644 --- a/apps/proxy/ts_proxy/server.py +++ b/apps/proxy/ts_proxy/server.py @@ -166,6 +166,7 @@ class ProxyServer: redis_password = os.environ.get("REDIS_PASSWORD", getattr(settings, 'REDIS_PASSWORD', '')) redis_user = os.environ.get("REDIS_USER", getattr(settings, 'REDIS_USER', '')) + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) pubsub_client = redis.Redis( host=redis_host, port=redis_port, @@ -175,7 +176,9 @@ class ProxyServer: socket_timeout=60, socket_connect_timeout=10, socket_keepalive=True, - health_check_interval=30 + health_check_interval=30, + decode_responses=True, + **ssl_params ) logger.info("Created fallback Redis PubSub client for event listener") @@ -196,8 +199,8 @@ class ProxyServer: continue try: - channel = message["channel"].decode("utf-8") - data = json.loads(message["data"].decode("utf-8")) + channel = message["channel"] + data = json.loads(message["data"]) event_type = data.get("event") channel_id = data.get("channel_id") @@ -224,26 +227,29 @@ class ProxyServer: # Handle stream switch request new_url = data.get("url") user_agent = data.get("user_agent") + event_stream_id = data.get("stream_id") + event_m3u_profile_id = data.get("m3u_profile_id") if new_url and channel_id in self.stream_managers: - # Update metadata in Redis + # Mark the switch as in-progress in Redis so other workers know to wait if self.redis_client: - metadata_key = RedisKeys.channel_metadata(channel_id) - self.redis_client.hset(metadata_key, "url", new_url) - if user_agent: - self.redis_client.hset(metadata_key, "user_agent", user_agent) - - # Set switch status status_key = RedisKeys.switch_status(channel_id) self.redis_client.set(status_key, "switching") - # Perform the stream switch + # Perform the stream switch, forwarding stream_id and m3u_profile_id stream_manager = self.stream_managers[channel_id] - success = stream_manager.update_url(new_url) + success = stream_manager.update_url(new_url, event_stream_id, event_m3u_profile_id) if success: logger.info(f"Stream switch initiated for channel {channel_id}") + # Confirm the URL in metadata now that the switch happened + if self.redis_client: + metadata_key = RedisKeys.channel_metadata(channel_id) + self.redis_client.hset(metadata_key, "url", new_url) + if user_agent: + self.redis_client.hset(metadata_key, "user_agent", user_agent) + # Publish confirmation switch_result = { "event": EventType.STREAM_SWITCHED, # Use constant instead of string @@ -263,6 +269,14 @@ class ProxyServer: else: logger.error(f"Failed to switch stream for channel {channel_id}") + # Roll back the URL in metadata to what the manager will + # actually reconnect to. The non-owner may have pre-written + # the desired URL; use stream_manager.url (the ground truth) + # so Redis is consistent with the live stream. + if self.redis_client: + metadata_key = RedisKeys.channel_metadata(channel_id) + self.redis_client.hset(metadata_key, "url", stream_manager.url) + # Publish failure switch_result = { "event": EventType.STREAM_SWITCHED, @@ -373,7 +387,7 @@ class ProxyServer: if result is None: return None try: - return result.decode('utf-8') + return result except (AttributeError, UnicodeDecodeError) as e: logger.error(f"Error decoding channel owner for {channel_id}: {e}, raw={result!r}") return None @@ -412,7 +426,7 @@ class ProxyServer: current_owner = self._execute_redis_command( lambda: self.redis_client.get(lock_key) ) - if current_owner and current_owner.decode('utf-8') == self.worker_id: + if current_owner and current_owner == self.worker_id: # Refresh TTL self._execute_redis_command( lambda: self.redis_client.expire(lock_key, ttl) @@ -437,7 +451,7 @@ class ProxyServer: # Only delete if we're the current owner to prevent race conditions current = self.redis_client.get(lock_key) - if current and current.decode('utf-8') == self.worker_id: + if current and current == self.worker_id: self.redis_client.delete(lock_key) logger.info(f"Released ownership of channel {channel_id}") @@ -471,7 +485,7 @@ class ProxyServer: return False return False - if current.decode('utf-8') == self.worker_id: + if current == self.worker_id: self.redis_client.expire(lock_key, ttl) return True @@ -488,15 +502,15 @@ class ProxyServer: metadata_key = RedisKeys.channel_metadata(channel_id) if self.redis_client.exists(metadata_key): metadata = self.redis_client.hgetall(metadata_key) - if b'state' in metadata: - state = metadata[b'state'].decode('utf-8') + if 'state' in metadata: + state = metadata['state'] active_states = [ChannelState.INITIALIZING, ChannelState.CONNECTING, ChannelState.WAITING_FOR_CLIENTS, ChannelState.ACTIVE, ChannelState.BUFFERING] if state in active_states: logger.info(f"Channel {channel_id} already being initialized with state {state}") # Create buffer and client manager only if we don't have them if channel_id not in self.stream_buffers: - self.stream_buffers[channel_id] = StreamBuffer(channel_id, redis_client=self.redis_client) + self.stream_buffers[channel_id] = StreamBuffer(channel_id, redis_client=RedisClient.get_buffer()) if channel_id not in self.client_managers: self.client_managers[channel_id] = ClientManager( channel_id, @@ -507,7 +521,7 @@ class ProxyServer: # Create buffer and client manager instances (or reuse if they exist) if channel_id not in self.stream_buffers: - buffer = StreamBuffer(channel_id, redis_client=self.redis_client) + buffer = StreamBuffer(channel_id, redis_client=RedisClient.get_buffer()) self.stream_buffers[channel_id] = buffer if channel_id not in self.client_managers: @@ -546,18 +560,18 @@ class ProxyServer: # If no url was passed, try to get from Redis if not url and existing_metadata: - url_bytes = existing_metadata.get(b'url') + url_bytes = existing_metadata.get('url') if url_bytes: - channel_url = url_bytes.decode('utf-8') + channel_url = url_bytes - ua_bytes = existing_metadata.get(b'user_agent') + ua_bytes = existing_metadata.get('user_agent') if ua_bytes: - channel_user_agent = ua_bytes.decode('utf-8') + channel_user_agent = ua_bytes # Get stream ID from metadata if not provided - if not channel_stream_id and b'stream_id' in existing_metadata: + if not channel_stream_id and 'stream_id' in existing_metadata: try: - channel_stream_id = int(existing_metadata[b'stream_id'].decode('utf-8')) + channel_stream_id = int(existing_metadata['stream_id']) logger.debug(f"Found stream_id {channel_stream_id} in metadata for channel {channel_id}") except (ValueError, TypeError) as e: logger.debug(f"Could not parse stream_id from metadata: {e}") @@ -572,7 +586,7 @@ class ProxyServer: # Create buffer but not stream manager (only if not already exists) if channel_id not in self.stream_buffers: - buffer = StreamBuffer(channel_id=channel_id, redis_client=self.redis_client) + buffer = StreamBuffer(channel_id=channel_id, redis_client=RedisClient.get_buffer()) self.stream_buffers[channel_id] = buffer # Create client manager with channel_id and redis_client (only if not already exists) @@ -595,7 +609,7 @@ class ProxyServer: # Create buffer but not stream manager (only if not already exists) if channel_id not in self.stream_buffers: - buffer = StreamBuffer(channel_id=channel_id, redis_client=self.redis_client) + buffer = StreamBuffer(channel_id=channel_id, redis_client=RedisClient.get_buffer()) self.stream_buffers[channel_id] = buffer # Create client manager with channel_id and redis_client (only if not already exists) @@ -634,12 +648,12 @@ class ProxyServer: # Verify the stream_id was set correctly in Redis stream_id_value = self.redis_client.hget(metadata_key, "stream_id") if stream_id_value: - logger.info(f"Verified stream_id {stream_id_value.decode('utf-8')} is set in Redis for channel {channel_id}") + logger.info(f"Verified stream_id {stream_id_value} is set in Redis for channel {channel_id}") else: logger.warning(f"Failed to set stream_id in Redis for channel {channel_id}") # Create stream buffer - buffer = StreamBuffer(channel_id=channel_id, redis_client=self.redis_client) + buffer = StreamBuffer(channel_id=channel_id, redis_client=RedisClient.get_buffer()) logger.debug(f"Created StreamBuffer for channel {channel_id}") self.stream_buffers[channel_id] = buffer @@ -733,8 +747,8 @@ class ProxyServer: metadata = self.redis_client.hgetall(metadata_key) # Get channel state and owner - state = metadata.get(b'state', b'unknown').decode('utf-8') - owner = metadata.get(b'owner', b'').decode('utf-8') + state = metadata.get('state', 'unknown') + owner = metadata.get('owner', '') # States that indicate the channel is running properly or shutting down valid_states = [ChannelState.ACTIVE, ChannelState.WAITING_FOR_CLIENTS, @@ -772,8 +786,8 @@ class ProxyServer: return False else: # Unknown or initializing state, check how long it's been in this state - if b'state_changed_at' in metadata: - state_changed_at = float(metadata[b'state_changed_at'].decode('utf-8')) + if 'state_changed_at' in metadata: + state_changed_at = float(metadata['state_changed_at']) state_age = time.time() - state_changed_at # If in initializing state for too long, consider it stale @@ -811,8 +825,8 @@ class ProxyServer: # If we have metadata, log details for debugging if metadata: - state = metadata.get(b'state', b'unknown').decode('utf-8') - owner = metadata.get(b'owner', b'unknown').decode('utf-8') + state = metadata.get('state', 'unknown') + owner = metadata.get('owner', 'unknown') logger.info(f"Zombie channel details - state: {state}, owner: {owner}") # Clean up Redis keys @@ -937,16 +951,16 @@ class ProxyServer: metadata = self.redis_client.hgetall(metadata_key) if metadata: # Calculate runtime from init_time - if b'init_time' in metadata: + if 'init_time' in metadata: try: - init_time = float(metadata[b'init_time'].decode('utf-8')) + init_time = float(metadata['init_time']) runtime = round(time.time() - init_time, 2) except Exception: pass # Get total bytes transferred - if b'total_bytes' in metadata: + if 'total_bytes' in metadata: try: - total_bytes = int(metadata[b'total_bytes'].decode('utf-8')) + total_bytes = int(metadata['total_bytes']) except Exception: pass @@ -1057,8 +1071,8 @@ class ProxyServer: if self.redis_client: metadata_key = RedisKeys.channel_metadata(channel_id) metadata = self.redis_client.hgetall(metadata_key) - if metadata and b'state' in metadata: - channel_state = metadata[b'state'].decode('utf-8') + if metadata and 'state' in metadata: + channel_state = metadata['state'] # Check if channel has any clients left total_clients = 0 @@ -1090,9 +1104,9 @@ class ProxyServer: # Get connection_ready_time from metadata (indicates if channel reached ready state) connection_ready_time = None - if metadata and b'connection_ready_time' in metadata: + if metadata and 'connection_ready_time' in metadata: try: - connection_ready_time = float(metadata[b'connection_ready_time'].decode('utf-8')) + connection_ready_time = float(metadata['connection_ready_time']) except (ValueError, TypeError): pass @@ -1104,15 +1118,15 @@ class ProxyServer: attempt_value = self.redis_client.get(attempt_key) if attempt_value: try: - connection_attempt_time = float(attempt_value.decode('utf-8')) + connection_attempt_time = float(attempt_value) except (ValueError, TypeError): pass # Also get init time as a fallback init_time = None - if metadata and b'init_time' in metadata: + if metadata and 'init_time' in metadata: try: - init_time = float(metadata[b'init_time'].decode('utf-8')) + init_time = float(metadata['init_time']) except (ValueError, TypeError): pass @@ -1183,7 +1197,7 @@ class ProxyServer: disconnect_value = self.redis_client.get(disconnect_key) if disconnect_value: try: - disconnect_time = float(disconnect_value.decode('utf-8')) + disconnect_time = float(disconnect_value) except (ValueError, TypeError) as e: logger.error(f"Invalid disconnect time for channel {channel_id}: {e}") @@ -1304,7 +1318,7 @@ class ProxyServer: for key in channel_keys: try: - channel_id = key.decode('utf-8').split(':')[2] + channel_id = key.split(':')[2] # Check if this channel has an owner owner = self.get_channel_owner(channel_id) @@ -1349,7 +1363,7 @@ class ProxyServer: for key in channel_keys: try: - channel_id = key.decode('utf-8').split(':')[2] + channel_id = key.split(':')[2] # Get metadata first metadata = self.redis_client.hgetall(key) @@ -1364,7 +1378,7 @@ class ProxyServer: continue # Get owner - owner = metadata.get(b'owner', b'').decode('utf-8') if b'owner' in metadata else '' + owner = metadata.get('owner', '') if 'owner' in metadata else '' # Check if owner is still alive owner_alive = False @@ -1378,7 +1392,7 @@ class ProxyServer: # If no owner and no clients, clean it up if not owner_alive and client_count == 0: - state = metadata.get(b'state', b'unknown').decode('utf-8') if b'state' in metadata else 'unknown' + state = metadata.get('state', 'unknown') logger.warning(f"Found orphaned metadata for channel {channel_id} (state: {state}, owner: {owner}, clients: {client_count}) - cleaning up") # If we have it locally, stop it properly to clean up transcode/proxy processes @@ -1397,7 +1411,7 @@ class ProxyServer: real_count = max(0, client_count - len(stale_ids)) if real_count <= 0: # No real clients remain — safe to clean up. - state = metadata.get(b'state', b'unknown').decode('utf-8') if b'state' in metadata else 'unknown' + state = metadata.get('state', 'unknown') logger.warning( f"Orphaned channel {channel_id} (state: {state}, " f"owner: {owner}) had {client_count} ghost client(s) " @@ -1492,8 +1506,8 @@ class ProxyServer: # Get current state for logging current_state = None metadata = self.redis_client.hgetall(metadata_key) - if metadata and b'state' in metadata: - current_state = metadata[b'state'].decode('utf-8') + if metadata and 'state' in metadata: + current_state = metadata['state'] # Only update if state is actually changing if current_state == new_state: diff --git a/apps/proxy/ts_proxy/services/channel_service.py b/apps/proxy/ts_proxy/services/channel_service.py index e765ece3..ff6eb416 100644 --- a/apps/proxy/ts_proxy/services/channel_service.py +++ b/apps/proxy/ts_proxy/services/channel_service.py @@ -61,7 +61,7 @@ class ChannelService: # Verify the stream_id was set stream_id_value = proxy_server.redis_client.hget(metadata_key, ChannelMetadataField.STREAM_ID) if stream_id_value: - logger.debug(f"Verified stream_id {stream_id_value.decode('utf-8')} is now set in Redis") + logger.debug(f"Verified stream_id {stream_id_value} is now set in Redis") else: logger.error(f"Failed to set stream_id {stream_id} in Redis before initialization") @@ -131,7 +131,7 @@ class ChannelService: try: # This is inefficient but used for diagnostics - in production would use more targeted checks redis_keys = proxy_server.redis_client.keys(f"ts_proxy:*:{channel_id}*") - redis_keys = [k.decode('utf-8') for k in redis_keys] if redis_keys else [] + redis_keys = [k for k in redis_keys] if redis_keys else [] except Exception as e: logger.error(f"Error checking Redis keys: {e}") @@ -168,15 +168,6 @@ class ChannelService: else: result = {'status': 'success'} - # Update metadata in Redis regardless of ownership - if proxy_server.redis_client: - try: - ChannelService._update_channel_metadata(channel_id, new_url, user_agent, stream_id, m3u_profile_id) - result['metadata_updated'] = True - except Exception as e: - logger.error(f"Error updating Redis metadata: {e}", exc_info=True) - result['metadata_updated'] = False - # If we're the owner, update directly if proxy_server.am_i_owner(channel_id) and channel_id in proxy_server.stream_managers: logger.info(f"This worker is the owner, changing stream URL for channel {channel_id}") @@ -187,14 +178,33 @@ class ChannelService: success = manager.update_url(new_url, stream_id, m3u_profile_id) logger.info(f"Stream URL changed from {old_url} to {new_url}, result: {success}") + # Update Redis metadata based on the actual outcome. + # On success, write the new values. On failure, restore whatever URL + # the manager will actually reconnect to (may be old_url if the + # exception happened before self.url was reassigned, or new_url if it + # happened after) so Redis never describes a URL that isn't in use. + if proxy_server.redis_client: + try: + if success: + ChannelService._update_channel_metadata(channel_id, new_url, user_agent, stream_id, m3u_profile_id) + else: + ChannelService._update_channel_metadata(channel_id, manager.url, user_agent) + result['metadata_updated'] = True + except Exception as e: + logger.error(f"Error updating Redis metadata: {e}", exc_info=True) + result['metadata_updated'] = False + result.update({ 'direct_update': True, 'success': success, 'worker_id': proxy_server.worker_id }) else: - # If we're not the owner, publish an event for the owner to pick up - logger.info(f"Not the owner, requesting URL change via Redis PubSub") + # Not the owner: publish the switch event. The owner will update metadata + # after the actual switch attempt succeeds (or roll back on failure). + # All needed info (url, user_agent, stream_id, m3u_profile_id) is carried + # in the pubsub message, so there is no reason to pre-write metadata here. + logger.debug(f"This worker is not the owner, publishing stream switch event for channel {channel_id}") if proxy_server.redis_client: ChannelService._publish_stream_switch_event(channel_id, new_url, user_agent, stream_id, m3u_profile_id) result.update({ @@ -236,8 +246,8 @@ class ChannelService: metadata_key = RedisKeys.channel_metadata(channel_id) try: metadata = proxy_server.redis_client.hgetall(metadata_key) - if metadata and b'state' in metadata: - state = metadata[b'state'].decode('utf-8') + if metadata and 'state' in metadata: + state = metadata['state'] channel_info = {"state": state} # Immediately mark as stopping in metadata so clients detect it faster @@ -382,8 +392,8 @@ class ChannelService: metadata = proxy_server.redis_client.hgetall(metadata_key) # Extract state and owner - state = metadata.get(ChannelMetadataField.STATE.encode(), b'unknown').decode('utf-8') - owner = metadata.get(ChannelMetadataField.OWNER.encode(), b'unknown').decode('utf-8') + state = metadata.get(ChannelMetadataField.STATE, 'unknown') + owner = metadata.get(ChannelMetadataField.OWNER, 'unknown') # Valid states indicate channel is running properly valid_states = [ChannelState.ACTIVE, ChannelState.WAITING_FOR_CLIENTS, ChannelState.CONNECTING] @@ -409,7 +419,7 @@ class ChannelService: } if last_data: - last_data_time = float(last_data.decode('utf-8')) + last_data_time = float(last_data) data_age = time.time() - last_data_time details["last_data_age"] = data_age @@ -432,13 +442,13 @@ class ChannelService: try: # Use factory to parse the line based on stream type parsed_data = LogParserFactory.parse(stream_type, stream_info_line) - + if not parsed_data: return # Update Redis and database with parsed data ChannelService._update_stream_info_in_redis( - channel_id, + channel_id, parsed_data.get('video_codec'), parsed_data.get('resolution'), parsed_data.get('width'), @@ -579,7 +589,7 @@ class ChannelService: metadata_key = RedisKeys.channel_metadata(channel_id) # First check if the key exists and what type it is - key_type = proxy_server.redis_client.type(metadata_key).decode('utf-8') + key_type = proxy_server.redis_client.type(metadata_key) logger.debug(f"Redis key {metadata_key} is of type: {key_type}") # Build metadata update dict diff --git a/apps/proxy/ts_proxy/stream_generator.py b/apps/proxy/ts_proxy/stream_generator.py index 2e16008b..0036be5c 100644 --- a/apps/proxy/ts_proxy/stream_generator.py +++ b/apps/proxy/ts_proxy/stream_generator.py @@ -25,7 +25,7 @@ class StreamGenerator: data delivery, and cleanup. """ - def __init__(self, channel_id, client_id, client_ip, client_user_agent, channel_initializing=False): + def __init__(self, channel_id, client_id, client_ip, client_user_agent, channel_initializing=False, user=None): """ Initialize the stream generator with client and channel details. @@ -35,12 +35,14 @@ class StreamGenerator: client_ip: Client's IP address client_user_agent: User agent string from client channel_initializing: Whether the channel is still initializing + user: Authenticated user making the request """ self.channel_id = channel_id self.client_id = client_id self.client_ip = client_ip self.client_user_agent = client_user_agent self.channel_initializing = channel_initializing + self.user = user # Performance and state tracking self.stream_start_time = time.time() @@ -112,7 +114,8 @@ class StreamGenerator: channel_name=channel_obj.name, client_ip=self.client_ip, client_id=self.client_id, - user_agent=self.client_user_agent[:100] if self.client_user_agent else None + user_agent=self.client_user_agent[:100] if self.client_user_agent else None, + username=self.user.username if self.user else None ) except Exception as e: logger.error(f"Could not log client connect event: {e}") @@ -141,13 +144,13 @@ class StreamGenerator: metadata_key = RedisKeys.channel_metadata(self.channel_id) metadata = proxy_server.redis_client.hgetall(metadata_key) - if metadata and b'state' in metadata: - state = metadata[b'state'].decode('utf-8') + if metadata and 'state' in metadata: + state = metadata['state'] if state in ['waiting_for_clients', 'active']: logger.info(f"[{self.client_id}] Channel {self.channel_id} now ready (state={state})") return True elif state in ['error', 'stopped', 'stopping']: # Added 'stopping' to error states - error_message = metadata.get(b'error_message', b'Unknown error').decode('utf-8') + error_message = metadata.get('error_message', 'Unknown error') logger.error(f"[{self.client_id}] Channel {self.channel_id} in error state: {state}, message: {error_message}") # Send error packet before giving up yield create_ts_packet('error', f"Error: {error_message}") @@ -155,9 +158,9 @@ class StreamGenerator: else: # Improved logging to track initialization progress init_time = "unknown" - if b'init_time' in metadata: + if 'init_time' in metadata: try: - init_time_float = float(metadata[b'init_time'].decode('utf-8')) + init_time_float = float(metadata['init_time']) init_duration = time.time() - init_time_float init_time = f"{init_duration:.1f}s ago" except: @@ -390,8 +393,8 @@ class StreamGenerator: # Channel state in metadata metadata_key = RedisKeys.channel_metadata(self.channel_id) metadata = proxy_server.redis_client.hgetall(metadata_key) - if metadata and b'state' in metadata: - state = metadata[b'state'].decode('utf-8') + if metadata and 'state' in metadata: + state = metadata['state'] if state in ['error', 'stopped', 'stopping']: logger.info(f"[{self.client_id}] Channel in {state} state, terminating stream") return False @@ -555,8 +558,6 @@ class StreamGenerator: if metadata: stream_id_bytes = proxy_server.redis_client.hget(metadata_key, ChannelMetadataField.STREAM_ID) if stream_id_bytes: - stream_id = int(stream_id_bytes.decode('utf-8')) - # Check if we're the last client if self.channel_id in proxy_server.client_managers: client_count = proxy_server.client_managers[self.channel_id].get_total_client_count() @@ -595,7 +596,8 @@ class StreamGenerator: client_id=self.client_id, user_agent=self.client_user_agent[:100] if self.client_user_agent else None, duration=round(elapsed, 2), - bytes_sent=self.bytes_sent + bytes_sent=self.bytes_sent, + username=self.user.username if self.user else None ) except Exception as e: logger.error(f"Could not log client disconnect event: {e}") @@ -630,10 +632,10 @@ class StreamGenerator: gevent.spawn(delayed_shutdown) -def create_stream_generator(channel_id, client_id, client_ip, client_user_agent, channel_initializing=False): +def create_stream_generator(channel_id, client_id, client_ip, client_user_agent, channel_initializing=False, user=None): """ Factory function to create a new stream generator. Returns a function that can be passed to StreamingHttpResponse. """ - generator = StreamGenerator(channel_id, client_id, client_ip, client_user_agent, channel_initializing) + generator = StreamGenerator(channel_id, client_id, client_ip, client_user_agent, channel_initializing, user=user) return generator.generate diff --git a/apps/proxy/ts_proxy/stream_manager.py b/apps/proxy/ts_proxy/stream_manager.py index 424e8231..d22fc7d0 100644 --- a/apps/proxy/ts_proxy/stream_manager.py +++ b/apps/proxy/ts_proxy/stream_manager.py @@ -90,7 +90,7 @@ class StreamManager: # Try to get stream_id specifically stream_id_bytes = buffer.redis_client.hget(metadata_key, "stream_id") if stream_id_bytes: - self.current_stream_id = int(stream_id_bytes.decode('utf-8')) + self.current_stream_id = int(stream_id_bytes) self.tried_stream_ids.add(self.current_stream_id) logger.info(f"Loaded stream ID {self.current_stream_id} from Redis for channel {buffer.channel_id}") else: @@ -123,6 +123,12 @@ class StreamManager: # Add HTTP reader thread property self.http_reader = None + # Output bitrate smoothing / throttled DB persistence + self._smoothed_output_bitrate = None + self._last_bitrate_db_save_time = 0 + self._bitrate_db_save_interval = 30 # seconds between DB writes + self._bitrate_warmup_samples = 10 # discard first N samples while EMA stabilizes (~5s) + def _create_session(self): """Create and configure requests session with optimal settings""" session = requests.Session() @@ -413,7 +419,7 @@ class StreamManager: is_owner = ( current_owner and self.worker_id - and current_owner.decode('utf-8') == self.worker_id + and current_owner == self.worker_id ) no_owner = current_owner is None @@ -423,7 +429,7 @@ class StreamManager: metadata_key, ChannelMetadataField.STATE ) current_state = ( - current_state_bytes.decode('utf-8') + current_state_bytes if current_state_bytes else None ) should_update = current_state in ChannelState.PRE_ACTIVE @@ -773,13 +779,25 @@ class StreamManager: if any(x is not None for x in [ffmpeg_speed, ffmpeg_fps, actual_fps, ffmpeg_output_bitrate]): self._update_ffmpeg_stats_in_redis(ffmpeg_speed, ffmpeg_fps, actual_fps, ffmpeg_output_bitrate) - # Also save ffmpeg_output_bitrate to database if we have stream_id + # Update local EMA and periodically flush to database if ffmpeg_output_bitrate is not None and self.current_stream_id: - from .services.channel_service import ChannelService - ChannelService._update_stream_stats_in_db( - self.current_stream_id, - ffmpeg_output_bitrate=ffmpeg_output_bitrate - ) + if self._bitrate_warmup_samples > 0: + # Discard early samples from the EMA + self._bitrate_warmup_samples -= 1 + else: + if self._smoothed_output_bitrate is None: + self._smoothed_output_bitrate = ffmpeg_output_bitrate + else: + self._smoothed_output_bitrate = 0.9 * self._smoothed_output_bitrate + 0.1 * ffmpeg_output_bitrate + + now = time.time() + if now - self._last_bitrate_db_save_time >= self._bitrate_db_save_interval: + from .services.channel_service import ChannelService + ChannelService._update_stream_stats_in_db( + self.current_stream_id, + ffmpeg_output_bitrate=round(self._smoothed_output_bitrate, 1) + ) + self._last_bitrate_db_save_time = now # Fix the f-string formatting actual_fps_str = f"{actual_fps:.1f}" if actual_fps is not None else "N/A" @@ -1057,6 +1075,20 @@ class StreamManager: # Set running to false to ensure thread exits self.running = False + # Flush the final bitrate to DB on stop only if warmup completed and we have + # a meaningful EMA. Short previews / channel hops that die during warmup do NOT + # write anything, preserving any previously correct value in the database. + if self._smoothed_output_bitrate is not None and self.current_stream_id: + final_bitrate = self._smoothed_output_bitrate + try: + from .services.channel_service import ChannelService + ChannelService._update_stream_stats_in_db( + self.current_stream_id, + ffmpeg_output_bitrate=round(final_bitrate, 1) + ) + except Exception as e: + logger.debug(f"Error flushing final bitrate to DB for channel {self.channel_id}: {e}") + def update_url(self, new_url, stream_id=None, m3u_profile_id=None): """Update stream URL and reconnect with proper cleanup for both HTTP and transcode sessions""" if new_url == self.url: @@ -1114,6 +1146,11 @@ class StreamManager: self.url = new_url self.connected = False + # Reset bitrate EMA on every URL change so stale data never carries over + self._smoothed_output_bitrate = None + self._last_bitrate_db_save_time = 0 + self._bitrate_warmup_samples = 10 + # Update stream ID if provided if stream_id: old_stream_id = self.current_stream_id @@ -1151,7 +1188,7 @@ class StreamManager: logger.error(f"Error during URL update for channel {self.channel_id}: {e}", exc_info=True) return False finally: - # CRITICAL FIX: Always reset the URL switching flag when done, whether successful or not + # Always reset the URL switching flag when done, whether successful or not self.url_switching = False logger.info(f"Stream switch completed for channel {self.channel_id}") @@ -1503,9 +1540,9 @@ class StreamManager: current_state = None try: metadata = redis_client.hgetall(metadata_key) - state_field = ChannelMetadataField.STATE.encode('utf-8') + state_field = ChannelMetadataField.STATE if metadata and state_field in metadata: - current_state = metadata[state_field].decode('utf-8') + current_state = metadata[state_field] except Exception as e: logger.error(f"Error checking current state: {e}") @@ -1656,7 +1693,7 @@ class StreamManager: new_user_agent = stream_info['user_agent'] new_transcode = stream_info['transcode'] - # CRITICAL FIX: Check if the new URL is the same as current URL + # Check if the new URL is the same as current URL # This can happen when current_stream_id is None and we accidentally select the same stream if new_url == self.url: logger.warning(f"Stream ID {stream_id} generates the same URL as current stream ({new_url}). " @@ -1665,7 +1702,7 @@ class StreamManager: logger.info(f"Switching from URL {self.url} to {new_url} for channel {self.channel_id}") - # IMPORTANT: Just update the URL, don't stop the channel or release resources + # Just update the URL, don't stop the channel or release resources switch_result = self.update_url(new_url, stream_id, profile_id) if not switch_result: logger.error(f"Failed to update URL for stream ID {stream_id} for channel {self.channel_id}") @@ -1710,4 +1747,4 @@ class StreamManager: """Safely reset the URL switching state if it gets stuck""" self.url_switching = False self.url_switch_start_time = 0 - logger.info(f"Reset URL switching state for channel {self.channel_id}") \ No newline at end of file + logger.info(f"Reset URL switching state for channel {self.channel_id}") diff --git a/apps/proxy/ts_proxy/url_utils.py b/apps/proxy/ts_proxy/url_utils.py index 14a714ea..5f4615a4 100644 --- a/apps/proxy/ts_proxy/url_utils.py +++ b/apps/proxy/ts_proxy/url_utils.py @@ -3,7 +3,7 @@ Utilities for handling stream URLs and transformations. """ import logging -import re +import regex from typing import Optional, Tuple, List from django.shortcuts import get_object_or_404 from apps.channels.models import Channel, Stream @@ -146,13 +146,14 @@ def transform_url(input_url: str, search_pattern: str, replace_pattern: str) -> logger.debug(f" base URL: {input_url}") logger.debug(f" search: {search_pattern}") - # Handle backreferences in the replacement pattern - safe_replace_pattern = re.sub(r'\$(\d+)', r'\\\1', replace_pattern) + # Convert JS-style backreferences in replace pattern: $ -> \g, $1 -> \1 + safe_replace_pattern = regex.sub(r'\$<([^>]+)>', r'\\g<\1>', replace_pattern) + safe_replace_pattern = regex.sub(r'\$(\d+)', r'\\\1', safe_replace_pattern) logger.debug(f" replace: {replace_pattern}") logger.debug(f" safe replace: {safe_replace_pattern}") - # Apply the transformation - stream_url = re.sub(search_pattern, safe_replace_pattern, input_url) + # Apply the transformation (regex module accepts JS-style (?...) natively) + stream_url = regex.sub(search_pattern, safe_replace_pattern, input_url) logger.info(f"Generated stream url: {stream_url}") return stream_url @@ -211,9 +212,9 @@ def get_stream_info_for_switch(channel_id: str, target_stream_id: Optional[int] existing_stream_id = redis_client.get(f"channel_stream:{channel.id}") if existing_stream_id: # Decode bytes to string/int for proper Redis key lookup - existing_stream_id = existing_stream_id.decode('utf-8') + existing_stream_id = existing_stream_id existing_profile_id = redis_client.get(f"stream_profile:{existing_stream_id}") - if existing_profile_id and int(existing_profile_id.decode('utf-8')) == profile.id: + if existing_profile_id and int(existing_profile_id) == profile.id: channel_using_profile = True logger.debug(f"Channel {channel.id} already using profile {profile.id}") @@ -349,9 +350,9 @@ def get_alternate_streams(channel_id: str, current_stream_id: Optional[int] = No existing_stream_id = redis_client.get(f"channel_stream:{channel.id}") if existing_stream_id: # Decode bytes to string/int for proper Redis key lookup - existing_stream_id = existing_stream_id.decode('utf-8') + existing_stream_id = existing_stream_id existing_profile_id = redis_client.get(f"stream_profile:{existing_stream_id}") - if existing_profile_id and int(existing_profile_id.decode('utf-8')) == profile.id: + if existing_profile_id and int(existing_profile_id) == profile.id: channel_using_profile = True logger.debug(f"Channel {channel.id} already using profile {profile.id}") diff --git a/apps/proxy/ts_proxy/utils.py b/apps/proxy/ts_proxy/utils.py index 20a6e140..ff9e1780 100644 --- a/apps/proxy/ts_proxy/utils.py +++ b/apps/proxy/ts_proxy/utils.py @@ -83,7 +83,7 @@ def create_ts_packet(packet_type='null', message=None): # Add message to payload if provided if message: - msg_bytes = message.encode('utf-8') + msg_bytes = message packet[4:4+min(len(msg_bytes), 180)] = msg_bytes[:180] return bytes(packet) @@ -113,4 +113,4 @@ def get_logger(component_name=None): # Default if detection fails logger_name = "ts_proxy" - return logging.getLogger(logger_name) \ No newline at end of file + return logging.getLogger(logger_name) diff --git a/apps/proxy/ts_proxy/views.py b/apps/proxy/ts_proxy/views.py index ccbb947d..da1c29f1 100644 --- a/apps/proxy/ts_proxy/views.py +++ b/apps/proxy/ts_proxy/views.py @@ -19,6 +19,7 @@ from apps.m3u.models import M3UAccount, M3UAccountProfile from apps.accounts.models import User from core.models import UserAgent, CoreSettings, PROXY_PROFILE_NAME from rest_framework.decorators import api_view, permission_classes +from rest_framework.permissions import AllowAny from rest_framework.response import Response from apps.accounts.permissions import ( IsAdmin, @@ -40,16 +41,21 @@ from .utils import get_logger from uuid import UUID import gevent from dispatcharr.utils import network_access_allowed +from apps.proxy.utils import check_user_stream_limits logger = get_logger() @api_view(["GET"]) -def stream_ts(request, channel_id): +@permission_classes([AllowAny]) +def stream_ts(request, channel_id, user=None): if not network_access_allowed(request, "STREAMS"): return JsonResponse({"error": "Forbidden"}, status=403) """Stream TS data to client with immediate response and keep-alive packets during initialization""" + if user is None and hasattr(request, 'user') and request.user.is_authenticated: + user = request.user + channel = get_stream_object(channel_id) client_user_agent = None @@ -71,6 +77,13 @@ def stream_ts(request, channel_id): ) break + if user: + if not check_user_stream_limits(user, client_id, media_id=channel_id): + return JsonResponse( + {"error": f"Stream limit exceeded ({user.stream_limit} concurrent streams allowed)"}, + status=429 + ) + # Check if we need to reinitialize the channel needs_initialization = True channel_state = None @@ -81,9 +94,9 @@ def stream_ts(request, channel_id): metadata_key = RedisKeys.channel_metadata(channel_id) if proxy_server.redis_client.exists(metadata_key): metadata = proxy_server.redis_client.hgetall(metadata_key) - state_field = ChannelMetadataField.STATE.encode("utf-8") + state_field = ChannelMetadataField.STATE if state_field in metadata: - channel_state = metadata[state_field].decode("utf-8") + channel_state = metadata[state_field] # Active/running states - channel is operational, don't reinitialize if channel_state in [ @@ -119,9 +132,9 @@ def stream_ts(request, channel_id): ) # Unknown/empty state - check if owner is alive else: - owner_field = ChannelMetadataField.OWNER.encode("utf-8") + owner_field = ChannelMetadataField.OWNER if owner_field in metadata: - owner = metadata[owner_field].decode("utf-8") + owner = metadata[owner_field] owner_heartbeat_key = f"ts_proxy:worker:{owner}:heartbeat" if proxy_server.redis_client.exists(owner_heartbeat_key): # Owner is still active with unknown state - don't reinitialize @@ -399,7 +412,7 @@ def stream_ts(request, channel_id): metadata_key, ChannelMetadataField.STATE ) if state_bytes: - current_state = state_bytes.decode("utf-8") + current_state = state_bytes logger.debug( f"[{client_id}] Current state of channel {channel_id}: {current_state}" ) @@ -475,12 +488,12 @@ def stream_ts(request, channel_id): ) if url_bytes: - url = url_bytes.decode("utf-8") + url = url_bytes if ua_bytes: - stream_user_agent = ua_bytes.decode("utf-8") + stream_user_agent = ua_bytes # Extract transcode setting from Redis if profile_bytes: - profile_str = profile_bytes.decode("utf-8") + profile_str = profile_bytes use_transcode = ( profile_str == PROXY_PROFILE_NAME or profile_str == "None" ) @@ -516,12 +529,12 @@ def stream_ts(request, channel_id): # Register client buffer = proxy_server.stream_buffers[channel_id] client_manager = proxy_server.client_managers[channel_id] - client_manager.add_client(client_id, client_ip, client_user_agent) + client_manager.add_client(client_id, client_ip, client_user_agent, user) logger.info(f"[{client_id}] Client registered with channel {channel_id}") # Create a stream generator for this client generate = create_stream_generator( - channel_id, client_id, client_ip, client_user_agent, channel_initializing + channel_id, client_id, client_ip, client_user_agent, channel_initializing, user=user ) # Return the StreamingHttpResponse from the main function @@ -543,6 +556,7 @@ def stream_ts(request, channel_id): @api_view(["GET"]) +@permission_classes([AllowAny]) def stream_xc(request, username, password, channel_id): user = get_object_or_404(User, username=username) @@ -557,7 +571,6 @@ def stream_xc(request, username, password, channel_id): if custom_properties["xc_password"] != password: return Response({"error": "Invalid credentials"}, status=401) - print(f"Fetchin channel with ID: {channel_id}") if user.user_level < 10: user_profile_count = user.channel_profiles.count() @@ -585,7 +598,7 @@ def stream_xc(request, username, password, channel_id): channel = get_object_or_404(Channel, id=channel_id) # @TODO: we've got the file 'type' via extension, support this when we support multiple outputs - return stream_ts(request._request, str(channel.uuid)) + return stream_ts(request._request, str(channel.uuid), user) @csrf_exempt @@ -713,7 +726,7 @@ def channel_status(request, channel_id=None): ) for key in keys: channel_id_match = re.search( - r"ts_proxy:channel:(.*):metadata", key.decode("utf-8") + r"ts_proxy:channel:(.*):metadata", key ) if channel_id_match: ch_id = channel_id_match.group(1) @@ -834,7 +847,7 @@ def next_stream(request, channel_id): metadata_key, ChannelMetadataField.STREAM_ID ) if stream_id_bytes: - current_stream_id = int(stream_id_bytes.decode("utf-8")) + current_stream_id = int(stream_id_bytes) logger.info( f"Found current stream ID {current_stream_id} in Redis for channel {channel_id}" ) @@ -844,7 +857,7 @@ def next_stream(request, channel_id): metadata_key, ChannelMetadataField.M3U_PROFILE ) if profile_id_bytes: - profile_id = int(profile_id_bytes.decode("utf-8")) + profile_id = int(profile_id_bytes) logger.info( f"Found M3U profile ID {profile_id} in Redis for channel {channel_id}" ) @@ -916,7 +929,8 @@ def next_stream(request, channel_id): channel_id, stream_info["url"], stream_info["user_agent"], - next_stream_id, # Pass the stream_id to be stored in Redis + next_stream_id, + stream_info.get("m3u_profile_id"), ) if result.get("status") == "error": diff --git a/apps/proxy/urls.py b/apps/proxy/urls.py index 34c026a9..3a320049 100644 --- a/apps/proxy/urls.py +++ b/apps/proxy/urls.py @@ -6,4 +6,4 @@ urlpatterns = [ path('ts/', include('apps.proxy.ts_proxy.urls')), path('hls/', include('apps.proxy.hls_proxy.urls')), path('vod/', include('apps.proxy.vod_proxy.urls')), -] \ No newline at end of file +] diff --git a/apps/proxy/utils.py b/apps/proxy/utils.py new file mode 100644 index 00000000..6503d1df --- /dev/null +++ b/apps/proxy/utils.py @@ -0,0 +1,185 @@ +import logging +from core.utils import RedisClient +from apps.proxy.vod_proxy.multi_worker_connection_manager import MultiWorkerVODConnectionManager, get_vod_client_stop_key +from core.models import CoreSettings +from apps.proxy.ts_proxy.services.channel_service import ChannelService + +logger = logging.getLogger("proxy") + + +def attempt_stream_termination(user_id, requesting_client_id, active_connections): + try: + logger.info("[stream limits]" f"[{requesting_client_id}] User {user_id} has {len(active_connections)} active connections, checking termination candidates") + + user_limit_settings = CoreSettings.get_user_limits_settings() + terminate_oldest = user_limit_settings.get("terminate_oldest", True) + prioritize_single = user_limit_settings.get("prioritize_single_client_channels", True) + ignore_same_channel = user_limit_settings.get("ignore_same_channel_connections", False) + + channel_counts = {} + for connection in active_connections: + media_id = connection['media_id'] + channel_counts[media_id] = channel_counts.get(media_id, 0) + 1 + + def prioritize(connection): + is_multi = channel_counts[connection['media_id']] > 1 + + # if we're ignoring same-channel connections, put them at the end + same_ch_key = 1 if (ignore_same_channel and is_multi) else 0 + + # key for prioritizing single-client channels + single_key = 0 if (prioritize_single and not is_multi) else 1 + + # sort by age setting + time_key = connection['connected_at'] if terminate_oldest else -connection['connected_at'] + + return (same_ch_key, single_key, time_key) + + termination_candidates = sorted(active_connections, key=prioritize) + + if not termination_candidates: + logger.warning("[stream limits]" f"[{requesting_client_id}] No termination candidates found for user {user_id}") + return False + + target = termination_candidates[0] + logger.info("[stream limits]" + f"[{requesting_client_id}] Terminating client {target['client_id']} " + f"on media {target['media_id']} (connected_at={target['connected_at']})" + ) + + # When counting by unique channel, freeing one connection from a multi-connection + # channel doesn't free a slot — terminate all connections to that channel so the + # unique-channel count actually drops by one. + targets = ( + [c for c in active_connections if c['media_id'] == target['media_id']] + if ignore_same_channel + else [target] + ) + + for t in targets: + if t['type'] == 'live': + result = ChannelService.stop_client(t['media_id'], t['client_id']) + if result.get("status") == "error": + logger.warning(f"[stream limits][{requesting_client_id}] Failed to stop client {t['client_id']} on channel {t['media_id']}") + else: + connection_manager = MultiWorkerVODConnectionManager.get_instance() + redis_client = connection_manager.redis_client + + if not redis_client: + return False + + connection_key = f"vod_persistent_connection:{t['client_id']}" + connection_data = redis_client.hgetall(connection_key) + if not connection_data: + logger.warning(f"VOD connection not found: {t['client_id']}") + continue + + stop_key = get_vod_client_stop_key(t['client_id']) + redis_client.setex(stop_key, 60, "true") # 60 second TTL + + return True + except Exception as e: + logger.error("[stream limits]" f"[{requesting_client_id}] Error during stream termination for user {user_id}: {e}") + return False + +def get_user_active_connections(user_id): + redis_client = RedisClient.get_client() + connections = [] + + try: + # Grab live streams + for key in redis_client.scan_iter(match="ts_proxy:channel:*:clients:*", count=1000): + parts = key.split(':') + if len(parts) >= 5: + channel_id = parts[2] + client_id = parts[4] + + client_user_id = redis_client.hget(key, 'user_id') + connected_at = redis_client.hget(key, 'connected_at') + + logger.debug(f"[stream limits] user_id = {user_id}") + logger.debug(f"[stream limits] channel_id = {channel_id}") + logger.debug(f"[stream limits] client_id = {client_id}") + + if client_user_id and int(client_user_id) == user_id: + try: + logger.debug(f"[stream limits] Found LIVE connection for user {user_id} on channel {channel_id} with client ID {client_id}") + connected_at = float(connected_at) if connected_at else 0 + connections.append({ + 'media_id': channel_id, + 'client_id': client_id, + 'connected_at': connected_at, + 'type': 'live', + }) + except (ValueError, TypeError): + pass + + # Grab VOD + for key in redis_client.scan_iter(match="vod_persistent_connection:*", count=1000): + parts = key.split(':') + if len(parts) >= 2: + client_id = parts[1] + + client_user_id = redis_client.hget(key, 'user_id') + connected_at = redis_client.hget(key, 'created_at') + content_uuid = redis_client.hget(key, 'content_uuid') + + logger.debug(f"[stream limits] user_id = {user_id}") + logger.debug(f"[stream limits] client_id = {client_id}") + + if client_user_id and int(client_user_id) == user_id: + try: + logger.debug(f"[stream limits] Found VOD connection for user {user_id} on content {content_uuid} with client ID {client_id}") + connected_at = float(connected_at) if connected_at else 0 + connections.append({ + 'media_id': content_uuid or client_id, + 'client_id': client_id, + 'connected_at': connected_at, + 'type': 'vod', + }) + except (ValueError, TypeError): + pass + + return connections + + except Exception as e: + logger.warning(f"Error getting active channel details for user {user_id}: {e}") + return [] + + +def check_user_stream_limits(user, client_id, media_id=None): + # Check user stream limits + if user and user.stream_limit > 0: + logger.debug("[stream limits]" f"[{client_id}] User {user.username} (ID: {user.id}) is requesting a stream (stream_limit: {user.stream_limit})") + user_limit_settings = CoreSettings.get_user_limits_settings() + ignore_same_channel = user_limit_settings.get("ignore_same_channel_connections", False) + + active_connections = get_user_active_connections(user.id) + unique_channel_count = set([conn['media_id'] for conn in active_connections]) + user_stream_count = len(unique_channel_count) if ignore_same_channel else len(active_connections) + + logger.debug(f"[stream limits]" f"[{client_id}] User {user.username} currently has {len(active_connections)} active connections across {len(unique_channel_count)} unique channels (counting method: {'unique channels' if ignore_same_channel else 'total connections'})") + + # If ignore_same_channel is enabled and this request is for a live channel the user + # is already watching, allow it through without counting against the limit. + # VOD is excluded: connections aren't shared so multiple VOD connections to the + # same content would mean multiple upstream connections. + live_channel_ids = {str(conn['media_id']) for conn in active_connections if conn['type'] == 'live'} + if ignore_same_channel and media_id and str(media_id) in live_channel_ids: + logger.debug(f"[stream limits][{client_id}] Same-channel reconnect for {media_id} allowed (ignore_same_channel=True)") + return True + + if user_stream_count >= user.stream_limit: + if user_limit_settings.get("terminate_on_limit_exceeded", True) == False: + return False + + if user_stream_count >= user.stream_limit: + logger.warning("[stream limits]" + f"[{client_id}] User {user.username} (ID: {user.id}) has reached stream limit " + f"({user_stream_count}/{user.stream_limit} streams), attempting to free up slot" + ) + + if not attempt_stream_termination(user.id, client_id, active_connections): + return False + + return True diff --git a/apps/proxy/vod_proxy/connection_manager.py b/apps/proxy/vod_proxy/connection_manager.py deleted file mode 100644 index 7bfccfca..00000000 --- a/apps/proxy/vod_proxy/connection_manager.py +++ /dev/null @@ -1,1505 +0,0 @@ -""" -VOD Connection Manager - Redis-based connection tracking for VOD streams -""" - -import time -import json -import logging -import threading -import random -import re -import requests -from typing import Optional, Dict, Any -from django.http import StreamingHttpResponse, HttpResponse -from core.utils import RedisClient -from apps.vod.models import Movie, Episode -from apps.m3u.models import M3UAccountProfile - -logger = logging.getLogger("vod_proxy") - - -class PersistentVODConnection: - """Handles a single persistent connection to a VOD provider for a session""" - - def __init__(self, session_id: str, stream_url: str, headers: dict): - self.session_id = session_id - self.stream_url = stream_url - self.base_headers = headers - self.session = None - self.current_response = None - self.content_length = None - self.content_type = 'video/mp4' - self.final_url = None - self.lock = threading.Lock() - self.request_count = 0 # Track number of requests on this connection - self.last_activity = time.time() # Track last activity for cleanup - self.cleanup_timer = None # Timer for delayed cleanup - self.active_streams = 0 # Count of active stream generators - - def _establish_connection(self, range_header=None): - """Establish or re-establish connection to provider""" - try: - if not self.session: - self.session = requests.Session() - - headers = self.base_headers.copy() - - # Validate range header against content length - if range_header and self.content_length: - logger.info(f"[{self.session_id}] Validating range {range_header} against content length {self.content_length}") - validated_range = self._validate_range_header(range_header, int(self.content_length)) - if validated_range is None: - # Range is not satisfiable, but don't raise error - return empty response - logger.warning(f"[{self.session_id}] Range not satisfiable: {range_header} for content length {self.content_length}") - return None - elif validated_range != range_header: - range_header = validated_range - logger.info(f"[{self.session_id}] Adjusted range header: {range_header}") - else: - logger.info(f"[{self.session_id}] Range header validated successfully: {range_header}") - elif range_header: - logger.info(f"[{self.session_id}] Range header provided but no content length available yet: {range_header}") - - if range_header: - headers['Range'] = range_header - logger.info(f"[{self.session_id}] Setting Range header: {range_header}") - - # Track request count for better logging - self.request_count += 1 - if self.request_count == 1: - logger.info(f"[{self.session_id}] Making initial request to provider") - target_url = self.stream_url - allow_redirects = True - else: - logger.info(f"[{self.session_id}] Making range request #{self.request_count} on SAME session (using final URL)") - # Use the final URL from first request to avoid redirect chain - target_url = self.final_url if self.final_url else self.stream_url - allow_redirects = False # No need to follow redirects again - logger.info(f"[{self.session_id}] Using cached final URL: {target_url}") - - response = self.session.get( - target_url, - headers=headers, - stream=True, - timeout=(10, 30), - allow_redirects=allow_redirects - ) - response.raise_for_status() - - # Log successful response - if self.request_count == 1: - logger.info(f"[{self.session_id}] Request #{self.request_count} successful: {response.status_code} (followed redirects)") - else: - logger.info(f"[{self.session_id}] Request #{self.request_count} successful: {response.status_code} (direct to final URL)") - - # Capture headers from final URL - if not self.content_length: - # First check if we have a pre-stored content length from HEAD request - try: - import redis - from django.conf import settings - redis_host = getattr(settings, 'REDIS_HOST', 'localhost') - redis_port = int(getattr(settings, 'REDIS_PORT', 6379)) - redis_db = int(getattr(settings, 'REDIS_DB', 0)) - redis_password = getattr(settings, 'REDIS_PASSWORD', '') - redis_user = getattr(settings, 'REDIS_USER', '') - r = redis.StrictRedis( - host=redis_host, - port=redis_port, - db=redis_db, - password=redis_password if redis_password else None, - username=redis_user if redis_user else None, - decode_responses=True - ) - content_length_key = f"vod_content_length:{self.session_id}" - stored_length = r.get(content_length_key) - if stored_length: - self.content_length = stored_length - logger.info(f"[{self.session_id}] *** USING PRE-STORED CONTENT LENGTH: {self.content_length} ***") - else: - # Fallback to response headers - self.content_length = response.headers.get('content-length') - logger.info(f"[{self.session_id}] *** USING RESPONSE CONTENT LENGTH: {self.content_length} ***") - except Exception as e: - logger.error(f"[{self.session_id}] Error checking Redis for content length: {e}") - # Fallback to response headers - self.content_length = response.headers.get('content-length') - - self.content_type = response.headers.get('content-type', 'video/mp4') - self.final_url = response.url - logger.info(f"[{self.session_id}] *** PERSISTENT CONNECTION - Final URL: {self.final_url} ***") - logger.info(f"[{self.session_id}] *** PERSISTENT CONNECTION - Content-Length: {self.content_length} ***") - - self.current_response = response - return response - - except Exception as e: - logger.error(f"[{self.session_id}] Error establishing connection: {e}") - self.cleanup() - raise - - def _validate_range_header(self, range_header, content_length): - """Validate and potentially adjust range header against content length""" - try: - if not range_header or not range_header.startswith('bytes='): - return range_header - - range_part = range_header.replace('bytes=', '') - if '-' not in range_part: - return range_header - - start_str, end_str = range_part.split('-', 1) - - # Parse start byte - if start_str: - start_byte = int(start_str) - if start_byte >= content_length: - # Start is beyond file end - not satisfiable - logger.warning(f"[{self.session_id}] Range start {start_byte} >= content length {content_length} - not satisfiable") - return None - else: - start_byte = 0 - - # Parse end byte - if end_str: - end_byte = int(end_str) - if end_byte >= content_length: - # Adjust end to file end - end_byte = content_length - 1 - logger.info(f"[{self.session_id}] Adjusted range end to {end_byte}") - else: - end_byte = content_length - 1 - - # Ensure start <= end - if start_byte > end_byte: - logger.warning(f"[{self.session_id}] Range start {start_byte} > end {end_byte} - not satisfiable") - return None - - validated_range = f"bytes={start_byte}-{end_byte}" - return validated_range - - except (ValueError, IndexError) as e: - logger.warning(f"[{self.session_id}] Could not validate range header {range_header}: {e}") - return range_header - - def get_stream(self, range_header=None): - """Get stream with optional range header - reuses connection for range requests""" - with self.lock: - # Update activity timestamp - self.last_activity = time.time() - - # Cancel any pending cleanup since connection is being reused - self.cancel_cleanup() - - # For range requests, we don't need to close the connection - # We can make a new request on the same session - if range_header: - logger.info(f"[{self.session_id}] Range request on existing connection: {range_header}") - # Close only the response stream, keep the session alive - if self.current_response: - logger.info(f"[{self.session_id}] Closing previous response stream (keeping connection alive)") - self.current_response.close() - self.current_response = None - - # Make new request (reuses connection if session exists) - response = self._establish_connection(range_header) - if response is None: - # Range not satisfiable - return None to indicate this - return None - - return self.current_response - - def cancel_cleanup(self): - """Cancel any pending cleanup - called when connection is reused""" - if self.cleanup_timer: - self.cleanup_timer.cancel() - self.cleanup_timer = None - logger.info(f"[{self.session_id}] Cancelled pending cleanup - connection being reused for new request") - - def increment_active_streams(self): - """Increment the count of active streams""" - with self.lock: - self.active_streams += 1 - logger.debug(f"[{self.session_id}] Active streams incremented to {self.active_streams}") - - def decrement_active_streams(self): - """Decrement the count of active streams""" - with self.lock: - if self.active_streams > 0: - self.active_streams -= 1 - logger.debug(f"[{self.session_id}] Active streams decremented to {self.active_streams}") - else: - logger.warning(f"[{self.session_id}] Attempted to decrement active streams when already at 0") - - def has_active_streams(self) -> bool: - """Check if connection has any active streams""" - with self.lock: - return self.active_streams > 0 - - def schedule_cleanup_if_not_streaming(self, delay_seconds: int = 10): - """Schedule cleanup only if no active streams""" - with self.lock: - if self.active_streams > 0: - logger.info(f"[{self.session_id}] Connection has {self.active_streams} active streams - NOT scheduling cleanup") - return False - - # No active streams, proceed with delayed cleanup - if self.cleanup_timer: - self.cleanup_timer.cancel() - - def delayed_cleanup(): - logger.info(f"[{self.session_id}] Delayed cleanup triggered - checking if connection is still needed") - # Use the singleton VODConnectionManager instance - manager = VODConnectionManager.get_instance() - manager.cleanup_persistent_connection(self.session_id) - - self.cleanup_timer = threading.Timer(delay_seconds, delayed_cleanup) - self.cleanup_timer.start() - logger.info(f"[{self.session_id}] Scheduled cleanup in {delay_seconds} seconds (connection not actively streaming)") - return True - - def get_headers(self): - """Get headers for response""" - return { - 'content_length': self.content_length, - 'content_type': self.content_type, - 'final_url': self.final_url - } - - def cleanup(self): - """Clean up connection resources""" - with self.lock: - # Cancel any pending cleanup timer - if self.cleanup_timer: - self.cleanup_timer.cancel() - self.cleanup_timer = None - logger.debug(f"[{self.session_id}] Cancelled cleanup timer during manual cleanup") - - # Clear active streams count - self.active_streams = 0 - - if self.current_response: - self.current_response.close() - self.current_response = None - if self.session: - self.session.close() - self.session = None - logger.info(f"[{self.session_id}] Persistent connection cleaned up") - - -class VODConnectionManager: - """Manages VOD connections using Redis for tracking""" - - _instance = None - _persistent_connections = {} # session_id -> PersistentVODConnection - - @classmethod - def get_instance(cls): - """Get the singleton instance of VODConnectionManager""" - if cls._instance is None: - cls._instance = cls() - return cls._instance - - def __init__(self): - self.redis_client = RedisClient.get_client() - self.connection_ttl = 3600 # 1 hour TTL for connections - self.session_ttl = 1800 # 30 minutes TTL for sessions - - def find_matching_idle_session(self, content_type: str, content_uuid: str, - client_ip: str, user_agent: str, - utc_start=None, utc_end=None, offset=None) -> Optional[str]: - """ - Find an existing session that matches content and client criteria with no active streams - - Args: - content_type: Type of content (movie, episode, series) - content_uuid: UUID of the content - client_ip: Client IP address - user_agent: Client user agent - utc_start: UTC start time for timeshift - utc_end: UTC end time for timeshift - offset: Offset in seconds - - Returns: - Session ID if matching idle session found, None otherwise - """ - if not self.redis_client: - return None - - try: - # Search for sessions with matching content - pattern = "vod_session:*" - cursor = 0 - matching_sessions = [] - - while True: - cursor, keys = self.redis_client.scan(cursor, match=pattern, count=100) - - for key in keys: - try: - session_data = self.redis_client.hgetall(key) - if not session_data: - continue - - # Extract session info - stored_content_type = session_data.get(b'content_type', b'').decode('utf-8') - stored_content_uuid = session_data.get(b'content_uuid', b'').decode('utf-8') - - # Check if content matches - if stored_content_type != content_type or stored_content_uuid != content_uuid: - continue - - # Extract session ID from key - session_id = key.decode('utf-8').replace('vod_session:', '') - - # Check if session has an active persistent connection - persistent_conn = self._persistent_connections.get(session_id) - if not persistent_conn: - # No persistent connection exists, skip - continue - - # Check if connection has no active streams - if persistent_conn.has_active_streams(): - logger.debug(f"[{session_id}] Session has active streams - skipping") - continue - - # Get stored client info for comparison - stored_client_ip = session_data.get(b'client_ip', b'').decode('utf-8') - stored_user_agent = session_data.get(b'user_agent', b'').decode('utf-8') - - # Check timeshift parameters match - stored_utc_start = session_data.get(b'utc_start', b'').decode('utf-8') - stored_utc_end = session_data.get(b'utc_end', b'').decode('utf-8') - stored_offset = session_data.get(b'offset', b'').decode('utf-8') - - current_utc_start = utc_start or "" - current_utc_end = utc_end or "" - current_offset = str(offset) if offset else "" - - # Calculate match score - score = 0 - match_reasons = [] - - # Content already matches (required) - score += 10 - match_reasons.append("content") - - # IP match (high priority) - if stored_client_ip and stored_client_ip == client_ip: - score += 5 - match_reasons.append("ip") - - # User-Agent match (medium priority) - if stored_user_agent and stored_user_agent == user_agent: - score += 3 - match_reasons.append("user-agent") - - # Timeshift parameters match (high priority for seeking) - if (stored_utc_start == current_utc_start and - stored_utc_end == current_utc_end and - stored_offset == current_offset): - score += 7 - match_reasons.append("timeshift") - - # Consider it a good match if we have at least content + one other criteria - if score >= 13: # content(10) + ip(5) or content(10) + user-agent(3) + something else - matching_sessions.append({ - 'session_id': session_id, - 'score': score, - 'reasons': match_reasons, - 'last_activity': float(session_data.get(b'last_activity', b'0').decode('utf-8')) - }) - - except Exception as e: - logger.debug(f"Error processing session key {key}: {e}") - continue - - if cursor == 0: - break - - # Sort by score (highest first), then by last activity (most recent first) - matching_sessions.sort(key=lambda x: (x['score'], x['last_activity']), reverse=True) - - if matching_sessions: - best_match = matching_sessions[0] - logger.info(f"Found matching idle session: {best_match['session_id']} " - f"(score: {best_match['score']}, reasons: {', '.join(best_match['reasons'])})") - return best_match['session_id'] - else: - logger.debug(f"No matching idle sessions found for {content_type} {content_uuid}") - return None - - except Exception as e: - logger.error(f"Error finding matching idle session: {e}") - return None - - def _get_connection_key(self, content_type: str, content_uuid: str, client_id: str) -> str: - """Get Redis key for a specific connection""" - return f"vod_proxy:connection:{content_type}:{content_uuid}:{client_id}" - - def _get_profile_connections_key(self, profile_id: int) -> str: - """Get Redis key for tracking connections per profile - STANDARDIZED with TS proxy""" - return f"profile_connections:{profile_id}" - - def _get_content_connections_key(self, content_type: str, content_uuid: str) -> str: - """Get Redis key for tracking connections per content""" - return f"vod_proxy:content:{content_type}:{content_uuid}:connections" - - def create_connection(self, content_type: str, content_uuid: str, content_name: str, - client_id: str, client_ip: str, user_agent: str, - m3u_profile: M3UAccountProfile) -> bool: - """ - Create a new VOD connection with profile limit checking - - Returns: - bool: True if connection was created, False if profile limit exceeded - """ - if not self.redis_client: - logger.error("Redis client not available for VOD connection tracking") - return False - - try: - # Atomically check and reserve a profile connection slot (INCR-first) - if not self._check_and_reserve_profile_slot(m3u_profile): - logger.warning(f"Profile {m3u_profile.name} connection limit exceeded") - return False - - connection_key = self._get_connection_key(content_type, content_uuid, client_id) - content_connections_key = self._get_content_connections_key(content_type, content_uuid) - - # Check if connection already exists to prevent duplicate counting - if self.redis_client.exists(connection_key): - logger.info(f"Connection already exists for {client_id} - {content_type} {content_name}") - # Update activity but don't increment profile counter - self.redis_client.hset(connection_key, "last_activity", str(time.time())) - # Roll back the reservation — connection already counted - if m3u_profile.max_streams > 0: - self.redis_client.decr(self._get_profile_connections_key(m3u_profile.id)) - return True - - # Connection data - connection_data = { - "content_type": content_type, - "content_uuid": content_uuid, - "content_name": content_name, - "client_id": client_id, - "client_ip": client_ip, - "user_agent": user_agent, - "m3u_profile_id": m3u_profile.id, - "m3u_profile_name": m3u_profile.name, - "connected_at": str(time.time()), - "last_activity": str(time.time()), - "bytes_sent": "0", - "position_seconds": "0", - "last_position_update": str(time.time()) - } - - # Use pipeline for atomic operations - pipe = self.redis_client.pipeline() - - # Store connection data - pipe.hset(connection_key, mapping=connection_data) - pipe.expire(connection_key, self.connection_ttl) - - # Profile counter already incremented atomically above — no pipe.incr needed - - # Add to content connections set - pipe.sadd(content_connections_key, client_id) - pipe.expire(content_connections_key, self.connection_ttl) - - # Execute all operations - pipe.execute() - - logger.info(f"Created VOD connection: {client_id} for {content_type} {content_name}") - return True - - except Exception as e: - # Roll back the profile reservation on failure - if m3u_profile.max_streams > 0: - self.redis_client.decr(self._get_profile_connections_key(m3u_profile.id)) - logger.error(f"Error creating VOD connection: {e}") - return False - - def _check_profile_limits(self, m3u_profile: M3UAccountProfile) -> bool: - """Check if profile has available connection slots""" - if m3u_profile.max_streams == 0: # Unlimited - return True - - try: - profile_connections_key = self._get_profile_connections_key(m3u_profile.id) - current_connections = int(self.redis_client.get(profile_connections_key) or 0) - - return current_connections < m3u_profile.max_streams - - except Exception as e: - logger.error(f"Error checking profile limits: {e}") - return False - - def _check_and_reserve_profile_slot(self, m3u_profile: M3UAccountProfile) -> bool: - """ - Atomically check and reserve a connection slot for the given profile. - - Uses an INCR-first-then-check pattern to eliminate the TOCTOU race - condition where separate GET > check > INCR operations could allow - concurrent requests to both pass the capacity check. - - For profiles with max_streams=0 (unlimited), no reservation is needed. - - Returns: - bool: True if slot was reserved (or unlimited), False if at capacity - """ - if m3u_profile.max_streams == 0: # Unlimited - return True - - try: - profile_connections_key = self._get_profile_connections_key(m3u_profile.id) - - # Atomically increment first — single Redis command eliminates race window - new_count = self.redis_client.incr(profile_connections_key) - - if new_count <= m3u_profile.max_streams: - logger.info(f"[PROFILE-RESERVE] Profile {m3u_profile.id} slot reserved: {new_count}/{m3u_profile.max_streams}") - return True - - # Over capacity — roll back the increment - self.redis_client.decr(profile_connections_key) - logger.info(f"[PROFILE-RESERVE] Profile {m3u_profile.id} at capacity: {new_count - 1}/{m3u_profile.max_streams}") - return False - - except Exception as e: - logger.error(f"Error reserving profile slot: {e}") - return False - - def update_connection_activity(self, content_type: str, content_uuid: str, - client_id: str, bytes_sent: int = 0, - position_seconds: int = 0) -> bool: - """Update connection activity""" - if not self.redis_client: - return False - - try: - connection_key = self._get_connection_key(content_type, content_uuid, client_id) - - update_data = { - "last_activity": str(time.time()) - } - - if bytes_sent > 0: - # Get current bytes and add to it - current_bytes = self.redis_client.hget(connection_key, "bytes_sent") - if current_bytes: - total_bytes = int(current_bytes.decode('utf-8')) + bytes_sent - else: - total_bytes = bytes_sent - update_data["bytes_sent"] = str(total_bytes) - - if position_seconds > 0: - update_data["position_seconds"] = str(position_seconds) - - # Update connection data - self.redis_client.hset(connection_key, mapping=update_data) - self.redis_client.expire(connection_key, self.connection_ttl) - - return True - - except Exception as e: - logger.error(f"Error updating connection activity: {e}") - return False - - def remove_connection(self, content_type: str, content_uuid: str, client_id: str) -> bool: - """Remove a VOD connection""" - if not self.redis_client: - return False - - try: - connection_key = self._get_connection_key(content_type, content_uuid, client_id) - - # Get connection data before removing - connection_data = self.redis_client.hgetall(connection_key) - if not connection_data: - return True # Already removed - - # Get profile ID for cleanup - profile_id = None - if b"m3u_profile_id" in connection_data: - try: - profile_id = int(connection_data[b"m3u_profile_id"].decode('utf-8')) - except ValueError: - pass - - # Use pipeline for atomic cleanup - pipe = self.redis_client.pipeline() - - # Remove connection data - pipe.delete(connection_key) - - # Decrement profile connections using standardized key - if profile_id: - profile_connections_key = self._get_profile_connections_key(profile_id) - current_count = int(self.redis_client.get(profile_connections_key) or 0) - if current_count > 0: - pipe.decr(profile_connections_key) - - # Remove from content connections set - content_connections_key = self._get_content_connections_key(content_type, content_uuid) - pipe.srem(content_connections_key, client_id) - - # Execute cleanup - pipe.execute() - - logger.info(f"Removed VOD connection: {client_id}") - return True - - except Exception as e: - logger.error(f"Error removing connection: {e}") - return False - - def get_connection_info(self, content_type: str, content_uuid: str, client_id: str) -> Optional[Dict[str, Any]]: - """Get connection information""" - if not self.redis_client: - return None - - try: - connection_key = self._get_connection_key(content_type, content_uuid, client_id) - connection_data = self.redis_client.hgetall(connection_key) - - if not connection_data: - return None - - # Convert bytes to strings and parse numbers - info = {} - for key, value in connection_data.items(): - key_str = key.decode('utf-8') - value_str = value.decode('utf-8') - - # Parse numeric fields - if key_str in ['connected_at', 'last_activity']: - info[key_str] = float(value_str) - elif key_str in ['bytes_sent', 'position_seconds', 'm3u_profile_id']: - info[key_str] = int(value_str) - else: - info[key_str] = value_str - - return info - - except Exception as e: - logger.error(f"Error getting connection info: {e}") - return None - - def get_profile_connections(self, profile_id: int) -> int: - """Get current connection count for a profile using standardized key""" - if not self.redis_client: - return 0 - - try: - profile_connections_key = self._get_profile_connections_key(profile_id) - return int(self.redis_client.get(profile_connections_key) or 0) - - except Exception as e: - logger.error(f"Error getting profile connections: {e}") - return 0 - - def get_content_connections(self, content_type: str, content_uuid: str) -> int: - """Get current connection count for content""" - if not self.redis_client: - return 0 - - try: - content_connections_key = self._get_content_connections_key(content_type, content_uuid) - return self.redis_client.scard(content_connections_key) or 0 - - except Exception as e: - logger.error(f"Error getting content connections: {e}") - return 0 - - def cleanup_stale_connections(self, max_age_seconds: int = 3600): - """Clean up stale connections that haven't been active recently""" - if not self.redis_client: - return - - try: - pattern = "vod_proxy:connection:*" - cursor = 0 - cleaned = 0 - current_time = time.time() - - while True: - cursor, keys = self.redis_client.scan(cursor, match=pattern, count=100) - - for key in keys: - try: - key_str = key.decode('utf-8') - last_activity = self.redis_client.hget(key, "last_activity") - - if last_activity: - last_activity_time = float(last_activity.decode('utf-8')) - if current_time - last_activity_time > max_age_seconds: - # Extract info for cleanup - parts = key_str.split(':') - if len(parts) >= 5: - content_type = parts[2] - content_uuid = parts[3] - client_id = parts[4] - self.remove_connection(content_type, content_uuid, client_id) - cleaned += 1 - except Exception as e: - logger.error(f"Error processing key {key}: {e}") - - if cursor == 0: - break - - if cleaned > 0: - logger.info(f"Cleaned up {cleaned} stale VOD connections") - - except Exception as e: - logger.error(f"Error during connection cleanup: {e}") - - def stream_content(self, content_obj, stream_url, m3u_profile, client_ip, user_agent, request, - utc_start=None, utc_end=None, offset=None, range_header=None): - """ - Stream VOD content with connection tracking and timeshift support - - Args: - content_obj: Movie or Episode object - stream_url: Final stream URL to proxy - m3u_profile: M3UAccountProfile instance - client_ip: Client IP address - user_agent: Client user agent - request: Django request object - utc_start: UTC start time for timeshift (e.g., '2023-01-01T12:00:00') - utc_end: UTC end time for timeshift - offset: Offset in seconds for seeking - range_header: HTTP Range header for partial content requests - - Returns: - StreamingHttpResponse or HttpResponse with error - """ - - try: - # Generate unique client ID - client_id = f"vod_{int(time.time() * 1000)}_{random.randint(1000, 9999)}" - - # Determine content type and get content info - if hasattr(content_obj, 'episodes'): # Series - content_type = 'series' - elif hasattr(content_obj, 'series'): # Episode - content_type = 'episode' - else: # Movie - content_type = 'movie' - - content_uuid = str(content_obj.uuid) - content_name = getattr(content_obj, 'name', getattr(content_obj, 'title', 'Unknown')) - - # Create connection tracking - connection_created = self.create_connection( - content_type=content_type, - content_uuid=content_uuid, - content_name=content_name, - client_id=client_id, - client_ip=client_ip, - user_agent=user_agent, - m3u_profile=m3u_profile - ) - - if not connection_created: - logger.error(f"Failed to create connection tracking for {content_type} {content_uuid}") - return HttpResponse("Connection limit exceeded", status=503) - - # Modify stream URL for timeshift functionality - modified_stream_url = self._apply_timeshift_parameters( - stream_url, utc_start, utc_end, offset - ) - - logger.info(f"[{client_id}] Modified stream URL for timeshift: {modified_stream_url}") - - # Create streaming generator with simplified header handling - upstream_response = None - - def stream_generator(): - nonlocal upstream_response - try: - logger.info(f"[{client_id}] Starting VOD stream for {content_type} {content_name}") - - # Prepare request headers - headers = {} - if user_agent: - headers['User-Agent'] = user_agent - - # Forward important headers - important_headers = [ - 'authorization', 'x-forwarded-for', 'x-real-ip', - 'referer', 'origin', 'accept' - ] - - for header_name in important_headers: - django_header = f'HTTP_{header_name.upper().replace("-", "_")}' - if hasattr(request, 'META') and django_header in request.META: - headers[header_name] = request.META[django_header] - logger.debug(f"[{client_id}] Forwarded header {header_name}") - - # Add client IP - if client_ip: - headers['X-Forwarded-For'] = client_ip - headers['X-Real-IP'] = client_ip - - # Add Range header if provided for seeking support - if range_header: - headers['Range'] = range_header - logger.info(f"[{client_id}] Added Range header: {range_header}") - - # Make request to upstream server with automatic redirect following - upstream_response = requests.get(modified_stream_url, headers=headers, stream=True, timeout=(10, 30), allow_redirects=True) - upstream_response.raise_for_status() - - # Log upstream response info - logger.info(f"[{client_id}] Upstream response status: {upstream_response.status_code}") - logger.info(f"[{client_id}] Upstream content-type: {upstream_response.headers.get('content-type', 'unknown')}") - if 'content-length' in upstream_response.headers: - logger.info(f"[{client_id}] Upstream content-length: {upstream_response.headers['content-length']}") - if 'content-range' in upstream_response.headers: - logger.info(f"[{client_id}] Upstream content-range: {upstream_response.headers['content-range']}") - - bytes_sent = 0 - chunk_count = 0 - - for chunk in upstream_response.iter_content(chunk_size=8192): - if chunk: - yield chunk - bytes_sent += len(chunk) - chunk_count += 1 - - # Update connection activity every 100 chunks - if chunk_count % 100 == 0: - self.update_connection_activity( - content_type=content_type, - content_uuid=content_uuid, - client_id=client_id, - bytes_sent=len(chunk) - ) - - logger.info(f"[{client_id}] VOD stream completed: {bytes_sent} bytes sent") - - except requests.RequestException as e: - logger.error(f"[{client_id}] Error streaming from source: {e}") - yield b"Error: Unable to stream content" - except Exception as e: - logger.error(f"[{client_id}] Error in stream generator: {e}") - finally: - # Clean up connection tracking - self.remove_connection(content_type, content_uuid, client_id) - if upstream_response: - upstream_response.close() - - def stream_generator(): - nonlocal upstream_response - try: - logger.info(f"[{client_id}] Starting VOD stream for {content_type} {content_name}") - - # Prepare request headers - headers = {} - if user_agent: - headers['User-Agent'] = user_agent - - # Forward important headers - important_headers = [ - 'authorization', 'x-forwarded-for', 'x-real-ip', - 'referer', 'origin', 'accept' - ] - - for header_name in important_headers: - django_header = f'HTTP_{header_name.upper().replace("-", "_")}' - if hasattr(request, 'META') and django_header in request.META: - headers[header_name] = request.META[django_header] - logger.debug(f"[{client_id}] Forwarded header {header_name}") - - # Add client IP - if client_ip: - headers['X-Forwarded-For'] = client_ip - headers['X-Real-IP'] = client_ip - - # Add Range header if provided for seeking support - if range_header: - headers['Range'] = range_header - logger.info(f"[{client_id}] Added Range header: {range_header}") - - # Make single request to upstream server with automatic redirect following - upstream_response = requests.get(modified_stream_url, headers=headers, stream=True, timeout=(10, 30), allow_redirects=True) - upstream_response.raise_for_status() - - # Log upstream response info - logger.info(f"[{client_id}] Upstream response status: {upstream_response.status_code}") - logger.info(f"[{client_id}] Final URL after redirects: {upstream_response.url}") - logger.info(f"[{client_id}] Upstream content-type: {upstream_response.headers.get('content-type', 'unknown')}") - if 'content-length' in upstream_response.headers: - logger.info(f"[{client_id}] Upstream content-length: {upstream_response.headers['content-length']}") - if 'content-range' in upstream_response.headers: - logger.info(f"[{client_id}] Upstream content-range: {upstream_response.headers['content-range']}") - - bytes_sent = 0 - chunk_count = 0 - - for chunk in upstream_response.iter_content(chunk_size=8192): - if chunk: - yield chunk - bytes_sent += len(chunk) - chunk_count += 1 - - # Update connection activity every 100 chunks - if chunk_count % 100 == 0: - self.update_connection_activity( - content_type=content_type, - content_uuid=content_uuid, - client_id=client_id, - bytes_sent=len(chunk) - ) - - logger.info(f"[{client_id}] VOD stream completed: {bytes_sent} bytes sent") - - except requests.RequestException as e: - logger.error(f"[{client_id}] Error streaming from source: {e}") - yield b"Error: Unable to stream content" - except Exception as e: - logger.error(f"[{client_id}] Error in stream generator: {e}") - finally: - # Clean up connection tracking - self.remove_connection(content_type, content_uuid, client_id) - if upstream_response: - upstream_response.close() - - # Create streaming response with sensible defaults - response = StreamingHttpResponse( - streaming_content=stream_generator(), - content_type='video/mp4' - ) - - # Set status code based on request type - if range_header: - response.status_code = 206 - logger.info(f"[{client_id}] Set response status to 206 for range request") - else: - response.status_code = 200 - logger.info(f"[{client_id}] Set response status to 200 for full request") - - # Set headers that VLC and other players expect - response['Cache-Control'] = 'no-cache' - response['Pragma'] = 'no-cache' - response['X-Content-Type-Options'] = 'nosniff' - response['Connection'] = 'keep-alive' - response['Accept-Ranges'] = 'bytes' - - # Log the critical headers we're sending to the client - logger.info(f"[{client_id}] Response headers to client - Status: {response.status_code}, Accept-Ranges: {response.get('Accept-Ranges', 'MISSING')}") - if 'Content-Length' in response: - logger.info(f"[{client_id}] Content-Length: {response['Content-Length']}") - if 'Content-Range' in response: - logger.info(f"[{client_id}] Content-Range: {response['Content-Range']}") - if 'Content-Type' in response: - logger.info(f"[{client_id}] Content-Type: {response['Content-Type']}") - - # Critical: Log what VLC needs to see for seeking to work - if response.status_code == 200: - logger.info(f"[{client_id}] VLC SEEKING INFO: Full content response (200). VLC should see Accept-Ranges and Content-Length to enable seeking.") - elif response.status_code == 206: - logger.info(f"[{client_id}] VLC SEEKING INFO: Partial content response (206). This confirms seeking is working if VLC requested a range.") - - return response - - except Exception as e: - logger.error(f"Error in stream_content: {e}", exc_info=True) - return HttpResponse(f"Streaming error: {str(e)}", status=500) - - def stream_content_with_session(self, session_id, content_obj, stream_url, m3u_profile, client_ip, user_agent, request, - utc_start=None, utc_end=None, offset=None, range_header=None): - """ - Stream VOD content with persistent connection per session - - Maintains 1 open connection to provider per session that handles all range requests - dynamically based on client Range headers for seeking functionality. - """ - - try: - # Use session_id as client_id for connection tracking - client_id = session_id - - # Determine content type and get content info - if hasattr(content_obj, 'episodes'): # Series - content_type = 'series' - elif hasattr(content_obj, 'series'): # Episode - content_type = 'episode' - else: # Movie - content_type = 'movie' - - content_uuid = str(content_obj.uuid) - content_name = getattr(content_obj, 'name', getattr(content_obj, 'title', 'Unknown')) - - # Check for existing connection or create new one - persistent_conn = self._persistent_connections.get(session_id) - - # Cancel any pending cleanup timer for this session regardless of new/existing - if persistent_conn: - persistent_conn.cancel_cleanup() - - # If no existing connection, try to find a matching idle session first - if not persistent_conn: - # Look for existing idle sessions that match content and client criteria - matching_session_id = self.find_matching_idle_session( - content_type, content_uuid, client_ip, user_agent, - utc_start, utc_end, offset - ) - - if matching_session_id: - logger.info(f"[{client_id}] Found matching idle session {matching_session_id} - redirecting client") - - # Update the session activity and client info - session_key = f"vod_session:{matching_session_id}" - if self.redis_client: - update_data = { - "last_activity": str(time.time()), - "client_ip": client_ip, # Update in case IP changed - "user_agent": user_agent # Update in case user agent changed - } - self.redis_client.hset(session_key, mapping=update_data) - self.redis_client.expire(session_key, self.session_ttl) - - # Get the existing persistent connection - persistent_conn = self._persistent_connections.get(matching_session_id) - if persistent_conn: - # Update the session_id to use the matching one - client_id = matching_session_id - session_id = matching_session_id - logger.info(f"[{client_id}] Successfully redirected to existing idle session") - else: - logger.warning(f"[{client_id}] Matching session found but no persistent connection - will create new") - - if not persistent_conn: - logger.info(f"[{client_id}] Creating NEW persistent connection for {content_type} {content_name}") - - # Create session in Redis for tracking - session_info = { - "content_type": content_type, - "content_uuid": content_uuid, - "content_name": content_name, - "created_at": str(time.time()), - "last_activity": str(time.time()), - "profile_id": str(m3u_profile.id), - "connection_counted": "True", - "client_ip": client_ip, - "user_agent": user_agent, - "utc_start": utc_start or "", - "utc_end": utc_end or "", - "offset": str(offset) if offset else "" - } - - session_key = f"vod_session:{session_id}" - if self.redis_client: - self.redis_client.hset(session_key, mapping=session_info) - self.redis_client.expire(session_key, self.session_ttl) - - logger.info(f"[{client_id}] Created new session: {session_info}") - - # Apply timeshift parameters to URL - modified_stream_url = self._apply_timeshift_parameters(stream_url, utc_start, utc_end, offset) - logger.info(f"[{client_id}] Modified stream URL for timeshift: {modified_stream_url}") - - # Prepare headers - headers = { - 'User-Agent': user_agent or 'VLC/3.0.21 LibVLC/3.0.21', - 'Accept': '*/*', - 'Connection': 'keep-alive' - } - - # Add any authentication headers from profile - if hasattr(m3u_profile, 'auth_headers') and m3u_profile.auth_headers: - headers.update(m3u_profile.auth_headers) - - # Create persistent connection - persistent_conn = PersistentVODConnection(session_id, modified_stream_url, headers) - self._persistent_connections[session_id] = persistent_conn - - # Track connection in profile - self.create_connection(content_type, content_uuid, content_name, client_id, client_ip, user_agent, m3u_profile) - else: - logger.info(f"[{client_id}] Using EXISTING persistent connection for {content_type} {content_name}") - # Update session activity - session_key = f"vod_session:{session_id}" - if self.redis_client: - self.redis_client.hset(session_key, "last_activity", str(time.time())) - self.redis_client.expire(session_key, self.session_ttl) - - logger.info(f"[{client_id}] Reusing existing session - no new connection created") - - # Log the incoming Range header for debugging - if range_header: - logger.info(f"[{client_id}] *** CLIENT RANGE REQUEST: {range_header} ***") - - # Parse range for seeking detection - try: - if 'bytes=' in range_header: - range_part = range_header.replace('bytes=', '') - if '-' in range_part: - start_byte, end_byte = range_part.split('-', 1) - if start_byte and int(start_byte) > 0: - start_pos_mb = int(start_byte) / (1024 * 1024) - logger.info(f"[{client_id}] *** VLC SEEKING TO: {start_pos_mb:.1f} MB ***") - else: - logger.info(f"[{client_id}] Range request from start") - except Exception as e: - logger.warning(f"[{client_id}] Could not parse range header: {e}") - else: - logger.info(f"[{client_id}] Full content request (no Range header)") - - # Get stream from persistent connection with current range - upstream_response = persistent_conn.get_stream(range_header) - - # Handle range not satisfiable - if upstream_response is None: - logger.warning(f"[{client_id}] Range not satisfiable - returning 416 error") - return HttpResponse( - "Requested Range Not Satisfiable", - status=416, - headers={ - 'Content-Range': f'bytes */{persistent_conn.content_length}' if persistent_conn.content_length else 'bytes */*' - } - ) - - connection_headers = persistent_conn.get_headers() - - # Ensure any pending cleanup is cancelled before starting stream - persistent_conn.cancel_cleanup() - - # Create streaming generator - def stream_generator(): - decremented = False # Track if we've already decremented the counter - - try: - logger.info(f"[{client_id}] Starting stream from persistent connection") - - # Increment active streams counter - persistent_conn.increment_active_streams() - - bytes_sent = 0 - chunk_count = 0 - - for chunk in upstream_response.iter_content(chunk_size=8192): - if chunk: - yield chunk - bytes_sent += len(chunk) - chunk_count += 1 - - # Update connection activity every 100 chunks - if chunk_count % 100 == 0: - self.update_connection_activity( - content_type=content_type, - content_uuid=content_uuid, - client_id=client_id, - bytes_sent=len(chunk) - ) - - logger.info(f"[{client_id}] Persistent stream completed normally: {bytes_sent} bytes sent") - # Stream completed normally - decrement counter - persistent_conn.decrement_active_streams() - decremented = True - - except GeneratorExit: - # Client disconnected - decrement counter and schedule cleanup only if no active streams - logger.info(f"[{client_id}] Client disconnected - checking if cleanup should be scheduled") - persistent_conn.decrement_active_streams() - decremented = True - scheduled = persistent_conn.schedule_cleanup_if_not_streaming(delay_seconds=10) - if not scheduled: - logger.info(f"[{client_id}] Cleanup not scheduled - connection still has active streams") - - except Exception as e: - logger.error(f"[{client_id}] Error in persistent stream: {e}") - # On error, decrement counter and cleanup the connection as it may be corrupted - persistent_conn.decrement_active_streams() - decremented = True - logger.info(f"[{client_id}] Cleaning up persistent connection due to error") - self.cleanup_persistent_connection(session_id) - yield b"Error: Stream interrupted" - - finally: - # Safety net: only decrement if we haven't already - if not decremented: - logger.warning(f"[{client_id}] Stream generator exited without decrement - applying safety net") - persistent_conn.decrement_active_streams() - # This runs regardless of how the generator exits - logger.debug(f"[{client_id}] Stream generator finished") - - # Create streaming response - response = StreamingHttpResponse( - streaming_content=stream_generator(), - content_type=connection_headers['content_type'] - ) - - # Set status code based on range request - if range_header: - response.status_code = 206 - logger.info(f"[{client_id}] Set response status to 206 for range request") - else: - response.status_code = 200 - logger.info(f"[{client_id}] Set response status to 200 for full request") - - # Set headers that VLC expects - response['Cache-Control'] = 'no-cache' - response['Pragma'] = 'no-cache' - response['X-Content-Type-Options'] = 'nosniff' - response['Connection'] = 'keep-alive' - response['Accept-Ranges'] = 'bytes' - - # CRITICAL: Forward Content-Length from persistent connection - if connection_headers['content_length']: - response['Content-Length'] = connection_headers['content_length'] - logger.info(f"[{client_id}] *** FORWARDED Content-Length: {connection_headers['content_length']} *** (VLC seeking enabled)") - else: - logger.warning(f"[{client_id}] *** NO Content-Length available *** (VLC seeking may not work)") - - # Handle range requests - set Content-Range for partial responses - if range_header and connection_headers['content_length']: - try: - if 'bytes=' in range_header: - range_part = range_header.replace('bytes=', '') - if '-' in range_part: - start_byte, end_byte = range_part.split('-', 1) - start = int(start_byte) if start_byte else 0 - end = int(end_byte) if end_byte else int(connection_headers['content_length']) - 1 - total_size = int(connection_headers['content_length']) - - content_range = f"bytes {start}-{end}/{total_size}" - response['Content-Range'] = content_range - logger.info(f"[{client_id}] Set Content-Range: {content_range}") - except Exception as e: - logger.warning(f"[{client_id}] Could not set Content-Range: {e}") - - # Log response headers - logger.info(f"[{client_id}] PERSISTENT Response - Status: {response.status_code}, Content-Length: {response.get('Content-Length', 'MISSING')}") - if 'Content-Range' in response: - logger.info(f"[{client_id}] PERSISTENT Content-Range: {response['Content-Range']}") - - # Log VLC seeking status - if response.status_code == 200: - if connection_headers['content_length']: - logger.info(f"[{client_id}] ✅ PERSISTENT VLC SEEKING: Full response with Content-Length - seeking should work!") - else: - logger.info(f"[{client_id}] ❌ PERSISTENT VLC SEEKING: Full response but no Content-Length - seeking won't work!") - elif response.status_code == 206: - logger.info(f"[{client_id}] ✅ PERSISTENT VLC SEEKING: Partial response - seeking is working!") - - return response - - except Exception as e: - logger.error(f"Error in persistent stream_content_with_session: {e}", exc_info=True) - # Cleanup persistent connection on error - if session_id in self._persistent_connections: - self._persistent_connections[session_id].cleanup() - del self._persistent_connections[session_id] - return HttpResponse(f"Streaming error: {str(e)}", status=500) - - def _apply_timeshift_parameters(self, original_url, utc_start=None, utc_end=None, offset=None): - """ - Apply timeshift parameters to the stream URL - - Args: - original_url: Original stream URL - utc_start: UTC start time (ISO format string) - utc_end: UTC end time (ISO format string) - offset: Offset in seconds - - Returns: - Modified URL with timeshift parameters - """ - try: - from urllib.parse import urlparse, parse_qs, urlencode, urlunparse - - parsed_url = urlparse(original_url) - query_params = parse_qs(parsed_url.query) - - logger.debug(f"Original URL: {original_url}") - logger.debug(f"Original query params: {query_params}") - - # Add timeshift parameters if provided - if utc_start: - # Support both utc_start and start parameter names - query_params['utc_start'] = [utc_start] - query_params['start'] = [utc_start] # Some providers use 'start' - logger.info(f"Added utc_start/start parameter: {utc_start}") - - if utc_end: - # Support both utc_end and end parameter names - query_params['utc_end'] = [utc_end] - query_params['end'] = [utc_end] # Some providers use 'end' - logger.info(f"Added utc_end/end parameter: {utc_end}") - - if offset: - try: - # Ensure offset is a valid number - offset_seconds = int(offset) - # Support multiple offset parameter names - query_params['offset'] = [str(offset_seconds)] - query_params['seek'] = [str(offset_seconds)] # Some providers use 'seek' - query_params['t'] = [str(offset_seconds)] # Some providers use 't' - logger.info(f"Added offset/seek/t parameter: {offset_seconds} seconds") - except (ValueError, TypeError): - logger.warning(f"Invalid offset value: {offset}, skipping") - - # Handle special URL patterns for VOD providers - # Some providers embed timeshift info in the path rather than query params - path = parsed_url.path - - # Check if this looks like an IPTV catchup URL pattern - catchup_pattern = r'/(\d{4}-\d{2}-\d{2})/(\d{2}-\d{2}-\d{2})' - if utc_start and re.search(catchup_pattern, path): - # Convert ISO format to provider-specific format if needed - try: - from datetime import datetime - start_dt = datetime.fromisoformat(utc_start.replace('Z', '+00:00')) - date_part = start_dt.strftime('%Y-%m-%d') - time_part = start_dt.strftime('%H-%M-%S') - - # Replace existing date/time in path - path = re.sub(catchup_pattern, f'/{date_part}/{time_part}', path) - logger.info(f"Modified path for catchup: {path}") - except Exception as e: - logger.warning(f"Could not parse timeshift date: {e}") - - # Reconstruct URL with new parameters - new_query = urlencode(query_params, doseq=True) - modified_url = urlunparse(( - parsed_url.scheme, - parsed_url.netloc, - path, # Use potentially modified path - parsed_url.params, - new_query, - parsed_url.fragment - )) - - logger.info(f"Modified URL: {modified_url}") - return modified_url - - except Exception as e: - logger.error(f"Error applying timeshift parameters: {e}") - return original_url - - def cleanup_persistent_connection(self, session_id: str): - """Clean up a specific persistent connection""" - if session_id in self._persistent_connections: - logger.info(f"[{session_id}] Cleaning up persistent connection") - self._persistent_connections[session_id].cleanup() - del self._persistent_connections[session_id] - - # Clean up ALL Redis keys associated with this session - session_key = f"vod_session:{session_id}" - if self.redis_client: - try: - session_data = self.redis_client.hgetall(session_key) - if session_data: - # Get session details for connection cleanup - content_type = session_data.get(b'content_type', b'').decode('utf-8') - content_uuid = session_data.get(b'content_uuid', b'').decode('utf-8') - profile_id = session_data.get(b'profile_id') - - # Generate client_id from session_id (matches what's used during streaming) - client_id = session_id - - # Remove individual connection tracking keys created during streaming - # Check if connection key exists first - remove_connection() - # handles the profile DECR when the key exists, but skips it - # if the key already expired by TTL - connection_key_exists = False - if content_type and content_uuid: - connection_key = self._get_connection_key(content_type, content_uuid, client_id) - connection_key_exists = bool(self.redis_client.exists(connection_key)) - logger.info(f"[{session_id}] Cleaning up connection tracking keys") - self.remove_connection(content_type, content_uuid, client_id) - - # Fallback DECR: only if the connection tracking key had already - # expired (remove_connection skips DECR in that case) - if not connection_key_exists: - if session_data.get(b'connection_counted') == b'True' and profile_id: - profile_key = self._get_profile_connections_key(int(profile_id.decode('utf-8'))) - current_count = int(self.redis_client.get(profile_key) or 0) - if current_count > 0: - self.redis_client.decr(profile_key) - logger.info(f"[{session_id}] Decremented profile {profile_id.decode('utf-8')} connections (fallback)") - - # Remove session tracking key - self.redis_client.delete(session_key) - logger.info(f"[{session_id}] Removed session tracking") - - # Clean up any additional session-related keys (pattern cleanup) - try: - # Look for any other keys that might be related to this session - pattern = f"*{session_id}*" - cursor = 0 - session_related_keys = [] - while True: - cursor, keys = self.redis_client.scan(cursor, match=pattern, count=100) - session_related_keys.extend(keys) - if cursor == 0: - break - - if session_related_keys: - # Filter out keys we already deleted - remaining_keys = [k for k in session_related_keys if k.decode('utf-8') != session_key] - if remaining_keys: - self.redis_client.delete(*remaining_keys) - logger.info(f"[{session_id}] Cleaned up {len(remaining_keys)} additional session-related keys") - except Exception as scan_error: - logger.warning(f"[{session_id}] Error during pattern cleanup: {scan_error}") - - except Exception as e: - logger.error(f"[{session_id}] Error cleaning up session: {e}") - - def cleanup_stale_persistent_connections(self, max_age_seconds: int = 1800): - """Clean up stale persistent connections that haven't been used recently""" - current_time = time.time() - stale_sessions = [] - - for session_id, conn in self._persistent_connections.items(): - try: - # Check connection's last activity time first - if hasattr(conn, 'last_activity'): - time_since_last_activity = current_time - conn.last_activity - if time_since_last_activity > max_age_seconds: - logger.info(f"[{session_id}] Connection inactive for {time_since_last_activity:.1f}s (max: {max_age_seconds}s)") - stale_sessions.append(session_id) - continue - - # Fallback to Redis session data if connection doesn't have last_activity - session_key = f"vod_session:{session_id}" - if self.redis_client: - session_data = self.redis_client.hgetall(session_key) - if session_data: - created_at = float(session_data.get(b'created_at', b'0').decode('utf-8')) - if current_time - created_at > max_age_seconds: - logger.info(f"[{session_id}] Session older than {max_age_seconds}s") - stale_sessions.append(session_id) - else: - # Session data missing, connection is stale - logger.info(f"[{session_id}] Session data missing from Redis") - stale_sessions.append(session_id) - - except Exception as e: - logger.error(f"[{session_id}] Error checking session age: {e}") - stale_sessions.append(session_id) - - # Clean up stale connections - for session_id in stale_sessions: - logger.info(f"[{session_id}] Cleaning up stale persistent connection") - self.cleanup_persistent_connection(session_id) - - if stale_sessions: - logger.info(f"Cleaned up {len(stale_sessions)} stale persistent connections") - else: - logger.debug(f"No stale persistent connections found (checked {len(self._persistent_connections)} connections)") - - -# Global instance -_connection_manager = None - -def get_connection_manager() -> VODConnectionManager: - """Get the global VOD connection manager instance""" - global _connection_manager - if _connection_manager is None: - _connection_manager = VODConnectionManager() - return _connection_manager diff --git a/apps/proxy/vod_proxy/multi_worker_connection_manager.py b/apps/proxy/vod_proxy/multi_worker_connection_manager.py index 6b048529..073df2ca 100644 --- a/apps/proxy/vod_proxy/multi_worker_connection_manager.py +++ b/apps/proxy/vod_proxy/multi_worker_connection_manager.py @@ -93,7 +93,7 @@ class SerializableConnectionState: content_name: str = None, client_ip: str = None, client_user_agent: str = None, utc_start: str = None, utc_end: str = None, offset: str = None, - worker_id: str = None, connection_type: str = "redis_backed"): + worker_id: str = None, connection_type: str = "redis_backed", user_id: str = "unknown"): self.session_id = session_id self.stream_url = stream_url self.headers = headers @@ -104,6 +104,7 @@ class SerializableConnectionState: self.last_activity = time.time() self.request_count = 0 self.active_streams = 0 + self.user_id = user_id # Session metadata (consolidated from vod_session key) self.content_obj_type = content_obj_type @@ -160,7 +161,8 @@ class SerializableConnectionState: 'last_seek_byte': str(self.last_seek_byte), 'last_seek_percentage': str(self.last_seek_percentage), 'total_content_size': str(self.total_content_size), - 'last_seek_timestamp': str(self.last_seek_timestamp) + 'last_seek_timestamp': str(self.last_seek_timestamp), + 'user_id': str(self.user_id), } @classmethod @@ -184,7 +186,8 @@ class SerializableConnectionState: utc_end=data.get('utc_end') or '', offset=data.get('offset') or '', worker_id=data.get('worker_id') or None, - connection_type=data.get('connection_type', 'redis_backed') + connection_type=data.get('connection_type', 'redis_backed'), + user_id=data.get('user_id', 'unknown') ) obj.last_activity = float(data.get('last_activity', time.time())) obj.request_count = int(data.get('request_count', 0)) @@ -224,7 +227,7 @@ class RedisBackedVODConnection: # Convert bytes keys/values to strings if needed if isinstance(list(data.keys())[0], bytes): - data = {k.decode('utf-8'): v.decode('utf-8') for k, v in data.items()} + data = {k: v for k, v in data.items()} return SerializableConnectionState.from_dict(data) except Exception as e: @@ -281,7 +284,7 @@ class RedisBackedVODConnection: content_name: str = None, client_ip: str = None, client_user_agent: str = None, utc_start: str = None, utc_end: str = None, offset: str = None, - worker_id: str = None) -> bool: + worker_id: str = None, user=None) -> bool: """Create a new connection state in Redis with consolidated session metadata""" if not self._acquire_lock(): logger.warning(f"[{self.session_id}] Could not acquire lock for connection creation") @@ -309,7 +312,8 @@ class RedisBackedVODConnection: utc_start=utc_start, utc_end=utc_end, offset=offset, - worker_id=worker_id + worker_id=worker_id, + user_id=user.id if user else "unknown" ) success = self._save_connection_state(state) @@ -365,6 +369,24 @@ class RedisBackedVODConnection: timeout=(10, 10), allow_redirects=allow_redirects ) + + # If the cached final_url returned an error (e.g. an ephemeral dispatcharr session + # that has since expired), clear it and retry from the original stream_url. + if response.status_code >= 400 and state.final_url: + logger.warning( + f"[{self.session_id}] Cached final_url returned {response.status_code}, " + f"clearing and retrying from stream_url" + ) + response.close() + state.final_url = None + response = self.local_session.get( + state.stream_url, + headers=headers, + stream=True, + timeout=(10, 10), + allow_redirects=True + ) + response.raise_for_status() # Update state with response info on first request @@ -522,6 +544,38 @@ class RedisBackedVODConnection: finally: self._release_lock() + def decrement_active_streams_and_check(self): + """Atomically decrement active streams and return (success, has_remaining_streams). + + Combines decrement + check under a single lock to eliminate the race window + between separate decrement_active_streams() and has_active_streams() calls. + + Returns: + (True, False) - decremented successfully, no streams remain + (True, True) - decremented successfully, other streams still active + (False, True) - lock contention, assume streams remain (safe default) + """ + if not self._acquire_lock(): + logger.warning(f"[{self.session_id}] DECR-AS-CHECK failed: could not acquire lock") + return False, True # Assume remaining to avoid skipping profile decrement + + try: + state = self._get_connection_state() + if state and state.active_streams > 0: + old = state.active_streams + state.active_streams -= 1 + state.last_activity = time.time() + self._save_connection_state(state) + logger.debug(f"[{self.session_id}] DECR-AS {old} -> {state.active_streams}") + return True, state.active_streams > 0 + if not state: + logger.warning(f"[{self.session_id}] DECR-AS-CHECK failed: no state") + return False, False + logger.warning(f"[{self.session_id}] DECR-AS-CHECK failed: active_streams already {state.active_streams}") + return False, False + finally: + self._release_lock() + def has_active_streams(self) -> bool: """Check if connection has any active streams""" state = self._get_connection_state() @@ -744,24 +798,29 @@ class MultiWorkerVODConnectionManager: return None def _decrement_profile_connections(self, m3u_profile_id: int): - """Decrement profile connection count""" + """Decrement profile connection count. + + Uses a single atomic DECR (no GET-before-DECR) to avoid the race condition + where two concurrent decrements both pass a >0 guard and both fire, sending + the counter negative. If the counter would go below zero it is clamped to 0. + """ try: profile_connections_key = self._get_profile_connections_key(m3u_profile_id) - current_count = int(self.redis_client.get(profile_connections_key) or 0) - if current_count > 0: - new_count = self.redis_client.decr(profile_connections_key) - logger.info(f"[PROFILE-DECR] Profile {m3u_profile_id} connections: {new_count}") - return new_count + new_count = self.redis_client.decr(profile_connections_key) + if new_count < 0: + self.redis_client.set(profile_connections_key, 0) + new_count = 0 + logger.warning(f"[PROFILE-DECR] Profile {m3u_profile_id} counter went negative, clamped to 0") else: - logger.warning(f"[PROFILE-DECR] Profile {m3u_profile_id} already at 0 connections") - return 0 + logger.info(f"[PROFILE-DECR] Profile {m3u_profile_id} connections: {new_count}") + return new_count except Exception as e: logger.error(f"Error decrementing profile connections: {e}") return None def stream_content_with_session(self, session_id, content_obj, stream_url, m3u_profile, client_ip, client_user_agent, request, - utc_start=None, utc_end=None, offset=None, range_header=None): + utc_start=None, utc_end=None, offset=None, range_header=None, user=None): """Stream content with Redis-backed persistent connection""" # Generate client ID @@ -858,7 +917,8 @@ class MultiWorkerVODConnectionManager: utc_start=utc_start, utc_end=utc_end, offset=str(offset) if offset else None, - worker_id=self.worker_id + worker_id=self.worker_id, + user=user ): logger.error(f"[{client_id}] Worker {self.worker_id} - Failed to create Redis connection") # Roll back the profile slot reservation since connection failed @@ -933,7 +993,8 @@ class MultiWorkerVODConnectionManager: # Create streaming generator def stream_generator(): - decremented = False + stream_decremented = False + profile_decremented = False stop_signal_detected = False try: logger.info(f"[{client_id}] Worker {self.worker_id} - Starting Redis-backed stream") @@ -986,16 +1047,16 @@ class MultiWorkerVODConnectionManager: logger.info(f"[{client_id}] Worker {self.worker_id} - Stream stopped by signal: {bytes_sent} bytes sent") else: logger.info(f"[{client_id}] Worker {self.worker_id} - Redis-backed stream completed: {bytes_sent} bytes sent") - redis_connection.decrement_active_streams() - decremented = True + stream_decremented, has_remaining = redis_connection.decrement_active_streams_and_check() # Schedule smart cleanup if no active streams after normal completion - if not redis_connection.has_active_streams(): + if stream_decremented and not has_remaining and not profile_decremented: # Decrement profile counter immediately — don't defer to daemon thread state = redis_connection._get_connection_state() profile_id = state.m3u_profile_id if state else m3u_profile.id if profile_id: self._decrement_profile_connections(profile_id) + profile_decremented = True logger.info(f"[{client_id}] Profile counter decremented for profile {profile_id} on normal completion") def delayed_cleanup(): @@ -1012,17 +1073,19 @@ class MultiWorkerVODConnectionManager: except GeneratorExit: logger.info(f"[{client_id}] Worker {self.worker_id} - Client disconnected from Redis-backed stream") - if not decremented: - redis_connection.decrement_active_streams() - decremented = True + if not stream_decremented: + stream_decremented, has_remaining = redis_connection.decrement_active_streams_and_check() + else: + has_remaining = redis_connection.has_active_streams() # Schedule smart cleanup if no active streams - if not redis_connection.has_active_streams(): + if not has_remaining and not profile_decremented: # Decrement profile counter immediately — don't defer to daemon thread state = redis_connection._get_connection_state() profile_id = state.m3u_profile_id if state else m3u_profile.id if profile_id: self._decrement_profile_connections(profile_id) + profile_decremented = True logger.info(f"[{client_id}] Profile counter decremented for profile {profile_id} on client disconnect") def delayed_cleanup(): @@ -1039,16 +1102,18 @@ class MultiWorkerVODConnectionManager: except Exception as e: logger.error(f"[{client_id}] Worker {self.worker_id} - Error in Redis-backed stream: {e}") - if not decremented: - redis_connection.decrement_active_streams() - decremented = True + if not stream_decremented: + stream_decremented, has_remaining = redis_connection.decrement_active_streams_and_check() + else: + has_remaining = redis_connection.has_active_streams() # Decrement profile counter immediately if no other active streams - if not redis_connection.has_active_streams(): + if not has_remaining and not profile_decremented: state = redis_connection._get_connection_state() profile_id = state.m3u_profile_id if state else m3u_profile.id if profile_id: self._decrement_profile_connections(profile_id) + profile_decremented = True logger.info(f"[{client_id}] Profile counter decremented for profile {profile_id} on stream error") # Smart cleanup on error - immediate cleanup since we're in error state # No connection_manager — profile already decremented above @@ -1056,8 +1121,31 @@ class MultiWorkerVODConnectionManager: yield b"Error: Stream interrupted" finally: - if not decremented: - redis_connection.decrement_active_streams() + if not stream_decremented: + stream_decremented, has_remaining = redis_connection.decrement_active_streams_and_check() + if stream_decremented and not has_remaining and not profile_decremented: + state = redis_connection._get_connection_state() + profile_id = state.m3u_profile_id if state else m3u_profile.id + if profile_id: + self._decrement_profile_connections(profile_id) + profile_decremented = True + logger.info(f"[{client_id}] Profile counter decremented for profile {profile_id} in finally block") + + # Delayed cleanup: wait 1s for seeking clients to reconnect + # before closing the provider connection and Redis keys. + # cleanup() re-checks active_streams under lock, so a + # reconnecting client that increments active_streams in + # time will prevent Redis key deletion. + def delayed_cleanup(): + time.sleep(1) + logger.info(f"[{client_id}] Worker {self.worker_id} - Checking for smart cleanup in finally block") + # No connection_manager — profile already decremented above + redis_connection.cleanup(current_worker_id=self.worker_id) + + import threading + cleanup_thread = threading.Thread(target=delayed_cleanup) + cleanup_thread.daemon = True + cleanup_thread.start() # Create streaming response response = StreamingHttpResponse( @@ -1272,14 +1360,14 @@ class MultiWorkerVODConnectionManager: # Convert bytes to strings if needed if isinstance(list(data.keys())[0], bytes): - data = {k.decode('utf-8'): v.decode('utf-8') for k, v in data.items()} + data = {k: v for k, v in data.items()} last_activity = float(data.get('last_activity', 0)) active_streams = int(data.get('active_streams', 0)) # Clean up if stale and no active streams if (current_time - last_activity > max_age_seconds) and active_streams == 0: - session_id = key.decode('utf-8').replace('vod_persistent_connection:', '') + session_id = key.replace('vod_persistent_connection:', '') logger.info(f"Cleaning up stale connection: {session_id}") # Clean up connection and related keys @@ -1376,7 +1464,7 @@ class MultiWorkerVODConnectionManager: if connection_data: # Convert bytes to strings if needed if isinstance(list(connection_data.keys())[0], bytes): - connection_data = {k.decode('utf-8'): v.decode('utf-8') for k, v in connection_data.items()} + connection_data = {k: v for k, v in connection_data.items()} profile_id = connection_data.get('m3u_profile_id') if profile_id: @@ -1438,7 +1526,7 @@ class MultiWorkerVODConnectionManager: # Convert bytes keys/values to strings if needed if isinstance(list(connection_data.keys())[0], bytes): - connection_data = {k.decode('utf-8'): v.decode('utf-8') for k, v in connection_data.items()} + connection_data = {k: v for k, v in connection_data.items()} # Check if content matches (using consolidated data) stored_content_type = connection_data.get('content_obj_type', '') @@ -1448,7 +1536,7 @@ class MultiWorkerVODConnectionManager: continue # Extract session ID - session_id = key.decode('utf-8').replace('vod_persistent_connection:', '') + session_id = key.replace('vod_persistent_connection:', '') # Check if Redis-backed connection exists and has no active streams redis_connection = RedisBackedVODConnection(session_id, self.redis_client) @@ -1526,4 +1614,4 @@ class MultiWorkerVODConnectionManager: return redis_connection.get_session_metadata() except Exception as e: logger.error(f"Error getting session info for {session_id}: {e}") - return None \ No newline at end of file + return None diff --git a/apps/proxy/vod_proxy/tests/__init__.py b/apps/proxy/vod_proxy/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/apps/proxy/vod_proxy/tests/test_profile_connections.py b/apps/proxy/vod_proxy/tests/test_profile_connections.py new file mode 100644 index 00000000..8e635479 --- /dev/null +++ b/apps/proxy/vod_proxy/tests/test_profile_connections.py @@ -0,0 +1,253 @@ +""" +Tests for VOD proxy profile connection counter fixes. + +Covers three race conditions in multi_worker_connection_manager: + 1. decrement_active_streams() return value was ignored — counter stuck on lock contention + 2. Non-atomic GET-then-DECR in _decrement_profile_connections() — counter could go negative + 3. has_active_streams() read without lock — race between decrement and check +""" + +from unittest.mock import MagicMock, patch, call +from django.test import TestCase + + +class FakeRedis: + """Minimal in-memory Redis stand-in for counter tests.""" + + def __init__(self): + self._data = {} + + def get(self, key): + val = self._data.get(key) + return str(val).encode() if val is not None else None + + def set(self, key, value, ex=None): + self._data[key] = int(value) + + def incr(self, key): + self._data[key] = self._data.get(key, 0) + 1 + return self._data[key] + + def decr(self, key): + self._data[key] = self._data.get(key, 0) - 1 + return self._data[key] + + def delete(self, key): + self._data.pop(key, None) + + def exists(self, key): + return key in self._data + + def pipeline(self): + return FakePipeline(self) + + +class FakePipeline: + def __init__(self, redis): + self._redis = redis + self._cmds = [] + + def incr(self, key): + self._cmds.append(('incr', key)) + return self + + def decr(self, key): + self._cmds.append(('decr', key)) + return self + + def execute(self): + results = [] + for cmd, key in self._cmds: + results.append(getattr(self._redis, cmd)(key)) + self._cmds = [] + return results + + +class MultiWorkerManagerImportMixin: + """Mixin to import the manager class with patched Django/Redis deps.""" + + @classmethod + def get_manager_class(cls): + import importlib + import sys + + # Stub out heavy Django deps so we can import the module standalone + for mod in ['apps.vod.models', 'apps.m3u.models', 'core.utils']: + if mod not in sys.modules: + sys.modules[mod] = MagicMock() + + from apps.proxy.vod_proxy.multi_worker_connection_manager import ( + MultiWorkerVODConnectionManager, + RedisBackedVODConnection, + ) + return MultiWorkerVODConnectionManager, RedisBackedVODConnection + + +class TestDecrementProfileConnectionsAtomic(TestCase): + """Bug 2: _decrement_profile_connections must be atomic (no GET-then-DECR).""" + + def _make_manager(self, redis): + _, _ = MultiWorkerManagerImportMixin.get_manager_class() + from apps.proxy.vod_proxy.multi_worker_connection_manager import MultiWorkerVODConnectionManager + mgr = MultiWorkerVODConnectionManager.__new__(MultiWorkerVODConnectionManager) + mgr.redis_client = redis + mgr.worker_id = 'test-worker' + return mgr + + def test_decrement_does_not_go_negative(self): + """Counter must be clamped to 0, never go negative.""" + redis = FakeRedis() + redis.set('profile_connections:1', 0) + mgr = self._make_manager(redis) + + result = mgr._decrement_profile_connections(1) + + self.assertEqual(result, 0) + self.assertEqual(int(redis._data.get('profile_connections:1', 0)), 0) + + def test_decrement_from_one_reaches_zero(self): + """Normal single decrement should reach 0.""" + redis = FakeRedis() + redis.set('profile_connections:1', 1) + mgr = self._make_manager(redis) + + result = mgr._decrement_profile_connections(1) + + self.assertEqual(result, 0) + + def test_concurrent_decrements_clamp_to_zero(self): + """Two concurrent decrements of a counter at 1 must not leave it at -1.""" + redis = FakeRedis() + redis.set('profile_connections:1', 1) + mgr = self._make_manager(redis) + + # Simulate two concurrent decrements (both fire before either reads back) + mgr._decrement_profile_connections(1) + mgr._decrement_profile_connections(1) + + final = int(redis._data.get('profile_connections:1', 0)) + self.assertGreaterEqual(final, 0, "Counter must not go negative after concurrent decrements") + + +class TestDecrementActiveStreamsAndCheck(TestCase): + """Bug 1 & 3: decrement_active_streams_and_check() must be atomic.""" + + def _make_connection(self, redis, session_id='test-session'): + from apps.proxy.vod_proxy.multi_worker_connection_manager import RedisBackedVODConnection + conn = RedisBackedVODConnection.__new__(RedisBackedVODConnection) + conn.session_id = session_id + conn.redis_client = redis + conn.connection_key = f'vod_connection:{session_id}' + conn.lock_key = f'vod_lock:{session_id}' + conn.local_session = None + conn._lock_acquired = False + return conn + + def _make_state(self, active_streams=1, profile_id=7): + from apps.proxy.vod_proxy.multi_worker_connection_manager import SerializableConnectionState + state = SerializableConnectionState.__new__(SerializableConnectionState) + state.session_id = 'test-session' + state.stream_url = 'http://example.com/stream.mkv' + state.headers = {} + state.m3u_profile_id = profile_id + state.active_streams = active_streams + state.last_activity = 0 + state.worker_id = 'test-worker' + state.content_type = None + state.content_length = None + state.final_url = None + state.request_count = 0 + state.bytes_sent = 0 + state.content_obj_type = None + state.content_uuid = None + state.content_name = None + state.client_ip = None + state.client_user_agent = None + state.utc_start = None + state.utc_end = None + state.offset = None + state.connection_type = 'redis' + state.created_at = 0 + return state + + def test_returns_success_and_no_remaining_when_last_stream(self): + """When active_streams goes 1->0, should return (True, False).""" + from apps.proxy.vod_proxy.multi_worker_connection_manager import RedisBackedVODConnection + conn = MagicMock(spec=RedisBackedVODConnection) + conn.session_id = 'test' + + state = MagicMock() + state.active_streams = 1 + + conn._acquire_lock.return_value = True + conn._get_connection_state.return_value = state + conn._save_connection_state.return_value = True + conn._release_lock.return_value = None + + # Call the real method on the mock instance + result = RedisBackedVODConnection.decrement_active_streams_and_check(conn) + + self.assertEqual(result, (True, False)) + self.assertEqual(state.active_streams, 0) + + def test_returns_success_and_remaining_when_other_streams_active(self): + """When active_streams goes 2->1, should return (True, True).""" + from apps.proxy.vod_proxy.multi_worker_connection_manager import RedisBackedVODConnection + conn = MagicMock(spec=RedisBackedVODConnection) + conn.session_id = 'test' + + state = MagicMock() + state.active_streams = 2 + + conn._acquire_lock.return_value = True + conn._get_connection_state.return_value = state + conn._save_connection_state.return_value = True + conn._release_lock.return_value = None + + result = RedisBackedVODConnection.decrement_active_streams_and_check(conn) + + self.assertEqual(result, (True, True)) + self.assertEqual(state.active_streams, 1) + + def test_returns_failure_and_assumes_remaining_on_lock_contention(self): + """Lock contention must return (False, True) — assume streams remain to be safe.""" + from apps.proxy.vod_proxy.multi_worker_connection_manager import RedisBackedVODConnection + conn = MagicMock(spec=RedisBackedVODConnection) + conn.session_id = 'test' + conn._acquire_lock.return_value = False + + result = RedisBackedVODConnection.decrement_active_streams_and_check(conn) + + self.assertEqual(result, (False, True)) + conn._get_connection_state.assert_not_called() + + def test_returns_failure_when_already_at_zero(self): + """When active_streams is already 0, should return (False, False).""" + from apps.proxy.vod_proxy.multi_worker_connection_manager import RedisBackedVODConnection + conn = MagicMock(spec=RedisBackedVODConnection) + conn.session_id = 'test' + + state = MagicMock() + state.active_streams = 0 + + conn._acquire_lock.return_value = True + conn._get_connection_state.return_value = state + conn._release_lock.return_value = None + + result = RedisBackedVODConnection.decrement_active_streams_and_check(conn) + + self.assertEqual(result, (False, False)) + conn._save_connection_state.assert_not_called() + + def test_lock_always_released_even_on_exception(self): + """Lock must be released even if an exception occurs inside.""" + from apps.proxy.vod_proxy.multi_worker_connection_manager import RedisBackedVODConnection + conn = MagicMock(spec=RedisBackedVODConnection) + conn.session_id = 'test' + conn._acquire_lock.return_value = True + conn._get_connection_state.side_effect = RuntimeError("Redis exploded") + + with self.assertRaises(RuntimeError): + RedisBackedVODConnection.decrement_active_streams_and_check(conn) + + conn._release_lock.assert_called_once() diff --git a/apps/proxy/vod_proxy/urls.py b/apps/proxy/vod_proxy/urls.py index f48f70e0..19f4ed4f 100644 --- a/apps/proxy/vod_proxy/urls.py +++ b/apps/proxy/vod_proxy/urls.py @@ -1,26 +1,20 @@ from django.urls import path from . import views +from .views import stream_vod app_name = 'vod_proxy' urlpatterns = [ # Generic VOD streaming with session ID in path (for compatibility) - path('//', views.VODStreamView.as_view(), name='vod_stream_with_session'), - path('////', views.VODStreamView.as_view(), name='vod_stream_with_session_and_profile'), + path('//', stream_vod, name='vod_stream_with_session'), + path('////', stream_vod, name='vod_stream_with_session_and_profile'), # Generic VOD streaming (supports movies, episodes, series) - legacy patterns - path('/', views.VODStreamView.as_view(), name='vod_stream'), - path('///', views.VODStreamView.as_view(), name='vod_stream_with_profile'), - - # VOD playlist generation - path('playlist/', views.VODPlaylistView.as_view(), name='vod_playlist'), - path('playlist//', views.VODPlaylistView.as_view(), name='vod_playlist_with_profile'), - - # Position tracking - path('position//', views.VODPositionView.as_view(), name='vod_position'), + path('/', stream_vod, name='vod_stream'), + path('///', stream_vod, name='vod_stream_with_profile'), # VOD Stats - path('stats/', views.VODStatsView.as_view(), name='vod_stats'), + path('stats/', views.vod_stats, name='vod_stats'), # Stop VOD client connection path('stop_client/', views.stop_vod_client, name='stop_vod_client'), diff --git a/apps/proxy/vod_proxy/views.py b/apps/proxy/vod_proxy/views.py index 948604d6..a1dcf9a3 100644 --- a/apps/proxy/vod_proxy/views.py +++ b/apps/proxy/vod_proxy/views.py @@ -7,114 +7,393 @@ import time import random import logging import requests -from django.http import StreamingHttpResponse, JsonResponse, Http404, HttpResponse +from django.http import JsonResponse, Http404, HttpResponse from django.shortcuts import get_object_or_404 from django.views.decorators.csrf import csrf_exempt -from django.utils.decorators import method_decorator -from django.views import View from apps.vod.models import Movie, Series, Episode -from apps.m3u.models import M3UAccount, M3UAccountProfile -from apps.proxy.vod_proxy.connection_manager import VODConnectionManager +from apps.m3u.models import M3UAccountProfile from apps.proxy.vod_proxy.multi_worker_connection_manager import MultiWorkerVODConnectionManager, infer_content_type_from_url, get_vod_client_stop_key -from .utils import get_client_info, create_vod_response +from .utils import get_client_info +from rest_framework.decorators import api_view, permission_classes +from rest_framework.permissions import AllowAny +from apps.accounts.models import User +from apps.accounts.permissions import IsAdmin +from apps.proxy.utils import check_user_stream_limits +from dispatcharr.utils import network_access_allowed logger = logging.getLogger(__name__) +_request_times = {} -@method_decorator(csrf_exempt, name='dispatch') -class VODStreamView(View): - """Handle VOD streaming requests with M3U profile support""" - def get(self, request, content_type, content_id, session_id=None, profile_id=None): - """ - Stream VOD content (movies or series episodes) with session-based connection reuse +def _get_content_and_relation(content_type, content_id, preferred_m3u_account_id=None, preferred_stream_id=None): + """Get the content object and its M3U relation""" + try: + logger.info(f"[CONTENT-LOOKUP] Looking up {content_type} with UUID {content_id}") + if preferred_m3u_account_id: + logger.info(f"[CONTENT-LOOKUP] Preferred M3U account ID: {preferred_m3u_account_id}") + if preferred_stream_id: + logger.info(f"[CONTENT-LOOKUP] Preferred stream ID: {preferred_stream_id}") - Args: - content_type: 'movie', 'series', or 'episode' - content_id: ID of the content - session_id: Optional session ID from URL path (for persistent connections) - profile_id: Optional M3U profile ID for authentication - """ - logger.info(f"[VOD-REQUEST] Starting VOD stream request: {content_type}/{content_id}, session: {session_id}, profile: {profile_id}") - logger.info(f"[VOD-REQUEST] Full request path: {request.get_full_path()}") - logger.info(f"[VOD-REQUEST] Request method: {request.method}") - logger.info(f"[VOD-REQUEST] Request headers: {dict(request.headers)}") + if content_type == 'movie': + content_obj = get_object_or_404(Movie, uuid=content_id) + logger.info(f"[CONTENT-FOUND] Movie: {content_obj.name} (ID: {content_obj.id})") - try: - client_ip, client_user_agent = get_client_info(request) + # Filter by preferred stream ID first (most specific) + relations_query = content_obj.m3u_relations.filter(m3u_account__is_active=True) + if preferred_stream_id: + specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() + if specific_relation: + logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") + return content_obj, specific_relation + else: + logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") - # Extract timeshift parameters from query string - # Support multiple timeshift parameter formats - utc_start = request.GET.get('utc_start') or request.GET.get('start') or request.GET.get('playliststart') - utc_end = request.GET.get('utc_end') or request.GET.get('end') or request.GET.get('playlistend') - offset = request.GET.get('offset') or request.GET.get('seek') or request.GET.get('t') + # Filter by preferred M3U account if specified + if preferred_m3u_account_id: + specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() + if specific_relation: + logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") + return content_obj, specific_relation + else: + logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") - # VLC specific timeshift parameters - if not utc_start and not offset: - # Check for VLC-style timestamp parameters - if 'timestamp' in request.GET: - offset = request.GET.get('timestamp') - elif 'time' in request.GET: - offset = request.GET.get('time') + # Get the highest priority active relation (fallback or default) + relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() - # Session ID now comes from URL path parameter - # Remove legacy query parameter extraction since we're using path-based routing + if relation: + logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") - # Extract Range header for seeking support - range_header = request.META.get('HTTP_RANGE') + return content_obj, relation - logger.info(f"[VOD-TIMESHIFT] Timeshift params - utc_start: {utc_start}, utc_end: {utc_end}, offset: {offset}") - logger.info(f"[VOD-SESSION] Session ID: {session_id}") + elif content_type == 'episode': + content_obj = get_object_or_404(Episode, uuid=content_id) + logger.info(f"[CONTENT-FOUND] Episode: {content_obj.name} (ID: {content_obj.id}, Series: {content_obj.series.name})") - # Log all query parameters for debugging - if request.GET: - logger.debug(f"[VOD-PARAMS] All query params: {dict(request.GET)}") + # Filter by preferred stream ID first (most specific) + relations_query = content_obj.m3u_relations.filter(m3u_account__is_active=True) + if preferred_stream_id: + specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() + if specific_relation: + logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") + return content_obj, specific_relation + else: + logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") - if range_header: - logger.info(f"[VOD-RANGE] Range header: {range_header}") + # Filter by preferred M3U account if specified + if preferred_m3u_account_id: + specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() + if specific_relation: + logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") + return content_obj, specific_relation + else: + logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") - # Parse the range to understand what position VLC is seeking to - try: - if 'bytes=' in range_header: - range_part = range_header.replace('bytes=', '') - if '-' in range_part: - start_byte, end_byte = range_part.split('-', 1) - if start_byte: - start_pos_mb = int(start_byte) / (1024 * 1024) - logger.info(f"[VOD-SEEK] Seeking to byte position: {start_byte} (~{start_pos_mb:.1f} MB)") - if int(start_byte) > 0: - logger.info(f"[VOD-SEEK] *** ACTUAL SEEK DETECTED *** Position: {start_pos_mb:.1f} MB") - else: - logger.info(f"[VOD-SEEK] Open-ended range request (from start)") - if end_byte: - end_pos_mb = int(end_byte) / (1024 * 1024) - logger.info(f"[VOD-SEEK] End position: {end_byte} bytes (~{end_pos_mb:.1f} MB)") - except Exception as e: - logger.warning(f"[VOD-SEEK] Could not parse range header: {e}") + # Get the highest priority active relation (fallback or default) + relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() - # Simple seek detection - track rapid requests - current_time = time.time() - request_key = f"{client_ip}:{content_type}:{content_id}" + if relation: + logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") - if not hasattr(self.__class__, '_request_times'): - self.__class__._request_times = {} + return content_obj, relation - if request_key in self.__class__._request_times: - time_diff = current_time - self.__class__._request_times[request_key] - if time_diff < 5.0: - logger.info(f"[VOD-SEEK] Rapid request detected ({time_diff:.1f}s) - likely seeking") + elif content_type == 'series': + # For series, get the first episode + series = get_object_or_404(Series, uuid=content_id) + logger.info(f"[CONTENT-FOUND] Series: {series.name} (ID: {series.id})") + episode = series.episodes.first() + if not episode: + logger.error(f"[CONTENT-ERROR] No episodes found for series {series.name}") + return None, None - self.__class__._request_times[request_key] = current_time + logger.info(f"[CONTENT-FOUND] First episode: {episode.name} (ID: {episode.id})") + + # Filter by preferred stream ID first (most specific) + relations_query = episode.m3u_relations.filter(m3u_account__is_active=True) + if preferred_stream_id: + specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() + if specific_relation: + logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") + return episode, specific_relation + else: + logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") + + # Filter by preferred M3U account if specified + if preferred_m3u_account_id: + specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() + if specific_relation: + logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") + return episode, specific_relation + else: + logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") + + # Get the highest priority active relation (fallback or default) + relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() + + if relation: + logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") + + return episode, relation + else: + logger.error(f"[CONTENT-ERROR] Invalid content type: {content_type}") + return None, None + + except Exception as e: + logger.error(f"Error getting content object: {e}") + return None, None + +def _get_stream_url_from_relation(relation): + """Get stream URL from the M3U relation""" + try: + # Log the relation type and available attributes + logger.info(f"[VOD-URL] Relation type: {type(relation).__name__}") + logger.info(f"[VOD-URL] Account type: {relation.m3u_account.account_type}") + logger.info(f"[VOD-URL] Stream ID: {getattr(relation, 'stream_id', 'N/A')}") + + # First try the get_stream_url method (this should build URLs dynamically) + if hasattr(relation, 'get_stream_url'): + url = relation.get_stream_url() + if url: + logger.info(f"[VOD-URL] Built URL from get_stream_url(): {url}") + return url else: - logger.info(f"[VOD-RANGE] No Range header - full content request") + logger.warning(f"[VOD-URL] get_stream_url() returned None") - logger.info(f"[VOD-CLIENT] Client info - IP: {client_ip}, User-Agent: {client_user_agent[:50]}...") + logger.error(f"[VOD-URL] Relation has no get_stream_url method or it failed") + return None + except Exception as e: + logger.error(f"[VOD-URL] Error getting stream URL from relation: {e}", exc_info=True) + return None - # If no session ID, create one and redirect to path-based URL - if not session_id: - new_session_id = f"vod_{int(time.time() * 1000)}_{random.randint(1000, 9999)}" - logger.info(f"[VOD-SESSION] Creating new session: {new_session_id}") +def _get_m3u_profile(m3u_account, profile_id, session_id=None): + """Get appropriate M3U profile for streaming using Redis-based viewer counts + Args: + m3u_account: M3UAccount instance + profile_id: Optional specific profile ID requested + session_id: Optional session ID to check for existing connections + + Returns: + tuple: (M3UAccountProfile, current_connections) or None if no profile found + """ + try: + from core.utils import RedisClient + redis_client = RedisClient.get_client() + + if not redis_client: + logger.warning("Redis not available, falling back to default profile") + default_profile = M3UAccountProfile.objects.filter( + m3u_account=m3u_account, + is_active=True, + is_default=True + ).first() + return (default_profile, 0) if default_profile else None + + # Check if this session already has an active connection + if session_id: + persistent_connection_key = f"vod_persistent_connection:{session_id}" + connection_data = redis_client.hgetall(persistent_connection_key) + + if connection_data: + existing_profile_id = connection_data.get('m3u_profile_id') + if existing_profile_id: + try: + existing_profile = M3UAccountProfile.objects.get( + id=int(existing_profile_id), + m3u_account=m3u_account, + is_active=True + ) + # Get current connections for logging + profile_connections_key = f"profile_connections:{existing_profile.id}" + current_connections = int(redis_client.get(profile_connections_key) or 0) + + logger.info(f"[PROFILE-SELECTION] Session {session_id} reusing existing profile {existing_profile.id}: {current_connections}/{existing_profile.max_streams} connections") + return (existing_profile, current_connections) + except (M3UAccountProfile.DoesNotExist, ValueError): + logger.warning(f"[PROFILE-SELECTION] Session {session_id} has invalid profile ID {existing_profile_id}, selecting new profile") + except Exception as e: + logger.warning(f"[PROFILE-SELECTION] Error checking existing profile for session {session_id}: {e}") + else: + logger.debug(f"[PROFILE-SELECTION] Session {session_id} exists but has no profile ID stored") # If specific profile requested, try to use it + if profile_id: + try: + profile = M3UAccountProfile.objects.get( + id=profile_id, + m3u_account=m3u_account, + is_active=True + ) + # Check Redis-based current connections + profile_connections_key = f"profile_connections:{profile.id}" + current_connections = int(redis_client.get(profile_connections_key) or 0) + + if profile.max_streams == 0 or current_connections < profile.max_streams: + logger.info(f"[PROFILE-SELECTION] Using requested profile {profile.id}: {current_connections}/{profile.max_streams} connections") + return (profile, current_connections) + else: + logger.warning(f"[PROFILE-SELECTION] Requested profile {profile.id} is at capacity: {current_connections}/{profile.max_streams}") + except M3UAccountProfile.DoesNotExist: + logger.warning(f"[PROFILE-SELECTION] Requested profile {profile_id} not found") + + # Get active profiles ordered by priority (default first) + m3u_profiles = M3UAccountProfile.objects.filter( + m3u_account=m3u_account, + is_active=True + ) + + default_profile = m3u_profiles.filter(is_default=True).first() + if not default_profile: + logger.error(f"[PROFILE-SELECTION] No default profile found for M3U account {m3u_account.id}") + return None + + # Check profiles in order: default first, then others + profiles = [default_profile] + list(m3u_profiles.filter(is_default=False)) + + for profile in profiles: + profile_connections_key = f"profile_connections:{profile.id}" + current_connections = int(redis_client.get(profile_connections_key) or 0) + + # Check if profile has available connection slots + if profile.max_streams == 0 or current_connections < profile.max_streams: + logger.info(f"[PROFILE-SELECTION] Selected profile {profile.id} ({profile.name}): {current_connections}/{profile.max_streams} connections") + return (profile, current_connections) + else: + logger.debug(f"[PROFILE-SELECTION] Profile {profile.id} at capacity: {current_connections}/{profile.max_streams}") + + # All profiles are at capacity - return None to trigger error response + logger.error(f"[PROFILE-SELECTION] All profiles at capacity for M3U account {m3u_account.id}, rejecting request") + return None + + except Exception as e: + logger.error(f"Error getting M3U profile: {e}") + return None + +def _transform_url(original_url, m3u_profile): + """Transform URL based on M3U profile settings""" + try: + import regex + + if not original_url: + return None + + search_pattern = m3u_profile.search_pattern + replace_pattern = m3u_profile.replace_pattern + # Convert JS-style backreferences in replace: $ -> \g, $1 -> \1 + safe_replace_pattern = regex.sub(r'\$<([^>]+)>', r'\\g<\1>', replace_pattern) + safe_replace_pattern = regex.sub(r'\$(\d+)', r'\\\1', safe_replace_pattern) + + if search_pattern and replace_pattern: + # regex module accepts JS-style (?...) named groups natively + transformed_url = regex.sub(search_pattern, safe_replace_pattern, original_url) + return transformed_url + + return original_url + + except Exception as e: + logger.error(f"Error transforming URL: {e}") + return original_url + +@api_view(["GET"]) +@permission_classes([AllowAny]) +def stream_vod(request, content_type, content_id, session_id=None, profile_id=None, user=None): + """ + Stream VOD content (movies or series episodes) with session-based connection reuse + + Args: + content_type: 'movie', 'series', or 'episode' + content_id: ID of the content + session_id: Optional session ID from URL path (for persistent connections) + profile_id: Optional M3U profile ID for authentication + """ + if not network_access_allowed(request, "STREAMS"): + return JsonResponse({"error": "Forbidden"}, status=403) + + logger.info(f"[VOD-REQUEST] Starting VOD stream request: {content_type}/{content_id}, session: {session_id}, profile: {profile_id}") + logger.info(f"[VOD-REQUEST] Full request path: {request.get_full_path()}") + logger.info(f"[VOD-REQUEST] Request method: {request.method}") + logger.info(f"[VOD-REQUEST] Request headers: {dict(request.headers)}") + + try: + client_ip, client_user_agent = get_client_info(request) + + # Extract timeshift parameters from query string + # Support multiple timeshift parameter formats + utc_start = request.GET.get('utc_start') or request.GET.get('start') or request.GET.get('playliststart') + utc_end = request.GET.get('utc_end') or request.GET.get('end') or request.GET.get('playlistend') + offset = request.GET.get('offset') or request.GET.get('seek') or request.GET.get('t') + + # VLC specific timeshift parameters + if not utc_start and not offset: + # Check for VLC-style timestamp parameters + if 'timestamp' in request.GET: + offset = request.GET.get('timestamp') + elif 'time' in request.GET: + offset = request.GET.get('time') + + # Session ID now comes from URL path parameter + # Remove legacy query parameter extraction since we're using path-based routing + + # Extract Range header for seeking support + range_header = request.META.get('HTTP_RANGE') + + logger.info(f"[VOD-TIMESHIFT] Timeshift params - utc_start: {utc_start}, utc_end: {utc_end}, offset: {offset}") + logger.info(f"[VOD-SESSION] Session ID: {session_id}") + + # Log all query parameters for debugging + if request.GET: + logger.debug(f"[VOD-PARAMS] All query params: {dict(request.GET)}") + + if range_header: + logger.info(f"[VOD-RANGE] Range header: {range_header}") + + # Parse the range to understand what position VLC is seeking to + try: + if 'bytes=' in range_header: + range_part = range_header.replace('bytes=', '') + if '-' in range_part: + start_byte, end_byte = range_part.split('-', 1) + if start_byte: + start_pos_mb = int(start_byte) / (1024 * 1024) + logger.info(f"[VOD-SEEK] Seeking to byte position: {start_byte} (~{start_pos_mb:.1f} MB)") + if int(start_byte) > 0: + logger.info(f"[VOD-SEEK] *** ACTUAL SEEK DETECTED *** Position: {start_pos_mb:.1f} MB") + else: + logger.info(f"[VOD-SEEK] Open-en`ded range request (from start)") + if end_byte: + end_pos_mb = int(end_byte) / (1024 * 1024) + logger.info(f"[VOD-SEEK] End position: {end_byte} bytes (~{end_pos_mb:.1f} MB)") + except Exception as e: + logger.warning(f"[VOD-SEEK] Could not parse range header: {e}") + + # Simple seek detection - track rapid requests + current_time = time.time() + request_key = f"{client_ip}:{content_type}:{content_id}" + + if request_key in _request_times: + time_diff = current_time - _request_times[request_key] + if time_diff < 5.0: + logger.info(f"[VOD-SEEK] Rapid request detected ({time_diff:.1f}s) - likely seeking") + + _request_times[request_key] = current_time + else: + logger.info(f"[VOD-RANGE] No Range header - full content request") + + logger.info(f"[VOD-CLIENT] Client info - IP: {client_ip}, User-Agent: {client_user_agent[:50]}...") + + # If no session ID, create one and redirect to path-based URL + if not session_id: + new_session_id = f"vod_{int(time.time() * 1000)}_{random.randint(1000, 9999)}" + logger.info(f"[VOD-SESSION] Creating new session: {new_session_id}") + + # Preserve any query parameters (except session_id) + query_params = dict(request.GET) + query_params.pop('session_id', None) # Remove if present + + if user: + redirect_url = f"{request.path}?session_id={new_session_id}" + if query_params: + query_string = urlencode(query_params, doseq=True) + redirect_url = f"{redirect_url}&{query_string}" + else: # Build redirect URL with session ID in path, preserve query parameters path_parts = request.path.rstrip('/').split('/') @@ -124,10 +403,6 @@ class VODStreamView(View): else: new_path = f"{'/'.join(path_parts)}/{new_session_id}" - # Preserve any query parameters (except session_id) - query_params = dict(request.GET) - query_params.pop('session_id', None) # Remove if present - if query_params: from urllib.parse import urlencode query_string = urlencode(query_params, doseq=True) @@ -135,899 +410,541 @@ class VODStreamView(View): else: redirect_url = new_path - logger.info(f"[VOD-SESSION] Redirecting to path-based URL: {redirect_url}") + logger.info(f"[VOD-SESSION] Redirecting to path-based URL: {redirect_url}") - return HttpResponse( - status=301, - headers={'Location': redirect_url} - ) - - # Extract preferred M3U account ID and stream ID from query parameters - preferred_m3u_account_id = request.GET.get('m3u_account_id') - preferred_stream_id = request.GET.get('stream_id') - - if preferred_m3u_account_id: - try: - preferred_m3u_account_id = int(preferred_m3u_account_id) - except (ValueError, TypeError): - logger.warning(f"[VOD-PARAM] Invalid m3u_account_id parameter: {preferred_m3u_account_id}") - preferred_m3u_account_id = None - - if preferred_stream_id: - logger.info(f"[VOD-PARAM] Preferred stream ID: {preferred_stream_id}") - - # Get the content object and its relation - content_obj, relation = self._get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id) - if not content_obj or not relation: - logger.error(f"[VOD-ERROR] Content or relation not found: {content_type} {content_id}") - raise Http404(f"Content not found: {content_type} {content_id}") - - logger.info(f"[VOD-CONTENT] Found content: {getattr(content_obj, 'name', 'Unknown')}") - - # Get M3U account from relation - m3u_account = relation.m3u_account - logger.info(f"[VOD-ACCOUNT] Using M3U account: {m3u_account.name}") - - # Get stream URL from relation - stream_url = self._get_stream_url_from_relation(relation) - logger.info(f"[VOD-CONTENT] Content URL: {stream_url or 'No URL found'}") - - if not stream_url: - logger.error(f"[VOD-ERROR] No stream URL available for {content_type} {content_id}") - return HttpResponse("No stream URL available", status=503) - - # Get M3U profile (returns profile and current connection count) - profile_result = self._get_m3u_profile(m3u_account, profile_id, session_id) - - if not profile_result or not profile_result[0]: - logger.error(f"[VOD-ERROR] No suitable M3U profile found for {content_type} {content_id}") - return HttpResponse("No available stream", status=503) - - m3u_profile, current_connections = profile_result - logger.info(f"[VOD-PROFILE] Using M3U profile: {m3u_profile.id} (max_streams: {m3u_profile.max_streams}, current: {current_connections})") - - # Connection tracking is handled by the connection manager - # Transform URL based on profile - final_stream_url = self._transform_url(stream_url, m3u_profile) - logger.info(f"[VOD-URL] Final stream URL: {final_stream_url}") - - # Validate stream URL - if not final_stream_url or not final_stream_url.startswith(('http://', 'https://')): - logger.error(f"[VOD-ERROR] Invalid stream URL: {final_stream_url}") - return HttpResponse("Invalid stream URL", status=500) - - # Get connection manager (Redis-backed for multi-worker support) - connection_manager = MultiWorkerVODConnectionManager.get_instance() - - # Stream the content with session-based connection reuse - logger.info("[VOD-STREAM] Calling connection manager to stream content") - response = connection_manager.stream_content_with_session( - session_id=session_id, - content_obj=content_obj, - stream_url=final_stream_url, - m3u_profile=m3u_profile, - client_ip=client_ip, - client_user_agent=client_user_agent, - request=request, - utc_start=utc_start, - utc_end=utc_end, - offset=offset, - range_header=range_header + return HttpResponse( + status=301, + headers={'Location': redirect_url} ) - logger.info(f"[VOD-SUCCESS] Stream response created successfully, type: {type(response)}") - return response + if user: + if not check_user_stream_limits(user, session_id, media_id=content_id): + return JsonResponse( + {"error": f"Stream limit exceeded ({user.stream_limit} concurrent streams allowed)"}, + status=429 + ) - except Exception as e: - logger.error(f"[VOD-EXCEPTION] Error streaming {content_type} {content_id}: {e}", exc_info=True) - return HttpResponse(f"Streaming error: {str(e)}", status=500) + # Extract preferred M3U account ID and stream ID from query parameters + preferred_m3u_account_id = request.GET.get('m3u_account_id') + preferred_stream_id = request.GET.get('stream_id') - def head(self, request, content_type, content_id, session_id=None, profile_id=None): - """ - Handle HEAD requests for FUSE filesystem integration + if preferred_m3u_account_id: + try: + preferred_m3u_account_id = int(preferred_m3u_account_id) + except (ValueError, TypeError): + logger.warning(f"[VOD-PARAM] Invalid m3u_account_id parameter: {preferred_m3u_account_id}") + preferred_m3u_account_id = None - Returns content length and session URL header for subsequent GET requests - """ - logger.info(f"[VOD-HEAD] HEAD request: {content_type}/{content_id}, session: {session_id}, profile: {profile_id}") + if preferred_stream_id: + logger.info(f"[VOD-PARAM] Preferred stream ID: {preferred_stream_id}") - try: - # Get client info for M3U profile selection - client_ip, client_user_agent = get_client_info(request) - logger.info(f"[VOD-HEAD] Client info - IP: {client_ip}, User-Agent: {client_user_agent[:50] if client_user_agent else 'None'}...") + # Get the content object and its relation + content_obj, relation = _get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id) + if not content_obj or not relation: + logger.error(f"[VOD-ERROR] Content or relation not found: {content_type} {content_id}") + raise Http404(f"Content not found: {content_type} {content_id}") - # If no session ID, create one (same logic as GET) - if not session_id: - new_session_id = f"vod_{int(time.time() * 1000)}_{random.randint(1000, 9999)}" - logger.info(f"[VOD-HEAD] Creating new session for HEAD: {new_session_id}") + logger.info(f"[VOD-CONTENT] Found content: {getattr(content_obj, 'name', 'Unknown')}") - # Build session URL for response header - path_parts = request.path.rstrip('/').split('/') - if profile_id: - session_url = f"{'/'.join(path_parts)}/{new_session_id}/{profile_id}/" - else: - session_url = f"{'/'.join(path_parts)}/{new_session_id}" + # Get M3U account from relation + m3u_account = relation.m3u_account + logger.info(f"[VOD-ACCOUNT] Using M3U account: {m3u_account.name}") - session_id = new_session_id + # Get stream URL from relation + stream_url = _get_stream_url_from_relation(relation) + logger.info(f"[VOD-CONTENT] Content URL: {stream_url or 'No URL found'}") + + if not stream_url: + logger.error(f"[VOD-ERROR] No stream URL available for {content_type} {content_id}") + return HttpResponse("No stream URL available", status=503) + + # Get M3U profile (returns profile and current connection count) + profile_result = _get_m3u_profile(m3u_account, profile_id, session_id) + + if not profile_result or not profile_result[0]: + logger.error(f"[VOD-ERROR] No suitable M3U profile found for {content_type} {content_id}") + return HttpResponse("No available stream", status=503) + + m3u_profile, current_connections = profile_result + logger.info(f"[VOD-PROFILE] Using M3U profile: {m3u_profile.id} (max_streams: {m3u_profile.max_streams}, current: {current_connections})") + + # Connection tracking is handled by the connection manager + # Transform URL based on profile + final_stream_url = _transform_url(stream_url, m3u_profile) + logger.info(f"[VOD-URL] Final stream URL: {final_stream_url}") + + # Validate stream URL + if not final_stream_url or not final_stream_url.startswith(('http://', 'https://')): + logger.error(f"[VOD-ERROR] Invalid stream URL: {final_stream_url}") + return HttpResponse("Invalid stream URL", status=500) + + # Get connection manager (Redis-backed for multi-worker support) + connection_manager = MultiWorkerVODConnectionManager.get_instance() + + # Stream the content with session-based connection reuse + logger.info("[VOD-STREAM] Calling connection manager to stream content") + response = connection_manager.stream_content_with_session( + session_id=session_id, + content_obj=content_obj, + stream_url=final_stream_url, + m3u_profile=m3u_profile, + client_ip=client_ip, + client_user_agent=client_user_agent, + request=request, + utc_start=utc_start, + utc_end=utc_end, + offset=offset, + range_header=range_header, + user=user, + ) + + logger.info(f"[VOD-SUCCESS] Stream response created successfully, type: {type(response)}") + return response + + except Exception as e: + logger.error(f"[VOD-EXCEPTION] Error streaming {content_type} {content_id}: {e}", exc_info=True) + return HttpResponse(f"Streaming error: {str(e)}", status=500) + +@api_view(["HEAD"]) +@permission_classes([AllowAny]) +def head_vod(request, content_type, content_id, session_id=None, profile_id=None): + """ + Handle HEAD requests for FUSE filesystem integration + + Returns content length and session URL header for subsequent GET requests + """ + if not network_access_allowed(request, "STREAMS"): + return JsonResponse({"error": "Forbidden"}, status=403) + + logger.info(f"[VOD-HEAD] HEAD request: {content_type}/{content_id}, session: {session_id}, profile: {profile_id}") + + try: + # Get client info for M3U profile selection + client_ip, client_user_agent = get_client_info(request) + logger.info(f"[VOD-HEAD] Client info - IP: {client_ip}, User-Agent: {client_user_agent[:50] if client_user_agent else 'None'}...") + + # If no session ID, create one (same logic as GET) + if not session_id: + new_session_id = f"vod_{int(time.time() * 1000)}_{random.randint(1000, 9999)}" + logger.info(f"[VOD-HEAD] Creating new session for HEAD: {new_session_id}") + + # Build session URL for response header + path_parts = request.path.rstrip('/').split('/') + if profile_id: + session_url = f"{'/'.join(path_parts)}/{new_session_id}/{profile_id}/" else: - # Session already in URL, construct the current session URL - session_url = request.path - logger.info(f"[VOD-HEAD] Using existing session: {session_id}") + session_url = f"{'/'.join(path_parts)}/{new_session_id}" - # Extract preferred M3U account ID and stream ID from query parameters - preferred_m3u_account_id = request.GET.get('m3u_account_id') - preferred_stream_id = request.GET.get('stream_id') + session_id = new_session_id + else: + # Session already in URL, construct the current session URL + session_url = request.path + logger.info(f"[VOD-HEAD] Using existing session: {session_id}") - if preferred_m3u_account_id: - try: - preferred_m3u_account_id = int(preferred_m3u_account_id) - except (ValueError, TypeError): - logger.warning(f"[VOD-HEAD] Invalid m3u_account_id parameter: {preferred_m3u_account_id}") - preferred_m3u_account_id = None + # Extract preferred M3U account ID and stream ID from query parameters + preferred_m3u_account_id = request.GET.get('m3u_account_id') + preferred_stream_id = request.GET.get('stream_id') - if preferred_stream_id: - logger.info(f"[VOD-HEAD] Preferred stream ID: {preferred_stream_id}") + if preferred_m3u_account_id: + try: + preferred_m3u_account_id = int(preferred_m3u_account_id) + except (ValueError, TypeError): + logger.warning(f"[VOD-HEAD] Invalid m3u_account_id parameter: {preferred_m3u_account_id}") + preferred_m3u_account_id = None - # Get content and relation (same as GET) - content_obj, relation = self._get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id) - if not content_obj or not relation: - logger.error(f"[VOD-HEAD] Content or relation not found: {content_type} {content_id}") - return HttpResponse("Content not found", status=404) + if preferred_stream_id: + logger.info(f"[VOD-HEAD] Preferred stream ID: {preferred_stream_id}") - # Get M3U account and stream URL - m3u_account = relation.m3u_account - stream_url = self._get_stream_url_from_relation(relation) - if not stream_url: - logger.error(f"[VOD-HEAD] No stream URL available for {content_type} {content_id}") - return HttpResponse("No stream URL available", status=503) + # Get content and relation (same as GET) + content_obj, relation = _get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id) + if not content_obj or not relation: + logger.error(f"[VOD-HEAD] Content or relation not found: {content_type} {content_id}") + return HttpResponse("Content not found", status=404) - # Get M3U profile (returns profile and current connection count) - profile_result = self._get_m3u_profile(m3u_account, profile_id, session_id) - if not profile_result or not profile_result[0]: - logger.error(f"[VOD-HEAD] No M3U profile found or all profiles at capacity") - return HttpResponse("No available stream", status=503) + # Get M3U account and stream URL + m3u_account = relation.m3u_account + stream_url = _get_stream_url_from_relation(relation) + if not stream_url: + logger.error(f"[VOD-HEAD] No stream URL available for {content_type} {content_id}") + return HttpResponse("No stream URL available", status=503) - m3u_profile, current_connections = profile_result + # Get M3U profile (returns profile and current connection count) + profile_result = _get_m3u_profile(m3u_account, profile_id, session_id) + if not profile_result or not profile_result[0]: + logger.error(f"[VOD-HEAD] No M3U profile found or all profiles at capacity") + return HttpResponse("No available stream", status=503) - # Transform URL if needed - final_stream_url = self._transform_url(stream_url, m3u_profile) + m3u_profile, current_connections = profile_result - # Make a small range GET request to get content length since providers don't support HEAD - # We'll use a tiny range to minimize data transfer but get the headers we need - # Use M3U account's user agent as primary, client user agent as fallback - m3u_user_agent = m3u_account.get_user_agent().user_agent if m3u_account.get_user_agent() else None - headers = { - 'User-Agent': m3u_user_agent or client_user_agent or 'Dispatcharr/1.0', - 'Accept': '*/*', - 'Range': 'bytes=0-1' # Request only first 2 bytes - } + # Transform URL if needed + final_stream_url = _transform_url(stream_url, m3u_profile) - logger.info(f"[VOD-HEAD] Making small range GET request to provider: {final_stream_url}") - response = requests.get(final_stream_url, headers=headers, timeout=30, allow_redirects=True, stream=True) + # Make a small range GET request to get content length since providers don't support HEAD + # We'll use a tiny range to minimize data transfer but get the headers we need + # Use M3U account's user agent as primary, client user agent as fallback + m3u_user_agent = m3u_account.get_user_agent().user_agent if m3u_account.get_user_agent() else None + headers = { + 'User-Agent': m3u_user_agent or client_user_agent or 'Dispatcharr/1.0', + 'Accept': '*/*', + 'Range': 'bytes=0-1' # Request only first 2 bytes + } - # Check for range support - should be 206 for partial content - if response.status_code == 206: - # Parse Content-Range header to get total file size - content_range = response.headers.get('Content-Range', '') - if content_range: - # Content-Range: bytes 0-1/1234567890 - total_size = content_range.split('/')[-1] - logger.info(f"[VOD-HEAD] Got file size from Content-Range: {total_size}") - else: - logger.warning(f"[VOD-HEAD] No Content-Range header in 206 response") - total_size = response.headers.get('Content-Length', '0') - elif response.status_code == 200: - # Server doesn't support range requests, use Content-Length from full response + logger.info(f"[VOD-HEAD] Making small range GET request to provider: {final_stream_url}") + response = requests.get(final_stream_url, headers=headers, timeout=30, allow_redirects=True, stream=True) + + # Check for range support - should be 206 for partial content + if response.status_code == 206: + # Parse Content-Range header to get total file size + content_range = response.headers.get('Content-Range', '') + if content_range: + # Content-Range: bytes 0-1/1234567890 + total_size = content_range.split('/')[-1] + logger.info(f"[VOD-HEAD] Got file size from Content-Range: {total_size}") + else: + logger.warning(f"[VOD-HEAD] No Content-Range header in 206 response") total_size = response.headers.get('Content-Length', '0') - logger.info(f"[VOD-HEAD] Server doesn't support ranges, got Content-Length: {total_size}") - else: - logger.error(f"[VOD-HEAD] Provider GET request failed: {response.status_code}") - return HttpResponse("Provider error", status=response.status_code) + elif response.status_code == 200: + # Server doesn't support range requests, use Content-Length from full response + total_size = response.headers.get('Content-Length', '0') + logger.info(f"[VOD-HEAD] Server doesn't support ranges, got Content-Length: {total_size}") + else: + logger.error(f"[VOD-HEAD] Provider GET request failed: {response.status_code}") + return HttpResponse("Provider error", status=response.status_code) - # Close the small range request - we don't need to keep this connection - response.close() + # Close the small range request - we don't need to keep this connection + response.close() - # Store the total content length in Redis for the persistent connection to use - try: - import redis - from django.conf import settings - redis_host = getattr(settings, 'REDIS_HOST', 'localhost') - redis_port = int(getattr(settings, 'REDIS_PORT', 6379)) - redis_db = int(getattr(settings, 'REDIS_DB', 0)) - redis_password = getattr(settings, 'REDIS_PASSWORD', '') - redis_user = getattr(settings, 'REDIS_USER', '') - r = redis.StrictRedis( - host=redis_host, - port=redis_port, - db=redis_db, - password=redis_password if redis_password else None, - username=redis_user if redis_user else None, - decode_responses=True - ) - content_length_key = f"vod_content_length:{session_id}" - r.set(content_length_key, total_size, ex=1800) # Store for 30 minutes - logger.info(f"[VOD-HEAD] Stored total content length {total_size} for session {session_id}") - except Exception as e: - logger.error(f"[VOD-HEAD] Failed to store content length in Redis: {e}") - - # Now create a persistent connection for the session (if one doesn't exist) - # This ensures the FUSE GET requests will reuse the same connection - - connection_manager = MultiWorkerVODConnectionManager.get_instance() - - logger.info(f"[VOD-HEAD] Pre-creating persistent connection for session: {session_id}") - - # We don't actually stream content here, just ensure connection is ready - # The actual GET requests from FUSE will use the persistent connection - - # Use the total_size we extracted from the range response - provider_content_type = response.headers.get('Content-Type') - - if provider_content_type: - content_type_header = provider_content_type - logger.info(f"[VOD-HEAD] Using provider Content-Type: {content_type_header}") - else: - # Provider didn't send Content-Type, infer from URL - inferred_content_type = infer_content_type_from_url(final_stream_url) - if inferred_content_type: - content_type_header = inferred_content_type - logger.info(f"[VOD-HEAD] Provider missing Content-Type, inferred from URL: {content_type_header}") - else: - content_type_header = 'video/mp4' - logger.info(f"[VOD-HEAD] No Content-Type from provider and could not infer from URL, using default: {content_type_header}") - - logger.info(f"[VOD-HEAD] Provider response - Total Size: {total_size}, Type: {content_type_header}") - - # Create response with content length and session URL header - head_response = HttpResponse() - head_response['Content-Length'] = total_size - head_response['Content-Type'] = content_type_header - head_response['Accept-Ranges'] = 'bytes' - - # Custom header with session URL for FUSE - head_response['X-Session-URL'] = session_url - head_response['X-Dispatcharr-Session'] = session_id - - logger.info(f"[VOD-HEAD] Returning HEAD response with session URL: {session_url}") - return head_response - - except Exception as e: - logger.error(f"[VOD-HEAD] Error in HEAD request: {e}", exc_info=True) - return HttpResponse(f"HEAD error: {str(e)}", status=500) - - def _get_content_and_relation(self, content_type, content_id, preferred_m3u_account_id=None, preferred_stream_id=None): - """Get the content object and its M3U relation""" + # Store the total content length in Redis for the persistent connection to use try: - logger.info(f"[CONTENT-LOOKUP] Looking up {content_type} with UUID {content_id}") - if preferred_m3u_account_id: - logger.info(f"[CONTENT-LOOKUP] Preferred M3U account ID: {preferred_m3u_account_id}") - if preferred_stream_id: - logger.info(f"[CONTENT-LOOKUP] Preferred stream ID: {preferred_stream_id}") - - if content_type == 'movie': - content_obj = get_object_or_404(Movie, uuid=content_id) - logger.info(f"[CONTENT-FOUND] Movie: {content_obj.name} (ID: {content_obj.id})") - - # Filter by preferred stream ID first (most specific) - relations_query = content_obj.m3u_relations.filter(m3u_account__is_active=True) - if preferred_stream_id: - specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() - if specific_relation: - logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") - return content_obj, specific_relation - else: - logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") - - # Filter by preferred M3U account if specified - if preferred_m3u_account_id: - specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() - if specific_relation: - logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") - return content_obj, specific_relation - else: - logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") - - # Get the highest priority active relation (fallback or default) - relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() - - if relation: - logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") - - return content_obj, relation - - elif content_type == 'episode': - content_obj = get_object_or_404(Episode, uuid=content_id) - logger.info(f"[CONTENT-FOUND] Episode: {content_obj.name} (ID: {content_obj.id}, Series: {content_obj.series.name})") - - # Filter by preferred stream ID first (most specific) - relations_query = content_obj.m3u_relations.filter(m3u_account__is_active=True) - if preferred_stream_id: - specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() - if specific_relation: - logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") - return content_obj, specific_relation - else: - logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") - - # Filter by preferred M3U account if specified - if preferred_m3u_account_id: - specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() - if specific_relation: - logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") - return content_obj, specific_relation - else: - logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") - - # Get the highest priority active relation (fallback or default) - relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() - - if relation: - logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") - - return content_obj, relation - - elif content_type == 'series': - # For series, get the first episode - series = get_object_or_404(Series, uuid=content_id) - logger.info(f"[CONTENT-FOUND] Series: {series.name} (ID: {series.id})") - episode = series.episodes.first() - if not episode: - logger.error(f"[CONTENT-ERROR] No episodes found for series {series.name}") - return None, None - - logger.info(f"[CONTENT-FOUND] First episode: {episode.name} (ID: {episode.id})") - - # Filter by preferred stream ID first (most specific) - relations_query = episode.m3u_relations.filter(m3u_account__is_active=True) - if preferred_stream_id: - specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() - if specific_relation: - logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") - return episode, specific_relation - else: - logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") - - # Filter by preferred M3U account if specified - if preferred_m3u_account_id: - specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() - if specific_relation: - logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") - return episode, specific_relation - else: - logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") - - # Get the highest priority active relation (fallback or default) - relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() - - if relation: - logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") - - return episode, relation - else: - logger.error(f"[CONTENT-ERROR] Invalid content type: {content_type}") - return None, None - - except Exception as e: - logger.error(f"Error getting content object: {e}") - return None, None - - def _get_stream_url_from_relation(self, relation): - """Get stream URL from the M3U relation""" - try: - # Log the relation type and available attributes - logger.info(f"[VOD-URL] Relation type: {type(relation).__name__}") - logger.info(f"[VOD-URL] Account type: {relation.m3u_account.account_type}") - logger.info(f"[VOD-URL] Stream ID: {getattr(relation, 'stream_id', 'N/A')}") - - # First try the get_stream_url method (this should build URLs dynamically) - if hasattr(relation, 'get_stream_url'): - url = relation.get_stream_url() - if url: - logger.info(f"[VOD-URL] Built URL from get_stream_url(): {url}") - return url - else: - logger.warning(f"[VOD-URL] get_stream_url() returned None") - - logger.error(f"[VOD-URL] Relation has no get_stream_url method or it failed") - return None - except Exception as e: - logger.error(f"[VOD-URL] Error getting stream URL from relation: {e}", exc_info=True) - return None - - def _get_m3u_profile(self, m3u_account, profile_id, session_id=None): - """Get appropriate M3U profile for streaming using Redis-based viewer counts - - Args: - m3u_account: M3UAccount instance - profile_id: Optional specific profile ID requested - session_id: Optional session ID to check for existing connections - - Returns: - tuple: (M3UAccountProfile, current_connections) or None if no profile found - """ - try: - from core.utils import RedisClient - redis_client = RedisClient.get_client() - - if not redis_client: - logger.warning("Redis not available, falling back to default profile") - default_profile = M3UAccountProfile.objects.filter( - m3u_account=m3u_account, - is_active=True, - is_default=True - ).first() - return (default_profile, 0) if default_profile else None - - # Check if this session already has an active connection - if session_id: - persistent_connection_key = f"vod_persistent_connection:{session_id}" - connection_data = redis_client.hgetall(persistent_connection_key) - - if connection_data: - # Decode Redis hash data - decoded_data = {} - for k, v in connection_data.items(): - k_str = k.decode('utf-8') if isinstance(k, bytes) else k - v_str = v.decode('utf-8') if isinstance(v, bytes) else v - decoded_data[k_str] = v_str - - existing_profile_id = decoded_data.get('m3u_profile_id') - if existing_profile_id: - try: - existing_profile = M3UAccountProfile.objects.get( - id=int(existing_profile_id), - m3u_account=m3u_account, - is_active=True - ) - # Get current connections for logging - profile_connections_key = f"profile_connections:{existing_profile.id}" - current_connections = int(redis_client.get(profile_connections_key) or 0) - - logger.info(f"[PROFILE-SELECTION] Session {session_id} reusing existing profile {existing_profile.id}: {current_connections}/{existing_profile.max_streams} connections") - return (existing_profile, current_connections) - except (M3UAccountProfile.DoesNotExist, ValueError): - logger.warning(f"[PROFILE-SELECTION] Session {session_id} has invalid profile ID {existing_profile_id}, selecting new profile") - except Exception as e: - logger.warning(f"[PROFILE-SELECTION] Error checking existing profile for session {session_id}: {e}") - else: - logger.debug(f"[PROFILE-SELECTION] Session {session_id} exists but has no profile ID stored") # If specific profile requested, try to use it - if profile_id: - try: - profile = M3UAccountProfile.objects.get( - id=profile_id, - m3u_account=m3u_account, - is_active=True - ) - # Check Redis-based current connections - profile_connections_key = f"profile_connections:{profile.id}" - current_connections = int(redis_client.get(profile_connections_key) or 0) - - if profile.max_streams == 0 or current_connections < profile.max_streams: - logger.info(f"[PROFILE-SELECTION] Using requested profile {profile.id}: {current_connections}/{profile.max_streams} connections") - return (profile, current_connections) - else: - logger.warning(f"[PROFILE-SELECTION] Requested profile {profile.id} is at capacity: {current_connections}/{profile.max_streams}") - except M3UAccountProfile.DoesNotExist: - logger.warning(f"[PROFILE-SELECTION] Requested profile {profile_id} not found") - - # Get active profiles ordered by priority (default first) - m3u_profiles = M3UAccountProfile.objects.filter( - m3u_account=m3u_account, - is_active=True + import redis + from django.conf import settings + redis_host = getattr(settings, 'REDIS_HOST', 'localhost') + redis_port = int(getattr(settings, 'REDIS_PORT', 6379)) + redis_db = int(getattr(settings, 'REDIS_DB', 0)) + redis_password = getattr(settings, 'REDIS_PASSWORD', '') + redis_user = getattr(settings, 'REDIS_USER', '') + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) + r = redis.StrictRedis( + host=redis_host, + port=redis_port, + db=redis_db, + password=redis_password if redis_password else None, + username=redis_user if redis_user else None, + decode_responses=True, + **ssl_params ) - - default_profile = m3u_profiles.filter(is_default=True).first() - if not default_profile: - logger.error(f"[PROFILE-SELECTION] No default profile found for M3U account {m3u_account.id}") - return None - - # Check profiles in order: default first, then others - profiles = [default_profile] + list(m3u_profiles.filter(is_default=False)) - - for profile in profiles: - profile_connections_key = f"profile_connections:{profile.id}" - current_connections = int(redis_client.get(profile_connections_key) or 0) - - # Check if profile has available connection slots - if profile.max_streams == 0 or current_connections < profile.max_streams: - logger.info(f"[PROFILE-SELECTION] Selected profile {profile.id} ({profile.name}): {current_connections}/{profile.max_streams} connections") - return (profile, current_connections) - else: - logger.debug(f"[PROFILE-SELECTION] Profile {profile.id} at capacity: {current_connections}/{profile.max_streams}") - - # All profiles are at capacity - return None to trigger error response - logger.error(f"[PROFILE-SELECTION] All profiles at capacity for M3U account {m3u_account.id}, rejecting request") - return None - + content_length_key = f"vod_content_length:{session_id}" + r.set(content_length_key, total_size, ex=1800) # Store for 30 minutes + logger.info(f"[VOD-HEAD] Stored total content length {total_size} for session {session_id}") except Exception as e: - logger.error(f"Error getting M3U profile: {e}") - return None + logger.error(f"[VOD-HEAD] Failed to store content length in Redis: {e}") - def _transform_url(self, original_url, m3u_profile): - """Transform URL based on M3U profile settings""" - try: - import re + # Now create a persistent connection for the session (if one doesn't exist) + # This ensures the FUSE GET requests will reuse the same connection - if not original_url: - return None + connection_manager = MultiWorkerVODConnectionManager.get_instance() - search_pattern = m3u_profile.search_pattern - replace_pattern = m3u_profile.replace_pattern - safe_replace_pattern = re.sub(r'\$(\d+)', r'\\\1', replace_pattern) + logger.info(f"[VOD-HEAD] Pre-creating persistent connection for session: {session_id}") - if search_pattern and replace_pattern: - transformed_url = re.sub(search_pattern, safe_replace_pattern, original_url) - return transformed_url + # We don't actually stream content here, just ensure connection is ready + # The actual GET requests from FUSE will use the persistent connection - return original_url + # Use the total_size we extracted from the range response + provider_content_type = response.headers.get('Content-Type') - except Exception as e: - logger.error(f"Error transforming URL: {e}") - return original_url + if provider_content_type: + content_type_header = provider_content_type + logger.info(f"[VOD-HEAD] Using provider Content-Type: {content_type_header}") + else: + # Provider didn't send Content-Type, infer from URL + inferred_content_type = infer_content_type_from_url(final_stream_url) + if inferred_content_type: + content_type_header = inferred_content_type + logger.info(f"[VOD-HEAD] Provider missing Content-Type, inferred from URL: {content_type_header}") + else: + content_type_header = 'video/mp4' + logger.info(f"[VOD-HEAD] No Content-Type from provider and could not infer from URL, using default: {content_type_header}") -@method_decorator(csrf_exempt, name='dispatch') -class VODPlaylistView(View): - """Generate M3U playlists for VOD content""" + logger.info(f"[VOD-HEAD] Provider response - Total Size: {total_size}, Type: {content_type_header}") - def get(self, request, profile_id=None): - """Generate VOD playlist""" - try: - # Get profile if specified - m3u_profile = None - if profile_id: + # Create response with content length and session URL header + head_response = HttpResponse() + head_response['Content-Length'] = total_size + head_response['Content-Type'] = content_type_header + head_response['Accept-Ranges'] = 'bytes' + + # Custom header with session URL for FUSE + head_response['X-Session-URL'] = session_url + head_response['X-Dispatcharr-Session'] = session_id + + logger.info(f"[VOD-HEAD] Returning HEAD response with session URL: {session_url}") + return head_response + + except Exception as e: + logger.error(f"[VOD-HEAD] Error in HEAD request: {e}", exc_info=True) + return HttpResponse(f"HEAD error: {str(e)}", status=500) + +@api_view(["GET"]) +@permission_classes([IsAdmin]) +def vod_stats(request): + """Get current VOD connection statistics""" + try: + connection_manager = MultiWorkerVODConnectionManager.get_instance() + redis_client = connection_manager.redis_client + + if not redis_client: + return JsonResponse({'error': 'Redis not available'}, status=500) + + # Get all VOD persistent connections (consolidated data) + pattern = "vod_persistent_connection:*" + cursor = 0 + connections = [] + current_time = time.time() + + while True: + cursor, keys = redis_client.scan(cursor, match=pattern, count=100) + + for key in keys: try: - m3u_profile = M3UAccountProfile.objects.get( - id=profile_id, - is_active=True - ) - except M3UAccountProfile.DoesNotExist: - return HttpResponse("Profile not found", status=404) + connection_data = redis_client.hgetall(key) - # Generate playlist content - playlist_content = self._generate_playlist(m3u_profile) + if connection_data: + # Extract session ID from key + session_id = key.replace('vod_persistent_connection:', '') - response = HttpResponse(playlist_content, content_type='application/vnd.apple.mpegurl') - response['Content-Disposition'] = 'attachment; filename="vod_playlist.m3u8"' - return response + # Decode Redis hash data + combined_data = {} + for k, v in connection_data.items(): + combined_data[k] = v - except Exception as e: - logger.error(f"Error generating VOD playlist: {e}") - return HttpResponse("Playlist generation error", status=500) + # Get content info from the connection data (using correct field names) + content_type = combined_data.get('content_obj_type', 'unknown') + content_uuid = combined_data.get('content_uuid', 'unknown') + client_id = session_id - def _generate_playlist(self, m3u_profile=None): - """Generate M3U playlist content for VOD""" - lines = ["#EXTM3U"] + # Get content info with enhanced metadata + content_name = "Unknown" + content_metadata = {} + try: + if content_type == 'movie': + content_obj = Movie.objects.select_related('logo').get(uuid=content_uuid) + content_name = content_obj.name - # Add movies - movies = Movie.objects.filter(is_active=True) - if m3u_profile: - movies = movies.filter(m3u_account=m3u_profile.m3u_account) + # Get duration from content object + duration_secs = None + if hasattr(content_obj, 'duration_secs') and content_obj.duration_secs: + duration_secs = content_obj.duration_secs - for movie in movies: - profile_param = f"?profile={m3u_profile.id}" if m3u_profile else "" - lines.append(f'#EXTINF:-1 tvg-id="{movie.tmdb_id}" group-title="Movies",{movie.title}') - lines.append(f'/proxy/vod/movie/{movie.uuid}/{profile_param}') + # If we don't have duration_secs, try to calculate it from file size and position data + if not duration_secs: + file_size_bytes = int(combined_data.get('total_content_size', 0)) + last_seek_byte = int(combined_data.get('last_seek_byte', 0)) + last_seek_percentage = float(combined_data.get('last_seek_percentage', 0.0)) - # Add series - series_list = Series.objects.filter(is_active=True) - if m3u_profile: - series_list = series_list.filter(m3u_account=m3u_profile.m3u_account) - - for series in series_list: - for episode in series.episodes.all(): - profile_param = f"?profile={m3u_profile.id}" if m3u_profile else "" - episode_title = f"{series.title} - S{episode.season_number:02d}E{episode.episode_number:02d}" - lines.append(f'#EXTINF:-1 tvg-id="{series.tmdb_id}" group-title="Series",{episode_title}') - lines.append(f'/proxy/vod/episode/{episode.uuid}/{profile_param}') - - return '\n'.join(lines) - - -@method_decorator(csrf_exempt, name='dispatch') -class VODPositionView(View): - """Handle VOD position updates""" - - def post(self, request, content_id): - """Update playback position for VOD content""" - try: - import json - data = json.loads(request.body) - client_id = data.get('client_id') - position = data.get('position', 0) - - # Find the content object - content_obj = None - try: - content_obj = Movie.objects.get(uuid=content_id) - except Movie.DoesNotExist: - try: - content_obj = Episode.objects.get(uuid=content_id) - except Episode.DoesNotExist: - return JsonResponse({'error': 'Content not found'}, status=404) - - # Here you could store the position in a model or cache - # For now, just return success - logger.info(f"Position update for {content_obj.__class__.__name__} {content_id}: {position}s") - - return JsonResponse({ - 'success': True, - 'content_id': str(content_id), - 'position': position - }) - - except Exception as e: - logger.error(f"Error updating VOD position: {e}") - return JsonResponse({'error': str(e)}, status=500) - - -@method_decorator(csrf_exempt, name='dispatch') -class VODStatsView(View): - """Get VOD connection statistics""" - - def get(self, request): - """Get current VOD connection statistics""" - try: - connection_manager = MultiWorkerVODConnectionManager.get_instance() - redis_client = connection_manager.redis_client - - if not redis_client: - return JsonResponse({'error': 'Redis not available'}, status=500) - - # Get all VOD persistent connections (consolidated data) - pattern = "vod_persistent_connection:*" - cursor = 0 - connections = [] - current_time = time.time() - - while True: - cursor, keys = redis_client.scan(cursor, match=pattern, count=100) - - for key in keys: - try: - key_str = key.decode('utf-8') if isinstance(key, bytes) else key - connection_data = redis_client.hgetall(key) - - if connection_data: - # Extract session ID from key - session_id = key_str.replace('vod_persistent_connection:', '') - - # Decode Redis hash data - combined_data = {} - for k, v in connection_data.items(): - k_str = k.decode('utf-8') if isinstance(k, bytes) else k - v_str = v.decode('utf-8') if isinstance(v, bytes) else v - combined_data[k_str] = v_str - - # Get content info from the connection data (using correct field names) - content_type = combined_data.get('content_obj_type', 'unknown') - content_uuid = combined_data.get('content_uuid', 'unknown') - client_id = session_id - - # Get content info with enhanced metadata - content_name = "Unknown" - content_metadata = {} - try: - if content_type == 'movie': - content_obj = Movie.objects.select_related('logo').get(uuid=content_uuid) - content_name = content_obj.name - - # Get duration from content object - duration_secs = None - if hasattr(content_obj, 'duration_secs') and content_obj.duration_secs: - duration_secs = content_obj.duration_secs - - # If we don't have duration_secs, try to calculate it from file size and position data - if not duration_secs: - file_size_bytes = int(combined_data.get('total_content_size', 0)) - last_seek_byte = int(combined_data.get('last_seek_byte', 0)) - last_seek_percentage = float(combined_data.get('last_seek_percentage', 0.0)) - - # Calculate position if we have the required data - if file_size_bytes and file_size_bytes > 0 and last_seek_percentage > 0: - # If we know the seek percentage and current time position, we can estimate duration - # But we need to know the current time position in seconds first - # For now, let's use a rough estimate based on file size and typical bitrates - # This is a fallback - ideally duration should be in the database - estimated_duration = 6000 # 100 minutes as default for movies - duration_secs = estimated_duration - - content_metadata = { - 'year': content_obj.year, - 'rating': content_obj.rating, - 'genre': content_obj.genre, - 'duration_secs': duration_secs, - 'description': content_obj.description, - 'logo_url': content_obj.logo.url if content_obj.logo else None, - 'tmdb_id': content_obj.tmdb_id, - 'imdb_id': content_obj.imdb_id - } - elif content_type == 'episode': - content_obj = Episode.objects.select_related('series', 'series__logo').get(uuid=content_uuid) - content_name = f"{content_obj.series.name} - {content_obj.name}" - - # Get duration from content object - duration_secs = None - if hasattr(content_obj, 'duration_secs') and content_obj.duration_secs: - duration_secs = content_obj.duration_secs - - # If we don't have duration_secs, estimate for episodes - if not duration_secs: - estimated_duration = 2400 # 40 minutes as default for episodes + # Calculate position if we have the required data + if file_size_bytes and file_size_bytes > 0 and last_seek_percentage > 0: + # If we know the seek percentage and current time position, we can estimate duration + # But we need to know the current time position in seconds first + # For now, let's use a rough estimate based on file size and typical bitrates + # This is a fallback - ideally duration should be in the database + estimated_duration = 6000 # 100 minutes as default for movies duration_secs = estimated_duration - content_metadata = { - 'series_name': content_obj.series.name, - 'episode_name': content_obj.name, - 'season_number': content_obj.season_number, - 'episode_number': content_obj.episode_number, - 'air_date': content_obj.air_date.isoformat() if content_obj.air_date else None, - 'rating': content_obj.rating, - 'duration_secs': duration_secs, - 'description': content_obj.description, - 'logo_url': content_obj.series.logo.url if content_obj.series.logo else None, - 'series_year': content_obj.series.year, - 'series_genre': content_obj.series.genre, - 'tmdb_id': content_obj.tmdb_id, - 'imdb_id': content_obj.imdb_id - } + content_metadata = { + 'year': content_obj.year, + 'rating': content_obj.rating, + 'genre': content_obj.genre, + 'duration_secs': duration_secs, + 'description': content_obj.description, + 'logo_url': content_obj.logo.url if content_obj.logo else None, + 'tmdb_id': content_obj.tmdb_id, + 'imdb_id': content_obj.imdb_id + } + elif content_type == 'episode': + content_obj = Episode.objects.select_related('series', 'series__logo').get(uuid=content_uuid) + content_name = f"{content_obj.series.name} - {content_obj.name}" + + # Get duration from content object + duration_secs = None + if hasattr(content_obj, 'duration_secs') and content_obj.duration_secs: + duration_secs = content_obj.duration_secs + + # If we don't have duration_secs, estimate for episodes + if not duration_secs: + estimated_duration = 2400 # 40 minutes as default for episodes + duration_secs = estimated_duration + + content_metadata = { + 'series_name': content_obj.series.name, + 'episode_name': content_obj.name, + 'season_number': content_obj.season_number, + 'episode_number': content_obj.episode_number, + 'air_date': content_obj.air_date.isoformat() if content_obj.air_date else None, + 'rating': content_obj.rating, + 'duration_secs': duration_secs, + 'description': content_obj.description, + 'logo_url': content_obj.series.logo.url if content_obj.series.logo else None, + 'series_year': content_obj.series.year, + 'series_genre': content_obj.series.genre, + 'tmdb_id': content_obj.tmdb_id, + 'imdb_id': content_obj.imdb_id + } + except: + pass + + # Get M3U profile information + m3u_profile_info = {} + m3u_profile_id = combined_data.get('m3u_profile_id') + if m3u_profile_id: + try: + from apps.m3u.models import M3UAccountProfile + profile = M3UAccountProfile.objects.select_related('m3u_account').get(id=m3u_profile_id) + m3u_profile_info = { + 'profile_name': profile.name, + 'account_name': profile.m3u_account.name, + 'account_id': profile.m3u_account.id, + 'max_streams': profile.m3u_account.max_streams, + 'm3u_profile_id': int(m3u_profile_id) + } + except Exception as e: + logger.warning(f"Could not fetch M3U profile {m3u_profile_id}: {e}") + + # Also try to get profile info from stored data if database lookup fails + if not m3u_profile_info and combined_data.get('m3u_profile_name'): + m3u_profile_info = { + 'profile_name': combined_data.get('m3u_profile_name', 'Unknown Profile'), + 'm3u_profile_id': combined_data.get('m3u_profile_id'), + 'account_name': 'Unknown Account' # We don't store account name directly + } + + # Calculate estimated current position based on seek percentage or last known position + last_known_position = int(combined_data.get('position_seconds', 0)) + last_position_update = combined_data.get('last_position_update') + last_seek_percentage = float(combined_data.get('last_seek_percentage', 0.0)) + last_seek_timestamp = float(combined_data.get('last_seek_timestamp', 0.0)) + estimated_position = last_known_position + + # If we have seek percentage and content duration, calculate position from that + if last_seek_percentage > 0 and content_metadata.get('duration_secs'): + try: + duration_secs = int(content_metadata['duration_secs']) + # Calculate position from seek percentage + seek_position = int((last_seek_percentage / 100) * duration_secs) + + # If we have a recent seek timestamp, add elapsed time since seek + if last_seek_timestamp > 0: + elapsed_since_seek = current_time - last_seek_timestamp + # Add elapsed time but don't exceed content duration + estimated_position = min( + seek_position + int(elapsed_since_seek), + duration_secs + ) + else: + estimated_position = seek_position + except (ValueError, TypeError): + pass + elif last_position_update and content_metadata.get('duration_secs'): + # Fallback: use time-based estimation from position_seconds + try: + update_timestamp = float(last_position_update) + elapsed_since_update = current_time - update_timestamp + # Add elapsed time to last known position, but don't exceed content duration + estimated_position = min( + last_known_position + int(elapsed_since_update), + int(content_metadata['duration_secs']) + ) + except (ValueError, TypeError): + # If timestamp parsing fails, fall back to last known position + estimated_position = last_known_position + + connection_info = { + 'content_type': content_type, + 'content_uuid': content_uuid, + 'content_name': content_name, + 'content_metadata': content_metadata, + 'm3u_profile': m3u_profile_info, + 'client_id': client_id, + 'client_ip': combined_data.get('client_ip', 'Unknown'), + 'user_id': combined_data.get('user_id', '0'), + 'user_agent': combined_data.get('client_user_agent', 'Unknown'), + 'connected_at': combined_data.get('created_at'), + 'last_activity': combined_data.get('last_activity'), + 'm3u_profile_id': m3u_profile_id, + 'position_seconds': estimated_position, # Use estimated position + 'last_known_position': last_known_position, # Include raw position for debugging + 'last_position_update': last_position_update, # Include timestamp for frontend use + 'bytes_sent': int(combined_data.get('bytes_sent', 0)), + # Seek/range information for position calculation and frontend display + 'last_seek_byte': int(combined_data.get('last_seek_byte', 0)), + 'last_seek_percentage': float(combined_data.get('last_seek_percentage', 0.0)), + 'total_content_size': int(combined_data.get('total_content_size', 0)), + 'last_seek_timestamp': float(combined_data.get('last_seek_timestamp', 0.0)) + } + + # Calculate connection duration + duration_calculated = False + if connection_info['connected_at']: + try: + connected_time = float(connection_info['connected_at']) + duration = current_time - connected_time + connection_info['duration'] = int(duration) + duration_calculated = True except: pass - # Get M3U profile information - m3u_profile_info = {} - m3u_profile_id = combined_data.get('m3u_profile_id') - if m3u_profile_id: - try: - from apps.m3u.models import M3UAccountProfile - profile = M3UAccountProfile.objects.select_related('m3u_account').get(id=m3u_profile_id) - m3u_profile_info = { - 'profile_name': profile.name, - 'account_name': profile.m3u_account.name, - 'account_id': profile.m3u_account.id, - 'max_streams': profile.m3u_account.max_streams, - 'm3u_profile_id': int(m3u_profile_id) - } - except Exception as e: - logger.warning(f"Could not fetch M3U profile {m3u_profile_id}: {e}") + # Fallback: use last_activity if connected_at is not available + if not duration_calculated and connection_info['last_activity']: + try: + last_activity_time = float(connection_info['last_activity']) + # Estimate connection duration using client_id timestamp if available + if connection_info['client_id'].startswith('vod_'): + # Extract timestamp from client_id (format: vod_timestamp_random) + parts = connection_info['client_id'].split('_') + if len(parts) >= 2: + client_start_time = float(parts[1]) / 1000.0 # Convert ms to seconds + duration = current_time - client_start_time + connection_info['duration'] = int(duration) + duration_calculated = True + except: + pass - # Also try to get profile info from stored data if database lookup fails - if not m3u_profile_info and combined_data.get('m3u_profile_name'): - m3u_profile_info = { - 'profile_name': combined_data.get('m3u_profile_name', 'Unknown Profile'), - 'm3u_profile_id': combined_data.get('m3u_profile_id'), - 'account_name': 'Unknown Account' # We don't store account name directly - } + # Final fallback + if not duration_calculated: + connection_info['duration'] = 0 - # Calculate estimated current position based on seek percentage or last known position - last_known_position = int(combined_data.get('position_seconds', 0)) - last_position_update = combined_data.get('last_position_update') - last_seek_percentage = float(combined_data.get('last_seek_percentage', 0.0)) - last_seek_timestamp = float(combined_data.get('last_seek_timestamp', 0.0)) - estimated_position = last_known_position + connections.append(connection_info) - # If we have seek percentage and content duration, calculate position from that - if last_seek_percentage > 0 and content_metadata.get('duration_secs'): - try: - duration_secs = int(content_metadata['duration_secs']) - # Calculate position from seek percentage - seek_position = int((last_seek_percentage / 100) * duration_secs) + except Exception as e: + logger.error(f"Error processing connection key {key}: {e}") - # If we have a recent seek timestamp, add elapsed time since seek - if last_seek_timestamp > 0: - elapsed_since_seek = current_time - last_seek_timestamp - # Add elapsed time but don't exceed content duration - estimated_position = min( - seek_position + int(elapsed_since_seek), - duration_secs - ) - else: - estimated_position = seek_position - except (ValueError, TypeError): - pass - elif last_position_update and content_metadata.get('duration_secs'): - # Fallback: use time-based estimation from position_seconds - try: - update_timestamp = float(last_position_update) - elapsed_since_update = current_time - update_timestamp - # Add elapsed time to last known position, but don't exceed content duration - estimated_position = min( - last_known_position + int(elapsed_since_update), - int(content_metadata['duration_secs']) - ) - except (ValueError, TypeError): - # If timestamp parsing fails, fall back to last known position - estimated_position = last_known_position + if cursor == 0: + break - connection_info = { - 'content_type': content_type, - 'content_uuid': content_uuid, - 'content_name': content_name, - 'content_metadata': content_metadata, - 'm3u_profile': m3u_profile_info, - 'client_id': client_id, - 'client_ip': combined_data.get('client_ip', 'Unknown'), - 'user_agent': combined_data.get('client_user_agent', 'Unknown'), - 'connected_at': combined_data.get('created_at'), - 'last_activity': combined_data.get('last_activity'), - 'm3u_profile_id': m3u_profile_id, - 'position_seconds': estimated_position, # Use estimated position - 'last_known_position': last_known_position, # Include raw position for debugging - 'last_position_update': last_position_update, # Include timestamp for frontend use - 'bytes_sent': int(combined_data.get('bytes_sent', 0)), - # Seek/range information for position calculation and frontend display - 'last_seek_byte': int(combined_data.get('last_seek_byte', 0)), - 'last_seek_percentage': float(combined_data.get('last_seek_percentage', 0.0)), - 'total_content_size': int(combined_data.get('total_content_size', 0)), - 'last_seek_timestamp': float(combined_data.get('last_seek_timestamp', 0.0)) - } + # Group connections by content + content_stats = {} + for conn in connections: + content_key = f"{conn['content_type']}:{conn['content_uuid']}" + if content_key not in content_stats: + content_stats[content_key] = { + 'content_type': conn['content_type'], + 'content_name': conn['content_name'], + 'content_uuid': conn['content_uuid'], + 'content_metadata': conn['content_metadata'], + 'connection_count': 0, + 'connections': [] + } + content_stats[content_key]['connection_count'] += 1 + content_stats[content_key]['connections'].append(conn) - # Calculate connection duration - duration_calculated = False - if connection_info['connected_at']: - try: - connected_time = float(connection_info['connected_at']) - duration = current_time - connected_time - connection_info['duration'] = int(duration) - duration_calculated = True - except: - pass + return JsonResponse({ + 'vod_connections': list(content_stats.values()), + 'total_connections': len(connections), + 'timestamp': current_time + }) - # Fallback: use last_activity if connected_at is not available - if not duration_calculated and connection_info['last_activity']: - try: - last_activity_time = float(connection_info['last_activity']) - # Estimate connection duration using client_id timestamp if available - if connection_info['client_id'].startswith('vod_'): - # Extract timestamp from client_id (format: vod_timestamp_random) - parts = connection_info['client_id'].split('_') - if len(parts) >= 2: - client_start_time = float(parts[1]) / 1000.0 # Convert ms to seconds - duration = current_time - client_start_time - connection_info['duration'] = int(duration) - duration_calculated = True - except: - pass - - # Final fallback - if not duration_calculated: - connection_info['duration'] = 0 - - connections.append(connection_info) - - except Exception as e: - logger.error(f"Error processing connection key {key}: {e}") - - if cursor == 0: - break - - # Group connections by content - content_stats = {} - for conn in connections: - content_key = f"{conn['content_type']}:{conn['content_uuid']}" - if content_key not in content_stats: - content_stats[content_key] = { - 'content_type': conn['content_type'], - 'content_name': conn['content_name'], - 'content_uuid': conn['content_uuid'], - 'content_metadata': conn['content_metadata'], - 'connection_count': 0, - 'connections': [] - } - content_stats[content_key]['connection_count'] += 1 - content_stats[content_key]['connections'].append(conn) - - return JsonResponse({ - 'vod_connections': list(content_stats.values()), - 'total_connections': len(connections), - 'timestamp': current_time - }) - - except Exception as e: - logger.error(f"Error getting VOD stats: {e}") - return JsonResponse({'error': str(e)}, status=500) - - -from rest_framework.decorators import api_view, permission_classes -from apps.accounts.permissions import IsAdmin + except Exception as e: + logger.error(f"Error getting VOD stats: {e}") + return JsonResponse({'error': str(e)}, status=500) @csrf_exempt @@ -1079,4 +996,67 @@ def stop_vod_client(request): logger.error(f"Error stopping VOD client: {e}", exc_info=True) return JsonResponse({'error': str(e)}, status=500) +@api_view(["GET"]) +@permission_classes([AllowAny]) +def stream_xc_movie(request, username, password, stream_id, extension): + if not network_access_allowed(request, "STREAMS"): + return JsonResponse({"error": "Forbidden"}, status=403) + from apps.vod.models import M3UMovieRelation + + session_id = request.GET.get('session_id') + profile_id = request.GET.get('profile_id') + + user = get_object_or_404(User, username=username) + + custom_properties = user.custom_properties or {} + + if "xc_password" not in custom_properties: + return Response({"error": "Invalid credentials"}, status=401) + + if custom_properties["xc_password"] != password: + return Response({"error": "Invalid credentials"}, status=401) + + # All authenticated users get access to VOD from all active M3U accounts + filters = {"movie_id": stream_id, "m3u_account__is_active": True} + + try: + # Order by account priority to get the best relation when multiple exist + movie_relation = M3UMovieRelation.objects.select_related('movie').filter(**filters).order_by('-m3u_account__priority', 'id').first() + if not movie_relation: + return JsonResponse({"error": "Movie not found"}, status=404) + except (M3UMovieRelation.DoesNotExist, M3UMovieRelation.MultipleObjectsReturned): + return JsonResponse({"error": "Movie not found"}, status=404) + + return stream_vod(request._request, 'movie', movie_relation.movie.uuid, session_id, profile_id, user) + +@api_view(["GET"]) +@permission_classes([AllowAny]) +def stream_xc_episode(request, username, password, stream_id, extension): + if not network_access_allowed(request, "STREAMS"): + return JsonResponse({"error": "Forbidden"}, status=403) + + from apps.vod.models import M3UEpisodeRelation + + session_id = request.GET.get('session_id') + profile_id = request.GET.get('profile_id') + + user = get_object_or_404(User, username=username) + + custom_properties = user.custom_properties or {} + + if "xc_password" not in custom_properties: + return Response({"error": "Invalid credentials"}, status=401) + + if custom_properties["xc_password"] != password: + return Response({"error": "Invalid credentials"}, status=401) + + # All authenticated users get access to series/episodes from all active M3U accounts + filters = {"episode_id": stream_id, "m3u_account__is_active": True} + + try: + episode_relation = M3UEpisodeRelation.objects.select_related('episode').filter(**filters).order_by('-m3u_account__priority', 'id').first() + except M3UEpisodeRelation.DoesNotExist: + return JsonResponse({"error": "Episode not found"}, status=404) + + return stream_vod(request._request, 'episode', episode_relation.episode.uuid, session_id, profile_id, user) diff --git a/apps/vod/api_views.py b/apps/vod/api_views.py index 3bd984e6..67813251 100644 --- a/apps/vod/api_views.py +++ b/apps/vod/api_views.py @@ -11,6 +11,7 @@ from django.db.models import Q import django_filters import logging import os +import time import requests from apps.accounts.permissions import ( Authenticated, @@ -36,6 +37,11 @@ from datetime import timedelta logger = logging.getLogger(__name__) +# Negative cache for remote VOD logo URLs that failed to fetch. +# Prevents repeated blocking requests to unreachable hosts. +_vod_logo_fetch_failures = {} +_VOD_LOGO_FAIL_TTL = 300 # seconds + class VODPagination(PageNumberPagination): page_size = 20 # Default page size to match frontend default @@ -578,13 +584,15 @@ class UnifiedContentViewSet(viewsets.ReadOnlyModelViewSet): "series.id IN (SELECT DISTINCT series_id FROM vod_m3useriesrelation msr JOIN m3u_m3uaccount ma ON msr.m3u_account_id = ma.id WHERE ma.is_active = true)" ] - params = [] + movie_params = [] + series_params = [] if search: where_conditions[0] += " AND LOWER(movies.name) LIKE %s" where_conditions[1] += " AND LOWER(series.name) LIKE %s" search_param = f"%{search.lower()}%" - params.extend([search_param, search_param]) + movie_params.append(search_param) + series_params.append(search_param) if category: if '|' in category: @@ -592,15 +600,20 @@ class UnifiedContentViewSet(viewsets.ReadOnlyModelViewSet): if cat_type == 'movie': where_conditions[0] += " AND movies.id IN (SELECT movie_id FROM vod_m3umovierelation mmr JOIN vod_vodcategory c ON mmr.category_id = c.id WHERE c.name = %s)" where_conditions[1] = "1=0" # Exclude series - params.append(cat_name) + movie_params.append(cat_name) + series_params = [] # no params needed for "1=0" elif cat_type == 'series': where_conditions[1] += " AND series.id IN (SELECT series_id FROM vod_m3useriesrelation msr JOIN vod_vodcategory c ON msr.category_id = c.id WHERE c.name = %s)" where_conditions[0] = "1=0" # Exclude movies - params.append(cat_name) + series_params.append(cat_name) + movie_params = [] # no params needed for "1=0" else: where_conditions[0] += " AND movies.id IN (SELECT movie_id FROM vod_m3umovierelation mmr JOIN vod_vodcategory c ON mmr.category_id = c.id WHERE c.name = %s)" where_conditions[1] += " AND series.id IN (SELECT series_id FROM vod_m3useriesrelation msr JOIN vod_vodcategory c ON msr.category_id = c.id WHERE c.name = %s)" - params.extend([category, category]) + movie_params.append(category) + series_params.append(category) + + params = movie_params + series_params # Use UNION ALL with ORDER BY and LIMIT/OFFSET for true unified pagination # This is much more efficient than Python sorting @@ -823,17 +836,62 @@ class VODLogoViewSet(viewsets.ModelViewSet): return HttpResponse(status=500) else: # It's a remote URL - proxy it + # Skip URLs that recently failed to avoid blocking workers + fail_expiry = _vod_logo_fetch_failures.get(logo.url) + if fail_expiry and time.monotonic() < fail_expiry: + return HttpResponse(status=404) + try: - response = requests.get(logo.url, stream=True, timeout=10) - response.raise_for_status() + _LOGO_TOTAL_TIMEOUT = 10 # seconds + _LOGO_MAX_BYTES = 5 * 1024 * 1024 # 5 MB - content_type = response.headers.get('Content-Type', 'image/png') - - return StreamingHttpResponse( - response.iter_content(chunk_size=8192), - content_type=content_type + remote_response = requests.get( + logo.url, + stream=True, + timeout=(3, 5), # (connect_timeout, read_timeout per chunk) ) + + if remote_response.status_code != 200: + now = time.monotonic() + _vod_logo_fetch_failures[logo.url] = now + _VOD_LOGO_FAIL_TTL + return HttpResponse(status=404) + + # Eagerly read the full image with a total time + size cap + # so the greenlet is released quickly. + chunks = [] + total = 0 + deadline = time.monotonic() + _LOGO_TOTAL_TIMEOUT + for chunk in remote_response.iter_content(chunk_size=8192): + total += len(chunk) + if total > _LOGO_MAX_BYTES: + remote_response.close() + return HttpResponse(status=404) + if time.monotonic() > deadline: + remote_response.close() + now = time.monotonic() + _vod_logo_fetch_failures[logo.url] = now + _VOD_LOGO_FAIL_TTL + return HttpResponse(status=404) + chunks.append(chunk) + body = b"".join(chunks) + + # Full read succeeded, clear any previous failure entry + _vod_logo_fetch_failures.pop(logo.url, None) + + content_type = remote_response.headers.get('Content-Type', 'image/png') + + response = HttpResponse(body, content_type=content_type) + response["Content-Length"] = str(len(body)) + if remote_response.headers.get("Cache-Control"): + response["Cache-Control"] = remote_response.headers.get("Cache-Control") + if remote_response.headers.get("Last-Modified"): + response["Last-Modified"] = remote_response.headers.get("Last-Modified") + response["Content-Disposition"] = 'inline; filename="{}"'.format( + os.path.basename(logo.url) + ) + return response except requests.exceptions.RequestException as e: + now = time.monotonic() + _vod_logo_fetch_failures[logo.url] = now + _VOD_LOGO_FAIL_TTL logger.error(f"Error fetching remote VOD logo {logo.url}: {str(e)}") return HttpResponse(status=404) @@ -896,4 +954,3 @@ class VODLogoViewSet(viewsets.ModelViewSet): {"error": str(e)}, status=status.HTTP_500_INTERNAL_SERVER_ERROR ) - diff --git a/core/api_views.py b/core/api_views.py index dd15886c..596da877 100644 --- a/core/api_views.py +++ b/core/api_views.py @@ -3,12 +3,13 @@ import json import ipaddress import logging +from django.conf import settings as django_settings from django.db import models from rest_framework import viewsets, status from rest_framework.response import Response from rest_framework.views import APIView from django.shortcuts import get_object_or_404 -from rest_framework.permissions import IsAuthenticated +from rest_framework.permissions import IsAuthenticated, AllowAny from rest_framework.decorators import api_view, permission_classes, action from drf_spectacular.utils import extend_schema, OpenApiParameter from drf_spectacular.types import OpenApiTypes @@ -34,6 +35,9 @@ import os from core.tasks import rehash_streams from apps.accounts.permissions import ( Authenticated, + IsAdmin, + IsStandardUser, + permission_classes_by_action, ) from dispatcharr.utils import get_client_ip @@ -49,6 +53,12 @@ class UserAgentViewSet(viewsets.ModelViewSet): queryset = UserAgent.objects.all() serializer_class = UserAgentSerializer + def get_permissions(self): + try: + return [perm() for perm in permission_classes_by_action[self.action]] + except KeyError: + return [Authenticated()] + class StreamProfileViewSet(viewsets.ModelViewSet): """ @@ -58,6 +68,12 @@ class StreamProfileViewSet(viewsets.ModelViewSet): queryset = StreamProfile.objects.all() serializer_class = StreamProfileSerializer + def get_permissions(self): + try: + return [perm() for perm in permission_classes_by_action[self.action]] + except KeyError: + return [Authenticated()] + class CoreSettingsViewSet(viewsets.ModelViewSet): """ @@ -68,6 +84,12 @@ class CoreSettingsViewSet(viewsets.ModelViewSet): queryset = CoreSettings.objects.all() serializer_class = CoreSettingsSerializer + def get_permissions(self): + try: + return [perm() for perm in permission_classes_by_action[self.action]] + except KeyError: + return [Authenticated()] + def update(self, request, *args, **kwargs): instance = self.get_object() old_value = instance.value @@ -170,6 +192,11 @@ class ProxySettingsViewSet(viewsets.ViewSet): """ serializer_class = ProxySettingsSerializer + def get_permissions(self): + if self.action in ('list', 'retrieve'): + return [IsStandardUser()] + return [IsAdmin()] + def _get_or_create_settings(self): """Get or create the proxy settings CoreSettings entry""" try: @@ -301,7 +328,9 @@ def environment(request): country_code = None country_name = None - # 4) Get environment mode from system environment variable + # 4) Get environment mode and TLS status from settings + postgres_ssl = getattr(django_settings, "POSTGRES_SSL", False) + return Response( { "authenticated": True, @@ -309,7 +338,17 @@ def environment(request): "local_ip": local_ip, "country_code": country_code, "country_name": country_name, - "env_mode": "dev" if os.getenv("DISPATCHARR_ENV") == "dev" else "prod", + "env_mode": os.getenv("DISPATCHARR_ENV", "aio"), + "redis_tls": { + "enabled": getattr(django_settings, "REDIS_SSL", False), + "verify": getattr(django_settings, "REDIS_SSL_VERIFY", True), + "mtls": bool(getattr(django_settings, "REDIS_SSL_CERT", "") and getattr(django_settings, "REDIS_SSL_KEY", "")), + }, + "postgres_tls": { + "enabled": postgres_ssl, + "ssl_mode": getattr(django_settings, "POSTGRES_SSL_MODE", "verify-full") if postgres_ssl else None, + "mtls": bool(getattr(django_settings, "POSTGRES_SSL_CERT", "") and getattr(django_settings, "POSTGRES_SSL_KEY", "")), + }, } ) @@ -317,8 +356,8 @@ def environment(request): @extend_schema( description="Get application version information", ) - @api_view(["GET"]) +@permission_classes([AllowAny]) def version(request): # Import version information from version import __version__, __timestamp__ diff --git a/core/management/commands/dropdb.py b/core/management/commands/dropdb.py index 1b39a58d..d61b9891 100644 --- a/core/management/commands/dropdb.py +++ b/core/management/commands/dropdb.py @@ -1,5 +1,6 @@ import sys import psycopg2 +from psycopg2 import sql from django.core.management.base import BaseCommand from django.conf import settings from django.db import connection @@ -15,6 +16,13 @@ class Command(BaseCommand): host = db_settings.get('HOST', 'localhost') port = db_settings.get('PORT', 5432) + # Read TLS parameters from Django OPTIONS (populated when POSTGRES_SSL=true) + db_options = db_settings.get('OPTIONS', {}) + ssl_kwargs = {} + for key in ('sslmode', 'sslrootcert', 'sslcert', 'sslkey'): + if key in db_options: + ssl_kwargs[key] = db_options[key] + self.stdout.write(self.style.WARNING( f"WARNING: This will irreversibly drop the entire database '{db_name}'!" )) @@ -30,13 +38,13 @@ class Command(BaseCommand): maintenance_db = 'postgres' try: self.stdout.write("Connecting to maintenance database...") - conn = psycopg2.connect(dbname=maintenance_db, user=user, password=password, host=host, port=port) + conn = psycopg2.connect(dbname=maintenance_db, user=user, password=password, host=host, port=port, **ssl_kwargs) conn.autocommit = True cur = conn.cursor() self.stdout.write(f"Dropping database '{db_name}'...") - cur.execute(f"DROP DATABASE IF EXISTS {db_name};") + cur.execute(sql.SQL("DROP DATABASE IF EXISTS {}").format(sql.Identifier(db_name))) self.stdout.write(f"Creating database '{db_name}'...") - cur.execute(f"CREATE DATABASE {db_name};") + cur.execute(sql.SQL("CREATE DATABASE {}").format(sql.Identifier(db_name))) cur.close() conn.close() self.stdout.write(self.style.SUCCESS(f"Database '{db_name}' has been dropped and recreated.")) diff --git a/core/migrations/022_default_user_limit_settings.py b/core/migrations/022_default_user_limit_settings.py new file mode 100644 index 00000000..4c2b3c33 --- /dev/null +++ b/core/migrations/022_default_user_limit_settings.py @@ -0,0 +1,24 @@ +# Generated by Django 5.1.6 on 2025-03-01 14:01 + +from django.db import migrations +from django.utils.text import slugify + + +def preload_user_limit_settings(apps, schema_editor): + CoreSettings = apps.get_model("core", "CoreSettings") + CoreSettings.objects.create( + key="user_limit_settings", + name="User Limit Settings", + value={}, + ) + + +class Migration(migrations.Migration): + + dependencies = [ + ("core", "0021_systemnotification_notificationdismissal"), + ] + + operations = [ + migrations.RunPython(preload_user_limit_settings), + ] diff --git a/core/models.py b/core/models.py index f1a604f7..5e86e98f 100644 --- a/core/models.py +++ b/core/models.py @@ -156,6 +156,7 @@ PROXY_SETTINGS_KEY = "proxy_settings" NETWORK_ACCESS_KEY = "network_access" SYSTEM_SETTINGS_KEY = "system_settings" EPG_SETTINGS_KEY = "epg_settings" +USER_LIMITS_SETTINGS_KEY = "user_limit_settings" class CoreSettings(models.Model): @@ -362,6 +363,15 @@ class CoreSettings(models.Model): cls._update_group(SYSTEM_SETTINGS_KEY, "System Settings", {"time_zone": value}) return value + @classmethod + def get_user_limits_settings(cls): + return cls._get_group(USER_LIMITS_SETTINGS_KEY, { + "terminate_on_limit_exceeded": True, + "prioritize_single_client_channels": True, + "ignore_same_channel_connections": False, + "terminate_oldest": True, + }) + class SystemEvent(models.Model): """ diff --git a/core/redis_pubsub.py b/core/redis_pubsub.py index 5d0032b0..b1f48b1f 100644 --- a/core/redis_pubsub.py +++ b/core/redis_pubsub.py @@ -201,10 +201,6 @@ class RedisPubSubManager: channel = message.get('channel') if channel: - # Decode binary channel name if needed - if isinstance(channel, bytes): - channel = channel.decode('utf-8') - # Find and call the appropriate handler handler = self.message_handlers.get(channel) if handler: diff --git a/core/tasks.py b/core/tasks.py index a18c2f74..042d63ea 100644 --- a/core/tasks.py +++ b/core/tasks.py @@ -13,7 +13,7 @@ from apps.epg.models import EPGSource from apps.m3u.tasks import refresh_single_m3u_account from apps.epg.tasks import refresh_epg_data from .models import CoreSettings -from apps.channels.models import Stream, ChannelStream +from apps.channels.models import ChannelStream from django.db import transaction logger = logging.getLogger(__name__) @@ -404,7 +404,7 @@ def fetch_channel_stats(): while True: cursor, keys = redis_client.scan(cursor, match=channel_pattern) for key in keys: - channel_id_match = re.search(r"ts_proxy:channel:(.*):metadata", key.decode('utf-8')) + channel_id_match = re.search(r"ts_proxy:channel:(.*):metadata", key) if channel_id_match: ch_id = channel_id_match.group(1) channel_info = ChannelStatus.get_basic_channel_info(ch_id) @@ -753,20 +753,6 @@ def _determine_stream_to_keep(stream_a, stream_b): return (stream_b, stream_a) -@shared_task -def cleanup_vod_persistent_connections(): - """Clean up stale VOD persistent connections""" - try: - from apps.proxy.vod_proxy.connection_manager import VODConnectionManager - - # Clean up connections older than 30 minutes - VODConnectionManager.cleanup_stale_persistent_connections(max_age_seconds=1800) - logger.info("VOD persistent connection cleanup completed") - - except Exception as e: - logger.error(f"Error during VOD persistent connection cleanup: {e}") - - @shared_task def check_for_version_update(): """ diff --git a/core/tests.py b/core/tests.py index 305ab308..b4770f92 100644 --- a/core/tests.py +++ b/core/tests.py @@ -1,3 +1,5 @@ +from unittest.mock import patch, MagicMock + from django.test import TestCase from core.models import CoreSettings, DVR_SETTINGS_KEY, EPG_SETTINGS_KEY @@ -148,3 +150,74 @@ class EpgIgnoreListsTest(TestCase): ]: self._set_epg_field_raw(field, "not a list") self.assertEqual(getter(), []) + + +class DropDBCommandTlsTest(TestCase): + """Verify dropdb management command passes TLS parameters to psycopg2.""" + databases = [] + + _DB_WITH_TLS = { + 'default': { + 'ENGINE': 'django.db.backends.postgresql', + 'NAME': 'testdb', + 'USER': 'testuser', + 'PASSWORD': 'testpass', + 'HOST': 'localhost', + 'PORT': 5432, + 'OPTIONS': { + 'sslmode': 'verify-full', + 'sslrootcert': '/certs/ca.crt', + 'sslcert': '/certs/client.crt', + 'sslkey': '/certs/client.key', + }, + } + } + + _DB_NO_TLS = { + 'default': { + 'ENGINE': 'django.db.backends.postgresql', + 'NAME': 'testdb', + 'USER': 'testuser', + 'PASSWORD': 'testpass', + 'HOST': 'localhost', + 'PORT': 5432, + } + } + + @patch('core.management.commands.dropdb.psycopg2.connect') + @patch('core.management.commands.dropdb.connection') + @patch('builtins.input', return_value='yes') + def test_dropdb_passes_ssl_kwargs_when_tls_enabled(self, _inp, _conn, mock_connect): + mock_pg = MagicMock() + mock_connect.return_value = mock_pg + mock_pg.cursor.return_value = MagicMock() + + with self.settings(DATABASES=self._DB_WITH_TLS): + from django.core.management import call_command + call_command('dropdb') + + mock_connect.assert_called_once_with( + dbname='postgres', user='testuser', password='testpass', + host='localhost', port=5432, + sslmode='verify-full', + sslrootcert='/certs/ca.crt', + sslcert='/certs/client.crt', + sslkey='/certs/client.key', + ) + + @patch('core.management.commands.dropdb.psycopg2.connect') + @patch('core.management.commands.dropdb.connection') + @patch('builtins.input', return_value='yes') + def test_dropdb_no_ssl_kwargs_when_tls_disabled(self, _inp, _conn, mock_connect): + mock_pg = MagicMock() + mock_connect.return_value = mock_pg + mock_pg.cursor.return_value = MagicMock() + + with self.settings(DATABASES=self._DB_NO_TLS): + from django.core.management import call_command + call_command('dropdb') + + mock_connect.assert_called_once_with( + dbname='postgres', user='testuser', password='testpass', + host='localhost', port=5432, + ) diff --git a/core/utils.py b/core/utils.py index ef5aa702..ba2458ec 100644 --- a/core/utils.py +++ b/core/utils.py @@ -3,6 +3,7 @@ import logging import time import os import threading +from pathlib import Path import re from django.conf import settings from redis.exceptions import ConnectionError, TimeoutError @@ -13,6 +14,8 @@ from django.core.validators import URLValidator from django.core.exceptions import ValidationError import gc +_REDIS_TLS_HINT = " (TLS is enabled — verify certificate paths and that Redis is configured for TLS)" + logger = logging.getLogger(__name__) # Import the command detector @@ -43,103 +46,116 @@ def natural_sort_key(text): class RedisClient: _client = None + _buffer = None _pubsub_client = None @classmethod - def get_client(cls, max_retries=5, retry_interval=1): - if cls._client is None: - retry_count = 0 - while retry_count < max_retries: + def _init_client(cls, decode_responses=True, max_retries=5, retry_interval=1): + retry_count = 0 + while retry_count < max_retries: + try: + # Get connection parameters from settings or environment + redis_host = os.environ.get("REDIS_HOST", getattr(settings, 'REDIS_HOST', 'localhost')) + redis_port = int(os.environ.get("REDIS_PORT", getattr(settings, 'REDIS_PORT', 6379))) + redis_db = int(os.environ.get("REDIS_DB", getattr(settings, 'REDIS_DB', 0))) + redis_password = os.environ.get("REDIS_PASSWORD", getattr(settings, 'REDIS_PASSWORD', '')) + redis_user = os.environ.get("REDIS_USER", getattr(settings, 'REDIS_USER', '')) + + # Use standardized settings + socket_timeout = getattr(settings, 'REDIS_SOCKET_TIMEOUT', 5) + socket_connect_timeout = getattr(settings, 'REDIS_SOCKET_CONNECT_TIMEOUT', 5) + health_check_interval = getattr(settings, 'REDIS_HEALTH_CHECK_INTERVAL', 30) + socket_keepalive = getattr(settings, 'REDIS_SOCKET_KEEPALIVE', True) + retry_on_timeout = getattr(settings, 'REDIS_RETRY_ON_TIMEOUT', True) + + # TLS params from settings (empty dict when TLS is disabled) + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) + + # Create Redis client with better defaults + client = redis.Redis( + host=redis_host, + port=redis_port, + db=redis_db, + password=redis_password if redis_password else None, + username=redis_user if redis_user else None, + socket_timeout=socket_timeout, + socket_connect_timeout=socket_connect_timeout, + socket_keepalive=socket_keepalive, + health_check_interval=health_check_interval, + retry_on_timeout=retry_on_timeout, + decode_responses=decode_responses, + **ssl_params + ) + + # Validate connection with ping + client.ping() + + # Disable persistence on first connection - improves performance + # Only try to disable if not in a read-only environment try: - # Get connection parameters from settings or environment - redis_host = os.environ.get("REDIS_HOST", getattr(settings, 'REDIS_HOST', 'localhost')) - redis_port = int(os.environ.get("REDIS_PORT", getattr(settings, 'REDIS_PORT', 6379))) - redis_db = int(os.environ.get("REDIS_DB", getattr(settings, 'REDIS_DB', 0))) - redis_password = os.environ.get("REDIS_PASSWORD", getattr(settings, 'REDIS_PASSWORD', '')) - redis_user = os.environ.get("REDIS_USER", getattr(settings, 'REDIS_USER', '')) + client.config_set('save', '') # Disable RDB snapshots + client.config_set('appendonly', 'no') # Disable AOF logging - # Use standardized settings - socket_timeout = getattr(settings, 'REDIS_SOCKET_TIMEOUT', 5) - socket_connect_timeout = getattr(settings, 'REDIS_SOCKET_CONNECT_TIMEOUT', 5) - health_check_interval = getattr(settings, 'REDIS_HEALTH_CHECK_INTERVAL', 30) - socket_keepalive = getattr(settings, 'REDIS_SOCKET_KEEPALIVE', True) - retry_on_timeout = getattr(settings, 'REDIS_RETRY_ON_TIMEOUT', True) + # Disable protected mode when in debug mode + if os.environ.get('DISPATCHARR_DEBUG', '').lower() == 'true': + client.config_set('protected-mode', 'no') # Disable protected mode in debug + logger.warning("Redis protected mode disabled for debug environment") - # Create Redis client with better defaults - client = redis.Redis( - host=redis_host, - port=redis_port, - db=redis_db, - password=redis_password if redis_password else None, - username=redis_user if redis_user else None, - socket_timeout=socket_timeout, - socket_connect_timeout=socket_connect_timeout, - socket_keepalive=socket_keepalive, - health_check_interval=health_check_interval, - retry_on_timeout=retry_on_timeout - ) - - # Validate connection with ping - client.ping() - - # Disable persistence on first connection - improves performance - # Only try to disable if not in a read-only environment - try: - client.config_set('save', '') # Disable RDB snapshots - client.config_set('appendonly', 'no') # Disable AOF logging - - # Set optimal memory settings with environment variable support - # Get max memory from environment or use a larger default (512MB instead of 256MB) - #max_memory = os.environ.get('REDIS_MAX_MEMORY', '512mb') - #eviction_policy = os.environ.get('REDIS_EVICTION_POLICY', 'allkeys-lru') - - # Apply memory settings - #client.config_set('maxmemory-policy', eviction_policy) - #client.config_set('maxmemory', max_memory) - - #logger.info(f"Redis configured with maxmemory={max_memory}, policy={eviction_policy}") - - # Disable protected mode when in debug mode - if os.environ.get('DISPATCHARR_DEBUG', '').lower() == 'true': - client.config_set('protected-mode', 'no') # Disable protected mode in debug - logger.warning("Redis protected mode disabled for debug environment") - - logger.trace("Redis persistence disabled for better performance") - except redis.exceptions.ResponseError as e: - # Improve error handling for Redis configuration errors - if "OOM" in str(e): - logger.error(f"Redis OOM during configuration: {e}") - # Try to increase maxmemory as an emergency measure - try: - client.config_set('maxmemory', '768mb') - logger.warning("Applied emergency Redis memory increase to 768MB") - except: - pass - else: - logger.error(f"Redis configuration error: {e}") - - logger.info(f"Connected to Redis at {redis_host}:{redis_port}/{redis_db}") - - cls._client = client - break - - except (ConnectionError, TimeoutError) as e: - retry_count += 1 - if retry_count >= max_retries: - logger.error(f"Failed to connect to Redis after {max_retries} attempts: {e}") - return None + logger.trace("Redis persistence disabled for better performance") + except redis.exceptions.ResponseError as e: + # Improve error handling for Redis configuration errors + if "OOM" in str(e): + logger.error(f"Redis OOM during configuration: {e}") + # Try to increase maxmemory as an emergency measure + try: + client.config_set('maxmemory', '768mb') + logger.warning("Applied emergency Redis memory increase to 768MB") + except: + pass else: - # Use exponential backoff for retries - wait_time = retry_interval * (2 ** (retry_count - 1)) - logger.warning(f"Redis connection failed. Retrying in {wait_time}s... ({retry_count}/{max_retries})") - time.sleep(wait_time) + logger.error(f"Redis configuration error: {e}") - except Exception as e: - logger.error(f"Unexpected error connecting to Redis: {e}") + logger.info(f"Connected to Redis at {redis_host}:{redis_port}/{redis_db}") + + return client + + except (ConnectionError, TimeoutError) as e: + retry_count += 1 + _tls_hint = _REDIS_TLS_HINT if ssl_params else "" + if retry_count >= max_retries: + logger.error(f"Failed to connect to Redis after {max_retries} attempts: {e}{_tls_hint}") return None + else: + # Use exponential backoff for retries + wait_time = retry_interval * (2 ** (retry_count - 1)) + logger.warning(f"Redis connection failed. Retrying in {wait_time}s... ({retry_count}/{max_retries})") + time.sleep(wait_time) + except Exception as e: + _tls_hint = "" + try: + _tls_hint = _REDIS_TLS_HINT if ssl_params else "" + except NameError: + pass + logger.error(f"Unexpected error connecting to Redis: {e}{_tls_hint}") + return None + + return None + + @classmethod + def get_client(cls, max_retries=5, retry_interval=1): + """Get Redis client optimized for non-binary data (decoded responses)""" + if cls._client is None: + cls._client = cls._init_client(decode_responses=True, max_retries=max_retries, retry_interval=retry_interval) return cls._client + @classmethod + def get_buffer(cls, max_retries=5, retry_interval=1): + """Get Redis client optimized for binary data (no decoding)""" + if cls._buffer is None: + cls._buffer = cls._init_client(decode_responses=False, max_retries=max_retries, retry_interval=retry_interval) + return cls._buffer + @classmethod def get_pubsub_client(cls, max_retries=5, retry_interval=1): """Get Redis client optimized for PubSub operations""" @@ -161,6 +177,8 @@ class RedisClient: health_check_interval = getattr(settings, 'REDIS_HEALTH_CHECK_INTERVAL', 30) retry_on_timeout = getattr(settings, 'REDIS_RETRY_ON_TIMEOUT', True) + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) + # Create Redis client with PubSub-optimized settings - no timeout client = redis.Redis( host=redis_host, @@ -172,7 +190,9 @@ class RedisClient: socket_connect_timeout=socket_connect_timeout, socket_keepalive=socket_keepalive, health_check_interval=health_check_interval, - retry_on_timeout=retry_on_timeout + retry_on_timeout=retry_on_timeout, + decode_responses=True, + **ssl_params ) # Validate connection with ping @@ -185,8 +205,9 @@ class RedisClient: except (ConnectionError, TimeoutError) as e: retry_count += 1 + _tls_hint = _REDIS_TLS_HINT if ssl_params else "" if retry_count >= max_retries: - logger.error(f"Failed to connect to Redis for PubSub after {max_retries} attempts: {e}") + logger.error(f"Failed to connect to Redis for PubSub after {max_retries} attempts: {e}{_tls_hint}") return None else: # Use exponential backoff for retries @@ -195,7 +216,8 @@ class RedisClient: time.sleep(wait_time) except Exception as e: - logger.error(f"Unexpected error connecting to Redis for PubSub: {e}") + _tls_hint = _REDIS_TLS_HINT if ssl_params else "" + logger.error(f"Unexpected error connecting to Redis for PubSub: {e}{_tls_hint}") return None return cls._pubsub_client @@ -410,6 +432,20 @@ def cleanup_memory(log_usage=False, force_collection=True): pass logger.trace("Memory cleanup complete for django") +def safe_upload_path(filename: str, base_dir) -> str: + """Return a safe absolute path for an uploaded file within base_dir. + + Strips all directory components from *filename* and verifies the resolved + path stays inside *base_dir*. Raises ValueError on path traversal attempts. + """ + safe_name = Path(filename).name + base = Path(base_dir).resolve() + file_path = (base / safe_name).resolve() + if not file_path.is_relative_to(base): + raise ValueError("Invalid filename.") + return str(file_path) + + def is_protected_path(file_path): """ Determine if a file path is in a protected directory that shouldn't be deleted. diff --git a/core/views.py b/core/views.py index 082cccc3..7321367c 100644 --- a/core/views.py +++ b/core/views.py @@ -4,7 +4,7 @@ from shlex import split as shlex_split import sys import subprocess import logging -import re +import regex import redis from django.conf import settings @@ -42,12 +42,14 @@ def stream_view(request, channel_uuid): redis_db = int(getattr(settings, "REDIS_DB", "0")) redis_password = getattr(settings, "REDIS_PASSWORD", "") redis_user = getattr(settings, "REDIS_USER", "") + ssl_params = getattr(settings, "REDIS_SSL_PARAMS", {}) redis_client = redis.Redis( host=redis_host, port=redis_port, db=redis_db, password=redis_password if redis_password else None, - username=redis_user if redis_user else None + username=redis_user if redis_user else None, + **ssl_params ) # Retrieve the channel by the provided stream_id. @@ -130,10 +132,13 @@ def stream_view(request, channel_uuid): # Prepare the pattern replacement. logger.debug("Executing the following pattern replacement:") logger.debug(f" search: {active_profile.search_pattern}") - safe_replace_pattern = re.sub(r'\$(\d+)', r'\\\1', active_profile.replace_pattern) + # Convert JS-style backreferences in replace: $ -> \g, $1 -> \1 + safe_replace_pattern = regex.sub(r'\$<([^>]+)>', r'\\g<\1>', active_profile.replace_pattern) + safe_replace_pattern = regex.sub(r'\$(\d+)', r'\\\1', safe_replace_pattern) logger.debug(f" replace: {active_profile.replace_pattern}") logger.debug(f" safe replace: {safe_replace_pattern}") - stream_url = re.sub(active_profile.search_pattern, safe_replace_pattern, input_url) + # regex module accepts JS-style (?...) named groups natively + stream_url = regex.sub(active_profile.search_pattern, safe_replace_pattern, input_url) logger.debug(f"Generated stream url: {stream_url}") # Get the stream profile set on the channel. diff --git a/core/xtream_codes.py b/core/xtream_codes.py index 9b56197a..822b37fd 100644 --- a/core/xtream_codes.py +++ b/core/xtream_codes.py @@ -62,7 +62,7 @@ class Client: url = f"{self.server_url}/{endpoint}" logger.debug(f"XC API Request: {url} with params: {params}") - response = self.session.get(url, params=params, timeout=30) + response = self.session.get(url, params=params, timeout=60) response.raise_for_status() # Check if response is empty diff --git a/dispatcharr/consumers.py b/dispatcharr/consumers.py index 4e21bdae..7b62d957 100644 --- a/dispatcharr/consumers.py +++ b/dispatcharr/consumers.py @@ -1,6 +1,6 @@ import json from channels.generic.websocket import AsyncWebsocketConsumer -import re, logging +import regex, logging logger = logging.getLogger(__name__) @@ -54,9 +54,9 @@ class MyWebSocketConsumer(AsyncWebsocketConsumer): # Apply the transformation using the replace_with_mark function try: - search_preview = re.sub(data["search"], replace_with_mark, data["url"]) + search_preview = regex.sub(data["search"], replace_with_mark, data["url"]) except Exception as e: - search_preview = data["search"] + search_preview = data["url"] logger.error(f"Failed to generate replace preview: {e}") result = transform_url(data["url"], data["search"], data["replace"]) diff --git a/dispatcharr/persistent_lock.py b/dispatcharr/persistent_lock.py index 2ed72bf4..45625857 100644 --- a/dispatcharr/persistent_lock.py +++ b/dispatcharr/persistent_lock.py @@ -48,7 +48,7 @@ class PersistentLock: Returns True if the expiration was successfully extended. """ current_value = self.redis_client.get(self.lock_key) - if current_value and current_value.decode("utf-8") == self.lock_token: + if current_value and current_value == self.lock_token: self.redis_client.expire(self.lock_key, self.lock_timeout) self.has_lock = False return True @@ -74,18 +74,40 @@ class PersistentLock: # Example usage (for testing purposes only): if __name__ == "__main__": import os + import sys # Connect to Redis using environment variables; adjust connection parameters as needed. redis_host = os.environ.get("REDIS_HOST", "localhost") redis_port = int(os.environ.get("REDIS_PORT", 6379)) redis_db = int(os.environ.get("REDIS_DB", 0)) redis_password = os.environ.get("REDIS_PASSWORD", "") redis_user = os.environ.get("REDIS_USER", "") + ssl_kwargs = {} + if os.environ.get("REDIS_SSL", "false").lower() == "true": + import ssl as _ssl + ssl_kwargs["ssl"] = True + ssl_kwargs["ssl_cert_reqs"] = ( + _ssl.CERT_REQUIRED if os.environ.get("REDIS_SSL_VERIFY", "true").lower() == "true" + else _ssl.CERT_NONE + ) + for env_var, key in [ + ("REDIS_SSL_CA_CERT", "ssl_ca_certs"), + ("REDIS_SSL_CERT", "ssl_certfile"), + ("REDIS_SSL_KEY", "ssl_keyfile"), + ]: + path = os.environ.get(env_var, "") + if path: + if not os.path.isfile(path): + print(f"Redis TLS: {env_var}={path!r} — file not found.") + sys.exit(1) + ssl_kwargs[key] = path + client = redis.Redis( host=redis_host, port=redis_port, db=redis_db, password=redis_password if redis_password else None, - username=redis_user if redis_user else None + username=redis_user if redis_user else None, + **ssl_kwargs ) lock = PersistentLock(client, "lock:example_account", lock_timeout=120) diff --git a/dispatcharr/settings.py b/dispatcharr/settings.py index b0d387b2..b2b1909f 100644 --- a/dispatcharr/settings.py +++ b/dispatcharr/settings.py @@ -1,7 +1,23 @@ import os +import ssl from pathlib import Path from datetime import timedelta from urllib.parse import quote_plus +from django.core.exceptions import ImproperlyConfigured + + +def _validate_tls_cert_paths(paths, service_name): + """Validate that configured TLS certificate file paths exist on disk. + + Raises ImproperlyConfigured with a clear message identifying the + service and missing file so operators can fix their environment. + """ + for env_var, file_path in paths: + if file_path and not Path(file_path).is_file(): + raise ImproperlyConfigured( + f"{service_name} TLS: {env_var}={file_path!r} — file not found. " + f"Check that the certificate file exists and the volume is mounted correctly." + ) BASE_DIR = Path(__file__).resolve().parent.parent @@ -12,6 +28,37 @@ REDIS_DB = os.environ.get("REDIS_DB", "0") REDIS_USER = os.environ.get("REDIS_USER", "") REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD", "") +# Redis TLS configuration +REDIS_SSL = os.environ.get("REDIS_SSL", "false").lower() == "true" +REDIS_SSL_VERIFY = os.environ.get("REDIS_SSL_VERIFY", "true").lower() == "true" +REDIS_SSL_CA_CERT = os.environ.get("REDIS_SSL_CA_CERT", "") +REDIS_SSL_CERT = os.environ.get("REDIS_SSL_CERT", "") +REDIS_SSL_KEY = os.environ.get("REDIS_SSL_KEY", "") + +# Reusable dict of SSL kwargs for redis.Redis() constructors +REDIS_SSL_PARAMS = {} +if REDIS_SSL: + _validate_tls_cert_paths([ + ("REDIS_SSL_CA_CERT", REDIS_SSL_CA_CERT), + ("REDIS_SSL_CERT", REDIS_SSL_CERT), + ("REDIS_SSL_KEY", REDIS_SSL_KEY), + ], "Redis") + + REDIS_SSL_PARAMS["ssl"] = True + REDIS_SSL_PARAMS["ssl_cert_reqs"] = ssl.CERT_REQUIRED if REDIS_SSL_VERIFY else ssl.CERT_NONE + if REDIS_SSL_CA_CERT: + REDIS_SSL_PARAMS["ssl_ca_certs"] = REDIS_SSL_CA_CERT + if REDIS_SSL_CERT: + REDIS_SSL_PARAMS["ssl_certfile"] = REDIS_SSL_CERT + if REDIS_SSL_KEY: + REDIS_SSL_PARAMS["ssl_keyfile"] = REDIS_SSL_KEY + + _mtls = "enabled" if REDIS_SSL_CERT and REDIS_SSL_KEY else "disabled" + _verify = "on" if REDIS_SSL_VERIFY else "off" + print(f"Redis TLS: enabled (verify={_verify}, mTLS={_mtls})") +else: + print("Redis TLS: disabled") + # Set DEBUG to True for development, False for production if os.environ.get("DISPATCHARR_DEBUG", "False").lower() == "true": DEBUG = True @@ -120,20 +167,46 @@ TEMPLATES = [ WSGI_APPLICATION = "dispatcharr.wsgi.application" ASGI_APPLICATION = "dispatcharr.asgi.application" +_redis_scheme = "rediss" if REDIS_SSL else "redis" + +# URL-encoded auth string shared by CHANNEL_LAYERS and Celery broker URLs +if REDIS_PASSWORD: + _encoded_password = quote_plus(REDIS_PASSWORD) + if REDIS_USER: + _redis_auth = f"{quote_plus(REDIS_USER)}:{_encoded_password}@" + else: + _redis_auth = f":{_encoded_password}@" +else: + _redis_auth = "" + +_channels_redis_url = f"{_redis_scheme}://{_redis_auth}{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" +# channels_redis accepts either a URL string or a dict with "address" + kwargs. +# When TLS is enabled, pass SSL params alongside the URL so the connection pool +# uses the correct CA cert and verification settings. +if REDIS_SSL: + # Filter out "ssl" key — the rediss:// scheme already enables SSL. + # Passing ssl=True as a kwarg to aioredis from_url causes an error. + _channels_ssl = {k: v for k, v in REDIS_SSL_PARAMS.items() if k != "ssl"} + _channels_host = {"address": _channels_redis_url, **_channels_ssl} +else: + _channels_host = _channels_redis_url + CHANNEL_LAYERS = { "default": { "BACKEND": "channels_redis.core.RedisChannelLayer", "CONFIG": { - "hosts": ["redis://{redis_auth}{host}:{port}/{db}".format( - redis_auth=f"{quote_plus(REDIS_USER)}:{quote_plus(REDIS_PASSWORD)}@" if REDIS_PASSWORD and REDIS_USER else f":{quote_plus(REDIS_PASSWORD)}@" if REDIS_PASSWORD else "", - host=REDIS_HOST, - port=REDIS_PORT, - db=REDIS_DB - )], # URL format supports authentication + "hosts": [_channels_host], }, }, } +# PostgreSQL TLS configuration (defined before DATABASES for module-level access) +POSTGRES_SSL = os.environ.get("POSTGRES_SSL", "false").lower() == "true" +POSTGRES_SSL_MODE = os.environ.get("POSTGRES_SSL_MODE", "verify-full") +POSTGRES_SSL_CA_CERT = os.environ.get("POSTGRES_SSL_CA_CERT", "") +POSTGRES_SSL_CERT = os.environ.get("POSTGRES_SSL_CERT", "") +POSTGRES_SSL_KEY = os.environ.get("POSTGRES_SSL_KEY", "") + if os.getenv("DB_ENGINE", None) == "sqlite": DATABASES = { "default": { @@ -154,6 +227,28 @@ else: } } + if POSTGRES_SSL: + _validate_tls_cert_paths([ + ("POSTGRES_SSL_CA_CERT", POSTGRES_SSL_CA_CERT), + ("POSTGRES_SSL_CERT", POSTGRES_SSL_CERT), + ("POSTGRES_SSL_KEY", POSTGRES_SSL_KEY), + ], "PostgreSQL") + + DATABASES["default"]["OPTIONS"] = { + "sslmode": POSTGRES_SSL_MODE, + } + if POSTGRES_SSL_CA_CERT: + DATABASES["default"]["OPTIONS"]["sslrootcert"] = POSTGRES_SSL_CA_CERT + if POSTGRES_SSL_CERT: + DATABASES["default"]["OPTIONS"]["sslcert"] = POSTGRES_SSL_CERT + if POSTGRES_SSL_KEY: + DATABASES["default"]["OPTIONS"]["sslkey"] = POSTGRES_SSL_KEY + + _mtls = "enabled" if POSTGRES_SSL_CERT and POSTGRES_SSL_KEY else "disabled" + print(f"PostgreSQL TLS: enabled (sslmode={POSTGRES_SSL_MODE}, mTLS={_mtls})") + else: + print("PostgreSQL TLS: disabled") + AUTH_PASSWORD_VALIDATORS = [ { "NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator", @@ -170,7 +265,14 @@ REST_FRAMEWORK = { "rest_framework_simplejwt.authentication.JWTAuthentication", "apps.accounts.authentication.ApiKeyAuthentication", ], + "DEFAULT_PERMISSION_CLASSES": [ + "apps.accounts.permissions.IsAdmin", + ], "DEFAULT_FILTER_BACKENDS": ["django_filters.rest_framework.DjangoFilterBackend"], + "DEFAULT_THROTTLE_CLASSES": [], + "DEFAULT_THROTTLE_RATES": { + "login": "3/minute", + }, } SPECTACULAR_SETTINGS = { @@ -178,17 +280,6 @@ SPECTACULAR_SETTINGS = { "DESCRIPTION": "API documentation for Dispatcharr", "VERSION": "1.0.0", "SERVE_INCLUDE_SCHEMA": False, - "SECURITY": [{"BearerAuth": []}], - "COMPONENTS": { - "securitySchemes": { - "BearerAuth": { - "type": "http", - "scheme": "bearer", - "bearerFormat": "JWT", - "description": "Enter your JWT access token. The 'Bearer ' prefix is added automatically.", - } - } - }, } LANGUAGE_CODE = "en-us" @@ -208,22 +299,52 @@ STATICFILES_DIRS = [ DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField" AUTH_USER_MODEL = "accounts.User" -# Build default Redis URL from components for Celery with optional authentication -# Build auth string conditionally with URL encoding for special characters -if REDIS_PASSWORD: - encoded_password = quote_plus(REDIS_PASSWORD) - if REDIS_USER: - encoded_user = quote_plus(REDIS_USER) - redis_auth = f"{encoded_user}:{encoded_password}@" - else: - redis_auth = f":{encoded_password}@" +_default_redis_url = f"{_redis_scheme}://{_redis_auth}{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" +# Celery/Kombu require SSL parameters in the URL query string because +# internal URL parsing can overwrite the CELERY_BROKER_USE_SSL dict. +if REDIS_SSL: + _celery_ssl_params = [ + f"ssl_cert_reqs={'CERT_REQUIRED' if REDIS_SSL_VERIFY else 'CERT_NONE'}", + ] + if REDIS_SSL_CA_CERT: + _celery_ssl_params.append(f"ssl_ca_certs={REDIS_SSL_CA_CERT}") + if REDIS_SSL_CERT: + _celery_ssl_params.append(f"ssl_certfile={REDIS_SSL_CERT}") + if REDIS_SSL_KEY: + _celery_ssl_params.append(f"ssl_keyfile={REDIS_SSL_KEY}") + _default_celery_url = f"{_default_redis_url}?{'&'.join(_celery_ssl_params)}" else: - redis_auth = "" - -_default_redis_url = f"redis://{redis_auth}{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" -CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL", _default_redis_url) + _default_celery_url = _default_redis_url +CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL", _default_celery_url) CELERY_RESULT_BACKEND = os.environ.get("CELERY_RESULT_BACKEND", CELERY_BROKER_URL) +# Validate that URL overrides don't conflict with TLS settings +for _url_var, _url_val in [ + ("CELERY_BROKER_URL", CELERY_BROKER_URL), + ("CELERY_RESULT_BACKEND", CELERY_RESULT_BACKEND), +]: + _is_override = os.environ.get(_url_var) is not None + if not _is_override: + continue + _url_is_ssl = _url_val.startswith("rediss://") + if REDIS_SSL and not _url_is_ssl: + raise ImproperlyConfigured( + f"REDIS_SSL is enabled but {_url_var} uses redis:// (plaintext). " + f"Change the URL scheme to rediss:// or remove the {_url_var} override." + ) + if not REDIS_SSL and _url_is_ssl: + raise ImproperlyConfigured( + f"{_url_var} uses rediss:// (TLS) but REDIS_SSL is not enabled. " + f"Set REDIS_SSL=true and configure the TLS certificate settings." + ) + +# Celery TLS configuration — required in addition to the rediss:// URL scheme. +# Uses the same cert params as REDIS_SSL_PARAMS, minus the "ssl" key that +# redis-py needs but Celery/Kombu does not. +if REDIS_SSL: + CELERY_BROKER_USE_SSL = {k: v for k, v in REDIS_SSL_PARAMS.items() if k != "ssl"} + CELERY_RESULT_BACKEND_USE_SSL = CELERY_BROKER_USE_SSL + # Configure Redis key prefix CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = { "global_keyprefix": "celery-tasks:", # Set the Redis key prefix for Celery @@ -288,7 +409,6 @@ BACKUP_DATA_DIRS = [ SERVER_IP = "127.0.0.1" CORS_ALLOW_ALL_ORIGINS = True -CORS_ALLOW_CREDENTIALS = True CSRF_TRUSTED_ORIGINS = ["http://*", "https://*"] APPEND_SLASH = True @@ -299,8 +419,19 @@ SIMPLE_JWT = { "BLACKLIST_AFTER_ROTATION": True, # Optional: Whether to blacklist refresh tokens } -# Redis connection settings +# Redis connection settings — _default_redis_url uses rediss:// when REDIS_SSL is enabled REDIS_URL = os.environ.get("REDIS_URL", _default_redis_url) +if os.environ.get("REDIS_URL") is not None: + if REDIS_SSL and not REDIS_URL.startswith("rediss://"): + raise ImproperlyConfigured( + "REDIS_SSL is enabled but REDIS_URL uses redis:// (plaintext). " + "Change the URL scheme to rediss:// or remove the REDIS_URL override." + ) + if not REDIS_SSL and REDIS_URL.startswith("rediss://"): + raise ImproperlyConfigured( + "REDIS_URL uses rediss:// (TLS) but REDIS_SSL is not enabled. " + "Set REDIS_SSL=true and configure the TLS certificate settings." + ) REDIS_SOCKET_TIMEOUT = 60 # Socket timeout in seconds REDIS_SOCKET_CONNECT_TIMEOUT = 5 # Connection timeout in seconds REDIS_HEALTH_CHECK_INTERVAL = 15 # Health check every 15 seconds diff --git a/dispatcharr/urls.py b/dispatcharr/urls.py index 092fb0cb..1319f31e 100644 --- a/dispatcharr/urls.py +++ b/dispatcharr/urls.py @@ -6,7 +6,7 @@ from django.views.generic import TemplateView, RedirectView from .routing import websocket_urlpatterns from apps.output.views import xc_player_api, xc_panel_api, xc_get, xc_xmltv from apps.proxy.ts_proxy.views import stream_xc -from apps.output.views import xc_movie_stream, xc_series_stream +from apps.proxy.vod_proxy.views import stream_xc_movie, stream_xc_episode urlpatterns = [ # API Routes @@ -44,13 +44,13 @@ urlpatterns = [ # XC VOD endpoints path( "movie///.", - xc_movie_stream, - name="xc_movie_stream", + stream_xc_movie, + name="stream_xc_movie", ), path( "series///.", - xc_series_stream, - name="xc_series_stream", + stream_xc_episode, + name="stream_xc_episode", ), # Admin path("admin", RedirectView.as_view(url="/admin/", permanent=True)), diff --git a/docker/Dockerfile b/docker/Dockerfile index 3e30a825..596c4025 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -12,7 +12,7 @@ COPY ./frontend /app/frontend # remove any node_modules that may have been copied from the host (x86) RUN rm -rf node_modules || true; \ npm install --no-audit --progress=false; -RUN npm run build; \ +RUN npm run build && \ rm -rf node_modules .cache # --- Redeclare build arguments for the next stage --- @@ -32,9 +32,9 @@ COPY . /app COPY ./docker/nginx.conf /etc/nginx/sites-enabled/default # Fix line endings and make entrypoint scripts executable RUN for f in /app/docker/entrypoint*.sh; do \ - if [ -f "$f" ]; then \ - sed -i 's/\r$//' "$f" && chmod +x "$f"; \ - fi; \ + if [ -f "$f" ]; then \ + sed -i 's/\r$//' "$f" && chmod +x "$f"; \ + fi; \ done # Clean out existing frontend folder RUN rm -rf /app/frontend diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index d674460a..547ac249 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -14,6 +14,7 @@ services: - 9191:9191 volumes: - ./data:/data + #- ./certs:/certs:ro # TLS certificates (optional) depends_on: db: condition: service_healthy @@ -33,15 +34,26 @@ services: - POSTGRES_DB=dispatcharr - POSTGRES_USER=dispatch - POSTGRES_PASSWORD=secret + # PostgreSQL TLS (optional) — mount certs via the volume above + #- POSTGRES_SSL=true # required to enable TLS + #- POSTGRES_SSL_MODE=verify-full # optional: verify-full (default) | verify-ca | require + #- POSTGRES_SSL_CA_CERT=/certs/postgres/ca.crt # optional: CA cert to verify the server + #- POSTGRES_SSL_CERT=/certs/postgres/client.crt # optional: client cert (only if server requires client auth) + #- POSTGRES_SSL_KEY=/certs/postgres/client.key # optional: client key (only if server requires client auth) # Redis Connection - REDIS_HOST=redis - REDIS_PORT=6379 - # Redis Authentication (Optional) # Uncomment and set if your Redis requires authentication: #- REDIS_PASSWORD=your_strong_redis_password #- REDIS_USER=your_redis_username # For Redis 6+ ACL - see Redis service below + # Redis TLS (optional) — mount certs via the volume above + #- REDIS_SSL=true # required to enable TLS + #- REDIS_SSL_VERIFY=true # optional: set false for self-signed certs without a CA + #- REDIS_SSL_CA_CERT=/certs/redis/ca.crt # optional: CA cert to verify the server + #- REDIS_SSL_CERT=/certs/redis/client.crt # optional: client cert (only if server requires client auth) + #- REDIS_SSL_KEY=/certs/redis/client.key # optional: client key (only if server requires client auth) # Logging - DISPATCHARR_LOG_LEVEL=info @@ -95,6 +107,7 @@ services: condition: service_started volumes: - ./data:/data + #- ./certs:/certs:ro # TLS certificates (optional) extra_hosts: - "host.docker.internal:host-gateway" entrypoint: ["/app/docker/entrypoint.celery.sh"] @@ -108,21 +121,30 @@ services: # Must match the web service port for DVR recording and internal API calls - DISPATCHARR_PORT=9191 - # PostgreSQL Connection + # PostgreSQL — must match web service settings - POSTGRES_HOST=db - POSTGRES_PORT=5432 - POSTGRES_DB=dispatcharr - POSTGRES_USER=dispatch - POSTGRES_PASSWORD=secret + # PostgreSQL TLS — must match web service + #- POSTGRES_SSL=true + #- POSTGRES_SSL_MODE=verify-full + #- POSTGRES_SSL_CA_CERT=/certs/postgres/ca.crt + #- POSTGRES_SSL_CERT=/certs/postgres/client.crt + #- POSTGRES_SSL_KEY=/certs/postgres/client.key - # Redis Connection + # Redis — must match web service settings - REDIS_HOST=redis - REDIS_PORT=6379 - - # Redis Authentication (Optional) - # Uncomment and set if your Redis requires authentication: #- REDIS_PASSWORD=your_strong_redis_password - #- REDIS_USER=your_redis_username # For Redis 6+ ACL - see Redis service below + #- REDIS_USER=your_redis_username + # Redis TLS — must match web service + #- REDIS_SSL=true + #- REDIS_SSL_VERIFY=true + #- REDIS_SSL_CA_CERT=/certs/redis/ca.crt + #- REDIS_SSL_CERT=/certs/redis/client.crt + #- REDIS_SSL_KEY=/certs/redis/client.key # Logging - DISPATCHARR_LOG_LEVEL=info diff --git a/docker/entrypoint.celery.sh b/docker/entrypoint.celery.sh index 7a69f430..b38b6ac3 100644 --- a/docker/entrypoint.celery.sh +++ b/docker/entrypoint.celery.sh @@ -36,6 +36,10 @@ if [ "$USE_LEGACY_NUMPY" = "true" ]; then fi fi +# Fix TLS client key permissions/ownership for PostgreSQL. +FIXED_KEY_PATH="/data/.pg-client-celery.key" +. /app/docker/init/00-fix-pg-ssl-key.sh + # Wait for migrations to complete # Uses 'migrate --check' which exits 0 only when all migrations are applied, # and exits 1 on unapplied migrations OR connection errors (safe either way) diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh index 0de0afd5..e5433650 100755 --- a/docker/entrypoint.sh +++ b/docker/entrypoint.sh @@ -2,9 +2,27 @@ set -e # Exit immediately if a command exits with a non-zero status +# Guard flag to prevent cleanup running twice (trap + explicit call) +_cleanup_done=false + # Function to clean up only running processes cleanup() { + if $_cleanup_done; then return; fi + _cleanup_done=true + set +e # Disable exit-on-error so cleanup always runs fully echo "🔥 Cleanup triggered! Stopping services..." + + # Explicitly stop uwsgi workers - children of 'su' wrapper, not tracked in pids[] + echo "⛔ Stopping uwsgi workers..." + pkill -TERM -f uwsgi 2>/dev/null || true + + # Stop celery, daphne, redis - also not tracked in pids[] + echo "⛔ Stopping celery, daphne, redis..." + pkill -TERM -f "celery" 2>/dev/null || true + pkill -TERM -f "daphne" 2>/dev/null || true + pkill -TERM -f "redis-server" 2>/dev/null || true + + # Stop tracked processes (postgres, nginx, su/uwsgi wrapper) for pid in "${pids[@]}"; do if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then echo "⛔ Stopping process (PID: $pid)..." @@ -13,14 +31,38 @@ cleanup() { echo "✅ Process (PID: $pid) already stopped." fi done + + # Wait up to 8 s for graceful shutdown, exit early once all are gone + # (leaves headroom within Docker's default 10 s stop_grace_period) + _shutdown_timeout=8 + _shutdown_elapsed=0 + while [ "$_shutdown_elapsed" -lt "$_shutdown_timeout" ]; do + pgrep -f "uwsgi|celery|daphne|redis-server|postgres" >/dev/null 2>&1 || break + sleep 1 + _shutdown_elapsed=$((_shutdown_elapsed + 1)) + done + + # Force kill anything still lingering + pkill -KILL -f uwsgi 2>/dev/null || true + pkill -KILL -f "celery" 2>/dev/null || true + pkill -KILL -f "daphne" 2>/dev/null || true + pkill -KILL -f "redis-server" 2>/dev/null || true + # Use pg_ctl immediate stop rather than SIGKILL. Avoids data corruption + # while still forcing a fast exit (crash recovery runs on next startup) + if pgrep -f "postgres" >/dev/null 2>&1; then + su - "$POSTGRES_USER" -c "$PG_BINDIR/pg_ctl -D ${POSTGRES_DIR} stop -m immediate" 2>/dev/null || true + fi + wait + echo "✅ All processes stopped cleanly." } # Catch termination signals (CTRL+C, Docker Stop, etc.) trap cleanup TERM INT -# Initialize an array to store PIDs +# Initialize an array to store PIDs and a map of PID->name pids=() +declare -A pid_names # Function to echo with timestamp echo_with_timestamp() { @@ -30,8 +72,23 @@ echo_with_timestamp() { # Set PostgreSQL environment variables export POSTGRES_DB=${POSTGRES_DB:-dispatcharr} export POSTGRES_USER=${POSTGRES_USER:-dispatch} -export POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-secret} -export POSTGRES_HOST=${POSTGRES_HOST:-localhost} +# AIO mode: default to 'secret' for internal DB. +# Modular mode + TLS: no default — cert-only auth (mTLS) uses no password. +# Modular mode + no TLS: preserve 'secret' default for backward compatibility. +if [[ "${DISPATCHARR_ENV:-}" == "modular" && "${POSTGRES_SSL:-}" == "true" ]]; then + export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-}" +else + export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-secret}" +fi +export DISPATCHARR_ENV=${DISPATCHARR_ENV:-aio} +if [[ "$DISPATCHARR_ENV" == "aio" ]]; then + # Use Unix socket for loopback values (unset, localhost, 127.0.0.1) + if [[ -z "$POSTGRES_HOST" || "$POSTGRES_HOST" == "localhost" || "$POSTGRES_HOST" == "127.0.0.1" ]]; then + export POSTGRES_HOST=/var/run/postgresql + fi +else + export POSTGRES_HOST=${POSTGRES_HOST:-localhost} +fi export POSTGRES_PORT=${POSTGRES_PORT:-5432} export PG_VERSION=$(ls /usr/lib/postgresql/ | sort -V | tail -n 1) export PG_BINDIR="/usr/lib/postgresql/${PG_VERSION}/bin" @@ -96,6 +153,20 @@ echo "Environment DISPATCHARR_LOG_LEVEL set to: '${DISPATCHARR_LOG_LEVEL}'" # Also make the log level available in /etc/environment for all login shells #grep -q "DISPATCHARR_LOG_LEVEL" /etc/environment || echo "DISPATCHARR_LOG_LEVEL=${DISPATCHARR_LOG_LEVEL}" >> /etc/environment +# Translate Dispatcharr POSTGRES_SSL_* env vars into libpq-recognized PGSSL* +# env vars. Called once before any external PostgreSQL connection; all child +# processes (psql, pg_dump, pg_isready, createdb, dropdb) inherit these +# automatically. No-op when POSTGRES_SSL is not "true". +setup_pg_ssl_env() { + if [ "${POSTGRES_SSL:-false}" != "true" ]; then + return 0 + fi + export PGSSLMODE="${POSTGRES_SSL_MODE:-verify-full}" + if [ -n "${POSTGRES_SSL_CA_CERT:-}" ]; then export PGSSLROOTCERT="$POSTGRES_SSL_CA_CERT"; fi + if [ -n "${POSTGRES_SSL_CERT:-}" ]; then export PGSSLCERT="$POSTGRES_SSL_CERT"; fi + if [ -n "${POSTGRES_SSL_KEY:-}" ]; then export PGSSLKEY="$POSTGRES_SSL_KEY"; fi +} + # READ-ONLY - don't let users change these export POSTGRES_DIR=/data/db @@ -112,6 +183,14 @@ variables=( CELERY_NICE_LEVEL UWSGI_NICE_LEVEL DJANGO_SECRET_KEY ) +# TLS variables are optional — only propagate when set to avoid noisy warnings +for _tls_var in POSTGRES_SSL POSTGRES_SSL_MODE POSTGRES_SSL_CA_CERT POSTGRES_SSL_CERT POSTGRES_SSL_KEY \ + REDIS_SSL REDIS_SSL_VERIFY REDIS_SSL_CA_CERT REDIS_SSL_CERT REDIS_SSL_KEY; do + if [ -n "${!_tls_var+x}" ]; then + variables+=("$_tls_var") + fi +done + # Truncate files before rewriting > /etc/profile.d/dispatcharr.sh @@ -146,6 +225,22 @@ fi echo "Starting user setup..." . /app/docker/init/01-user-setup.sh +# Fix TLS client key permissions/ownership BEFORE any external PG connections. +# Must run after 01-user-setup.sh (user exists for chown) and before +# 02-postgres.sh / pg_isready (which make the first external PG connections). +FIXED_KEY_PATH="/data/.pg-client.key" +. /app/docker/init/00-fix-pg-ssl-key.sh +# Propagate the fixed path to login shells (su - strips env vars) +if [ "${POSTGRES_SSL_KEY:-}" = "$FIXED_KEY_PATH" ]; then + sed -i "/^POSTGRES_SSL_KEY=/d" /etc/environment + echo "POSTGRES_SSL_KEY='$FIXED_KEY_PATH'" >> /etc/environment + sed -i "s|export POSTGRES_SSL_KEY=.*|export POSTGRES_SSL_KEY='$FIXED_KEY_PATH'|" /etc/profile.d/dispatcharr.sh +fi + +# Export libpq TLS env vars so all subsequent psql/pg_dump/pg_isready calls +# (in 02-postgres.sh, modular-mode checks, etc.) use TLS automatically. +setup_pg_ssl_env + # Initialize PostgreSQL (script handles modular vs internal mode internally) echo "Setting up PostgreSQL..." . /app/docker/init/02-postgres.sh @@ -165,7 +260,7 @@ if [[ "$DISPATCHARR_ENV" != "modular" ]]; then done postgres_pid=$(su - "$POSTGRES_USER" -c "$PG_BINDIR/pg_ctl -D ${POSTGRES_DIR} status" | sed -n 's/.*PID: \([0-9]\+\).*/\1/p') echo "✅ Postgres started with PID $postgres_pid" - pids+=("$postgres_pid") + if [ -n "$postgres_pid" ]; then pids+=("$postgres_pid"); pid_names[$postgres_pid]="postgres"; fi # Unconditional startup guarantees — run on every AIO startup. # Each is idempotent and handles all scenarios (fresh, upgrade, restart). @@ -207,13 +302,13 @@ if [[ "$DISPATCHARR_ENV" = "dev" ]]; then su - "$POSTGRES_USER" -c "cd /app/frontend && npm run dev &" npm_pid=$(pgrep vite | sort | head -n1) echo "✅ vite started with PID $npm_pid" - pids+=("$npm_pid") + if [ -n "$npm_pid" ]; then pids+=("$npm_pid"); pid_names[$npm_pid]="vite"; fi else echo "🚀 Starting nginx..." nginx - nginx_pid=$(pgrep nginx | sort | head -n1) + nginx_pid=$(pgrep nginx | sort | head -n1) echo "✅ nginx started with PID $nginx_pid" - pids+=("$nginx_pid") + if [ -n "$nginx_pid" ]; then pids+=("$nginx_pid"); pid_names[$nginx_pid]="nginx"; fi fi @@ -262,39 +357,7 @@ fi # This preserves both the nice value and environment variables nice -n "$UWSGI_NICE_LEVEL" su - "$POSTGRES_USER" -c "cd /app && exec $VIRTUAL_ENV/bin/uwsgi $uwsgi_args" & uwsgi_pid=$! echo "✅ uwsgi started with PID $uwsgi_pid (nice $UWSGI_NICE_LEVEL)" -pids+=("$uwsgi_pid") - -# sed -i 's/protected-mode yes/protected-mode no/g' /etc/redis/redis.conf -# su - "$POSTGRES_USER" -c "redis-server --protected-mode no &" -# redis_pid=$(pgrep redis) -# echo "✅ redis started with PID $redis_pid" -# pids+=("$redis_pid") - -# echo "🚀 Starting gunicorn..." -# su - "$POSTGRES_USER" -c "cd /app && gunicorn dispatcharr.asgi:application \ -# --bind 0.0.0.0:5656 \ -# --worker-class uvicorn.workers.UvicornWorker \ -# --workers 2 \ -# --threads 1 \ -# --timeout 0 \ -# --keep-alive 30 \ -# --access-logfile - \ -# --error-logfile - &" -# gunicorn_pid=$(pgrep gunicorn | sort | head -n1) -# echo "✅ gunicorn started with PID $gunicorn_pid" -# pids+=("$gunicorn_pid") - -# echo "Starting celery and beat..." -# su - "$POSTGRES_USER" -c "cd /app && celery -A dispatcharr worker -l info --autoscale=8,2 &" -# celery_pid=$(pgrep celery | sort | head -n1) -# echo "✅ celery started with PID $celery_pid" -# pids+=("$celery_pid") - -# su - "$POSTGRES_USER" -c "cd /app && celery -A dispatcharr beat -l info &" -# beat_pid=$(pgrep beat | sort | head -n1) -# echo "✅ celery beat started with PID $beat_pid" -# pids+=("$beat_pid") - +pids+=("$uwsgi_pid"); pid_names[$uwsgi_pid]="uwsgi" # Wait for services to fully initialize before checking hardware echo "⏳ Waiting for services to fully initialize before hardware check..." @@ -307,18 +370,23 @@ echo "🔍 Running hardware acceleration check..." # Wait for at least one process to exit and log the process that exited first if [ ${#pids[@]} -gt 0 ]; then echo "⏳ Dispatcharr is running. Monitoring processes..." + set +e while kill -0 "${pids[@]}" 2>/dev/null; do sleep 1 # Wait for a second before checking again done - echo "🚨 One of the processes exited! Checking which one..." + # Only report unexpected exits — skip if cleanup was already triggered by + # the trap (i.e. docker stop sent SIGTERM and we shut down intentionally) + if ! $_cleanup_done; then + echo "🚨 One of the processes exited unexpectedly! Checking which one..." - for pid in "${pids[@]}"; do - if ! kill -0 "$pid" 2>/dev/null; then - process_name=$(ps -p "$pid" -o comm=) - echo "❌ Process $process_name (PID: $pid) has exited!" - fi - done + for pid in "${pids[@]}"; do + if ! kill -0 "$pid" 2>/dev/null; then + process_name=${pid_names[$pid]:-unknown} + echo "❌ Process $process_name (PID: $pid) has exited!" + fi + done + fi else echo "❌ No processes started. Exiting." exit 1 diff --git a/docker/init/00-fix-pg-ssl-key.sh b/docker/init/00-fix-pg-ssl-key.sh new file mode 100644 index 00000000..d7065815 --- /dev/null +++ b/docker/init/00-fix-pg-ssl-key.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# +# Fix TLS client key permissions and ownership for PostgreSQL. +# libpq requires the client key to be 0600 or stricter. +# +# Triggers on: +# - Permissions too open (Docker Desktop mounts files as 0777) +# - Wrong ownership (Kubernetes secrets / Docker volumes mount as root; +# the application user can't read a root-owned 0600 key) +# - Read-only source (volume mounted :ro — can't chmod in place) +# +# Usage: source this script with FIXED_KEY_PATH set to the destination. +# FIXED_KEY_PATH="/data/.pg-client.key" +# . /app/docker/init/00-fix-pg-ssl-key.sh +# +# After sourcing, POSTGRES_SSL_KEY is updated to the fixed path if a copy +# was needed. The caller is responsible for propagating the new value to +# /etc/environment or profile.d if required. + +: "${FIXED_KEY_PATH:?FIXED_KEY_PATH must be set before sourcing fix-pg-ssl-key.sh}" + +if [ -n "${POSTGRES_SSL_KEY:-}" ] && [ -f "$POSTGRES_SSL_KEY" ]; then + _key_perms=$(stat -c '%a' "$POSTGRES_SSL_KEY" 2>/dev/null) + _key_owner=$(stat -c '%u' "$POSTGRES_SSL_KEY" 2>/dev/null) + _needs_fix=false + + if [ "$_key_perms" != "600" ] && [ "$_key_perms" != "640" ]; then + _needs_fix=true + elif [ "$(id -u)" = "0" ] && [ -n "${PUID:-}" ] && [ "$_key_owner" != "$PUID" ]; then + _needs_fix=true + fi + + if [ "$_needs_fix" = true ]; then + cp "$POSTGRES_SSL_KEY" "$FIXED_KEY_PATH" + chmod 600 "$FIXED_KEY_PATH" + if [ "$(id -u)" = "0" ] && [ -n "${PUID:-}" ]; then + chown "${PUID}:${PGID:-$PUID}" "$FIXED_KEY_PATH" + fi + export POSTGRES_SSL_KEY="$FIXED_KEY_PATH" + echo "Fixed PostgreSQL client key (perms: ${_key_perms}, owner: ${_key_owner} → ${PUID:-root}:600)" + fi + + unset _key_perms _key_owner _needs_fix +fi diff --git a/docker/init/99-init-dev.sh b/docker/init/99-init-dev.sh index 5afbc96d..d2e6dd0d 100644 --- a/docker/init/99-init-dev.sh +++ b/docker/init/99-init-dev.sh @@ -1,25 +1,33 @@ #!/bin/bash -echo "🚀 Development Mode - Setting up Frontend..." +if [ ! -e "/tmp/init" ]; then + echo "🚀 Development Mode - Setting up Frontend..." -# Install Node.js -if ! command -v node 2>&1 >/dev/null -then - echo "=== setting up nodejs ===" - curl -sL https://deb.nodesource.com/setup_23.x -o /tmp/nodesource_setup.sh - bash /tmp/nodesource_setup.sh - apt-get update - apt-get install -y --no-install-recommends \ - nodejs -fi - -# Install frontend dependencies -cd /app/frontend && npm install -# Install Python dependencies using UV -cd /app && uv sync --python $UV_PROJECT_ENVIRONMENT/bin/python --no-install-project --no-dev - -# Install debugpy for remote debugging -if [ "$DISPATCHARR_DEBUG" = "true" ]; then - echo "=== setting up debugpy ===" - uv pip install --python $UV_PROJECT_ENVIRONMENT/bin/python debugpy + # Install Node.js + if ! command -v node 2>&1 >/dev/null + then + echo "=== setting up nodejs ===" + curl -sL https://deb.nodesource.com/setup_23.x -o /tmp/nodesource_setup.sh + bash /tmp/nodesource_setup.sh + apt-get update + apt-get install -y --no-install-recommends \ + nodejs + fi + + # Install frontend dependencies + cd /app/frontend && npm install + # Install Python dependencies using UV + cd /app && uv sync --python $UV_PROJECT_ENVIRONMENT/bin/python --no-install-project --no-dev + + # Install debugpy for remote debugging + if [ "$DISPATCHARR_DEBUG" = "true" ]; then + echo "=== setting up debugpy ===" + uv pip install --python $UV_PROJECT_ENVIRONMENT/bin/python debugpy + fi + + if [[ "$DISPATCHARR_ENV" = "dev" ]]; then + touch /tmp/init + fi +else + echo "Development mode initialization already done. Skipping dev setup." fi diff --git a/docker/nginx.conf b/docker/nginx.conf index e08d08f2..c5cb429e 100644 --- a/docker/nginx.conf +++ b/docker/nginx.conf @@ -42,7 +42,7 @@ server { alias /data/backups/; } - location /api/logos/(?\d+)/cache/ { + location ~ ^/api/channels/logos/(?\d+)/cache/ { proxy_pass http://127.0.0.1:5656; proxy_cache logo_cache; proxy_cache_key "$scheme$request_uri"; # Cache per logo URL @@ -50,7 +50,7 @@ server { proxy_cache_use_stale error timeout updating; # Serve stale if Django is slow } - location ~ ^/api/channels/logos/(?\d+)/cache/ { + location ~ ^/api/vod/vodlogos/(?\d+)/cache/ { proxy_pass http://127.0.0.1:5656; proxy_cache logo_cache; proxy_cache_key "$scheme$request_uri"; # Cache per logo URL @@ -91,12 +91,9 @@ server { location /proxy/ { include uwsgi_params; uwsgi_pass unix:/app/uwsgi.sock; - proxy_http_version 1.1; - proxy_set_header Connection ""; - proxy_buffering off; - proxy_cache off; - proxy_read_timeout 300s; - proxy_send_timeout 300s; + uwsgi_buffering off; + uwsgi_read_timeout 300s; + uwsgi_send_timeout 300s; client_max_body_size 0; } } diff --git a/docker/tests/test-tls-postgres.sh b/docker/tests/test-tls-postgres.sh new file mode 100644 index 00000000..8a7630dc --- /dev/null +++ b/docker/tests/test-tls-postgres.sh @@ -0,0 +1,892 @@ +#!/bin/bash +# +# Integration test suite for TLS/mTLS in modular mode. +# Validates that Dispatcharr connects correctly to external PostgreSQL and +# Redis services using various TLS configurations. +# +# Prerequisites: +# - Docker Desktop (or Docker Engine) running +# - Internet access (pulls postgres:17, redis:latest) +# - ~10-15 minutes for a full run +# +# Usage: +# cd +# bash docker/tests/test-tls-postgres.sh [--skip-build] [--keep-on-fail] [scenario_name] +# +# Options: +# --skip-build Skip Docker image build (use existing dispatcharr:tls-test image) +# --keep-on-fail Don't clean up containers/volumes on failure (for debugging) +# scenario_name Run only the named scenario +# +# Scenarios: +# modular_mtls_no_password PG mTLS cert-only auth, no password +# modular_mtls_with_password PG mTLS + password auth combined +# modular_tls_server_only PG server-side TLS only (no client cert) +# modular_tls_key_permission PG mTLS with 0777 client key (Docker Desktop scenario) +# modular_no_tls_regression Non-TLS modular mode still works +# modular_pg_verify_full PG mTLS with verify-full (CN must match hostname) +# modular_redis_tls Redis with TLS (server-side verification) +# modular_full_tls_celery PG mTLS + Redis TLS with separate Celery container +# +# Exit codes: +# 0 All tests passed +# 1 One or more tests failed (or build failed) + +set -uo pipefail + +# Prevent Git Bash (MINGW) from converting Unix paths +export MSYS_NO_PATHCONV=1 + +############################################################################### +# Configuration +############################################################################### +IMAGE_NAME="dispatcharr:tls-test" +TEST_PREFIX="tls_test" +STARTUP_TIMEOUT=120 +SKIP_BUILD=false +KEEP_ON_FAIL=false +SINGLE_SCENARIO="" +PASS=0 +FAIL=0 +SKIP=0 +ERRORS=() +CERT_DIR="" + +# Colors (disabled if not a terminal) +if [ -t 1 ]; then + RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m' + CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m' +else + RED=''; GREEN=''; YELLOW=''; CYAN=''; BOLD=''; NC='' +fi + +############################################################################### +# Parse arguments +############################################################################### +for arg in "$@"; do + case "$arg" in + --skip-build) SKIP_BUILD=true ;; + --keep-on-fail) KEEP_ON_FAIL=true ;; + -*) echo "Unknown option: $arg"; exit 1 ;; + *) SINGLE_SCENARIO="$arg" ;; + esac +done + +############################################################################### +# Helpers +############################################################################### +CURRENT_SCENARIO="" +CLEANUP_ITEMS=() + +log_pass() { echo -e " ${GREEN}✅ $1${NC}"; PASS=$((PASS + 1)); } +log_fail() { echo -e " ${RED}❌ $1${NC}"; FAIL=$((FAIL + 1)); ERRORS+=("[$CURRENT_SCENARIO] $1"); } +log_skip() { echo -e " ${YELLOW}⏭️ $1${NC}"; SKIP=$((SKIP + 1)); } +log_info() { echo -e " ${CYAN}ℹ️ $1${NC}"; } +section() { echo -e "\n${BOLD}━━━ $1 ━━━${NC}"; SCENARIO_FAIL_BEFORE=$FAIL; } + +track_container() { CLEANUP_ITEMS+=("container:$1"); } +track_volume() { CLEANUP_ITEMS+=("volume:$1"); } +track_network() { CLEANUP_ITEMS+=("network:$1"); } + +fresh_volume() { + local vol="$1" + docker rm -f $(docker ps -aq --filter "volume=${vol}") 2>/dev/null || true + docker volume rm "$vol" 2>/dev/null || true + docker volume create "$vol" >/dev/null + track_volume "$vol" +} + +cleanup_scenario() { + if [ "$KEEP_ON_FAIL" = true ] && [ "$FAIL" -gt "${SCENARIO_FAIL_BEFORE:-0}" ]; then + log_info "Keeping resources for debugging (--keep-on-fail)" + CLEANUP_ITEMS=() + return + fi + for item in "${CLEANUP_ITEMS[@]}"; do + local type="${item%%:*}" + local name="${item#*:}" + case "$type" in + container) docker stop "$name" 2>/dev/null; docker rm -f "$name" 2>/dev/null ;; + volume) docker volume rm "$name" 2>/dev/null ;; + network) docker network rm "$name" 2>/dev/null ;; + esac + done + CLEANUP_ITEMS=() +} + +trap 'cleanup_scenario; [ -n "$CERT_DIR" ] && rm -rf "$CERT_DIR"' EXIT + +wait_for_ready() { + local name="$1" + local timeout="${2:-$STARTUP_TIMEOUT}" + local elapsed=0 + + while [ $elapsed -lt $timeout ]; do + if ! docker ps -q -f "name=^${name}$" 2>/dev/null | grep -q .; then + echo " Container $name exited unexpectedly" + return 1 + fi + if docker logs "$name" 2>&1 | grep -q "uwsgi started with PID"; then + return 0 + fi + sleep 3 + ((elapsed+=3)) + done + echo " Timeout (${timeout}s) waiting for $name" + return 1 +} + +_capture_logs() { + local container="$1" logfile="$2" + docker logs "$container" > "$logfile" 2>&1 +} + +check_log_contains() { + local container="$1" pattern="$2" description="$3" + local tmplog; tmplog=$(mktemp) + _capture_logs "$container" "$tmplog" + if grep -q "$pattern" "$tmplog"; then + log_pass "$description" + else + log_fail "$description (pattern not found: $pattern)" + fi + rm -f "$tmplog" +} + +check_log_absent() { + local container="$1" pattern="$2" description="$3" + local tmplog; tmplog=$(mktemp) + _capture_logs "$container" "$tmplog" + if grep -q "$pattern" "$tmplog"; then + log_fail "$description (unexpected pattern found: $pattern)" + else + log_pass "$description" + fi + rm -f "$tmplog" +} + +check_migrations_done() { + local container="$1" + local tmplog; tmplog=$(mktemp) + _capture_logs "$container" "$tmplog" + if grep -qE "Running migrations|No migrations to apply|Operations to perform|Applying .+\.\.\. OK" "$tmplog"; then + log_pass "Django migrations completed" + elif grep -q "uwsgi started with PID" "$tmplog"; then + log_pass "Django migrations completed (confirmed via uwsgi startup)" + else + log_fail "Django migrations did not complete" + fi + rm -f "$tmplog" +} + +check_no_permission_errors() { + local container="$1" + local tmplog; tmplog=$(mktemp) + _capture_logs "$container" "$tmplog" + local errors + errors=$(grep -iE "permission denied|operation not permitted" "$tmplog" \ + | grep -v "GPU acceleration" | grep -v "Warning:" | head -5) + rm -f "$tmplog" + if [ -n "$errors" ]; then + log_fail "Permission errors in logs:" + echo "$errors" | while read -r line; do echo " $line"; done + else + log_pass "No permission errors in logs" + fi +} + +dump_logs_on_fail() { + local container="$1" + if [ $FAIL -gt ${SCENARIO_FAIL_BEFORE:-0} ]; then + echo -e " ${YELLOW}--- Container logs ($container) ---${NC}" + docker logs "$container" 2>&1 | tail -30 | sed 's/^/ /' + echo -e " ${YELLOW}--- End logs ---${NC}" + fi +} + +############################################################################### +# Certificate generation +############################################################################### +generate_test_certs() { + CERT_DIR=$(mktemp -d) + log_info "Generating test certificates in $CERT_DIR" + + # Generate certs inside a container for cross-platform compatibility. + # Shared CA for both PG and Redis. CN of server certs must match their + # Docker container hostnames for verify-full mode. + docker run --rm --entrypoint sh \ + -v "$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR"):/certs" \ + -w /certs alpine/openssl -c ' + # Shared CA + openssl req -new -x509 -days 1 -nodes \ + -keyout ca.key -out ca.crt -subj "/CN=Test CA" 2>/dev/null && + + # PostgreSQL server cert (CN = PG container hostname) + openssl req -new -nodes \ + -keyout pg-server.key -out pg-server.csr -subj "/CN='"${TEST_PREFIX}"'_pg" 2>/dev/null && + openssl x509 -req -days 1 -in pg-server.csr \ + -CA ca.crt -CAkey ca.key -CAcreateserial -out pg-server.crt 2>/dev/null && + # PostgreSQL client cert (CN = POSTGRES_USER) + openssl req -new -nodes \ + -keyout pg-client.key -out pg-client.csr -subj "/CN=dispatch" 2>/dev/null && + openssl x509 -req -days 1 -in pg-client.csr \ + -CA ca.crt -CAkey ca.key -CAcreateserial -out pg-client.crt 2>/dev/null && + + # Redis server cert (CN = Redis container hostname) + openssl req -new -nodes \ + -keyout redis-server.key -out redis-server.csr -subj "/CN='"${TEST_PREFIX}"'_redis" 2>/dev/null && + openssl x509 -req -days 1 -in redis-server.csr \ + -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-server.crt 2>/dev/null && + + # Backwards-compat aliases (existing PG-only scenarios use these names) + cp pg-server.crt server.crt && cp pg-server.key server.key && + cp pg-client.crt client.crt && cp pg-client.key client.key && + + chmod 600 pg-server.key pg-client.key redis-server.key server.key client.key + ' || { log_fail "Certificate generation failed"; return 1; } + + log_pass "Test certificates generated" +} + +############################################################################### +# Start a TLS-enabled Redis container +############################################################################### +start_tls_redis() { + local name="$1" net="$2" + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + # Redis needs certs owned by redis user (uid 999 in the official image). + # Mount certs, copy to a writable location, fix ownership, then start + # with TLS flags. + docker run -d --name "$name" --network "$net" \ + -v "${cert_mount}:/certs:ro" \ + redis:latest \ + sh -c ' + cp /certs/redis-server.crt /certs/redis-server.key /certs/ca.crt /tmp/ && + chmod 600 /tmp/redis-server.key && + chown redis:redis /tmp/redis-server.crt /tmp/redis-server.key /tmp/ca.crt && + exec redis-server \ + --tls-port 6379 --port 0 \ + --tls-cert-file /tmp/redis-server.crt \ + --tls-key-file /tmp/redis-server.key \ + --tls-ca-cert-file /tmp/ca.crt \ + --tls-auth-clients no + ' >/dev/null + + # Wait for Redis TLS to be ready + local elapsed=0 + while [ $elapsed -lt 20 ]; do + if docker exec "$name" redis-cli --tls \ + --cert /certs/redis-server.crt --key /certs/redis-server.key --cacert /certs/ca.crt \ + ping 2>/dev/null | grep -q "PONG"; then + break + fi + sleep 2; elapsed=$((elapsed + 2)) + done +} + +############################################################################### +# Start a TLS-enabled PostgreSQL container +############################################################################### +start_tls_postgres() { + local name="$1" net="$2" hba_auth="$3" + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + docker run -d --name "$name" --network "$net" \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=tempsetup \ + -e POSTGRES_DB=dispatcharr \ + -v "${cert_mount}:/certs:ro" \ + postgres:17 >/dev/null + + # Wait for PG to initialize + local elapsed=0 + while [ $elapsed -lt 30 ]; do + if docker exec "$name" su postgres -c "/usr/lib/postgresql/17/bin/pg_isready" 2>/dev/null | grep -q "accepting"; then + break + fi + sleep 2; ((elapsed+=2)) + done + + # Configure SSL and pg_hba.conf + docker exec "$name" bash -c " + cp /certs/server.crt /certs/server.key /certs/ca.crt /var/lib/postgresql/ + chown postgres:postgres /var/lib/postgresql/server.crt /var/lib/postgresql/server.key /var/lib/postgresql/ca.crt + chmod 600 /var/lib/postgresql/server.key + echo \"ssl = on\" >> /var/lib/postgresql/data/postgresql.conf + echo \"ssl_cert_file = '/var/lib/postgresql/server.crt'\" >> /var/lib/postgresql/data/postgresql.conf + echo \"ssl_key_file = '/var/lib/postgresql/server.key'\" >> /var/lib/postgresql/data/postgresql.conf + echo \"ssl_ca_file = '/var/lib/postgresql/ca.crt'\" >> /var/lib/postgresql/data/postgresql.conf + cat > /var/lib/postgresql/data/pg_hba.conf << HBA +local all all trust +hostssl all all 0.0.0.0/0 ${hba_auth} +hostssl all all ::0/0 ${hba_auth} +HBA + su postgres -c '/usr/lib/postgresql/17/bin/pg_ctl reload -D /var/lib/postgresql/data' + " >/dev/null 2>&1 + sleep 1 +} + +############################################################################### +# Test scenarios +############################################################################### + +test_modular_mtls_no_password() { + CURRENT_SCENARIO="modular_mtls_no_password" + section "Modular mode — mTLS cert-only auth (no password)" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + start_tls_postgres "$pg_name" "$net" "cert" + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + # No POSTGRES_PASSWORD — cert-only auth + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e POSTGRES_SSL=true \ + -e POSTGRES_SSL_MODE=verify-ca \ + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt \ + -e POSTGRES_SSL_CERT=/certs/client.crt \ + -e POSTGRES_SSL_KEY=/certs/client.key \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with mTLS cert-only auth" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed with mTLS" + check_log_contains "$name" "PostgreSQL TLS: enabled" \ + "Django sees TLS enabled" + check_migrations_done "$name" + check_no_permission_errors "$name" + else + log_fail "Container failed to start with mTLS cert-only auth" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_mtls_with_password() { + CURRENT_SCENARIO="modular_mtls_with_password" + section "Modular mode — mTLS + password auth" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + # cert + md5 password + start_tls_postgres "$pg_name" "$net" "cert" + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=tempsetup \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e POSTGRES_SSL=true \ + -e POSTGRES_SSL_MODE=verify-ca \ + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt \ + -e POSTGRES_SSL_CERT=/certs/client.crt \ + -e POSTGRES_SSL_KEY=/certs/client.key \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with mTLS + password" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed with mTLS + password" + check_migrations_done "$name" + else + log_fail "Container failed to start with mTLS + password" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_tls_server_only() { + CURRENT_SCENARIO="modular_tls_server_only" + section "Modular mode — server-only TLS (no client cert)" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + # md5 auth over TLS (no client cert required) + start_tls_postgres "$pg_name" "$net" "md5" + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=tempsetup \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e POSTGRES_SSL=true \ + -e POSTGRES_SSL_MODE=verify-ca \ + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with server-only TLS" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed with server-only TLS" + check_migrations_done "$name" + else + log_fail "Container failed to start with server-only TLS" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_tls_key_permission() { + CURRENT_SCENARIO="modular_tls_key_permission" + section "Modular mode — mTLS with 0777 client key (Docker Desktop scenario)" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + start_tls_postgres "$pg_name" "$net" "cert" + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + # Create a copy of certs with 0777 key permissions + local bad_perms_dir + bad_perms_dir=$(mktemp -d) + cp "$CERT_DIR"/ca.crt "$CERT_DIR"/client.crt "$CERT_DIR"/client.key "$bad_perms_dir/" + chmod 777 "$bad_perms_dir/client.key" + + local cert_mount + cert_mount="$(cygpath -w "$bad_perms_dir" 2>/dev/null || echo "$bad_perms_dir")" + + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e POSTGRES_SSL=true \ + -e POSTGRES_SSL_MODE=verify-ca \ + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt \ + -e POSTGRES_SSL_CERT=/certs/client.crt \ + -e POSTGRES_SSL_KEY=/certs/client.key \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with 0777 client key" + check_log_contains "$name" "Fixed PostgreSQL client key" \ + "Key permission fix triggered" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed after key fix" + check_migrations_done "$name" + else + log_fail "Container failed to start with 0777 client key" + fi + dump_logs_on_fail "$name" + rm -rf "$bad_perms_dir" + cleanup_scenario +} + +test_modular_no_tls_regression() { + CURRENT_SCENARIO="modular_no_tls_regression" + section "Modular mode — no TLS (regression check)" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + # Plain PostgreSQL — no TLS + docker run -d --name "$pg_name" --network "$net" \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=secret \ + -e POSTGRES_DB=dispatcharr \ + postgres:17 >/dev/null + + local elapsed=0 + while [ $elapsed -lt 30 ]; do + if docker exec "$pg_name" su postgres -c "/usr/lib/postgresql/17/bin/pg_isready" 2>/dev/null | grep -q "accepting"; then + break + fi + sleep 2; ((elapsed+=2)) + done + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=secret \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started without TLS (regression check)" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed without TLS" + check_log_absent "$name" "Fixed PostgreSQL client key" \ + "No key fix when TLS disabled" + check_migrations_done "$name" + else + log_fail "Container failed to start without TLS" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_pg_verify_full() { + CURRENT_SCENARIO="modular_pg_verify_full" + section "Modular mode — PG mTLS with verify-full (CN must match hostname)" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + start_tls_postgres "$pg_name" "$net" "cert" + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + # verify-full requires server cert CN to match the hostname used to connect. + # Our PG server cert CN is "${TEST_PREFIX}_pg", which matches the container name + # used in POSTGRES_HOST. + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e POSTGRES_SSL=true \ + -e POSTGRES_SSL_MODE=verify-full \ + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt \ + -e POSTGRES_SSL_CERT=/certs/client.crt \ + -e POSTGRES_SSL_KEY=/certs/client.key \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with verify-full" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed with verify-full" + check_log_contains "$name" "sslmode=verify-full" \ + "Django reports verify-full mode" + check_migrations_done "$name" + else + log_fail "Container failed to start with verify-full" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_redis_tls() { + CURRENT_SCENARIO="modular_redis_tls" + section "Modular mode — Redis with TLS" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + # Plain PG (no TLS) — isolate Redis TLS testing + docker run -d --name "$pg_name" --network "$net" \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=secret \ + -e POSTGRES_DB=dispatcharr \ + postgres:17 >/dev/null + + local elapsed=0 + while [ $elapsed -lt 30 ]; do + if docker exec "$pg_name" su postgres -c "/usr/lib/postgresql/17/bin/pg_isready" 2>/dev/null | grep -q "accepting"; then + break + fi + sleep 2; elapsed=$((elapsed + 2)) + done + + start_tls_redis "$redis_name" "$net" + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=secret \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e REDIS_SSL=true \ + -e REDIS_SSL_VERIFY=false \ + -e REDIS_SSL_CA_CERT=/certs/ca.crt \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with Redis TLS" + check_log_contains "$name" "Redis TLS: enabled" \ + "Django reports Redis TLS enabled" + check_log_contains "$name" "Redis at ${redis_name}" \ + "Redis connected via TLS" + check_migrations_done "$name" + else + log_fail "Container failed to start with Redis TLS" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_full_tls_celery() { + CURRENT_SCENARIO="modular_full_tls_celery" + section "Modular mode — PG mTLS + Redis TLS with Celery container" + + local name="${TEST_PREFIX}_app" + local celery_name="${TEST_PREFIX}_celery" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name" + track_container "$name"; track_container "$celery_name" + + start_tls_postgres "$pg_name" "$net" "cert" + start_tls_redis "$redis_name" "$net" + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + # Shared env vars for both web and celery containers + local -a tls_env=( + -e DISPATCHARR_ENV=modular + -e POSTGRES_HOST="$pg_name" + -e POSTGRES_PORT=5432 + -e POSTGRES_USER=dispatch + -e POSTGRES_DB=dispatcharr + -e REDIS_HOST="$redis_name" + -e POSTGRES_SSL=true + -e POSTGRES_SSL_MODE=verify-ca + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt + -e POSTGRES_SSL_CERT=/certs/client.crt + -e POSTGRES_SSL_KEY=/certs/client.key + -e REDIS_SSL=true + -e REDIS_SSL_VERIFY=false + -e REDIS_SSL_CA_CERT=/certs/ca.crt + ) + + # Start web container first (generates JWT, runs migrations) + docker run -d --name "$name" --network "$net" \ + "${tls_env[@]}" \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if ! wait_for_ready "$name"; then + log_fail "Web container failed to start with full TLS" + dump_logs_on_fail "$name" + cleanup_scenario + return + fi + log_pass "Web container started with PG mTLS + Redis TLS" + + # Start Celery container (shares /data volume for JWT, waits for migrations) + docker run -d --name "$celery_name" --network "$net" \ + "${tls_env[@]}" \ + -e DJANGO_SETTINGS_MODULE=dispatcharr.settings \ + -e PYTHONUNBUFFERED=1 \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + --entrypoint /app/docker/entrypoint.celery.sh \ + "$IMAGE_NAME" >/dev/null + + # Wait for Celery to start (look for "starting Celery" message) + local elapsed=0 + local celery_ok=false + while [ $elapsed -lt 90 ]; do + if ! docker ps -q -f "name=^${celery_name}$" 2>/dev/null | grep -q .; then + echo " Celery container exited unexpectedly" + break + fi + if docker logs "$celery_name" 2>&1 | grep -q "starting Celery"; then + celery_ok=true + break + fi + sleep 3; elapsed=$((elapsed + 3)) + done + + if [ "$celery_ok" = true ]; then + log_pass "Celery container started with PG mTLS + Redis TLS" + check_log_contains "$celery_name" "Migrations complete" \ + "Celery confirmed migrations complete via TLS" + check_log_contains "$celery_name" "PostgreSQL TLS: enabled" \ + "Celery sees PostgreSQL TLS enabled" + check_log_contains "$celery_name" "Redis TLS: enabled" \ + "Celery sees Redis TLS enabled" + else + log_fail "Celery container failed to start with full TLS" + echo -e " ${YELLOW}--- Celery logs ---${NC}" + docker logs "$celery_name" 2>&1 | tail -20 | sed 's/^/ /' + echo -e " ${YELLOW}--- End logs ---${NC}" + fi + + dump_logs_on_fail "$name" + cleanup_scenario +} + +############################################################################### +# Main +############################################################################### +echo -e "${BOLD}╔═══════════════════════════════════════════════════════════╗${NC}" +echo -e "${BOLD}║ Dispatcharr — TLS Integration Tests ║${NC}" +echo -e "${BOLD}╚═══════════════════════════════════════════════════════════╝${NC}" + +# Build image +if [ "$SKIP_BUILD" = false ]; then + echo -e "\n${BOLD}Building test image...${NC}" + if ! docker build -t "$IMAGE_NAME" -f docker/Dockerfile . 2>&1 | tail -5; then + echo -e "${RED}Build failed${NC}" + exit 1 + fi + echo -e "${GREEN}Build complete${NC}" +else + echo -e "\n${YELLOW}Skipping build (--skip-build)${NC}" +fi + +# Generate certificates +generate_test_certs || exit 1 + +# Run scenarios +SCENARIOS=( + modular_mtls_no_password + modular_mtls_with_password + modular_tls_server_only + modular_tls_key_permission + modular_no_tls_regression + modular_pg_verify_full + modular_redis_tls + modular_full_tls_celery +) + +for scenario in "${SCENARIOS[@]}"; do + if [ -n "$SINGLE_SCENARIO" ] && [ "$scenario" != "$SINGLE_SCENARIO" ]; then + continue + fi + "test_${scenario}" +done + +# Clean up certs +rm -rf "$CERT_DIR" + +# Summary +echo -e "\n${BOLD}═══════════════════════════════════════════════════════════${NC}" +echo -e " ${GREEN}Passed: $PASS${NC} ${RED}Failed: $FAIL${NC} ${YELLOW}Skipped: $SKIP${NC}" +if [ ${#ERRORS[@]} -gt 0 ]; then + echo -e "\n ${RED}Failures:${NC}" + for err in "${ERRORS[@]}"; do + echo -e " ${RED}• $err${NC}" + done +fi +echo -e "${BOLD}═══════════════════════════════════════════════════════════${NC}" + +[ $FAIL -eq 0 ] diff --git a/docker/uwsgi.dev.ini b/docker/uwsgi.dev.ini index c4c5d0aa..51aae9a5 100644 --- a/docker/uwsgi.dev.ini +++ b/docker/uwsgi.dev.ini @@ -8,7 +8,7 @@ exec-pre = python /app/scripts/wait_for_redis.py ; Start Redis first -attach-daemon = redis-server +attach-daemon = redis-server --protected-mode no ; Then start other services with configurable nice level (default: 5 for low priority) ; Users can override via CELERY_NICE_LEVEL environment variable in docker-compose attach-daemon = nice -n $(CELERY_NICE_LEVEL) celery -A dispatcharr worker --autoscale=6,1 @@ -57,4 +57,4 @@ logformat-strftime = true log-date = %%Y-%%m-%%d %%H:%%M:%%S,000 # Use formatted time with environment variable for log level log-format = %(ftime) $(DISPATCHARR_LOG_LEVEL) uwsgi.requests Worker ID: %(wid) %(method) %(status) %(uri) %(msecs)ms -log-buffering = 1024 # Add buffer size limit for logging \ No newline at end of file +log-buffering = 1024 # Add buffer size limit for logging diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 9a0adea6..e5350fab 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -2518,9 +2518,9 @@ } }, "node_modules/@xmldom/xmldom": { - "version": "0.8.11", - "resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.11.tgz", - "integrity": "sha512-cQzWCtO6C8TQiYl1ruKNn2U6Ao4o4WBBcbL61yJl84x+j5sOWWFU9X7DpND8XZG3daDppSsigMdfAIl2upQBRw==", + "version": "0.8.12", + "resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.12.tgz", + "integrity": "sha512-9k/gHF6n/pAi/9tqr3m3aqkuiNosYTurLLUtc7xQ9sxB/wm7WPygCv8GYa6mS0fLJEHhqMC1ATYhz++U/lRHqg==", "license": "MIT", "engines": { "node": ">=10.0.0" @@ -2704,16 +2704,16 @@ } }, "node_modules/brace-expansion": { - "version": "5.0.2", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.2.tgz", - "integrity": "sha512-Pdk8c9poy+YhOgVWw1JNN22/HcivgKWwpxKq04M/jTmHyCZn12WPJebZxdjSa5TmBqISrUSgNYU3eRORljfCCw==", + "version": "5.0.5", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.5.tgz", + "integrity": "sha512-VZznLgtwhn+Mact9tfiwx64fA9erHH/MCXEUfB/0bX/6Fz6ny5EGTXYltMocqg4xFAQZtnO3DHWWXi8RiuN7cQ==", "dev": true, "license": "MIT", "dependencies": { "balanced-match": "^4.0.2" }, "engines": { - "node": "20 || >=22" + "node": "18 || 20 || >=22" } }, "node_modules/cac": { @@ -2850,9 +2850,9 @@ } }, "node_modules/cosmiconfig/node_modules/yaml": { - "version": "1.10.2", - "resolved": "https://registry.npmjs.org/yaml/-/yaml-1.10.2.tgz", - "integrity": "sha512-r3vXyErRCYJ7wg28yvBY5VSoAF8ZvlcW9/BwUzEtUsjvX/DKs24dIkuwjtuprwJJHsbyUbLApepYTR1BN4uHrg==", + "version": "1.10.3", + "resolved": "https://registry.npmjs.org/yaml/-/yaml-1.10.3.tgz", + "integrity": "sha512-vIYeF1u3CjlhAFekPPAk2h/Kv4T3mAkMox5OymRiJQB0spDP10LHvt+K7G9Ny6NuuMAb25/6n1qyUjAcGNf/AA==", "license": "ISC", "engines": { "node": ">= 6" @@ -3554,9 +3554,9 @@ } }, "node_modules/flatted": { - "version": "3.4.1", - "resolved": "https://registry.npmjs.org/flatted/-/flatted-3.4.1.tgz", - "integrity": "sha512-IxfVbRFVlV8V/yRaGzk0UVIcsKKHMSfYw66T/u4nTwlWteQePsxe//LjudR1AMX4tZW3WFCh3Zqa/sjlqpbURQ==", + "version": "3.4.2", + "resolved": "https://registry.npmjs.org/flatted/-/flatted-3.4.2.tgz", + "integrity": "sha512-PjDse7RzhcPkIJwy5t7KPWQSZ9cAbzQXcafsetQoD7sOJRQlGikNbx7yZp2OotDnJyrDcbyRq3Ttb18iYOqkxA==", "dev": true, "license": "ISC" }, @@ -3986,9 +3986,9 @@ } }, "node_modules/lodash": { - "version": "4.17.23", - "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.23.tgz", - "integrity": "sha512-LgVTMpQtIopCi79SJeDiP0TfWi5CNEc/L/aRdTh3yIvmZXTnheWpKjSZhnvMl8iXbC1tFg9gdHHDMLoV7CnG+w==", + "version": "4.18.1", + "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.18.1.tgz", + "integrity": "sha512-dMInicTPVE8d1e5otfwmmjlxkZoUpiVLwyeTdUsi/Caj/gfzzblBcCE5sRHV/AsjuCmxWrte2TNGSYuCeCq+0Q==", "license": "MIT" }, "node_modules/lodash.clamp": { @@ -4362,9 +4362,9 @@ "license": "ISC" }, "node_modules/picomatch": { - "version": "4.0.3", - "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.3.tgz", - "integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==", + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", + "integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", "dev": true, "license": "MIT", "engines": { @@ -5490,9 +5490,9 @@ } }, "node_modules/vite": { - "version": "7.3.1", - "resolved": "https://registry.npmjs.org/vite/-/vite-7.3.1.tgz", - "integrity": "sha512-w+N7Hifpc3gRjZ63vYBXA56dvvRlNWRczTdmCBBa+CotUzAPf5b7YMdMR/8CQoeYE5LX3W4wj6RYTgonm1b9DA==", + "version": "7.3.2", + "resolved": "https://registry.npmjs.org/vite/-/vite-7.3.2.tgz", + "integrity": "sha512-Bby3NOsna2jsjfLVOHKes8sGwgl4TT0E6vvpYgnAYDIF/tie7MRaFthmKuHx1NSXjiTueXH3do80FMQgvEktRg==", "dev": true, "license": "MIT", "dependencies": { @@ -5794,6 +5794,24 @@ "dev": true, "license": "MIT" }, + "node_modules/yaml": { + "version": "2.8.3", + "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.8.3.tgz", + "integrity": "sha512-AvbaCLOO2Otw/lW5bmh9d/WEdcDFdQp2Z2ZUH3pX9U2ihyUY0nvLv7J6TrWowklRGPYbB/IuIMfYgxaCPg5Bpg==", + "dev": true, + "license": "ISC", + "optional": true, + "peer": true, + "bin": { + "yaml": "bin.mjs" + }, + "engines": { + "node": ">= 14.6" + }, + "funding": { + "url": "https://github.com/sponsors/eemeli" + } + }, "node_modules/yocto-queue": { "version": "0.1.0", "resolved": "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz", diff --git a/frontend/src/App.jsx b/frontend/src/App.jsx index 7950e8ae..659d8070 100644 --- a/frontend/src/App.jsx +++ b/frontend/src/App.jsx @@ -14,6 +14,7 @@ import Stats from './pages/Stats'; import DVR from './pages/DVR'; import Settings from './pages/Settings'; import PluginsPage from './pages/Plugins'; +import PluginBrowsePage from './pages/PluginBrowse'; import ConnectPage from './pages/Connect'; import ConnectLogsPage from './pages/ConnectLogs'; import Users from './pages/Users'; @@ -153,6 +154,10 @@ const App = () => { } /> } /> } /> + } + /> } /> } /> s.contentType); const metadata = useVideoStore((s) => s.metadata); const hideVideo = useVideoStore((s) => s.hideVideo); + const accessToken = useAuthStore((s) => s.accessToken); const videoRef = useRef(null); const playerRef = useRef(null); @@ -133,7 +135,8 @@ export default function FloatingVideo() { const [isLoading, setIsLoading] = useState(false); const [loadError, setLoadError] = useState(null); - const [showOverlay, setShowOverlay] = useState(true); + const [showOverlay, setShowOverlay] = useState(false); + const [showControls, setShowControls] = useState(false); const [videoSize, setVideoSize] = useState(() => { const prefs = getPlayerPrefs(); const saved = prefs.size; @@ -223,7 +226,8 @@ export default function FloatingVideo() { setIsLoading(true); setLoadError(null); - setShowOverlay(true); // Show overlay initially + setShowOverlay(false); + setShowControls(false); console.log('Initializing VOD player for:', streamUrl); @@ -246,7 +250,8 @@ export default function FloatingVideo() { console.log('Auto-play prevented:', e); setLoadError('Auto-play was prevented. Click play to start.'); }); - // Start overlay timer when video is ready + // Show overlay briefly when video is ready, then auto-hide + setShowOverlay(true); startOverlayTimer(); }; const handleError = (e) => { @@ -298,8 +303,7 @@ export default function FloatingVideo() { setIsLoading(true); setLoadError(null); - - console.log('Initializing live stream player for:', streamUrl); + setShowControls(false); try { if (!mpegts.getFeatureList().mseLivePlayback) { @@ -310,20 +314,34 @@ export default function FloatingVideo() { return; } - const player = mpegts.createPlayer({ - type: 'mpegts', - url: streamUrl, - isLive: true, - enableWorker: true, - enableStashBuffer: false, - liveBufferLatencyChasing: true, - liveSync: true, - cors: true, - autoCleanupSourceBuffer: true, - autoCleanupMaxBackwardDuration: 10, - autoCleanupMinBackwardDuration: 5, - reuseRedirectedURL: true, - }); + // mpegts.js workers run in WorkerGlobalScope where relative URLs are + // not resolved against the page origin. Always pass an absolute URL. + const absoluteStreamUrl = + streamUrl.startsWith('/') && typeof window !== 'undefined' + ? `${window.location.origin}${streamUrl}` + : streamUrl; + + const player = mpegts.createPlayer( + { + type: 'mpegts', + url: absoluteStreamUrl, + isLive: true, + cors: true, + }, + { + enableWorker: true, + enableStashBuffer: false, + liveBufferLatencyChasing: false, + liveSync: false, + autoCleanupSourceBuffer: true, + autoCleanupMaxBackwardDuration: 120, + autoCleanupMinBackwardDuration: 60, + reuseRedirectedURL: true, + headers: accessToken + ? { Authorization: `Bearer ${accessToken}` } + : undefined, + } + ); player.attachMediaElement(videoRef.current); @@ -834,6 +852,7 @@ export default function FloatingVideo() { { + setShowControls(true); if (contentType === 'vod' && !isLoading) { setShowOverlay(true); if (overlayTimeoutRef.current) { @@ -850,7 +869,7 @@ export default function FloatingVideo() { {/* Enhanced video element with better controls for VOD */}