diff --git a/CHANGELOG.md b/CHANGELOG.md index 71b8b8c1..1f8aad67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,67 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Security + +- Updated `requests` 2.32.5 → 2.33.0, resolving the following CVE: + - **CVE-2026-25645** (moderate): Insecure temp file reuse in `extract_zipped_paths()` utility function. +- Updated frontend npm dependencies to resolve 4 audit vulnerabilities (2 moderate, 2 high): + - Updated `brace-expansion` 5.0.2 → 5.0.5, resolving **moderate** zero-step sequence causing process hang and memory exhaustion ([GHSA-f886-m6hf-6m8v](https://github.com/advisories/GHSA-f886-m6hf-6m8v)) + - Updated `flatted` 3.4.1 → 3.4.2, resolving **high** Prototype Pollution via `parse()` in NodeJS flatted ([GHSA-rf6f-7fwh-wjgh](https://github.com/advisories/GHSA-rf6f-7fwh-wjgh)) + - Updated `picomatch` 4.0.3 → 4.0.4, resolving **high** method injection in POSIX character classes causing incorrect glob matching ([GHSA-3v7f-55p6-f55p](https://github.com/advisories/GHSA-3v7f-55p6-f55p)) and a ReDoS vulnerability via extglob quantifiers ([GHSA-c2c7-rcm5-vvqj](https://github.com/advisories/GHSA-c2c7-rcm5-vvqj)) + - Updated `yaml` 1.10.2 → 1.10.3, resolving **moderate** stack overflow via deeply nested YAML collections ([GHSA-48c2-rrv3-qjmp](https://github.com/advisories/GHSA-48c2-rrv3-qjmp)) + +### Added + +- Connection cards on the Stats page now show the **username** of the connected user. For live channel connections a new User column appears between IP Address and Connected; for VOD connections the username is shown inline next to the IP address in the Client summary row. The username is resolved from the user store using the `user_id` stored in Redis client metadata. (Closes #766, Closes #586) +- `ip_address` and `user_id` were not included in the client info returned by `get_detailed_channel_info()` despite being available in the Redis hash. Both fields are now extracted and returned. `user_id` is now also included in the VOD stats response. +- Web UI stream preview now sends an `Authorization: Bearer` header with each mpegts.js request, identifying the logged-in user. Live channel previews initiated from the web UI now appear on the Stats page with the correct username rather than as unknown user. +- `client_connect` and `client_disconnect` system events now include the **username** of the connected user. The username is stored alongside the client metadata in Redis and included in the event payload for `log_system_event` calls (making it available to webhook and script integrations). +- Donate button added to the sidebar footer. A heart icon links to the project's Open Collective page, visible in both expanded and collapsed states. Hovering shows a "Support Dispatcharr" tooltip. The version string is also now clickable to copy it to the clipboard. +- User stream limits: administrators can now set a maximum number of concurrent streams per user account. When a user reaches their limit, the system can automatically terminate an existing stream to free a slot based on configurable rules. Limit enforcement applies to both live channels and VOD. (Closes #544) + - Each user account has a new **Stream Limit** field (0 = unlimited) configurable from the user edit form in Settings → Users. + - Global enforcement behaviour is configurable in Settings → User Limits: + - **Terminate on Limit Exceeded**: automatically stop an existing stream when the user's limit is reached (vs. rejecting the new connection). + - **Terminate Oldest**: prefer terminating the oldest stream when freeing a slot; disable to prefer the newest. + - **Prioritize Single-Client Channels**: prefer terminating streams on channels that only this user is watching. + - **Ignore Same-Channel Connections**: count multiple connections to the same live channel as one stream toward the limit. Same-channel reconnects are always allowed through. When this is enabled and a channel must be freed, all connections to the chosen channel are terminated together so that the unique-channel count actually decreases. VOD is explicitly excluded from this bypass since VOD connections are not shared upstream. +- TLS and mutual TLS (mTLS) support for Redis and PostgreSQL connections in modular deployments. Supports encrypted connections, server certificate verification (Redis: on/off; PostgreSQL: verify-full, verify-ca, require), CA certificate configuration, and client certificate authentication. Configured via environment variables in the docker compose file. Includes startup validation for certificate paths and TLS/URL scheme conflicts, and a read-only Connection Security panel in System Settings. (Closes #950) — Thanks [@CodeBormen](https://github.com/CodeBormen) +- Status filter for M3U group and VOD category filter modals: A new **All / Enabled / Disabled** segmented control is now shown alongside the text search input in the Live, VOD - Movies, and VOD - Series tabs of the M3U Group Filter modal. The status filter works in combination with the text search and also scopes the "Select Visible" / "Deselect Visible" buttons so they only act on the currently visible subset. (Closes #312) + +### Changed + +- M3U Profile form (XC accounts): added a **Simple / Advanced** mode toggle for credential-based URL rewriting. In Simple mode users enter just a new username and password; the search and replace patterns are built automatically from the account's current credentials. In Advanced mode the full regex fields are shown as before. The selected mode is saved to `custom_properties.xcMode` and auto-detected on existing profiles (a profile whose search pattern matches the account's current `username/password` is recognised as Simple automatically). The Live Regex Demonstration panel is hidden in Simple mode. +- XtreamCodes VOD endpoints (`/movie/` and `/series/`) no longer redirect clients to a UUID-based proxy URL. Requests are now handled directly in the proxy layer via `stream_xc_movie` and `stream_xc_episode`, which call `stream_vod()` internally. The original XC path is preserved for the client throughout the stream. +- `CustomTable` column layout now supports flexible (`grow`) columns alongside fixed-width ones: + - Column definitions accept a `grow` property (boolean or number) to opt into flex layout. A numeric value sets the flex-grow weight, allowing relative sizing between grow columns (e.g. `grow: 2` gives a column twice the share of spare space as `grow: 1`). + - `maxSize` is now respected on grow columns, capping how wide they expand via `maxWidth`. + - The wrapper's `minWidth` calculation now uses `minSize` (not TanStack's 150px default) for grow columns, preventing the table from overflowing its container when columns would otherwise be sized larger than available space. +- Dependency updates: + - `requests` 2.32.5 → 2.33.0 (security patch; see Security section) + - `celery` 5.6.2 → 5.6.3 + - `torch` 2.10.0+cpu → 2.11.0+cpu + - `sentence-transformers` 5.2.3 → 5.3.0 + - `yt-dlp` 2026.3.13 → 2026.3.17 +- Docker base image cleanup: removed `python-is-python3`, `python3-pip`, and `streamlink` from the apt package list in `DispatcharrBase`. `python3-pip` and `streamlink` were pulling outdated system Python packages (e.g. `requests 2.31.0`, `cryptography 41.0.7`, `lxml 5.2.1`) into the system Python's site-packages despite the app running entirely in the uv-managed venv at `/dispatcharrpy`. `streamlink` is already installed in the venv via `pyproject.toml`. `python-is-python3` is unnecessary as `PATH` resolves bare `python` to the venv binary. +- M3U table **Max Streams** column now reflects the combined limit across all active profiles. When a playlist has multiple active profiles, the column displays their summed total (or ∞ if any profile is unlimited) and a hover tooltip lists each profile's individual limit by name. (Closes #816) +- Toggling an M3U profile's active state now immediately updates the playlist store (including the `playlists` array), so the **Max Streams** total in the M3U table reflects the change without a page reload. +- M3U account form: **Max Streams** field changed from a plain text input to a number input with increment/decrement controls, consistent with other integer fields. +- M3U account form: removed unused `useMantineTheme` import and `theme` variable. +- Moved `guideUtils.js` from `frontend/src/pages/` to `frontend/src/utils/` to be consistent with other utility modules (e.g. `networkUtils.js`). Updated all imports across `GuideRow.jsx`, `HourTimeline.jsx`, `ProgramDetailModal.jsx`, `RecordingCardUtils.js`, `Guide.jsx`, and related test files. +- Frontend cleanup: removed unused imports from `M3UGroupFilter`, `LiveGroupFilter`, and `VODCategoryFilter` (`Yup`, `M3UProfiles`, several unused Mantine components, dead `OptionWithTooltip` component, duplicate lucide-react imports, and `Divider` in `VODCategoryFilter`). No behaviour changes. +- Network Access settings: leaving a field blank no longer shows a validation error. The default CIDR range for that field is saved automatically and a "Defaults Restored" warning is displayed listing which fields were reset. (Closes #726) + +### Fixed + +- M3U profile URL rewriting now uses the `regex` module instead of `re` across all URL transform code paths (`url_utils.transform_url`, `core/views.py`, `vod_proxy/_transform_url`, `tasks.get_transformed_credentials`, and the WebSocket live-preview handler in `consumers.py`). The `regex` module natively accepts JavaScript/PCRE-style named capture groups (`(?...)`) without any conversion, eliminating the root cause of patterns that matched in the frontend live preview but failed on the backend with a `re.error`. As a further improvement, `regex` also supports variable-length lookbehind assertions (e.g. `(?<=a+)`), which `re` rejects with an error; patterns using these will now work correctly on the backend as well. Replace-pattern JS tokens are still normalised before calling `regex.sub`: `$` → `\g` and `$1`/`$2`/… → `\1`/`\2`/… (Python replacement syntax). Also fixed a bug in the WebSocket preview handler where a pattern error was incorrectly returning the search pattern string as the preview output instead of the original URL. (Fixes #1005) +- Web UI stream preview (`FloatingVideo`) was calling `mpegts.createPlayer()` with all `Config` options (e.g. `enableWorker`, `liveSync`, `headers`) merged into the first `MediaDataSource` argument. mpegts.js only reads `Config` from the optional second argument; unrecognised fields in the first are silently ignored. As a result all player configuration was effectively the library defaults — worker offloading was disabled, latency management had no effect, and the `Authorization: Bearer` header (required for user identification) was never sent. Fixed by splitting into the correct two-argument call. Both `liveBufferLatencyChasing` and `liveSync` have been disabled, eliminating playback-rate fluctuations that caused audible stuttering on live streams. SourceBuffer cleanup thresholds were also relaxed from 10s/5s to 120s/60s to prevent frequent SourceBuffer pauses. +- HTML named entities in XMLTV EPG files are now correctly preserved during lxml parsing. Some EPG providers (particularly French and other European sources) use HTML named entities like `é`, `î`, `ü` in channel names, program titles, and metadata. These are not valid XML entities — lxml 6.0.2 with `recover=True` silently drops them, causing characters to go missing (e.g., "Chaîne Télé" becomes "Chane Tl"). This is now fixed by injecting an XML `` internal subset declaring all 252 HTML 4 named entities directly into the byte stream that lxml reads, using a lightweight in-memory wrapper (`_PrependStream`) with zero disk I/O. libxml2 resolves the entities during its normal C-level parse pass — no Python-level preprocessing or temporary files are involved. The DOCTYPE block (~8 KB) is built once at module load from Python's stdlib `html.entities.name2codepoint` and reused for every parse. Files that already declare their own `` are passed through unchanged. (Closes #1095) — Thanks [@CodeBormen](https://github.com/CodeBormen) for helping with this! +- Duplicate recordings created when EPG sources refresh and re-evaluate series rules (Fixes #940) — Thanks [@CodeBormen](https://github.com/CodeBormen): + - **Program ID instability**: `parse_programs_for_source()` deletes and recreates all `ProgramData` rows with new auto-increment IDs on every EPG refresh. The dedup set used these IDs, so it never matched after a refresh. Deduplication now uses a stable `(tvg_id, start_time, end_time)` composite key sourced from `Recording.custom_properties.program`. + - **Secondary guard using wrong times**: The DB guard compared unadjusted program times against offset-adjusted `Recording.start_time`/`end_time`, so it never matched when any DVR pre/post offset was configured. It now queries `custom_properties__program__start_time/end_time` (the original, unadjusted program times stored at recording creation). + - **No concurrency guard**: Each EPG source refresh fired `evaluate_series_rules.delay()` independently. Concurrent tasks loaded the dedup set before others committed, allowing races. Evaluation is now serialized with `acquire_task_lock` (reusing the existing EPG task pattern). Gracefully degrades if Redis is unavailable — the primary and secondary dedup guards still protect. +- EPG refresh tasks (`refresh_epg_data`) were being killed mid-transaction on large EPG sources. The `soft_time_limit=1700s` introduced in v0.21.0 raised `SoftTimeLimitExceeded`, a subclass of `Exception`, which was swallowed by the existing `except Exception` handler in `parse_programs_for_source`, leaving the database in a partial state with no logged error. `soft_time_limit` has been removed from `refresh_epg_data` and `time_limit` raised to 14400s (4 hours) as a true last-resort ceiling; the existing `TaskLockRenewer` daemon thread continues to renew the Redis lock every 120s for legitimately long-running tasks. + ## [0.21.1] - 2026-03-18 ### Fixed diff --git a/apps/accounts/api_views.py b/apps/accounts/api_views.py index 76892929..c698f07a 100644 --- a/apps/accounts/api_views.py +++ b/apps/accounts/api_views.py @@ -287,11 +287,10 @@ class UserViewSet(viewsets.ModelViewSet): if request.method == "PATCH": ALLOWED_FIELDS = {"custom_properties", "first_name", "last_name", "email", "password"} disallowed = set(request.data.keys()) - ALLOWED_FIELDS - if disallowed: - return Response( - {"detail": f"Fields not allowed for self-update: {', '.join(disallowed)}"}, - status=400, - ) + + for key in disallowed: + request.data.pop(key, None) + serializer = UserSerializer(user, data=request.data, partial=True) serializer.is_valid(raise_exception=True) serializer.save() diff --git a/apps/accounts/migrations/0006_user_stream_limit.py b/apps/accounts/migrations/0006_user_stream_limit.py new file mode 100644 index 00000000..5714e1ef --- /dev/null +++ b/apps/accounts/migrations/0006_user_stream_limit.py @@ -0,0 +1,18 @@ +# Generated by Django 5.2.11 on 2026-03-19 13:46 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('accounts', '0005_alter_user_managers'), + ] + + operations = [ + migrations.AddField( + model_name='user', + name='stream_limit', + field=models.IntegerField(default=0), + ), + ] diff --git a/apps/accounts/models.py b/apps/accounts/models.py index e04d66ed..2a32f95a 100644 --- a/apps/accounts/models.py +++ b/apps/accounts/models.py @@ -30,6 +30,7 @@ class User(AbstractUser): user_level = models.IntegerField(default=UserLevel.STREAMER) custom_properties = models.JSONField(default=dict, blank=True, null=True) api_key = models.CharField(max_length=200, blank=True, null=True, db_index=True) + stream_limit = models.IntegerField(default=0) def __str__(self): return self.username diff --git a/apps/accounts/serializers.py b/apps/accounts/serializers.py index c90ffef4..8a9f609e 100644 --- a/apps/accounts/serializers.py +++ b/apps/accounts/serializers.py @@ -65,6 +65,7 @@ class UserSerializer(serializers.ModelSerializer): "channel_profiles", "custom_properties", "avatar_config", + "stream_limit", "is_staff", "is_superuser", "last_login", diff --git a/apps/backups/services.py b/apps/backups/services.py index b638e701..c70160e1 100644 --- a/apps/backups/services.py +++ b/apps/backups/services.py @@ -28,10 +28,36 @@ def _is_postgresql() -> bool: def _get_pg_env() -> dict: - """Get environment variables for PostgreSQL commands.""" + """Get environment variables for PostgreSQL commands. + + Includes PGPASSWORD for password auth and PGSSL* variables for TLS. + Reads TLS config from DATABASES['default']['OPTIONS'], which is + populated by settings.py when POSTGRES_SSL=true. + """ db_config = settings.DATABASES["default"] env = os.environ.copy() - env["PGPASSWORD"] = db_config.get("PASSWORD", "") + + password = db_config.get("PASSWORD", "") + if password: + env["PGPASSWORD"] = password + else: + env.pop("PGPASSWORD", None) + + # Propagate TLS configuration from Django OPTIONS to libpq env vars. + options = db_config.get("OPTIONS", {}) + _ssl_env_map = { + "sslmode": "PGSSLMODE", + "sslrootcert": "PGSSLROOTCERT", + "sslcert": "PGSSLCERT", + "sslkey": "PGSSLKEY", + } + # Always strip inherited PGSSL* vars first, then set only what is explicitly configured + for opt_key, env_key in _ssl_env_map.items(): + env.pop(env_key, None) + value = options.get(opt_key) + if value: + env[env_key] = value + return env diff --git a/apps/backups/tests.py b/apps/backups/tests.py index e5743623..bdea2be5 100644 --- a/apps/backups/tests.py +++ b/apps/backups/tests.py @@ -15,6 +15,96 @@ from . import services User = get_user_model() +class PgEnvTlsTestCase(TestCase): + """Test that _get_pg_env includes TLS and password env vars correctly.""" + databases = [] + + @patch('apps.backups.services.settings') + def test_pg_env_includes_ssl_vars_when_tls_enabled(self, mock_settings): + mock_settings.DATABASES = { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": "testdb", + "USER": "testuser", + "PASSWORD": "testpass", + "HOST": "localhost", + "PORT": 5432, + "OPTIONS": { + "sslmode": "verify-full", + "sslrootcert": "/certs/ca.crt", + "sslcert": "/certs/client.crt", + "sslkey": "/certs/client.key", + }, + } + } + env = services._get_pg_env() + self.assertEqual(env["PGSSLMODE"], "verify-full") + self.assertEqual(env["PGSSLROOTCERT"], "/certs/ca.crt") + self.assertEqual(env["PGSSLCERT"], "/certs/client.crt") + self.assertEqual(env["PGSSLKEY"], "/certs/client.key") + self.assertEqual(env["PGPASSWORD"], "testpass") + + @patch('apps.backups.services.settings') + def test_pg_env_no_ssl_vars_when_tls_disabled(self, mock_settings): + mock_settings.DATABASES = { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": "testdb", + "USER": "testuser", + "PASSWORD": "testpass", + "HOST": "localhost", + "PORT": 5432, + } + } + env = services._get_pg_env() + self.assertNotIn("PGSSLMODE", env) + self.assertNotIn("PGSSLROOTCERT", env) + self.assertNotIn("PGSSLCERT", env) + self.assertNotIn("PGSSLKEY", env) + self.assertEqual(env["PGPASSWORD"], "testpass") + + @patch('apps.backups.services.settings') + def test_pg_env_no_password_when_empty(self, mock_settings): + """Cert-only auth: PGPASSWORD must not be set when password is empty.""" + mock_settings.DATABASES = { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": "testdb", + "USER": "testuser", + "PASSWORD": "", + "HOST": "localhost", + "PORT": 5432, + "OPTIONS": {"sslmode": "verify-full"}, + } + } + env = services._get_pg_env() + self.assertNotIn("PGPASSWORD", env) + self.assertEqual(env["PGSSLMODE"], "verify-full") + + @patch('apps.backups.services.settings') + def test_pg_env_partial_ssl_options(self, mock_settings): + """Server-only TLS: only sslmode and CA cert, no client cert/key.""" + mock_settings.DATABASES = { + "default": { + "ENGINE": "django.db.backends.postgresql", + "NAME": "testdb", + "USER": "testuser", + "PASSWORD": "pass", + "HOST": "localhost", + "PORT": 5432, + "OPTIONS": { + "sslmode": "verify-ca", + "sslrootcert": "/certs/ca.crt", + }, + } + } + env = services._get_pg_env() + self.assertEqual(env["PGSSLMODE"], "verify-ca") + self.assertEqual(env["PGSSLROOTCERT"], "/certs/ca.crt") + self.assertNotIn("PGSSLCERT", env) + self.assertNotIn("PGSSLKEY", env) + + class BackupServicesTestCase(TestCase): """Test cases for backup services""" diff --git a/apps/channels/api_views.py b/apps/channels/api_views.py index 08283e73..268cc828 100644 --- a/apps/channels/api_views.py +++ b/apps/channels/api_views.py @@ -2674,7 +2674,49 @@ class RecordingViewSet(viewsets.ModelViewSet): recording_id = instance.pk channel_name = instance.channel.name - # Capture state before the DB row is deleted + # Attempt to close the DVR client connection for this channel if active + try: + channel_uuid = str(instance.channel.uuid) + # Lazy imports to avoid module overhead if proxy isn't used + from core.utils import RedisClient + from apps.proxy.ts_proxy.redis_keys import RedisKeys + from apps.proxy.ts_proxy.services.channel_service import ChannelService + + r = RedisClient.get_client() + if r: + client_set_key = RedisKeys.clients(channel_uuid) + client_ids = r.smembers(client_set_key) or [] + stopped = 0 + for cid in client_ids: + try: + meta_key = RedisKeys.client_metadata(channel_uuid, cid) + ua = r.hget(meta_key, "user_agent") + # Identify DVR recording client by its user agent + if ua and "Dispatcharr-DVR" in ua: + try: + ChannelService.stop_client(channel_uuid, cid) + stopped += 1 + except Exception as inner_e: + logger.debug(f"Failed to stop DVR client {cid} for channel {channel_uuid}: {inner_e}") + except Exception as inner: + logger.debug(f"Error while checking client metadata: {inner}") + if stopped: + logger.info(f"Stopped {stopped} DVR client(s) for channel {channel_uuid} due to recording cancellation") + # If no clients remain after stopping DVR clients, proactively stop the channel + try: + remaining = r.scard(client_set_key) or 0 + except Exception: + remaining = 0 + if remaining == 0: + try: + ChannelService.stop_channel(channel_uuid) + logger.info(f"Stopped channel {channel_uuid} (no clients remain)") + except Exception as sc_e: + logger.debug(f"Unable to stop channel {channel_uuid}: {sc_e}") + except Exception as e: + logger.debug(f"Unable to stop DVR clients for cancelled recording: {e}") + + # Capture paths before deletion cp = instance.custom_properties or {} rec_status = cp.get("status", "") file_path = cp.get("file_path") diff --git a/apps/channels/models.py b/apps/channels/models.py index 3bebd224..51629db3 100644 --- a/apps/channels/models.py +++ b/apps/channels/models.py @@ -204,7 +204,7 @@ class Stream(models.Model): return stream_profile - def get_stream(self): + def get_stream(self, requester=None): """ Finds an available profile for this stream and reserves a connection slot. @@ -400,6 +400,144 @@ class Channel(models.Model): return stream_profile + def _pick_channel_to_preempt( + self, + profile_id, + requester_level, + redis_client, + exclude_channel_ids=None, + cooldown_seconds=30, + ): + """ + Pick the lowest-impact channel to terminate on the given profile. + Returns: Optional[int] channel_id to preempt + """ + exclude_channel_ids = set(exclude_channel_ids or []) + candidates = [] + + # 1) Try to get active channel IDs for this profile from an index set if available + ch_set_key = f"ts_proxy:profile:{profile_id}:channels" + try: + ch_ids = { (int(x) if not isinstance(x, int) else x) for x in (redis_client.smembers(ch_set_key) or set()) } + except Exception: + ch_ids = set() + + logger.debug("Candidate channels for preemption:") + logger.debug(ch_ids) + + # 2) Fallback: scan metadata keys and filter by m3u_profile == profile_id + if not ch_ids: + cursor = 0 + pattern = "ts_proxy:channel:*:metadata" + while True: + cursor, keys = redis_client.scan(cursor=cursor, match=pattern, count=500) + if keys: + # Prefer HGET m3u_profile if metadata is a hash + pipe = redis_client.pipeline() + for k in keys: + pipe.hget(k, "m3u_profile") + prof_vals = pipe.execute() + for k, prof_val in zip(keys, prof_vals): + try: + pid = int(prof_val) if prof_val is not None else None + except Exception: + pid = None + + if pid == profile_id: + parts = k.split(":") # ts_proxy:channel:{id}:metadata + if len(parts) >= 4: + try: + ch_ids.add(int(parts[2])) + except Exception: + pass + if cursor == 0: + break + + logger.debug("Candidate channels for preemption:") + logger.debug(ch_ids) + + if not ch_ids: + return None + + # 3) Score candidates + for ch_id in ch_ids: + if ch_id in exclude_channel_ids: + continue + + # Skip if recently preempted + last_preempt_key = f"ts_proxy:channel:{ch_id}:last_preempt" + try: + last_preempt = float(redis_client.get(last_preempt_key) or 0.0) + except Exception: + last_preempt = 0.0 + if last_preempt and (time.time() - last_preempt) < cooldown_seconds: + continue + + # Clients and their levels + clients_key = f"ts_proxy:channel:{ch_id}:clients" + member_ids = list(redis_client.smembers(clients_key) or []) + viewer_count = len(member_ids) + max_viewer_level = 0 + if viewer_count: + pipe = redis_client.pipeline() + for cid in member_ids: + pipe.hget(f"ts_proxy:channel:{ch_id}:clients:{cid}", "user_level") + levels_raw = pipe.execute() + levels = [] + for lv in levels_raw: + try: + levels.append(int(lv or 0)) + except Exception: + levels.append(0) + max_viewer_level = max(levels or [0]) + + # Only preempt if requester strictly outranks this channel's viewers + if requester_level <= max_viewer_level: + continue + + # Metadata (protected/recording/started_at_ts) + meta_key = f"ts_proxy:channel:{ch_id}:metadata" + try: + protected, recording, started_at_ts = redis_client.hmget( + meta_key, "protected", "recording", "started_at_ts" + ) + except Exception: + protected = recording = started_at_ts = None + + protected = str(protected or "0") in ("1", "true", "True") + recording = str(recording or "0") in ("1", "true", "True") + if protected or recording: + continue + + try: + started_at_ts = float(started_at_ts) if started_at_ts is not None else None + except Exception: + started_at_ts = None + if started_at_ts is None: + started_at_ts = time.time() # treat unknown as newest + + # Score: lower is safer to terminate + has_viewers = 1 if viewer_count > 0 else 0 + score = (has_viewers, max_viewer_level, viewer_count, started_at_ts) + candidates.append((score, ch_id)) + + logger.debug("Candidate channels after scoring:") + logger.debug(candidates) + + if not candidates: + return None + + candidates.sort(key=lambda x: x[0]) + victim_id = candidates[0][1] + + # Mark preempt timestamp to avoid thrashing + try: + redis_client.set(f"ts_proxy:channel:{victim_id}:last_preempt", str(time.time()), ex=3600) + except Exception: + pass + + return victim_id + def _check_and_reserve_profile_slot(self, profile, redis_client): """ Atomically check and reserve a connection slot for the given profile. @@ -432,7 +570,7 @@ class Channel(models.Model): redis_client.decr(profile_connections_key) return (False, new_count - 1) - def get_stream(self): + def get_stream(self, requester=None): """ Finds an available stream for the requested channel and returns the selected stream and profile. @@ -513,6 +651,17 @@ class Channel(models.Model): None, ) # Return newly assigned stream and matched profile else: + # At capacity: try to preempt a lower-impact channel on this profile + victim_channel_id = self._pick_channel_to_preempt( + profile_id=profile.id, + requester_level=requester.user_level if requester else 100, + redis_client=redis_client, + exclude_channel_ids=None, + ) + if victim_channel_id: + logger.info(f"Preempting channel {victim_channel_id} for new stream on profile {profile.id}") + # return self.id, profile.id, victim_channel_id + # This profile is at max connections has_streams_but_maxed_out = True logger.debug( diff --git a/apps/channels/tasks.py b/apps/channels/tasks.py index b5969cd1..59122d9f 100755 --- a/apps/channels/tasks.py +++ b/apps/channels/tasks.py @@ -19,6 +19,7 @@ from rapidfuzz import fuzz from apps.channels.models import Channel from apps.epg.models import EPGData from core.models import CoreSettings +from core.utils import acquire_task_lock, release_task_lock from django.db import OperationalError, close_old_connections from channels.layers import get_channel_layer @@ -1070,12 +1071,39 @@ def match_single_channel_epg(channel_id): def evaluate_series_rules_impl(tvg_id: str | None = None): """Synchronous implementation of series rule evaluation; returns details for debugging.""" + result = {"scheduled": 0, "details": []} + + # Serialize all invocations to prevent concurrent evaluations from + # racing to create duplicate recordings (e.g. multiple EPG sources + # refreshing simultaneously each firing evaluate_series_rules.delay()). + # If Redis is unavailable, proceed without lock — the primary and + # secondary dedup guards still prevent duplicates. + lock_acquired = False + try: + lock_acquired = acquire_task_lock('evaluate_series_rules', 'all') + if not lock_acquired: + result["details"].append({"status": "skipped", "reason": "concurrent evaluation in progress"}) + return result + except (ConnectionError, OSError, AttributeError): + logger.warning("Could not acquire series rule evaluation lock (Redis unavailable), proceeding without lock") + + try: + return _evaluate_series_rules_locked(tvg_id, result) + finally: + if lock_acquired: + try: + release_task_lock('evaluate_series_rules', 'all') + except (ConnectionError, OSError, AttributeError): + logger.warning("Could not release series rule evaluation lock") + + +def _evaluate_series_rules_locked(tvg_id, result): + """Inner implementation of series rule evaluation, called under lock.""" from django.utils import timezone from apps.channels.models import Recording, Channel from apps.epg.models import EPGData, ProgramData rules = CoreSettings.get_dvr_series_rules() - result = {"scheduled": 0, "details": []} if not isinstance(rules, list) or not rules: return result @@ -1089,14 +1117,23 @@ def evaluate_series_rules_impl(tvg_id: str | None = None): now = timezone.now() horizon = now + timedelta(days=7) - # Preload existing recordings' program ids to avoid duplicates - existing_program_ids = set() - for rec in Recording.objects.all().only("custom_properties"): + # Preload existing recordings keyed by stable program attributes that + # survive EPG refreshes (tvg_id + original start/end times stored in + # custom_properties). ProgramData.id changes on every EPG refresh so + # it cannot be used for deduplication. Only load future recordings + # to bound the set size — past recordings cannot collide with newly + # scheduled future programs. + existing_program_keys = set() + for cp in Recording.objects.filter( + end_time__gte=now, + ).values_list("custom_properties", flat=True): try: - pid = rec.custom_properties.get("program", {}).get("id") if rec.custom_properties else None - if pid is not None: - # Normalize to string for consistent comparisons - existing_program_ids.add(str(pid)) + prog_data = (cp or {}).get("program", {}) + tvg_id_val = prog_data.get("tvg_id") + st = prog_data.get("start_time") + et = prog_data.get("end_time") + if tvg_id_val and st and et: + existing_program_keys.add((str(tvg_id_val), str(st), str(et))) except Exception: continue @@ -1191,17 +1228,21 @@ def evaluate_series_rules_impl(tvg_id: str | None = None): created_here = 0 for prog in unique_programs: try: - # Skip if already scheduled by program id - if str(prog.id) in existing_program_ids: + # Skip if a recording already exists for this exact airing + # (keyed by tvg_id + original program times, which are stable + # across EPG refreshes unlike ProgramData.id). + prog_key = (str(prog.tvg_id), prog.start_time.isoformat(), prog.end_time.isoformat()) + if prog_key in existing_program_keys: continue - # Extra guard: skip if a recording exists for the same channel + timeslot + # Extra guard: DB query using the same stable attributes + # stored in custom_properties (unadjusted program times, + # not offset-adjusted Recording.start_time/end_time). try: - from django.db.models import Q if Recording.objects.filter( - channel=channel, - start_time=prog.start_time, - end_time=prog.end_time, - ).filter(Q(custom_properties__program__id=prog.id) | Q(custom_properties__program__title=prog.title)).exists(): + custom_properties__program__tvg_id=prog.tvg_id, + custom_properties__program__start_time=prog.start_time.isoformat(), + custom_properties__program__end_time=prog.end_time.isoformat(), + ).exists(): continue except Exception: continue # already scheduled/recorded @@ -1245,7 +1286,7 @@ def evaluate_series_rules_impl(tvg_id: str | None = None): } }, ) - existing_program_ids.add(str(prog.id)) + existing_program_keys.add(prog_key) created_here += 1 try: prefetch_recording_artwork.apply_async(args=[rec.id], countdown=1) @@ -2452,15 +2493,12 @@ def run_recording(recording_id, channel_id, start_time_str, end_time_str): metadata_key = RedisKeys.channel_metadata(str(channel.uuid)) md = r.hgetall(metadata_key) if md: - def _gv(bkey): - return md.get(bkey.encode('utf-8')) - def _d(bkey, cast=str): - v = _gv(bkey) + v = md.get(bkey) try: if v is None: return None - s = v.decode('utf-8') + s = v return cast(s) if cast is not str else s except Exception: return None diff --git a/apps/channels/tests/test_series_rule_dedup.py b/apps/channels/tests/test_series_rule_dedup.py new file mode 100644 index 00000000..33aff695 --- /dev/null +++ b/apps/channels/tests/test_series_rule_dedup.py @@ -0,0 +1,718 @@ +"""Tests for series rule evaluation deduplication. + +Unit tests verify the dedup logic in evaluate_series_rules_impl. +Integration tests exercise the full path: EPG refresh → series rule +evaluation → Recording creation → post_save signal chain. +""" +from datetime import timedelta +from unittest.mock import patch, MagicMock + +from django.test import TestCase +from django.utils import timezone + +from apps.channels.models import Channel, Recording +from apps.epg.models import EPGSource, EPGData, ProgramData +from core.models import CoreSettings + + +def _set_series_rules(rules): + """Helper to store series rules in CoreSettings.""" + CoreSettings.set_dvr_series_rules(rules) + + +def _set_dvr_offsets(pre_min=0, post_min=0): + """Helper to store DVR pre/post offsets.""" + CoreSettings._update_group("dvr_settings", "DVR Settings", { + "pre_offset_minutes": pre_min, + "post_offset_minutes": post_min, + }) + + +class SeriesRuleDedupBaseTestCase(TestCase): + """Shared setup for series rule dedup tests.""" + + def setUp(self): + self.now = timezone.now() + self.epg_source = EPGSource.objects.create( + name="Test EPG", source_type="xmltv" + ) + self.epg = EPGData.objects.create( + tvg_id="test.channel.1", + name="Test Channel EPG", + epg_source=self.epg_source, + ) + self.channel = Channel.objects.create( + channel_number=1, name="Test Channel", epg_data=self.epg + ) + + _set_series_rules([{ + "tvg_id": "test.channel.1", + "mode": "all", + "title": "Test Show", + }]) + _set_dvr_offsets(pre_min=0, post_min=0) + + def _create_program(self, hours_from_now=1, title="Test Show", + sub_title="Episode 1", tvg_id="test.channel.1"): + """Create a ProgramData at the given offset.""" + start = self.now + timedelta(hours=hours_from_now) + end = start + timedelta(hours=1) + return ProgramData.objects.create( + epg=self.epg, + tvg_id=tvg_id, + start_time=start, + end_time=end, + title=title, + sub_title=sub_title, + ) + + def _simulate_epg_refresh(self, programs_data): + """Delete all ProgramData and recreate with new IDs (simulates EPG refresh).""" + ProgramData.objects.filter(epg=self.epg).delete() + new_programs = [] + for data in programs_data: + prog = ProgramData.objects.create(epg=self.epg, **data) + new_programs.append(prog) + return new_programs + + def _program_data_for_refresh(self, prog): + """Build the dict needed by _simulate_epg_refresh from a ProgramData.""" + return { + "tvg_id": prog.tvg_id, + "start_time": prog.start_time, + "end_time": prog.end_time, + "title": prog.title, + "sub_title": prog.sub_title, + } + + +# --------------------------------------------------------------------------- +# Unit tests: dedup logic in evaluate_series_rules_impl +# --------------------------------------------------------------------------- + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class ProgramIdStabilityTests(SeriesRuleDedupBaseTestCase): + """Verify dedup works after EPG refresh changes ProgramData IDs.""" + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_no_duplicate_after_epg_refresh(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Same program should not be recorded twice after EPG refresh.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2) + old_id = prog.id + result1 = evaluate_series_rules_impl() + self.assertEqual(result1["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 1) + + new_programs = self._simulate_epg_refresh( + [self._program_data_for_refresh(prog)] + ) + self.assertNotEqual(old_id, new_programs[0].id) + + result2 = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + self.assertEqual(result2["scheduled"], 0) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_no_duplicate_with_offsets_after_refresh(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Dedup works when DVR offsets shift Recording times away from program times.""" + from apps.channels.tasks import evaluate_series_rules_impl + + _set_dvr_offsets(pre_min=5, post_min=5) + prog = self._create_program(hours_from_now=2) + result1 = evaluate_series_rules_impl() + self.assertEqual(result1["scheduled"], 1) + + rec = Recording.objects.first() + self.assertEqual(rec.start_time, prog.start_time - timedelta(minutes=5)) + self.assertEqual(rec.end_time, prog.end_time + timedelta(minutes=5)) + + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + result2 = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_different_episodes_still_recorded(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Different episodes on the same channel should each get a recording.""" + from apps.channels.tasks import evaluate_series_rules_impl + + self._create_program(hours_from_now=2, sub_title="Episode 1") + self._create_program(hours_from_now=4, sub_title="Episode 2") + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 2) + self.assertEqual(Recording.objects.count(), 2) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_new_episode_after_refresh_is_recorded(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """A genuinely new episode appearing after EPG refresh should be recorded.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2, sub_title="Episode 1") + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + self._simulate_epg_refresh([ + self._program_data_for_refresh(prog), + { + "tvg_id": "test.channel.1", + "start_time": prog.end_time, + "end_time": prog.end_time + timedelta(hours=1), + "title": "Test Show", + "sub_title": "Episode 2", + }, + ]) + + result2 = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 2) + self.assertEqual(result2["scheduled"], 1) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_multiple_epg_refreshes_no_duplicates(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Multiple consecutive EPG refreshes should not accumulate duplicates.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + for _ in range(5): + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + evaluate_series_rules_impl() + + self.assertEqual(Recording.objects.count(), 1) + + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class ConcurrencyGuardTests(SeriesRuleDedupBaseTestCase): + """Verify the task lock prevents concurrent evaluation.""" + + def test_lock_acquired_and_released(self, mock_schedule, mock_artwork): + """evaluate_series_rules_impl acquires and releases the task lock.""" + from apps.channels.tasks import evaluate_series_rules_impl + + self._create_program(hours_from_now=2) + + with patch("apps.channels.tasks.acquire_task_lock", return_value=True) as mock_lock, \ + patch("apps.channels.tasks.release_task_lock") as mock_release: + evaluate_series_rules_impl() + mock_lock.assert_called_once_with('evaluate_series_rules', 'all') + mock_release.assert_called_once_with('evaluate_series_rules', 'all') + + def test_skips_when_lock_held(self, mock_schedule, mock_artwork): + """Returns early with skip reason when lock is already held.""" + from apps.channels.tasks import evaluate_series_rules_impl + + self._create_program(hours_from_now=2) + + with patch("apps.channels.tasks.acquire_task_lock", return_value=False): + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 0) + self.assertTrue( + any(d.get("reason") == "concurrent evaluation in progress" + for d in result["details"]), + ) + self.assertEqual(Recording.objects.count(), 0) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_lock_released_on_exception(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Lock is released even if the inner implementation raises.""" + from apps.channels.tasks import evaluate_series_rules_impl + + with patch("apps.channels.tasks._evaluate_series_rules_locked", + side_effect=RuntimeError("test error")): + with self.assertRaises(RuntimeError): + evaluate_series_rules_impl() + mock_release.assert_called_once_with('evaluate_series_rules', 'all') + + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class SecondaryGuardTests(SeriesRuleDedupBaseTestCase): + """Verify the secondary DB guard uses stable program attributes.""" + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_secondary_guard_catches_duplicate_with_offsets(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Secondary guard works with stale program IDs and DVR offsets.""" + from apps.channels.tasks import evaluate_series_rules_impl + + _set_dvr_offsets(pre_min=10, post_min=10) + prog = self._create_program(hours_from_now=2) + + # Pre-existing recording with a stale program ID (from previous EPG refresh) + Recording.objects.create( + channel=self.channel, + start_time=prog.start_time - timedelta(minutes=10), + end_time=prog.end_time + timedelta(minutes=10), + custom_properties={ + "program": { + "id": 99999, + "tvg_id": prog.tvg_id, + "title": prog.title, + "start_time": prog.start_time.isoformat(), + "end_time": prog.end_time.isoformat(), + } + }, + ) + + result = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + self.assertEqual(result["scheduled"], 0) + + +# --------------------------------------------------------------------------- +# Integration tests: full path from EPG refresh through recording creation +# --------------------------------------------------------------------------- + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class IntegrationEPGRefreshTests(SeriesRuleDedupBaseTestCase): + """End-to-end tests simulating the EPG refresh → evaluate → record flow. + + These exercise the full signal chain: evaluate_series_rules_impl creates + a Recording, the post_save signal fires schedule_recording_task, and + subsequent evaluations (after EPG refresh) must not create duplicates. + """ + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_single_episode_no_duplicates(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Simulate: create rule → evaluate → EPG refresh → re-evaluate. + + The full recording lifecycle must result in exactly 1 recording. + """ + from apps.channels.tasks import evaluate_series_rules_impl + + # Initial EPG data + prog = self._create_program(hours_from_now=2, sub_title="Pilot") + + # First evaluation creates the recording + result1 = evaluate_series_rules_impl() + self.assertEqual(result1["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 1) + + # Verify the recording was created with correct program metadata + rec = Recording.objects.first() + self.assertEqual(rec.custom_properties["program"]["tvg_id"], "test.channel.1") + self.assertEqual(rec.custom_properties["program"]["title"], "Test Show") + self.assertEqual( + rec.custom_properties["program"]["start_time"], + prog.start_time.isoformat() + ) + + # Verify the post_save signal scheduled a task + mock_schedule.assert_called() + initial_schedule_count = mock_schedule.call_count + + # Simulate EPG refresh (programs get new DB IDs) + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + + # Re-evaluate after refresh (this is what EPG refresh triggers) + result2 = evaluate_series_rules_impl() + self.assertEqual(result2["scheduled"], 0) + self.assertEqual(Recording.objects.count(), 1) + + # No additional task scheduling should have occurred + self.assertEqual(mock_schedule.call_count, initial_schedule_count) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_with_offsets_no_duplicates(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Full flow with DVR offsets: recording times differ from program times.""" + from apps.channels.tasks import evaluate_series_rules_impl + + _set_dvr_offsets(pre_min=5, post_min=10) + prog = self._create_program(hours_from_now=3, sub_title="Episode 1") + + result1 = evaluate_series_rules_impl() + self.assertEqual(result1["scheduled"], 1) + + rec = Recording.objects.first() + # Verify offset-adjusted recording times + self.assertEqual(rec.start_time, prog.start_time - timedelta(minutes=5)) + self.assertEqual(rec.end_time, prog.end_time + timedelta(minutes=10)) + # Verify original (unadjusted) program times in custom_properties + self.assertEqual( + rec.custom_properties["program"]["start_time"], + prog.start_time.isoformat() + ) + self.assertEqual( + rec.custom_properties["program"]["end_time"], + prog.end_time.isoformat() + ) + + # EPG refresh + re-evaluate + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + result2 = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + self.assertEqual(result2["scheduled"], 0) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_multiple_episodes_across_refreshes(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """New episodes appear across multiple EPG refreshes; each recorded once.""" + from apps.channels.tasks import evaluate_series_rules_impl + + ep1 = self._create_program(hours_from_now=2, sub_title="Episode 1") + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + # EPG refresh adds episode 2 alongside episode 1 + ep1_data = self._program_data_for_refresh(ep1) + ep2_start = ep1.end_time + ep2_data = { + "tvg_id": "test.channel.1", + "start_time": ep2_start, + "end_time": ep2_start + timedelta(hours=1), + "title": "Test Show", + "sub_title": "Episode 2", + } + self._simulate_epg_refresh([ep1_data, ep2_data]) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 2) + + # Another EPG refresh adds episode 3 + ep3_start = ep2_start + timedelta(hours=1) + ep3_data = { + "tvg_id": "test.channel.1", + "start_time": ep3_start, + "end_time": ep3_start + timedelta(hours=1), + "title": "Test Show", + "sub_title": "Episode 3", + } + self._simulate_epg_refresh([ep1_data, ep2_data, ep3_data]) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 3) + + # Final EPG refresh with no new episodes — count must stay at 3 + self._simulate_epg_refresh([ep1_data, ep2_data, ep3_data]) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 3) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_multiple_series_rules(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Multiple series rules on different channels, each evaluated correctly.""" + from apps.channels.tasks import evaluate_series_rules_impl + + # Second channel with its own EPG + epg2 = EPGData.objects.create( + tvg_id="test.channel.2", + name="Channel 2 EPG", + epg_source=self.epg_source, + ) + channel2 = Channel.objects.create( + channel_number=2, name="Test Channel 2", epg_data=epg2 + ) + + _set_series_rules([ + {"tvg_id": "test.channel.1", "mode": "all", "title": "Show A"}, + {"tvg_id": "test.channel.2", "mode": "all", "title": "Show B"}, + ]) + + # Programs on both channels + start1 = self.now + timedelta(hours=2) + prog1 = ProgramData.objects.create( + epg=self.epg, tvg_id="test.channel.1", + start_time=start1, end_time=start1 + timedelta(hours=1), + title="Show A", sub_title="Episode 1", + ) + start2 = self.now + timedelta(hours=3) + prog2 = ProgramData.objects.create( + epg=epg2, tvg_id="test.channel.2", + start_time=start2, end_time=start2 + timedelta(hours=1), + title="Show B", sub_title="Episode 1", + ) + + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 2) + self.assertEqual(Recording.objects.filter(channel=self.channel).count(), 1) + self.assertEqual(Recording.objects.filter(channel=channel2).count(), 1) + + # EPG refresh for both channels + ProgramData.objects.filter(epg=self.epg).delete() + ProgramData.objects.filter(epg=epg2).delete() + ProgramData.objects.create( + epg=self.epg, tvg_id="test.channel.1", + start_time=start1, end_time=start1 + timedelta(hours=1), + title="Show A", sub_title="Episode 1", + ) + ProgramData.objects.create( + epg=epg2, tvg_id="test.channel.2", + start_time=start2, end_time=start2 + timedelta(hours=1), + title="Show B", sub_title="Episode 1", + ) + + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 2, + "No duplicates across multiple series rules after EPG refresh") + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_rapid_epg_refreshes_simulate_user_report( + self, mock_release, mock_lock, mock_schedule, mock_artwork + ): + """Reproduce the user-reported scenario: series rule + multiple EPG refreshes + causing count to balloon from 6 to 25 and 5 simultaneous recordings. + + Simulates 6 episodes with 5 EPG refreshes (each assigning new ProgramData IDs). + """ + from apps.channels.tasks import evaluate_series_rules_impl + + # Create 6 episodes (the user had "next of 6") + episodes = [] + for i in range(6): + start = self.now + timedelta(hours=2 + i * 2) + episodes.append({ + "tvg_id": "test.channel.1", + "start_time": start, + "end_time": start + timedelta(hours=1), + "title": "Test Show", + "sub_title": f"Episode {i + 1}", + }) + + # Create initial ProgramData + for ep in episodes: + ProgramData.objects.create(epg=self.epg, **ep) + + # First evaluation: should create exactly 6 recordings + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 6) + + # Simulate 5 EPG refreshes (the user saw count balloon to 25) + for refresh_num in range(5): + self._simulate_epg_refresh(episodes) + result = evaluate_series_rules_impl() + self.assertEqual( + Recording.objects.count(), 6, + f"After EPG refresh #{refresh_num + 1}, expected 6 recordings " + f"but got {Recording.objects.count()}" + ) + self.assertEqual(result["scheduled"], 0) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_recording_survives_program_removal_and_readd( + self, mock_release, mock_lock, mock_schedule, mock_artwork + ): + """Program temporarily disappears from EPG then reappears — no duplicate.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2, sub_title="Episode 1") + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + # EPG refresh removes the program entirely + self._simulate_epg_refresh([]) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1, + "Existing recording preserved when program disappears from EPG") + + # EPG refresh adds the program back (new ID) + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1, + "No duplicate when program reappears with new ID") + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_celery_task_wrapper_calls_impl(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """The @shared_task evaluate_series_rules delegates to _impl correctly.""" + from apps.channels.tasks import evaluate_series_rules + + self._create_program(hours_from_now=2) + result = evaluate_series_rules() + self.assertEqual(result["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 1) + + # Call again (simulating a second EPG refresh trigger) + result2 = evaluate_series_rules() + self.assertEqual(result2["scheduled"], 0) + self.assertEqual(Recording.objects.count(), 1) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_tvg_id_scoped_evaluation(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Scoped evaluation (tvg_id parameter) still prevents duplicates.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2) + result1 = evaluate_series_rules_impl(tvg_id="test.channel.1") + self.assertEqual(result1["scheduled"], 1) + + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + result2 = evaluate_series_rules_impl(tvg_id="test.channel.1") + self.assertEqual(result2["scheduled"], 0) + self.assertEqual(Recording.objects.count(), 1) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_full_flow_offset_change_between_refreshes(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Changing DVR offsets between EPG refreshes doesn't create duplicates. + + Even though Recording.start_time/end_time change when offsets change, + the dedup key uses the original program times from custom_properties. + """ + from apps.channels.tasks import evaluate_series_rules_impl + + _set_dvr_offsets(pre_min=5, post_min=5) + prog = self._create_program(hours_from_now=2) + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + rec = Recording.objects.first() + original_start = rec.start_time + original_end = rec.end_time + + # Change offsets + _set_dvr_offsets(pre_min=10, post_min=15) + + # EPG refresh + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + result = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1, + "Changing offsets between refreshes should not create duplicates") + self.assertEqual(result["scheduled"], 0) + + +# --------------------------------------------------------------------------- +# Edge case tests: Redis unavailability, non-series recordings, robustness +# --------------------------------------------------------------------------- + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class RedisUnavailabilityTests(SeriesRuleDedupBaseTestCase): + """Verify evaluation works when Redis is unavailable (lock cannot be acquired).""" + + def test_proceeds_when_redis_down(self, mock_schedule, mock_artwork): + """Evaluation succeeds (with dedup guards) when Redis raises on lock acquire.""" + from apps.channels.tasks import evaluate_series_rules_impl + + self._create_program(hours_from_now=2) + + with patch("apps.channels.tasks.acquire_task_lock", + side_effect=ConnectionError("Redis unavailable")): + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 1) + + def test_dedup_still_works_without_lock(self, mock_schedule, mock_artwork): + """Dedup guards prevent duplicates even when the lock is unavailable.""" + from apps.channels.tasks import evaluate_series_rules_impl + + prog = self._create_program(hours_from_now=2) + + # First call: Redis down, proceeds without lock + with patch("apps.channels.tasks.acquire_task_lock", + side_effect=ConnectionError("Redis unavailable")): + evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1) + + # EPG refresh + self._simulate_epg_refresh([self._program_data_for_refresh(prog)]) + + # Second call: Redis still down + with patch("apps.channels.tasks.acquire_task_lock", + side_effect=ConnectionError("Redis unavailable")): + result = evaluate_series_rules_impl() + self.assertEqual(Recording.objects.count(), 1, + "Dedup guards prevent duplicates even without lock") + self.assertEqual(result["scheduled"], 0) + + def test_lock_not_released_when_not_acquired(self, mock_schedule, mock_artwork): + """release_task_lock is not called if acquire raised an exception.""" + from apps.channels.tasks import evaluate_series_rules_impl + + self._create_program(hours_from_now=2) + + with patch("apps.channels.tasks.acquire_task_lock", + side_effect=ConnectionError("Redis unavailable")), \ + patch("apps.channels.tasks.release_task_lock") as mock_release: + evaluate_series_rules_impl() + mock_release.assert_not_called() + + +@patch("apps.channels.tasks.prefetch_recording_artwork") +@patch("apps.channels.signals.schedule_recording_task", return_value="mock-task-id") +class NonSeriesRecordingTests(SeriesRuleDedupBaseTestCase): + """Verify non-series recordings don't interfere with series rule dedup.""" + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_manual_recording_without_program_data_ignored(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Recordings without custom_properties.program are skipped by dedup key builder.""" + from apps.channels.tasks import evaluate_series_rules_impl + + # Manual recording with no program metadata + Recording.objects.create( + channel=self.channel, + start_time=self.now + timedelta(hours=2), + end_time=self.now + timedelta(hours=3), + custom_properties={}, + ) + + prog = self._create_program(hours_from_now=2) + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 2) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_recurring_rule_recording_does_not_interfere(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Recordings from recurring rules (custom_properties.rule) don't block series rules.""" + from apps.channels.tasks import evaluate_series_rules_impl + + Recording.objects.create( + channel=self.channel, + start_time=self.now + timedelta(hours=2), + end_time=self.now + timedelta(hours=3), + custom_properties={"rule": {"id": 1, "name": "Daily News"}}, + ) + + prog = self._create_program(hours_from_now=2) + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 1) + self.assertEqual(Recording.objects.count(), 2) + + @patch("apps.channels.tasks.acquire_task_lock", return_value=True) + @patch("apps.channels.tasks.release_task_lock") + def test_recording_with_null_custom_properties_ignored(self, mock_release, mock_lock, + mock_schedule, mock_artwork): + """Recordings with None custom_properties don't crash the dedup key builder.""" + from apps.channels.tasks import evaluate_series_rules_impl + + Recording.objects.create( + channel=self.channel, + start_time=self.now + timedelta(hours=2), + end_time=self.now + timedelta(hours=3), + custom_properties=None, + ) + + prog = self._create_program(hours_from_now=2) + result = evaluate_series_rules_impl() + self.assertEqual(result["scheduled"], 1) diff --git a/apps/epg/tasks.py b/apps/epg/tasks.py index f4c85256..0d05552f 100644 --- a/apps/epg/tasks.py +++ b/apps/epg/tasks.py @@ -2,6 +2,7 @@ import logging import gzip +import html.entities import os import uuid import requests @@ -15,7 +16,7 @@ import zipfile from celery import shared_task from django.conf import settings -from django.db import transaction +from django.db import connection, transaction from django.utils import timezone from apps.channels.models import Channel from core.models import UserAgent, CoreSettings @@ -28,6 +29,103 @@ from core.utils import acquire_task_lock, release_task_lock, TaskLockRenewer, se logger = logging.getLogger(__name__) +# DOCTYPE internal subset for XMLTV files. Declares all 252 HTML 4 named +# entities so lxml/libxml2 can resolve references like é correctly +# instead of silently dropping them in recovery mode. +# The 5 XML-predefined entities (amp, lt, gt, quot, apos) are always +# recognised by the XML spec and must not be redeclared. +_XML_ENTITIES = frozenset({'amp', 'lt', 'gt', 'quot', 'apos'}) + + +def _build_html_entity_doctype() -> bytes: + """Build a DOCTYPE internal subset declaring all HTML 4 named entities.""" + lines = [b'\n'.encode('ascii')) + lines.append(b']>\n') + return b''.join(lines) + + +_HTML_ENTITY_DOCTYPE = _build_html_entity_doctype() + + +class _PrependStream: + """Wraps an open binary file and prepends a bytes prefix to its content. + + Used by _open_xmltv_file to inject a DOCTYPE entity block before the + file content reaches lxml's iterparse, with zero disk I/O. + """ + + __slots__ = ('_prefix', '_prefix_pos', '_file') + + def __init__(self, prefix: bytes, file_obj): + self._prefix = prefix + self._prefix_pos = 0 + self._file = file_obj + + def read(self, size=-1): + prefix_len = len(self._prefix) + if self._prefix_pos >= prefix_len: + return self._file.read(size) + remaining = prefix_len - self._prefix_pos + if size < 0: + chunk = self._prefix[self._prefix_pos:] + self._file.read() + self._prefix_pos = prefix_len + return chunk + if size <= remaining: + chunk = self._prefix[self._prefix_pos:self._prefix_pos + size] + self._prefix_pos += size + return chunk + chunk = self._prefix[self._prefix_pos:] + self._prefix_pos = prefix_len + return chunk + self._file.read(size - remaining) + + def close(self): + self._file.close() + + def __enter__(self): + return self + + def __exit__(self, *_): + self.close() + + +def _open_xmltv_file(file_path: str): + """Open an XMLTV file for lxml iterparse, injecting an HTML entity DOCTYPE. + + Prepends a block that declares all 252 HTML 4 named + entities so lxml/libxml2 resolves references like é correctly + instead of silently dropping them in recovery mode. This involves zero + disk I/O — the DOCTYPE is streamed in-memory before the file content. + + If the file already contains a declaration the file is returned + unchanged; a second DOCTYPE would be invalid XML. + + The caller is responsible for closing the returned object. + """ + f = open(file_path, 'rb') + start = f.read(512) + + # Do not inject if the file already declares a DOCTYPE. + if b'') + if decl_end >= 0: + xml_decl = start[:decl_end + 2] + f.seek(decl_end + 2) + return _PrependStream(xml_decl + b'\n' + _HTML_ENTITY_DOCTYPE, f) + + # No XML declaration — insert DOCTYPE at the very start of the file. + f.seek(0) + return _PrependStream(_HTML_ENTITY_DOCTYPE, f) + def validate_icon_url_fast(icon_url, max_length=None): """ @@ -146,7 +244,7 @@ def refresh_all_epg_data(): return "EPG data refreshed." -@shared_task(time_limit=1800, soft_time_limit=1700) +@shared_task(time_limit=14400) def refresh_epg_data(source_id): if not acquire_task_lock('refresh_epg_data', source_id): logger.debug(f"EPG refresh for {source_id} already running") @@ -397,42 +495,41 @@ def fetch_xmltv(source): # Download to temporary file with open(temp_download_path, 'wb') as f: - for chunk in response.iter_content(chunk_size=16384): # Increased chunk size for better performance - if chunk: - f.write(chunk) + for chunk in response.iter_content(chunk_size=16384): + f.write(chunk) - downloaded += len(chunk) - elapsed_time = time.time() - start_time + downloaded += len(chunk) + elapsed_time = time.time() - start_time - # Calculate download speed in KB/s - speed = downloaded / elapsed_time / 1024 if elapsed_time > 0 else 0 + # Calculate download speed in KB/s + speed = downloaded / elapsed_time / 1024 if elapsed_time > 0 else 0 - # Calculate progress percentage - if total_size and total_size > 0: - progress = min(100, int((downloaded / total_size) * 100)) - else: - # If no content length header, estimate progress - progress = min(95, int((downloaded / (10 * 1024 * 1024)) * 100)) # Assume 10MB if unknown + # Calculate progress percentage + if total_size and total_size > 0: + progress = min(100, int((downloaded / total_size) * 100)) + else: + # If no content length header, estimate progress + progress = min(95, int((downloaded / (10 * 1024 * 1024)) * 100)) # Assume 10MB if unknown - # Time remaining (in seconds) - time_remaining = (total_size - downloaded) / (speed * 1024) if speed > 0 and total_size > 0 else 0 + # Time remaining (in seconds) + time_remaining = (total_size - downloaded) / (speed * 1024) if speed > 0 and total_size > 0 else 0 - # Only send updates at specified intervals to avoid flooding - current_time = time.time() - if current_time - last_update_time >= update_interval and progress > 0: - last_update_time = current_time - send_epg_update( - source.id, - "downloading", - progress, - speed=round(speed, 2), - elapsed_time=round(elapsed_time, 1), - time_remaining=round(time_remaining, 1), - downloaded=f"{downloaded / (1024 * 1024):.2f} MB" - ) + # Only send updates at specified intervals to avoid flooding + current_time = time.time() + if current_time - last_update_time >= update_interval and progress > 0: + last_update_time = current_time + send_epg_update( + source.id, + "downloading", + progress, + speed=round(speed, 2), + elapsed_time=round(elapsed_time, 1), + time_remaining=round(time_remaining, 1), + downloaded=f"{downloaded / (1024 * 1024):.2f} MB" + ) - # Explicitly delete the chunk to free memory immediately - del chunk + # Explicitly delete the chunk to free memory immediately + del chunk # Send completion notification send_epg_update(source.id, "downloading", 100) @@ -526,6 +623,7 @@ def fetch_xmltv(source): source.save(update_fields=['status']) logger.info(f"Cached EPG file saved to {source.file_path}") + return True except requests.exceptions.HTTPError as e: @@ -888,7 +986,7 @@ def parse_channels_only(source): # Open the file - no need to check file type since it's always XML now logger.debug(f"Opening file for channel parsing: {file_path}") - source_file = open(file_path, 'rb') + source_file = _open_xmltv_file(file_path) if process: logger.debug(f"[parse_channels_only] Memory after opening file: {process.memory_info().rss / 1024 / 1024:.2f} MB") @@ -1271,7 +1369,7 @@ def parse_programs_for_tvg_id(epg_id): try: # Open the file directly - no need to check compression logger.debug(f"Opening file for parsing: {file_path}") - source_file = open(file_path, 'rb') + source_file = _open_xmltv_file(file_path) # Stream parse the file using lxml's iterparse program_parser = etree.iterparse(source_file, events=('end',), tag='programme', remove_blank_text=True, recover=True) @@ -1544,7 +1642,7 @@ def parse_programs_for_source(epg_source, tvg_id=None): try: logger.debug(f"Opening file for single-pass parsing: {file_path}") - source_file = open(file_path, 'rb') + source_file = _open_xmltv_file(file_path) # Stream parse the file using lxml's iterparse program_parser = etree.iterparse(source_file, events=('end',), tag='programme', remove_blank_text=True, recover=True) @@ -1657,6 +1755,10 @@ def parse_programs_for_source(epg_source, tvg_id=None): batch_size = 1000 try: with transaction.atomic(): + # Kill any individual statement that hangs longer than 10 minutes. + # SET LOCAL automatically resets when this transaction ends (commit or rollback). + with connection.cursor() as cursor: + cursor.execute("SET LOCAL statement_timeout = '10min'") # Delete existing programs for mapped EPGs deleted_count = ProgramData.objects.filter(epg_id__in=mapped_epg_ids).delete()[0] logger.debug(f"Deleted {deleted_count} existing programs") diff --git a/apps/epg/tests/test_entity_resolution.py b/apps/epg/tests/test_entity_resolution.py new file mode 100644 index 00000000..353d9b51 --- /dev/null +++ b/apps/epg/tests/test_entity_resolution.py @@ -0,0 +1,195 @@ +import os +import tempfile + +from django.test import TestCase + +from apps.epg.tasks import ( + _NAMED_ENTITY_RE, + _detect_xml_encoding, + _replace_html_entity, + _resolve_html_entities, +) + + +class ReplaceHtmlEntityTests(TestCase): + """Tests for the regex callback that resolves individual HTML entities.""" + + def _sub(self, text): + return _NAMED_ENTITY_RE.sub(_replace_html_entity, text) + + def test_french_accented(self): + self.assertEqual(self._sub("Chaîne Télé"), "Chaîne Télé") + + def test_german_umlauts(self): + self.assertEqual(self._sub("München Übersicht ß"), "München Übersicht ß") + + def test_spanish(self): + self.assertEqual(self._sub("España ¿Qué?"), "España ¿Qué?") + + def test_portuguese(self): + self.assertEqual(self._sub("Comunicação"), "Comunicação") + + def test_scandinavian(self): + self.assertEqual(self._sub("Norsk ø å æ"), "Norsk ø å æ") + + def test_greek_letters(self): + self.assertEqual(self._sub("αβγ"), "αβγ") + + def test_currency_and_symbols(self): + self.assertEqual(self._sub("© € £ ¥"), "© € £ ¥") + + def test_preserves_xml_amp(self): + self.assertEqual(self._sub("A & B"), "A & B") + + def test_preserves_xml_lt_gt(self): + self.assertEqual(self._sub("<tag>"), "<tag>") + + def test_preserves_xml_quot_apos(self): + self.assertEqual(self._sub(""hello'"), ""hello'") + + def test_preserves_uppercase_xml_entities(self): + """&, <, >, " resolve to XML-special chars; must not be replaced.""" + self.assertEqual(self._sub("&"), "&") + self.assertEqual(self._sub("<"), "<") + self.assertEqual(self._sub(">"), ">") + self.assertEqual(self._sub("""), """) + + def test_partial_entity_match_preserved(self): + """html.unescape can partially match & inside &ersand; — must not corrupt.""" + self.assertEqual(self._sub("&ersand;"), "&ersand;") + + def test_mixed_html_and_xml_entities(self): + self.assertEqual( + self._sub("Résumé & Co <test>"), + "Résumé & Co <test>", + ) + + def test_plain_ascii_unchanged(self): + self.assertEqual(self._sub("Plain ASCII text"), "Plain ASCII text") + + def test_direct_utf8_unchanged(self): + self.assertEqual(self._sub("日本語テレビ"), "日本語テレビ") + + def test_unknown_entity_preserved(self): + self.assertEqual(self._sub("&zzfakeentity;"), "&zzfakeentity;") + + +class ResolveHtmlEntitiesFileTests(TestCase): + """Tests for the file-level preprocessing function.""" + + def _make_file(self, content): + fd, path = tempfile.mkstemp(suffix=".xml") + with os.fdopen(fd, "w", encoding="utf-8") as f: + f.write(content) + self.addCleanup(lambda: os.unlink(path) if os.path.exists(path) else None) + return path + + def test_resolves_entities_in_file(self): + path = self._make_file( + '\nTélé' + ) + _resolve_html_entities(path) + with open(path, "r", encoding="utf-8") as f: + content = f.read() + self.assertIn("Télé", content) + self.assertNotIn("é", content) + + def test_preserves_xml_entities_in_file(self): + path = self._make_file("A & B <C>") + _resolve_html_entities(path) + with open(path, "r", encoding="utf-8") as f: + content = f.read() + self.assertIn("&", content) + self.assertIn("<", content) + self.assertIn(">", content) + + def test_no_temp_file_left_on_success(self): + path = self._make_file("test") + _resolve_html_entities(path) + self.assertFalse(os.path.exists(path + ".entity_tmp")) + + def test_plain_file_unchanged(self): + original = '\nPlain' + path = self._make_file(original) + _resolve_html_entities(path) + with open(path, "r", encoding="utf-8") as f: + content = f.read() + self.assertEqual(content, original) + + def test_utf8_content_preserved(self): + original = "日本語テレビ" + path = self._make_file(original) + _resolve_html_entities(path) + with open(path, "r", encoding="utf-8") as f: + content = f.read() + self.assertIn("日本語テレビ", content) + + def test_iso_8859_1_encoding(self): + """Files declaring ISO-8859-1 should be read in that encoding.""" + xml = '\nChaîne' + fd, path = tempfile.mkstemp(suffix=".xml") + with os.fdopen(fd, "wb") as f: + f.write(xml.encode("iso-8859-1")) + self.addCleanup(lambda: os.unlink(path) if os.path.exists(path) else None) + + _resolve_html_entities(path) + with open(path, "r", encoding="iso-8859-1") as f: + content = f.read() + self.assertIn("Cha\u00eene", content) + self.assertNotIn("î", content) + + def test_detect_encoding_utf8_default(self): + """Headers without an encoding declaration default to UTF-8.""" + self.assertEqual(_detect_xml_encoding(b''), "utf-8") + + def test_detect_encoding_iso_8859_1(self): + """Encoding is read from the XML declaration.""" + self.assertEqual( + _detect_xml_encoding(b''), + "ISO-8859-1", + ) + + def test_detect_encoding_single_quotes(self): + """Encoding detection works with single-quoted attributes.""" + self.assertEqual( + _detect_xml_encoding(b""), + "windows-1252", + ) + + def test_detect_encoding_unknown_falls_back(self): + """Unrecognized encoding falls back to UTF-8.""" + self.assertEqual( + _detect_xml_encoding(b''), + "utf-8", + ) + + def test_iso_8859_1_with_entities_roundtrip(self): + """ISO-8859-1 file with entities: resolved without corrupting existing accented chars.""" + # Mix of direct ISO-8859-1 chars and HTML entities + xml_str = '\nD\xe9j\xe0 émission' + fd, path = tempfile.mkstemp(suffix=".xml") + with os.fdopen(fd, "wb") as f: + f.write(xml_str.encode("iso-8859-1")) + self.addCleanup(lambda: os.unlink(path) if os.path.exists(path) else None) + + _resolve_html_entities(path) + with open(path, "r", encoding="iso-8859-1") as f: + content = f.read() + self.assertIn("D\xe9j\xe0", content, "Existing accented chars should be preserved") + self.assertIn("\xe9mission", content, "Entity should be resolved") + self.assertNotIn("é", content) + + def test_mismatched_encoding_leaves_file_untouched(self): + """File declaring UTF-8 but containing Latin-1 bytes is left alone.""" + # \xe9 is valid ISO-8859-1 but invalid as a standalone UTF-8 byte + raw = b'\n\xe9' + fd, path = tempfile.mkstemp(suffix=".xml") + with os.fdopen(fd, "wb") as f: + f.write(raw) + self.addCleanup(lambda: os.unlink(path) if os.path.exists(path) else None) + + original_bytes = raw # save for comparison + _resolve_html_entities(path) + with open(path, "rb") as f: + result_bytes = f.read() + self.assertEqual(result_bytes, original_bytes, "File should be untouched on decode error") diff --git a/apps/m3u/tasks.py b/apps/m3u/tasks.py index 9c31fa75..354a2ef4 100644 --- a/apps/m3u/tasks.py +++ b/apps/m3u/tasks.py @@ -1,6 +1,7 @@ # apps/m3u/tasks.py import logging import re +import regex import requests import os import gc @@ -2399,11 +2400,13 @@ def get_transformed_credentials(account, profile=None): # Apply profile-specific transformations if profile is provided if profile and profile.search_pattern and profile.replace_pattern: try: - # Handle backreferences in the replacement pattern - safe_replace_pattern = re.sub(r'\$(\d+)', r'\\\1', profile.replace_pattern) + # Handle backreferences: convert JS-style $ -> \g, $1 -> \1 + # regex module accepts JS-style (?...) named groups natively + safe_replace_pattern = regex.sub(r'\$<([^>]+)>', r'\\g<\1>', profile.replace_pattern) + safe_replace_pattern = regex.sub(r'\$(\d+)', r'\\\1', safe_replace_pattern) # Apply transformation to the complete URL - transformed_complete_url = re.sub(profile.search_pattern, safe_replace_pattern, complete_url) + transformed_complete_url = regex.sub(profile.search_pattern, safe_replace_pattern, complete_url) logger.info(f"Transformed complete URL: {complete_url} -> {transformed_complete_url}") # Extract components from the transformed URL diff --git a/apps/proxy/tasks.py b/apps/proxy/tasks.py index 68843712..d42e4d9a 100644 --- a/apps/proxy/tasks.py +++ b/apps/proxy/tasks.py @@ -31,7 +31,7 @@ def fetch_channel_stats(): while True: cursor, keys = redis_client.scan(cursor, match=channel_pattern) for key in keys: - channel_id_match = re.search(r"ts_proxy:channel:(.*):metadata", key.decode('utf-8')) + channel_id_match = re.search(r"ts_proxy:channel:(.*):metadata", key) if channel_id_match: ch_id = channel_id_match.group(1) channel_info = ChannelStatus.get_basic_channel_info(ch_id) diff --git a/apps/proxy/ts_proxy/channel_status.py b/apps/proxy/ts_proxy/channel_status.py index b5012b77..c4a54cdc 100644 --- a/apps/proxy/ts_proxy/channel_status.py +++ b/apps/proxy/ts_proxy/channel_status.py @@ -39,19 +39,19 @@ class ChannelStatus: info = { 'channel_id': channel_id, - 'state': metadata.get(ChannelMetadataField.STATE.encode('utf-8'), b'unknown').decode('utf-8'), - 'url': metadata.get(ChannelMetadataField.URL.encode('utf-8'), b'').decode('utf-8'), - 'stream_profile': metadata.get(ChannelMetadataField.STREAM_PROFILE.encode('utf-8'), b'').decode('utf-8'), - 'started_at': metadata.get(ChannelMetadataField.INIT_TIME.encode('utf-8'), b'0').decode('utf-8'), - 'owner': metadata.get(ChannelMetadataField.OWNER.encode('utf-8'), b'unknown').decode('utf-8'), - 'buffer_index': int(buffer_index_value.decode('utf-8')) if buffer_index_value else 0, + 'state': metadata.get(ChannelMetadataField.STATE, 'unknown'), + 'url': metadata.get(ChannelMetadataField.URL, ''), + 'stream_profile': metadata.get(ChannelMetadataField.STREAM_PROFILE, ''), + 'started_at': metadata.get(ChannelMetadataField.INIT_TIME, '0'), + 'owner': metadata.get(ChannelMetadataField.OWNER, 'unknown'), + 'buffer_index': int(buffer_index_value) if buffer_index_value else 0, } # Add stream ID and name information - stream_id_bytes = metadata.get(ChannelMetadataField.STREAM_ID.encode('utf-8')) + stream_id_bytes = metadata.get(ChannelMetadataField.STREAM_ID) if stream_id_bytes: try: - stream_id = int(stream_id_bytes.decode('utf-8')) + stream_id = int(stream_id_bytes) info['stream_id'] = stream_id # Look up stream name from database @@ -66,10 +66,10 @@ class ChannelStatus: logger.warning(f"Invalid stream_id format in Redis: {stream_id_bytes}") # Add M3U profile information - m3u_profile_id_bytes = metadata.get(ChannelMetadataField.M3U_PROFILE.encode('utf-8')) + m3u_profile_id_bytes = metadata.get(ChannelMetadataField.M3U_PROFILE) if m3u_profile_id_bytes: try: - m3u_profile_id = int(m3u_profile_id_bytes.decode('utf-8')) + m3u_profile_id = int(m3u_profile_id_bytes) info['m3u_profile_id'] = m3u_profile_id # Look up M3U profile name from database @@ -84,22 +84,22 @@ class ChannelStatus: logger.warning(f"Invalid m3u_profile_id format in Redis: {m3u_profile_id_bytes}") # Add timing information - state_changed_field = ChannelMetadataField.STATE_CHANGED_AT.encode('utf-8') + state_changed_field = ChannelMetadataField.STATE_CHANGED_AT if state_changed_field in metadata: - state_changed_at = float(metadata[state_changed_field].decode('utf-8')) + state_changed_at = float(metadata[state_changed_field]) info['state_changed_at'] = state_changed_at info['state_duration'] = time.time() - state_changed_at - init_time_field = ChannelMetadataField.INIT_TIME.encode('utf-8') + init_time_field = ChannelMetadataField.INIT_TIME if init_time_field in metadata: - created_at = float(metadata[init_time_field].decode('utf-8')) + created_at = float(metadata[init_time_field]) info['started_at'] = created_at info['uptime'] = time.time() - created_at # Add data throughput information - total_bytes_field = ChannelMetadataField.TOTAL_BYTES.encode('utf-8') + total_bytes_field = ChannelMetadataField.TOTAL_BYTES if total_bytes_field in metadata: - total_bytes = int(metadata[total_bytes_field].decode('utf-8')) + total_bytes = int(metadata[total_bytes_field]) info['total_bytes'] = total_bytes # Format total bytes in human-readable form @@ -130,7 +130,7 @@ class ChannelStatus: stale_client_ids = [] for client_id in client_ids: - client_id_str = client_id.decode('utf-8') + client_id_str = client_id client_key = RedisKeys.client_metadata(channel_id, client_id_str) client_data = proxy_server.redis_client.hgetall(client_key) @@ -141,33 +141,35 @@ class ChannelStatus: client_info = { 'client_id': client_id_str, - 'user_agent': client_data.get(b'user_agent', b'unknown').decode('utf-8'), - 'worker_id': client_data.get(b'worker_id', b'unknown').decode('utf-8'), + 'user_agent': client_data.get('user_agent', 'unknown'), + 'worker_id': client_data.get('worker_id', 'unknown'), + 'ip_address': client_data.get('ip_address', 'unknown'), + 'user_id': client_data.get('user_id', '0'), } - if b'connected_at' in client_data: - connected_at = float(client_data[b'connected_at'].decode('utf-8')) + if 'connected_at' in client_data: + connected_at = float(client_data['connected_at']) client_info['connected_at'] = connected_at client_info['connection_duration'] = time.time() - connected_at - if b'last_active' in client_data: - last_active = float(client_data[b'last_active'].decode('utf-8')) + if 'last_active' in client_data: + last_active = float(client_data['last_active']) client_info['last_active'] = last_active client_info['last_active_ago'] = time.time() - last_active # Add transfer rate statistics - if b'bytes_sent' in client_data: - client_info['bytes_sent'] = int(client_data[b'bytes_sent'].decode('utf-8')) + if 'bytes_sent' in client_data: + client_info['bytes_sent'] = int(client_data['bytes_sent']) # Add average transfer rate - if b'avg_rate_KBps' in client_data: - client_info['avg_rate_KBps'] = float(client_data[b'avg_rate_KBps'].decode('utf-8')) - elif b'transfer_rate_KBps' in client_data: # For backward compatibility - client_info['avg_rate_KBps'] = float(client_data[b'transfer_rate_KBps'].decode('utf-8')) + if 'avg_rate_KBps' in client_data: + client_info['avg_rate_KBps'] = float(client_data['avg_rate_KBps']) + elif 'transfer_rate_KBps' in client_data: # For backward compatibility + client_info['avg_rate_KBps'] = float(client_data['transfer_rate_KBps']) # Add current transfer rate - if b'current_rate_KBps' in client_data: - client_info['current_rate_KBps'] = float(client_data[b'current_rate_KBps'].decode('utf-8')) + if 'current_rate_KBps' in client_data: + client_info['current_rate_KBps'] = float(client_data['current_rate_KBps']) clients.append(client_info) @@ -249,7 +251,7 @@ class ChannelStatus: while True: cursor, keys = proxy_server.redis_client.scan(cursor, match=buffer_key_pattern, count=100) if keys: - all_buffer_keys.extend([k.decode('utf-8') for k in keys]) + all_buffer_keys.extend([k for k in keys]) if cursor == 0 or len(all_buffer_keys) >= 20: # Limit to 20 keys break @@ -279,61 +281,64 @@ class ChannelStatus: } # Add FFmpeg stream information - video_codec = metadata.get(ChannelMetadataField.VIDEO_CODEC.encode('utf-8')) + video_codec = metadata.get(ChannelMetadataField.VIDEO_CODEC) if video_codec: - info['video_codec'] = video_codec.decode('utf-8') + info['video_codec'] = video_codec - resolution = metadata.get(ChannelMetadataField.RESOLUTION.encode('utf-8')) + resolution = metadata.get(ChannelMetadataField.RESOLUTION) if resolution: - info['resolution'] = resolution.decode('utf-8') + info['resolution'] = resolution - source_fps = metadata.get(ChannelMetadataField.SOURCE_FPS.encode('utf-8')) + source_fps = metadata.get(ChannelMetadataField.SOURCE_FPS) if source_fps: - info['source_fps'] = float(source_fps.decode('utf-8')) + info['source_fps'] = source_fps - pixel_format = metadata.get(ChannelMetadataField.PIXEL_FORMAT.encode('utf-8')) + pixel_format = metadata.get(ChannelMetadataField.PIXEL_FORMAT) if pixel_format: - info['pixel_format'] = pixel_format.decode('utf-8') + info['pixel_format'] = pixel_format - source_bitrate = metadata.get(ChannelMetadataField.SOURCE_BITRATE.encode('utf-8')) + source_bitrate = metadata.get(ChannelMetadataField.SOURCE_BITRATE) if source_bitrate: - info['source_bitrate'] = float(source_bitrate.decode('utf-8')) + info['source_bitrate'] = source_bitrate - audio_codec = metadata.get(ChannelMetadataField.AUDIO_CODEC.encode('utf-8')) + audio_codec = metadata.get(ChannelMetadataField.AUDIO_CODEC) if audio_codec: - info['audio_codec'] = audio_codec.decode('utf-8') + info['audio_codec'] = audio_codec - sample_rate = metadata.get(ChannelMetadataField.SAMPLE_RATE.encode('utf-8')) + sample_rate = metadata.get(ChannelMetadataField.SAMPLE_RATE) if sample_rate: - info['sample_rate'] = int(sample_rate.decode('utf-8')) + info['sample_rate'] = sample_rate - audio_channels = metadata.get(ChannelMetadataField.AUDIO_CHANNELS.encode('utf-8')) + audio_channels = metadata.get(ChannelMetadataField.AUDIO_CHANNELS) if audio_channels: - info['audio_channels'] = audio_channels.decode('utf-8') + info['audio_channels'] = audio_channels - audio_bitrate = metadata.get(ChannelMetadataField.AUDIO_BITRATE.encode('utf-8')) + audio_bitrate = metadata.get(ChannelMetadataField.AUDIO_BITRATE) if audio_bitrate: - info['audio_bitrate'] = float(audio_bitrate.decode('utf-8')) + info['audio_bitrate'] = audio_bitrate + # Add FFmpeg performance stats - ffmpeg_speed = metadata.get(ChannelMetadataField.FFMPEG_SPEED.encode('utf-8')) + ffmpeg_speed = metadata.get(ChannelMetadataField.FFMPEG_SPEED) if ffmpeg_speed: - info['ffmpeg_speed'] = float(ffmpeg_speed.decode('utf-8')) + info['ffmpeg_speed'] = ffmpeg_speed - ffmpeg_fps = metadata.get(ChannelMetadataField.FFMPEG_FPS.encode('utf-8')) + ffmpeg_fps = metadata.get(ChannelMetadataField.FFMPEG_FPS) if ffmpeg_fps: - info['ffmpeg_fps'] = float(ffmpeg_fps.decode('utf-8')) + info['ffmpeg_fps'] = ffmpeg_fps - actual_fps = metadata.get(ChannelMetadataField.ACTUAL_FPS.encode('utf-8')) + actual_fps = metadata.get(ChannelMetadataField.ACTUAL_FPS) if actual_fps: - info['actual_fps'] = float(actual_fps.decode('utf-8')) + info['actual_fps'] = actual_fps - ffmpeg_bitrate = metadata.get(ChannelMetadataField.FFMPEG_BITRATE.encode('utf-8')) + ffmpeg_bitrate = metadata.get(ChannelMetadataField.FFMPEG_BITRATE) if ffmpeg_bitrate: - info['ffmpeg_bitrate'] = float(ffmpeg_bitrate.decode('utf-8')) - stream_type = metadata.get(ChannelMetadataField.STREAM_TYPE.encode('utf-8')) + info['ffmpeg_bitrate'] = ffmpeg_bitrate + + stream_type = metadata.get(ChannelMetadataField.STREAM_TYPE) if stream_type: - info['stream_type'] = stream_type.decode('utf-8') + info['stream_type'] = stream_type + return info @@ -378,33 +383,27 @@ class ChannelStatus: client_count = proxy_server.redis_client.scard(client_set_key) or 0 # Calculate uptime - init_time_bytes = metadata.get(ChannelMetadataField.INIT_TIME.encode('utf-8'), b'0') - created_at = float(init_time_bytes.decode('utf-8')) + init_time_bytes = metadata.get(ChannelMetadataField.INIT_TIME, '0') + created_at = float(init_time_bytes) uptime = time.time() - created_at if created_at > 0 else 0 - # Safely decode bytes or use defaults - def safe_decode(bytes_value, default="unknown"): - if bytes_value is None: - return default - return bytes_value.decode('utf-8') - # Simplified info info = { 'channel_id': channel_id, - 'state': safe_decode(metadata.get(ChannelMetadataField.STATE.encode('utf-8'))), - 'url': safe_decode(metadata.get(ChannelMetadataField.URL.encode('utf-8')), ""), - 'stream_profile': safe_decode(metadata.get(ChannelMetadataField.STREAM_PROFILE.encode('utf-8')), ""), - 'owner': safe_decode(metadata.get(ChannelMetadataField.OWNER.encode('utf-8'))), - 'buffer_index': int(buffer_index_value.decode('utf-8')) if buffer_index_value else 0, + 'state': metadata.get(ChannelMetadataField.STATE), + 'url': metadata.get(ChannelMetadataField.URL, ""), + 'stream_profile': metadata.get(ChannelMetadataField.STREAM_PROFILE, ""), + 'owner': metadata.get(ChannelMetadataField.OWNER), + 'buffer_index': int(buffer_index_value) if buffer_index_value else 0, 'client_count': client_count, 'uptime': uptime } # Add stream ID and name information - stream_id_bytes = metadata.get(ChannelMetadataField.STREAM_ID.encode('utf-8')) + stream_id_bytes = metadata.get(ChannelMetadataField.STREAM_ID) if stream_id_bytes: try: - stream_id = int(stream_id_bytes.decode('utf-8')) + stream_id = int(stream_id_bytes) info['stream_id'] = stream_id # Look up stream name from database @@ -419,9 +418,9 @@ class ChannelStatus: logger.warning(f"Invalid stream_id format in Redis: {stream_id_bytes}") # Add data throughput information to basic info - total_bytes_bytes = proxy_server.redis_client.hget(metadata_key, ChannelMetadataField.TOTAL_BYTES.encode('utf-8')) + total_bytes_bytes = proxy_server.redis_client.hget(metadata_key, ChannelMetadataField.TOTAL_BYTES) if total_bytes_bytes: - total_bytes = int(total_bytes_bytes.decode('utf-8')) + total_bytes = int(total_bytes_bytes) info['total_bytes'] = total_bytes # Calculate and add bitrate @@ -458,25 +457,28 @@ class ChannelStatus: if client_id in stale_client_ids: continue - client_id_str = client_id.decode('utf-8') - client_key = RedisKeys.client_metadata(channel_id, client_id_str) + client_key = RedisKeys.client_metadata(channel_id, client_id) client_info = { - 'client_id': client_id_str, + 'client_id': client_id, } user_agent_bytes = proxy_server.redis_client.hget(client_key, 'user_agent') - client_info['user_agent'] = safe_decode(user_agent_bytes) + client_info['user_agent'] = user_agent_bytes ip_address_bytes = proxy_server.redis_client.hget(client_key, 'ip_address') if ip_address_bytes: - client_info['ip_address'] = safe_decode(ip_address_bytes) + client_info['ip_address'] = ip_address_bytes connected_at_bytes = proxy_server.redis_client.hget(client_key, 'connected_at') if connected_at_bytes: - connected_at = float(connected_at_bytes.decode('utf-8')) + connected_at = float(connected_at_bytes) client_info['connected_since'] = time.time() - connected_at + user_id_bytes = proxy_server.redis_client.hget(client_key, 'user_id') + if user_id_bytes: + client_info['user_id'] = user_id_bytes + clients.append(client_info) # Add clients to info @@ -484,10 +486,10 @@ class ChannelStatus: info['client_count'] = client_count # Add M3U profile information - m3u_profile_id_bytes = metadata.get(ChannelMetadataField.M3U_PROFILE.encode('utf-8')) - if m3u_profile_id_bytes: + m3u_profile_id = metadata.get(ChannelMetadataField.M3U_PROFILE) + if m3u_profile_id: try: - m3u_profile_id = int(m3u_profile_id_bytes.decode('utf-8')) + m3u_profile_id = int(m3u_profile_id) info['m3u_profile_id'] = m3u_profile_id # Look up M3U profile name from database @@ -499,32 +501,36 @@ class ChannelStatus: except (ImportError, DatabaseError) as e: logger.warning(f"Failed to get M3U profile name for ID {m3u_profile_id}: {e}") except ValueError: - logger.warning(f"Invalid m3u_profile_id format in Redis: {m3u_profile_id_bytes}") + logger.warning(f"Invalid m3u_profile_id format in Redis: {m3u_profile_id}") # Add stream info to basic info as well - video_codec = metadata.get(ChannelMetadataField.VIDEO_CODEC.encode('utf-8')) + video_codec = metadata.get(ChannelMetadataField.VIDEO_CODEC) if video_codec: - info['video_codec'] = video_codec.decode('utf-8') + info['video_codec'] = video_codec - resolution = metadata.get(ChannelMetadataField.RESOLUTION.encode('utf-8')) + resolution = metadata.get(ChannelMetadataField.RESOLUTION) if resolution: - info['resolution'] = resolution.decode('utf-8') + info['resolution'] = resolution - source_fps = metadata.get(ChannelMetadataField.SOURCE_FPS.encode('utf-8')) + source_fps = metadata.get(ChannelMetadataField.SOURCE_FPS) if source_fps: - info['source_fps'] = float(source_fps.decode('utf-8')) - ffmpeg_speed = metadata.get(ChannelMetadataField.FFMPEG_SPEED.encode('utf-8')) + info['source_fps'] = float(source_fps) + + ffmpeg_speed = metadata.get(ChannelMetadataField.FFMPEG_SPEED) if ffmpeg_speed: - info['ffmpeg_speed'] = float(ffmpeg_speed.decode('utf-8')) - audio_codec = metadata.get(ChannelMetadataField.AUDIO_CODEC.encode('utf-8')) + info['ffmpeg_speed'] = float(ffmpeg_speed) + + audio_codec = metadata.get(ChannelMetadataField.AUDIO_CODEC) if audio_codec: - info['audio_codec'] = audio_codec.decode('utf-8') - audio_channels = metadata.get(ChannelMetadataField.AUDIO_CHANNELS.encode('utf-8')) + info['audio_codec'] = audio_codec + + audio_channels = metadata.get(ChannelMetadataField.AUDIO_CHANNELS) if audio_channels: - info['audio_channels'] = audio_channels.decode('utf-8') - stream_type = metadata.get(ChannelMetadataField.STREAM_TYPE.encode('utf-8')) + info['audio_channels'] = audio_channels + + stream_type = metadata.get(ChannelMetadataField.STREAM_TYPE) if stream_type: - info['stream_type'] = stream_type.decode('utf-8') + info['stream_type'] = stream_type return info except Exception as e: diff --git a/apps/proxy/ts_proxy/client_manager.py b/apps/proxy/ts_proxy/client_manager.py index ea2aa5b0..af7eb7d3 100644 --- a/apps/proxy/ts_proxy/client_manager.py +++ b/apps/proxy/ts_proxy/client_manager.py @@ -1,10 +1,8 @@ """Client connection management for TS streams""" import threading -import logging import time import json -import gevent from typing import Set, Optional from apps.proxy.config import TSConfig as Config from redis.exceptions import ConnectionError, TimeoutError @@ -58,7 +56,8 @@ class ClientManager: from django.conf import settings redis_url = getattr(settings, 'REDIS_URL', 'redis://localhost:6379/0') - redis_client = redis.Redis.from_url(redis_url, decode_responses=True) + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) + redis_client = redis.Redis.from_url(redis_url, decode_responses=True, **ssl_params) all_channels = [] cursor = 0 @@ -129,7 +128,7 @@ class ClientManager: # Check for stale activity using last_active field last_active = self.redis_client.hget(client_key, "last_active") if last_active: - last_active_time = float(last_active.decode('utf-8')) + last_active_time = float(last_active) ghost_timeout = self.heartbeat_interval * getattr(Config, 'GHOST_CLIENT_MULTIPLIER', 5.0) if current_time - last_active_time > ghost_timeout: @@ -229,7 +228,7 @@ class ClientManager: except Exception as e: logger.error(f"Error notifying owner of client activity: {e}") - def add_client(self, client_id, client_ip, user_agent=None): + def add_client(self, client_id, client_ip, user_agent=None, user=None): """Add a client with duplicate prevention""" if client_id in self._registered_clients: logger.debug(f"Client {client_id} already registered, skipping") @@ -247,7 +246,9 @@ class ClientManager: "ip_address": client_ip, "connected_at": current_time, "last_active": current_time, - "worker_id": self.worker_id or "unknown" + "worker_id": self.worker_id or "unknown", + "user_id": str(user.id) if user is not None else "0", + # "user_level": user.user_level if user is not None else 100, # default to a high value since no user means the non-user specific M3U/HDHR } try: @@ -277,7 +278,8 @@ class ClientManager: "channel_id": self.channel_id, "client_id": client_id, "worker_id": self.worker_id or "unknown", - "timestamp": time.time() + "timestamp": time.time(), + "username": user.username if user is not None else "unknown" } if user_agent: @@ -308,8 +310,6 @@ class ClientManager: def remove_client(self, client_id): """Remove a client from this channel and Redis""" - client_ip = None - with self.lock: if client_id in self.clients: self.clients.remove(client_id) @@ -320,13 +320,11 @@ class ClientManager: self.last_active_time = time.time() if self.redis_client: - # Get client IP before removing the data + # Get client data before removing the data client_key = f"ts_proxy:channel:{self.channel_id}:clients:{client_id}" - client_data = self.redis_client.hgetall(client_key) - if client_data and b'ip_address' in client_data: - client_ip = client_data[b'ip_address'].decode('utf-8') - elif client_data and 'ip_address' in client_data: - client_ip = client_data['ip_address'] + client_username = self.redis_client.hget(client_key, "username") or "unknown" + if isinstance(client_username, bytes): + client_username = client_username.decode("utf-8") # Remove from channel's client set self.redis_client.srem(self.client_set_key, client_id) @@ -367,7 +365,8 @@ class ClientManager: "client_id": client_id, "worker_id": self.worker_id or "unknown", "timestamp": time.time(), - "remaining_clients": remaining + "remaining_clients": remaining, + "username": client_username }) self.redis_client.publish(RedisKeys.events_channel(self.channel_id), event_data) @@ -434,8 +433,7 @@ class ClientManager: client_id_list = list(client_ids) pipe = redis_client.pipeline() for cid in client_id_list: - cid_str = cid.decode('utf-8') - pipe.exists(RedisKeys.client_metadata(channel_id, cid_str)) + pipe.exists(RedisKeys.client_metadata(channel_id, cid)) results = pipe.execute() stale_ids = [ diff --git a/apps/proxy/ts_proxy/server.py b/apps/proxy/ts_proxy/server.py index b72b350a..5316ec58 100644 --- a/apps/proxy/ts_proxy/server.py +++ b/apps/proxy/ts_proxy/server.py @@ -166,6 +166,7 @@ class ProxyServer: redis_password = os.environ.get("REDIS_PASSWORD", getattr(settings, 'REDIS_PASSWORD', '')) redis_user = os.environ.get("REDIS_USER", getattr(settings, 'REDIS_USER', '')) + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) pubsub_client = redis.Redis( host=redis_host, port=redis_port, @@ -175,7 +176,9 @@ class ProxyServer: socket_timeout=60, socket_connect_timeout=10, socket_keepalive=True, - health_check_interval=30 + health_check_interval=30, + decode_responses=True, + **ssl_params ) logger.info("Created fallback Redis PubSub client for event listener") @@ -196,8 +199,8 @@ class ProxyServer: continue try: - channel = message["channel"].decode("utf-8") - data = json.loads(message["data"].decode("utf-8")) + channel = message["channel"] + data = json.loads(message["data"]) event_type = data.get("event") channel_id = data.get("channel_id") @@ -373,7 +376,7 @@ class ProxyServer: if result is None: return None try: - return result.decode('utf-8') + return result except (AttributeError, UnicodeDecodeError) as e: logger.error(f"Error decoding channel owner for {channel_id}: {e}, raw={result!r}") return None @@ -412,7 +415,7 @@ class ProxyServer: current_owner = self._execute_redis_command( lambda: self.redis_client.get(lock_key) ) - if current_owner and current_owner.decode('utf-8') == self.worker_id: + if current_owner and current_owner == self.worker_id: # Refresh TTL self._execute_redis_command( lambda: self.redis_client.expire(lock_key, ttl) @@ -437,7 +440,7 @@ class ProxyServer: # Only delete if we're the current owner to prevent race conditions current = self.redis_client.get(lock_key) - if current and current.decode('utf-8') == self.worker_id: + if current and current == self.worker_id: self.redis_client.delete(lock_key) logger.info(f"Released ownership of channel {channel_id}") @@ -471,7 +474,7 @@ class ProxyServer: return False return False - if current.decode('utf-8') == self.worker_id: + if current == self.worker_id: self.redis_client.expire(lock_key, ttl) return True @@ -488,15 +491,15 @@ class ProxyServer: metadata_key = RedisKeys.channel_metadata(channel_id) if self.redis_client.exists(metadata_key): metadata = self.redis_client.hgetall(metadata_key) - if b'state' in metadata: - state = metadata[b'state'].decode('utf-8') + if 'state' in metadata: + state = metadata['state'] active_states = [ChannelState.INITIALIZING, ChannelState.CONNECTING, ChannelState.WAITING_FOR_CLIENTS, ChannelState.ACTIVE, ChannelState.BUFFERING] if state in active_states: logger.info(f"Channel {channel_id} already being initialized with state {state}") # Create buffer and client manager only if we don't have them if channel_id not in self.stream_buffers: - self.stream_buffers[channel_id] = StreamBuffer(channel_id, redis_client=self.redis_client) + self.stream_buffers[channel_id] = StreamBuffer(channel_id, redis_client=RedisClient.get_buffer()) if channel_id not in self.client_managers: self.client_managers[channel_id] = ClientManager( channel_id, @@ -507,7 +510,7 @@ class ProxyServer: # Create buffer and client manager instances (or reuse if they exist) if channel_id not in self.stream_buffers: - buffer = StreamBuffer(channel_id, redis_client=self.redis_client) + buffer = StreamBuffer(channel_id, redis_client=RedisClient.get_buffer()) self.stream_buffers[channel_id] = buffer if channel_id not in self.client_managers: @@ -546,18 +549,18 @@ class ProxyServer: # If no url was passed, try to get from Redis if not url and existing_metadata: - url_bytes = existing_metadata.get(b'url') + url_bytes = existing_metadata.get('url') if url_bytes: - channel_url = url_bytes.decode('utf-8') + channel_url = url_bytes - ua_bytes = existing_metadata.get(b'user_agent') + ua_bytes = existing_metadata.get('user_agent') if ua_bytes: - channel_user_agent = ua_bytes.decode('utf-8') + channel_user_agent = ua_bytes # Get stream ID from metadata if not provided - if not channel_stream_id and b'stream_id' in existing_metadata: + if not channel_stream_id and 'stream_id' in existing_metadata: try: - channel_stream_id = int(existing_metadata[b'stream_id'].decode('utf-8')) + channel_stream_id = int(existing_metadata['stream_id']) logger.debug(f"Found stream_id {channel_stream_id} in metadata for channel {channel_id}") except (ValueError, TypeError) as e: logger.debug(f"Could not parse stream_id from metadata: {e}") @@ -572,7 +575,7 @@ class ProxyServer: # Create buffer but not stream manager (only if not already exists) if channel_id not in self.stream_buffers: - buffer = StreamBuffer(channel_id=channel_id, redis_client=self.redis_client) + buffer = StreamBuffer(channel_id=channel_id, redis_client=RedisClient.get_buffer()) self.stream_buffers[channel_id] = buffer # Create client manager with channel_id and redis_client (only if not already exists) @@ -595,7 +598,7 @@ class ProxyServer: # Create buffer but not stream manager (only if not already exists) if channel_id not in self.stream_buffers: - buffer = StreamBuffer(channel_id=channel_id, redis_client=self.redis_client) + buffer = StreamBuffer(channel_id=channel_id, redis_client=RedisClient.get_buffer()) self.stream_buffers[channel_id] = buffer # Create client manager with channel_id and redis_client (only if not already exists) @@ -634,12 +637,12 @@ class ProxyServer: # Verify the stream_id was set correctly in Redis stream_id_value = self.redis_client.hget(metadata_key, "stream_id") if stream_id_value: - logger.info(f"Verified stream_id {stream_id_value.decode('utf-8')} is set in Redis for channel {channel_id}") + logger.info(f"Verified stream_id {stream_id_value} is set in Redis for channel {channel_id}") else: logger.warning(f"Failed to set stream_id in Redis for channel {channel_id}") # Create stream buffer - buffer = StreamBuffer(channel_id=channel_id, redis_client=self.redis_client) + buffer = StreamBuffer(channel_id=channel_id, redis_client=RedisClient.get_buffer()) logger.debug(f"Created StreamBuffer for channel {channel_id}") self.stream_buffers[channel_id] = buffer @@ -733,8 +736,8 @@ class ProxyServer: metadata = self.redis_client.hgetall(metadata_key) # Get channel state and owner - state = metadata.get(b'state', b'unknown').decode('utf-8') - owner = metadata.get(b'owner', b'').decode('utf-8') + state = metadata.get('state', 'unknown') + owner = metadata.get('owner', '') # States that indicate the channel is running properly or shutting down valid_states = [ChannelState.ACTIVE, ChannelState.WAITING_FOR_CLIENTS, @@ -772,8 +775,8 @@ class ProxyServer: return False else: # Unknown or initializing state, check how long it's been in this state - if b'state_changed_at' in metadata: - state_changed_at = float(metadata[b'state_changed_at'].decode('utf-8')) + if 'state_changed_at' in metadata: + state_changed_at = float(metadata['state_changed_at']) state_age = time.time() - state_changed_at # If in initializing state for too long, consider it stale @@ -811,8 +814,8 @@ class ProxyServer: # If we have metadata, log details for debugging if metadata: - state = metadata.get(b'state', b'unknown').decode('utf-8') - owner = metadata.get(b'owner', b'unknown').decode('utf-8') + state = metadata.get('state', 'unknown') + owner = metadata.get('owner', 'unknown') logger.info(f"Zombie channel details - state: {state}, owner: {owner}") # Clean up Redis keys @@ -937,16 +940,16 @@ class ProxyServer: metadata = self.redis_client.hgetall(metadata_key) if metadata: # Calculate runtime from init_time - if b'init_time' in metadata: + if 'init_time' in metadata: try: - init_time = float(metadata[b'init_time'].decode('utf-8')) + init_time = float(metadata['init_time']) runtime = round(time.time() - init_time, 2) except Exception: pass # Get total bytes transferred - if b'total_bytes' in metadata: + if 'total_bytes' in metadata: try: - total_bytes = int(metadata[b'total_bytes'].decode('utf-8')) + total_bytes = int(metadata['total_bytes']) except Exception: pass @@ -1057,8 +1060,8 @@ class ProxyServer: if self.redis_client: metadata_key = RedisKeys.channel_metadata(channel_id) metadata = self.redis_client.hgetall(metadata_key) - if metadata and b'state' in metadata: - channel_state = metadata[b'state'].decode('utf-8') + if metadata and 'state' in metadata: + channel_state = metadata['state'] # Check if channel has any clients left total_clients = 0 @@ -1090,9 +1093,9 @@ class ProxyServer: # Get connection_ready_time from metadata (indicates if channel reached ready state) connection_ready_time = None - if metadata and b'connection_ready_time' in metadata: + if metadata and 'connection_ready_time' in metadata: try: - connection_ready_time = float(metadata[b'connection_ready_time'].decode('utf-8')) + connection_ready_time = float(metadata['connection_ready_time']) except (ValueError, TypeError): pass @@ -1104,15 +1107,15 @@ class ProxyServer: attempt_value = self.redis_client.get(attempt_key) if attempt_value: try: - connection_attempt_time = float(attempt_value.decode('utf-8')) + connection_attempt_time = float(attempt_value) except (ValueError, TypeError): pass # Also get init time as a fallback init_time = None - if metadata and b'init_time' in metadata: + if metadata and 'init_time' in metadata: try: - init_time = float(metadata[b'init_time'].decode('utf-8')) + init_time = float(metadata['init_time']) except (ValueError, TypeError): pass @@ -1183,7 +1186,7 @@ class ProxyServer: disconnect_value = self.redis_client.get(disconnect_key) if disconnect_value: try: - disconnect_time = float(disconnect_value.decode('utf-8')) + disconnect_time = float(disconnect_value) except (ValueError, TypeError) as e: logger.error(f"Invalid disconnect time for channel {channel_id}: {e}") @@ -1304,7 +1307,7 @@ class ProxyServer: for key in channel_keys: try: - channel_id = key.decode('utf-8').split(':')[2] + channel_id = key.split(':')[2] # Check if this channel has an owner owner = self.get_channel_owner(channel_id) @@ -1349,7 +1352,7 @@ class ProxyServer: for key in channel_keys: try: - channel_id = key.decode('utf-8').split(':')[2] + channel_id = key.split(':')[2] # Get metadata first metadata = self.redis_client.hgetall(key) @@ -1364,7 +1367,7 @@ class ProxyServer: continue # Get owner - owner = metadata.get(b'owner', b'').decode('utf-8') if b'owner' in metadata else '' + owner = metadata.get('owner', '') if 'owner' in metadata else '' # Check if owner is still alive owner_alive = False @@ -1378,7 +1381,7 @@ class ProxyServer: # If no owner and no clients, clean it up if not owner_alive and client_count == 0: - state = metadata.get(b'state', b'unknown').decode('utf-8') if b'state' in metadata else 'unknown' + state = metadata.get('state', 'unknown') logger.warning(f"Found orphaned metadata for channel {channel_id} (state: {state}, owner: {owner}, clients: {client_count}) - cleaning up") # If we have it locally, stop it properly to clean up transcode/proxy processes @@ -1397,7 +1400,7 @@ class ProxyServer: real_count = max(0, client_count - len(stale_ids)) if real_count <= 0: # No real clients remain — safe to clean up. - state = metadata.get(b'state', b'unknown').decode('utf-8') if b'state' in metadata else 'unknown' + state = metadata.get('state', 'unknown') logger.warning( f"Orphaned channel {channel_id} (state: {state}, " f"owner: {owner}) had {client_count} ghost client(s) " @@ -1492,8 +1495,8 @@ class ProxyServer: # Get current state for logging current_state = None metadata = self.redis_client.hgetall(metadata_key) - if metadata and b'state' in metadata: - current_state = metadata[b'state'].decode('utf-8') + if metadata and 'state' in metadata: + current_state = metadata['state'] # Only update if state is actually changing if current_state == new_state: diff --git a/apps/proxy/ts_proxy/services/channel_service.py b/apps/proxy/ts_proxy/services/channel_service.py index e765ece3..d0478e8f 100644 --- a/apps/proxy/ts_proxy/services/channel_service.py +++ b/apps/proxy/ts_proxy/services/channel_service.py @@ -61,7 +61,7 @@ class ChannelService: # Verify the stream_id was set stream_id_value = proxy_server.redis_client.hget(metadata_key, ChannelMetadataField.STREAM_ID) if stream_id_value: - logger.debug(f"Verified stream_id {stream_id_value.decode('utf-8')} is now set in Redis") + logger.debug(f"Verified stream_id {stream_id_value} is now set in Redis") else: logger.error(f"Failed to set stream_id {stream_id} in Redis before initialization") @@ -131,7 +131,7 @@ class ChannelService: try: # This is inefficient but used for diagnostics - in production would use more targeted checks redis_keys = proxy_server.redis_client.keys(f"ts_proxy:*:{channel_id}*") - redis_keys = [k.decode('utf-8') for k in redis_keys] if redis_keys else [] + redis_keys = [k for k in redis_keys] if redis_keys else [] except Exception as e: logger.error(f"Error checking Redis keys: {e}") @@ -236,8 +236,8 @@ class ChannelService: metadata_key = RedisKeys.channel_metadata(channel_id) try: metadata = proxy_server.redis_client.hgetall(metadata_key) - if metadata and b'state' in metadata: - state = metadata[b'state'].decode('utf-8') + if metadata and 'state' in metadata: + state = metadata['state'] channel_info = {"state": state} # Immediately mark as stopping in metadata so clients detect it faster @@ -382,8 +382,8 @@ class ChannelService: metadata = proxy_server.redis_client.hgetall(metadata_key) # Extract state and owner - state = metadata.get(ChannelMetadataField.STATE.encode(), b'unknown').decode('utf-8') - owner = metadata.get(ChannelMetadataField.OWNER.encode(), b'unknown').decode('utf-8') + state = metadata.get(ChannelMetadataField.STATE, 'unknown') + owner = metadata.get(ChannelMetadataField.OWNER, 'unknown') # Valid states indicate channel is running properly valid_states = [ChannelState.ACTIVE, ChannelState.WAITING_FOR_CLIENTS, ChannelState.CONNECTING] @@ -409,7 +409,7 @@ class ChannelService: } if last_data: - last_data_time = float(last_data.decode('utf-8')) + last_data_time = float(last_data) data_age = time.time() - last_data_time details["last_data_age"] = data_age @@ -432,13 +432,13 @@ class ChannelService: try: # Use factory to parse the line based on stream type parsed_data = LogParserFactory.parse(stream_type, stream_info_line) - + if not parsed_data: return # Update Redis and database with parsed data ChannelService._update_stream_info_in_redis( - channel_id, + channel_id, parsed_data.get('video_codec'), parsed_data.get('resolution'), parsed_data.get('width'), @@ -579,7 +579,7 @@ class ChannelService: metadata_key = RedisKeys.channel_metadata(channel_id) # First check if the key exists and what type it is - key_type = proxy_server.redis_client.type(metadata_key).decode('utf-8') + key_type = proxy_server.redis_client.type(metadata_key) logger.debug(f"Redis key {metadata_key} is of type: {key_type}") # Build metadata update dict diff --git a/apps/proxy/ts_proxy/stream_generator.py b/apps/proxy/ts_proxy/stream_generator.py index 2e16008b..0036be5c 100644 --- a/apps/proxy/ts_proxy/stream_generator.py +++ b/apps/proxy/ts_proxy/stream_generator.py @@ -25,7 +25,7 @@ class StreamGenerator: data delivery, and cleanup. """ - def __init__(self, channel_id, client_id, client_ip, client_user_agent, channel_initializing=False): + def __init__(self, channel_id, client_id, client_ip, client_user_agent, channel_initializing=False, user=None): """ Initialize the stream generator with client and channel details. @@ -35,12 +35,14 @@ class StreamGenerator: client_ip: Client's IP address client_user_agent: User agent string from client channel_initializing: Whether the channel is still initializing + user: Authenticated user making the request """ self.channel_id = channel_id self.client_id = client_id self.client_ip = client_ip self.client_user_agent = client_user_agent self.channel_initializing = channel_initializing + self.user = user # Performance and state tracking self.stream_start_time = time.time() @@ -112,7 +114,8 @@ class StreamGenerator: channel_name=channel_obj.name, client_ip=self.client_ip, client_id=self.client_id, - user_agent=self.client_user_agent[:100] if self.client_user_agent else None + user_agent=self.client_user_agent[:100] if self.client_user_agent else None, + username=self.user.username if self.user else None ) except Exception as e: logger.error(f"Could not log client connect event: {e}") @@ -141,13 +144,13 @@ class StreamGenerator: metadata_key = RedisKeys.channel_metadata(self.channel_id) metadata = proxy_server.redis_client.hgetall(metadata_key) - if metadata and b'state' in metadata: - state = metadata[b'state'].decode('utf-8') + if metadata and 'state' in metadata: + state = metadata['state'] if state in ['waiting_for_clients', 'active']: logger.info(f"[{self.client_id}] Channel {self.channel_id} now ready (state={state})") return True elif state in ['error', 'stopped', 'stopping']: # Added 'stopping' to error states - error_message = metadata.get(b'error_message', b'Unknown error').decode('utf-8') + error_message = metadata.get('error_message', 'Unknown error') logger.error(f"[{self.client_id}] Channel {self.channel_id} in error state: {state}, message: {error_message}") # Send error packet before giving up yield create_ts_packet('error', f"Error: {error_message}") @@ -155,9 +158,9 @@ class StreamGenerator: else: # Improved logging to track initialization progress init_time = "unknown" - if b'init_time' in metadata: + if 'init_time' in metadata: try: - init_time_float = float(metadata[b'init_time'].decode('utf-8')) + init_time_float = float(metadata['init_time']) init_duration = time.time() - init_time_float init_time = f"{init_duration:.1f}s ago" except: @@ -390,8 +393,8 @@ class StreamGenerator: # Channel state in metadata metadata_key = RedisKeys.channel_metadata(self.channel_id) metadata = proxy_server.redis_client.hgetall(metadata_key) - if metadata and b'state' in metadata: - state = metadata[b'state'].decode('utf-8') + if metadata and 'state' in metadata: + state = metadata['state'] if state in ['error', 'stopped', 'stopping']: logger.info(f"[{self.client_id}] Channel in {state} state, terminating stream") return False @@ -555,8 +558,6 @@ class StreamGenerator: if metadata: stream_id_bytes = proxy_server.redis_client.hget(metadata_key, ChannelMetadataField.STREAM_ID) if stream_id_bytes: - stream_id = int(stream_id_bytes.decode('utf-8')) - # Check if we're the last client if self.channel_id in proxy_server.client_managers: client_count = proxy_server.client_managers[self.channel_id].get_total_client_count() @@ -595,7 +596,8 @@ class StreamGenerator: client_id=self.client_id, user_agent=self.client_user_agent[:100] if self.client_user_agent else None, duration=round(elapsed, 2), - bytes_sent=self.bytes_sent + bytes_sent=self.bytes_sent, + username=self.user.username if self.user else None ) except Exception as e: logger.error(f"Could not log client disconnect event: {e}") @@ -630,10 +632,10 @@ class StreamGenerator: gevent.spawn(delayed_shutdown) -def create_stream_generator(channel_id, client_id, client_ip, client_user_agent, channel_initializing=False): +def create_stream_generator(channel_id, client_id, client_ip, client_user_agent, channel_initializing=False, user=None): """ Factory function to create a new stream generator. Returns a function that can be passed to StreamingHttpResponse. """ - generator = StreamGenerator(channel_id, client_id, client_ip, client_user_agent, channel_initializing) + generator = StreamGenerator(channel_id, client_id, client_ip, client_user_agent, channel_initializing, user=user) return generator.generate diff --git a/apps/proxy/ts_proxy/stream_manager.py b/apps/proxy/ts_proxy/stream_manager.py index 424e8231..7b07c744 100644 --- a/apps/proxy/ts_proxy/stream_manager.py +++ b/apps/proxy/ts_proxy/stream_manager.py @@ -90,7 +90,7 @@ class StreamManager: # Try to get stream_id specifically stream_id_bytes = buffer.redis_client.hget(metadata_key, "stream_id") if stream_id_bytes: - self.current_stream_id = int(stream_id_bytes.decode('utf-8')) + self.current_stream_id = int(stream_id_bytes) self.tried_stream_ids.add(self.current_stream_id) logger.info(f"Loaded stream ID {self.current_stream_id} from Redis for channel {buffer.channel_id}") else: @@ -413,7 +413,7 @@ class StreamManager: is_owner = ( current_owner and self.worker_id - and current_owner.decode('utf-8') == self.worker_id + and current_owner == self.worker_id ) no_owner = current_owner is None @@ -423,7 +423,7 @@ class StreamManager: metadata_key, ChannelMetadataField.STATE ) current_state = ( - current_state_bytes.decode('utf-8') + current_state_bytes if current_state_bytes else None ) should_update = current_state in ChannelState.PRE_ACTIVE @@ -1503,9 +1503,9 @@ class StreamManager: current_state = None try: metadata = redis_client.hgetall(metadata_key) - state_field = ChannelMetadataField.STATE.encode('utf-8') + state_field = ChannelMetadataField.STATE if metadata and state_field in metadata: - current_state = metadata[state_field].decode('utf-8') + current_state = metadata[state_field] except Exception as e: logger.error(f"Error checking current state: {e}") @@ -1710,4 +1710,4 @@ class StreamManager: """Safely reset the URL switching state if it gets stuck""" self.url_switching = False self.url_switch_start_time = 0 - logger.info(f"Reset URL switching state for channel {self.channel_id}") \ No newline at end of file + logger.info(f"Reset URL switching state for channel {self.channel_id}") diff --git a/apps/proxy/ts_proxy/url_utils.py b/apps/proxy/ts_proxy/url_utils.py index 14a714ea..5f4615a4 100644 --- a/apps/proxy/ts_proxy/url_utils.py +++ b/apps/proxy/ts_proxy/url_utils.py @@ -3,7 +3,7 @@ Utilities for handling stream URLs and transformations. """ import logging -import re +import regex from typing import Optional, Tuple, List from django.shortcuts import get_object_or_404 from apps.channels.models import Channel, Stream @@ -146,13 +146,14 @@ def transform_url(input_url: str, search_pattern: str, replace_pattern: str) -> logger.debug(f" base URL: {input_url}") logger.debug(f" search: {search_pattern}") - # Handle backreferences in the replacement pattern - safe_replace_pattern = re.sub(r'\$(\d+)', r'\\\1', replace_pattern) + # Convert JS-style backreferences in replace pattern: $ -> \g, $1 -> \1 + safe_replace_pattern = regex.sub(r'\$<([^>]+)>', r'\\g<\1>', replace_pattern) + safe_replace_pattern = regex.sub(r'\$(\d+)', r'\\\1', safe_replace_pattern) logger.debug(f" replace: {replace_pattern}") logger.debug(f" safe replace: {safe_replace_pattern}") - # Apply the transformation - stream_url = re.sub(search_pattern, safe_replace_pattern, input_url) + # Apply the transformation (regex module accepts JS-style (?...) natively) + stream_url = regex.sub(search_pattern, safe_replace_pattern, input_url) logger.info(f"Generated stream url: {stream_url}") return stream_url @@ -211,9 +212,9 @@ def get_stream_info_for_switch(channel_id: str, target_stream_id: Optional[int] existing_stream_id = redis_client.get(f"channel_stream:{channel.id}") if existing_stream_id: # Decode bytes to string/int for proper Redis key lookup - existing_stream_id = existing_stream_id.decode('utf-8') + existing_stream_id = existing_stream_id existing_profile_id = redis_client.get(f"stream_profile:{existing_stream_id}") - if existing_profile_id and int(existing_profile_id.decode('utf-8')) == profile.id: + if existing_profile_id and int(existing_profile_id) == profile.id: channel_using_profile = True logger.debug(f"Channel {channel.id} already using profile {profile.id}") @@ -349,9 +350,9 @@ def get_alternate_streams(channel_id: str, current_stream_id: Optional[int] = No existing_stream_id = redis_client.get(f"channel_stream:{channel.id}") if existing_stream_id: # Decode bytes to string/int for proper Redis key lookup - existing_stream_id = existing_stream_id.decode('utf-8') + existing_stream_id = existing_stream_id existing_profile_id = redis_client.get(f"stream_profile:{existing_stream_id}") - if existing_profile_id and int(existing_profile_id.decode('utf-8')) == profile.id: + if existing_profile_id and int(existing_profile_id) == profile.id: channel_using_profile = True logger.debug(f"Channel {channel.id} already using profile {profile.id}") diff --git a/apps/proxy/ts_proxy/utils.py b/apps/proxy/ts_proxy/utils.py index 20a6e140..ff9e1780 100644 --- a/apps/proxy/ts_proxy/utils.py +++ b/apps/proxy/ts_proxy/utils.py @@ -83,7 +83,7 @@ def create_ts_packet(packet_type='null', message=None): # Add message to payload if provided if message: - msg_bytes = message.encode('utf-8') + msg_bytes = message packet[4:4+min(len(msg_bytes), 180)] = msg_bytes[:180] return bytes(packet) @@ -113,4 +113,4 @@ def get_logger(component_name=None): # Default if detection fails logger_name = "ts_proxy" - return logging.getLogger(logger_name) \ No newline at end of file + return logging.getLogger(logger_name) diff --git a/apps/proxy/ts_proxy/views.py b/apps/proxy/ts_proxy/views.py index ccbb947d..5750616b 100644 --- a/apps/proxy/ts_proxy/views.py +++ b/apps/proxy/ts_proxy/views.py @@ -40,16 +40,20 @@ from .utils import get_logger from uuid import UUID import gevent from dispatcharr.utils import network_access_allowed +from apps.proxy.utils import check_user_stream_limits logger = get_logger() @api_view(["GET"]) -def stream_ts(request, channel_id): +def stream_ts(request, channel_id, user=None): if not network_access_allowed(request, "STREAMS"): return JsonResponse({"error": "Forbidden"}, status=403) """Stream TS data to client with immediate response and keep-alive packets during initialization""" + if user is None and hasattr(request, 'user') and request.user.is_authenticated: + user = request.user + channel = get_stream_object(channel_id) client_user_agent = None @@ -71,6 +75,13 @@ def stream_ts(request, channel_id): ) break + if user: + if not check_user_stream_limits(user, client_id, media_id=channel_id): + return JsonResponse( + {"error": f"Stream limit exceeded ({user.stream_limit} concurrent streams allowed)"}, + status=429 + ) + # Check if we need to reinitialize the channel needs_initialization = True channel_state = None @@ -81,9 +92,9 @@ def stream_ts(request, channel_id): metadata_key = RedisKeys.channel_metadata(channel_id) if proxy_server.redis_client.exists(metadata_key): metadata = proxy_server.redis_client.hgetall(metadata_key) - state_field = ChannelMetadataField.STATE.encode("utf-8") + state_field = ChannelMetadataField.STATE if state_field in metadata: - channel_state = metadata[state_field].decode("utf-8") + channel_state = metadata[state_field] # Active/running states - channel is operational, don't reinitialize if channel_state in [ @@ -119,9 +130,9 @@ def stream_ts(request, channel_id): ) # Unknown/empty state - check if owner is alive else: - owner_field = ChannelMetadataField.OWNER.encode("utf-8") + owner_field = ChannelMetadataField.OWNER if owner_field in metadata: - owner = metadata[owner_field].decode("utf-8") + owner = metadata[owner_field] owner_heartbeat_key = f"ts_proxy:worker:{owner}:heartbeat" if proxy_server.redis_client.exists(owner_heartbeat_key): # Owner is still active with unknown state - don't reinitialize @@ -399,7 +410,7 @@ def stream_ts(request, channel_id): metadata_key, ChannelMetadataField.STATE ) if state_bytes: - current_state = state_bytes.decode("utf-8") + current_state = state_bytes logger.debug( f"[{client_id}] Current state of channel {channel_id}: {current_state}" ) @@ -475,12 +486,12 @@ def stream_ts(request, channel_id): ) if url_bytes: - url = url_bytes.decode("utf-8") + url = url_bytes if ua_bytes: - stream_user_agent = ua_bytes.decode("utf-8") + stream_user_agent = ua_bytes # Extract transcode setting from Redis if profile_bytes: - profile_str = profile_bytes.decode("utf-8") + profile_str = profile_bytes use_transcode = ( profile_str == PROXY_PROFILE_NAME or profile_str == "None" ) @@ -516,12 +527,12 @@ def stream_ts(request, channel_id): # Register client buffer = proxy_server.stream_buffers[channel_id] client_manager = proxy_server.client_managers[channel_id] - client_manager.add_client(client_id, client_ip, client_user_agent) + client_manager.add_client(client_id, client_ip, client_user_agent, user) logger.info(f"[{client_id}] Client registered with channel {channel_id}") # Create a stream generator for this client generate = create_stream_generator( - channel_id, client_id, client_ip, client_user_agent, channel_initializing + channel_id, client_id, client_ip, client_user_agent, channel_initializing, user=user ) # Return the StreamingHttpResponse from the main function @@ -557,7 +568,6 @@ def stream_xc(request, username, password, channel_id): if custom_properties["xc_password"] != password: return Response({"error": "Invalid credentials"}, status=401) - print(f"Fetchin channel with ID: {channel_id}") if user.user_level < 10: user_profile_count = user.channel_profiles.count() @@ -585,7 +595,7 @@ def stream_xc(request, username, password, channel_id): channel = get_object_or_404(Channel, id=channel_id) # @TODO: we've got the file 'type' via extension, support this when we support multiple outputs - return stream_ts(request._request, str(channel.uuid)) + return stream_ts(request._request, str(channel.uuid), user) @csrf_exempt @@ -713,7 +723,7 @@ def channel_status(request, channel_id=None): ) for key in keys: channel_id_match = re.search( - r"ts_proxy:channel:(.*):metadata", key.decode("utf-8") + r"ts_proxy:channel:(.*):metadata", key ) if channel_id_match: ch_id = channel_id_match.group(1) @@ -834,7 +844,7 @@ def next_stream(request, channel_id): metadata_key, ChannelMetadataField.STREAM_ID ) if stream_id_bytes: - current_stream_id = int(stream_id_bytes.decode("utf-8")) + current_stream_id = int(stream_id_bytes) logger.info( f"Found current stream ID {current_stream_id} in Redis for channel {channel_id}" ) @@ -844,7 +854,7 @@ def next_stream(request, channel_id): metadata_key, ChannelMetadataField.M3U_PROFILE ) if profile_id_bytes: - profile_id = int(profile_id_bytes.decode("utf-8")) + profile_id = int(profile_id_bytes) logger.info( f"Found M3U profile ID {profile_id} in Redis for channel {channel_id}" ) diff --git a/apps/proxy/urls.py b/apps/proxy/urls.py index 34c026a9..3a320049 100644 --- a/apps/proxy/urls.py +++ b/apps/proxy/urls.py @@ -6,4 +6,4 @@ urlpatterns = [ path('ts/', include('apps.proxy.ts_proxy.urls')), path('hls/', include('apps.proxy.hls_proxy.urls')), path('vod/', include('apps.proxy.vod_proxy.urls')), -] \ No newline at end of file +] diff --git a/apps/proxy/utils.py b/apps/proxy/utils.py new file mode 100644 index 00000000..6503d1df --- /dev/null +++ b/apps/proxy/utils.py @@ -0,0 +1,185 @@ +import logging +from core.utils import RedisClient +from apps.proxy.vod_proxy.multi_worker_connection_manager import MultiWorkerVODConnectionManager, get_vod_client_stop_key +from core.models import CoreSettings +from apps.proxy.ts_proxy.services.channel_service import ChannelService + +logger = logging.getLogger("proxy") + + +def attempt_stream_termination(user_id, requesting_client_id, active_connections): + try: + logger.info("[stream limits]" f"[{requesting_client_id}] User {user_id} has {len(active_connections)} active connections, checking termination candidates") + + user_limit_settings = CoreSettings.get_user_limits_settings() + terminate_oldest = user_limit_settings.get("terminate_oldest", True) + prioritize_single = user_limit_settings.get("prioritize_single_client_channels", True) + ignore_same_channel = user_limit_settings.get("ignore_same_channel_connections", False) + + channel_counts = {} + for connection in active_connections: + media_id = connection['media_id'] + channel_counts[media_id] = channel_counts.get(media_id, 0) + 1 + + def prioritize(connection): + is_multi = channel_counts[connection['media_id']] > 1 + + # if we're ignoring same-channel connections, put them at the end + same_ch_key = 1 if (ignore_same_channel and is_multi) else 0 + + # key for prioritizing single-client channels + single_key = 0 if (prioritize_single and not is_multi) else 1 + + # sort by age setting + time_key = connection['connected_at'] if terminate_oldest else -connection['connected_at'] + + return (same_ch_key, single_key, time_key) + + termination_candidates = sorted(active_connections, key=prioritize) + + if not termination_candidates: + logger.warning("[stream limits]" f"[{requesting_client_id}] No termination candidates found for user {user_id}") + return False + + target = termination_candidates[0] + logger.info("[stream limits]" + f"[{requesting_client_id}] Terminating client {target['client_id']} " + f"on media {target['media_id']} (connected_at={target['connected_at']})" + ) + + # When counting by unique channel, freeing one connection from a multi-connection + # channel doesn't free a slot — terminate all connections to that channel so the + # unique-channel count actually drops by one. + targets = ( + [c for c in active_connections if c['media_id'] == target['media_id']] + if ignore_same_channel + else [target] + ) + + for t in targets: + if t['type'] == 'live': + result = ChannelService.stop_client(t['media_id'], t['client_id']) + if result.get("status") == "error": + logger.warning(f"[stream limits][{requesting_client_id}] Failed to stop client {t['client_id']} on channel {t['media_id']}") + else: + connection_manager = MultiWorkerVODConnectionManager.get_instance() + redis_client = connection_manager.redis_client + + if not redis_client: + return False + + connection_key = f"vod_persistent_connection:{t['client_id']}" + connection_data = redis_client.hgetall(connection_key) + if not connection_data: + logger.warning(f"VOD connection not found: {t['client_id']}") + continue + + stop_key = get_vod_client_stop_key(t['client_id']) + redis_client.setex(stop_key, 60, "true") # 60 second TTL + + return True + except Exception as e: + logger.error("[stream limits]" f"[{requesting_client_id}] Error during stream termination for user {user_id}: {e}") + return False + +def get_user_active_connections(user_id): + redis_client = RedisClient.get_client() + connections = [] + + try: + # Grab live streams + for key in redis_client.scan_iter(match="ts_proxy:channel:*:clients:*", count=1000): + parts = key.split(':') + if len(parts) >= 5: + channel_id = parts[2] + client_id = parts[4] + + client_user_id = redis_client.hget(key, 'user_id') + connected_at = redis_client.hget(key, 'connected_at') + + logger.debug(f"[stream limits] user_id = {user_id}") + logger.debug(f"[stream limits] channel_id = {channel_id}") + logger.debug(f"[stream limits] client_id = {client_id}") + + if client_user_id and int(client_user_id) == user_id: + try: + logger.debug(f"[stream limits] Found LIVE connection for user {user_id} on channel {channel_id} with client ID {client_id}") + connected_at = float(connected_at) if connected_at else 0 + connections.append({ + 'media_id': channel_id, + 'client_id': client_id, + 'connected_at': connected_at, + 'type': 'live', + }) + except (ValueError, TypeError): + pass + + # Grab VOD + for key in redis_client.scan_iter(match="vod_persistent_connection:*", count=1000): + parts = key.split(':') + if len(parts) >= 2: + client_id = parts[1] + + client_user_id = redis_client.hget(key, 'user_id') + connected_at = redis_client.hget(key, 'created_at') + content_uuid = redis_client.hget(key, 'content_uuid') + + logger.debug(f"[stream limits] user_id = {user_id}") + logger.debug(f"[stream limits] client_id = {client_id}") + + if client_user_id and int(client_user_id) == user_id: + try: + logger.debug(f"[stream limits] Found VOD connection for user {user_id} on content {content_uuid} with client ID {client_id}") + connected_at = float(connected_at) if connected_at else 0 + connections.append({ + 'media_id': content_uuid or client_id, + 'client_id': client_id, + 'connected_at': connected_at, + 'type': 'vod', + }) + except (ValueError, TypeError): + pass + + return connections + + except Exception as e: + logger.warning(f"Error getting active channel details for user {user_id}: {e}") + return [] + + +def check_user_stream_limits(user, client_id, media_id=None): + # Check user stream limits + if user and user.stream_limit > 0: + logger.debug("[stream limits]" f"[{client_id}] User {user.username} (ID: {user.id}) is requesting a stream (stream_limit: {user.stream_limit})") + user_limit_settings = CoreSettings.get_user_limits_settings() + ignore_same_channel = user_limit_settings.get("ignore_same_channel_connections", False) + + active_connections = get_user_active_connections(user.id) + unique_channel_count = set([conn['media_id'] for conn in active_connections]) + user_stream_count = len(unique_channel_count) if ignore_same_channel else len(active_connections) + + logger.debug(f"[stream limits]" f"[{client_id}] User {user.username} currently has {len(active_connections)} active connections across {len(unique_channel_count)} unique channels (counting method: {'unique channels' if ignore_same_channel else 'total connections'})") + + # If ignore_same_channel is enabled and this request is for a live channel the user + # is already watching, allow it through without counting against the limit. + # VOD is excluded: connections aren't shared so multiple VOD connections to the + # same content would mean multiple upstream connections. + live_channel_ids = {str(conn['media_id']) for conn in active_connections if conn['type'] == 'live'} + if ignore_same_channel and media_id and str(media_id) in live_channel_ids: + logger.debug(f"[stream limits][{client_id}] Same-channel reconnect for {media_id} allowed (ignore_same_channel=True)") + return True + + if user_stream_count >= user.stream_limit: + if user_limit_settings.get("terminate_on_limit_exceeded", True) == False: + return False + + if user_stream_count >= user.stream_limit: + logger.warning("[stream limits]" + f"[{client_id}] User {user.username} (ID: {user.id}) has reached stream limit " + f"({user_stream_count}/{user.stream_limit} streams), attempting to free up slot" + ) + + if not attempt_stream_termination(user.id, client_id, active_connections): + return False + + return True diff --git a/apps/proxy/vod_proxy/connection_manager.py b/apps/proxy/vod_proxy/connection_manager.py index 7bfccfca..1692c315 100644 --- a/apps/proxy/vod_proxy/connection_manager.py +++ b/apps/proxy/vod_proxy/connection_manager.py @@ -103,13 +103,15 @@ class PersistentVODConnection: redis_db = int(getattr(settings, 'REDIS_DB', 0)) redis_password = getattr(settings, 'REDIS_PASSWORD', '') redis_user = getattr(settings, 'REDIS_USER', '') + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) r = redis.StrictRedis( host=redis_host, port=redis_port, db=redis_db, password=redis_password if redis_password else None, username=redis_user if redis_user else None, - decode_responses=True + decode_responses=True, + **ssl_params ) content_length_key = f"vod_content_length:{self.session_id}" stored_length = r.get(content_length_key) @@ -342,15 +344,15 @@ class VODConnectionManager: continue # Extract session info - stored_content_type = session_data.get(b'content_type', b'').decode('utf-8') - stored_content_uuid = session_data.get(b'content_uuid', b'').decode('utf-8') + stored_content_type = session_data.get('content_type', '') + stored_content_uuid = session_data.get('content_uuid', '') # Check if content matches if stored_content_type != content_type or stored_content_uuid != content_uuid: continue # Extract session ID from key - session_id = key.decode('utf-8').replace('vod_session:', '') + session_id = key.replace('vod_session:', '') # Check if session has an active persistent connection persistent_conn = self._persistent_connections.get(session_id) @@ -364,13 +366,13 @@ class VODConnectionManager: continue # Get stored client info for comparison - stored_client_ip = session_data.get(b'client_ip', b'').decode('utf-8') - stored_user_agent = session_data.get(b'user_agent', b'').decode('utf-8') + stored_client_ip = session_data.get('client_ip', '') + stored_user_agent = session_data.get('user_agent', '') # Check timeshift parameters match - stored_utc_start = session_data.get(b'utc_start', b'').decode('utf-8') - stored_utc_end = session_data.get(b'utc_end', b'').decode('utf-8') - stored_offset = session_data.get(b'offset', b'').decode('utf-8') + stored_utc_start = session_data.get('utc_start', '') + stored_utc_end = session_data.get('utc_end', '') + stored_offset = session_data.get('offset', '') current_utc_start = utc_start or "" current_utc_end = utc_end or "" @@ -407,7 +409,7 @@ class VODConnectionManager: 'session_id': session_id, 'score': score, 'reasons': match_reasons, - 'last_activity': float(session_data.get(b'last_activity', b'0').decode('utf-8')) + 'last_activity': float(session_data.get('last_activity', '0')) }) except Exception as e: @@ -588,7 +590,7 @@ class VODConnectionManager: # Get current bytes and add to it current_bytes = self.redis_client.hget(connection_key, "bytes_sent") if current_bytes: - total_bytes = int(current_bytes.decode('utf-8')) + bytes_sent + total_bytes = int(current_bytes) + bytes_sent else: total_bytes = bytes_sent update_data["bytes_sent"] = str(total_bytes) @@ -623,7 +625,7 @@ class VODConnectionManager: profile_id = None if b"m3u_profile_id" in connection_data: try: - profile_id = int(connection_data[b"m3u_profile_id"].decode('utf-8')) + profile_id = int(connection_data["m3u_profile_id"]) except ValueError: pass @@ -669,16 +671,14 @@ class VODConnectionManager: # Convert bytes to strings and parse numbers info = {} for key, value in connection_data.items(): - key_str = key.decode('utf-8') - value_str = value.decode('utf-8') # Parse numeric fields - if key_str in ['connected_at', 'last_activity']: - info[key_str] = float(value_str) - elif key_str in ['bytes_sent', 'position_seconds', 'm3u_profile_id']: - info[key_str] = int(value_str) + if key in ['connected_at', 'last_activity']: + info[key] = float(value) + elif key in ['bytes_sent', 'position_seconds', 'm3u_profile_id']: + info[key] = int(valuvaluee_str) else: - info[key_str] = value_str + info[key] = value return info @@ -728,14 +728,13 @@ class VODConnectionManager: for key in keys: try: - key_str = key.decode('utf-8') last_activity = self.redis_client.hget(key, "last_activity") if last_activity: - last_activity_time = float(last_activity.decode('utf-8')) + last_activity_time = float(last_activity) if current_time - last_activity_time > max_age_seconds: # Extract info for cleanup - parts = key_str.split(':') + parts = key.split(':') if len(parts) >= 5: content_type = parts[2] content_uuid = parts[3] @@ -1009,7 +1008,7 @@ class VODConnectionManager: return HttpResponse(f"Streaming error: {str(e)}", status=500) def stream_content_with_session(self, session_id, content_obj, stream_url, m3u_profile, client_ip, user_agent, request, - utc_start=None, utc_end=None, offset=None, range_header=None): + utc_start=None, utc_end=None, offset=None, range_header=None, user=None): """ Stream VOD content with persistent connection per session @@ -1394,9 +1393,9 @@ class VODConnectionManager: session_data = self.redis_client.hgetall(session_key) if session_data: # Get session details for connection cleanup - content_type = session_data.get(b'content_type', b'').decode('utf-8') - content_uuid = session_data.get(b'content_uuid', b'').decode('utf-8') - profile_id = session_data.get(b'profile_id') + content_type = session_data.get('content_type', '') + content_uuid = session_data.get('content_uuid', '') + profile_id = session_data.get('profile_id') # Generate client_id from session_id (matches what's used during streaming) client_id = session_id @@ -1415,12 +1414,12 @@ class VODConnectionManager: # Fallback DECR: only if the connection tracking key had already # expired (remove_connection skips DECR in that case) if not connection_key_exists: - if session_data.get(b'connection_counted') == b'True' and profile_id: - profile_key = self._get_profile_connections_key(int(profile_id.decode('utf-8'))) + if session_data.get('connection_counted') == 'True' and profile_id: + profile_key = self._get_profile_connections_key(int(profile_id)) current_count = int(self.redis_client.get(profile_key) or 0) if current_count > 0: self.redis_client.decr(profile_key) - logger.info(f"[{session_id}] Decremented profile {profile_id.decode('utf-8')} connections (fallback)") + logger.info(f"[{session_id}] Decremented profile {profile_id} connections (fallback)") # Remove session tracking key self.redis_client.delete(session_key) @@ -1440,7 +1439,7 @@ class VODConnectionManager: if session_related_keys: # Filter out keys we already deleted - remaining_keys = [k for k in session_related_keys if k.decode('utf-8') != session_key] + remaining_keys = [k for k in session_related_keys if k != session_key] if remaining_keys: self.redis_client.delete(*remaining_keys) logger.info(f"[{session_id}] Cleaned up {len(remaining_keys)} additional session-related keys") @@ -1470,7 +1469,7 @@ class VODConnectionManager: if self.redis_client: session_data = self.redis_client.hgetall(session_key) if session_data: - created_at = float(session_data.get(b'created_at', b'0').decode('utf-8')) + created_at = float(session_data.get('created_at', '0')) if current_time - created_at > max_age_seconds: logger.info(f"[{session_id}] Session older than {max_age_seconds}s") stale_sessions.append(session_id) diff --git a/apps/proxy/vod_proxy/multi_worker_connection_manager.py b/apps/proxy/vod_proxy/multi_worker_connection_manager.py index 6b048529..d7787016 100644 --- a/apps/proxy/vod_proxy/multi_worker_connection_manager.py +++ b/apps/proxy/vod_proxy/multi_worker_connection_manager.py @@ -93,7 +93,7 @@ class SerializableConnectionState: content_name: str = None, client_ip: str = None, client_user_agent: str = None, utc_start: str = None, utc_end: str = None, offset: str = None, - worker_id: str = None, connection_type: str = "redis_backed"): + worker_id: str = None, connection_type: str = "redis_backed", user_id: str = "unknown"): self.session_id = session_id self.stream_url = stream_url self.headers = headers @@ -104,6 +104,7 @@ class SerializableConnectionState: self.last_activity = time.time() self.request_count = 0 self.active_streams = 0 + self.user_id = user_id # Session metadata (consolidated from vod_session key) self.content_obj_type = content_obj_type @@ -160,7 +161,8 @@ class SerializableConnectionState: 'last_seek_byte': str(self.last_seek_byte), 'last_seek_percentage': str(self.last_seek_percentage), 'total_content_size': str(self.total_content_size), - 'last_seek_timestamp': str(self.last_seek_timestamp) + 'last_seek_timestamp': str(self.last_seek_timestamp), + 'user_id': str(self.user_id), } @classmethod @@ -184,7 +186,8 @@ class SerializableConnectionState: utc_end=data.get('utc_end') or '', offset=data.get('offset') or '', worker_id=data.get('worker_id') or None, - connection_type=data.get('connection_type', 'redis_backed') + connection_type=data.get('connection_type', 'redis_backed'), + user_id=data.get('user_id', 'unknown') ) obj.last_activity = float(data.get('last_activity', time.time())) obj.request_count = int(data.get('request_count', 0)) @@ -224,7 +227,7 @@ class RedisBackedVODConnection: # Convert bytes keys/values to strings if needed if isinstance(list(data.keys())[0], bytes): - data = {k.decode('utf-8'): v.decode('utf-8') for k, v in data.items()} + data = {k: v for k, v in data.items()} return SerializableConnectionState.from_dict(data) except Exception as e: @@ -281,7 +284,7 @@ class RedisBackedVODConnection: content_name: str = None, client_ip: str = None, client_user_agent: str = None, utc_start: str = None, utc_end: str = None, offset: str = None, - worker_id: str = None) -> bool: + worker_id: str = None, user=None) -> bool: """Create a new connection state in Redis with consolidated session metadata""" if not self._acquire_lock(): logger.warning(f"[{self.session_id}] Could not acquire lock for connection creation") @@ -309,7 +312,8 @@ class RedisBackedVODConnection: utc_start=utc_start, utc_end=utc_end, offset=offset, - worker_id=worker_id + worker_id=worker_id, + user_id=user.id if user else "unknown" ) success = self._save_connection_state(state) @@ -365,6 +369,24 @@ class RedisBackedVODConnection: timeout=(10, 10), allow_redirects=allow_redirects ) + + # If the cached final_url returned an error (e.g. an ephemeral dispatcharr session + # that has since expired), clear it and retry from the original stream_url. + if response.status_code >= 400 and state.final_url: + logger.warning( + f"[{self.session_id}] Cached final_url returned {response.status_code}, " + f"clearing and retrying from stream_url" + ) + response.close() + state.final_url = None + response = self.local_session.get( + state.stream_url, + headers=headers, + stream=True, + timeout=(10, 10), + allow_redirects=True + ) + response.raise_for_status() # Update state with response info on first request @@ -761,7 +783,7 @@ class MultiWorkerVODConnectionManager: def stream_content_with_session(self, session_id, content_obj, stream_url, m3u_profile, client_ip, client_user_agent, request, - utc_start=None, utc_end=None, offset=None, range_header=None): + utc_start=None, utc_end=None, offset=None, range_header=None, user=None): """Stream content with Redis-backed persistent connection""" # Generate client ID @@ -858,7 +880,8 @@ class MultiWorkerVODConnectionManager: utc_start=utc_start, utc_end=utc_end, offset=str(offset) if offset else None, - worker_id=self.worker_id + worker_id=self.worker_id, + user=user ): logger.error(f"[{client_id}] Worker {self.worker_id} - Failed to create Redis connection") # Roll back the profile slot reservation since connection failed @@ -1272,14 +1295,14 @@ class MultiWorkerVODConnectionManager: # Convert bytes to strings if needed if isinstance(list(data.keys())[0], bytes): - data = {k.decode('utf-8'): v.decode('utf-8') for k, v in data.items()} + data = {k: v for k, v in data.items()} last_activity = float(data.get('last_activity', 0)) active_streams = int(data.get('active_streams', 0)) # Clean up if stale and no active streams if (current_time - last_activity > max_age_seconds) and active_streams == 0: - session_id = key.decode('utf-8').replace('vod_persistent_connection:', '') + session_id = key.replace('vod_persistent_connection:', '') logger.info(f"Cleaning up stale connection: {session_id}") # Clean up connection and related keys @@ -1376,7 +1399,7 @@ class MultiWorkerVODConnectionManager: if connection_data: # Convert bytes to strings if needed if isinstance(list(connection_data.keys())[0], bytes): - connection_data = {k.decode('utf-8'): v.decode('utf-8') for k, v in connection_data.items()} + connection_data = {k: v for k, v in connection_data.items()} profile_id = connection_data.get('m3u_profile_id') if profile_id: @@ -1438,7 +1461,7 @@ class MultiWorkerVODConnectionManager: # Convert bytes keys/values to strings if needed if isinstance(list(connection_data.keys())[0], bytes): - connection_data = {k.decode('utf-8'): v.decode('utf-8') for k, v in connection_data.items()} + connection_data = {k: v for k, v in connection_data.items()} # Check if content matches (using consolidated data) stored_content_type = connection_data.get('content_obj_type', '') @@ -1448,7 +1471,7 @@ class MultiWorkerVODConnectionManager: continue # Extract session ID - session_id = key.decode('utf-8').replace('vod_persistent_connection:', '') + session_id = key.replace('vod_persistent_connection:', '') # Check if Redis-backed connection exists and has no active streams redis_connection = RedisBackedVODConnection(session_id, self.redis_client) @@ -1526,4 +1549,4 @@ class MultiWorkerVODConnectionManager: return redis_connection.get_session_metadata() except Exception as e: logger.error(f"Error getting session info for {session_id}: {e}") - return None \ No newline at end of file + return None diff --git a/apps/proxy/vod_proxy/urls.py b/apps/proxy/vod_proxy/urls.py index f48f70e0..f79e6b48 100644 --- a/apps/proxy/vod_proxy/urls.py +++ b/apps/proxy/vod_proxy/urls.py @@ -1,16 +1,17 @@ from django.urls import path from . import views +from .views import stream_vod app_name = 'vod_proxy' urlpatterns = [ # Generic VOD streaming with session ID in path (for compatibility) - path('//', views.VODStreamView.as_view(), name='vod_stream_with_session'), - path('////', views.VODStreamView.as_view(), name='vod_stream_with_session_and_profile'), + path('//', stream_vod, name='vod_stream_with_session'), + path('////', stream_vod, name='vod_stream_with_session_and_profile'), # Generic VOD streaming (supports movies, episodes, series) - legacy patterns - path('/', views.VODStreamView.as_view(), name='vod_stream'), - path('///', views.VODStreamView.as_view(), name='vod_stream_with_profile'), + path('/', stream_vod, name='vod_stream'), + path('///', stream_vod, name='vod_stream_with_profile'), # VOD playlist generation path('playlist/', views.VODPlaylistView.as_view(), name='vod_playlist'), diff --git a/apps/proxy/vod_proxy/views.py b/apps/proxy/vod_proxy/views.py index 948604d6..e6b6e144 100644 --- a/apps/proxy/vod_proxy/views.py +++ b/apps/proxy/vod_proxy/views.py @@ -17,104 +17,379 @@ from apps.m3u.models import M3UAccount, M3UAccountProfile from apps.proxy.vod_proxy.connection_manager import VODConnectionManager from apps.proxy.vod_proxy.multi_worker_connection_manager import MultiWorkerVODConnectionManager, infer_content_type_from_url, get_vod_client_stop_key from .utils import get_client_info, create_vod_response +from rest_framework.decorators import api_view +from apps.accounts.models import User +from apps.proxy.utils import check_user_stream_limits logger = logging.getLogger(__name__) +_request_times = {} -@method_decorator(csrf_exempt, name='dispatch') -class VODStreamView(View): - """Handle VOD streaming requests with M3U profile support""" - def get(self, request, content_type, content_id, session_id=None, profile_id=None): - """ - Stream VOD content (movies or series episodes) with session-based connection reuse +def _get_content_and_relation(content_type, content_id, preferred_m3u_account_id=None, preferred_stream_id=None): + """Get the content object and its M3U relation""" + try: + logger.info(f"[CONTENT-LOOKUP] Looking up {content_type} with UUID {content_id}") + if preferred_m3u_account_id: + logger.info(f"[CONTENT-LOOKUP] Preferred M3U account ID: {preferred_m3u_account_id}") + if preferred_stream_id: + logger.info(f"[CONTENT-LOOKUP] Preferred stream ID: {preferred_stream_id}") - Args: - content_type: 'movie', 'series', or 'episode' - content_id: ID of the content - session_id: Optional session ID from URL path (for persistent connections) - profile_id: Optional M3U profile ID for authentication - """ - logger.info(f"[VOD-REQUEST] Starting VOD stream request: {content_type}/{content_id}, session: {session_id}, profile: {profile_id}") - logger.info(f"[VOD-REQUEST] Full request path: {request.get_full_path()}") - logger.info(f"[VOD-REQUEST] Request method: {request.method}") - logger.info(f"[VOD-REQUEST] Request headers: {dict(request.headers)}") + if content_type == 'movie': + content_obj = get_object_or_404(Movie, uuid=content_id) + logger.info(f"[CONTENT-FOUND] Movie: {content_obj.name} (ID: {content_obj.id})") - try: - client_ip, client_user_agent = get_client_info(request) + # Filter by preferred stream ID first (most specific) + relations_query = content_obj.m3u_relations.filter(m3u_account__is_active=True) + if preferred_stream_id: + specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() + if specific_relation: + logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") + return content_obj, specific_relation + else: + logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") - # Extract timeshift parameters from query string - # Support multiple timeshift parameter formats - utc_start = request.GET.get('utc_start') or request.GET.get('start') or request.GET.get('playliststart') - utc_end = request.GET.get('utc_end') or request.GET.get('end') or request.GET.get('playlistend') - offset = request.GET.get('offset') or request.GET.get('seek') or request.GET.get('t') + # Filter by preferred M3U account if specified + if preferred_m3u_account_id: + specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() + if specific_relation: + logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") + return content_obj, specific_relation + else: + logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") - # VLC specific timeshift parameters - if not utc_start and not offset: - # Check for VLC-style timestamp parameters - if 'timestamp' in request.GET: - offset = request.GET.get('timestamp') - elif 'time' in request.GET: - offset = request.GET.get('time') + # Get the highest priority active relation (fallback or default) + relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() - # Session ID now comes from URL path parameter - # Remove legacy query parameter extraction since we're using path-based routing + if relation: + logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") - # Extract Range header for seeking support - range_header = request.META.get('HTTP_RANGE') + return content_obj, relation - logger.info(f"[VOD-TIMESHIFT] Timeshift params - utc_start: {utc_start}, utc_end: {utc_end}, offset: {offset}") - logger.info(f"[VOD-SESSION] Session ID: {session_id}") + elif content_type == 'episode': + content_obj = get_object_or_404(Episode, uuid=content_id) + logger.info(f"[CONTENT-FOUND] Episode: {content_obj.name} (ID: {content_obj.id}, Series: {content_obj.series.name})") - # Log all query parameters for debugging - if request.GET: - logger.debug(f"[VOD-PARAMS] All query params: {dict(request.GET)}") + # Filter by preferred stream ID first (most specific) + relations_query = content_obj.m3u_relations.filter(m3u_account__is_active=True) + if preferred_stream_id: + specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() + if specific_relation: + logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") + return content_obj, specific_relation + else: + logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") - if range_header: - logger.info(f"[VOD-RANGE] Range header: {range_header}") + # Filter by preferred M3U account if specified + if preferred_m3u_account_id: + specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() + if specific_relation: + logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") + return content_obj, specific_relation + else: + logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") - # Parse the range to understand what position VLC is seeking to - try: - if 'bytes=' in range_header: - range_part = range_header.replace('bytes=', '') - if '-' in range_part: - start_byte, end_byte = range_part.split('-', 1) - if start_byte: - start_pos_mb = int(start_byte) / (1024 * 1024) - logger.info(f"[VOD-SEEK] Seeking to byte position: {start_byte} (~{start_pos_mb:.1f} MB)") - if int(start_byte) > 0: - logger.info(f"[VOD-SEEK] *** ACTUAL SEEK DETECTED *** Position: {start_pos_mb:.1f} MB") - else: - logger.info(f"[VOD-SEEK] Open-ended range request (from start)") - if end_byte: - end_pos_mb = int(end_byte) / (1024 * 1024) - logger.info(f"[VOD-SEEK] End position: {end_byte} bytes (~{end_pos_mb:.1f} MB)") - except Exception as e: - logger.warning(f"[VOD-SEEK] Could not parse range header: {e}") + # Get the highest priority active relation (fallback or default) + relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() - # Simple seek detection - track rapid requests - current_time = time.time() - request_key = f"{client_ip}:{content_type}:{content_id}" + if relation: + logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") - if not hasattr(self.__class__, '_request_times'): - self.__class__._request_times = {} + return content_obj, relation - if request_key in self.__class__._request_times: - time_diff = current_time - self.__class__._request_times[request_key] - if time_diff < 5.0: - logger.info(f"[VOD-SEEK] Rapid request detected ({time_diff:.1f}s) - likely seeking") + elif content_type == 'series': + # For series, get the first episode + series = get_object_or_404(Series, uuid=content_id) + logger.info(f"[CONTENT-FOUND] Series: {series.name} (ID: {series.id})") + episode = series.episodes.first() + if not episode: + logger.error(f"[CONTENT-ERROR] No episodes found for series {series.name}") + return None, None - self.__class__._request_times[request_key] = current_time + logger.info(f"[CONTENT-FOUND] First episode: {episode.name} (ID: {episode.id})") + + # Filter by preferred stream ID first (most specific) + relations_query = episode.m3u_relations.filter(m3u_account__is_active=True) + if preferred_stream_id: + specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() + if specific_relation: + logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") + return episode, specific_relation + else: + logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") + + # Filter by preferred M3U account if specified + if preferred_m3u_account_id: + specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() + if specific_relation: + logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") + return episode, specific_relation + else: + logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") + + # Get the highest priority active relation (fallback or default) + relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() + + if relation: + logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") + + return episode, relation + else: + logger.error(f"[CONTENT-ERROR] Invalid content type: {content_type}") + return None, None + + except Exception as e: + logger.error(f"Error getting content object: {e}") + return None, None + +def _get_stream_url_from_relation(relation): + """Get stream URL from the M3U relation""" + try: + # Log the relation type and available attributes + logger.info(f"[VOD-URL] Relation type: {type(relation).__name__}") + logger.info(f"[VOD-URL] Account type: {relation.m3u_account.account_type}") + logger.info(f"[VOD-URL] Stream ID: {getattr(relation, 'stream_id', 'N/A')}") + + # First try the get_stream_url method (this should build URLs dynamically) + if hasattr(relation, 'get_stream_url'): + url = relation.get_stream_url() + if url: + logger.info(f"[VOD-URL] Built URL from get_stream_url(): {url}") + return url else: - logger.info(f"[VOD-RANGE] No Range header - full content request") + logger.warning(f"[VOD-URL] get_stream_url() returned None") - logger.info(f"[VOD-CLIENT] Client info - IP: {client_ip}, User-Agent: {client_user_agent[:50]}...") + logger.error(f"[VOD-URL] Relation has no get_stream_url method or it failed") + return None + except Exception as e: + logger.error(f"[VOD-URL] Error getting stream URL from relation: {e}", exc_info=True) + return None - # If no session ID, create one and redirect to path-based URL - if not session_id: - new_session_id = f"vod_{int(time.time() * 1000)}_{random.randint(1000, 9999)}" - logger.info(f"[VOD-SESSION] Creating new session: {new_session_id}") +def _get_m3u_profile(m3u_account, profile_id, session_id=None): + """Get appropriate M3U profile for streaming using Redis-based viewer counts + Args: + m3u_account: M3UAccount instance + profile_id: Optional specific profile ID requested + session_id: Optional session ID to check for existing connections + + Returns: + tuple: (M3UAccountProfile, current_connections) or None if no profile found + """ + try: + from core.utils import RedisClient + redis_client = RedisClient.get_client() + + if not redis_client: + logger.warning("Redis not available, falling back to default profile") + default_profile = M3UAccountProfile.objects.filter( + m3u_account=m3u_account, + is_active=True, + is_default=True + ).first() + return (default_profile, 0) if default_profile else None + + # Check if this session already has an active connection + if session_id: + persistent_connection_key = f"vod_persistent_connection:{session_id}" + connection_data = redis_client.hgetall(persistent_connection_key) + + if connection_data: + existing_profile_id = connection_data.get('m3u_profile_id') + if existing_profile_id: + try: + existing_profile = M3UAccountProfile.objects.get( + id=int(existing_profile_id), + m3u_account=m3u_account, + is_active=True + ) + # Get current connections for logging + profile_connections_key = f"profile_connections:{existing_profile.id}" + current_connections = int(redis_client.get(profile_connections_key) or 0) + + logger.info(f"[PROFILE-SELECTION] Session {session_id} reusing existing profile {existing_profile.id}: {current_connections}/{existing_profile.max_streams} connections") + return (existing_profile, current_connections) + except (M3UAccountProfile.DoesNotExist, ValueError): + logger.warning(f"[PROFILE-SELECTION] Session {session_id} has invalid profile ID {existing_profile_id}, selecting new profile") + except Exception as e: + logger.warning(f"[PROFILE-SELECTION] Error checking existing profile for session {session_id}: {e}") + else: + logger.debug(f"[PROFILE-SELECTION] Session {session_id} exists but has no profile ID stored") # If specific profile requested, try to use it + if profile_id: + try: + profile = M3UAccountProfile.objects.get( + id=profile_id, + m3u_account=m3u_account, + is_active=True + ) + # Check Redis-based current connections + profile_connections_key = f"profile_connections:{profile.id}" + current_connections = int(redis_client.get(profile_connections_key) or 0) + + if profile.max_streams == 0 or current_connections < profile.max_streams: + logger.info(f"[PROFILE-SELECTION] Using requested profile {profile.id}: {current_connections}/{profile.max_streams} connections") + return (profile, current_connections) + else: + logger.warning(f"[PROFILE-SELECTION] Requested profile {profile.id} is at capacity: {current_connections}/{profile.max_streams}") + except M3UAccountProfile.DoesNotExist: + logger.warning(f"[PROFILE-SELECTION] Requested profile {profile_id} not found") + + # Get active profiles ordered by priority (default first) + m3u_profiles = M3UAccountProfile.objects.filter( + m3u_account=m3u_account, + is_active=True + ) + + default_profile = m3u_profiles.filter(is_default=True).first() + if not default_profile: + logger.error(f"[PROFILE-SELECTION] No default profile found for M3U account {m3u_account.id}") + return None + + # Check profiles in order: default first, then others + profiles = [default_profile] + list(m3u_profiles.filter(is_default=False)) + + for profile in profiles: + profile_connections_key = f"profile_connections:{profile.id}" + current_connections = int(redis_client.get(profile_connections_key) or 0) + + # Check if profile has available connection slots + if profile.max_streams == 0 or current_connections < profile.max_streams: + logger.info(f"[PROFILE-SELECTION] Selected profile {profile.id} ({profile.name}): {current_connections}/{profile.max_streams} connections") + return (profile, current_connections) + else: + logger.debug(f"[PROFILE-SELECTION] Profile {profile.id} at capacity: {current_connections}/{profile.max_streams}") + + # All profiles are at capacity - return None to trigger error response + logger.error(f"[PROFILE-SELECTION] All profiles at capacity for M3U account {m3u_account.id}, rejecting request") + return None + + except Exception as e: + logger.error(f"Error getting M3U profile: {e}") + return None + +def _transform_url(original_url, m3u_profile): + """Transform URL based on M3U profile settings""" + try: + import regex + + if not original_url: + return None + + search_pattern = m3u_profile.search_pattern + replace_pattern = m3u_profile.replace_pattern + # Convert JS-style backreferences in replace: $ -> \g, $1 -> \1 + safe_replace_pattern = regex.sub(r'\$<([^>]+)>', r'\\g<\1>', replace_pattern) + safe_replace_pattern = regex.sub(r'\$(\d+)', r'\\\1', safe_replace_pattern) + + if search_pattern and replace_pattern: + # regex module accepts JS-style (?...) named groups natively + transformed_url = regex.sub(search_pattern, safe_replace_pattern, original_url) + return transformed_url + + return original_url + + except Exception as e: + logger.error(f"Error transforming URL: {e}") + return original_url + +@api_view(["GET"]) +def stream_vod(request, content_type, content_id, session_id=None, profile_id=None, user=None): + """ + Stream VOD content (movies or series episodes) with session-based connection reuse + + Args: + content_type: 'movie', 'series', or 'episode' + content_id: ID of the content + session_id: Optional session ID from URL path (for persistent connections) + profile_id: Optional M3U profile ID for authentication + """ + logger.info(f"[VOD-REQUEST] Starting VOD stream request: {content_type}/{content_id}, session: {session_id}, profile: {profile_id}") + logger.info(f"[VOD-REQUEST] Full request path: {request.get_full_path()}") + logger.info(f"[VOD-REQUEST] Request method: {request.method}") + logger.info(f"[VOD-REQUEST] Request headers: {dict(request.headers)}") + + try: + client_ip, client_user_agent = get_client_info(request) + + # Extract timeshift parameters from query string + # Support multiple timeshift parameter formats + utc_start = request.GET.get('utc_start') or request.GET.get('start') or request.GET.get('playliststart') + utc_end = request.GET.get('utc_end') or request.GET.get('end') or request.GET.get('playlistend') + offset = request.GET.get('offset') or request.GET.get('seek') or request.GET.get('t') + + # VLC specific timeshift parameters + if not utc_start and not offset: + # Check for VLC-style timestamp parameters + if 'timestamp' in request.GET: + offset = request.GET.get('timestamp') + elif 'time' in request.GET: + offset = request.GET.get('time') + + # Session ID now comes from URL path parameter + # Remove legacy query parameter extraction since we're using path-based routing + + # Extract Range header for seeking support + range_header = request.META.get('HTTP_RANGE') + + logger.info(f"[VOD-TIMESHIFT] Timeshift params - utc_start: {utc_start}, utc_end: {utc_end}, offset: {offset}") + logger.info(f"[VOD-SESSION] Session ID: {session_id}") + + # Log all query parameters for debugging + if request.GET: + logger.debug(f"[VOD-PARAMS] All query params: {dict(request.GET)}") + + if range_header: + logger.info(f"[VOD-RANGE] Range header: {range_header}") + + # Parse the range to understand what position VLC is seeking to + try: + if 'bytes=' in range_header: + range_part = range_header.replace('bytes=', '') + if '-' in range_part: + start_byte, end_byte = range_part.split('-', 1) + if start_byte: + start_pos_mb = int(start_byte) / (1024 * 1024) + logger.info(f"[VOD-SEEK] Seeking to byte position: {start_byte} (~{start_pos_mb:.1f} MB)") + if int(start_byte) > 0: + logger.info(f"[VOD-SEEK] *** ACTUAL SEEK DETECTED *** Position: {start_pos_mb:.1f} MB") + else: + logger.info(f"[VOD-SEEK] Open-en`ded range request (from start)") + if end_byte: + end_pos_mb = int(end_byte) / (1024 * 1024) + logger.info(f"[VOD-SEEK] End position: {end_byte} bytes (~{end_pos_mb:.1f} MB)") + except Exception as e: + logger.warning(f"[VOD-SEEK] Could not parse range header: {e}") + + # Simple seek detection - track rapid requests + current_time = time.time() + request_key = f"{client_ip}:{content_type}:{content_id}" + + if request_key in _request_times: + time_diff = current_time - _request_times[request_key] + if time_diff < 5.0: + logger.info(f"[VOD-SEEK] Rapid request detected ({time_diff:.1f}s) - likely seeking") + + _request_times[request_key] = current_time + else: + logger.info(f"[VOD-RANGE] No Range header - full content request") + + logger.info(f"[VOD-CLIENT] Client info - IP: {client_ip}, User-Agent: {client_user_agent[:50]}...") + + # If no session ID, create one and redirect to path-based URL + if not session_id: + new_session_id = f"vod_{int(time.time() * 1000)}_{random.randint(1000, 9999)}" + logger.info(f"[VOD-SESSION] Creating new session: {new_session_id}") + + # Preserve any query parameters (except session_id) + query_params = dict(request.GET) + query_params.pop('session_id', None) # Remove if present + + if user: + redirect_url = f"{request.path}?session_id={new_session_id}" + if query_params: + query_string = urlencode(query_params, doseq=True) + redirect_url = f"{redirect_url}&{query_string}" + else: # Build redirect URL with session ID in path, preserve query parameters path_parts = request.path.rstrip('/').split('/') @@ -124,10 +399,6 @@ class VODStreamView(View): else: new_path = f"{'/'.join(path_parts)}/{new_session_id}" - # Preserve any query parameters (except session_id) - query_params = dict(request.GET) - query_params.pop('session_id', None) # Remove if present - if query_params: from urllib.parse import urlencode query_string = urlencode(query_params, doseq=True) @@ -135,534 +406,275 @@ class VODStreamView(View): else: redirect_url = new_path - logger.info(f"[VOD-SESSION] Redirecting to path-based URL: {redirect_url}") + logger.info(f"[VOD-SESSION] Redirecting to path-based URL: {redirect_url}") - return HttpResponse( - status=301, - headers={'Location': redirect_url} - ) - - # Extract preferred M3U account ID and stream ID from query parameters - preferred_m3u_account_id = request.GET.get('m3u_account_id') - preferred_stream_id = request.GET.get('stream_id') - - if preferred_m3u_account_id: - try: - preferred_m3u_account_id = int(preferred_m3u_account_id) - except (ValueError, TypeError): - logger.warning(f"[VOD-PARAM] Invalid m3u_account_id parameter: {preferred_m3u_account_id}") - preferred_m3u_account_id = None - - if preferred_stream_id: - logger.info(f"[VOD-PARAM] Preferred stream ID: {preferred_stream_id}") - - # Get the content object and its relation - content_obj, relation = self._get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id) - if not content_obj or not relation: - logger.error(f"[VOD-ERROR] Content or relation not found: {content_type} {content_id}") - raise Http404(f"Content not found: {content_type} {content_id}") - - logger.info(f"[VOD-CONTENT] Found content: {getattr(content_obj, 'name', 'Unknown')}") - - # Get M3U account from relation - m3u_account = relation.m3u_account - logger.info(f"[VOD-ACCOUNT] Using M3U account: {m3u_account.name}") - - # Get stream URL from relation - stream_url = self._get_stream_url_from_relation(relation) - logger.info(f"[VOD-CONTENT] Content URL: {stream_url or 'No URL found'}") - - if not stream_url: - logger.error(f"[VOD-ERROR] No stream URL available for {content_type} {content_id}") - return HttpResponse("No stream URL available", status=503) - - # Get M3U profile (returns profile and current connection count) - profile_result = self._get_m3u_profile(m3u_account, profile_id, session_id) - - if not profile_result or not profile_result[0]: - logger.error(f"[VOD-ERROR] No suitable M3U profile found for {content_type} {content_id}") - return HttpResponse("No available stream", status=503) - - m3u_profile, current_connections = profile_result - logger.info(f"[VOD-PROFILE] Using M3U profile: {m3u_profile.id} (max_streams: {m3u_profile.max_streams}, current: {current_connections})") - - # Connection tracking is handled by the connection manager - # Transform URL based on profile - final_stream_url = self._transform_url(stream_url, m3u_profile) - logger.info(f"[VOD-URL] Final stream URL: {final_stream_url}") - - # Validate stream URL - if not final_stream_url or not final_stream_url.startswith(('http://', 'https://')): - logger.error(f"[VOD-ERROR] Invalid stream URL: {final_stream_url}") - return HttpResponse("Invalid stream URL", status=500) - - # Get connection manager (Redis-backed for multi-worker support) - connection_manager = MultiWorkerVODConnectionManager.get_instance() - - # Stream the content with session-based connection reuse - logger.info("[VOD-STREAM] Calling connection manager to stream content") - response = connection_manager.stream_content_with_session( - session_id=session_id, - content_obj=content_obj, - stream_url=final_stream_url, - m3u_profile=m3u_profile, - client_ip=client_ip, - client_user_agent=client_user_agent, - request=request, - utc_start=utc_start, - utc_end=utc_end, - offset=offset, - range_header=range_header + return HttpResponse( + status=301, + headers={'Location': redirect_url} ) - logger.info(f"[VOD-SUCCESS] Stream response created successfully, type: {type(response)}") - return response + if user: + if not check_user_stream_limits(user, session_id, media_id=content_id): + return JsonResponse( + {"error": f"Stream limit exceeded ({user.stream_limit} concurrent streams allowed)"}, + status=429 + ) - except Exception as e: - logger.error(f"[VOD-EXCEPTION] Error streaming {content_type} {content_id}: {e}", exc_info=True) - return HttpResponse(f"Streaming error: {str(e)}", status=500) + # Extract preferred M3U account ID and stream ID from query parameters + preferred_m3u_account_id = request.GET.get('m3u_account_id') + preferred_stream_id = request.GET.get('stream_id') - def head(self, request, content_type, content_id, session_id=None, profile_id=None): - """ - Handle HEAD requests for FUSE filesystem integration - - Returns content length and session URL header for subsequent GET requests - """ - logger.info(f"[VOD-HEAD] HEAD request: {content_type}/{content_id}, session: {session_id}, profile: {profile_id}") - - try: - # Get client info for M3U profile selection - client_ip, client_user_agent = get_client_info(request) - logger.info(f"[VOD-HEAD] Client info - IP: {client_ip}, User-Agent: {client_user_agent[:50] if client_user_agent else 'None'}...") - - # If no session ID, create one (same logic as GET) - if not session_id: - new_session_id = f"vod_{int(time.time() * 1000)}_{random.randint(1000, 9999)}" - logger.info(f"[VOD-HEAD] Creating new session for HEAD: {new_session_id}") - - # Build session URL for response header - path_parts = request.path.rstrip('/').split('/') - if profile_id: - session_url = f"{'/'.join(path_parts)}/{new_session_id}/{profile_id}/" - else: - session_url = f"{'/'.join(path_parts)}/{new_session_id}" - - session_id = new_session_id - else: - # Session already in URL, construct the current session URL - session_url = request.path - logger.info(f"[VOD-HEAD] Using existing session: {session_id}") - - # Extract preferred M3U account ID and stream ID from query parameters - preferred_m3u_account_id = request.GET.get('m3u_account_id') - preferred_stream_id = request.GET.get('stream_id') - - if preferred_m3u_account_id: - try: - preferred_m3u_account_id = int(preferred_m3u_account_id) - except (ValueError, TypeError): - logger.warning(f"[VOD-HEAD] Invalid m3u_account_id parameter: {preferred_m3u_account_id}") - preferred_m3u_account_id = None - - if preferred_stream_id: - logger.info(f"[VOD-HEAD] Preferred stream ID: {preferred_stream_id}") - - # Get content and relation (same as GET) - content_obj, relation = self._get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id) - if not content_obj or not relation: - logger.error(f"[VOD-HEAD] Content or relation not found: {content_type} {content_id}") - return HttpResponse("Content not found", status=404) - - # Get M3U account and stream URL - m3u_account = relation.m3u_account - stream_url = self._get_stream_url_from_relation(relation) - if not stream_url: - logger.error(f"[VOD-HEAD] No stream URL available for {content_type} {content_id}") - return HttpResponse("No stream URL available", status=503) - - # Get M3U profile (returns profile and current connection count) - profile_result = self._get_m3u_profile(m3u_account, profile_id, session_id) - if not profile_result or not profile_result[0]: - logger.error(f"[VOD-HEAD] No M3U profile found or all profiles at capacity") - return HttpResponse("No available stream", status=503) - - m3u_profile, current_connections = profile_result - - # Transform URL if needed - final_stream_url = self._transform_url(stream_url, m3u_profile) - - # Make a small range GET request to get content length since providers don't support HEAD - # We'll use a tiny range to minimize data transfer but get the headers we need - # Use M3U account's user agent as primary, client user agent as fallback - m3u_user_agent = m3u_account.get_user_agent().user_agent if m3u_account.get_user_agent() else None - headers = { - 'User-Agent': m3u_user_agent or client_user_agent or 'Dispatcharr/1.0', - 'Accept': '*/*', - 'Range': 'bytes=0-1' # Request only first 2 bytes - } - - logger.info(f"[VOD-HEAD] Making small range GET request to provider: {final_stream_url}") - response = requests.get(final_stream_url, headers=headers, timeout=30, allow_redirects=True, stream=True) - - # Check for range support - should be 206 for partial content - if response.status_code == 206: - # Parse Content-Range header to get total file size - content_range = response.headers.get('Content-Range', '') - if content_range: - # Content-Range: bytes 0-1/1234567890 - total_size = content_range.split('/')[-1] - logger.info(f"[VOD-HEAD] Got file size from Content-Range: {total_size}") - else: - logger.warning(f"[VOD-HEAD] No Content-Range header in 206 response") - total_size = response.headers.get('Content-Length', '0') - elif response.status_code == 200: - # Server doesn't support range requests, use Content-Length from full response - total_size = response.headers.get('Content-Length', '0') - logger.info(f"[VOD-HEAD] Server doesn't support ranges, got Content-Length: {total_size}") - else: - logger.error(f"[VOD-HEAD] Provider GET request failed: {response.status_code}") - return HttpResponse("Provider error", status=response.status_code) - - # Close the small range request - we don't need to keep this connection - response.close() - - # Store the total content length in Redis for the persistent connection to use + if preferred_m3u_account_id: try: - import redis - from django.conf import settings - redis_host = getattr(settings, 'REDIS_HOST', 'localhost') - redis_port = int(getattr(settings, 'REDIS_PORT', 6379)) - redis_db = int(getattr(settings, 'REDIS_DB', 0)) - redis_password = getattr(settings, 'REDIS_PASSWORD', '') - redis_user = getattr(settings, 'REDIS_USER', '') - r = redis.StrictRedis( - host=redis_host, - port=redis_port, - db=redis_db, - password=redis_password if redis_password else None, - username=redis_user if redis_user else None, - decode_responses=True - ) - content_length_key = f"vod_content_length:{session_id}" - r.set(content_length_key, total_size, ex=1800) # Store for 30 minutes - logger.info(f"[VOD-HEAD] Stored total content length {total_size} for session {session_id}") - except Exception as e: - logger.error(f"[VOD-HEAD] Failed to store content length in Redis: {e}") + preferred_m3u_account_id = int(preferred_m3u_account_id) + except (ValueError, TypeError): + logger.warning(f"[VOD-PARAM] Invalid m3u_account_id parameter: {preferred_m3u_account_id}") + preferred_m3u_account_id = None - # Now create a persistent connection for the session (if one doesn't exist) - # This ensures the FUSE GET requests will reuse the same connection + if preferred_stream_id: + logger.info(f"[VOD-PARAM] Preferred stream ID: {preferred_stream_id}") - connection_manager = MultiWorkerVODConnectionManager.get_instance() + # Get the content object and its relation + content_obj, relation = _get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id) + if not content_obj or not relation: + logger.error(f"[VOD-ERROR] Content or relation not found: {content_type} {content_id}") + raise Http404(f"Content not found: {content_type} {content_id}") - logger.info(f"[VOD-HEAD] Pre-creating persistent connection for session: {session_id}") + logger.info(f"[VOD-CONTENT] Found content: {getattr(content_obj, 'name', 'Unknown')}") - # We don't actually stream content here, just ensure connection is ready - # The actual GET requests from FUSE will use the persistent connection + # Get M3U account from relation + m3u_account = relation.m3u_account + logger.info(f"[VOD-ACCOUNT] Using M3U account: {m3u_account.name}") - # Use the total_size we extracted from the range response - provider_content_type = response.headers.get('Content-Type') + # Get stream URL from relation + stream_url = _get_stream_url_from_relation(relation) + logger.info(f"[VOD-CONTENT] Content URL: {stream_url or 'No URL found'}") - if provider_content_type: - content_type_header = provider_content_type - logger.info(f"[VOD-HEAD] Using provider Content-Type: {content_type_header}") - else: - # Provider didn't send Content-Type, infer from URL - inferred_content_type = infer_content_type_from_url(final_stream_url) - if inferred_content_type: - content_type_header = inferred_content_type - logger.info(f"[VOD-HEAD] Provider missing Content-Type, inferred from URL: {content_type_header}") - else: - content_type_header = 'video/mp4' - logger.info(f"[VOD-HEAD] No Content-Type from provider and could not infer from URL, using default: {content_type_header}") + if not stream_url: + logger.error(f"[VOD-ERROR] No stream URL available for {content_type} {content_id}") + return HttpResponse("No stream URL available", status=503) - logger.info(f"[VOD-HEAD] Provider response - Total Size: {total_size}, Type: {content_type_header}") + # Get M3U profile (returns profile and current connection count) + profile_result = _get_m3u_profile(m3u_account, profile_id, session_id) - # Create response with content length and session URL header - head_response = HttpResponse() - head_response['Content-Length'] = total_size - head_response['Content-Type'] = content_type_header - head_response['Accept-Ranges'] = 'bytes' + if not profile_result or not profile_result[0]: + logger.error(f"[VOD-ERROR] No suitable M3U profile found for {content_type} {content_id}") + return HttpResponse("No available stream", status=503) - # Custom header with session URL for FUSE - head_response['X-Session-URL'] = session_url - head_response['X-Dispatcharr-Session'] = session_id + m3u_profile, current_connections = profile_result + logger.info(f"[VOD-PROFILE] Using M3U profile: {m3u_profile.id} (max_streams: {m3u_profile.max_streams}, current: {current_connections})") - logger.info(f"[VOD-HEAD] Returning HEAD response with session URL: {session_url}") - return head_response + # Connection tracking is handled by the connection manager + # Transform URL based on profile + final_stream_url = _transform_url(stream_url, m3u_profile) + logger.info(f"[VOD-URL] Final stream URL: {final_stream_url}") - except Exception as e: - logger.error(f"[VOD-HEAD] Error in HEAD request: {e}", exc_info=True) - return HttpResponse(f"HEAD error: {str(e)}", status=500) + # Validate stream URL + if not final_stream_url or not final_stream_url.startswith(('http://', 'https://')): + logger.error(f"[VOD-ERROR] Invalid stream URL: {final_stream_url}") + return HttpResponse("Invalid stream URL", status=500) - def _get_content_and_relation(self, content_type, content_id, preferred_m3u_account_id=None, preferred_stream_id=None): - """Get the content object and its M3U relation""" - try: - logger.info(f"[CONTENT-LOOKUP] Looking up {content_type} with UUID {content_id}") - if preferred_m3u_account_id: - logger.info(f"[CONTENT-LOOKUP] Preferred M3U account ID: {preferred_m3u_account_id}") - if preferred_stream_id: - logger.info(f"[CONTENT-LOOKUP] Preferred stream ID: {preferred_stream_id}") + # Get connection manager (Redis-backed for multi-worker support) + connection_manager = MultiWorkerVODConnectionManager.get_instance() - if content_type == 'movie': - content_obj = get_object_or_404(Movie, uuid=content_id) - logger.info(f"[CONTENT-FOUND] Movie: {content_obj.name} (ID: {content_obj.id})") + # Stream the content with session-based connection reuse + logger.info("[VOD-STREAM] Calling connection manager to stream content") + response = connection_manager.stream_content_with_session( + session_id=session_id, + content_obj=content_obj, + stream_url=final_stream_url, + m3u_profile=m3u_profile, + client_ip=client_ip, + client_user_agent=client_user_agent, + request=request, + utc_start=utc_start, + utc_end=utc_end, + offset=offset, + range_header=range_header, + user=user, + ) - # Filter by preferred stream ID first (most specific) - relations_query = content_obj.m3u_relations.filter(m3u_account__is_active=True) - if preferred_stream_id: - specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() - if specific_relation: - logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") - return content_obj, specific_relation - else: - logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") + logger.info(f"[VOD-SUCCESS] Stream response created successfully, type: {type(response)}") + return response - # Filter by preferred M3U account if specified - if preferred_m3u_account_id: - specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() - if specific_relation: - logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") - return content_obj, specific_relation - else: - logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") + except Exception as e: + logger.error(f"[VOD-EXCEPTION] Error streaming {content_type} {content_id}: {e}", exc_info=True) + return HttpResponse(f"Streaming error: {str(e)}", status=500) - # Get the highest priority active relation (fallback or default) - relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() +@api_view(["HEAD"]) +def head_vod(request, content_type, content_id, session_id=None, profile_id=None): + """ + Handle HEAD requests for FUSE filesystem integration - if relation: - logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") + Returns content length and session URL header for subsequent GET requests + """ + logger.info(f"[VOD-HEAD] HEAD request: {content_type}/{content_id}, session: {session_id}, profile: {profile_id}") - return content_obj, relation + try: + # Get client info for M3U profile selection + client_ip, client_user_agent = get_client_info(request) + logger.info(f"[VOD-HEAD] Client info - IP: {client_ip}, User-Agent: {client_user_agent[:50] if client_user_agent else 'None'}...") - elif content_type == 'episode': - content_obj = get_object_or_404(Episode, uuid=content_id) - logger.info(f"[CONTENT-FOUND] Episode: {content_obj.name} (ID: {content_obj.id}, Series: {content_obj.series.name})") + # If no session ID, create one (same logic as GET) + if not session_id: + new_session_id = f"vod_{int(time.time() * 1000)}_{random.randint(1000, 9999)}" + logger.info(f"[VOD-HEAD] Creating new session for HEAD: {new_session_id}") - # Filter by preferred stream ID first (most specific) - relations_query = content_obj.m3u_relations.filter(m3u_account__is_active=True) - if preferred_stream_id: - specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() - if specific_relation: - logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") - return content_obj, specific_relation - else: - logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") - - # Filter by preferred M3U account if specified - if preferred_m3u_account_id: - specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() - if specific_relation: - logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") - return content_obj, specific_relation - else: - logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") - - # Get the highest priority active relation (fallback or default) - relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() - - if relation: - logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") - - return content_obj, relation - - elif content_type == 'series': - # For series, get the first episode - series = get_object_or_404(Series, uuid=content_id) - logger.info(f"[CONTENT-FOUND] Series: {series.name} (ID: {series.id})") - episode = series.episodes.first() - if not episode: - logger.error(f"[CONTENT-ERROR] No episodes found for series {series.name}") - return None, None - - logger.info(f"[CONTENT-FOUND] First episode: {episode.name} (ID: {episode.id})") - - # Filter by preferred stream ID first (most specific) - relations_query = episode.m3u_relations.filter(m3u_account__is_active=True) - if preferred_stream_id: - specific_relation = relations_query.filter(stream_id=preferred_stream_id).first() - if specific_relation: - logger.info(f"[STREAM-SELECTED] Using specific stream: {specific_relation.stream_id} from provider: {specific_relation.m3u_account.name}") - return episode, specific_relation - else: - logger.warning(f"[STREAM-FALLBACK] Preferred stream ID {preferred_stream_id} not found, falling back to account/priority selection") - - # Filter by preferred M3U account if specified - if preferred_m3u_account_id: - specific_relation = relations_query.filter(m3u_account__id=preferred_m3u_account_id).first() - if specific_relation: - logger.info(f"[PROVIDER-SELECTED] Using preferred provider: {specific_relation.m3u_account.name}") - return episode, specific_relation - else: - logger.warning(f"[PROVIDER-FALLBACK] Preferred M3U account {preferred_m3u_account_id} not found, using highest priority") - - # Get the highest priority active relation (fallback or default) - relation = relations_query.select_related('m3u_account').order_by('-m3u_account__priority', 'id').first() - - if relation: - logger.info(f"[PROVIDER-SELECTED] Using provider: {relation.m3u_account.name} (priority: {relation.m3u_account.priority})") - - return episode, relation - else: - logger.error(f"[CONTENT-ERROR] Invalid content type: {content_type}") - return None, None - - except Exception as e: - logger.error(f"Error getting content object: {e}") - return None, None - - def _get_stream_url_from_relation(self, relation): - """Get stream URL from the M3U relation""" - try: - # Log the relation type and available attributes - logger.info(f"[VOD-URL] Relation type: {type(relation).__name__}") - logger.info(f"[VOD-URL] Account type: {relation.m3u_account.account_type}") - logger.info(f"[VOD-URL] Stream ID: {getattr(relation, 'stream_id', 'N/A')}") - - # First try the get_stream_url method (this should build URLs dynamically) - if hasattr(relation, 'get_stream_url'): - url = relation.get_stream_url() - if url: - logger.info(f"[VOD-URL] Built URL from get_stream_url(): {url}") - return url - else: - logger.warning(f"[VOD-URL] get_stream_url() returned None") - - logger.error(f"[VOD-URL] Relation has no get_stream_url method or it failed") - return None - except Exception as e: - logger.error(f"[VOD-URL] Error getting stream URL from relation: {e}", exc_info=True) - return None - - def _get_m3u_profile(self, m3u_account, profile_id, session_id=None): - """Get appropriate M3U profile for streaming using Redis-based viewer counts - - Args: - m3u_account: M3UAccount instance - profile_id: Optional specific profile ID requested - session_id: Optional session ID to check for existing connections - - Returns: - tuple: (M3UAccountProfile, current_connections) or None if no profile found - """ - try: - from core.utils import RedisClient - redis_client = RedisClient.get_client() - - if not redis_client: - logger.warning("Redis not available, falling back to default profile") - default_profile = M3UAccountProfile.objects.filter( - m3u_account=m3u_account, - is_active=True, - is_default=True - ).first() - return (default_profile, 0) if default_profile else None - - # Check if this session already has an active connection - if session_id: - persistent_connection_key = f"vod_persistent_connection:{session_id}" - connection_data = redis_client.hgetall(persistent_connection_key) - - if connection_data: - # Decode Redis hash data - decoded_data = {} - for k, v in connection_data.items(): - k_str = k.decode('utf-8') if isinstance(k, bytes) else k - v_str = v.decode('utf-8') if isinstance(v, bytes) else v - decoded_data[k_str] = v_str - - existing_profile_id = decoded_data.get('m3u_profile_id') - if existing_profile_id: - try: - existing_profile = M3UAccountProfile.objects.get( - id=int(existing_profile_id), - m3u_account=m3u_account, - is_active=True - ) - # Get current connections for logging - profile_connections_key = f"profile_connections:{existing_profile.id}" - current_connections = int(redis_client.get(profile_connections_key) or 0) - - logger.info(f"[PROFILE-SELECTION] Session {session_id} reusing existing profile {existing_profile.id}: {current_connections}/{existing_profile.max_streams} connections") - return (existing_profile, current_connections) - except (M3UAccountProfile.DoesNotExist, ValueError): - logger.warning(f"[PROFILE-SELECTION] Session {session_id} has invalid profile ID {existing_profile_id}, selecting new profile") - except Exception as e: - logger.warning(f"[PROFILE-SELECTION] Error checking existing profile for session {session_id}: {e}") - else: - logger.debug(f"[PROFILE-SELECTION] Session {session_id} exists but has no profile ID stored") # If specific profile requested, try to use it + # Build session URL for response header + path_parts = request.path.rstrip('/').split('/') if profile_id: - try: - profile = M3UAccountProfile.objects.get( - id=profile_id, - m3u_account=m3u_account, - is_active=True - ) - # Check Redis-based current connections - profile_connections_key = f"profile_connections:{profile.id}" - current_connections = int(redis_client.get(profile_connections_key) or 0) + session_url = f"{'/'.join(path_parts)}/{new_session_id}/{profile_id}/" + else: + session_url = f"{'/'.join(path_parts)}/{new_session_id}" - if profile.max_streams == 0 or current_connections < profile.max_streams: - logger.info(f"[PROFILE-SELECTION] Using requested profile {profile.id}: {current_connections}/{profile.max_streams} connections") - return (profile, current_connections) - else: - logger.warning(f"[PROFILE-SELECTION] Requested profile {profile.id} is at capacity: {current_connections}/{profile.max_streams}") - except M3UAccountProfile.DoesNotExist: - logger.warning(f"[PROFILE-SELECTION] Requested profile {profile_id} not found") + session_id = new_session_id + else: + # Session already in URL, construct the current session URL + session_url = request.path + logger.info(f"[VOD-HEAD] Using existing session: {session_id}") - # Get active profiles ordered by priority (default first) - m3u_profiles = M3UAccountProfile.objects.filter( - m3u_account=m3u_account, - is_active=True - ) + # Extract preferred M3U account ID and stream ID from query parameters + preferred_m3u_account_id = request.GET.get('m3u_account_id') + preferred_stream_id = request.GET.get('stream_id') - default_profile = m3u_profiles.filter(is_default=True).first() - if not default_profile: - logger.error(f"[PROFILE-SELECTION] No default profile found for M3U account {m3u_account.id}") - return None + if preferred_m3u_account_id: + try: + preferred_m3u_account_id = int(preferred_m3u_account_id) + except (ValueError, TypeError): + logger.warning(f"[VOD-HEAD] Invalid m3u_account_id parameter: {preferred_m3u_account_id}") + preferred_m3u_account_id = None - # Check profiles in order: default first, then others - profiles = [default_profile] + list(m3u_profiles.filter(is_default=False)) + if preferred_stream_id: + logger.info(f"[VOD-HEAD] Preferred stream ID: {preferred_stream_id}") - for profile in profiles: - profile_connections_key = f"profile_connections:{profile.id}" - current_connections = int(redis_client.get(profile_connections_key) or 0) + # Get content and relation (same as GET) + content_obj, relation = _get_content_and_relation(content_type, content_id, preferred_m3u_account_id, preferred_stream_id) + if not content_obj or not relation: + logger.error(f"[VOD-HEAD] Content or relation not found: {content_type} {content_id}") + return HttpResponse("Content not found", status=404) - # Check if profile has available connection slots - if profile.max_streams == 0 or current_connections < profile.max_streams: - logger.info(f"[PROFILE-SELECTION] Selected profile {profile.id} ({profile.name}): {current_connections}/{profile.max_streams} connections") - return (profile, current_connections) - else: - logger.debug(f"[PROFILE-SELECTION] Profile {profile.id} at capacity: {current_connections}/{profile.max_streams}") + # Get M3U account and stream URL + m3u_account = relation.m3u_account + stream_url = _get_stream_url_from_relation(relation) + if not stream_url: + logger.error(f"[VOD-HEAD] No stream URL available for {content_type} {content_id}") + return HttpResponse("No stream URL available", status=503) - # All profiles are at capacity - return None to trigger error response - logger.error(f"[PROFILE-SELECTION] All profiles at capacity for M3U account {m3u_account.id}, rejecting request") - return None + # Get M3U profile (returns profile and current connection count) + profile_result = _get_m3u_profile(m3u_account, profile_id, session_id) + if not profile_result or not profile_result[0]: + logger.error(f"[VOD-HEAD] No M3U profile found or all profiles at capacity") + return HttpResponse("No available stream", status=503) - except Exception as e: - logger.error(f"Error getting M3U profile: {e}") - return None + m3u_profile, current_connections = profile_result - def _transform_url(self, original_url, m3u_profile): - """Transform URL based on M3U profile settings""" + # Transform URL if needed + final_stream_url = _transform_url(stream_url, m3u_profile) + + # Make a small range GET request to get content length since providers don't support HEAD + # We'll use a tiny range to minimize data transfer but get the headers we need + # Use M3U account's user agent as primary, client user agent as fallback + m3u_user_agent = m3u_account.get_user_agent().user_agent if m3u_account.get_user_agent() else None + headers = { + 'User-Agent': m3u_user_agent or client_user_agent or 'Dispatcharr/1.0', + 'Accept': '*/*', + 'Range': 'bytes=0-1' # Request only first 2 bytes + } + + logger.info(f"[VOD-HEAD] Making small range GET request to provider: {final_stream_url}") + response = requests.get(final_stream_url, headers=headers, timeout=30, allow_redirects=True, stream=True) + + # Check for range support - should be 206 for partial content + if response.status_code == 206: + # Parse Content-Range header to get total file size + content_range = response.headers.get('Content-Range', '') + if content_range: + # Content-Range: bytes 0-1/1234567890 + total_size = content_range.split('/')[-1] + logger.info(f"[VOD-HEAD] Got file size from Content-Range: {total_size}") + else: + logger.warning(f"[VOD-HEAD] No Content-Range header in 206 response") + total_size = response.headers.get('Content-Length', '0') + elif response.status_code == 200: + # Server doesn't support range requests, use Content-Length from full response + total_size = response.headers.get('Content-Length', '0') + logger.info(f"[VOD-HEAD] Server doesn't support ranges, got Content-Length: {total_size}") + else: + logger.error(f"[VOD-HEAD] Provider GET request failed: {response.status_code}") + return HttpResponse("Provider error", status=response.status_code) + + # Close the small range request - we don't need to keep this connection + response.close() + + # Store the total content length in Redis for the persistent connection to use try: - import re - - if not original_url: - return None - - search_pattern = m3u_profile.search_pattern - replace_pattern = m3u_profile.replace_pattern - safe_replace_pattern = re.sub(r'\$(\d+)', r'\\\1', replace_pattern) - - if search_pattern and replace_pattern: - transformed_url = re.sub(search_pattern, safe_replace_pattern, original_url) - return transformed_url - - return original_url - + import redis + from django.conf import settings + redis_host = getattr(settings, 'REDIS_HOST', 'localhost') + redis_port = int(getattr(settings, 'REDIS_PORT', 6379)) + redis_db = int(getattr(settings, 'REDIS_DB', 0)) + redis_password = getattr(settings, 'REDIS_PASSWORD', '') + redis_user = getattr(settings, 'REDIS_USER', '') + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) + r = redis.StrictRedis( + host=redis_host, + port=redis_port, + db=redis_db, + password=redis_password if redis_password else None, + username=redis_user if redis_user else None, + decode_responses=True, + **ssl_params + ) + content_length_key = f"vod_content_length:{session_id}" + r.set(content_length_key, total_size, ex=1800) # Store for 30 minutes + logger.info(f"[VOD-HEAD] Stored total content length {total_size} for session {session_id}") except Exception as e: - logger.error(f"Error transforming URL: {e}") - return original_url + logger.error(f"[VOD-HEAD] Failed to store content length in Redis: {e}") + + # Now create a persistent connection for the session (if one doesn't exist) + # This ensures the FUSE GET requests will reuse the same connection + + connection_manager = MultiWorkerVODConnectionManager.get_instance() + + logger.info(f"[VOD-HEAD] Pre-creating persistent connection for session: {session_id}") + + # We don't actually stream content here, just ensure connection is ready + # The actual GET requests from FUSE will use the persistent connection + + # Use the total_size we extracted from the range response + provider_content_type = response.headers.get('Content-Type') + + if provider_content_type: + content_type_header = provider_content_type + logger.info(f"[VOD-HEAD] Using provider Content-Type: {content_type_header}") + else: + # Provider didn't send Content-Type, infer from URL + inferred_content_type = infer_content_type_from_url(final_stream_url) + if inferred_content_type: + content_type_header = inferred_content_type + logger.info(f"[VOD-HEAD] Provider missing Content-Type, inferred from URL: {content_type_header}") + else: + content_type_header = 'video/mp4' + logger.info(f"[VOD-HEAD] No Content-Type from provider and could not infer from URL, using default: {content_type_header}") + + logger.info(f"[VOD-HEAD] Provider response - Total Size: {total_size}, Type: {content_type_header}") + + # Create response with content length and session URL header + head_response = HttpResponse() + head_response['Content-Length'] = total_size + head_response['Content-Type'] = content_type_header + head_response['Accept-Ranges'] = 'bytes' + + # Custom header with session URL for FUSE + head_response['X-Session-URL'] = session_url + head_response['X-Dispatcharr-Session'] = session_id + + logger.info(f"[VOD-HEAD] Returning HEAD response with session URL: {session_url}") + return head_response + + except Exception as e: + logger.error(f"[VOD-HEAD] Error in HEAD request: {e}", exc_info=True) + return HttpResponse(f"HEAD error: {str(e)}", status=500) @method_decorator(csrf_exempt, name='dispatch') class VODPlaylistView(View): @@ -783,19 +795,16 @@ class VODStatsView(View): for key in keys: try: - key_str = key.decode('utf-8') if isinstance(key, bytes) else key connection_data = redis_client.hgetall(key) if connection_data: # Extract session ID from key - session_id = key_str.replace('vod_persistent_connection:', '') + session_id = key.replace('vod_persistent_connection:', '') # Decode Redis hash data combined_data = {} for k, v in connection_data.items(): - k_str = k.decode('utf-8') if isinstance(k, bytes) else k - v_str = v.decode('utf-8') if isinstance(v, bytes) else v - combined_data[k_str] = v_str + combined_data[k] = v # Get content info from the connection data (using correct field names) content_type = combined_data.get('content_obj_type', 'unknown') @@ -945,6 +954,7 @@ class VODStatsView(View): 'm3u_profile': m3u_profile_info, 'client_id': client_id, 'client_ip': combined_data.get('client_ip', 'Unknown'), + 'user_id': combined_data.get('user_id', '0'), 'user_agent': combined_data.get('client_user_agent', 'Unknown'), 'connected_at': combined_data.get('created_at'), 'last_activity': combined_data.get('last_activity'), @@ -1079,4 +1089,59 @@ def stop_vod_client(request): logger.error(f"Error stopping VOD client: {e}", exc_info=True) return JsonResponse({'error': str(e)}, status=500) +@api_view(["GET"]) +def stream_xc_movie(request, username, password, stream_id, extension): + from apps.vod.models import M3UMovieRelation + session_id = request.GET.get('session_id') + profile_id = request.GET.get('profile_id') + + user = get_object_or_404(User, username=username) + + custom_properties = user.custom_properties or {} + + if "xc_password" not in custom_properties: + return Response({"error": "Invalid credentials"}, status=401) + + if custom_properties["xc_password"] != password: + return Response({"error": "Invalid credentials"}, status=401) + + # All authenticated users get access to VOD from all active M3U accounts + filters = {"movie_id": stream_id, "m3u_account__is_active": True} + + try: + # Order by account priority to get the best relation when multiple exist + movie_relation = M3UMovieRelation.objects.select_related('movie').filter(**filters).order_by('-m3u_account__priority', 'id').first() + if not movie_relation: + return JsonResponse({"error": "Movie not found"}, status=404) + except (M3UMovieRelation.DoesNotExist, M3UMovieRelation.MultipleObjectsReturned): + return JsonResponse({"error": "Movie not found"}, status=404) + + return stream_vod(request._request, 'movie', movie_relation.movie.uuid, session_id, profile_id, user) + +@api_view(["GET"]) +def stream_xc_episode(request, username, password, stream_id, extension): + from apps.vod.models import M3UEpisodeRelation + + session_id = request.GET.get('session_id') + profile_id = request.GET.get('profile_id') + + user = get_object_or_404(User, username=username) + + custom_properties = user.custom_properties or {} + + if "xc_password" not in custom_properties: + return Response({"error": "Invalid credentials"}, status=401) + + if custom_properties["xc_password"] != password: + return Response({"error": "Invalid credentials"}, status=401) + + # All authenticated users get access to series/episodes from all active M3U accounts + filters = {"episode_id": stream_id, "m3u_account__is_active": True} + + try: + episode_relation = M3UEpisodeRelation.objects.select_related('episode').filter(**filters).order_by('-m3u_account__priority', 'id').first() + except M3UEpisodeRelation.DoesNotExist: + return JsonResponse({"error": "Episode not found"}, status=404) + + return stream_vod(request._request, 'episode', episode_relation.episode.uuid, session_id, profile_id, user) diff --git a/apps/vod/api_views.py b/apps/vod/api_views.py index 3bd984e6..edb1b110 100644 --- a/apps/vod/api_views.py +++ b/apps/vod/api_views.py @@ -578,13 +578,15 @@ class UnifiedContentViewSet(viewsets.ReadOnlyModelViewSet): "series.id IN (SELECT DISTINCT series_id FROM vod_m3useriesrelation msr JOIN m3u_m3uaccount ma ON msr.m3u_account_id = ma.id WHERE ma.is_active = true)" ] - params = [] + movie_params = [] + series_params = [] if search: where_conditions[0] += " AND LOWER(movies.name) LIKE %s" where_conditions[1] += " AND LOWER(series.name) LIKE %s" search_param = f"%{search.lower()}%" - params.extend([search_param, search_param]) + movie_params.append(search_param) + series_params.append(search_param) if category: if '|' in category: @@ -592,15 +594,20 @@ class UnifiedContentViewSet(viewsets.ReadOnlyModelViewSet): if cat_type == 'movie': where_conditions[0] += " AND movies.id IN (SELECT movie_id FROM vod_m3umovierelation mmr JOIN vod_vodcategory c ON mmr.category_id = c.id WHERE c.name = %s)" where_conditions[1] = "1=0" # Exclude series - params.append(cat_name) + movie_params.append(cat_name) + series_params = [] # no params needed for "1=0" elif cat_type == 'series': where_conditions[1] += " AND series.id IN (SELECT series_id FROM vod_m3useriesrelation msr JOIN vod_vodcategory c ON msr.category_id = c.id WHERE c.name = %s)" where_conditions[0] = "1=0" # Exclude movies - params.append(cat_name) + series_params.append(cat_name) + movie_params = [] # no params needed for "1=0" else: where_conditions[0] += " AND movies.id IN (SELECT movie_id FROM vod_m3umovierelation mmr JOIN vod_vodcategory c ON mmr.category_id = c.id WHERE c.name = %s)" where_conditions[1] += " AND series.id IN (SELECT series_id FROM vod_m3useriesrelation msr JOIN vod_vodcategory c ON msr.category_id = c.id WHERE c.name = %s)" - params.extend([category, category]) + movie_params.append(category) + series_params.append(category) + + params = movie_params + series_params # Use UNION ALL with ORDER BY and LIMIT/OFFSET for true unified pagination # This is much more efficient than Python sorting @@ -896,4 +903,3 @@ class VODLogoViewSet(viewsets.ModelViewSet): {"error": str(e)}, status=status.HTTP_500_INTERNAL_SERVER_ERROR ) - diff --git a/core/api_views.py b/core/api_views.py index dd15886c..a81ddf54 100644 --- a/core/api_views.py +++ b/core/api_views.py @@ -3,6 +3,7 @@ import json import ipaddress import logging +from django.conf import settings as django_settings from django.db import models from rest_framework import viewsets, status from rest_framework.response import Response @@ -301,7 +302,9 @@ def environment(request): country_code = None country_name = None - # 4) Get environment mode from system environment variable + # 4) Get environment mode and TLS status from settings + postgres_ssl = getattr(django_settings, "POSTGRES_SSL", False) + return Response( { "authenticated": True, @@ -309,7 +312,17 @@ def environment(request): "local_ip": local_ip, "country_code": country_code, "country_name": country_name, - "env_mode": "dev" if os.getenv("DISPATCHARR_ENV") == "dev" else "prod", + "env_mode": os.getenv("DISPATCHARR_ENV", "aio"), + "redis_tls": { + "enabled": getattr(django_settings, "REDIS_SSL", False), + "verify": getattr(django_settings, "REDIS_SSL_VERIFY", True), + "mtls": bool(getattr(django_settings, "REDIS_SSL_CERT", "") and getattr(django_settings, "REDIS_SSL_KEY", "")), + }, + "postgres_tls": { + "enabled": postgres_ssl, + "ssl_mode": getattr(django_settings, "POSTGRES_SSL_MODE", "verify-full") if postgres_ssl else None, + "mtls": bool(getattr(django_settings, "POSTGRES_SSL_CERT", "") and getattr(django_settings, "POSTGRES_SSL_KEY", "")), + }, } ) diff --git a/core/management/commands/dropdb.py b/core/management/commands/dropdb.py index 1b39a58d..d61b9891 100644 --- a/core/management/commands/dropdb.py +++ b/core/management/commands/dropdb.py @@ -1,5 +1,6 @@ import sys import psycopg2 +from psycopg2 import sql from django.core.management.base import BaseCommand from django.conf import settings from django.db import connection @@ -15,6 +16,13 @@ class Command(BaseCommand): host = db_settings.get('HOST', 'localhost') port = db_settings.get('PORT', 5432) + # Read TLS parameters from Django OPTIONS (populated when POSTGRES_SSL=true) + db_options = db_settings.get('OPTIONS', {}) + ssl_kwargs = {} + for key in ('sslmode', 'sslrootcert', 'sslcert', 'sslkey'): + if key in db_options: + ssl_kwargs[key] = db_options[key] + self.stdout.write(self.style.WARNING( f"WARNING: This will irreversibly drop the entire database '{db_name}'!" )) @@ -30,13 +38,13 @@ class Command(BaseCommand): maintenance_db = 'postgres' try: self.stdout.write("Connecting to maintenance database...") - conn = psycopg2.connect(dbname=maintenance_db, user=user, password=password, host=host, port=port) + conn = psycopg2.connect(dbname=maintenance_db, user=user, password=password, host=host, port=port, **ssl_kwargs) conn.autocommit = True cur = conn.cursor() self.stdout.write(f"Dropping database '{db_name}'...") - cur.execute(f"DROP DATABASE IF EXISTS {db_name};") + cur.execute(sql.SQL("DROP DATABASE IF EXISTS {}").format(sql.Identifier(db_name))) self.stdout.write(f"Creating database '{db_name}'...") - cur.execute(f"CREATE DATABASE {db_name};") + cur.execute(sql.SQL("CREATE DATABASE {}").format(sql.Identifier(db_name))) cur.close() conn.close() self.stdout.write(self.style.SUCCESS(f"Database '{db_name}' has been dropped and recreated.")) diff --git a/core/migrations/022_default_user_limit_settings.py b/core/migrations/022_default_user_limit_settings.py new file mode 100644 index 00000000..4c2b3c33 --- /dev/null +++ b/core/migrations/022_default_user_limit_settings.py @@ -0,0 +1,24 @@ +# Generated by Django 5.1.6 on 2025-03-01 14:01 + +from django.db import migrations +from django.utils.text import slugify + + +def preload_user_limit_settings(apps, schema_editor): + CoreSettings = apps.get_model("core", "CoreSettings") + CoreSettings.objects.create( + key="user_limit_settings", + name="User Limit Settings", + value={}, + ) + + +class Migration(migrations.Migration): + + dependencies = [ + ("core", "0021_systemnotification_notificationdismissal"), + ] + + operations = [ + migrations.RunPython(preload_user_limit_settings), + ] diff --git a/core/models.py b/core/models.py index f1a604f7..5e86e98f 100644 --- a/core/models.py +++ b/core/models.py @@ -156,6 +156,7 @@ PROXY_SETTINGS_KEY = "proxy_settings" NETWORK_ACCESS_KEY = "network_access" SYSTEM_SETTINGS_KEY = "system_settings" EPG_SETTINGS_KEY = "epg_settings" +USER_LIMITS_SETTINGS_KEY = "user_limit_settings" class CoreSettings(models.Model): @@ -362,6 +363,15 @@ class CoreSettings(models.Model): cls._update_group(SYSTEM_SETTINGS_KEY, "System Settings", {"time_zone": value}) return value + @classmethod + def get_user_limits_settings(cls): + return cls._get_group(USER_LIMITS_SETTINGS_KEY, { + "terminate_on_limit_exceeded": True, + "prioritize_single_client_channels": True, + "ignore_same_channel_connections": False, + "terminate_oldest": True, + }) + class SystemEvent(models.Model): """ diff --git a/core/redis_pubsub.py b/core/redis_pubsub.py index 5d0032b0..b1f48b1f 100644 --- a/core/redis_pubsub.py +++ b/core/redis_pubsub.py @@ -201,10 +201,6 @@ class RedisPubSubManager: channel = message.get('channel') if channel: - # Decode binary channel name if needed - if isinstance(channel, bytes): - channel = channel.decode('utf-8') - # Find and call the appropriate handler handler = self.message_handlers.get(channel) if handler: diff --git a/core/tasks.py b/core/tasks.py index a18c2f74..13100d0b 100644 --- a/core/tasks.py +++ b/core/tasks.py @@ -404,7 +404,7 @@ def fetch_channel_stats(): while True: cursor, keys = redis_client.scan(cursor, match=channel_pattern) for key in keys: - channel_id_match = re.search(r"ts_proxy:channel:(.*):metadata", key.decode('utf-8')) + channel_id_match = re.search(r"ts_proxy:channel:(.*):metadata", key) if channel_id_match: ch_id = channel_id_match.group(1) channel_info = ChannelStatus.get_basic_channel_info(ch_id) diff --git a/core/tests.py b/core/tests.py index 305ab308..b4770f92 100644 --- a/core/tests.py +++ b/core/tests.py @@ -1,3 +1,5 @@ +from unittest.mock import patch, MagicMock + from django.test import TestCase from core.models import CoreSettings, DVR_SETTINGS_KEY, EPG_SETTINGS_KEY @@ -148,3 +150,74 @@ class EpgIgnoreListsTest(TestCase): ]: self._set_epg_field_raw(field, "not a list") self.assertEqual(getter(), []) + + +class DropDBCommandTlsTest(TestCase): + """Verify dropdb management command passes TLS parameters to psycopg2.""" + databases = [] + + _DB_WITH_TLS = { + 'default': { + 'ENGINE': 'django.db.backends.postgresql', + 'NAME': 'testdb', + 'USER': 'testuser', + 'PASSWORD': 'testpass', + 'HOST': 'localhost', + 'PORT': 5432, + 'OPTIONS': { + 'sslmode': 'verify-full', + 'sslrootcert': '/certs/ca.crt', + 'sslcert': '/certs/client.crt', + 'sslkey': '/certs/client.key', + }, + } + } + + _DB_NO_TLS = { + 'default': { + 'ENGINE': 'django.db.backends.postgresql', + 'NAME': 'testdb', + 'USER': 'testuser', + 'PASSWORD': 'testpass', + 'HOST': 'localhost', + 'PORT': 5432, + } + } + + @patch('core.management.commands.dropdb.psycopg2.connect') + @patch('core.management.commands.dropdb.connection') + @patch('builtins.input', return_value='yes') + def test_dropdb_passes_ssl_kwargs_when_tls_enabled(self, _inp, _conn, mock_connect): + mock_pg = MagicMock() + mock_connect.return_value = mock_pg + mock_pg.cursor.return_value = MagicMock() + + with self.settings(DATABASES=self._DB_WITH_TLS): + from django.core.management import call_command + call_command('dropdb') + + mock_connect.assert_called_once_with( + dbname='postgres', user='testuser', password='testpass', + host='localhost', port=5432, + sslmode='verify-full', + sslrootcert='/certs/ca.crt', + sslcert='/certs/client.crt', + sslkey='/certs/client.key', + ) + + @patch('core.management.commands.dropdb.psycopg2.connect') + @patch('core.management.commands.dropdb.connection') + @patch('builtins.input', return_value='yes') + def test_dropdb_no_ssl_kwargs_when_tls_disabled(self, _inp, _conn, mock_connect): + mock_pg = MagicMock() + mock_connect.return_value = mock_pg + mock_pg.cursor.return_value = MagicMock() + + with self.settings(DATABASES=self._DB_NO_TLS): + from django.core.management import call_command + call_command('dropdb') + + mock_connect.assert_called_once_with( + dbname='postgres', user='testuser', password='testpass', + host='localhost', port=5432, + ) diff --git a/core/utils.py b/core/utils.py index ef5aa702..583075c3 100644 --- a/core/utils.py +++ b/core/utils.py @@ -13,6 +13,8 @@ from django.core.validators import URLValidator from django.core.exceptions import ValidationError import gc +_REDIS_TLS_HINT = " (TLS is enabled — verify certificate paths and that Redis is configured for TLS)" + logger = logging.getLogger(__name__) # Import the command detector @@ -43,103 +45,116 @@ def natural_sort_key(text): class RedisClient: _client = None + _buffer = None _pubsub_client = None @classmethod - def get_client(cls, max_retries=5, retry_interval=1): - if cls._client is None: - retry_count = 0 - while retry_count < max_retries: + def _init_client(cls, decode_responses=True, max_retries=5, retry_interval=1): + retry_count = 0 + while retry_count < max_retries: + try: + # Get connection parameters from settings or environment + redis_host = os.environ.get("REDIS_HOST", getattr(settings, 'REDIS_HOST', 'localhost')) + redis_port = int(os.environ.get("REDIS_PORT", getattr(settings, 'REDIS_PORT', 6379))) + redis_db = int(os.environ.get("REDIS_DB", getattr(settings, 'REDIS_DB', 0))) + redis_password = os.environ.get("REDIS_PASSWORD", getattr(settings, 'REDIS_PASSWORD', '')) + redis_user = os.environ.get("REDIS_USER", getattr(settings, 'REDIS_USER', '')) + + # Use standardized settings + socket_timeout = getattr(settings, 'REDIS_SOCKET_TIMEOUT', 5) + socket_connect_timeout = getattr(settings, 'REDIS_SOCKET_CONNECT_TIMEOUT', 5) + health_check_interval = getattr(settings, 'REDIS_HEALTH_CHECK_INTERVAL', 30) + socket_keepalive = getattr(settings, 'REDIS_SOCKET_KEEPALIVE', True) + retry_on_timeout = getattr(settings, 'REDIS_RETRY_ON_TIMEOUT', True) + + # TLS params from settings (empty dict when TLS is disabled) + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) + + # Create Redis client with better defaults + client = redis.Redis( + host=redis_host, + port=redis_port, + db=redis_db, + password=redis_password if redis_password else None, + username=redis_user if redis_user else None, + socket_timeout=socket_timeout, + socket_connect_timeout=socket_connect_timeout, + socket_keepalive=socket_keepalive, + health_check_interval=health_check_interval, + retry_on_timeout=retry_on_timeout, + decode_responses=decode_responses, + **ssl_params + ) + + # Validate connection with ping + client.ping() + + # Disable persistence on first connection - improves performance + # Only try to disable if not in a read-only environment try: - # Get connection parameters from settings or environment - redis_host = os.environ.get("REDIS_HOST", getattr(settings, 'REDIS_HOST', 'localhost')) - redis_port = int(os.environ.get("REDIS_PORT", getattr(settings, 'REDIS_PORT', 6379))) - redis_db = int(os.environ.get("REDIS_DB", getattr(settings, 'REDIS_DB', 0))) - redis_password = os.environ.get("REDIS_PASSWORD", getattr(settings, 'REDIS_PASSWORD', '')) - redis_user = os.environ.get("REDIS_USER", getattr(settings, 'REDIS_USER', '')) + client.config_set('save', '') # Disable RDB snapshots + client.config_set('appendonly', 'no') # Disable AOF logging - # Use standardized settings - socket_timeout = getattr(settings, 'REDIS_SOCKET_TIMEOUT', 5) - socket_connect_timeout = getattr(settings, 'REDIS_SOCKET_CONNECT_TIMEOUT', 5) - health_check_interval = getattr(settings, 'REDIS_HEALTH_CHECK_INTERVAL', 30) - socket_keepalive = getattr(settings, 'REDIS_SOCKET_KEEPALIVE', True) - retry_on_timeout = getattr(settings, 'REDIS_RETRY_ON_TIMEOUT', True) + # Disable protected mode when in debug mode + if os.environ.get('DISPATCHARR_DEBUG', '').lower() == 'true': + client.config_set('protected-mode', 'no') # Disable protected mode in debug + logger.warning("Redis protected mode disabled for debug environment") - # Create Redis client with better defaults - client = redis.Redis( - host=redis_host, - port=redis_port, - db=redis_db, - password=redis_password if redis_password else None, - username=redis_user if redis_user else None, - socket_timeout=socket_timeout, - socket_connect_timeout=socket_connect_timeout, - socket_keepalive=socket_keepalive, - health_check_interval=health_check_interval, - retry_on_timeout=retry_on_timeout - ) - - # Validate connection with ping - client.ping() - - # Disable persistence on first connection - improves performance - # Only try to disable if not in a read-only environment - try: - client.config_set('save', '') # Disable RDB snapshots - client.config_set('appendonly', 'no') # Disable AOF logging - - # Set optimal memory settings with environment variable support - # Get max memory from environment or use a larger default (512MB instead of 256MB) - #max_memory = os.environ.get('REDIS_MAX_MEMORY', '512mb') - #eviction_policy = os.environ.get('REDIS_EVICTION_POLICY', 'allkeys-lru') - - # Apply memory settings - #client.config_set('maxmemory-policy', eviction_policy) - #client.config_set('maxmemory', max_memory) - - #logger.info(f"Redis configured with maxmemory={max_memory}, policy={eviction_policy}") - - # Disable protected mode when in debug mode - if os.environ.get('DISPATCHARR_DEBUG', '').lower() == 'true': - client.config_set('protected-mode', 'no') # Disable protected mode in debug - logger.warning("Redis protected mode disabled for debug environment") - - logger.trace("Redis persistence disabled for better performance") - except redis.exceptions.ResponseError as e: - # Improve error handling for Redis configuration errors - if "OOM" in str(e): - logger.error(f"Redis OOM during configuration: {e}") - # Try to increase maxmemory as an emergency measure - try: - client.config_set('maxmemory', '768mb') - logger.warning("Applied emergency Redis memory increase to 768MB") - except: - pass - else: - logger.error(f"Redis configuration error: {e}") - - logger.info(f"Connected to Redis at {redis_host}:{redis_port}/{redis_db}") - - cls._client = client - break - - except (ConnectionError, TimeoutError) as e: - retry_count += 1 - if retry_count >= max_retries: - logger.error(f"Failed to connect to Redis after {max_retries} attempts: {e}") - return None + logger.trace("Redis persistence disabled for better performance") + except redis.exceptions.ResponseError as e: + # Improve error handling for Redis configuration errors + if "OOM" in str(e): + logger.error(f"Redis OOM during configuration: {e}") + # Try to increase maxmemory as an emergency measure + try: + client.config_set('maxmemory', '768mb') + logger.warning("Applied emergency Redis memory increase to 768MB") + except: + pass else: - # Use exponential backoff for retries - wait_time = retry_interval * (2 ** (retry_count - 1)) - logger.warning(f"Redis connection failed. Retrying in {wait_time}s... ({retry_count}/{max_retries})") - time.sleep(wait_time) + logger.error(f"Redis configuration error: {e}") - except Exception as e: - logger.error(f"Unexpected error connecting to Redis: {e}") + logger.info(f"Connected to Redis at {redis_host}:{redis_port}/{redis_db}") + + return client + + except (ConnectionError, TimeoutError) as e: + retry_count += 1 + _tls_hint = _REDIS_TLS_HINT if ssl_params else "" + if retry_count >= max_retries: + logger.error(f"Failed to connect to Redis after {max_retries} attempts: {e}{_tls_hint}") return None + else: + # Use exponential backoff for retries + wait_time = retry_interval * (2 ** (retry_count - 1)) + logger.warning(f"Redis connection failed. Retrying in {wait_time}s... ({retry_count}/{max_retries})") + time.sleep(wait_time) + except Exception as e: + _tls_hint = "" + try: + _tls_hint = _REDIS_TLS_HINT if ssl_params else "" + except NameError: + pass + logger.error(f"Unexpected error connecting to Redis: {e}{_tls_hint}") + return None + + return None + + @classmethod + def get_client(cls, max_retries=5, retry_interval=1): + """Get Redis client optimized for non-binary data (decoded responses)""" + if cls._client is None: + cls._client = cls._init_client(decode_responses=True, max_retries=max_retries, retry_interval=retry_interval) return cls._client + @classmethod + def get_buffer(cls, max_retries=5, retry_interval=1): + """Get Redis client optimized for binary data (no decoding)""" + if cls._buffer is None: + cls._buffer = cls._init_client(decode_responses=False, max_retries=max_retries, retry_interval=retry_interval) + return cls._buffer + @classmethod def get_pubsub_client(cls, max_retries=5, retry_interval=1): """Get Redis client optimized for PubSub operations""" @@ -161,6 +176,8 @@ class RedisClient: health_check_interval = getattr(settings, 'REDIS_HEALTH_CHECK_INTERVAL', 30) retry_on_timeout = getattr(settings, 'REDIS_RETRY_ON_TIMEOUT', True) + ssl_params = getattr(settings, 'REDIS_SSL_PARAMS', {}) + # Create Redis client with PubSub-optimized settings - no timeout client = redis.Redis( host=redis_host, @@ -172,7 +189,9 @@ class RedisClient: socket_connect_timeout=socket_connect_timeout, socket_keepalive=socket_keepalive, health_check_interval=health_check_interval, - retry_on_timeout=retry_on_timeout + retry_on_timeout=retry_on_timeout, + decode_responses=True, + **ssl_params ) # Validate connection with ping @@ -185,8 +204,9 @@ class RedisClient: except (ConnectionError, TimeoutError) as e: retry_count += 1 + _tls_hint = _REDIS_TLS_HINT if ssl_params else "" if retry_count >= max_retries: - logger.error(f"Failed to connect to Redis for PubSub after {max_retries} attempts: {e}") + logger.error(f"Failed to connect to Redis for PubSub after {max_retries} attempts: {e}{_tls_hint}") return None else: # Use exponential backoff for retries @@ -195,7 +215,8 @@ class RedisClient: time.sleep(wait_time) except Exception as e: - logger.error(f"Unexpected error connecting to Redis for PubSub: {e}") + _tls_hint = _REDIS_TLS_HINT if ssl_params else "" + logger.error(f"Unexpected error connecting to Redis for PubSub: {e}{_tls_hint}") return None return cls._pubsub_client diff --git a/core/views.py b/core/views.py index 082cccc3..7321367c 100644 --- a/core/views.py +++ b/core/views.py @@ -4,7 +4,7 @@ from shlex import split as shlex_split import sys import subprocess import logging -import re +import regex import redis from django.conf import settings @@ -42,12 +42,14 @@ def stream_view(request, channel_uuid): redis_db = int(getattr(settings, "REDIS_DB", "0")) redis_password = getattr(settings, "REDIS_PASSWORD", "") redis_user = getattr(settings, "REDIS_USER", "") + ssl_params = getattr(settings, "REDIS_SSL_PARAMS", {}) redis_client = redis.Redis( host=redis_host, port=redis_port, db=redis_db, password=redis_password if redis_password else None, - username=redis_user if redis_user else None + username=redis_user if redis_user else None, + **ssl_params ) # Retrieve the channel by the provided stream_id. @@ -130,10 +132,13 @@ def stream_view(request, channel_uuid): # Prepare the pattern replacement. logger.debug("Executing the following pattern replacement:") logger.debug(f" search: {active_profile.search_pattern}") - safe_replace_pattern = re.sub(r'\$(\d+)', r'\\\1', active_profile.replace_pattern) + # Convert JS-style backreferences in replace: $ -> \g, $1 -> \1 + safe_replace_pattern = regex.sub(r'\$<([^>]+)>', r'\\g<\1>', active_profile.replace_pattern) + safe_replace_pattern = regex.sub(r'\$(\d+)', r'\\\1', safe_replace_pattern) logger.debug(f" replace: {active_profile.replace_pattern}") logger.debug(f" safe replace: {safe_replace_pattern}") - stream_url = re.sub(active_profile.search_pattern, safe_replace_pattern, input_url) + # regex module accepts JS-style (?...) named groups natively + stream_url = regex.sub(active_profile.search_pattern, safe_replace_pattern, input_url) logger.debug(f"Generated stream url: {stream_url}") # Get the stream profile set on the channel. diff --git a/core/xtream_codes.py b/core/xtream_codes.py index 9b56197a..822b37fd 100644 --- a/core/xtream_codes.py +++ b/core/xtream_codes.py @@ -62,7 +62,7 @@ class Client: url = f"{self.server_url}/{endpoint}" logger.debug(f"XC API Request: {url} with params: {params}") - response = self.session.get(url, params=params, timeout=30) + response = self.session.get(url, params=params, timeout=60) response.raise_for_status() # Check if response is empty diff --git a/dispatcharr/consumers.py b/dispatcharr/consumers.py index 4e21bdae..7b62d957 100644 --- a/dispatcharr/consumers.py +++ b/dispatcharr/consumers.py @@ -1,6 +1,6 @@ import json from channels.generic.websocket import AsyncWebsocketConsumer -import re, logging +import regex, logging logger = logging.getLogger(__name__) @@ -54,9 +54,9 @@ class MyWebSocketConsumer(AsyncWebsocketConsumer): # Apply the transformation using the replace_with_mark function try: - search_preview = re.sub(data["search"], replace_with_mark, data["url"]) + search_preview = regex.sub(data["search"], replace_with_mark, data["url"]) except Exception as e: - search_preview = data["search"] + search_preview = data["url"] logger.error(f"Failed to generate replace preview: {e}") result = transform_url(data["url"], data["search"], data["replace"]) diff --git a/dispatcharr/persistent_lock.py b/dispatcharr/persistent_lock.py index 2ed72bf4..45625857 100644 --- a/dispatcharr/persistent_lock.py +++ b/dispatcharr/persistent_lock.py @@ -48,7 +48,7 @@ class PersistentLock: Returns True if the expiration was successfully extended. """ current_value = self.redis_client.get(self.lock_key) - if current_value and current_value.decode("utf-8") == self.lock_token: + if current_value and current_value == self.lock_token: self.redis_client.expire(self.lock_key, self.lock_timeout) self.has_lock = False return True @@ -74,18 +74,40 @@ class PersistentLock: # Example usage (for testing purposes only): if __name__ == "__main__": import os + import sys # Connect to Redis using environment variables; adjust connection parameters as needed. redis_host = os.environ.get("REDIS_HOST", "localhost") redis_port = int(os.environ.get("REDIS_PORT", 6379)) redis_db = int(os.environ.get("REDIS_DB", 0)) redis_password = os.environ.get("REDIS_PASSWORD", "") redis_user = os.environ.get("REDIS_USER", "") + ssl_kwargs = {} + if os.environ.get("REDIS_SSL", "false").lower() == "true": + import ssl as _ssl + ssl_kwargs["ssl"] = True + ssl_kwargs["ssl_cert_reqs"] = ( + _ssl.CERT_REQUIRED if os.environ.get("REDIS_SSL_VERIFY", "true").lower() == "true" + else _ssl.CERT_NONE + ) + for env_var, key in [ + ("REDIS_SSL_CA_CERT", "ssl_ca_certs"), + ("REDIS_SSL_CERT", "ssl_certfile"), + ("REDIS_SSL_KEY", "ssl_keyfile"), + ]: + path = os.environ.get(env_var, "") + if path: + if not os.path.isfile(path): + print(f"Redis TLS: {env_var}={path!r} — file not found.") + sys.exit(1) + ssl_kwargs[key] = path + client = redis.Redis( host=redis_host, port=redis_port, db=redis_db, password=redis_password if redis_password else None, - username=redis_user if redis_user else None + username=redis_user if redis_user else None, + **ssl_kwargs ) lock = PersistentLock(client, "lock:example_account", lock_timeout=120) diff --git a/dispatcharr/settings.py b/dispatcharr/settings.py index b0d387b2..ec61eb4d 100644 --- a/dispatcharr/settings.py +++ b/dispatcharr/settings.py @@ -1,7 +1,23 @@ import os +import ssl from pathlib import Path from datetime import timedelta from urllib.parse import quote_plus +from django.core.exceptions import ImproperlyConfigured + + +def _validate_tls_cert_paths(paths, service_name): + """Validate that configured TLS certificate file paths exist on disk. + + Raises ImproperlyConfigured with a clear message identifying the + service and missing file so operators can fix their environment. + """ + for env_var, file_path in paths: + if file_path and not Path(file_path).is_file(): + raise ImproperlyConfigured( + f"{service_name} TLS: {env_var}={file_path!r} — file not found. " + f"Check that the certificate file exists and the volume is mounted correctly." + ) BASE_DIR = Path(__file__).resolve().parent.parent @@ -12,6 +28,37 @@ REDIS_DB = os.environ.get("REDIS_DB", "0") REDIS_USER = os.environ.get("REDIS_USER", "") REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD", "") +# Redis TLS configuration +REDIS_SSL = os.environ.get("REDIS_SSL", "false").lower() == "true" +REDIS_SSL_VERIFY = os.environ.get("REDIS_SSL_VERIFY", "true").lower() == "true" +REDIS_SSL_CA_CERT = os.environ.get("REDIS_SSL_CA_CERT", "") +REDIS_SSL_CERT = os.environ.get("REDIS_SSL_CERT", "") +REDIS_SSL_KEY = os.environ.get("REDIS_SSL_KEY", "") + +# Reusable dict of SSL kwargs for redis.Redis() constructors +REDIS_SSL_PARAMS = {} +if REDIS_SSL: + _validate_tls_cert_paths([ + ("REDIS_SSL_CA_CERT", REDIS_SSL_CA_CERT), + ("REDIS_SSL_CERT", REDIS_SSL_CERT), + ("REDIS_SSL_KEY", REDIS_SSL_KEY), + ], "Redis") + + REDIS_SSL_PARAMS["ssl"] = True + REDIS_SSL_PARAMS["ssl_cert_reqs"] = ssl.CERT_REQUIRED if REDIS_SSL_VERIFY else ssl.CERT_NONE + if REDIS_SSL_CA_CERT: + REDIS_SSL_PARAMS["ssl_ca_certs"] = REDIS_SSL_CA_CERT + if REDIS_SSL_CERT: + REDIS_SSL_PARAMS["ssl_certfile"] = REDIS_SSL_CERT + if REDIS_SSL_KEY: + REDIS_SSL_PARAMS["ssl_keyfile"] = REDIS_SSL_KEY + + _mtls = "enabled" if REDIS_SSL_CERT and REDIS_SSL_KEY else "disabled" + _verify = "on" if REDIS_SSL_VERIFY else "off" + print(f"Redis TLS: enabled (verify={_verify}, mTLS={_mtls})") +else: + print("Redis TLS: disabled") + # Set DEBUG to True for development, False for production if os.environ.get("DISPATCHARR_DEBUG", "False").lower() == "true": DEBUG = True @@ -120,20 +167,46 @@ TEMPLATES = [ WSGI_APPLICATION = "dispatcharr.wsgi.application" ASGI_APPLICATION = "dispatcharr.asgi.application" +_redis_scheme = "rediss" if REDIS_SSL else "redis" + +# URL-encoded auth string shared by CHANNEL_LAYERS and Celery broker URLs +if REDIS_PASSWORD: + _encoded_password = quote_plus(REDIS_PASSWORD) + if REDIS_USER: + _redis_auth = f"{quote_plus(REDIS_USER)}:{_encoded_password}@" + else: + _redis_auth = f":{_encoded_password}@" +else: + _redis_auth = "" + +_channels_redis_url = f"{_redis_scheme}://{_redis_auth}{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" +# channels_redis accepts either a URL string or a dict with "address" + kwargs. +# When TLS is enabled, pass SSL params alongside the URL so the connection pool +# uses the correct CA cert and verification settings. +if REDIS_SSL: + # Filter out "ssl" key — the rediss:// scheme already enables SSL. + # Passing ssl=True as a kwarg to aioredis from_url causes an error. + _channels_ssl = {k: v for k, v in REDIS_SSL_PARAMS.items() if k != "ssl"} + _channels_host = {"address": _channels_redis_url, **_channels_ssl} +else: + _channels_host = _channels_redis_url + CHANNEL_LAYERS = { "default": { "BACKEND": "channels_redis.core.RedisChannelLayer", "CONFIG": { - "hosts": ["redis://{redis_auth}{host}:{port}/{db}".format( - redis_auth=f"{quote_plus(REDIS_USER)}:{quote_plus(REDIS_PASSWORD)}@" if REDIS_PASSWORD and REDIS_USER else f":{quote_plus(REDIS_PASSWORD)}@" if REDIS_PASSWORD else "", - host=REDIS_HOST, - port=REDIS_PORT, - db=REDIS_DB - )], # URL format supports authentication + "hosts": [_channels_host], }, }, } +# PostgreSQL TLS configuration (defined before DATABASES for module-level access) +POSTGRES_SSL = os.environ.get("POSTGRES_SSL", "false").lower() == "true" +POSTGRES_SSL_MODE = os.environ.get("POSTGRES_SSL_MODE", "verify-full") +POSTGRES_SSL_CA_CERT = os.environ.get("POSTGRES_SSL_CA_CERT", "") +POSTGRES_SSL_CERT = os.environ.get("POSTGRES_SSL_CERT", "") +POSTGRES_SSL_KEY = os.environ.get("POSTGRES_SSL_KEY", "") + if os.getenv("DB_ENGINE", None) == "sqlite": DATABASES = { "default": { @@ -154,6 +227,28 @@ else: } } + if POSTGRES_SSL: + _validate_tls_cert_paths([ + ("POSTGRES_SSL_CA_CERT", POSTGRES_SSL_CA_CERT), + ("POSTGRES_SSL_CERT", POSTGRES_SSL_CERT), + ("POSTGRES_SSL_KEY", POSTGRES_SSL_KEY), + ], "PostgreSQL") + + DATABASES["default"]["OPTIONS"] = { + "sslmode": POSTGRES_SSL_MODE, + } + if POSTGRES_SSL_CA_CERT: + DATABASES["default"]["OPTIONS"]["sslrootcert"] = POSTGRES_SSL_CA_CERT + if POSTGRES_SSL_CERT: + DATABASES["default"]["OPTIONS"]["sslcert"] = POSTGRES_SSL_CERT + if POSTGRES_SSL_KEY: + DATABASES["default"]["OPTIONS"]["sslkey"] = POSTGRES_SSL_KEY + + _mtls = "enabled" if POSTGRES_SSL_CERT and POSTGRES_SSL_KEY else "disabled" + print(f"PostgreSQL TLS: enabled (sslmode={POSTGRES_SSL_MODE}, mTLS={_mtls})") + else: + print("PostgreSQL TLS: disabled") + AUTH_PASSWORD_VALIDATORS = [ { "NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator", @@ -208,22 +303,52 @@ STATICFILES_DIRS = [ DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField" AUTH_USER_MODEL = "accounts.User" -# Build default Redis URL from components for Celery with optional authentication -# Build auth string conditionally with URL encoding for special characters -if REDIS_PASSWORD: - encoded_password = quote_plus(REDIS_PASSWORD) - if REDIS_USER: - encoded_user = quote_plus(REDIS_USER) - redis_auth = f"{encoded_user}:{encoded_password}@" - else: - redis_auth = f":{encoded_password}@" +_default_redis_url = f"{_redis_scheme}://{_redis_auth}{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" +# Celery/Kombu require SSL parameters in the URL query string because +# internal URL parsing can overwrite the CELERY_BROKER_USE_SSL dict. +if REDIS_SSL: + _celery_ssl_params = [ + f"ssl_cert_reqs={'CERT_REQUIRED' if REDIS_SSL_VERIFY else 'CERT_NONE'}", + ] + if REDIS_SSL_CA_CERT: + _celery_ssl_params.append(f"ssl_ca_certs={REDIS_SSL_CA_CERT}") + if REDIS_SSL_CERT: + _celery_ssl_params.append(f"ssl_certfile={REDIS_SSL_CERT}") + if REDIS_SSL_KEY: + _celery_ssl_params.append(f"ssl_keyfile={REDIS_SSL_KEY}") + _default_celery_url = f"{_default_redis_url}?{'&'.join(_celery_ssl_params)}" else: - redis_auth = "" - -_default_redis_url = f"redis://{redis_auth}{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" -CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL", _default_redis_url) + _default_celery_url = _default_redis_url +CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL", _default_celery_url) CELERY_RESULT_BACKEND = os.environ.get("CELERY_RESULT_BACKEND", CELERY_BROKER_URL) +# Validate that URL overrides don't conflict with TLS settings +for _url_var, _url_val in [ + ("CELERY_BROKER_URL", CELERY_BROKER_URL), + ("CELERY_RESULT_BACKEND", CELERY_RESULT_BACKEND), +]: + _is_override = os.environ.get(_url_var) is not None + if not _is_override: + continue + _url_is_ssl = _url_val.startswith("rediss://") + if REDIS_SSL and not _url_is_ssl: + raise ImproperlyConfigured( + f"REDIS_SSL is enabled but {_url_var} uses redis:// (plaintext). " + f"Change the URL scheme to rediss:// or remove the {_url_var} override." + ) + if not REDIS_SSL and _url_is_ssl: + raise ImproperlyConfigured( + f"{_url_var} uses rediss:// (TLS) but REDIS_SSL is not enabled. " + f"Set REDIS_SSL=true and configure the TLS certificate settings." + ) + +# Celery TLS configuration — required in addition to the rediss:// URL scheme. +# Uses the same cert params as REDIS_SSL_PARAMS, minus the "ssl" key that +# redis-py needs but Celery/Kombu does not. +if REDIS_SSL: + CELERY_BROKER_USE_SSL = {k: v for k, v in REDIS_SSL_PARAMS.items() if k != "ssl"} + CELERY_RESULT_BACKEND_USE_SSL = CELERY_BROKER_USE_SSL + # Configure Redis key prefix CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = { "global_keyprefix": "celery-tasks:", # Set the Redis key prefix for Celery @@ -299,8 +424,19 @@ SIMPLE_JWT = { "BLACKLIST_AFTER_ROTATION": True, # Optional: Whether to blacklist refresh tokens } -# Redis connection settings +# Redis connection settings — _default_redis_url uses rediss:// when REDIS_SSL is enabled REDIS_URL = os.environ.get("REDIS_URL", _default_redis_url) +if os.environ.get("REDIS_URL") is not None: + if REDIS_SSL and not REDIS_URL.startswith("rediss://"): + raise ImproperlyConfigured( + "REDIS_SSL is enabled but REDIS_URL uses redis:// (plaintext). " + "Change the URL scheme to rediss:// or remove the REDIS_URL override." + ) + if not REDIS_SSL and REDIS_URL.startswith("rediss://"): + raise ImproperlyConfigured( + "REDIS_URL uses rediss:// (TLS) but REDIS_SSL is not enabled. " + "Set REDIS_SSL=true and configure the TLS certificate settings." + ) REDIS_SOCKET_TIMEOUT = 60 # Socket timeout in seconds REDIS_SOCKET_CONNECT_TIMEOUT = 5 # Connection timeout in seconds REDIS_HEALTH_CHECK_INTERVAL = 15 # Health check every 15 seconds diff --git a/dispatcharr/urls.py b/dispatcharr/urls.py index 092fb0cb..1319f31e 100644 --- a/dispatcharr/urls.py +++ b/dispatcharr/urls.py @@ -6,7 +6,7 @@ from django.views.generic import TemplateView, RedirectView from .routing import websocket_urlpatterns from apps.output.views import xc_player_api, xc_panel_api, xc_get, xc_xmltv from apps.proxy.ts_proxy.views import stream_xc -from apps.output.views import xc_movie_stream, xc_series_stream +from apps.proxy.vod_proxy.views import stream_xc_movie, stream_xc_episode urlpatterns = [ # API Routes @@ -44,13 +44,13 @@ urlpatterns = [ # XC VOD endpoints path( "movie///.", - xc_movie_stream, - name="xc_movie_stream", + stream_xc_movie, + name="stream_xc_movie", ), path( "series///.", - xc_series_stream, - name="xc_series_stream", + stream_xc_episode, + name="stream_xc_episode", ), # Admin path("admin", RedirectView.as_view(url="/admin/", permanent=True)), diff --git a/docker/Dockerfile b/docker/Dockerfile index 3e30a825..596c4025 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -12,7 +12,7 @@ COPY ./frontend /app/frontend # remove any node_modules that may have been copied from the host (x86) RUN rm -rf node_modules || true; \ npm install --no-audit --progress=false; -RUN npm run build; \ +RUN npm run build && \ rm -rf node_modules .cache # --- Redeclare build arguments for the next stage --- @@ -32,9 +32,9 @@ COPY . /app COPY ./docker/nginx.conf /etc/nginx/sites-enabled/default # Fix line endings and make entrypoint scripts executable RUN for f in /app/docker/entrypoint*.sh; do \ - if [ -f "$f" ]; then \ - sed -i 's/\r$//' "$f" && chmod +x "$f"; \ - fi; \ + if [ -f "$f" ]; then \ + sed -i 's/\r$//' "$f" && chmod +x "$f"; \ + fi; \ done # Clean out existing frontend folder RUN rm -rf /app/frontend diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index d674460a..547ac249 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -14,6 +14,7 @@ services: - 9191:9191 volumes: - ./data:/data + #- ./certs:/certs:ro # TLS certificates (optional) depends_on: db: condition: service_healthy @@ -33,15 +34,26 @@ services: - POSTGRES_DB=dispatcharr - POSTGRES_USER=dispatch - POSTGRES_PASSWORD=secret + # PostgreSQL TLS (optional) — mount certs via the volume above + #- POSTGRES_SSL=true # required to enable TLS + #- POSTGRES_SSL_MODE=verify-full # optional: verify-full (default) | verify-ca | require + #- POSTGRES_SSL_CA_CERT=/certs/postgres/ca.crt # optional: CA cert to verify the server + #- POSTGRES_SSL_CERT=/certs/postgres/client.crt # optional: client cert (only if server requires client auth) + #- POSTGRES_SSL_KEY=/certs/postgres/client.key # optional: client key (only if server requires client auth) # Redis Connection - REDIS_HOST=redis - REDIS_PORT=6379 - # Redis Authentication (Optional) # Uncomment and set if your Redis requires authentication: #- REDIS_PASSWORD=your_strong_redis_password #- REDIS_USER=your_redis_username # For Redis 6+ ACL - see Redis service below + # Redis TLS (optional) — mount certs via the volume above + #- REDIS_SSL=true # required to enable TLS + #- REDIS_SSL_VERIFY=true # optional: set false for self-signed certs without a CA + #- REDIS_SSL_CA_CERT=/certs/redis/ca.crt # optional: CA cert to verify the server + #- REDIS_SSL_CERT=/certs/redis/client.crt # optional: client cert (only if server requires client auth) + #- REDIS_SSL_KEY=/certs/redis/client.key # optional: client key (only if server requires client auth) # Logging - DISPATCHARR_LOG_LEVEL=info @@ -95,6 +107,7 @@ services: condition: service_started volumes: - ./data:/data + #- ./certs:/certs:ro # TLS certificates (optional) extra_hosts: - "host.docker.internal:host-gateway" entrypoint: ["/app/docker/entrypoint.celery.sh"] @@ -108,21 +121,30 @@ services: # Must match the web service port for DVR recording and internal API calls - DISPATCHARR_PORT=9191 - # PostgreSQL Connection + # PostgreSQL — must match web service settings - POSTGRES_HOST=db - POSTGRES_PORT=5432 - POSTGRES_DB=dispatcharr - POSTGRES_USER=dispatch - POSTGRES_PASSWORD=secret + # PostgreSQL TLS — must match web service + #- POSTGRES_SSL=true + #- POSTGRES_SSL_MODE=verify-full + #- POSTGRES_SSL_CA_CERT=/certs/postgres/ca.crt + #- POSTGRES_SSL_CERT=/certs/postgres/client.crt + #- POSTGRES_SSL_KEY=/certs/postgres/client.key - # Redis Connection + # Redis — must match web service settings - REDIS_HOST=redis - REDIS_PORT=6379 - - # Redis Authentication (Optional) - # Uncomment and set if your Redis requires authentication: #- REDIS_PASSWORD=your_strong_redis_password - #- REDIS_USER=your_redis_username # For Redis 6+ ACL - see Redis service below + #- REDIS_USER=your_redis_username + # Redis TLS — must match web service + #- REDIS_SSL=true + #- REDIS_SSL_VERIFY=true + #- REDIS_SSL_CA_CERT=/certs/redis/ca.crt + #- REDIS_SSL_CERT=/certs/redis/client.crt + #- REDIS_SSL_KEY=/certs/redis/client.key # Logging - DISPATCHARR_LOG_LEVEL=info diff --git a/docker/entrypoint.celery.sh b/docker/entrypoint.celery.sh index 7a69f430..b38b6ac3 100644 --- a/docker/entrypoint.celery.sh +++ b/docker/entrypoint.celery.sh @@ -36,6 +36,10 @@ if [ "$USE_LEGACY_NUMPY" = "true" ]; then fi fi +# Fix TLS client key permissions/ownership for PostgreSQL. +FIXED_KEY_PATH="/data/.pg-client-celery.key" +. /app/docker/init/00-fix-pg-ssl-key.sh + # Wait for migrations to complete # Uses 'migrate --check' which exits 0 only when all migrations are applied, # and exits 1 on unapplied migrations OR connection errors (safe either way) diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh index 0de0afd5..ea8a95b9 100755 --- a/docker/entrypoint.sh +++ b/docker/entrypoint.sh @@ -30,7 +30,14 @@ echo_with_timestamp() { # Set PostgreSQL environment variables export POSTGRES_DB=${POSTGRES_DB:-dispatcharr} export POSTGRES_USER=${POSTGRES_USER:-dispatch} -export POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-secret} +# AIO mode: default to 'secret' for internal DB. +# Modular mode + TLS: no default — cert-only auth (mTLS) uses no password. +# Modular mode + no TLS: preserve 'secret' default for backward compatibility. +if [[ "${DISPATCHARR_ENV:-}" == "modular" && "${POSTGRES_SSL:-}" == "true" ]]; then + export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-}" +else + export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-secret}" +fi export POSTGRES_HOST=${POSTGRES_HOST:-localhost} export POSTGRES_PORT=${POSTGRES_PORT:-5432} export PG_VERSION=$(ls /usr/lib/postgresql/ | sort -V | tail -n 1) @@ -96,6 +103,20 @@ echo "Environment DISPATCHARR_LOG_LEVEL set to: '${DISPATCHARR_LOG_LEVEL}'" # Also make the log level available in /etc/environment for all login shells #grep -q "DISPATCHARR_LOG_LEVEL" /etc/environment || echo "DISPATCHARR_LOG_LEVEL=${DISPATCHARR_LOG_LEVEL}" >> /etc/environment +# Translate Dispatcharr POSTGRES_SSL_* env vars into libpq-recognized PGSSL* +# env vars. Called once before any external PostgreSQL connection; all child +# processes (psql, pg_dump, pg_isready, createdb, dropdb) inherit these +# automatically. No-op when POSTGRES_SSL is not "true". +setup_pg_ssl_env() { + if [ "${POSTGRES_SSL:-false}" != "true" ]; then + return 0 + fi + export PGSSLMODE="${POSTGRES_SSL_MODE:-verify-full}" + if [ -n "${POSTGRES_SSL_CA_CERT:-}" ]; then export PGSSLROOTCERT="$POSTGRES_SSL_CA_CERT"; fi + if [ -n "${POSTGRES_SSL_CERT:-}" ]; then export PGSSLCERT="$POSTGRES_SSL_CERT"; fi + if [ -n "${POSTGRES_SSL_KEY:-}" ]; then export PGSSLKEY="$POSTGRES_SSL_KEY"; fi +} + # READ-ONLY - don't let users change these export POSTGRES_DIR=/data/db @@ -112,6 +133,14 @@ variables=( CELERY_NICE_LEVEL UWSGI_NICE_LEVEL DJANGO_SECRET_KEY ) +# TLS variables are optional — only propagate when set to avoid noisy warnings +for _tls_var in POSTGRES_SSL POSTGRES_SSL_MODE POSTGRES_SSL_CA_CERT POSTGRES_SSL_CERT POSTGRES_SSL_KEY \ + REDIS_SSL REDIS_SSL_VERIFY REDIS_SSL_CA_CERT REDIS_SSL_CERT REDIS_SSL_KEY; do + if [ -n "${!_tls_var+x}" ]; then + variables+=("$_tls_var") + fi +done + # Truncate files before rewriting > /etc/profile.d/dispatcharr.sh @@ -146,6 +175,22 @@ fi echo "Starting user setup..." . /app/docker/init/01-user-setup.sh +# Fix TLS client key permissions/ownership BEFORE any external PG connections. +# Must run after 01-user-setup.sh (user exists for chown) and before +# 02-postgres.sh / pg_isready (which make the first external PG connections). +FIXED_KEY_PATH="/data/.pg-client.key" +. /app/docker/init/00-fix-pg-ssl-key.sh +# Propagate the fixed path to login shells (su - strips env vars) +if [ "${POSTGRES_SSL_KEY:-}" = "$FIXED_KEY_PATH" ]; then + sed -i "/^POSTGRES_SSL_KEY=/d" /etc/environment + echo "POSTGRES_SSL_KEY='$FIXED_KEY_PATH'" >> /etc/environment + sed -i "s|export POSTGRES_SSL_KEY=.*|export POSTGRES_SSL_KEY='$FIXED_KEY_PATH'|" /etc/profile.d/dispatcharr.sh +fi + +# Export libpq TLS env vars so all subsequent psql/pg_dump/pg_isready calls +# (in 02-postgres.sh, modular-mode checks, etc.) use TLS automatically. +setup_pg_ssl_env + # Initialize PostgreSQL (script handles modular vs internal mode internally) echo "Setting up PostgreSQL..." . /app/docker/init/02-postgres.sh diff --git a/docker/init/00-fix-pg-ssl-key.sh b/docker/init/00-fix-pg-ssl-key.sh new file mode 100644 index 00000000..d7065815 --- /dev/null +++ b/docker/init/00-fix-pg-ssl-key.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# +# Fix TLS client key permissions and ownership for PostgreSQL. +# libpq requires the client key to be 0600 or stricter. +# +# Triggers on: +# - Permissions too open (Docker Desktop mounts files as 0777) +# - Wrong ownership (Kubernetes secrets / Docker volumes mount as root; +# the application user can't read a root-owned 0600 key) +# - Read-only source (volume mounted :ro — can't chmod in place) +# +# Usage: source this script with FIXED_KEY_PATH set to the destination. +# FIXED_KEY_PATH="/data/.pg-client.key" +# . /app/docker/init/00-fix-pg-ssl-key.sh +# +# After sourcing, POSTGRES_SSL_KEY is updated to the fixed path if a copy +# was needed. The caller is responsible for propagating the new value to +# /etc/environment or profile.d if required. + +: "${FIXED_KEY_PATH:?FIXED_KEY_PATH must be set before sourcing fix-pg-ssl-key.sh}" + +if [ -n "${POSTGRES_SSL_KEY:-}" ] && [ -f "$POSTGRES_SSL_KEY" ]; then + _key_perms=$(stat -c '%a' "$POSTGRES_SSL_KEY" 2>/dev/null) + _key_owner=$(stat -c '%u' "$POSTGRES_SSL_KEY" 2>/dev/null) + _needs_fix=false + + if [ "$_key_perms" != "600" ] && [ "$_key_perms" != "640" ]; then + _needs_fix=true + elif [ "$(id -u)" = "0" ] && [ -n "${PUID:-}" ] && [ "$_key_owner" != "$PUID" ]; then + _needs_fix=true + fi + + if [ "$_needs_fix" = true ]; then + cp "$POSTGRES_SSL_KEY" "$FIXED_KEY_PATH" + chmod 600 "$FIXED_KEY_PATH" + if [ "$(id -u)" = "0" ] && [ -n "${PUID:-}" ]; then + chown "${PUID}:${PGID:-$PUID}" "$FIXED_KEY_PATH" + fi + export POSTGRES_SSL_KEY="$FIXED_KEY_PATH" + echo "Fixed PostgreSQL client key (perms: ${_key_perms}, owner: ${_key_owner} → ${PUID:-root}:600)" + fi + + unset _key_perms _key_owner _needs_fix +fi diff --git a/docker/init/99-init-dev.sh b/docker/init/99-init-dev.sh index 5afbc96d..d2e6dd0d 100644 --- a/docker/init/99-init-dev.sh +++ b/docker/init/99-init-dev.sh @@ -1,25 +1,33 @@ #!/bin/bash -echo "🚀 Development Mode - Setting up Frontend..." +if [ ! -e "/tmp/init" ]; then + echo "🚀 Development Mode - Setting up Frontend..." -# Install Node.js -if ! command -v node 2>&1 >/dev/null -then - echo "=== setting up nodejs ===" - curl -sL https://deb.nodesource.com/setup_23.x -o /tmp/nodesource_setup.sh - bash /tmp/nodesource_setup.sh - apt-get update - apt-get install -y --no-install-recommends \ - nodejs -fi - -# Install frontend dependencies -cd /app/frontend && npm install -# Install Python dependencies using UV -cd /app && uv sync --python $UV_PROJECT_ENVIRONMENT/bin/python --no-install-project --no-dev - -# Install debugpy for remote debugging -if [ "$DISPATCHARR_DEBUG" = "true" ]; then - echo "=== setting up debugpy ===" - uv pip install --python $UV_PROJECT_ENVIRONMENT/bin/python debugpy + # Install Node.js + if ! command -v node 2>&1 >/dev/null + then + echo "=== setting up nodejs ===" + curl -sL https://deb.nodesource.com/setup_23.x -o /tmp/nodesource_setup.sh + bash /tmp/nodesource_setup.sh + apt-get update + apt-get install -y --no-install-recommends \ + nodejs + fi + + # Install frontend dependencies + cd /app/frontend && npm install + # Install Python dependencies using UV + cd /app && uv sync --python $UV_PROJECT_ENVIRONMENT/bin/python --no-install-project --no-dev + + # Install debugpy for remote debugging + if [ "$DISPATCHARR_DEBUG" = "true" ]; then + echo "=== setting up debugpy ===" + uv pip install --python $UV_PROJECT_ENVIRONMENT/bin/python debugpy + fi + + if [[ "$DISPATCHARR_ENV" = "dev" ]]; then + touch /tmp/init + fi +else + echo "Development mode initialization already done. Skipping dev setup." fi diff --git a/docker/tests/test-tls-postgres.sh b/docker/tests/test-tls-postgres.sh new file mode 100644 index 00000000..8a7630dc --- /dev/null +++ b/docker/tests/test-tls-postgres.sh @@ -0,0 +1,892 @@ +#!/bin/bash +# +# Integration test suite for TLS/mTLS in modular mode. +# Validates that Dispatcharr connects correctly to external PostgreSQL and +# Redis services using various TLS configurations. +# +# Prerequisites: +# - Docker Desktop (or Docker Engine) running +# - Internet access (pulls postgres:17, redis:latest) +# - ~10-15 minutes for a full run +# +# Usage: +# cd +# bash docker/tests/test-tls-postgres.sh [--skip-build] [--keep-on-fail] [scenario_name] +# +# Options: +# --skip-build Skip Docker image build (use existing dispatcharr:tls-test image) +# --keep-on-fail Don't clean up containers/volumes on failure (for debugging) +# scenario_name Run only the named scenario +# +# Scenarios: +# modular_mtls_no_password PG mTLS cert-only auth, no password +# modular_mtls_with_password PG mTLS + password auth combined +# modular_tls_server_only PG server-side TLS only (no client cert) +# modular_tls_key_permission PG mTLS with 0777 client key (Docker Desktop scenario) +# modular_no_tls_regression Non-TLS modular mode still works +# modular_pg_verify_full PG mTLS with verify-full (CN must match hostname) +# modular_redis_tls Redis with TLS (server-side verification) +# modular_full_tls_celery PG mTLS + Redis TLS with separate Celery container +# +# Exit codes: +# 0 All tests passed +# 1 One or more tests failed (or build failed) + +set -uo pipefail + +# Prevent Git Bash (MINGW) from converting Unix paths +export MSYS_NO_PATHCONV=1 + +############################################################################### +# Configuration +############################################################################### +IMAGE_NAME="dispatcharr:tls-test" +TEST_PREFIX="tls_test" +STARTUP_TIMEOUT=120 +SKIP_BUILD=false +KEEP_ON_FAIL=false +SINGLE_SCENARIO="" +PASS=0 +FAIL=0 +SKIP=0 +ERRORS=() +CERT_DIR="" + +# Colors (disabled if not a terminal) +if [ -t 1 ]; then + RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m' + CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m' +else + RED=''; GREEN=''; YELLOW=''; CYAN=''; BOLD=''; NC='' +fi + +############################################################################### +# Parse arguments +############################################################################### +for arg in "$@"; do + case "$arg" in + --skip-build) SKIP_BUILD=true ;; + --keep-on-fail) KEEP_ON_FAIL=true ;; + -*) echo "Unknown option: $arg"; exit 1 ;; + *) SINGLE_SCENARIO="$arg" ;; + esac +done + +############################################################################### +# Helpers +############################################################################### +CURRENT_SCENARIO="" +CLEANUP_ITEMS=() + +log_pass() { echo -e " ${GREEN}✅ $1${NC}"; PASS=$((PASS + 1)); } +log_fail() { echo -e " ${RED}❌ $1${NC}"; FAIL=$((FAIL + 1)); ERRORS+=("[$CURRENT_SCENARIO] $1"); } +log_skip() { echo -e " ${YELLOW}⏭️ $1${NC}"; SKIP=$((SKIP + 1)); } +log_info() { echo -e " ${CYAN}ℹ️ $1${NC}"; } +section() { echo -e "\n${BOLD}━━━ $1 ━━━${NC}"; SCENARIO_FAIL_BEFORE=$FAIL; } + +track_container() { CLEANUP_ITEMS+=("container:$1"); } +track_volume() { CLEANUP_ITEMS+=("volume:$1"); } +track_network() { CLEANUP_ITEMS+=("network:$1"); } + +fresh_volume() { + local vol="$1" + docker rm -f $(docker ps -aq --filter "volume=${vol}") 2>/dev/null || true + docker volume rm "$vol" 2>/dev/null || true + docker volume create "$vol" >/dev/null + track_volume "$vol" +} + +cleanup_scenario() { + if [ "$KEEP_ON_FAIL" = true ] && [ "$FAIL" -gt "${SCENARIO_FAIL_BEFORE:-0}" ]; then + log_info "Keeping resources for debugging (--keep-on-fail)" + CLEANUP_ITEMS=() + return + fi + for item in "${CLEANUP_ITEMS[@]}"; do + local type="${item%%:*}" + local name="${item#*:}" + case "$type" in + container) docker stop "$name" 2>/dev/null; docker rm -f "$name" 2>/dev/null ;; + volume) docker volume rm "$name" 2>/dev/null ;; + network) docker network rm "$name" 2>/dev/null ;; + esac + done + CLEANUP_ITEMS=() +} + +trap 'cleanup_scenario; [ -n "$CERT_DIR" ] && rm -rf "$CERT_DIR"' EXIT + +wait_for_ready() { + local name="$1" + local timeout="${2:-$STARTUP_TIMEOUT}" + local elapsed=0 + + while [ $elapsed -lt $timeout ]; do + if ! docker ps -q -f "name=^${name}$" 2>/dev/null | grep -q .; then + echo " Container $name exited unexpectedly" + return 1 + fi + if docker logs "$name" 2>&1 | grep -q "uwsgi started with PID"; then + return 0 + fi + sleep 3 + ((elapsed+=3)) + done + echo " Timeout (${timeout}s) waiting for $name" + return 1 +} + +_capture_logs() { + local container="$1" logfile="$2" + docker logs "$container" > "$logfile" 2>&1 +} + +check_log_contains() { + local container="$1" pattern="$2" description="$3" + local tmplog; tmplog=$(mktemp) + _capture_logs "$container" "$tmplog" + if grep -q "$pattern" "$tmplog"; then + log_pass "$description" + else + log_fail "$description (pattern not found: $pattern)" + fi + rm -f "$tmplog" +} + +check_log_absent() { + local container="$1" pattern="$2" description="$3" + local tmplog; tmplog=$(mktemp) + _capture_logs "$container" "$tmplog" + if grep -q "$pattern" "$tmplog"; then + log_fail "$description (unexpected pattern found: $pattern)" + else + log_pass "$description" + fi + rm -f "$tmplog" +} + +check_migrations_done() { + local container="$1" + local tmplog; tmplog=$(mktemp) + _capture_logs "$container" "$tmplog" + if grep -qE "Running migrations|No migrations to apply|Operations to perform|Applying .+\.\.\. OK" "$tmplog"; then + log_pass "Django migrations completed" + elif grep -q "uwsgi started with PID" "$tmplog"; then + log_pass "Django migrations completed (confirmed via uwsgi startup)" + else + log_fail "Django migrations did not complete" + fi + rm -f "$tmplog" +} + +check_no_permission_errors() { + local container="$1" + local tmplog; tmplog=$(mktemp) + _capture_logs "$container" "$tmplog" + local errors + errors=$(grep -iE "permission denied|operation not permitted" "$tmplog" \ + | grep -v "GPU acceleration" | grep -v "Warning:" | head -5) + rm -f "$tmplog" + if [ -n "$errors" ]; then + log_fail "Permission errors in logs:" + echo "$errors" | while read -r line; do echo " $line"; done + else + log_pass "No permission errors in logs" + fi +} + +dump_logs_on_fail() { + local container="$1" + if [ $FAIL -gt ${SCENARIO_FAIL_BEFORE:-0} ]; then + echo -e " ${YELLOW}--- Container logs ($container) ---${NC}" + docker logs "$container" 2>&1 | tail -30 | sed 's/^/ /' + echo -e " ${YELLOW}--- End logs ---${NC}" + fi +} + +############################################################################### +# Certificate generation +############################################################################### +generate_test_certs() { + CERT_DIR=$(mktemp -d) + log_info "Generating test certificates in $CERT_DIR" + + # Generate certs inside a container for cross-platform compatibility. + # Shared CA for both PG and Redis. CN of server certs must match their + # Docker container hostnames for verify-full mode. + docker run --rm --entrypoint sh \ + -v "$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR"):/certs" \ + -w /certs alpine/openssl -c ' + # Shared CA + openssl req -new -x509 -days 1 -nodes \ + -keyout ca.key -out ca.crt -subj "/CN=Test CA" 2>/dev/null && + + # PostgreSQL server cert (CN = PG container hostname) + openssl req -new -nodes \ + -keyout pg-server.key -out pg-server.csr -subj "/CN='"${TEST_PREFIX}"'_pg" 2>/dev/null && + openssl x509 -req -days 1 -in pg-server.csr \ + -CA ca.crt -CAkey ca.key -CAcreateserial -out pg-server.crt 2>/dev/null && + # PostgreSQL client cert (CN = POSTGRES_USER) + openssl req -new -nodes \ + -keyout pg-client.key -out pg-client.csr -subj "/CN=dispatch" 2>/dev/null && + openssl x509 -req -days 1 -in pg-client.csr \ + -CA ca.crt -CAkey ca.key -CAcreateserial -out pg-client.crt 2>/dev/null && + + # Redis server cert (CN = Redis container hostname) + openssl req -new -nodes \ + -keyout redis-server.key -out redis-server.csr -subj "/CN='"${TEST_PREFIX}"'_redis" 2>/dev/null && + openssl x509 -req -days 1 -in redis-server.csr \ + -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-server.crt 2>/dev/null && + + # Backwards-compat aliases (existing PG-only scenarios use these names) + cp pg-server.crt server.crt && cp pg-server.key server.key && + cp pg-client.crt client.crt && cp pg-client.key client.key && + + chmod 600 pg-server.key pg-client.key redis-server.key server.key client.key + ' || { log_fail "Certificate generation failed"; return 1; } + + log_pass "Test certificates generated" +} + +############################################################################### +# Start a TLS-enabled Redis container +############################################################################### +start_tls_redis() { + local name="$1" net="$2" + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + # Redis needs certs owned by redis user (uid 999 in the official image). + # Mount certs, copy to a writable location, fix ownership, then start + # with TLS flags. + docker run -d --name "$name" --network "$net" \ + -v "${cert_mount}:/certs:ro" \ + redis:latest \ + sh -c ' + cp /certs/redis-server.crt /certs/redis-server.key /certs/ca.crt /tmp/ && + chmod 600 /tmp/redis-server.key && + chown redis:redis /tmp/redis-server.crt /tmp/redis-server.key /tmp/ca.crt && + exec redis-server \ + --tls-port 6379 --port 0 \ + --tls-cert-file /tmp/redis-server.crt \ + --tls-key-file /tmp/redis-server.key \ + --tls-ca-cert-file /tmp/ca.crt \ + --tls-auth-clients no + ' >/dev/null + + # Wait for Redis TLS to be ready + local elapsed=0 + while [ $elapsed -lt 20 ]; do + if docker exec "$name" redis-cli --tls \ + --cert /certs/redis-server.crt --key /certs/redis-server.key --cacert /certs/ca.crt \ + ping 2>/dev/null | grep -q "PONG"; then + break + fi + sleep 2; elapsed=$((elapsed + 2)) + done +} + +############################################################################### +# Start a TLS-enabled PostgreSQL container +############################################################################### +start_tls_postgres() { + local name="$1" net="$2" hba_auth="$3" + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + docker run -d --name "$name" --network "$net" \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=tempsetup \ + -e POSTGRES_DB=dispatcharr \ + -v "${cert_mount}:/certs:ro" \ + postgres:17 >/dev/null + + # Wait for PG to initialize + local elapsed=0 + while [ $elapsed -lt 30 ]; do + if docker exec "$name" su postgres -c "/usr/lib/postgresql/17/bin/pg_isready" 2>/dev/null | grep -q "accepting"; then + break + fi + sleep 2; ((elapsed+=2)) + done + + # Configure SSL and pg_hba.conf + docker exec "$name" bash -c " + cp /certs/server.crt /certs/server.key /certs/ca.crt /var/lib/postgresql/ + chown postgres:postgres /var/lib/postgresql/server.crt /var/lib/postgresql/server.key /var/lib/postgresql/ca.crt + chmod 600 /var/lib/postgresql/server.key + echo \"ssl = on\" >> /var/lib/postgresql/data/postgresql.conf + echo \"ssl_cert_file = '/var/lib/postgresql/server.crt'\" >> /var/lib/postgresql/data/postgresql.conf + echo \"ssl_key_file = '/var/lib/postgresql/server.key'\" >> /var/lib/postgresql/data/postgresql.conf + echo \"ssl_ca_file = '/var/lib/postgresql/ca.crt'\" >> /var/lib/postgresql/data/postgresql.conf + cat > /var/lib/postgresql/data/pg_hba.conf << HBA +local all all trust +hostssl all all 0.0.0.0/0 ${hba_auth} +hostssl all all ::0/0 ${hba_auth} +HBA + su postgres -c '/usr/lib/postgresql/17/bin/pg_ctl reload -D /var/lib/postgresql/data' + " >/dev/null 2>&1 + sleep 1 +} + +############################################################################### +# Test scenarios +############################################################################### + +test_modular_mtls_no_password() { + CURRENT_SCENARIO="modular_mtls_no_password" + section "Modular mode — mTLS cert-only auth (no password)" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + start_tls_postgres "$pg_name" "$net" "cert" + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + # No POSTGRES_PASSWORD — cert-only auth + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e POSTGRES_SSL=true \ + -e POSTGRES_SSL_MODE=verify-ca \ + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt \ + -e POSTGRES_SSL_CERT=/certs/client.crt \ + -e POSTGRES_SSL_KEY=/certs/client.key \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with mTLS cert-only auth" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed with mTLS" + check_log_contains "$name" "PostgreSQL TLS: enabled" \ + "Django sees TLS enabled" + check_migrations_done "$name" + check_no_permission_errors "$name" + else + log_fail "Container failed to start with mTLS cert-only auth" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_mtls_with_password() { + CURRENT_SCENARIO="modular_mtls_with_password" + section "Modular mode — mTLS + password auth" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + # cert + md5 password + start_tls_postgres "$pg_name" "$net" "cert" + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=tempsetup \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e POSTGRES_SSL=true \ + -e POSTGRES_SSL_MODE=verify-ca \ + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt \ + -e POSTGRES_SSL_CERT=/certs/client.crt \ + -e POSTGRES_SSL_KEY=/certs/client.key \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with mTLS + password" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed with mTLS + password" + check_migrations_done "$name" + else + log_fail "Container failed to start with mTLS + password" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_tls_server_only() { + CURRENT_SCENARIO="modular_tls_server_only" + section "Modular mode — server-only TLS (no client cert)" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + # md5 auth over TLS (no client cert required) + start_tls_postgres "$pg_name" "$net" "md5" + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=tempsetup \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e POSTGRES_SSL=true \ + -e POSTGRES_SSL_MODE=verify-ca \ + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with server-only TLS" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed with server-only TLS" + check_migrations_done "$name" + else + log_fail "Container failed to start with server-only TLS" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_tls_key_permission() { + CURRENT_SCENARIO="modular_tls_key_permission" + section "Modular mode — mTLS with 0777 client key (Docker Desktop scenario)" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + start_tls_postgres "$pg_name" "$net" "cert" + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + # Create a copy of certs with 0777 key permissions + local bad_perms_dir + bad_perms_dir=$(mktemp -d) + cp "$CERT_DIR"/ca.crt "$CERT_DIR"/client.crt "$CERT_DIR"/client.key "$bad_perms_dir/" + chmod 777 "$bad_perms_dir/client.key" + + local cert_mount + cert_mount="$(cygpath -w "$bad_perms_dir" 2>/dev/null || echo "$bad_perms_dir")" + + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e POSTGRES_SSL=true \ + -e POSTGRES_SSL_MODE=verify-ca \ + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt \ + -e POSTGRES_SSL_CERT=/certs/client.crt \ + -e POSTGRES_SSL_KEY=/certs/client.key \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with 0777 client key" + check_log_contains "$name" "Fixed PostgreSQL client key" \ + "Key permission fix triggered" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed after key fix" + check_migrations_done "$name" + else + log_fail "Container failed to start with 0777 client key" + fi + dump_logs_on_fail "$name" + rm -rf "$bad_perms_dir" + cleanup_scenario +} + +test_modular_no_tls_regression() { + CURRENT_SCENARIO="modular_no_tls_regression" + section "Modular mode — no TLS (regression check)" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + # Plain PostgreSQL — no TLS + docker run -d --name "$pg_name" --network "$net" \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=secret \ + -e POSTGRES_DB=dispatcharr \ + postgres:17 >/dev/null + + local elapsed=0 + while [ $elapsed -lt 30 ]; do + if docker exec "$pg_name" su postgres -c "/usr/lib/postgresql/17/bin/pg_isready" 2>/dev/null | grep -q "accepting"; then + break + fi + sleep 2; ((elapsed+=2)) + done + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=secret \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started without TLS (regression check)" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed without TLS" + check_log_absent "$name" "Fixed PostgreSQL client key" \ + "No key fix when TLS disabled" + check_migrations_done "$name" + else + log_fail "Container failed to start without TLS" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_pg_verify_full() { + CURRENT_SCENARIO="modular_pg_verify_full" + section "Modular mode — PG mTLS with verify-full (CN must match hostname)" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + start_tls_postgres "$pg_name" "$net" "cert" + + docker run -d --name "$redis_name" --network "$net" redis:latest >/dev/null + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + # verify-full requires server cert CN to match the hostname used to connect. + # Our PG server cert CN is "${TEST_PREFIX}_pg", which matches the container name + # used in POSTGRES_HOST. + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e POSTGRES_SSL=true \ + -e POSTGRES_SSL_MODE=verify-full \ + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt \ + -e POSTGRES_SSL_CERT=/certs/client.crt \ + -e POSTGRES_SSL_KEY=/certs/client.key \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with verify-full" + check_log_contains "$name" "PostgreSQL version check passed" \ + "Version check passed with verify-full" + check_log_contains "$name" "sslmode=verify-full" \ + "Django reports verify-full mode" + check_migrations_done "$name" + else + log_fail "Container failed to start with verify-full" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_redis_tls() { + CURRENT_SCENARIO="modular_redis_tls" + section "Modular mode — Redis with TLS" + + local name="${TEST_PREFIX}_app" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name"; track_container "$name" + + # Plain PG (no TLS) — isolate Redis TLS testing + docker run -d --name "$pg_name" --network "$net" \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=secret \ + -e POSTGRES_DB=dispatcharr \ + postgres:17 >/dev/null + + local elapsed=0 + while [ $elapsed -lt 30 ]; do + if docker exec "$pg_name" su postgres -c "/usr/lib/postgresql/17/bin/pg_isready" 2>/dev/null | grep -q "accepting"; then + break + fi + sleep 2; elapsed=$((elapsed + 2)) + done + + start_tls_redis "$redis_name" "$net" + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + docker run -d --name "$name" --network "$net" \ + -e DISPATCHARR_ENV=modular \ + -e POSTGRES_HOST="$pg_name" \ + -e POSTGRES_PORT=5432 \ + -e POSTGRES_USER=dispatch \ + -e POSTGRES_PASSWORD=secret \ + -e POSTGRES_DB=dispatcharr \ + -e REDIS_HOST="$redis_name" \ + -e REDIS_SSL=true \ + -e REDIS_SSL_VERIFY=false \ + -e REDIS_SSL_CA_CERT=/certs/ca.crt \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if wait_for_ready "$name"; then + log_pass "Container started with Redis TLS" + check_log_contains "$name" "Redis TLS: enabled" \ + "Django reports Redis TLS enabled" + check_log_contains "$name" "Redis at ${redis_name}" \ + "Redis connected via TLS" + check_migrations_done "$name" + else + log_fail "Container failed to start with Redis TLS" + fi + dump_logs_on_fail "$name" + cleanup_scenario +} + +test_modular_full_tls_celery() { + CURRENT_SCENARIO="modular_full_tls_celery" + section "Modular mode — PG mTLS + Redis TLS with Celery container" + + local name="${TEST_PREFIX}_app" + local celery_name="${TEST_PREFIX}_celery" + local pg_name="${TEST_PREFIX}_pg" + local redis_name="${TEST_PREFIX}_redis" + local net="${TEST_PREFIX}_net" + local vol="${name}_data" + cleanup_scenario + + docker network create "$net" >/dev/null 2>&1 + fresh_volume "$vol" + track_network "$net" + track_container "$pg_name"; track_container "$redis_name" + track_container "$name"; track_container "$celery_name" + + start_tls_postgres "$pg_name" "$net" "cert" + start_tls_redis "$redis_name" "$net" + + local cert_mount + cert_mount="$(cygpath -w "$CERT_DIR" 2>/dev/null || echo "$CERT_DIR")" + + # Shared env vars for both web and celery containers + local -a tls_env=( + -e DISPATCHARR_ENV=modular + -e POSTGRES_HOST="$pg_name" + -e POSTGRES_PORT=5432 + -e POSTGRES_USER=dispatch + -e POSTGRES_DB=dispatcharr + -e REDIS_HOST="$redis_name" + -e POSTGRES_SSL=true + -e POSTGRES_SSL_MODE=verify-ca + -e POSTGRES_SSL_CA_CERT=/certs/ca.crt + -e POSTGRES_SSL_CERT=/certs/client.crt + -e POSTGRES_SSL_KEY=/certs/client.key + -e REDIS_SSL=true + -e REDIS_SSL_VERIFY=false + -e REDIS_SSL_CA_CERT=/certs/ca.crt + ) + + # Start web container first (generates JWT, runs migrations) + docker run -d --name "$name" --network "$net" \ + "${tls_env[@]}" \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + "$IMAGE_NAME" >/dev/null + + if ! wait_for_ready "$name"; then + log_fail "Web container failed to start with full TLS" + dump_logs_on_fail "$name" + cleanup_scenario + return + fi + log_pass "Web container started with PG mTLS + Redis TLS" + + # Start Celery container (shares /data volume for JWT, waits for migrations) + docker run -d --name "$celery_name" --network "$net" \ + "${tls_env[@]}" \ + -e DJANGO_SETTINGS_MODULE=dispatcharr.settings \ + -e PYTHONUNBUFFERED=1 \ + -v "${cert_mount}:/certs:ro" \ + -v "${vol}:/data" \ + --entrypoint /app/docker/entrypoint.celery.sh \ + "$IMAGE_NAME" >/dev/null + + # Wait for Celery to start (look for "starting Celery" message) + local elapsed=0 + local celery_ok=false + while [ $elapsed -lt 90 ]; do + if ! docker ps -q -f "name=^${celery_name}$" 2>/dev/null | grep -q .; then + echo " Celery container exited unexpectedly" + break + fi + if docker logs "$celery_name" 2>&1 | grep -q "starting Celery"; then + celery_ok=true + break + fi + sleep 3; elapsed=$((elapsed + 3)) + done + + if [ "$celery_ok" = true ]; then + log_pass "Celery container started with PG mTLS + Redis TLS" + check_log_contains "$celery_name" "Migrations complete" \ + "Celery confirmed migrations complete via TLS" + check_log_contains "$celery_name" "PostgreSQL TLS: enabled" \ + "Celery sees PostgreSQL TLS enabled" + check_log_contains "$celery_name" "Redis TLS: enabled" \ + "Celery sees Redis TLS enabled" + else + log_fail "Celery container failed to start with full TLS" + echo -e " ${YELLOW}--- Celery logs ---${NC}" + docker logs "$celery_name" 2>&1 | tail -20 | sed 's/^/ /' + echo -e " ${YELLOW}--- End logs ---${NC}" + fi + + dump_logs_on_fail "$name" + cleanup_scenario +} + +############################################################################### +# Main +############################################################################### +echo -e "${BOLD}╔═══════════════════════════════════════════════════════════╗${NC}" +echo -e "${BOLD}║ Dispatcharr — TLS Integration Tests ║${NC}" +echo -e "${BOLD}╚═══════════════════════════════════════════════════════════╝${NC}" + +# Build image +if [ "$SKIP_BUILD" = false ]; then + echo -e "\n${BOLD}Building test image...${NC}" + if ! docker build -t "$IMAGE_NAME" -f docker/Dockerfile . 2>&1 | tail -5; then + echo -e "${RED}Build failed${NC}" + exit 1 + fi + echo -e "${GREEN}Build complete${NC}" +else + echo -e "\n${YELLOW}Skipping build (--skip-build)${NC}" +fi + +# Generate certificates +generate_test_certs || exit 1 + +# Run scenarios +SCENARIOS=( + modular_mtls_no_password + modular_mtls_with_password + modular_tls_server_only + modular_tls_key_permission + modular_no_tls_regression + modular_pg_verify_full + modular_redis_tls + modular_full_tls_celery +) + +for scenario in "${SCENARIOS[@]}"; do + if [ -n "$SINGLE_SCENARIO" ] && [ "$scenario" != "$SINGLE_SCENARIO" ]; then + continue + fi + "test_${scenario}" +done + +# Clean up certs +rm -rf "$CERT_DIR" + +# Summary +echo -e "\n${BOLD}═══════════════════════════════════════════════════════════${NC}" +echo -e " ${GREEN}Passed: $PASS${NC} ${RED}Failed: $FAIL${NC} ${YELLOW}Skipped: $SKIP${NC}" +if [ ${#ERRORS[@]} -gt 0 ]; then + echo -e "\n ${RED}Failures:${NC}" + for err in "${ERRORS[@]}"; do + echo -e " ${RED}• $err${NC}" + done +fi +echo -e "${BOLD}═══════════════════════════════════════════════════════════${NC}" + +[ $FAIL -eq 0 ] diff --git a/docker/uwsgi.dev.ini b/docker/uwsgi.dev.ini index c4c5d0aa..51aae9a5 100644 --- a/docker/uwsgi.dev.ini +++ b/docker/uwsgi.dev.ini @@ -8,7 +8,7 @@ exec-pre = python /app/scripts/wait_for_redis.py ; Start Redis first -attach-daemon = redis-server +attach-daemon = redis-server --protected-mode no ; Then start other services with configurable nice level (default: 5 for low priority) ; Users can override via CELERY_NICE_LEVEL environment variable in docker-compose attach-daemon = nice -n $(CELERY_NICE_LEVEL) celery -A dispatcharr worker --autoscale=6,1 @@ -57,4 +57,4 @@ logformat-strftime = true log-date = %%Y-%%m-%%d %%H:%%M:%%S,000 # Use formatted time with environment variable for log level log-format = %(ftime) $(DISPATCHARR_LOG_LEVEL) uwsgi.requests Worker ID: %(wid) %(method) %(status) %(uri) %(msecs)ms -log-buffering = 1024 # Add buffer size limit for logging \ No newline at end of file +log-buffering = 1024 # Add buffer size limit for logging diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 9a0adea6..6880b99a 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -2704,16 +2704,16 @@ } }, "node_modules/brace-expansion": { - "version": "5.0.2", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.2.tgz", - "integrity": "sha512-Pdk8c9poy+YhOgVWw1JNN22/HcivgKWwpxKq04M/jTmHyCZn12WPJebZxdjSa5TmBqISrUSgNYU3eRORljfCCw==", + "version": "5.0.5", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.5.tgz", + "integrity": "sha512-VZznLgtwhn+Mact9tfiwx64fA9erHH/MCXEUfB/0bX/6Fz6ny5EGTXYltMocqg4xFAQZtnO3DHWWXi8RiuN7cQ==", "dev": true, "license": "MIT", "dependencies": { "balanced-match": "^4.0.2" }, "engines": { - "node": "20 || >=22" + "node": "18 || 20 || >=22" } }, "node_modules/cac": { @@ -2850,9 +2850,9 @@ } }, "node_modules/cosmiconfig/node_modules/yaml": { - "version": "1.10.2", - "resolved": "https://registry.npmjs.org/yaml/-/yaml-1.10.2.tgz", - "integrity": "sha512-r3vXyErRCYJ7wg28yvBY5VSoAF8ZvlcW9/BwUzEtUsjvX/DKs24dIkuwjtuprwJJHsbyUbLApepYTR1BN4uHrg==", + "version": "1.10.3", + "resolved": "https://registry.npmjs.org/yaml/-/yaml-1.10.3.tgz", + "integrity": "sha512-vIYeF1u3CjlhAFekPPAk2h/Kv4T3mAkMox5OymRiJQB0spDP10LHvt+K7G9Ny6NuuMAb25/6n1qyUjAcGNf/AA==", "license": "ISC", "engines": { "node": ">= 6" @@ -3554,9 +3554,9 @@ } }, "node_modules/flatted": { - "version": "3.4.1", - "resolved": "https://registry.npmjs.org/flatted/-/flatted-3.4.1.tgz", - "integrity": "sha512-IxfVbRFVlV8V/yRaGzk0UVIcsKKHMSfYw66T/u4nTwlWteQePsxe//LjudR1AMX4tZW3WFCh3Zqa/sjlqpbURQ==", + "version": "3.4.2", + "resolved": "https://registry.npmjs.org/flatted/-/flatted-3.4.2.tgz", + "integrity": "sha512-PjDse7RzhcPkIJwy5t7KPWQSZ9cAbzQXcafsetQoD7sOJRQlGikNbx7yZp2OotDnJyrDcbyRq3Ttb18iYOqkxA==", "dev": true, "license": "ISC" }, @@ -4362,9 +4362,9 @@ "license": "ISC" }, "node_modules/picomatch": { - "version": "4.0.3", - "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.3.tgz", - "integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==", + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", + "integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", "dev": true, "license": "MIT", "engines": { @@ -5794,6 +5794,24 @@ "dev": true, "license": "MIT" }, + "node_modules/yaml": { + "version": "2.8.3", + "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.8.3.tgz", + "integrity": "sha512-AvbaCLOO2Otw/lW5bmh9d/WEdcDFdQp2Z2ZUH3pX9U2ihyUY0nvLv7J6TrWowklRGPYbB/IuIMfYgxaCPg5Bpg==", + "dev": true, + "license": "ISC", + "optional": true, + "peer": true, + "bin": { + "yaml": "bin.mjs" + }, + "engines": { + "node": ">= 14.6" + }, + "funding": { + "url": "https://github.com/sponsors/eemeli" + } + }, "node_modules/yocto-queue": { "version": "0.1.0", "resolved": "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz", diff --git a/frontend/src/api.js b/frontend/src/api.js index b1c103d8..0f76c2a2 100644 --- a/frontend/src/api.js +++ b/frontend/src/api.js @@ -1601,9 +1601,7 @@ export default class API { }); const playlist = await API.getPlaylist(accountId); - usePlaylistsStore - .getState() - .updateProfiles(playlist.id, playlist.profiles); + usePlaylistsStore.getState().updatePlaylist(playlist); } catch (e) { errorNotification(`Failed to update profile for account ${accountId}`, e); } @@ -3000,12 +2998,15 @@ export default class API { } } - static async updateUser(id, body) { + static async updateUser(id, body, self = false) { try { - const response = await request(`${host}/api/accounts/users/${id}/`, { - method: 'PATCH', - body, - }); + const response = await request( + `${host}/api/accounts/users/${self ? 'me' : id}/`, + { + method: 'PATCH', + body, + } + ); useUsersStore.getState().updateUser(response); diff --git a/frontend/src/components/FloatingVideo.jsx b/frontend/src/components/FloatingVideo.jsx index 0a63e53f..f5bdc303 100644 --- a/frontend/src/components/FloatingVideo.jsx +++ b/frontend/src/components/FloatingVideo.jsx @@ -2,6 +2,7 @@ import React, { useCallback, useEffect, useRef, useState } from 'react'; import Draggable from 'react-draggable'; import useVideoStore from '../store/useVideoStore'; +import useAuthStore from '../store/auth'; import mpegts from 'mpegts.js'; import { CloseButton, Flex, Loader, Text, Box } from '@mantine/core'; import { @@ -117,6 +118,7 @@ export default function FloatingVideo() { const contentType = useVideoStore((s) => s.contentType); const metadata = useVideoStore((s) => s.metadata); const hideVideo = useVideoStore((s) => s.hideVideo); + const accessToken = useAuthStore((s) => s.accessToken); const videoRef = useRef(null); const playerRef = useRef(null); @@ -300,7 +302,8 @@ export default function FloatingVideo() { setLoadError(null); console.log('Initializing live stream player for:', streamUrl); - + console.log('Access token present:', !!accessToken); + console.log('Current access token from auth store:', accessToken); try { if (!mpegts.getFeatureList().mseLivePlayback) { setIsLoading(false); @@ -310,20 +313,34 @@ export default function FloatingVideo() { return; } - const player = mpegts.createPlayer({ - type: 'mpegts', - url: streamUrl, - isLive: true, - enableWorker: true, - enableStashBuffer: false, - liveBufferLatencyChasing: true, - liveSync: true, - cors: true, - autoCleanupSourceBuffer: true, - autoCleanupMaxBackwardDuration: 10, - autoCleanupMinBackwardDuration: 5, - reuseRedirectedURL: true, - }); + // mpegts.js workers run in WorkerGlobalScope where relative URLs are + // not resolved against the page origin. Always pass an absolute URL. + const absoluteStreamUrl = + streamUrl.startsWith('/') && typeof window !== 'undefined' + ? `${window.location.origin}${streamUrl}` + : streamUrl; + + const player = mpegts.createPlayer( + { + type: 'mpegts', + url: absoluteStreamUrl, + isLive: true, + cors: true, + }, + { + enableWorker: true, + enableStashBuffer: false, + liveBufferLatencyChasing: false, + liveSync: false, + autoCleanupSourceBuffer: true, + autoCleanupMaxBackwardDuration: 120, + autoCleanupMinBackwardDuration: 60, + reuseRedirectedURL: true, + headers: accessToken + ? { Authorization: `Bearer ${accessToken}` } + : undefined, + } + ); player.attachMediaElement(videoRef.current); diff --git a/frontend/src/components/GuideRow.jsx b/frontend/src/components/GuideRow.jsx index a1cdaed9..a13ddcdf 100644 --- a/frontend/src/components/GuideRow.jsx +++ b/frontend/src/components/GuideRow.jsx @@ -5,7 +5,7 @@ import { MINUTE_BLOCK_WIDTH, MINUTE_INCREMENT, PROGRAM_HEIGHT, -} from '../pages/guideUtils.js'; +} from '../utils/guideUtils.js'; import { Box, Flex, Text, Tooltip } from '@mantine/core'; import { Play } from 'lucide-react'; import logo from '../images/logo.png'; diff --git a/frontend/src/components/HourTimeline.jsx b/frontend/src/components/HourTimeline.jsx index 0d5e70b4..5e1fc621 100644 --- a/frontend/src/components/HourTimeline.jsx +++ b/frontend/src/components/HourTimeline.jsx @@ -1,7 +1,7 @@ import React from 'react'; import { Box, Text } from '@mantine/core'; import { format } from '../utils/dateTimeUtils.js'; -import { HOUR_WIDTH } from '../pages/guideUtils.js'; +import { HOUR_WIDTH } from '../utils/guideUtils.js'; const HourBlock = React.memo( ({ hourData, timeFormat, formatDayLabel, handleTimeClick }) => { diff --git a/frontend/src/components/ProgramDetailModal.jsx b/frontend/src/components/ProgramDetailModal.jsx index 07ff77e0..89388cb4 100644 --- a/frontend/src/components/ProgramDetailModal.jsx +++ b/frontend/src/components/ProgramDetailModal.jsx @@ -16,7 +16,7 @@ import API from '../api'; import useVideoStore from '../store/useVideoStore'; import useSettingsStore from '../store/settings'; import { getShowVideoUrl } from '../utils/cards/RecordingCardUtils'; -import { formatSeasonEpisode } from '../pages/guideUtils'; +import { formatSeasonEpisode } from '../utils/guideUtils'; import { format, initializeTime, diff --git a/frontend/src/components/Sidebar.jsx b/frontend/src/components/Sidebar.jsx index 73ee13c3..f5332d3a 100644 --- a/frontend/src/components/Sidebar.jsx +++ b/frontend/src/components/Sidebar.jsx @@ -1,12 +1,7 @@ import React, { useRef, useState, useMemo } from 'react'; import { Link, useLocation } from 'react-router-dom'; import { copyToClipboard } from '../utils'; -import { - Copy, - LogOut, - ChevronDown, - ChevronRight, -} from 'lucide-react'; +import { Copy, LogOut, ChevronDown, ChevronRight, Heart } from 'lucide-react'; import { getOrderedNavItems } from '../config/navigation'; import { Avatar, @@ -19,6 +14,7 @@ import { ActionIcon, AppShellNavbar, ScrollArea, + Tooltip, } from '@mantine/core'; import logo from '../images/logo.png'; import useChannelsStore from '../store/channels'; @@ -29,6 +25,21 @@ import { USER_LEVELS } from '../constants'; import UserForm from './forms/User'; import NotificationCenter from './NotificationCenter'; +const DonateButton = ({ tooltipPosition = 'top' }) => ( + + + + + +); + const NavLink = ({ item, isActive, collapsed }) => { const IconComponent = item.icon; return ( @@ -328,24 +339,56 @@ const Sidebar = ({ collapsed, toggleDrawer, drawerWidth, miniDrawerWidth }) => { {!collapsed && ( - - v{appVersion?.version || '0.0.0'} - {appVersion?.timestamp ? `-${appVersion.timestamp}` : ''} - - {isAuthenticated && } + + + copyToClipboard( + `v${appVersion?.version || '0.0.0'}${appVersion?.timestamp ? `-${appVersion.timestamp}` : ''}`, + { + successTitle: 'Copied', + successMessage: 'Version copied to clipboard', + } + ) + } + > + v{appVersion?.version || '0.0.0'} + {appVersion?.timestamp ? `-${appVersion.timestamp}` : ''} + + + + + {isAuthenticated && } + )} - {collapsed && isAuthenticated && ( + {collapsed && ( - + {isAuthenticated && } + )} diff --git a/frontend/src/components/__tests__/FloatingVideo.test.jsx b/frontend/src/components/__tests__/FloatingVideo.test.jsx index 63f3ee3d..b511b60b 100644 --- a/frontend/src/components/__tests__/FloatingVideo.test.jsx +++ b/frontend/src/components/__tests__/FloatingVideo.test.jsx @@ -150,6 +150,10 @@ describe('FloatingVideo', () => { type: 'mpegts', url: 'http://example.com/stream.ts', isLive: true, + }), + expect.objectContaining({ + enableWorker: true, + enableStashBuffer: false, }) ); }); diff --git a/frontend/src/components/__tests__/GuideRow.test.jsx b/frontend/src/components/__tests__/GuideRow.test.jsx index a78d6978..703ff76a 100644 --- a/frontend/src/components/__tests__/GuideRow.test.jsx +++ b/frontend/src/components/__tests__/GuideRow.test.jsx @@ -6,7 +6,7 @@ import { CHANNEL_WIDTH, HOUR_WIDTH, PROGRAM_HEIGHT, -} from '../../pages/guideUtils'; +} from '../../utils/guideUtils'; // Mock logo import vi.mock('../../images/logo.png', () => ({ diff --git a/frontend/src/components/__tests__/HourTimeline.test.jsx b/frontend/src/components/__tests__/HourTimeline.test.jsx index 372ec155..393e559e 100644 --- a/frontend/src/components/__tests__/HourTimeline.test.jsx +++ b/frontend/src/components/__tests__/HourTimeline.test.jsx @@ -3,7 +3,7 @@ import { render, screen, fireEvent } from '@testing-library/react'; import '@testing-library/jest-dom'; import HourTimeline from '../HourTimeline'; import { format } from '../../utils/dateTimeUtils'; -import { HOUR_WIDTH } from '../../pages/guideUtils'; +import { HOUR_WIDTH } from '../../utils/guideUtils'; // Mock date utilities vi.mock('../../utils/dateTimeUtils', () => ({ @@ -108,7 +108,9 @@ describe('HourTimeline', () => { /> ); - const hourBlocks = container.querySelectorAll('[style*="cursor: pointer"]'); + const hourBlocks = container.querySelectorAll( + '[style*="cursor: pointer"]' + ); hourBlocks.forEach((block) => { expect(block).toHaveAttribute('w', `${HOUR_WIDTH}`); expect(block).toHaveAttribute('h', '40px'); @@ -126,16 +128,16 @@ describe('HourTimeline', () => { /> ); - const hourBlocks = container.querySelectorAll('[style*="cursor: pointer"]'); + const hourBlocks = container.querySelectorAll( + '[style*="cursor: pointer"]' + ); expect(hourBlocks[0]).toHaveStyle({ backgroundColor: '#1B2421', }); }); it('should apply special styling for new day blocks', () => { - const newDayTimeline = [ - { time: mockTime3, isNewDay: true }, - ]; + const newDayTimeline = [{ time: mockTime3, isNewDay: true }]; const { container } = render( { }); it('should apply bold font weight to day label on new day', () => { - const newDayTimeline = [ - { time: mockTime3, isNewDay: true }, - ]; + const newDayTimeline = [{ time: mockTime3, isNewDay: true }]; const { container } = render( { /> ); - const markers = container.querySelectorAll('[style*="background-color: rgb(113, 128, 150);"]'); + const markers = container.querySelectorAll( + '[style*="background-color: rgb(113, 128, 150);"]' + ); expect(markers.length).toBe(3); // 15, 30, 45 minute markers }); @@ -212,7 +214,9 @@ describe('HourTimeline', () => { /> ); - const markers = container.querySelectorAll('[style*="backgroundColor: #718096"]'); + const markers = container.querySelectorAll( + '[style*="backgroundColor: #718096"]' + ); const positions = ['25%', '50%', '75%']; markers.forEach((marker, index) => { @@ -238,10 +242,15 @@ describe('HourTimeline', () => { /> ); - const hourBlocks = container.querySelectorAll('[style*="cursor: pointer"]'); + const hourBlocks = container.querySelectorAll( + '[style*="cursor: pointer"]' + ); fireEvent.click(hourBlocks[0]); - expect(mockHandleTimeClick).toHaveBeenCalledWith(mockTime1, expect.any(Object)); + expect(mockHandleTimeClick).toHaveBeenCalledWith( + mockTime1, + expect.any(Object) + ); }); it('should call handleTimeClick with correct time for each block', () => { @@ -254,13 +263,21 @@ describe('HourTimeline', () => { /> ); - const hourBlocks = container.querySelectorAll('[style*="cursor: pointer"]'); + const hourBlocks = container.querySelectorAll( + '[style*="cursor: pointer"]' + ); fireEvent.click(hourBlocks[0]); - expect(mockHandleTimeClick).toHaveBeenCalledWith(mockTime1, expect.any(Object)); + expect(mockHandleTimeClick).toHaveBeenCalledWith( + mockTime1, + expect.any(Object) + ); fireEvent.click(hourBlocks[1]); - expect(mockHandleTimeClick).toHaveBeenCalledWith(mockTime2, expect.any(Object)); + expect(mockHandleTimeClick).toHaveBeenCalledWith( + mockTime2, + expect.any(Object) + ); }); }); diff --git a/frontend/src/components/__tests__/M3URefreshNotification.test.jsx b/frontend/src/components/__tests__/M3URefreshNotification.test.jsx index 2c5a5572..43375fa7 100644 --- a/frontend/src/components/__tests__/M3URefreshNotification.test.jsx +++ b/frontend/src/components/__tests__/M3URefreshNotification.test.jsx @@ -57,6 +57,7 @@ vi.mock('@mantine/core', async () => { // Mock lucide-react icons vi.mock('lucide-react', () => ({ + ListOrdered: () =>
, CircleCheck: () =>
, })); diff --git a/frontend/src/components/__tests__/NotificationCenter.test.jsx b/frontend/src/components/__tests__/NotificationCenter.test.jsx index 9dfb13f5..2231c951 100644 --- a/frontend/src/components/__tests__/NotificationCenter.test.jsx +++ b/frontend/src/components/__tests__/NotificationCenter.test.jsx @@ -33,8 +33,12 @@ vi.mock('@mantine/core', async () => { {children}
), - PopoverTarget: ({ children }) =>
{children}
, - PopoverDropdown: ({ children }) =>
{children}
, + PopoverTarget: ({ children }) => ( +
{children}
+ ), + PopoverDropdown: ({ children }) => ( +
{children}
+ ), Indicator: ({ children, label, disabled, processing }) => (
{
), ActionIcon: ({ children, onClick, 'aria-label': ariaLabel, ...props }) => ( - ), - ScrollAreaAutosize: ({ children }) =>
{children}
, - Badge: ({ children, ...props }) => {children}, - Card: ({ children, ...props }) =>
{children}
, - ThemeIcon: ({ children, ...props }) =>
{children}
, - Group: ({ children, ...props }) =>
{children}
, - Stack: ({ children, ...props }) =>
{children}
, - Box: ({ children, ...props }) =>
{children}
, - Text: ({ children, ...props }) => {children}, + ScrollAreaAutosize: ({ children }) => ( +
{children}
+ ), + Badge: ({ children, ...props }) => ( + + {children} + + ), + Card: ({ children, ...props }) => ( +
+ {children} +
+ ), + ThemeIcon: ({ children, ...props }) => ( +
+ {children} +
+ ), + Group: ({ children, ...props }) => ( +
+ {children} +
+ ), + Stack: ({ children, ...props }) => ( +
+ {children} +
+ ), + Box: ({ children, ...props }) => ( +
+ {children} +
+ ), + Text: ({ children, ...props }) => ( + + {children} + + ), Button: ({ children, onClick, ...props }) => ( +
{children}
); }, - Box: ({ children, ...props }) =>
{children}
, + Box: ({ children, ...props }) => ( +
+ {children} +
+ ), Button: ({ children, onClick, disabled, ...props }) => ( - ), - Flex: ({ children, ...props }) =>
{children}
, - Group: ({ children, ...props }) =>
{children}
, + Flex: ({ children, ...props }) => ( +
+ {children} +
+ ), + Group: ({ children, ...props }) => ( +
+ {children} +
+ ), Image: ({ src, alt, ...props }) => ( {alt} ), - Text: ({ children, ...props }) =>
{children}
, - Title: ({ children, order, ...props }) => ( -
{children}
+ Text: ({ children, ...props }) => ( +
+ {children} +
), - Select: ({ value, onChange, data, label, placeholder, disabled, ...props }) => ( + Title: ({ children, order, ...props }) => ( +
+ {children} +
+ ), + Select: ({ + value, + onChange, + data, + label, + placeholder, + disabled, + ...props + }) => (
), - Badge: ({ children, ...props }) => {children}, + Badge: ({ children, ...props }) => ( + + {children} + + ), Loader: (props) =>
, - Stack: ({ children, ...props }) =>
{children}
, + Stack: ({ children, ...props }) => ( +
+ {children} +
+ ), ActionIcon: ({ children, onClick, disabled, ...props }) => ( - ), Tabs: ({ children, value, onChange, ...props }) => (
-
{ - const tab = e.target.closest('[data-tab-value]'); - if (tab) onChange?.(tab.dataset.tabValue); - }}> +
{ + const tab = e.target.closest('[data-tab-value]'); + if (tab) onChange?.(tab.dataset.tabValue); + }} + > {children}
), TabsList: ({ children }) =>
{children}
, TabsTab: ({ children, value }) => ( - + ), TabsPanel: ({ children, value }) => ( -
{children}
+
+ {children} +
+ ), + Table: ({ children, ...props }) => ( + + {children} +
+ ), + TableThead: ({ children }) => ( + {children} + ), + TableTbody: ({ children }) => ( + {children} ), - Table: ({ children, ...props }) => {children}
, - TableThead: ({ children }) => {children}, - TableTbody: ({ children }) => {children}, TableTr: ({ children, onClick, ...props }) => ( - {children} + + {children} + + ), + TableTh: ({ children, ...props }) => ( + + {children} + + ), + TableTd: ({ children, ...props }) => ( + + {children} + ), - TableTh: ({ children, ...props }) => {children}, - TableTd: ({ children, ...props }) => {children}, Divider: (props) =>
, }; }); @@ -168,7 +239,7 @@ describe('SeriesModal', () => { m3u_account: { name: 'Provider 2' }, stream_name: 'Test Series 720p', quality_info: null, - } + }, ]; beforeEach(() => { @@ -187,9 +258,15 @@ describe('SeriesModal', () => { environment: { env_mode: 'prod' }, }; - useVODStore.mockImplementation((selector) => selector ? selector(mockVODStore) : mockVODStore); - useVideoStore.mockImplementation((selector) => selector ? selector(mockVideoStore) : mockVideoStore); - useSettingsStore.mockImplementation((selector) => selector ? selector(mockSettingsStore) : mockSettingsStore); + useVODStore.mockImplementation((selector) => + selector ? selector(mockVODStore) : mockVODStore + ); + useVideoStore.mockImplementation((selector) => + selector ? selector(mockVideoStore) : mockVideoStore + ); + useSettingsStore.mockImplementation((selector) => + selector ? selector(mockSettingsStore) : mockSettingsStore + ); copyToClipboard.mockResolvedValue(undefined); }); @@ -325,23 +402,37 @@ describe('SeriesModal', () => { it('should display IMDB link when imdb_id exists', async () => { render( - + ); await waitFor(() => { const link = screen.getByText(/IMDB/i).closest('a'); - expect(link).toHaveAttribute('href', 'https://www.imdb.com/title/tt1234567'); + expect(link).toHaveAttribute( + 'href', + 'https://www.imdb.com/title/tt1234567' + ); }); }); it('should display TMDB link when tmdb_id exists', async () => { render( - + ); await waitFor(() => { const link = screen.getByText(/TMDB/i).closest('a'); - expect(link).toHaveAttribute('href', 'https://www.themoviedb.org/tv/12345'); + expect(link).toHaveAttribute( + 'href', + 'https://www.themoviedb.org/tv/12345' + ); }); }); }); @@ -446,7 +537,12 @@ describe('SeriesModal', () => { }); it('should sort episodes by episode number', async () => { - const episode2 = { ...mockEpisode, id: 2, episode_number: 2, name: 'Second Episode' }; + const episode2 = { + ...mockEpisode, + id: 2, + episode_number: 2, + name: 'Second Episode', + }; mockVODStore.fetchSeriesInfo.mockResolvedValue({ ...mockDetailedSeries, episodesList: [episode2, mockEpisode], @@ -592,7 +688,12 @@ describe('SeriesModal', () => { describe('Season Tabs', () => { it('should create tabs for each season', async () => { - const season2Episode = { ...mockEpisode, id: 2, season_number: 2, episode_num: 1 }; + const season2Episode = { + ...mockEpisode, + id: 2, + season_number: 2, + episode_num: 1, + }; mockVODStore.fetchSeriesInfo.mockResolvedValue({ ...mockDetailedSeries, episodesList: [mockEpisode, season2Episode], @@ -670,7 +771,6 @@ describe('SeriesModal', () => { ); - await waitFor(() => { expect(screen.getByText(/Provider 1 - 1080p/)).toBeInTheDocument(); }); @@ -797,7 +897,7 @@ describe('SeriesModal', () => { ); await waitFor(() => { - expect(screen.getByText('Test Account')).toBeInTheDocument() + expect(screen.getByText('Test Account')).toBeInTheDocument(); }); }); }); diff --git a/frontend/src/components/__tests__/Sidebar.test.jsx b/frontend/src/components/__tests__/Sidebar.test.jsx index cc5e8be2..e92b2f39 100644 --- a/frontend/src/components/__tests__/Sidebar.test.jsx +++ b/frontend/src/components/__tests__/Sidebar.test.jsx @@ -56,6 +56,7 @@ vi.mock('lucide-react', () => ({ ChevronRight: () =>
, MonitorCog: () =>
, Blocks: () =>
, + Heart: () =>
, })); // Mock UserForm component @@ -114,6 +115,7 @@ vi.mock('@mantine/core', async () => { ), ScrollArea: ({ children }) =>
{children}
, + Tooltip: ({ children }) => <>{children}, }; }); diff --git a/frontend/src/components/cards/StreamConnectionCard.jsx b/frontend/src/components/cards/StreamConnectionCard.jsx index 56d4d174..86dbc240 100644 --- a/frontend/src/components/cards/StreamConnectionCard.jsx +++ b/frontend/src/components/cards/StreamConnectionCard.jsx @@ -2,6 +2,7 @@ import { useLocation } from 'react-router-dom'; import React, { useEffect, useMemo, useState } from 'react'; import usePlaylistsStore from '../../store/playlists.jsx'; import useSettingsStore from '../../store/settings.jsx'; +import useUsersStore from '../../store/users.jsx'; import { ActionIcon, Badge, @@ -123,6 +124,8 @@ const StreamConnectionCard = ({ // Get M3U account data from the playlists store const m3uAccounts = usePlaylistsStore((s) => s.playlists); + // Get users for resolving user_id → username on client rows + const users = useUsersStore((s) => s.users); // Get settings for speed threshold and environment mode const settings = useSettingsStore((s) => s.settings); const env_mode = @@ -138,6 +141,15 @@ const StreamConnectionCard = ({ return getM3uAccountsMap(m3uAccounts); }, [m3uAccounts]); + // Create a map of user IDs to usernames for quick lookup + const usersMap = useMemo(() => { + const map = {}; + users.forEach((u) => { + map[String(u.id)] = u.username; + }); + return map; + }, [users]); + // Update M3U profile information when channel data changes useEffect(() => { // If the channel data includes M3U profile information, update our state @@ -314,7 +326,8 @@ const StreamConnectionCard = ({ { header: 'IP Address', accessorKey: 'ip_address', - size: 150, + grow: true, + minSize: 85, cell: ({ cell }) => ( @@ -323,10 +336,29 @@ const StreamConnectionCard = ({ ), }, + { + id: 'user', + header: 'User', + grow: true, + minSize: 60, + accessorFn: (row) => { + const uid = row.user_id ? String(row.user_id) : null; + if (!uid || uid === '0') return 'Anonymous'; + return usersMap[uid] || `User ${uid}`; + }, + cell: ({ cell }) => ( + + {cell.getValue()} + + ), + }, // Updated Connected column with tooltip { id: 'connected', header: 'Connected', + grow: 1.5, + minSize: 70, + maxSize: 150, accessorFn: connectedAccessor(fullDateTimeFormat), cell: ({ cell }) => ( - {cell.getValue()} + + {cell.getValue()} + ), }, @@ -344,6 +378,8 @@ const StreamConnectionCard = ({ { id: 'duration', header: 'Duration', + size: 82, + minSize: 60, accessorFn: durationAccessor(), cell: ({ cell, row }) => { const exactDuration = @@ -356,7 +392,9 @@ const StreamConnectionCard = ({ : 'Unknown duration' } > - {cell.getValue()} + + {cell.getValue()} + ); }, @@ -364,10 +402,11 @@ const StreamConnectionCard = ({ { id: 'actions', header: 'Actions', - size: 100, + size: 60, + minSize: 40, }, ], - [fullDateTimeFormat] + [fullDateTimeFormat, usersMap] ); const channelClientsTable = useTable({ @@ -383,6 +422,7 @@ const StreamConnectionCard = ({ }), headerCellRenderFns: { ip_address: renderHeaderCell, + user: renderHeaderCell, connected: renderHeaderCell, duration: renderHeaderCell, actions: renderHeaderCell, @@ -610,8 +650,9 @@ const StreamConnectionCard = ({ {currentProgram && isProgramDescExpanded && currentProgram.start_time && - currentProgram.end_time && - } + currentProgram.end_time && ( + + )} {/* Add stream selection dropdown and preview button */} {availableStreams.length > 0 && ( diff --git a/frontend/src/components/cards/VodConnectionCard.jsx b/frontend/src/components/cards/VodConnectionCard.jsx index 5e081c03..90b4c034 100644 --- a/frontend/src/components/cards/VodConnectionCard.jsx +++ b/frontend/src/components/cards/VodConnectionCard.jsx @@ -1,5 +1,5 @@ // Format duration for content length -import React, { useCallback, useEffect, useState } from 'react'; +import React, { useCallback, useEffect, useMemo, useState } from 'react'; import logo from '../../images/logo.png'; import { ActionIcon, @@ -38,6 +38,7 @@ import { getMovieDisplayTitle, getMovieSubtitle, } from '../../utils/cards/VodConnectionCardUtils.js'; +import useUsersStore from '../../store/users.jsx'; const ClientDetails = ({ connection, connectionStartTime }) => { return ( @@ -142,7 +143,10 @@ const ClientDetails = ({ connection, connectionStartTime }) => { }; const ConnectionProgress = ({ connection, durationSecs }) => { - const { totalTime, currentTime, percentage } = calculateProgress(connection, durationSecs); + const { totalTime, currentTime, percentage } = calculateProgress( + connection, + durationSecs + ); return totalTime > 0 ? ( @@ -172,6 +176,14 @@ const ConnectionProgress = ({ connection, durationSecs }) => { const VodConnectionCard = ({ vodContent, stopVODClient }) => { const { fullDateTimeFormat } = useDateTimeFormat(); const [isClientExpanded, setIsClientExpanded] = useState(false); + const users = useUsersStore((s) => s.users); + const usersMap = useMemo(() => { + const map = {}; + users.forEach((u) => { + map[String(u.id)] = u.username; + }); + return map; + }, [users]); const [, setUpdateTrigger] = useState(0); // Force re-renders for progress updates // Get metadata from the VOD content @@ -377,13 +389,12 @@ const VodConnectionCard = ({ vodContent, stopVODClient }) => { {/* Progress bar - show current position in content */} - {connection && - metadata.duration_secs && + {connection && metadata.duration_secs && ( - } + )} {/* Client information section - collapsible like channel cards */} {connection && ( @@ -403,11 +414,21 @@ const VodConnectionCard = ({ vodContent, stopVODClient }) => { > - Client: + Client IP: {connection.client_ip || 'Unknown IP'} + {usersMap[String(connection.user_id)] && ( + <> + + User: + + + {usersMap[String(connection.user_id)]} + + + )} diff --git a/frontend/src/components/cards/__tests__/RecordingCard.test.jsx b/frontend/src/components/cards/__tests__/RecordingCard.test.jsx index 9566f2b5..24e8dc46 100644 --- a/frontend/src/components/cards/__tests__/RecordingCard.test.jsx +++ b/frontend/src/components/cards/__tests__/RecordingCard.test.jsx @@ -109,13 +109,12 @@ vi.mock('@mantine/core', async () => ({ {children} ), - Tooltip: ({ children, label }) => ( -
{children}
- ), + Tooltip: ({ children, label }) =>
{children}
, })); // ── lucide-react ─────────────────────────────────────────────────────────────── vi.mock('lucide-react', () => ({ + ListOrdered: () => , AlertTriangle: () => , Plus: () => , Square: () => , @@ -140,8 +139,13 @@ vi.mock('../../../images/logo.png', () => ({ default: 'default-logo.png' })); import useChannelsStore from '../../../store/channels.jsx'; import useSettingsStore from '../../../store/settings.jsx'; import useVideoStore from '../../../store/useVideoStore.jsx'; -import { useDateTimeFormat, useTimeHelpers, format, isAfter, isBefore } - from '../../../utils/dateTimeUtils.js'; +import { + useDateTimeFormat, + useTimeHelpers, + format, + isAfter, + isBefore, +} from '../../../utils/dateTimeUtils.js'; import { notifications } from '@mantine/notifications'; import * as RecordingCardUtils from '../../../utils/cards/RecordingCardUtils.js'; import dayjs from 'dayjs'; @@ -187,7 +191,11 @@ const makeChannel = () => ({ }); /** Wire up all store/utility mocks with sensible defaults */ -const setupMocks = ({ now = NOW, recording = makeRecording(), channel = makeChannel() } = {}) => { +const setupMocks = ({ + now = NOW, + recording = makeRecording(), + channel = makeChannel(), +} = {}) => { const nowMoment = makeMoment(now); const startMoment = makeMoment(recording.start_time); const endMoment = makeMoment(recording.end_time); @@ -226,9 +234,13 @@ const setupMocks = ({ now = NOW, recording = makeRecording(), channel = makeChan vi.mocked(RecordingCardUtils.getPosterUrl).mockReturnValue('/poster.jpg'); vi.mocked(RecordingCardUtils.getChannelLogoUrl).mockReturnValue('/logo.png'); - vi.mocked(RecordingCardUtils.getRecordingUrl).mockReturnValue('/recordings/test.ts'); + vi.mocked(RecordingCardUtils.getRecordingUrl).mockReturnValue( + '/recordings/test.ts' + ); vi.mocked(RecordingCardUtils.getSeasonLabel).mockReturnValue(''); - vi.mocked(RecordingCardUtils.getSeriesInfo).mockReturnValue({ seriesId: 's1' }); + vi.mocked(RecordingCardUtils.getSeriesInfo).mockReturnValue({ + seriesId: 's1', + }); vi.mocked(RecordingCardUtils.getShowVideoUrl).mockReturnValue('/live/ch-1'); return { mockShowVideo, mockFetchRecordings }; @@ -237,10 +249,18 @@ const setupMocks = ({ now = NOW, recording = makeRecording(), channel = makeChan describe('RecordingCard', () => { beforeEach(() => { vi.clearAllMocks(); - vi.mocked(RecordingCardUtils.stopRecordingById).mockResolvedValue(undefined); - vi.mocked(RecordingCardUtils.deleteRecordingById).mockResolvedValue(undefined); - vi.mocked(RecordingCardUtils.deleteSeriesAndRule).mockResolvedValue(undefined); - vi.mocked(RecordingCardUtils.extendRecordingById).mockResolvedValue(undefined); + vi.mocked(RecordingCardUtils.stopRecordingById).mockResolvedValue( + undefined + ); + vi.mocked(RecordingCardUtils.deleteRecordingById).mockResolvedValue( + undefined + ); + vi.mocked(RecordingCardUtils.deleteSeriesAndRule).mockResolvedValue( + undefined + ); + vi.mocked(RecordingCardUtils.extendRecordingById).mockResolvedValue( + undefined + ); vi.mocked(RecordingCardUtils.runComSkip).mockResolvedValue(undefined); vi.mocked(RecordingCardUtils.removeRecording).mockReturnValue(undefined); }); @@ -250,17 +270,23 @@ describe('RecordingCard', () => { describe('rendering', () => { it('renders the recording title', () => { setupMocks(); - render(); + render( + + ); expect(screen.getByText('Test Show')).toBeInTheDocument(); }); it('renders "Custom Recording" when no program title', () => { setupMocks({ - recording: makeRecording({ custom_properties: { status: 'completed', program: {} } }), + recording: makeRecording({ + custom_properties: { status: 'completed', program: {} }, + }), }); render( ); @@ -269,7 +295,9 @@ describe('RecordingCard', () => { it('renders channel info', () => { setupMocks(); - render(); + render( + + ); expect(screen.getByText('501 • HBO')).toBeInTheDocument(); }); @@ -281,27 +309,35 @@ describe('RecordingCard', () => { it('shows description via RecordingSynopsis for non-series completed recording', () => { setupMocks(); - render(); + render( + + ); expect(screen.getByTestId('recording-synopsis')).toBeInTheDocument(); expect(screen.getByText('A test description')).toBeInTheDocument(); }); it('shows sub_title when present', () => { setupMocks(); - render(); + render( + + ); expect(screen.getByText('Pilot')).toBeInTheDocument(); }); it('shows season/episode label when getSeasonLabel returns a value', () => { setupMocks(); vi.mocked(RecordingCardUtils.getSeasonLabel).mockReturnValue('S01E02'); - render(); + render( + + ); expect(screen.getByText('S01E02')).toBeInTheDocument(); }); it('renders the poster image', () => { setupMocks(); - render(); + render( + + ); const img = screen.getByAltText('Test Show'); expect(img).toHaveAttribute('src', '/poster.jpg'); }); @@ -312,7 +348,9 @@ describe('RecordingCard', () => { describe('status badge', () => { it('shows "Completed" badge for a completed recording', () => { setupMocks(); - render(); + render( + + ); expect(screen.getByText('Completed')).toBeInTheDocument(); }); @@ -320,7 +358,10 @@ describe('RecordingCard', () => { const recording = makeRecording({ start_time: PAST, end_time: FUTURE, - custom_properties: { status: 'recording', program: { title: 'Live Show' } }, + custom_properties: { + status: 'recording', + program: { title: 'Live Show' }, + }, }); setupMocks({ recording }); render(); @@ -331,7 +372,10 @@ describe('RecordingCard', () => { const recording = makeRecording({ start_time: FUTURE, end_time: FUTURE, - custom_properties: { status: 'scheduled', program: { title: 'Future Show' } }, + custom_properties: { + status: 'scheduled', + program: { title: 'Future Show' }, + }, }); setupMocks({ recording }); render(); @@ -373,8 +417,7 @@ describe('RecordingCard', () => { // ── Series group ─────────────────────────────────────────────────────────── describe('series group', () => { - const makeSeriesRecording = () => - makeRecording({ _group_count: 3 }); + const makeSeriesRecording = () => makeRecording({ _group_count: 3 }); it('shows "Series" badge when _group_count > 1', () => { const recording = makeSeriesRecording(); @@ -394,7 +437,9 @@ describe('RecordingCard', () => { const recording = makeSeriesRecording(); setupMocks({ recording }); render(); - expect(screen.queryByTestId('recording-synopsis')).not.toBeInTheDocument(); + expect( + screen.queryByTestId('recording-synopsis') + ).not.toBeInTheDocument(); }); }); @@ -500,7 +545,9 @@ describe('RecordingCard', () => { it('does not show "Watch Live" for a completed recording', () => { setupMocks(); - render(); + render( + + ); expect(screen.queryByText('Watch Live')).not.toBeInTheDocument(); }); @@ -523,7 +570,9 @@ describe('RecordingCard', () => { it('shows "Watch" button for a completed recording', () => { setupMocks(); - render(); + render( + + ); expect(screen.getByText('Watch')).toBeInTheDocument(); }); @@ -531,7 +580,10 @@ describe('RecordingCard', () => { const recording = makeRecording({ start_time: FUTURE, end_time: FUTURE, - custom_properties: { status: 'scheduled', program: { title: 'Future' } }, + custom_properties: { + status: 'scheduled', + program: { title: 'Future' }, + }, }); setupMocks({ recording }); render(); @@ -540,7 +592,9 @@ describe('RecordingCard', () => { it('calls showVideo with vod params when Watch is clicked', () => { const { mockShowVideo } = setupMocks(); - render(); + render( + + ); fireEvent.click(screen.getByText('Watch')); expect(mockShowVideo).toHaveBeenCalledWith( '/recordings/test.ts', @@ -552,7 +606,9 @@ describe('RecordingCard', () => { it('does not call showVideo when Watch is clicked but no file url', () => { const { mockShowVideo } = setupMocks(); vi.mocked(RecordingCardUtils.getRecordingUrl).mockReturnValue(null); - render(); + render( + + ); fireEvent.click(screen.getByText('Watch')); expect(mockShowVideo).not.toHaveBeenCalled(); }); @@ -575,7 +631,9 @@ describe('RecordingCard', () => { describe('"Remove commercials" button', () => { it('shows "Remove commercials" for a completed recording without comskip', () => { setupMocks(); - render(); + render( + + ); expect(screen.getByText('Remove commercials')).toBeInTheDocument(); }); @@ -597,7 +655,10 @@ describe('RecordingCard', () => { const recording = makeRecording({ start_time: FUTURE, end_time: FUTURE, - custom_properties: { status: 'scheduled', program: { title: 'Future' } }, + custom_properties: { + status: 'scheduled', + program: { title: 'Future' }, + }, }); setupMocks({ recording }); render(); @@ -606,20 +667,31 @@ describe('RecordingCard', () => { it('calls runComSkip and shows notification on success', async () => { setupMocks(); - render(); + render( + + ); fireEvent.click(screen.getByText('Remove commercials')); await waitFor(() => { - expect(RecordingCardUtils.runComSkip).toHaveBeenCalledWith(makeRecording()); + expect(RecordingCardUtils.runComSkip).toHaveBeenCalledWith( + makeRecording() + ); expect(notifications.show).toHaveBeenCalledWith( - expect.objectContaining({ title: 'Removing commercials', color: 'blue.5' }) + expect.objectContaining({ + title: 'Removing commercials', + color: 'blue.5', + }) ); }); }); it('does not show notification when runComSkip throws', async () => { - vi.mocked(RecordingCardUtils.runComSkip).mockRejectedValue(new Error('fail')); + vi.mocked(RecordingCardUtils.runComSkip).mockRejectedValue( + new Error('fail') + ); setupMocks(); - render(); + render( + + ); fireEvent.click(screen.getByText('Remove commercials')); await waitFor(() => { expect(notifications.show).not.toHaveBeenCalled(); @@ -634,7 +706,11 @@ describe('RecordingCard', () => { makeRecording({ start_time: PAST, end_time: FUTURE, - custom_properties: { status: 'recording', program: { title: 'Live Show' }, file_url: '/f.ts' }, + custom_properties: { + status: 'recording', + program: { title: 'Live Show' }, + file_url: '/f.ts', + }, }); it('shows extend menu for in-progress recording', () => { @@ -650,9 +726,15 @@ describe('RecordingCard', () => { render(); fireEvent.click(screen.getByText('+15 minutes')); await waitFor(() => { - expect(RecordingCardUtils.extendRecordingById).toHaveBeenCalledWith('rec-1', 15); + expect(RecordingCardUtils.extendRecordingById).toHaveBeenCalledWith( + 'rec-1', + 15 + ); expect(notifications.show).toHaveBeenCalledWith( - expect.objectContaining({ title: 'Recording extended', color: 'teal' }) + expect.objectContaining({ + title: 'Recording extended', + color: 'teal', + }) ); }); }); @@ -663,7 +745,10 @@ describe('RecordingCard', () => { render(); fireEvent.click(screen.getByText('+30 minutes')); await waitFor(() => { - expect(RecordingCardUtils.extendRecordingById).toHaveBeenCalledWith('rec-1', 30); + expect(RecordingCardUtils.extendRecordingById).toHaveBeenCalledWith( + 'rec-1', + 30 + ); }); }); @@ -673,12 +758,17 @@ describe('RecordingCard', () => { render(); fireEvent.click(screen.getByText('+1 hour')); await waitFor(() => { - expect(RecordingCardUtils.extendRecordingById).toHaveBeenCalledWith('rec-1', 60); + expect(RecordingCardUtils.extendRecordingById).toHaveBeenCalledWith( + 'rec-1', + 60 + ); }); }); it('shows error notification when extendRecordingById throws', async () => { - vi.mocked(RecordingCardUtils.extendRecordingById).mockRejectedValue(new Error('Network error')); + vi.mocked(RecordingCardUtils.extendRecordingById).mockRejectedValue( + new Error('Network error') + ); const recording = makeInProgress(); setupMocks({ recording }); render(); @@ -698,7 +788,11 @@ describe('RecordingCard', () => { makeRecording({ start_time: PAST, end_time: FUTURE, - custom_properties: { status: 'recording', program: { title: 'Live Show' }, file_url: '/f.ts' }, + custom_properties: { + status: 'recording', + program: { title: 'Live Show' }, + file_url: '/f.ts', + }, }); it('shows stop modal when stop button is clicked', () => { @@ -711,7 +805,9 @@ describe('RecordingCard', () => { fireEvent.click(stopButton); expect(screen.getByTestId('modal')).toBeInTheDocument(); - expect(screen.getByTestId('modal-title')).toHaveTextContent('Stop Recording'); + expect(screen.getByTestId('modal-title')).toHaveTextContent( + 'Stop Recording' + ); }); it('closes stop modal when Go Back is clicked', () => { @@ -736,7 +832,9 @@ describe('RecordingCard', () => { fireEvent.click(screen.getAllByText('Stop Recording')[1]); await waitFor(() => { - expect(RecordingCardUtils.stopRecordingById).toHaveBeenCalledWith('rec-1'); + expect(RecordingCardUtils.stopRecordingById).toHaveBeenCalledWith( + 'rec-1' + ); expect(mockFetchRecordings).toHaveBeenCalled(); }); }); @@ -761,48 +859,71 @@ describe('RecordingCard', () => { describe('delete recording', () => { it('shows delete modal for a completed non-series recording', () => { setupMocks(); - render(); + render( + + ); - const deleteButton = screen.getByTestId('icon-square-x').closest('button'); + const deleteButton = screen + .getByTestId('icon-square-x') + .closest('button'); fireEvent.click(deleteButton); expect(screen.getByTestId('modal')).toBeInTheDocument(); - expect(screen.getByTestId('modal-title')).toHaveTextContent('Delete Recording'); + expect(screen.getByTestId('modal-title')).toHaveTextContent( + 'Delete Recording' + ); }); it('shows "Cancel Recording" title for upcoming recording delete', () => { const recording = makeRecording({ start_time: FUTURE, end_time: FUTURE, - custom_properties: { status: 'scheduled', program: { title: 'Future Show' } }, + custom_properties: { + status: 'scheduled', + program: { title: 'Future Show' }, + }, }); setupMocks({ recording }); render(); - const deleteButton = screen.getByTestId('icon-square-x').closest('button'); + const deleteButton = screen + .getByTestId('icon-square-x') + .closest('button'); fireEvent.click(deleteButton); - expect(screen.getByTestId('modal-title')).toHaveTextContent('Cancel Recording'); + expect(screen.getByTestId('modal-title')).toHaveTextContent( + 'Cancel Recording' + ); }); it('calls removeRecording when delete is confirmed', async () => { setupMocks(); - render(); + render( + + ); - const deleteButton = screen.getByTestId('icon-square-x').closest('button'); + const deleteButton = screen + .getByTestId('icon-square-x') + .closest('button'); fireEvent.click(deleteButton); fireEvent.click(screen.getByText('Delete')); await waitFor(() => { - expect(RecordingCardUtils.removeRecording).toHaveBeenCalledWith('rec-1'); + expect(RecordingCardUtils.removeRecording).toHaveBeenCalledWith( + 'rec-1' + ); }); }); it('closes delete modal after confirming', async () => { setupMocks(); - render(); + render( + + ); - const deleteButton = screen.getByTestId('icon-square-x').closest('button'); + const deleteButton = screen + .getByTestId('icon-square-x') + .closest('button'); fireEvent.click(deleteButton); fireEvent.click(screen.getByText('Delete')); @@ -813,9 +934,13 @@ describe('RecordingCard', () => { it('closes delete modal on Go Back click', () => { setupMocks(); - render(); + render( + + ); - const deleteButton = screen.getByTestId('icon-square-x').closest('button'); + const deleteButton = screen + .getByTestId('icon-square-x') + .closest('button'); fireEvent.click(deleteButton); fireEvent.click(screen.getByText('Go Back')); @@ -842,10 +967,14 @@ describe('RecordingCard', () => { setupMocks({ recording }); render(); - const deleteButton = screen.getByTestId('icon-square-x').closest('button'); + const deleteButton = screen + .getByTestId('icon-square-x') + .closest('button'); fireEvent.click(deleteButton); - expect(screen.getByTestId('modal-title')).toHaveTextContent('Cancel Series'); + expect(screen.getByTestId('modal-title')).toHaveTextContent( + 'Cancel Series' + ); }); it('calls deleteRecordingById when "Only this upcoming" is clicked', async () => { @@ -853,12 +982,16 @@ describe('RecordingCard', () => { const { mockFetchRecordings } = setupMocks({ recording }); render(); - const deleteButton = screen.getByTestId('icon-square-x').closest('button'); + const deleteButton = screen + .getByTestId('icon-square-x') + .closest('button'); fireEvent.click(deleteButton); fireEvent.click(screen.getByText('Only this upcoming')); await waitFor(() => { - expect(RecordingCardUtils.deleteRecordingById).toHaveBeenCalledWith('rec-1'); + expect(RecordingCardUtils.deleteRecordingById).toHaveBeenCalledWith( + 'rec-1' + ); expect(mockFetchRecordings).toHaveBeenCalled(); }); }); @@ -868,7 +1001,9 @@ describe('RecordingCard', () => { const { mockFetchRecordings } = setupMocks({ recording }); render(); - const deleteButton = screen.getByTestId('icon-square-x').closest('button'); + const deleteButton = screen + .getByTestId('icon-square-x') + .closest('button'); fireEvent.click(deleteButton); fireEvent.click(screen.getByText('Entire series + rule')); @@ -883,7 +1018,9 @@ describe('RecordingCard', () => { setupMocks({ recording }); render(); - const deleteButton = screen.getByTestId('icon-square-x').closest('button'); + const deleteButton = screen + .getByTestId('icon-square-x') + .closest('button'); fireEvent.click(deleteButton); fireEvent.click(screen.getByText('Only this upcoming')); @@ -918,13 +1055,18 @@ describe('RecordingCard', () => { _group_count: 3, start_time: FUTURE, end_time: FUTURE, - custom_properties: { status: 'scheduled', program: { title: 'Series' } }, + custom_properties: { + status: 'scheduled', + program: { title: 'Series' }, + }, }); const { mockFetchRecordings } = setupMocks({ recording }); mockFetchRecordings.mockRejectedValue(new Error('network')); render(); - const deleteButton = screen.getByTestId('icon-square-x').closest('button'); + const deleteButton = screen + .getByTestId('icon-square-x') + .closest('button'); fireEvent.click(deleteButton); await expect( @@ -932,4 +1074,4 @@ describe('RecordingCard', () => { ).resolves.not.toThrow(); }); }); -}); \ No newline at end of file +}); diff --git a/frontend/src/components/cards/__tests__/SeriesCard.test.jsx b/frontend/src/components/cards/__tests__/SeriesCard.test.jsx index fb95c882..49b8524f 100644 --- a/frontend/src/components/cards/__tests__/SeriesCard.test.jsx +++ b/frontend/src/components/cards/__tests__/SeriesCard.test.jsx @@ -30,7 +30,13 @@ vi.mock('@mantine/core', async () => ({ ), Stack: ({ children, gap }) =>
{children}
, Text: ({ children, size, fw, c, lineClamp, style }) => ( - + {children} ), @@ -38,6 +44,7 @@ vi.mock('@mantine/core', async () => ({ // ── lucide-react ─────────────────────────────────────────────────────────────── vi.mock('lucide-react', () => ({ + ListOrdered: () => , Calendar: () => , Play: () => , Star: () => , @@ -78,7 +85,12 @@ describe('SeriesCard', () => { }); it('renders a fallback image when poster_url is missing', () => { - render(); + render( + + ); const img = screen.getByRole('img'); expect(img).toBeInTheDocument(); }); @@ -105,7 +117,12 @@ describe('SeriesCard', () => { it('renders play icon', () => { //this only renders when logo.url is missing, but we want to test that the icon itself renders correctly - render(); + render( + + ); expect(screen.getByTestId('icon-play')).toBeInTheDocument(); }); @@ -119,27 +136,52 @@ describe('SeriesCard', () => { describe('optional fields', () => { it('does not crash when year is missing', () => { - render(); + render( + + ); expect(screen.getByTestId('series-card')).toBeInTheDocument(); }); it('does not crash when rating is missing', () => { - render(); + render( + + ); expect(screen.getByTestId('series-card')).toBeInTheDocument(); }); it('does not crash when genre is missing', () => { - render(); + render( + + ); expect(screen.getByTestId('series-card')).toBeInTheDocument(); }); it('does not crash when description is missing', () => { - render(); + render( + + ); expect(screen.getByTestId('series-card')).toBeInTheDocument(); }); it('does not crash when seasons is missing', () => { - render(); + render( + + ); expect(screen.getByTestId('series-card')).toBeInTheDocument(); }); @@ -161,4 +203,4 @@ describe('SeriesCard', () => { expect(onClick).toHaveBeenCalledWith(series); }); }); -}); \ No newline at end of file +}); diff --git a/frontend/src/components/cards/__tests__/StreamConnectionCard.test.jsx b/frontend/src/components/cards/__tests__/StreamConnectionCard.test.jsx index babcf1e6..2f006244 100644 --- a/frontend/src/components/cards/__tests__/StreamConnectionCard.test.jsx +++ b/frontend/src/components/cards/__tests__/StreamConnectionCard.test.jsx @@ -95,7 +95,12 @@ vi.mock('@mantine/core', () => ({ Center: ({ children }) =>
{children}
, Group: ({ children }) =>
{children}
, Progress: ({ value, size, color }) => ( -
+
), Select: ({ value, onChange, label, data, disabled, placeholder }) => ( { - return { - label: USER_LEVEL_LABELS[value], - value: `${value}`, - }; - })} - {...form.getInputProps('user_level')} - key={form.key('user_level')} - /> - )} + <> + ), Badge: ({ children, size, variant, color, style }) => ( - + {children} ), @@ -208,8 +213,8 @@ vi.mock('../../components/ProgramDetailModal', () => ({ ) : null, })); -vi.mock('../guideUtils', async () => { - const actual = await vi.importActual('../guideUtils'); +vi.mock('../../utils/guideUtils', async () => { + const actual = await vi.importActual('../../utils/guideUtils'); return { ...actual, fetchPrograms: vi.fn(), diff --git a/frontend/src/pages/__tests__/Settings.test.jsx b/frontend/src/pages/__tests__/Settings.test.jsx index fc0bc323..7a01b95a 100644 --- a/frontend/src/pages/__tests__/Settings.test.jsx +++ b/frontend/src/pages/__tests__/Settings.test.jsx @@ -78,6 +78,13 @@ vi.mock('../../components/forms/settings/NavOrderForm', () => ({
), })); +vi.mock('../../components/forms/settings/UserLimitsForm', () => ({ + default: ({ active }) => ( +
+ UserLimitsForm {active ? 'active' : 'inactive'} +
+ ), +})); vi.mock('../../components/ErrorBoundary', () => ({ default: ({ children }) =>
{children}
, })); diff --git a/frontend/src/pages/__tests__/guideUtils.test.js b/frontend/src/pages/__tests__/guideUtils.test.js index 14d03745..0c7f5cd4 100644 --- a/frontend/src/pages/__tests__/guideUtils.test.js +++ b/frontend/src/pages/__tests__/guideUtils.test.js @@ -1,7 +1,7 @@ import { describe, it, expect, vi, beforeEach } from 'vitest'; import dayjs from 'dayjs'; import utc from 'dayjs/plugin/utc'; -import * as guideUtils from '../guideUtils'; +import * as guideUtils from '../../utils/guideUtils'; import * as dateTimeUtils from '../../utils/dateTimeUtils'; import API from '../../api'; @@ -676,11 +676,26 @@ describe('guideUtils', () => { it('should exclude terminal status recordings', () => { const recordings = [ - { id: 1, custom_properties: { program: { id: 'p1' }, status: 'completed' } }, - { id: 2, custom_properties: { program: { id: 'p2' }, status: 'stopped' } }, - { id: 3, custom_properties: { program: { id: 'p3' }, status: 'interrupted' } }, - { id: 4, custom_properties: { program: { id: 'p4' }, status: 'failed' } }, - { id: 5, custom_properties: { program: { id: 'p5' }, status: 'recording' } }, + { + id: 1, + custom_properties: { program: { id: 'p1' }, status: 'completed' }, + }, + { + id: 2, + custom_properties: { program: { id: 'p2' }, status: 'stopped' }, + }, + { + id: 3, + custom_properties: { program: { id: 'p3' }, status: 'interrupted' }, + }, + { + id: 4, + custom_properties: { program: { id: 'p4' }, status: 'failed' }, + }, + { + id: 5, + custom_properties: { program: { id: 'p5' }, status: 'recording' }, + }, { id: 6, custom_properties: { program: { id: 'p6' } } }, ]; diff --git a/frontend/src/store/__tests__/settings.test.jsx b/frontend/src/store/__tests__/settings.test.jsx index 016fce07..9eee6f26 100644 --- a/frontend/src/store/__tests__/settings.test.jsx +++ b/frontend/src/store/__tests__/settings.test.jsx @@ -14,7 +14,7 @@ describe('useSettingsStore', () => { public_ip: '', country_code: '', country_name: '', - env_mode: 'prod', + env_mode: 'aio', }, version: { version: '', @@ -33,7 +33,7 @@ describe('useSettingsStore', () => { public_ip: '', country_code: '', country_name: '', - env_mode: 'prod', + env_mode: 'aio', }); expect(result.current.isLoading).toBe(false); expect(result.current.error).toBe(null); @@ -87,7 +87,7 @@ describe('useSettingsStore', () => { public_ip: '', country_code: '', country_name: '', - env_mode: 'prod', + env_mode: 'aio', }); }); diff --git a/frontend/src/store/settings.jsx b/frontend/src/store/settings.jsx index d7d46a3f..1b28f475 100644 --- a/frontend/src/store/settings.jsx +++ b/frontend/src/store/settings.jsx @@ -8,7 +8,7 @@ const useSettingsStore = create((set, get) => ({ public_ip: '', country_code: '', country_name: '', - env_mode: 'prod', + env_mode: 'aio', }, version: { version: '', @@ -40,7 +40,7 @@ const useSettingsStore = create((set, get) => ({ public_ip: '', country_code: '', country_name: '', - env_mode: 'prod', + env_mode: 'aio', }, }; diff --git a/frontend/src/utils/cards/RecordingCardUtils.js b/frontend/src/utils/cards/RecordingCardUtils.js index 43e3fb95..48e14393 100644 --- a/frontend/src/utils/cards/RecordingCardUtils.js +++ b/frontend/src/utils/cards/RecordingCardUtils.js @@ -1,7 +1,7 @@ import API from '../../api.js'; import useChannelsStore from '../../store/channels.jsx'; import defaultLogo from '../../images/logo.png'; -import { formatSeasonEpisode } from '../../pages/guideUtils.js'; +import { formatSeasonEpisode } from '../guideUtils.js'; export const removeRecording = (id) => { // Optimistically remove immediately from UI @@ -111,7 +111,8 @@ export const getRecordingUrl = (customProps, env_mode) => { }; export const getSeasonLabel = (season, episode, onscreen) => { - if (season != null && episode != null) return formatSeasonEpisode(season, episode); + if (season != null && episode != null) + return formatSeasonEpisode(season, episode); return onscreen || null; }; diff --git a/frontend/src/utils/forms/settings/NetworkAccessFormUtils.js b/frontend/src/utils/forms/settings/NetworkAccessFormUtils.js index 1a871571..ac863b0b 100644 --- a/frontend/src/utils/forms/settings/NetworkAccessFormUtils.js +++ b/frontend/src/utils/forms/settings/NetworkAccessFormUtils.js @@ -16,6 +16,10 @@ export const getNetworkAccessFormInitialValues = () => { export const getNetworkAccessFormValidation = () => { return Object.keys(NETWORK_ACCESS_OPTIONS).reduce((acc, key) => { acc[key] = (value) => { + if (!value || value.trim() === '') { + return null; // Empty values will be replaced with defaults on submit + } + if ( value .split(',') diff --git a/frontend/src/utils/forms/settings/__tests__/NetworkAccessFormUtils.test.js b/frontend/src/utils/forms/settings/__tests__/NetworkAccessFormUtils.test.js index 11438d00..2cc885ec 100644 --- a/frontend/src/utils/forms/settings/__tests__/NetworkAccessFormUtils.test.js +++ b/frontend/src/utils/forms/settings/__tests__/NetworkAccessFormUtils.test.js @@ -131,7 +131,8 @@ describe('NetworkAccessFormUtils', () => { NetworkAccessFormUtils.getNetworkAccessFormValidation(); const validator = validation['network-access-admin']; - expect(validator('')).toBe('Invalid CIDR range'); + // Empty values are allowed — defaults are substituted on submit + expect(validator('')).toBe(null); }); it('should return empty object when NETWORK_ACCESS_OPTIONS is empty', () => { diff --git a/frontend/src/pages/guideUtils.js b/frontend/src/utils/guideUtils.js similarity index 100% rename from frontend/src/pages/guideUtils.js rename to frontend/src/utils/guideUtils.js diff --git a/scripts/wait_for_redis.py b/scripts/wait_for_redis.py index e1814411..9c2d6eaf 100644 --- a/scripts/wait_for_redis.py +++ b/scripts/wait_for_redis.py @@ -34,10 +34,49 @@ def _flush_non_celery_keys(client): logger.info(f"Modular mode: selectively cleared {deleted} non-Celery Redis key(s)") +def _build_ssl_kwargs(): + """Build Redis SSL kwargs from environment variables. + + Validates that any configured certificate file paths exist on disk + and logs a summary of the active TLS configuration. + """ + ssl_kwargs = {} + if os.environ.get("REDIS_SSL", "false").lower() == "true": + import ssl as _ssl + ssl_kwargs["ssl"] = True + ssl_kwargs["ssl_cert_reqs"] = ( + _ssl.CERT_REQUIRED if os.environ.get("REDIS_SSL_VERIFY", "true").lower() == "true" + else _ssl.CERT_NONE + ) + cert_paths = { + "REDIS_SSL_CA_CERT": os.environ.get("REDIS_SSL_CA_CERT", ""), + "REDIS_SSL_CERT": os.environ.get("REDIS_SSL_CERT", ""), + "REDIS_SSL_KEY": os.environ.get("REDIS_SSL_KEY", ""), + } + for env_var, path in cert_paths.items(): + if path and not os.path.isfile(path): + logger.error( + f"Redis TLS: {env_var}={path!r} — file not found. " + f"Check that the certificate file exists and the volume is mounted correctly." + ) + sys.exit(1) + + if cert_paths["REDIS_SSL_CA_CERT"]: + ssl_kwargs["ssl_ca_certs"] = cert_paths["REDIS_SSL_CA_CERT"] + if cert_paths["REDIS_SSL_CERT"]: + ssl_kwargs["ssl_certfile"] = cert_paths["REDIS_SSL_CERT"] + if cert_paths["REDIS_SSL_KEY"]: + ssl_kwargs["ssl_keyfile"] = cert_paths["REDIS_SSL_KEY"] + + logger.info("Redis TLS: enabled") + return ssl_kwargs + + def wait_for_redis(host='localhost', port=6379, db=0, password='', username='', max_retries=30, retry_interval=2): """Wait for Redis to become available""" redis_client = None retry_count = 0 + ssl_kwargs = _build_ssl_kwargs() logger.info(f"Waiting for Redis at {host}:{port}/{db}...") @@ -50,7 +89,8 @@ def wait_for_redis(host='localhost', port=6379, db=0, password='', username='', password=password if password else None, username=username if username else None, socket_timeout=2, - socket_connect_timeout=2 + socket_connect_timeout=2, + **ssl_kwargs ) redis_client.ping() # Clear stale state on startup. In AIO mode, every service restarts @@ -66,14 +106,16 @@ def wait_for_redis(host='localhost', port=6379, db=0, password='', username='', return True except (redis.exceptions.ConnectionError, redis.exceptions.TimeoutError) as e: retry_count += 1 + _tls_hint = " (TLS is enabled — verify certificate paths and that Redis is configured for TLS)" if ssl_kwargs else "" if retry_count >= max_retries: - logger.error(f"❌ Failed to connect to Redis after {max_retries} attempts: {e}") + logger.error(f"❌ Failed to connect to Redis after {max_retries} attempts: {e}{_tls_hint}") return False logger.info(f"⏳ Redis not available yet, retrying in {retry_interval}s... ({retry_count}/{max_retries})") time.sleep(retry_interval) except Exception as e: - logger.error(f"❌ Unexpected error connecting to Redis: {e}") + _tls_hint = " (TLS is enabled — verify certificate paths and that Redis is configured for TLS)" if ssl_kwargs else "" + logger.error(f"❌ Unexpected error connecting to Redis: {e}{_tls_hint}") return False return False