diff --git a/CHANGELOG.md b/CHANGELOG.md index a8bd0551..414a6850 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,9 +9,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Configurable sidebar navigation ordering and visibility — Thanks [@jcasimir](https://github.com/jcasimir) + - Sidebar nav items can be reordered via drag-and-drop in Settings → UI Settings → Navigation. + - Individual nav items can be hidden from the sidebar using the eye toggle. Hiding an item preserves its position in the order. + - A "Reset to Default" button restores the role-appropriate default order and clears all hidden items. + - Order and visibility are saved per-user with optimistic updates and automatic rollback on failure. Changes appear in the sidebar immediately without a page reload. + - Admin users see a grouped navigation: flat items (`Channels`, `VODs`, `M3U & EPG Manager`, `TV Guide`, `DVR`, `Stats`, `Plugins`) plus collapsible `Integrations` (Connections, Logs) and `System` (Users, Logo Manager, Settings) groups. The `System` group cannot be hidden. + - Non-admin users see `Channels`, `TV Guide`, and `Settings`, with the `Settings` item not hideable. +- Unit tests for `NotificationCenter`, `NotificationCenterUtils`, and `M3URefreshNotification` components. — Thanks [@nick4810](https://github.com/nick4810) +- Unit tests for DVR port resolution (`build_dvr_candidates`) and selective Redis flush behavior in modular mode. — Thanks [@CodeBormen](https://github.com/CodeBormen) - Floating video player now displays the channel, stream, or VOD title in the player header bar. Title is passed through from all preview entry points: channel table, stream table, stream connection card, guide, DVR, recording cards, recording details modal, VOD modal, and series modal. - New Client Buffer proxy setting: new clients joining an active channel are now positioned a configurable number of seconds behind live rather than a fixed chunk count. The start position is determined by wall-clock chunk receive time (stored as a Redis sorted set alongside the buffer), so the buffer depth is consistent in seconds regardless of stream bitrate. Setting the value to `0` starts clients at live with no buffer. Defaults to 5 seconds. Existing chunk-count gating for the first client connecting to a channel is unchanged. The setting is exposed in Settings → Proxy as "New Client Buffer (seconds)". - Channel table filter for channels that have stale streams: A new "Has Stale Streams" filter option in the channel table header menu highlights and filters channels containing at least one stale stream. Channels with stale streams are visually distinguished with an orange tint. The filter is mutually exclusive with "Only Empty Channels". - Thanks [@JCBird1012](https://github.com/JCBird1012) +- Sort icons added to the Group and EPG column headers in the Channels table, and to the Group column header in the Streams table. Clicking a sort icon cycles through ascending/descending/unsorted states. EPG sorting required a backend change (`epg_data__name` added to `ChannelViewSet.ordering_fields`); Group sorting was already supported by the API in both tables. (Closes #854) — Thanks [@CodeBormen](https://github.com/CodeBormen) - DVR enhancements — Thanks [@CodeBormen](https://github.com/CodeBormen) - **Stop Recording**: A new Stop button (distinct from Cancel) cleanly ends an in-progress recording early and keeps the partial file available for playback. The API returns immediately; stream teardown and task revocation happen in a background thread to prevent 504 timeouts. When multiple recordings run simultaneously, stopping one only terminates that recording's proxy client by ID, leaving all others unaffected. (Closes #454) - **Extend Recording**: In-progress recordings can be extended by 15, 30, or 60 minutes without interrupting the stream. @@ -25,22 +35,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Channels page default splitter ratio changed from 50/50 to 60/40 (channels/streams) so all channel action buttons are visible without scrolling on 1080p displays. +- Frontend component refactoring and cleanup — Thanks [@nick4810](https://github.com/nick4810) + - `FloatingVideo`, `SeriesModal`, `VODModal`, `SystemEvents`, `M3URefreshNotification`, and `NotificationCenter` significantly reduced in size by separating business logic into dedicated utility modules under `utils/components/` (`FloatingVideoUtils.js`, `SeriesModalUtils.js`, `VODModalUtils.js`, `NotificationCenterUtils.js`). + - `FloatingVideo` resize handle elements extracted into a standalone `ResizeHandles` sub-component. + - `YouTubeTrailerModal` extracted into a standalone component (`components/modals/YouTubeTrailerModal.jsx`). + - `NotificationCenter` and `Sidebar` updated from Mantine dot-notation sub-components (`Popover.Target`, `Popover.Dropdown`, `ScrollArea.Autosize`, `AppShell.Navbar`) to Mantine v7 named imports (`PopoverTarget`, `PopoverDropdown`, `ScrollAreaAutosize`, `AppShellNavbar`). + - `M3URefreshNotification` now uses the centralized `showNotification()` utility (from `notificationUtils.js`) instead of calling `notifications.show()` directly, bringing it in line with the rest of the app. State updates also converted to functional updater form (`prev => ...`) to eliminate potential stale-closure bugs. + - `SystemEvents` now imports `format` from `dateTimeUtils` for consistent date/time formatting. + - Removed a dead `onLogout` handler in `Sidebar` that called `logout()` and `window.location.reload()` but was never wired to any UI element. - EPG output when no `days` parameter is specified now excludes already-ended programs instead of returning all historical data. ### Fixed +- **Security**: Any authenticated user could self-escalate privileges by sending `user_level`, `is_staff`, or `is_superuser` in a `PATCH /api/accounts/users/me/` request. The `/me/` endpoint now enforces an allowlist (`custom_properties`, `first_name`, `last_name`, `email`, `password`); any other fields return HTTP 400. Privilege-sensitive fields can only be changed via the admin-only `PATCH /api/accounts/users/{id}/` endpoint. +- Double error notification when saving user preferences: `API.updateMe` was catching errors internally and displaying a notification before re-throwing, causing callers to display a second notification for the same failure. +- Navigation preference saves from concurrent sessions could overwrite each other due to a double-merge race: the frontend was pre-merging `custom_properties` before sending, then the backend merged again against the DB value, causing the second session's write to silently drop keys set by the first. The frontend now sends only the delta; the backend merges authoritatively against the stored value. +- Stale nav item IDs (e.g. from a previous nav structure) are now scrubbed from `navOrder` and `hiddenNav` on the next preference save, preventing unbounded growth of the `custom_properties` JSON field. +- Version update notification persisting after upgrading to the notified version (e.g. "v0.20.2 available" shown while already running v0.20.2). Root cause: `check_for_version_update.delay()` was called from `AppConfig.ready()`, which fires inside Celery prefork pool subprocesses before the broker connection is established, causing the dispatch to fail silently with no log output. Fixed by moving the startup dispatch to the `worker_ready` signal in `celery.py` (consistent with the existing `recover_recordings_on_startup` pattern), and deleting the stale `version-{current_version}` notification at the top of the production check path so it is cleared even when GitHub is unreachable. A WebSocket update is sent immediately on deletion so the frontend badge clears without waiting for the API response. +- VOD orphan cleanup crashing with a `ForeignKeyViolation` (`IntegrityError`) when a concurrent refresh task created a new `M3UMovieRelation` or `M3USeriesRelation` for a movie/series between the orphan-detection query and the `DELETE` SQL. Both `orphaned_movies.delete()` and `orphaned_series.delete()` are now wrapped in `try/except IntegrityError`; affected records are skipped with a warning and will be cleaned up on the next scheduled run. +- XC stream refresh crashing with a `null value in column "name"` database error when a provider returns streams with a null or empty name. Affected streams are now assigned a generated fallback name in the format ` - ` so the refresh completes successfully and the stream remains accessible. A warning is logged for each affected stream. +- 504 Gateway Timeout when saving M3U group settings on slower hardware (e.g. Synology NAS). Replaced per-row `update_or_create()` loops with `bulk_create(update_conflicts=True)` wrapped in `transaction.atomic()` for both `ChannelGroupM3UAccount` and `M3UVODCategoryRelation`, reducing hundreds of individual DB round-trips to a single query per model. (Fixes #745) — Thanks [@nickgerrer](https://github.com/nickgerrer) +- Improved frontend table stability during M3U imports: Fixed incorrect default `state` initialization (`[]` → `{}`) in `CustomTable` to match TanStack Table v8's expected state object shape. Added `autoResetPageIndex: false` and `autoResetExpanded: false` to prevent TanStack Table from issuing internal state resets on data updates. Memoized `processedData` in `M3UsTable` to avoid redundant sort/filter recomputation on re-renders. - Thanks [@marcinolek](https://github.com/marcinolek) +- `debian_install.sh` hardened for non-UTF8 environments (common in minimal LXC containers) - Thanks [@marcinolek](https://github.com/marcinolek) + - Added `setup_locales` step that installs the `locales` package, enables `en_US.UTF-8`, regenerates locales, and exports `LANG`/`LC_ALL` before any other work runs, preventing PostgreSQL from defaulting to `SQL_ASCII` encoding. + - PostgreSQL database creation now explicitly passes `-E UTF8` to `createdb`. + - `PATH` in the Celery worker, Celery Beat, and Daphne systemd service files extended to include `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin`, fixing failures where background tasks could not locate `ffmpeg` or `ffprobe`. +- `is_adult` field parsing now guards against invalid values (e.g. the string `"None"`) that providers may send instead of a valid integer, preventing a `ValueError` crash during M3U/XC stream refresh. A new `parse_is_adult()` helper wraps the cast in a `try/except`, returning `False` for anything that cannot be interpreted as `1`. (Fixes #1061) — Thanks [@JCBird1012](https://github.com/JCBird1012) +- M3U EXTINF attribute parsing for values containing `=` or `==` (e.g. base64-padded `tvg-logo` URLs, catchup tokens with query strings). The previous regex used `[^\s]+` for the key pattern, allowing `=` signs inside a quoted value to be greedily absorbed into the next attribute's key name, causing that attribute and all subsequent ones on the line to be silently dropped. Changed to `[^\s=]+` so the key match always stops at the first `=`. (Fixes #1055) - Thanks [@JCBird1012](https://github.com/JCBird1012) - Celery worker memory leak during M3U/XC refresh causing 20–80 MB growth per cycle with no reclamation (Fixes #1012, #1053) - Thanks [@CodeBormen](https://github.com/CodeBormen) - Restructured `refresh_single_m3u_account()` with a `try/finally` that guarantees `del` of large data structures runs before Celery's `gc.collect()`, and lock release on all exit paths (success, exception, early return) - Re-enabled batch data cleanup in `process_m3u_batch_direct()` (was commented out) - Added `CELERY_WORKER_MAX_MEMORY_PER_CHILD = 512 MB` as a safety net against pymalloc arena fragmentation - EPG output was filtering programs using `start_time__gte=now` when the `days` parameter was specified, which caused currently-airing programs (started before the request time but not yet ended) to be omitted from the XML output. This produced a gap in clients' guides immediately after an EPG refresh, lasting until the next program started. Fixed by changing the filter to `end_time__gte=now` so any program that has not yet finished is included. - TS proxy connection slot leaks and TOCTOU races in stream initialization (Fixes #947) - Thanks [@CodeBormen](https://github.com/CodeBormen) - - **TOCTOU race in slot reservation**: `get_stream()` previously used a `GET`→check→`INCR` sequence, allowing concurrent requests to both read the same count below the limit and both reserve a slot, silently exceeding `max_streams`. Replaced with an atomic `INCR`-first pattern: increment unconditionally, check the result, roll back with `DECR` if over capacity. + - **TOCTOU race in slot reservation**: `get_stream()` previously used a `GET`→check→`INCR` sequence, allowing concurrent requests to both read the same count below the limit and both reserve a slot, silently exceeding `max_streams`. Replaced with an atomic `INCR`-first pattern: increment unconditionally, check the result, roll back with `DECR` if over capacity. — Thanks [@patchy8736](https://github.com/patchy8736) - **Leak on URL generation failure**: `generate_stream_url()` called `get_stream()` (which `INCR`s the counter) but had no cleanup path if subsequent DB lookups or URL construction failed. The post-`get_stream()` block is now wrapped in a `try/except` that calls `release_stream()` on any error. - **Leak on retry-loop timeout**: the retry loop in `stream_ts()` called a bare `get_stream()` on the first failure to classify the error reason. If a slot was available, this `INCR`'d the counter and set Redis keys that were never released when the loop timed out. A `release_stream()` call is now issued before returning 503. - **Leak on `initialize_channel()` failure**: when `initialize_channel()` returned `False`, the connection slot allocated by the preceding `get_stream()` was never released. A `connection_allocated` flag now tracks whether this request performed the `INCR` (fresh initialization vs. joining an existing channel), and `release_stream()` is called guarded by that flag to prevent incorrect decrements when attaching to an already-running channel. - **Safety net for unexpected exceptions**: the outer `except` in `stream_ts()` now checks `connection_allocated` and calls `release_stream()` as a last-resort cleanup for any unhandled exception that escapes before the channel is handed off to the stream lifecycle. - - **`release_stream()` now returns `bool`** and adds a metadata-hash fallback: if the primary `channel_stream` / `stream_profile` Redis keys have already been cleaned up by the proxy, it recovers `stream_id` and `profile_id` from the channel's metadata hash and clears those fields atomically to prevent duplicate `DECR`s on repeated calls. + - **`release_stream()` now returns `bool`** and adds a metadata-hash fallback: if the primary `channel_stream` / `stream_profile` Redis keys have already been cleaned up by the proxy, it recovers `stream_id` and `profile_id` from the channel's metadata hash and clears those fields atomically to prevent duplicate `DECR`s on repeated calls. — Thanks [@patchy8736](https://github.com/patchy8736) - **`update_stream_profile()` uses a Redis pipeline** for the old-profile DECR + key update + new-profile INCR sequence, preventing counter drift if the process crashes between operations. - **`stream_generator._cleanup()`** now falls back to `Stream.objects.get()` when the channel UUID resolves to a preview flow rather than a normal channel, rather than silently skipping the slot release. - **VOD `cleanup_persistent_connection()`** fallback DECR is now conditional: it only decrements the profile counter when the connection tracking key had already expired by TTL (i.e., `remove_connection()` would have skipped the DECR), preventing double-decrements when the key is still present. @@ -73,6 +107,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Duplicate series rule evaluation race**: Creating a series rule fired `evaluate_series_rules.delay()` in the API view while the frontend immediately called the synchronous evaluate endpoint, racing to create duplicate recordings for the same program. Removed the redundant async call from the API; the frontend's explicit evaluate call is now the sole evaluation path. - **Recording card S/E badge overlap**: Season/episode badges were overlapping and metadata was hidden on the recording card. - **Orphaned recording fallback in series modal**: When a series rule no longer exists, the recurring rule modal now shows a "Delete Recording" button for the orphaned recording instead of failing silently. +- Modular mode deployment hardening — Thanks [@CodeBormen](https://github.com/CodeBormen) + - **Postgres version check with restricted DB users**: The version check was connecting to the hardcoded `postgres` database, which fails when the configured user lacks access to it. Changed to use `$POSTGRES_DB` so the check works with least-privilege database users. (Fixes #1045) + - **DVR recording broken in modular mode**: Internal TS stream URL candidates hardcoded port `9191`, so recordings failed when `DISPATCHARR_PORT` was set to any other value. URL construction now reads `DISPATCHARR_PORT` from the environment via the new `build_dvr_candidates()` helper. `DISPATCHARR_PORT` is also now explicitly passed to the Celery container in `docker-compose.yml`. + - **Selective Redis flush in modular mode**: `wait_for_redis.py` now performs a targeted flush in modular mode — clearing stale stream locks, proxy metadata, and server-state keys — while preserving Celery broker and result-backend keys. Previously either a full `flushdb()` (which wiped Celery queues) or no flush at all was performed. + - **Redis wait stripping environment variables**: The modular-mode Redis readiness check ran as a uWSGI `exec-pre` hook, which executes under `su -` and strips Docker environment variables, making `DISPATCHARR_ENV` and `REDIS_HOST` unavailable. Moved to the container entrypoint so all env vars are present. + - **Stale environment variables after container restart**: `/etc/profile.d/dispatcharr.sh` was only written on the first container run; restarts with changed env vars (e.g. a rotated `POSTGRES_PASSWORD`) retained stale values. The file is now truncated and rewritten on every startup. `/etc/environment` entries are likewise updated rather than skipped when a key already exists. All exported values are now quoted to prevent breakage from special characters. + - **Celery entrypoint startup timeouts**: The JWT key wait and migration wait loops had no timeout, leaving the Celery worker hanging indefinitely if the web container was stuck. Each loop now times out (120 s for JWT, 300 s for migrations) and exits with a clear diagnostic message. The migration readiness check is also replaced from a fragile `showmigrations | grep` pattern to `migrate --check`, which exits cleanly on both unapplied migrations and connection errors. + - **Service startup ordering**: `depends_on` entries for `db` and `redis` in `docker-compose.yml` upgraded from plain name-link ordering to `condition: service_healthy`, ensuring containers wait for actual readiness signals before starting. + - **`host.docker.internal` resolution on Linux**: Added `extra_hosts: host.docker.internal:host-gateway` to the web service in `docker-compose.yml` so Linux hosts resolve `host.docker.internal` the same way Docker Desktop does on macOS/Windows. ## [0.20.2] - 2026-03-03 diff --git a/apps/accounts/api_views.py b/apps/accounts/api_views.py index cf2d9225..76892929 100644 --- a/apps/accounts/api_views.py +++ b/apps/accounts/api_views.py @@ -278,11 +278,24 @@ class UserViewSet(viewsets.ModelViewSet): return super().destroy(request, *args, **kwargs) @extend_schema( - description="Get active user information", + description="Get or update active user information. PATCH updates custom_properties with merge semantics.", + methods=["GET", "PATCH"], ) - @action(detail=False, methods=["get"], url_path="me") + @action(detail=False, methods=["get", "patch"], url_path="me") def me(self, request): user = request.user + if request.method == "PATCH": + ALLOWED_FIELDS = {"custom_properties", "first_name", "last_name", "email", "password"} + disallowed = set(request.data.keys()) - ALLOWED_FIELDS + if disallowed: + return Response( + {"detail": f"Fields not allowed for self-update: {', '.join(disallowed)}"}, + status=400, + ) + serializer = UserSerializer(user, data=request.data, partial=True) + serializer.is_valid(raise_exception=True) + serializer.save() + return Response(serializer.data) serializer = UserSerializer(user) return Response(serializer.data) diff --git a/apps/accounts/serializers.py b/apps/accounts/serializers.py index 05e11ebb..c90ffef4 100644 --- a/apps/accounts/serializers.py +++ b/apps/accounts/serializers.py @@ -1,9 +1,32 @@ +import json + from rest_framework import serializers from django.contrib.auth.models import Group, Permission from .models import User from apps.channels.models import ChannelProfile +# Valid navigation item IDs for validation +VALID_NAV_ITEM_IDS = { + 'channels', 'vods', 'sources', 'guide', 'dvr', + 'stats', 'plugins', 'integrations', 'system', 'settings' +} +MAX_CUSTOM_PROPS_SIZE = 10240 # 10KB limit + + +def validate_nav_array(value, field_name): + """Validate that a value is an array of valid nav item ID strings.""" + if not isinstance(value, list): + raise serializers.ValidationError(f"{field_name} must be an array") + if len(value) > 50: + raise serializers.ValidationError(f"{field_name} exceeds maximum length of 50 items") + for item in value: + if not isinstance(item, str): + raise serializers.ValidationError(f"{field_name} items must be strings") + if item not in VALID_NAV_ITEM_IDS: + raise serializers.ValidationError(f"'{item}' is not a valid navigation item ID") + + # 🔹 Fix for Permission serialization class PermissionSerializer(serializers.ModelSerializer): class Meta: @@ -24,7 +47,7 @@ class GroupSerializer(serializers.ModelSerializer): # 🔹 Fix for User serialization class UserSerializer(serializers.ModelSerializer): - password = serializers.CharField(write_only=True) + password = serializers.CharField(write_only=True, required=False) channel_profiles = serializers.PrimaryKeyRelatedField( queryset=ChannelProfile.objects.all(), many=True, required=False ) @@ -50,6 +73,32 @@ class UserSerializer(serializers.ModelSerializer): "last_name", ] + def validate_custom_properties(self, value): + """Validate custom_properties structure and size.""" + if value is None: + return {} + if not isinstance(value, dict): + raise serializers.ValidationError("custom_properties must be a dictionary") + + # Size limit check + try: + if len(json.dumps(value)) > MAX_CUSTOM_PROPS_SIZE: + raise serializers.ValidationError( + f"custom_properties exceeds maximum size of {MAX_CUSTOM_PROPS_SIZE} bytes" + ) + except (TypeError, ValueError): + raise serializers.ValidationError("custom_properties contains non-serializable data") + + # Validate navOrder if present + if 'navOrder' in value: + validate_nav_array(value['navOrder'], 'navOrder') + + # Validate hiddenNav if present + if 'hiddenNav' in value: + validate_nav_array(value['hiddenNav'], 'hiddenNav') + + return value + def create(self, validated_data): channel_profiles = validated_data.pop("channel_profiles", []) @@ -65,6 +114,22 @@ class UserSerializer(serializers.ModelSerializer): password = validated_data.pop("password", None) channel_profiles = validated_data.pop("channel_profiles", None) + # Merge custom_properties instead of replacing (prevents data loss) + # Strip null values — sending null for a key omits it rather than overwriting with null + custom_properties = validated_data.pop("custom_properties", None) + if custom_properties is not None: + existing = instance.custom_properties or {} + cleaned = {k: v for k, v in custom_properties.items() if v is not None} + merged = {**existing, **cleaned} + # Scrub stale nav IDs so the DB self-heals on next save + for nav_field in ('navOrder', 'hiddenNav'): + if nav_field in merged and isinstance(merged[nav_field], list): + merged[nav_field] = [ + item for item in merged[nav_field] + if item in VALID_NAV_ITEM_IDS + ] + instance.custom_properties = merged + for attr, value in validated_data.items(): setattr(instance, attr, value) diff --git a/apps/channels/api_views.py b/apps/channels/api_views.py index 4460c277..550f6302 100644 --- a/apps/channels/api_views.py +++ b/apps/channels/api_views.py @@ -474,7 +474,7 @@ class ChannelViewSet(viewsets.ModelViewSet): filter_backends = [DjangoFilterBackend, SearchFilter, OrderingFilter] filterset_class = ChannelFilter search_fields = ["name", "channel_group__name"] - ordering_fields = ["channel_number", "name", "channel_group__name"] + ordering_fields = ["channel_number", "name", "channel_group__name", "epg_data__name"] ordering = ["-channel_number"] def create(self, request, *args, **kwargs): diff --git a/apps/channels/tasks.py b/apps/channels/tasks.py index 29fe5fc2..06cd45cf 100755 --- a/apps/channels/tasks.py +++ b/apps/channels/tasks.py @@ -1642,6 +1642,33 @@ def _build_output_paths(channel, program, start_time, end_time): return final_path, temp_ts_path, os.path.basename(final_path) +def build_dvr_candidates(): + """Build ordered list of candidate base URLs for DVR TS streaming. + + Reads environment variables to determine which URLs to try: + - DISPATCHARR_INTERNAL_TS_BASE_URL: explicit override (first priority) + - DISPATCHARR_PORT: the external port (default 9191) + - DISPATCHARR_ENV/DISPATCHARR_DEBUG/REDIS_HOST: dev-mode detection + - DISPATCHARR_INTERNAL_API_BASE: override for the docker service URL + """ + explicit = os.environ.get('DISPATCHARR_INTERNAL_TS_BASE_URL') + dispatcharr_port = os.environ.get('DISPATCHARR_PORT', '9191') + is_dev = (os.environ.get('DISPATCHARR_ENV', '').lower() == 'dev') or \ + (os.environ.get('DISPATCHARR_DEBUG', '').lower() == 'true') or \ + (os.environ.get('REDIS_HOST', 'redis') in ('localhost', '127.0.0.1')) + candidates = [] + if explicit: + candidates.append(explicit) + if is_dev: + # Debug container typically exposes API on 5656 (uwsgi internal port) + candidates.extend(['http://127.0.0.1:5656', f'http://127.0.0.1:{dispatcharr_port}']) + # Docker service name fallback — use DISPATCHARR_PORT so modular mode works with custom ports + candidates.append(os.environ.get('DISPATCHARR_INTERNAL_API_BASE', f'http://web:{dispatcharr_port}')) + # Last-resort localhost ports + candidates.extend(['http://localhost:5656', f'http://localhost:{dispatcharr_port}']) + return candidates + + @shared_task def run_recording(recording_id, channel_id, start_time_str, end_time_str): """ @@ -1820,22 +1847,7 @@ def run_recording(recording_id, channel_id, start_time_str, end_time_str): from requests.exceptions import ReadTimeout, ConnectionError as ReqConnectionError, ChunkedEncodingError - # Determine internal base URL(s) for TS streaming - # Prefer explicit override, then try common ports for debug and docker - explicit = os.environ.get('DISPATCHARR_INTERNAL_TS_BASE_URL') - is_dev = (os.environ.get('DISPATCHARR_ENV', '').lower() == 'dev') or \ - (os.environ.get('DISPATCHARR_DEBUG', '').lower() == 'true') or \ - (os.environ.get('REDIS_HOST', 'redis') in ('localhost', '127.0.0.1')) - candidates = [] - if explicit: - candidates.append(explicit) - if is_dev: - # Debug container typically exposes API on 5656 - candidates.extend(['http://127.0.0.1:5656', 'http://127.0.0.1:9191']) - # Docker service name fallback - candidates.append(os.environ.get('DISPATCHARR_INTERNAL_API_BASE', 'http://web:9191')) - # Last-resort localhost ports - candidates.extend(['http://localhost:5656', 'http://localhost:9191']) + candidates = build_dvr_candidates() chosen_base = None last_error = None diff --git a/apps/channels/tests/test_dvr_port_resolution.py b/apps/channels/tests/test_dvr_port_resolution.py new file mode 100644 index 00000000..c7373959 --- /dev/null +++ b/apps/channels/tests/test_dvr_port_resolution.py @@ -0,0 +1,59 @@ +import os +from django.test import SimpleTestCase +from unittest.mock import patch + +from apps.channels.tasks import build_dvr_candidates + + +class DVRPortResolutionTests(SimpleTestCase): + """ + Tests that DVR recording candidate URLs respect the DISPATCHARR_PORT + environment variable instead of hardcoding port 9191. + """ + + @patch.dict(os.environ, {'REDIS_HOST': 'redis'}, clear=True) + def test_default_port_uses_9191(self): + """Without DISPATCHARR_PORT set, candidates default to 9191.""" + candidates = build_dvr_candidates() + self.assertIn('http://web:9191', candidates) + self.assertIn('http://localhost:9191', candidates) + + @patch.dict(os.environ, {'DISPATCHARR_PORT': '8080', 'REDIS_HOST': 'redis'}, clear=True) + def test_custom_port_reflected_in_candidates(self): + """DISPATCHARR_PORT=8080 replaces all hardcoded 9191 references.""" + candidates = build_dvr_candidates() + self.assertIn('http://web:8080', candidates) + self.assertIn('http://localhost:8080', candidates) + self.assertNotIn('http://web:9191', candidates) + self.assertNotIn('http://localhost:9191', candidates) + + @patch.dict(os.environ, { + 'DISPATCHARR_PORT': '7777', + 'DISPATCHARR_ENV': 'dev', + 'REDIS_HOST': 'redis', + }, clear=True) + def test_dev_mode_includes_5656_and_custom_port(self): + """Dev mode includes both uwsgi internal port (5656) and custom port.""" + candidates = build_dvr_candidates() + self.assertIn('http://127.0.0.1:5656', candidates) + self.assertIn('http://127.0.0.1:7777', candidates) + + @patch.dict(os.environ, { + 'DISPATCHARR_INTERNAL_TS_BASE_URL': 'http://custom:1234', + 'REDIS_HOST': 'redis', + }, clear=True) + def test_explicit_override_is_first(self): + """DISPATCHARR_INTERNAL_TS_BASE_URL should be the first candidate.""" + candidates = build_dvr_candidates() + self.assertEqual(candidates[0], 'http://custom:1234') + + @patch.dict(os.environ, { + 'DISPATCHARR_PORT': '3000', + 'DISPATCHARR_INTERNAL_API_BASE': 'http://myhost:4000', + 'REDIS_HOST': 'redis', + }, clear=True) + def test_internal_api_base_overrides_web_fallback(self): + """DISPATCHARR_INTERNAL_API_BASE replaces the http://web:{port} default.""" + candidates = build_dvr_candidates() + self.assertIn('http://myhost:4000', candidates) + self.assertNotIn('http://web:3000', candidates) diff --git a/apps/m3u/api_views.py b/apps/m3u/api_views.py index 26e182d9..34f3cd77 100644 --- a/apps/m3u/api_views.py +++ b/apps/m3u/api_views.py @@ -8,6 +8,7 @@ from apps.accounts.permissions import ( ) from drf_spectacular.utils import extend_schema, OpenApiParameter from drf_spectacular.types import OpenApiTypes +from django.db import transaction from django.shortcuts import get_object_or_404 from django.http import JsonResponse from django.core.cache import cache @@ -264,38 +265,50 @@ class M3UAccountViewSet(viewsets.ModelViewSet): category_settings = request.data.get("category_settings", []) try: - for setting in group_settings: - group_id = setting.get("channel_group") - enabled = setting.get("enabled", True) - auto_sync = setting.get("auto_channel_sync", False) - sync_start = setting.get("auto_sync_channel_start") - custom_properties = setting.get("custom_properties", {}) - - if group_id: - ChannelGroupM3UAccount.objects.update_or_create( - channel_group_id=group_id, + with transaction.atomic(): + group_objects = [ + ChannelGroupM3UAccount( + channel_group_id=setting["channel_group"], m3u_account=account, - defaults={ - "enabled": enabled, - "auto_channel_sync": auto_sync, - "auto_sync_channel_start": sync_start, - "custom_properties": custom_properties, - }, + enabled=setting.get("enabled", True), + auto_channel_sync=setting.get("auto_channel_sync", False), + auto_sync_channel_start=setting.get("auto_sync_channel_start"), + custom_properties=setting.get("custom_properties", {}), + ) + for setting in group_settings + if setting.get("channel_group") + ] + + if group_objects: + ChannelGroupM3UAccount.objects.bulk_create( + group_objects, + update_conflicts=True, + unique_fields=["channel_group", "m3u_account"], + update_fields=[ + "enabled", + "auto_channel_sync", + "auto_sync_channel_start", + "custom_properties", + ], ) - for setting in category_settings: - category_id = setting.get("id") - enabled = setting.get("enabled", True) - custom_properties = setting.get("custom_properties", {}) - - if category_id: - M3UVODCategoryRelation.objects.update_or_create( - category_id=category_id, + category_objects = [ + M3UVODCategoryRelation( + category_id=setting["id"], m3u_account=account, - defaults={ - "enabled": enabled, - "custom_properties": custom_properties, - }, + enabled=setting.get("enabled", True), + custom_properties=setting.get("custom_properties", {}), + ) + for setting in category_settings + if setting.get("id") + ] + + if category_objects: + M3UVODCategoryRelation.objects.bulk_create( + category_objects, + update_conflicts=True, + unique_fields=["m3u_account", "category"], + update_fields=["enabled", "custom_properties"], ) return Response({"message": "Group settings updated successfully"}) diff --git a/apps/m3u/tasks.py b/apps/m3u/tasks.py index c0c90c57..5259cefc 100644 --- a/apps/m3u/tasks.py +++ b/apps/m3u/tasks.py @@ -462,6 +462,13 @@ def get_case_insensitive_attr(attributes, key, default=""): return default +def parse_is_adult(value): + try: + return int(value) == 1 + except (TypeError, ValueError): + return False + + def parse_extinf_line(line: str) -> dict: """ Parse an EXTINF line from an M3U file. @@ -482,8 +489,10 @@ def parse_extinf_line(line: str) -> dict: attrs = {} last_attr_end = 0 - # Use a single regex that handles both quote types - for match in re.finditer(r'([^\s]+)=(["\'])([^\2]*?)\2', content): + # Use a single regex that handles both quote types. + # Keys must stop at '=' so values like base64-padded URLs ending with '==' + # don't get folded into the preceding attribute name. + for match in re.finditer(r'([^\s=]+)\s*=\s*(["\'])(.*?)\2', content): key = match.group(1) value = match.group(3) attrs[key] = value @@ -749,6 +758,14 @@ def collect_xc_streams(account_id, enabled_groups): # Filter streams based on enabled categories filtered_count = 0 for stream in all_xc_streams: + # Fall back to a generated name if the provider returns null/empty + stream_name = stream.get("name") or f"{account.name} - {stream.get('stream_id', 'Unknown')}" + if not stream.get("name"): + logger.warning( + f"XC stream has null/empty name; using generated name '{stream_name}' " + f"(stream_id={stream.get('stream_id', 'unknown')})" + ) + # Get the category_id for this stream category_id = str(stream.get("category_id", "")) @@ -758,7 +775,7 @@ def collect_xc_streams(account_id, enabled_groups): # Convert XC stream to our standard format with all properties preserved stream_data = { - "name": stream["name"], + "name": stream_name, "url": xc_client.get_stream_url(stream["stream_id"]), "attributes": { "tvg-id": stream.get("epg_channel_id", ""), @@ -842,7 +859,12 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): ) for stream in streams: - name = stream["name"] + name = stream.get("name") or f"{account.name} - {stream.get('stream_id', 'Unknown')}" + if not stream.get("name"): + logger.warning( + f"XC stream has null/empty name in category {group_name}; " + f"using generated name '{name}' (stream_id={stream.get('stream_id', 'unknown')})" + ) raw_stream_id = stream.get("stream_id", "") provider_stream_id = None if raw_stream_id: @@ -875,7 +897,7 @@ def process_xc_category_direct(account_id, batch, groups, hash_keys): "channel_group_id": int(group_id), "stream_hash": stream_hash, "custom_properties": stream, - "is_adult": int(stream.get("is_adult", 0)) == 1, + "is_adult": parse_is_adult(stream.get("is_adult", 0)), "is_stale": False, "stream_id": provider_stream_id, "stream_chno": stream_chno, @@ -1093,7 +1115,7 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys): "channel_group_id": int(groups.get(group_title)), "stream_hash": stream_hash, "custom_properties": stream_info["attributes"], - "is_adult": int(stream_info["attributes"].get("is_adult", 0)) == 1, + "is_adult": parse_is_adult(stream_info["attributes"].get("is_adult", 0)), "is_stale": False, "stream_id": provider_stream_id, "stream_chno": channel_num, diff --git a/apps/m3u/tests/test_extinf_parsing.py b/apps/m3u/tests/test_extinf_parsing.py new file mode 100644 index 00000000..367f569b --- /dev/null +++ b/apps/m3u/tests/test_extinf_parsing.py @@ -0,0 +1,41 @@ +from django.test import SimpleTestCase + +from apps.m3u.tasks import parse_extinf_line + + +class ParseExtinfLineTests(SimpleTestCase): + def test_preserves_equals_padding_in_tvg_logo(self): + line = ( + '#EXTINF:-1 tvg-id="cp_891ee08a2cdfde210ec2c9137127103b" ' + 'tvg-chno="1001" ' + 'tvg-name="UK Sky Sports Premier League" ' + 'tvg-logo="https://e3.365dm.com/tvlogos/channels/1303-Logo.png?' + 'U2t5IFNwb3J0cyBQcmVtaWVyIExlYWd1ZQ==" ' + 'group-title="Team Games",UK Sky Sports Premier League' + ) + + parsed = parse_extinf_line(line) + + self.assertIsNotNone(parsed) + self.assertEqual( + parsed["attributes"]["tvg-logo"], + "https://e3.365dm.com/tvlogos/channels/1303-Logo.png?U2t5IFNwb3J0cyBQcmVtaWVyIExlYWd1ZQ==", + ) + self.assertEqual(parsed["attributes"]["group-title"], "Team Games") + self.assertEqual(parsed["name"], "UK Sky Sports Premier League") + + def test_supports_single_quoted_attributes(self): + line = ( + "#EXTINF:-1 tvg-name='Channel One' tvg-logo='https://example.com/logo==.png' " + "group-title='Sports',Channel One" + ) + + parsed = parse_extinf_line(line) + + self.assertIsNotNone(parsed) + self.assertEqual( + parsed["attributes"]["tvg-logo"], + "https://example.com/logo==.png", + ) + self.assertEqual(parsed["attributes"]["group-title"], "Sports") + self.assertEqual(parsed["display_name"], "Channel One") diff --git a/apps/proxy/ts_proxy/url_utils.py b/apps/proxy/ts_proxy/url_utils.py index fe06ad48..14a714ea 100644 --- a/apps/proxy/ts_proxy/url_utils.py +++ b/apps/proxy/ts_proxy/url_utils.py @@ -39,84 +39,44 @@ def generate_stream_url(channel_id: str) -> Tuple[str, str, bool, Optional[int]] # Handle direct stream preview (custom streams) if isinstance(channel_or_stream, Stream): - from core.utils import RedisClient - stream = channel_or_stream logger.info(f"Previewing stream directly: {stream.id} ({stream.name})") - # For custom streams, we need to get the M3U account and profile - m3u_account = stream.m3u_account - if not m3u_account: + if not stream.m3u_account: logger.error(f"Stream {stream.id} has no M3U account") return None, None, False, None - # Get active profiles for this M3U account - m3u_profiles = m3u_account.profiles.filter(is_active=True) - default_profile = next((obj for obj in m3u_profiles if obj.is_default), None) - - if not default_profile: - logger.error(f"No default active profile found for M3U account {m3u_account.id}") + # Use get_stream() to atomically reserve a slot and write the + # channel_stream / stream_profile Redis keys, matching the channel + # path so stream_name and stream_stats work correctly. + stream_id, profile_id, error_reason = stream.get_stream() + if not stream_id or not profile_id: + logger.error(f"No profile available for stream {stream.id}: {error_reason}") return None, None, False, None - # Check profiles in order: default first, then others - profiles = [default_profile] + [obj for obj in m3u_profiles if not obj.is_default] + try: + profile = M3UAccountProfile.objects.get(id=profile_id) + m3u_account = stream.m3u_account - # Try to find an available profile with connection capacity - redis_client = RedisClient.get_client() - selected_profile = None + stream_user_agent = m3u_account.get_user_agent().user_agent + if stream_user_agent is None: + stream_user_agent = UserAgent.objects.get(id=CoreSettings.get_default_user_agent_id()) + logger.debug(f"No user agent found for account, using default: {stream_user_agent}") - for profile in profiles: - logger.info(profile) + stream_url = transform_url(stream.url, profile.search_pattern, profile.replace_pattern) - # Check connection availability - if redis_client: - profile_connections_key = f"profile_connections:{profile.id}" - current_connections = int(redis_client.get(profile_connections_key) or 0) + stream_profile = stream.get_stream_profile() + logger.debug(f"Using stream profile: {stream_profile.name}") - # Check if profile has available slots (or unlimited connections) - if profile.max_streams == 0 or current_connections < profile.max_streams: - selected_profile = profile - logger.debug(f"Selected profile {profile.id} with {current_connections}/{profile.max_streams} connections for stream preview") - break - else: - logger.debug(f"Profile {profile.id} at max connections: {current_connections}/{profile.max_streams}") - else: - # No Redis available, use first active profile - selected_profile = profile - break + transcode = not stream_profile.is_proxy() + stream_profile_id = stream_profile.id - if not selected_profile: - logger.error(f"No profiles available with connection capacity for M3U account {m3u_account.id}") + return stream_url, stream_user_agent, transcode, stream_profile_id + except Exception as e: + logger.error(f"Error generating stream URL for stream {stream.id}: {e}") + stream.release_stream() return None, None, False, None - # Get the appropriate user agent - stream_user_agent = m3u_account.get_user_agent().user_agent - if stream_user_agent is None: - stream_user_agent = UserAgent.objects.get(id=CoreSettings.get_default_user_agent_id()) - logger.debug(f"No user agent found for account, using default: {stream_user_agent}") - - # Get stream URL with the selected profile's URL transformation - stream_url = transform_url(stream.url, selected_profile.search_pattern, selected_profile.replace_pattern) - - # Check if the stream has its own stream_profile set, otherwise use default - if stream.stream_profile: - stream_profile = stream.stream_profile - logger.debug(f"Using stream's own stream profile: {stream_profile.name}") - else: - stream_profile = StreamProfile.objects.get( - id=CoreSettings.get_default_stream_profile_id() - ) - logger.debug(f"Using default stream profile: {stream_profile.name}") - - # Check if transcoding is needed - if stream_profile.is_proxy() or stream_profile is None: - transcode = False - else: - transcode = True - - stream_profile_id = stream_profile.id - - return stream_url, stream_user_agent, transcode, stream_profile_id # Handle channel preview (existing logic) channel = channel_or_stream diff --git a/apps/vod/tasks.py b/apps/vod/tasks.py index 1b12c7b7..ff195fc9 100644 --- a/apps/vod/tasks.py +++ b/apps/vod/tasks.py @@ -1645,14 +1645,30 @@ def cleanup_orphaned_vod_content(stale_days=0, scan_start_time=None, account_id= orphaned_movie_count = orphaned_movies.count() if orphaned_movie_count > 0: logger.info(f"Deleting {orphaned_movie_count} orphaned movies with no M3U relations") - orphaned_movies.delete() + try: + orphaned_movies.delete() + except IntegrityError: + # A concurrent refresh task created a new relation for one of these movies + # between our query and the DELETE. Skip and let the next cleanup run handle it. + logger.warning( + "Skipped some orphaned movie deletions due to concurrent modifications; " + "they will be retried on the next cleanup run." + ) + orphaned_movie_count = 0 # Clean up series with no relations (orphaned) orphaned_series = Series.objects.filter(m3u_relations__isnull=True) orphaned_series_count = orphaned_series.count() if orphaned_series_count > 0: logger.info(f"Deleting {orphaned_series_count} orphaned series with no M3U relations") - orphaned_series.delete() + try: + orphaned_series.delete() + except IntegrityError: + logger.warning( + "Skipped some orphaned series deletions due to concurrent modifications; " + "they will be retried on the next cleanup run." + ) + orphaned_series_count = 0 result = (f"Cleaned up {stale_movie_count} stale movie relations, " f"{stale_series_count} stale series relations, " diff --git a/core/apps.py b/core/apps.py index f2780bd1..ee182e89 100644 --- a/core/apps.py +++ b/core/apps.py @@ -31,7 +31,6 @@ class CoreConfig(AppConfig): return self._sync_developer_notifications() - self._check_version_update() def _sync_developer_notifications(self): """Sync developer notifications from JSON file to database.""" @@ -47,14 +46,3 @@ class CoreConfig(AppConfig): except Exception as e: logger.warning(f"Failed to sync developer notifications on startup: {e}") - def _check_version_update(self): - """Check for version updates on startup.""" - import logging - - logger = logging.getLogger(__name__) - - try: - from core.tasks import check_for_version_update - check_for_version_update.delay() - except Exception as e: - logger.warning(f"Failed to check for version updates on startup: {e}") diff --git a/core/tasks.py b/core/tasks.py index 44ddc68a..a18c2f74 100644 --- a/core/tasks.py +++ b/core/tasks.py @@ -789,6 +789,7 @@ def check_for_version_update(): try: is_dev_build = __timestamp__ is not None DISPATCHARR_HEADERS = {'User-Agent': f'Dispatcharr/{__version__}'} + if is_dev_build: # Check Docker Hub for newer dev builds docker_hub_url = "https://hub.docker.com/v2/repositories/dispatcharr/dispatcharr/tags/dev" @@ -878,7 +879,21 @@ def check_for_version_update(): } ) else: - # Production build - check GitHub for stable releases + # Production build - check GitHub for stable releases. + # Delete any stale notification for the currently running version upfront; + # a "vX is available" notification is meaningless once the user is already on vX. + # Notify the frontend immediately so the badge clears without waiting for the API call. + deleted_count = SystemNotification.objects.filter( + notification_key=f"version-{__version__}", + notification_type='version_update', + ).delete()[0] + if deleted_count > 0: + send_websocket_update( + 'updates', + 'update', + {'success': True, 'type': 'notifications_cleared', 'count': deleted_count} + ) + github_api_url = "https://api.github.com/repos/Dispatcharr/Dispatcharr/releases/latest" headers = {"Accept": "application/vnd.github.v3+json", **DISPATCHARR_HEADERS} response = requests.get( diff --git a/debian_install.sh b/debian_install.sh index 3630161d..305424f8 100755 --- a/debian_install.sh +++ b/debian_install.sh @@ -12,9 +12,20 @@ fi trap 'echo -e "\n[ERROR] Line $LINENO failed. Exiting." >&2; exit 1' ERR ############################################################################## -# 0) Warning / Disclaimer +# 0) Locales & Warning / Disclaimer ############################################################################## +setup_locales() { + echo ">>> Setting up locales..." + apt-get update + apt-get install -y locales + sed -i '/en_US.UTF-8 UTF-8/s/^# //g' /etc/locale.gen + locale-gen + update-locale LANG=en_US.UTF-8 + export LANG=en_US.UTF-8 + export LC_ALL=en_US.UTF-8 +} + show_disclaimer() { echo "**************************************************************" echo "WARNING: While we do not anticipate any problems, we disclaim all" @@ -106,8 +117,8 @@ setup_postgresql() { db_exists=$(sudo -u postgres psql -tAc "SELECT 1 FROM pg_database WHERE datname='$POSTGRES_DB'") if [[ "$db_exists" != "1" ]]; then - echo ">>> Creating database '${POSTGRES_DB}'..." - sudo -u postgres createdb "$POSTGRES_DB" + echo ">>> Creating database '${POSTGRES_DB}' with UTF8 encoding..." + sudo -u postgres createdb -E UTF8 "$POSTGRES_DB" else echo ">>> Database '${POSTGRES_DB}' already exists, skipping creation." fi @@ -326,7 +337,7 @@ User=${DISPATCH_USER} Group=${DISPATCH_GROUP} WorkingDirectory=${APP_DIR} EnvironmentFile=/opt/dispatcharr/.env -Environment="PATH=${APP_DIR}/env/bin" +Environment="PATH=${APP_DIR}/env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin" Environment="POSTGRES_DB=${POSTGRES_DB}" Environment="POSTGRES_USER=${POSTGRES_USER}" Environment="POSTGRES_PASSWORD=${POSTGRES_PASSWORD}" @@ -354,7 +365,7 @@ User=${DISPATCH_USER} Group=${DISPATCH_GROUP} WorkingDirectory=${APP_DIR} EnvironmentFile=/opt/dispatcharr/.env -Environment="PATH=${APP_DIR}/env/bin" +Environment="PATH=${APP_DIR}/env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin" Environment="POSTGRES_DB=${POSTGRES_DB}" Environment="POSTGRES_USER=${POSTGRES_USER}" Environment="POSTGRES_PASSWORD=${POSTGRES_PASSWORD}" @@ -382,7 +393,7 @@ User=${DISPATCH_USER} Group=${DISPATCH_GROUP} WorkingDirectory=${APP_DIR} EnvironmentFile=/opt/dispatcharr/.env -Environment="PATH=${APP_DIR}/env/bin" +Environment="PATH=${APP_DIR}/env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin" Environment="POSTGRES_DB=${POSTGRES_DB}" Environment="POSTGRES_USER=${POSTGRES_USER}" Environment="POSTGRES_PASSWORD=${POSTGRES_PASSWORD}" @@ -474,6 +485,7 @@ EOF ############################################################################## main() { + setup_locales show_disclaimer configure_variables install_packages diff --git a/dispatcharr/celery.py b/dispatcharr/celery.py index 436ffc35..8ccb3c33 100644 --- a/dispatcharr/celery.py +++ b/dispatcharr/celery.py @@ -54,7 +54,7 @@ app.conf.update( def cleanup_task_memory(**kwargs): """Clean up memory and database connections after each task completes""" from django.db import connection - + # Get task name from kwargs task_name = kwargs.get('task').name if kwargs.get('task') else '' @@ -153,6 +153,9 @@ def setup_celery_logging(**kwargs): @worker_ready.connect def on_worker_ready(**kwargs): - """Resume or finalize interrupted DVR recordings after a worker restart.""" + """Tasks to run once the worker is fully connected and ready.""" from apps.channels.tasks import recover_recordings_on_startup recover_recordings_on_startup.delay() + + from core.tasks import check_for_version_update + check_for_version_update.delay() diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index 8d086500..d674460a 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -15,8 +15,12 @@ services: volumes: - ./data:/data depends_on: - - db - - redis + db: + condition: service_healthy + redis: + condition: service_healthy + extra_hosts: + - "host.docker.internal:host-gateway" # --- Environment Configuration --- environment: @@ -83,9 +87,12 @@ services: container_name: dispatcharr_celery restart: unless-stopped depends_on: - - db - - redis - - web + db: + condition: service_healthy + redis: + condition: service_healthy + web: + condition: service_started volumes: - ./data:/data extra_hosts: @@ -97,6 +104,10 @@ services: # Deployment Mode - DISPATCHARR_ENV=modular + # Internal Service Communication + # Must match the web service port for DVR recording and internal API calls + - DISPATCHARR_PORT=9191 + # PostgreSQL Connection - POSTGRES_HOST=db - POSTGRES_PORT=5432 diff --git a/docker/entrypoint.celery.sh b/docker/entrypoint.celery.sh index e5c2f340..1c14f9b6 100644 --- a/docker/entrypoint.celery.sh +++ b/docker/entrypoint.celery.sh @@ -9,9 +9,19 @@ echo_with_timestamp() { echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" } -# Wait for Django secret key +# Wait for Django secret key (generated by the web container on startup) +JWT_TIMEOUT=120 +JWT_WAITED=0 echo 'Waiting for Django secret key...' -while [ ! -f /data/jwt ]; do sleep 1; done +while [ ! -f /data/jwt ]; do + if [ $JWT_WAITED -ge $JWT_TIMEOUT ]; then + echo "❌ ERROR: Timed out waiting for /data/jwt after ${JWT_TIMEOUT}s." + echo " Is the web container running? Does it have the /data volume mounted?" + exit 1 + fi + sleep 1 + JWT_WAITED=$((JWT_WAITED + 1)) +done export DJANGO_SECRET_KEY="$(tr -d '\r\n' < /data/jwt)" # --- NumPy version switching for legacy hardware --- @@ -26,11 +36,21 @@ if [ "$USE_LEGACY_NUMPY" = "true" ]; then fi fi -# Wait for migrations to complete (check that NO unapplied migrations remain) +# Wait for migrations to complete +# Uses 'migrate --check' which exits 0 only when all migrations are applied, +# and exits 1 on unapplied migrations OR connection errors (safe either way) +MIG_TIMEOUT=300 +MIG_WAITED=0 echo 'Waiting for migrations to complete...' -until ! python manage.py showmigrations 2>&1 | grep -q '\[ \]'; do +until python manage.py migrate --check >/dev/null 2>&1; do + if [ $MIG_WAITED -ge $MIG_TIMEOUT ]; then + echo "❌ ERROR: Timed out waiting for migrations after ${MIG_TIMEOUT}s." + echo " Check web container logs for migration errors." + exit 1 + fi echo_with_timestamp 'Migrations not ready yet, waiting...' sleep 2 + MIG_WAITED=$((MIG_WAITED + 2)) done # Start Celery diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh index b79af952..7d2a1bf6 100755 --- a/docker/entrypoint.sh +++ b/docker/entrypoint.sh @@ -99,31 +99,35 @@ echo "Environment DISPATCHARR_LOG_LEVEL set to: '${DISPATCHARR_LOG_LEVEL}'" # READ-ONLY - don't let users change these export POSTGRES_DIR=/data/db -# Global variables, stored so other users inherit them -if [[ ! -f /etc/profile.d/dispatcharr.sh ]]; then - # Define all variables to process - variables=( - PATH VIRTUAL_ENV DJANGO_SETTINGS_MODULE PYTHONUNBUFFERED PYTHONDONTWRITEBYTECODE - POSTGRES_DB POSTGRES_USER POSTGRES_PASSWORD POSTGRES_HOST POSTGRES_PORT - DISPATCHARR_ENV DISPATCHARR_DEBUG DISPATCHARR_LOG_LEVEL - REDIS_HOST REDIS_PORT REDIS_DB REDIS_PASSWORD REDIS_USER POSTGRES_DIR DISPATCHARR_PORT - DISPATCHARR_VERSION DISPATCHARR_TIMESTAMP LIBVA_DRIVERS_PATH LIBVA_DRIVER_NAME LD_LIBRARY_PATH - CELERY_NICE_LEVEL UWSGI_NICE_LEVEL DJANGO_SECRET_KEY - ) +# Global variables, stored so other users inherit them. +# Rewritten every startup so that container restarts with changed env vars +# pick up the new values (not stale ones from a previous run). +# Define all variables to process +variables=( + PATH VIRTUAL_ENV DJANGO_SETTINGS_MODULE PYTHONUNBUFFERED PYTHONDONTWRITEBYTECODE + POSTGRES_DB POSTGRES_USER POSTGRES_PASSWORD POSTGRES_HOST POSTGRES_PORT + DISPATCHARR_ENV DISPATCHARR_DEBUG DISPATCHARR_LOG_LEVEL + REDIS_HOST REDIS_PORT REDIS_DB REDIS_PASSWORD REDIS_USER POSTGRES_DIR DISPATCHARR_PORT + DISPATCHARR_VERSION DISPATCHARR_TIMESTAMP LIBVA_DRIVERS_PATH LIBVA_DRIVER_NAME LD_LIBRARY_PATH + CELERY_NICE_LEVEL UWSGI_NICE_LEVEL DJANGO_SECRET_KEY +) - # Process each variable for both profile.d and environment - for var in "${variables[@]}"; do - # Check if the variable is set in the environment - if [ -n "${!var+x}" ]; then - # Add to profile.d - echo "export ${var}=${!var}" >> /etc/profile.d/dispatcharr.sh - # Add to /etc/environment if not already there - grep -q "^${var}=" /etc/environment || echo "${var}=${!var}" >> /etc/environment - else - echo "Warning: Environment variable $var is not set" - fi - done -fi +# Truncate files before rewriting +> /etc/profile.d/dispatcharr.sh + +# Process each variable for both profile.d and environment +for var in "${variables[@]}"; do + # Check if the variable is set in the environment + if [ -n "${!var+x}" ]; then + # Add to profile.d (quoted to handle special characters in values) + echo "export ${var}='${!var}'" >> /etc/profile.d/dispatcharr.sh + # Add/update in /etc/environment + sed -i "/^${var}=/d" /etc/environment + echo "${var}='${!var}'" >> /etc/environment + else + echo "Warning: Environment variable $var is not set" + fi +done chmod +x /etc/profile.d/dispatcharr.sh @@ -175,23 +179,17 @@ else check_external_postgres_version || exit 1 fi -# Wait for Redis to be ready (modular mode uses external Redis) +# Wait for Redis to be ready and flush stale state. +# In modular mode Redis is external — call wait_for_redis.py here +# because uWSGI's exec-pre runs under 'su -' which strips env vars +# (DISPATCHARR_ENV, REDIS_HOST, etc.). +# In AIO mode Redis is started by uWSGI (attach-daemon), so the +# exec-pre in uwsgi.ini handles the wait + flush there instead. if [[ "$DISPATCHARR_ENV" == "modular" ]]; then echo "🔗 Modular mode: Using external Redis at ${REDIS_HOST}:${REDIS_PORT}" - echo_with_timestamp "Waiting for external Redis to be ready..." - until python3 -c " -import socket, sys -try: - s = socket.create_connection(('${REDIS_HOST}', ${REDIS_PORT}), timeout=2) - s.close() - sys.exit(0) -except Exception: - sys.exit(1) -" 2>/dev/null; do - echo_with_timestamp "Waiting for Redis at ${REDIS_HOST}:${REDIS_PORT}..." - sleep 1 - done - echo "✅ External Redis is ready" + echo_with_timestamp "Waiting for Redis to be ready..." + python3 /app/scripts/wait_for_redis.py + echo "✅ Redis is ready" fi # Ensure database encoding is UTF8 (handles both internal and external databases) diff --git a/docker/init/02-postgres.sh b/docker/init/02-postgres.sh index 87aa94ef..2ece526f 100644 --- a/docker/init/02-postgres.sh +++ b/docker/init/02-postgres.sh @@ -204,14 +204,19 @@ check_external_postgres_version() { MIN_REQUIRED_VERSION=$PG_VERSION # Query external PostgreSQL version - EXTERNAL_VERSION=$(PGPASSWORD="$POSTGRES_PASSWORD" psql -w -h "$POSTGRES_HOST" -p "$POSTGRES_PORT" -U "$POSTGRES_USER" -d "postgres" -tAc "SHOW server_version;" 2>/dev/null | grep -oE '^[0-9]+') + # Use $POSTGRES_DB — restricted users may not have access to the default 'postgres' database + PG_VERSION_ERR=$(mktemp) + EXTERNAL_VERSION=$(PGPASSWORD="$POSTGRES_PASSWORD" psql -w -h "$POSTGRES_HOST" -p "$POSTGRES_PORT" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc "SHOW server_version;" 2>"$PG_VERSION_ERR" | grep -oE '^[0-9]+') if [ -z "$EXTERNAL_VERSION" ]; then echo "❌ ERROR: Unable to determine external PostgreSQL version" - echo " Could not connect to database at ${POSTGRES_HOST}:${POSTGRES_PORT}" + echo " Could not connect to database '$POSTGRES_DB' at ${POSTGRES_HOST}:${POSTGRES_PORT} as user '$POSTGRES_USER'" + echo " Error: $(cat "$PG_VERSION_ERR")" echo " Please verify your database connection settings." + rm -f "$PG_VERSION_ERR" return 1 fi + rm -f "$PG_VERSION_ERR" # Compare versions if [[ "$EXTERNAL_VERSION" -lt "$MIN_REQUIRED_VERSION" ]]; then diff --git a/docker/uwsgi.modular.ini b/docker/uwsgi.modular.ini index 3220a8d8..57ecaea8 100644 --- a/docker/uwsgi.modular.ini +++ b/docker/uwsgi.modular.ini @@ -5,8 +5,8 @@ ; exec-pre = touch /data/logs/uwsgi.log ; exec-pre = chmod 666 /data/logs/uwsgi.log -; First run Redis availability check script once -exec-pre = python /app/scripts/wait_for_redis.py +; Redis wait + flush is handled by the entrypoint in modular mode +; (uWSGI exec-pre runs under 'su -' which strips Docker env vars) ; Start Daphne for WebSocket support (required for real-time features) ; Redis and Celery run in separate containers in modular mode diff --git a/frontend/src/api.js b/frontend/src/api.js index 35233204..9e4ac982 100644 --- a/frontend/src/api.js +++ b/frontend/src/api.js @@ -2675,11 +2675,14 @@ export default class API { static async extendRecording(id, extraMinutes) { try { - const resp = await request(`${host}/api/channels/recordings/${id}/extend/`, { - method: 'POST', - body: JSON.stringify({ extra_minutes: extraMinutes }), - headers: { 'Content-Type': 'application/json' }, - }); + const resp = await request( + `${host}/api/channels/recordings/${id}/extend/`, + { + method: 'POST', + body: JSON.stringify({ extra_minutes: extraMinutes }), + headers: { 'Content-Type': 'application/json' }, + } + ); return resp; } catch (e) { errorNotification(`Failed to extend recording ${id}`, e); @@ -2899,6 +2902,13 @@ export default class API { return await request(`${host}/api/accounts/users/me/`); } + static async updateMe(data) { + return await request(`${host}/api/accounts/users/me/`, { + method: 'PATCH', + body: data, + }); + } + static async getUsers() { try { const response = await request(`${host}/api/accounts/users/`); diff --git a/frontend/src/components/M3URefreshNotification.jsx b/frontend/src/components/M3URefreshNotification.jsx index 8fd461b4..bd6319bf 100644 --- a/frontend/src/components/M3URefreshNotification.jsx +++ b/frontend/src/components/M3URefreshNotification.jsx @@ -11,12 +11,12 @@ import { useNavigate } from 'react-router-dom'; import { CircleCheck } from 'lucide-react'; import { showNotification } from '../utils/notificationUtils.js'; -const M3uSetupSuccess = (data) => { +const M3uSetupSuccess = ({ data }) => { const navigate = useNavigate(); const onClickRefresh = () => { API.refreshPlaylist(data.account); - } + }; const onClickConfigure = () => { // Store the ID we want to edit in the store first @@ -25,7 +25,7 @@ const M3uSetupSuccess = (data) => { // Then navigate to the content sources page // Using the exact path that matches your app's routing structure navigate('/sources'); - } + }; return ( @@ -41,14 +41,14 @@ const M3uSetupSuccess = (data) => { ); -} +}; export default function M3URefreshNotification() { const playlists = usePlaylistsStore((s) => s.playlists); const refreshProgress = usePlaylistsStore((s) => s.refreshProgress); const fetchStreams = useStreamsStore((s) => s.fetchStreams); const fetchChannelGroups = useChannelsStore((s) => s.fetchChannelGroups); - const fetchChannels = useChannelsStore((s) => s.fetchChannels); + const fetchChannelIds = useChannelsStore((s) => s.fetchChannelIds); const fetchPlaylists = usePlaylistsStore((s) => s.fetchPlaylists); const fetchEPGData = useEPGsStore((s) => s.fetchEPGData); const fetchCategories = useVODStore((s) => s.fetchCategories); @@ -69,7 +69,7 @@ export default function M3URefreshNotification() { } // Update notification status - setNotificationStatus(prev => ({ + setNotificationStatus((prev) => ({ ...prev, [data.account]: data, })); @@ -134,7 +134,7 @@ export default function M3URefreshNotification() { if (action == 'parsing') { fetchStreams(); API.requeryChannels(); - fetchChannels(); + fetchChannelIds(); } else if (action == 'processing_groups') { fetchStreams(); fetchChannelGroups(); @@ -148,9 +148,10 @@ export default function M3URefreshNotification() { const handleProgressNotification = (playlist, data) => { const baseMessage = getActionMessage(data.action); - const message = data.progress == 0 - ? `${baseMessage} starting...` - : `${baseMessage} complete!`; + const message = + data.progress == 0 + ? `${baseMessage} starting...` + : `${baseMessage} complete!`; if (data.progress == 100) { triggerPostCompletionFetches(data.action); diff --git a/frontend/src/components/Sidebar.jsx b/frontend/src/components/Sidebar.jsx index 0b016e7d..73ee13c3 100644 --- a/frontend/src/components/Sidebar.jsx +++ b/frontend/src/components/Sidebar.jsx @@ -1,26 +1,13 @@ -import React, { useRef, useState } from 'react'; +import React, { useRef, useState, useMemo } from 'react'; import { Link, useLocation } from 'react-router-dom'; import { copyToClipboard } from '../utils'; import { - ListOrdered, - Play, - Database, - LayoutGrid, - Settings as LucideSettings, Copy, - ChartLine, - Video, - PlugZap, LogOut, - User, - FileImage, - Webhook, - Logs, ChevronDown, ChevronRight, - MonitorCog, - Blocks, } from 'lucide-react'; +import { getOrderedNavItems } from '../config/navigation'; import { Avatar, Group, @@ -43,6 +30,7 @@ import UserForm from './forms/User'; import NotificationCenter from './NotificationCenter'; const NavLink = ({ item, isActive, collapsed }) => { + const IconComponent = item.icon; return ( { to={item.path} className={`navlink ${isActive ? 'navlink-active' : ''} ${collapsed ? 'navlink-collapsed' : ''}`} > - {item.icon} + {IconComponent && } {!collapsed && ( { ); }; -function NavGroup({ label, icon, paths, location, collapsed }) { +function NavGroup({ label, icon: IconComponent, paths, location, collapsed }) { const [open, setOpen] = useState(() => - location.pathname.startsWith('/connect') + paths.some((p) => location.pathname.startsWith(p.path)) ); const parentActive = paths @@ -93,7 +81,7 @@ function NavGroup({ label, icon, paths, location, collapsed }) { className={`navlink ${parentActive ? 'navlink-parent-active' : ''} ${open ? 'navlink-collapsed' : ''}`} style={{ width: '100%' }} > - {icon} + {IconComponent && } {!collapsed && ( { const isAuthenticated = useAuthStore((s) => s.isAuthenticated); const authUser = useAuthStore((s) => s.user); const logout = useAuthStore((s) => s.logout); + const getNavOrder = useAuthStore((s) => s.getNavOrder); + const getHiddenNav = useAuthStore((s) => s.getHiddenNav); const publicIPRef = useRef(null); @@ -158,82 +148,15 @@ const Sidebar = ({ collapsed, toggleDrawer, drawerWidth, miniDrawerWidth }) => { const closeUserForm = () => setUserFormOpen(false); - // Navigation Items - const navItems = - authUser && authUser.user_level == USER_LEVELS.ADMIN - ? [ - { - label: 'Channels', - icon: , - path: '/channels', - badge: `(${Array.isArray(channelIds) ? channelIds.length : 0})`, - }, - { - label: 'VODs', - path: '/vods', - icon: