Commit graph

51 commits

Author SHA1 Message Date
SergeantPanda
b671a72707 fix: resolve decode_responses migration bugs and user-limit regressions
Fix a wave of bugs introduced by the user-priority branch's migration
from manual .decode('utf-8') calls to decode_responses=True on the
metadata Redis client:

core/utils.py
- Rewrite _init_client: fix SyntaxError at line 138, missing
  redis_password/redis_user params, cls._client corruption when
  decode_responses=False was requested, and dead retry backoff logic

apps/proxy/utils.py
- Fix get_user_limit_settings() → get_user_limits_settings() (2 sites)
- Fix VOD scan key vod_persistent_connections:* → vod_persistent_connection:*
- Fix undefined channel_id in VOD loop (use content_uuid from Redis hash)
- Remove leftover print(active_connections) debug statement
- Refactor manual cursor SCAN while-loops to scan_iter()

apps/channels/tasks.py
- Fix closure bug in _d(): md.get(key) → md.get(bkey) (was reading
  the outer loop variable instead of the local bytes key)

apps/proxy/ts_proxy/server.py
- Replace stale b'init_time', b'total_bytes', b'state' byte-string
  key lookups and .decode() calls with plain string equivalents

apps/proxy/ts_proxy/services/channel_service.py
- Fix ChannelMetadataField.STATE.encode() / .OWNER.encode() →
  .STATE / .OWNER (pre-existing, broke under decoded client)

apps/proxy/ts_proxy/views.py
- Fix ChannelMetadataField.OWNER.encode("utf-8") → .OWNER

apps/proxy/ts_proxy/client_manager.py
- Fix cid.decode('utf-8') in remove_ghost_clients(): smembers()
  already returns str with decode_responses=True
2026-03-26 18:41:42 -05:00
dekzter
90a600b48f merged in dev 2026-03-26 12:17:43 -04:00
dekzter
a19be96ced fleshed out user limits and termination logic 2026-03-25 17:33:26 -04:00
None
b30a24e2fb feat: add TLS connection support for Redis and PostgreSQL (#950)
- Add 10 env vars for Redis/PostgreSQL TLS (REDIS_SSL, REDIS_SSL_VERIFY, REDIS_SSL_CA_CERT, REDIS_SSL_CERT, REDIS_SSL_KEY, POSTGRES_SSL, POSTGRES_SSL_MODE, POSTGRES_SSL_CA_CERT, POSTGRES_SSL_CERT, POSTGRES_SSL_KEY)
- Propagate SSL params to all Redis connection points: RedisClient, stream view, PubSub, client manager, Celery broker/result backend, Django Channels, and pre-Django startup scripts
- Add Celery SSL config via URL query string params (required by Kombu's internal URL parsing)
- Add PostgreSQL sslmode and certificate options to DATABASES config
- Add startup validation: cert file existence, URL scheme conflict detection, TLS status logging
- Add TLS error hints to Redis connection failure messages
- Add Connection Security panel in System Settings (modular mode only) showing encryption, verification, and mTLS status
- Add PG client key permission fix in web and celery entrypoints for Docker Desktop compatibility (0777 to 0600)
- Add TLS env var passthrough in entrypoint for su - login shells
- Document TLS env vars in modular docker-compose.yml
- Change env_mode from dev/prod to actual DISPATCHARR_ENV value for modular mode detection
2026-03-21 17:24:09 -05:00
dekzter
f69a462253 merged in dev 2026-03-13 08:22:44 -04:00
None
f28530a47b fix(dvr): add recording extend, fix cross-recording interference, artwork race, and keepalive gaps
File changes:

Backend

apps/channels/api_views.py — Added extend() action that uses queryset .update() to bypass signals, letting the running task pick up the new end_time via polling. Moved WS recording_stopped notification to synchronous (before HTTP response). Refactored destroy() to delete DB row first, send WS event immediately, defer cleanup to background thread.

apps/channels/signals.py — Added pre_save guard to skip task revocation when status is "recording". Added post_save reentrancy guard to skip artwork prefetch for internal field-only saves (custom_properties, task_id, end_time).

apps/channels/tasks.py — Added 2-second polling loop in streaming that checks for stop, deletion, and extended end_time. Added refresh_from_db() + merge before metadata save to prevent overwriting concurrent artwork prefetch. Replaced inline PeriodicTask cleanup with revoke_task()/_dvr_task_name(). Consolidated redundant DB queries in error path. Fixed TOCTOU os.path.exists + os.remove. Preserved "stopped" status in final-status logic instead of overwriting with "completed".

core/utils.py — Made send_websocket_update gevent-safe by detecting monkey-patching and offloading async_to_sync to gevent's native threadpool, fixing the event loop deadlock that caused cross-recording interference.

apps/proxy/ts_proxy/client_manager.py — Moved _trigger_stats_update to a background thread so remove_client() returns immediately.

apps/proxy/ts_proxy/stream_generator.py — Added Redis-based health check for non-owner workers in _should_send_keepalive(), so keepalives fire correctly even when stream_manager is None.

Frontend

frontend/src/WebSocket.jsx — Added 400ms debounced scheduleRecordingFetch() replacing all inline fetchRecordings() calls. Added recording_extended and recording_updated event handlers. recording_cancelled now does surgical removeRecording() by ID when available.

frontend/src/api.js — Added extendRecording(id, extraMinutes).

frontend/src/components/cards/RecordingCard.jsx — Added Extend menu (+15m/+30m/+1h) for in-progress recordings. Widened Comskip button to stopped/interrupted statuses.

frontend/src/components/forms/ProgramRecordingModal.jsx — Removed manual fetchRecordings() calls, relying on WS-driven debounced refresh.

frontend/src/components/forms/RecordingDetailsModal.jsx — Added stopped to playable statuses. Widened Comskip button. Removed manual fetchRecordings().

frontend/src/components/forms/RecurringRuleModal.jsx — Removed all manual fetchRecordings() calls, relying on WS events.

frontend/src/pages/Guide.jsx — Removed isLoading subscription that caused loading flash on debounced refresh. Removed manual fetchRecordings() after saving a series rule.

frontend/src/store/channels.jsx — Stripped isLoading/error state from fetchRecordings to prevent full-page loading indicators. Added no-op guards in removeRecording.

frontend/src/utils/cards/RecordingCardUtils.js — Added extendRecordingById wrapper.

Tests

apps/channels/tests/test_recording_extend.py (new) — 14 tests: extend endpoint validation, stacked extensions, signal bypass, pre_save guard behavior.

apps/channels/tests/test_recording_stop_cancel.py (new) — 18 tests: stop endpoint, DVR client teardown, cancel was_in_progress flag, run_recording race guards, signal reentrancy guards.

apps/channels/tests/test_ts_proxy_keepalive.py (new) — 21 tests: keepalive owner/non-owner logic, stats update error handling, client removal non-blocking, timing invariants.

apps/channels/tests/test_recording_scheduling.py — Updated mock assertion to match new _stop_dvr_clients call signature.

Summary:

Adds an Extend action for in-progress recordings that bypasses Django signals and lets the running task dynamically pick up the new end_time via its 2-second DB polling loop. Fixes cross-recording interference caused by send_websocket_update deadlocking the gevent event loop. Fixes an artwork race condition where run_recording overwrote concurrent prefetch data. Adds Redis-based keepalive health checks for non-owner workers to prevent DVR read timeouts during source transitions. Replaces manual fetchRecordings() calls throughout the frontend with a debounced WS-driven refresh. 53 new tests.
2026-03-04 10:21:02 -06:00
PFalko
8bd38ad71c Fix stream ownership bugs causing streams to die after 30-200s
Three interrelated bugs cause TS proxy streams to terminate prematurely
in multi-worker uWSGI deployments:

1. Double ProxyServer instantiation: ProxyConfig.ready() in apps/proxy/apps.py
   calls TSProxyServer() directly, bypassing get_instance(). The subsequent
   TSProxyConfig.ready() call to get_instance() creates a second instance.
   Each instance starts its own cleanup thread, but only one holds channel
   data — the orphaned cleanup thread cannot extend ownership.

2. Redis flushdb() on every client init: RedisClient.get_client() in
   core/utils.py calls flushdb() whenever a new connection is created.
   Celery autoscale workers spawning mid-stream nuke all Redis keys
   including ownership, client records, and channel metadata.

3. No recovery from expired ownership: get_channel_owner() has a TOCTOU
   bug (two separate GET calls in a lambda). extend_ownership() silently
   fails when keys expire. Non-owner cleanup unconditionally kills streams
   even when the worker holds the stream_manager.

Fixes:
- Use TSProxyServer.get_instance() in ProxyConfig.ready()
- Remove flushdb() from Redis client initialization
- Use sentinel pattern for gevent-safe singleton (threading.Lock does not
  work with gevent greenlets)
- Single GET in get_channel_owner() to avoid TOCTOU race
- Re-acquire expired ownership keys in extend_ownership()
- Attempt re-acquisition before cleanup in non-owner path

Relates to #992, #980
2026-02-27 16:40:28 +01:00
dekzter
6f4665e6eb merged in dev 2026-02-26 08:20:20 -05:00
SergeantPanda
e9a2cb4dbb
Merge pull request #1014 from CodeBormen/fix/861-M3U-downloading-endlessly
Some checks are pending
CI Pipeline / prepare (push) Waiting to run
CI Pipeline / docker (amd64, ubuntu-24.04) (push) Blocked by required conditions
CI Pipeline / docker (arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
CI Pipeline / create-manifest (push) Blocked by required conditions
Build and Push Multi-Arch Docker Image / build-and-push (push) Waiting to run
Frontend Tests / test (push) Waiting to run
Fix #861: Prevent M3U/EPG tasks from downloading endlessly
2026-02-24 14:28:36 -06:00
dekzter
9bd9985d63 remove empty keys 2026-02-24 12:19:41 -05:00
dekzter
9470600474 pass all details into event triggers, updated custom script default path 2026-02-24 12:13:21 -05:00
None
c841cfc8fc Fix #861: Prevent M3U endless downloading caused by Redis lock expiry
Large M3U downloads that exceed the 300s Redis lock TTL caused Celery Beat
to re-queue duplicate tasks, creating overlapping downloads that never complete.

Files changed:

- core/utils.py: Add TaskLockRenewer class — a daemon thread that periodically renews Redis lock TTL (every 120s) to prevent expiry during long-running tasks.

- apps/m3u/tasks.py: Apply TaskLockRenewer to refresh_single_m3u_account and refresh_m3u_groups; add HTTP timeout (30s connect, 60s read) to M3U download (the only download path missing one); stream M3U download to disk instead of accumulating in memory; add Celery task time limits (1 hour hard limit).

- apps/epg/tasks.py: Apply TaskLockRenewer to refresh_epg_data and parse_programs_for_tvg_id; add Celery task time limits (30 min / 1 hour).

Verified with a throttled test server: locks renewed at T+120s, T+240s, T+360s; 50,000 streams processed with no duplicate tasks.
2026-02-23 00:34:28 -06:00
dekzter
bc72ea7310 merged in dev 2026-02-11 19:51:21 -05:00
dekzter
e6ffeb19de added event support to plugins 2026-02-11 19:50:11 -05:00
dekzter
24f812dc4d initial connect feature 2026-02-08 09:29:22 -05:00
None
e217960500 feat: Add Redis authentication support for modular deployment
Add support for authentication when connecting to external Redis instances in modular deployment mode. This enables secure Redis deployments using either password-only authentication (Redis <6) or
username + password authentication (Redis 6+ ACL).

Changes:
- Add REDIS_PASSWORD and REDIS_USER environment variables
- Implement URL encoding for special characters in passwords
- Update all Redis connection points to support auth
- Add comprehensive documentation and examples to docker-compose.yml
- Maintains full backward compatibility (empty defaults = no auth)

All authentication mechanisms have been fully tested including pasword-only authentication, Redis 6+ ACL authentication with username + password, volume-mounted configuration files, and special character handling.

Existing deployments are not effected, authentication support is entirely opt-in using docker-compose.

Also, acknowledging the fact that the docker-compose file for modular deployments has been getting out of hand with auth support, the docker-compose.yml file was re-formatted for better visibility in configuration. This seemed like a better way to go than mandating a .env file.
2026-02-03 21:42:06 -06:00
SergeantPanda
b01eb9585c feat: add system notifications and update checks
Real-time notifications for system events and alerts
Per-user notification management and dismissal
Update check on startup and every 24 hours to notify users of available versions
Notification center UI component
Automatic cleanup of expired notifications
2026-02-03 09:24:02 -06:00
SergeantPanda
36967c10ce Refactor CoreSettings to use JSONField for value storage and update related logic for proper type handling. Adjusted serializers and forms to accommodate new data structure, ensuring seamless integration across the application. 2026-01-13 12:18:34 -06:00
dekzter
4a23883d0c merged in upstream 2025-12-05 08:34:29 -05:00
SergeantPanda
89a23164ff Enhancement: Add system event logging and viewer with M3U/EPG endpoint caching
System Event Logging:
- Add SystemEvent model with 15 event types tracking channel operations, client connections, M3U/EPG activities, and buffering events
- Log detailed metrics for M3U/EPG refresh operations (streams/programs created/updated/deleted)
- Track M3U/EPG downloads with client information (IP address, user agent, profile, channel count)
- Record channel lifecycle events (start, stop, reconnect) with stream and client details
- Monitor client connections/disconnections and buffering events with stream metadata

Event Viewer UI:
- Add SystemEvents component with real-time updates via WebSocket
- Implement pagination, filtering by event type, and configurable auto-refresh
- Display events with color-coded badges and type-specific icons
- Integrate event viewer into Stats page with modal display
- Add event management settings (retention period, refresh rate)

M3U/EPG Endpoint Optimizations:
- Implement content caching with 5-minute TTL to reduce duplicate processing
- Add client-based event deduplication (2-second window) using IP and user agent hashing
- Support HEAD requests for efficient preflight checks
- Cache streamed EPG responses while maintaining streaming behavior for first request
2025-11-20 17:41:06 -06:00
SergeantPanda
79895a1ce4 Enhancement: Update URL validation to support authentication for non-FQDN hostnames in rtsp/rtp/udp URLs 2025-11-12 16:58:07 -06:00
SergeantPanda
b9e819e343 Enhancement: Adds support for UDP streams. Closes #617 2025-11-11 18:30:59 -06:00
SergeantPanda
21723e29bc Update URL validation to support FQDNs for rtsp/rtp protocols and improve regex pattern for flexibility. 2025-11-11 17:17:01 -06:00
dekzter
50e9075bb5 initial run of a binary and encoded redis client - no more encoding / decoding data into redis, huge PITA (still some outstanding spots I need to patch) 2025-10-25 08:15:39 -04:00
Ragchuck
ae8b85a3e2 feat: added support for rtsp 2025-10-15 22:06:01 +02:00
SergeantPanda
7b5a617bf8 Use custom validator for urls fields to allow for non fqdn hostnames.
Fixes #63
2025-08-01 11:28:51 -05:00
SergeantPanda
7c442064e6 Implement natural sorting for channel names in auto channel sync 2025-07-28 18:01:50 -05:00
SergeantPanda
45239b744c Delete cached files when deleting epg account. 2025-05-26 16:19:57 -05:00
SergeantPanda
f87ab4b071 Rolled back some earlier memory omptimizations that were causing issues with extremely large m3us. 2025-05-22 21:52:28 -05:00
SergeantPanda
451c892457 Changed logging levels. 2025-05-20 19:49:55 -05:00
SergeantPanda
06b1dec2b6 Better logic for cleanup task. Skip gathering memory if we aren't going to log it anyway. 2025-05-19 10:02:42 -05:00
SergeantPanda
6087ecadf0 Cleaning up added gc's 2025-05-19 09:42:21 -05:00
SergeantPanda
f821743163 Created a utility to clean up django memory. 2025-05-18 19:46:52 -05:00
SergeantPanda
ed665584e9 The struggle is real 2025-05-18 17:05:03 -05:00
SergeantPanda
f18ca4de37 Initial rework of EPG processesing. 2025-05-16 19:26:06 -05:00
SergeantPanda
1772bc7257 Configure Redis to not write to disk and also run in unprotect mode if running in debug mode. 2025-05-15 13:16:18 -05:00
dekzter
5570562960 epg match run externally to keep memory usage low 2025-04-07 11:57:00 -04:00
dekzter
ecc96f8b69 memory optimization, m3u processing, re-added group filtering before m3u ingestion 2025-04-06 11:36:16 -04:00
dekzter
a38930b0c9 syntax error 2025-04-05 20:10:12 -04:00
dekzter
cdf9df03bd lazy-load sentencetransformer instance 2025-04-05 20:05:45 -04:00
dekzter
7351264e8a centralized and lazy-loaded redis client singleton, check for manage.py commands so we don't init proxyservers (redis connection), put manage commmands before starting uwsgi 2025-04-04 16:18:12 -04:00
dekzter
83b11254ac clear redis when we get the client 2025-04-03 13:24:32 -04:00
dekzter
c5e0de5d48 logos, channel numbers, websocket regex test so we can properly test against python regex and not javascript 2025-04-02 16:27:28 -04:00
dekzter
2995d2c456 added centralized task locking 2025-03-31 10:03:49 -04:00
SergeantPanda
3c3961bb3a Bypass redis for management commands. 2025-03-22 12:23:54 -05:00
SergeantPanda
071efaf017 Lowered max retries and timeout for redis client. 2025-03-22 11:36:28 -05:00
SergeantPanda
d622c96aba Improved connection handling for redis pubsub. 2025-03-22 08:48:39 -05:00
SergeantPanda
efaa7f7195 Singular redis-client. 2025-03-21 10:55:13 -05:00
SergeantPanda
fdcd3f2dbb Added error checking to global redis. 2025-03-14 15:42:28 -05:00
dekzter
05d9ea30e2 new migrations, setting proxy and redirect profiles, setting locked profiles 2025-03-14 16:12:37 -04:00