Commit graph

83 commits

Author SHA1 Message Date
SergeantPanda
7989fa3673 fix(proxy): enhance database connection management during VOD playback and stats updates
- Added calls to `close_old_connections()` in `stream_vod()` and `build_vod_stats_data()` to prevent connection leaks during long-lived streaming responses and background stats refreshes.
- Improved connection handling in various components to ensure proper resource cleanup and prevent blocking issues.
2026-06-15 17:32:38 -05:00
SergeantPanda
843940f552 refactor(http headers): centralize User-Agent construction and streamline API requests
- Introduced `dispatcharr_user_agent`, `dispatcharr_dvr_user_agent`, and `dispatcharr_http_headers` functions in `core.utils` to standardize User-Agent strings and HTTP headers for outbound requests.
- Updated various components, including `LogoViewSet`, `EPGSourceViewSet`, and VOD proxy views, to utilize the new header functions, ensuring consistent User-Agent usage across the application.
- Enhanced the handling of Schedules Direct API requests by including proper User-Agent headers, addressing previous API compliance issues.
2026-06-11 09:45:35 -05:00
SergeantPanda
705041ea81 refactor(channel_status, vod_proxy): remove stream URL credential extraction
- Eliminated redundant extraction of provider usernames from stream URLs in both ChannelStatus and VOD stats data functions, simplifying the codebase.
- Updated frontend components to reflect the removal of provider username display, streamlining the UI.
- Enhanced overall readability and maintainability of the code by reducing complexity in handling stream URLs.
2026-06-09 14:46:26 -05:00
SergeantPanda
bc046b3c92 refactor(m3u, channels): streamline stream assignment and server group management
- Renamed `_clear_stream_assignment_keys` to `_release_stale_stream_assignment` for clarity and updated its logic to release pool counters.
- Introduced new functions for managing credential slots during profile switches, enhancing the handling of shared connection limits across server groups.
- Removed the `max_streams` field from the `ServerGroup` model and updated related components to reflect this change, simplifying the server group management.
- Updated frontend components to integrate server group management, allowing for dynamic creation and editing of server groups.
- Enhanced error handling in stream URL generation to provide more informative feedback on connection issues.
- Added tests for stale assignment release and credential management during profile switches.
2026-06-09 13:40:16 -05:00
SergeantPanda
0f0e855507 Merge branch 'dev' of https://github.com/Dispatcharr/Dispatcharr into pr/Goldenfreddy0703/1254 2026-06-09 11:16:39 -05:00
Goldenfreddy0703
abdf2d7864 Fix credential release on deleted profile and round 3 review items.
Store credential Redis keys at reserve so release works when the profile row is deleted. Return reserve failure reasons to avoid fingerprint DB queries on logging paths. Document unlimited profile bypass in pool logic and Server Group UI.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-04 19:51:20 -04:00
R3XCHRIS
ff60bbec1a fix(vod_proxy): fall back to stream_id when content UUID is dead
Issue: When `process_movie_batch` / `process_series_batch`
(apps/vod/tasks.py) creates duplicate vod_movie / vod_episode records
during a refresh, existing M3U*Relation rows get repointed to the new
records. The old UUIDs that external players (Emby / Jellyfin /
ChannelsDVR) cached in `.strm` URLs are left orphaned, and the proxy
404s on every subsequent request — even though the same request
already carries a stable `stream_id` that uniquely identifies a live
relation.

This patch makes `_get_content_and_relation` use a soft UUID lookup
and, if it misses, fall back to resolving content via
`M3UMovieRelation.stream_id` / `M3UEpisodeRelation.stream_id`. The
fallback respects the strictest match first: when
`preferred_m3u_account_id` is also present, that account is queried
first; otherwise the highest-priority active relation matching the
stream_id wins (same ordering the existing relation-selection logic
uses).

This is read-side graceful degradation, complementary to the
write-side root fix proposed in closed #973 (re-use the relation's
existing movie/series during refresh rather than rematching by TMDB
ID). The two PRs together would fully resolve #961: #973 prevents new
orphaning, this PR makes any URL that ships a stream_id survive past
orphaning. URLs without a stream_id (e.g. XC-compat
`/movie/<user>/<pass>/<id>.mkv`) are NOT covered by this patch — they
need the root fix.

Series-UUID lookups (`/proxy/vod/series/<UUID>`) are left UUID-only by
design — players cache episode and movie URLs, not series URLs. Same
pattern can be added as a follow-up if needed.

Tests: 7 new mock-based regression tests covering both branches,
verified against `:dev`. UUID-first happy path is unchanged
(M3U*Relation table never queried when the UUID resolves).

Related: #961 (open), closed #973 (root fix abandoned for inactivity).
2026-06-02 19:29:58 +01:00
SergeantPanda
450b384419 fix(vod_proxy): Fix stream_vod to handle session_id in path for VOD proxy URLs 2026-05-30 20:48:17 -05:00
SergeantPanda
72d1520400 feat(auth): add QueryParamJWTAuthentication for token retrieval via query parameters
feat(vod): include token in stream URL for episode and movie requests (Fixes #1224)
2026-05-30 11:14:59 -05:00
Goldenfreddy0703
bb9b95b4a4 Refactor connection pools to manual ServerGroups with profile rotation.
Drop auto-fingerprint migration and restore per-profile selection for live/VOD.
Enforce shared limits on reserve using login-scoped group counters, and add
Server Groups UI for manual account assignment per maintainer feedback (#1137).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-23 20:18:45 -04:00
Goldenfreddy0703
d89f634213 Enforce shared credential connection pools across M3U accounts (#1137)
Group M3U/XC accounts and profiles that share the same provider login into auto-assigned ServerGroups keyed by credential fingerprint. Enforce combined Redis limits for live TV and VOD via apps/m3u/connection_pool.py.

- Per-profile fingerprinting (XC transforms and STD stream URLs)

- VOD profile selection tries alternates when default credential pool is full (fixes live then VOD failure)

- Stats UI shows provider login from active stream URL

- Tests: apps/m3u/tests/test_connection_pool.py (11 tests, all passing)

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-20 03:30:48 -04:00
Seth Van Niekerk
0d3eed81ba
Code review changes 2026-05-02 16:59:50 -04:00
Seth Van Niekerk
a01814069a
code review & pivot 2026-05-01 09:10:37 -04:00
Seth Van Niekerk
8894d51877
restrict XC access per user by configurable IP allowlist 2026-04-28 16:35:32 -04:00
SergeantPanda
29739ede36 Enhancement: VOD start/stop notifications and system events.
Some checks are pending
CI Pipeline / prepare (push) Waiting to run
CI Pipeline / docker (amd64, ubuntu-24.04) (push) Blocked by required conditions
CI Pipeline / docker (arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
CI Pipeline / create-manifest (push) Blocked by required conditions
Build and Push Multi-Arch Docker Image / build-and-push (push) Waiting to run
Frontend Tests / test (push) Waiting to run
2026-04-23 18:03:02 -05:00
SergeantPanda
f98bb183da Bug Fix: Fixed a provider TCP connection leak in the VOD proxy 2026-04-13 19:33:22 -05:00
SergeantPanda
55048b60e3 fix(vod-proxy): prevent double profile decrement and clean up stream counter naming 2026-04-13 18:40:33 -05:00
SergeantPanda
d73071b20f Merge branch 'dev' of https://github.com/Dispatcharr/Dispatcharr into pr/firestaerter3/1125 2026-04-13 17:42:43 -05:00
SergeantPanda
c3fb9c0073 security: Fixed missing network_access_allowed checks in the VOD proxy. stream_vod, head_vod, stream_xc_movie, and stream_xc_episode were not checking the STREAMS network policy, unlike the equivalent TS proxy endpoints. 2026-04-10 11:09:27 -05:00
SergeantPanda
fa9a7868ff security: proxy streaming endpoints (stream_ts, stream_xc, stream_vod, head_vod, stream_xc_movie, stream_xc_episode) use @permission_classes([AllowAny]) (access is controlled by the per-stream-type network allow-list inside the view body); the UserAgentViewSet, StreamProfileViewSet, CoreSettingsViewSet, and ProxySettingsViewSet gained get_permissions() methods mapping read actions to IsStandardUser and write actions to IsAdmin; and AuthViewSet.logout was updated to return [Authenticated()]. 2026-04-10 10:47:50 -05:00
SergeantPanda
dcefc6541c chore(cleanup): - Removed dead VODConnectionManager class (apps/proxy/vod_proxy/connection_manager.py) and its associated helpers, which had been superseded by MultiWorkerVODConnectionManager. All active code already used the multi-worker implementation. Removed the unused VODConnectionManager import from vod_proxy/views.py, the unscheduled cleanup_vod_connections task from apps/proxy/tasks.py, and the unscheduled cleanup_vod_persistent_connections task from core/tasks.py.
Some checks are pending
CI Pipeline / prepare (push) Waiting to run
CI Pipeline / docker (amd64, ubuntu-24.04) (push) Blocked by required conditions
CI Pipeline / docker (arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
CI Pipeline / create-manifest (push) Blocked by required conditions
Build and Push Multi-Arch Docker Image / build-and-push (push) Waiting to run
Frontend Tests / test (push) Waiting to run
- Removed dead VOD URL routes: `VODPlaylistView` (playlist generation), `VODPositionView` (position tracking), and the class-based `VODStatsView` (replaced by the existing function-based `vod_stats` view).
- Removed dead `updateVODPosition()` API method from `frontend/src/api.js`, which called the now-removed position tracking endpoint.
2026-04-09 19:59:50 -05:00
None
763f42cfff fix: pass TLS parameters to all external connection points in modular mode
- Add setup_pg_ssl_env() to export libpq PGSSL* env vars; all child psql/pg_dump/pg_isready commands inherit TLS automatically

- Move TLS client key permission fix before external PG connections (was running after, defeating the fix)

- Enhance key fix to trigger on ownership mismatch (root-owned 0600 key unreadable by app user)

- Extract key fix to shared script (docker/init/00-fix-pg-ssl-key.sh) sourced by both entrypoints

- Default POSTGRES_PASSWORD to empty in modular mode (cert-only auth sends no spurious password)

- Propagate TLS OPTIONS to backup pg_dump/pg_restore subprocess calls via _get_pg_env()

- Pass SSL kwargs to dropdb management command's psycopg2.connect()

- Add REDIS_SSL_PARAMS to VOD proxy Redis connections missed in original TLS PR

- Add 8-scenario TLS integration test suite (PG mTLS, Redis TLS, verify-full, key fix, Celery, regression)

- Add 6 unit tests for _get_pg_env() and dropdb TLS parameter passing
2026-03-30 20:19:27 -05:00
SergeantPanda
086cc74959 Bug Fix: M3U profile URL rewriting now uses the regex module instead of re across all URL transform code paths (url_utils.transform_url, core/views.py, vod_proxy/_transform_url, tasks.get_transformed_credentials, and the WebSocket live-preview handler in consumers.py). The regex module natively accepts JavaScript/PCRE-style named capture groups ((?<name>...)) without any conversion, eliminating the root cause of patterns that matched in the frontend live preview but failed on the backend with a re.error. As a further improvement, regex also supports variable-length lookbehind assertions (e.g. (?<=a+)), which re rejects with an error; patterns using these will now work correctly on the backend as well. Replace-pattern JS tokens are still normalised before calling regex.sub: $<name>\g<name> and $1/$2/… → \1/\2/… (Python replacement syntax). Also fixed a bug in the WebSocket preview handler where a pattern error was incorrectly returning the search pattern string as the preview output instead of the original URL. (Fixes #1005)
Some checks failed
CI Pipeline / prepare (push) Waiting to run
CI Pipeline / docker (amd64, ubuntu-24.04) (push) Blocked by required conditions
CI Pipeline / docker (arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
CI Pipeline / create-manifest (push) Blocked by required conditions
Build and Push Multi-Arch Docker Image / build-and-push (push) Waiting to run
Frontend Tests / test (push) Has been cancelled
2026-03-29 17:55:08 -05:00
SergeantPanda
38f5156ca6 Enhancement: Show username in VOD cards. 2026-03-28 16:46:06 -05:00
SergeantPanda
aa6fb033d3 fix: update user stream limit checks to include media_id and rename user_limits_settings key 2026-03-26 20:02:54 -05:00
dekzter
a19be96ced fleshed out user limits and termination logic 2026-03-25 17:33:26 -04:00
firestaerter3
5ad4824c4b fix(vod-proxy): eliminate profile_connections counter stuck-at-nonzero races
Three race conditions in multi_worker_connection_manager could leave
the Redis profile_connections:<id> counter permanently elevated with no
active streams, causing all VOD requests to 503 "All profiles at capacity".

Bug 1 — decrement_active_streams() return value was ignored
All three generator exit paths (normal, GeneratorExit, Exception) called
decrement_active_streams() and unconditionally set decremented = True
regardless of whether the lock was acquired. On lock contention the decrement
was silently skipped, active_streams remained > 0, the subsequent
has_active_streams() check returned True, and _decrement_profile_connections()
was never called. The counter was then stuck until manual DEL.

Fix: add decrement_active_streams_and_check() which performs the decrement and
the "are there remaining streams?" check atomically under a single lock,
eliminating the race window. All three exit paths and the finally block now
use this method and propagate its success/remaining return values.

Bug 2 — non-atomic GET-then-DECR in _decrement_profile_connections()
The previous implementation read the counter with GET then conditionally
called DECR. Two concurrent decrements could both pass the > 0 guard and
both fire, driving the counter to -1. A subsequent _check_and_reserve_profile_slot()
INCR would then produce 0 which passes the <= max_streams check, allowing
an extra stream to bypass the limit on the next request.

Fix: replace GET-then-DECR with a direct DECR (matching the INCR-first
pattern already used by _check_and_reserve_profile_slot) and clamp the
result to 0 if it goes negative.

Bug 3 — has_active_streams() read state without holding the lock
The separate has_active_streams() call after decrement_active_streams()
released its lock left a window where another concurrent stream could
increment active_streams back to 1, causing the profile decrement to be
skipped. This is resolved as a consequence of Bug 1's fix: the new
decrement_active_streams_and_check() method reads active_streams while
the lock is still held, eliminating the window entirely.

Adds tests covering all three scenarios in
apps/proxy/vod_proxy/tests/test_profile_connections.py.

Fixes #1121.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-19 23:23:36 +01:00
dekzter
f69a462253 merged in dev 2026-03-13 08:22:44 -04:00
None
0e7eb916c2 Merge remote-tracking branch 'upstream/dev' into fix/947-Connection-capacity-leak-failed-stream-initialization-permanently-consumes-connection-slot 2026-02-27 20:59:30 -06:00
dekzter
6f4665e6eb merged in dev 2026-02-26 08:20:20 -05:00
None
58ccf2d6e1 Fix: VOD proxy connection counter leaks on client disconnect and seeking
Fix multiple VOD proxy connection counter leak paths that caused
profile_connections to remain permanently elevated after client
disconnects, leaving profiles locked at capacity.

Changes:
- Fix get_stream() race condition: save state under lock with fresh active_streams to prevent overwriting concurrent decrements during scrubbing/seeking (root cause of counter stuck after timeshift)
- Add early active_streams increment in session reuse path to prevent cleanup race between GeneratorExit and new request
- Add rollback on get_stream() returning None (range not satisfiable)
- Add profile_id fallback to closure variable when Redis state is deleted before generator cleanup runs
- Change increment/decrement return types and add warning logs for lock acquisition and state failures
- Clean up debug traceback logging from increment/decrement methods
2026-02-17 20:24:00 -06:00
None
4842bb87fa Fix: VOD proxy connection counter leak on client disconnect
Three fixes:
- TOCTOU: Replace GET-check-INCR with atomic INCR-first-then-check pattern in both connection managers to prevent concurrent requests exceeding max_streams
- Immediate DECR: Decrement profile counter directly in stream generator exit paths (normal completion, client disconnect, error) instead of deferring to daemon thread cleanup which may never execute
- Rollback: Decrement profile counter on create_connection() failure so the reserved slot is released

Fixes #962
2026-02-13 19:37:06 -06:00
None
cf96287c90 Add guard to remove_connection()
- multi_worker_connection_manager.py: Add missing count > 0 guard to multi-worker remove_connection()

This method doesn't appear to be called anywhere but added the guard in case it is put back into service
2026-02-10 19:06:51 -06:00
None
9e8f227f49 Fix: VOD proxy double-DECR of profile connection counter during cleanup
- multi_worker_connection_manager.py: Error handler at line 1042: removed connection_manager=self from cleanup() call since the DECR was already done at line 1036. Prevents double-DECR.

- connection_manager.py: Added connection_key_exists check before calling remove_connection(). The fallback DECR from session data now only fires when the connection tracking key has already expired. Prevents double-DECR.
2026-02-10 18:58:17 -06:00
None
e217960500 feat: Add Redis authentication support for modular deployment
Add support for authentication when connecting to external Redis instances in modular deployment mode. This enables secure Redis deployments using either password-only authentication (Redis <6) or
username + password authentication (Redis 6+ ACL).

Changes:
- Add REDIS_PASSWORD and REDIS_USER environment variables
- Implement URL encoding for special characters in passwords
- Update all Redis connection points to support auth
- Add comprehensive documentation and examples to docker-compose.yml
- Maintains full backward compatibility (empty defaults = no auth)

All authentication mechanisms have been fully tested including pasword-only authentication, Redis 6+ ACL authentication with username + password, volume-mounted configuration files, and special character handling.

Existing deployments are not effected, authentication support is entirely opt-in using docker-compose.

Also, acknowledging the fact that the docker-compose file for modular deployments has been getting out of hand with auth support, the docker-compose.yml file was re-formatted for better visibility in configuration. This seemed like a better way to go than mandating a .env file.
2026-02-03 21:42:06 -06:00
SergeantPanda
9612a67412 Change: VOD upstream read timeout reduced from 30 seconds to 10 seconds to minimize lock hold time when clients disconnect during connection phase
Some checks are pending
CI Pipeline / prepare (push) Waiting to run
CI Pipeline / docker (amd64, ubuntu-24.04) (push) Blocked by required conditions
CI Pipeline / docker (arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
CI Pipeline / create-manifest (push) Blocked by required conditions
Build and Push Multi-Arch Docker Image / build-and-push (push) Waiting to run
2026-01-04 15:21:22 -06:00
SergeantPanda
4e65ffd113 Bug fix: Fixed VOD profile connection count not being decremented when stream connection fails (timeout, 404, etc.), preventing profiles from reaching capacity limits and rejecting valid stream requests 2026-01-04 15:00:08 -06:00
SergeantPanda
de31826137 refactor: externalize Redis and Celery configuration via environment variables
Replace hardcoded localhost:6379 values throughout codebase with environment-based configuration. Add REDIS_PORT support and allow REDIS_URL override for external Redis services. Configure Celery broker/result backend to use Redis settings with environment variable overrides.

Closes #762
2025-12-18 16:54:59 -06:00
SergeantPanda
2558ea0b0b Enhancement: Add VOD client stop functionality to Stats page 2025-12-17 16:54:10 -06:00
dekzter
1618332962 byte to string fix 2025-11-12 17:52:11 -05:00
dekzter
50e9075bb5 initial run of a binary and encoded redis client - no more encoding / decoding data into redis, huge PITA (still some outstanding spots I need to patch) 2025-10-25 08:15:39 -04:00
SergeantPanda
70f7484fb5 Better connection tracking for apps that do not reuse connections during seeking operations. 2025-09-26 09:18:43 -05:00
SergeantPanda
7bb4df78c8 Enhancement: Update M3U profile retrieval to include current connection counts and session handling during vod session start. 2025-09-26 09:18:42 -05:00
SergeantPanda
d961d4cad1 Bug fix: VOD will now select the correct M3U profile while starting.Fixes #461 2025-09-26 09:18:22 -05:00
SergeantPanda
c0ddec6b4b Reduced cleanup time on vod from 10 seconds to 1. 2025-09-09 13:06:40 -05:00
SergeantPanda
64e500c524 Add logging for profile connection cleanup and implement delayed cleanup for idle Redis connections 2025-09-09 12:58:35 -05:00
SergeantPanda
0938a3c592 Better headers for the client including content range that includes total length as well. 2025-09-07 18:51:58 -05:00
SergeantPanda
b45c6eda38 Better content-type detection 2025-09-07 18:25:39 -05:00
SergeantPanda
be9823c5ce Added debug logs for active clients 2025-09-07 17:14:29 -05:00
SergeantPanda
adf960753c Changed user-agent for head request to use m3u account 2025-09-07 15:17:19 -05:00