This update modifies permission classes in multiple viewsets to restrict access based on user roles. The `IsAdmin` permission is now enforced for several actions, including group management and permission listing, ensuring that only administrators can perform sensitive operations. Additionally, a new utility function, `resolve_safe_local_data_path`, is introduced to enhance security when accessing local file paths. The changes improve overall security and maintainability of the codebase.
This update introduces a new `is_catchup_enabled` function to determine if catch-up is allowed for users based on their custom properties and system settings. The `UserViewSet` is modified to restrict admin-managed properties, including catch-up access. Additionally, various views and tests are updated to incorporate catch-up checks, ensuring that users without access receive appropriate error responses. The frontend is enhanced with a catch-up toggle in user and system settings forms, allowing for better management of catch-up capabilities.
This update introduces Redis caching for grouped settings in the CoreSettings model, improving performance by reducing database queries. A new method, `get_network_access_settings`, is added to streamline access to network settings. Additionally, the `setting_flag_enabled` function is introduced to handle boolean settings more robustly. The cache is invalidated on save and delete operations, ensuring data consistency. Tests have been added to validate the caching behavior and the new settings functionality.
- Improved multi-provider failover by preferring streams with sufficient catch-up days.
- Added new helper functions for calculating programme age and ordering catch-up streams.
- Removed deprecated XMLTV settings from the frontend and backend.
- Updated tests to validate new stream ordering logic and catch-up functionality.
DB sessions were never actually pinned to UTC. Three stacked failures:
1. The connection_created receiver (_force_utc0 in core/apps.py) was a
nested closure connected with Django's default weak reference. It was
garbage-collected as soon as CoreConfig.ready() returned, so
SET TIME ZONE 'UTC0' fired into a dead weakref and never executed -
in any version since it landed (verified: the signal's receivers list
on a live 0.27.2 shows a dead weakref).
2. Sessions were UTC on older stacks anyway because Django's own
init_connection_state configured the timezone. Since native
psycopg-pool support (Django 5.1+), that path is gated on
`not self.pool` - and the geventpool mixin's `pool` property is
always truthy, so Django silently skips timezone (and role)
configuration for this backend on every connection. This is what
actually regressed at the psycopg2->3 / Django upgrade: the masking
layer disappeared, not the (already dead) signal.
3. psycopg3 logs "unknown PostgreSQL timezone: 'UTC0'" because the
POSIX spec is unresolvable in Python zoneinfo (cosmetic, but it
means 'UTC0' buys nothing on psycopg3).
Net effect: every session ran at the server-default timezone (verified
live: SHOW TimeZone through the pool returns 'Etc/UTC'). Deployments
whose Postgres default is non-UTC (e.g. /etc/localtime bind-mounts,
the original issue 651 report) get the EPG offset corruption back.
Fix: pin the GUC in the libpq startup packet in the pool backend's
get_connection_params():
- covers every connection the gevent pool creates; no signal, GC, or
Django-flow dependency (same pattern as the client_encoding pin the
pool already applies)
- startup-packet GUCs are the session default: they survive ROLLBACK,
and RESET TimeZone returns to UTC rather than the server default
- 'UTC' resolves cleanly in psycopg3's zoneinfo lookup, and PostgreSQL
resolves it against its own bundled tzdata, not host-mounted files
The dead signal is removed. Regression tests exercise the session
timezone through the real pool backend explicitly (the test runner's
default engine is the vanilla Django backend), and assert the
RESET-returns-to-UTC session-default property.
Tested: live 0.27.2 AIO (psycopg 3.3.4, PostgreSQL 17) - unpatched
backend+pool session shows 'Etc/UTC'; patched shows 'UTC' incl. after
RESET and ROLLBACK. Full A/B against the dev image with the server
default forced to Europe/Zurich: stock renders a 12:00Z source
programme as start="20260715140000 +0000" in /output/epg; patched
renders it correctly; the regression tests fail on stock and pass
when patched.
Add end-to-end catch-up support for XC clients and native apps: provider
proxy with failover and per-viewer session pooling, REST session minting
for tokenless playback URLs, catch-up admin stats, combined connection
stats, and Stats UI with dedicated cards plus websocket updates.
Includes Redis namespace consolidation under timeshift:* (dropping legacy
timeshift_ id prefixes), dedicated catch-up stop by session_id, and
cleaner channel/client metadata split for stats keys.
Closes#133
Enhanced the `/output/epg` cold rebuild process to prevent freezing of the gevent uWSGI worker. The `_stream_build` function now yields control to the gevent hub after processing each cached chunk, allowing other requests to be handled concurrently. This change improves responsiveness during CPU-bound XMLTV generation. Additionally, introduced a new utility function, `_cooperative_yield`, to facilitate yielding in CPU-bound loops.
Modified the file type validation in scan_and_process_files() to recognize .xz files alongside .xml, .gz, and .zip. Updated logging messages to reflect the inclusion of .xz in the skipped file notifications.
- Moved the `programme_index` from the `EPGSource` model to a new `EPGSourceIndex` table, ensuring that the large JSON blob is only loaded when explicitly accessed, thus improving query performance and memory efficiency.
- Updated related queries and API views to utilize the new structure, including adjustments to EPG generation and import logic to prevent unnecessary data loading.
- Enhanced memory management in the EPG grid endpoint to reduce worker RSS during response handling.
- Streamlined `generate_epg()` to incrementally stream EPG data without loading the entire guide, improving memory efficiency.
- Reduced database load by deferring the fetching of large `programme_index` blobs, enhancing response times for EPG generation.
- Introduced a composite index on `(epg_id, id)` in `ProgramData` to optimize query performance during EPG exports.
- Updated tests to ensure proper functionality and performance of new features.
This PR updates the non_fqdn_pattern regex to allow underscores in non-FQDN hostnames. Such hostnames are commonly used in Docker networks, but Dispatcharr currently rejects them due to the overly strict regex.
- Implemented `ensure_custom_properties_dict()` to normalize custom properties across various models and serializers, addressing issues with legacy JSON-encoded strings.
- Updated M3U account and channel group models to ensure custom properties are consistently stored as dictionaries during save operations.
- Enhanced Celery task management by ensuring old DB connections are closed before and after tasks, improving reliability and preventing errors during account refresh operations. (Fixes#1338)
- Implemented `normalize_server_url()` to standardize account server URLs, ensuring that on-demand live URLs are built correctly without including API endpoints or query parameters.
- Updated `get_transformed_credentials()` and stream URL generation in `M3UMovieRelation` and `M3UEpisodeRelation` to utilize the new normalization function, improving URL handling for Xtream Codes accounts.
- Improved database connection management by ensuring `close_old_connections()` is called in various methods to prevent connection leaks.
- Updated event dispatching to run asynchronously in gevent, preventing blocking during live-proxy and streaming paths.
- Replaced individual EPG program parse tasks with a centralized dispatch function to streamline guide refresh for newly assigned EPG IDs.
- Implemented batching for guide fetches when multiple EPGs are mapped, reducing redundant API calls and improving efficiency.
- Updated related utility functions to support the new fetching strategy and added tests to ensure correct behavior under various scenarios.
Implements all four points from the latest review, plus hardening from a
pre-submission audit pass.
1. Access control: timeshift_proxy now enforces
network_access_allowed(request, "STREAMS", user) — same key and placement
as the live XC stream endpoint.
2. Catch-up failover: the proxy walks the channel's catch-up streams in
channelstream order (get_channel_catchup_streams), mirroring live
playback. Each attempt carries its own provider context: account
credentials, provider stream id, reported server_info timezone (the
UTC->provider conversion is recomputed per attempt), user-agent, and the
per-account URL-format cache. The first streamable response wins; if all
providers fail the last failure is returned.
Ban-safety is per account: a decisive auth/ban-class failure (401/403/406)
marks the account and skips its remaining streams (e.g. FHD/HD variants of
the same channel) instead of hammering a banning provider, while other
accounts — different hosts — are still tried. Streams from disabled M3U
accounts are excluded, same as live dispatch. Redirects stay enabled on
purpose (XC providers legitimately 302 to load-balanced streaming nodes);
the 3xx decisive branch is kept as defense-in-depth and documented as such.
3. apps/proxy/live_proxy/views.py restored byte-identical to upstream — the
leftover channel-id wrapper from the removed provider-stream-id fallback
is gone (zero-line diff).
4. Single remaining setting relocated: xmltv_prev_days_override now lives in
proxy_settings (backend default in get_proxy_settings, consumed by the
XMLTV prev_days resolution). The timeshift_settings group,
TIMESHIFT_DEFAULTS, get_timeshift_settings, the Settings → Timeshift form
and tab are all removed; the field appears under Settings → Proxy Settings
(0 = auto-detect, capped at 30).
Audit hardening in the same pass:
- Updated the proxy-settings defaults unit test for the new key (would have
failed CI otherwise).
- Migration backfills use schema_editor.connection instead of the global
connection (multi-database correctness).
- CHANGELOG and module docstring brought in line with the final architecture
(PATH-first cascade, failover, setting under Proxy Settings).
- Tests grown to 69 backend tests: failover success/exhaustion/skip
semantics, decisive-account skip vs soft-failure retry, per-stream
timezone conversion (different zones per provider), 406/connection-error
cascade paths, stream-limit and no-eligible-stream outcomes,
network-gate 403, server_info strict-UTC guarantee, EPG duration window
resolution, and DB-backed coverage of xc_password auth, user_level access
and the failover stream ordering (catch-up-only, active accounts,
channelstream order). The format-cache test now runs on an isolated
locmem cache.
- Introduced `dispatcharr_user_agent`, `dispatcharr_dvr_user_agent`, and `dispatcharr_http_headers` functions in `core.utils` to standardize User-Agent strings and HTTP headers for outbound requests.
- Updated various components, including `LogoViewSet`, `EPGSourceViewSet`, and VOD proxy views, to utilize the new header functions, ensuring consistent User-Agent usage across the application.
- Enhanced the handling of Schedules Direct API requests by including proper User-Agent headers, addressing previous API compliance issues.
Two coupled fixes that make catch-up play the requested programme:
1. Try the PATH catch-up form (/timeshift/.../{start}/{id}.ts) BEFORE the
timeshift.php query form. Empirical testing showed some XC providers
return the LIVE stream on the query form (HTTP 200, silently ignoring
`start`) — a valid MPEG-TS indistinguishable from a real archive, so the
cascade accepted it and the user always got live content. The PATH form
actually seeks. Candidate ordering now lives in
build_timeshift_candidate_urls() with the full rationale.
2. Strict-UTC XC API surface + single proxy-time timezone conversion:
- xc_get_epg start/end always UTC, server_info.timezone always "UTC",
time_now UTC — the timezone triple is consistently UTC.
- Removed the global default_timezone setting and the
_convert_xmltv_to_local_timezone rewrite entirely.
- The proxy converts the client's UTC timestamp to the SERVING provider's
own zone (server_info.timezone captured on account refresh, read from
the account's default profile) right before building the upstream URL —
DST-correct via ZoneInfo, no-op for UTC/unknown zones. Verified against
a real provider: it interprets the URL timestamp as its LOCAL wall
clock, so pure UTC pass-through seeks 1-2h off.
- EPG duration lookup keeps the ORIGINAL UTC timestamp (programmes are
stored in UTC) — exactly one conversion in the whole chain.
Also per review feedback:
- Removed the debug_logging toggle — verbose timeshift logging now follows
the standard logger DEBUG level (DISPATCHARR_LOG_LEVEL).
- Removed the dead default_language setting (never read anywhere; the EPG
lang field is hardcoded). TIMESHIFT_DEFAULTS is down to
xmltv_prev_days_override, and the Settings UI shows that single field
with accurate help text.
- Validate the timestamp up front (400 on malformed input) instead of
forwarding it verbatim into the upstream URL.
- Redact the upstream URL in the connection-error log (requests exceptions
embed the full URL, which carries XC credentials).
- URL-encode XC credentials in both URL builders.
- Request identity encoding upstream (the TS-sync peek reads raw bytes).
- Poll the stop key on the 5-second heartbeat cadence instead of every 100
chunks (~25 MB), so stream-limit terminations free the provider slot fast.
- New tests: proxy timestamp wiring (converted value reaches the URL
builder, original UTC value reaches the duration lookup), _redact_url,
decisive 3xx break (anti ban), invalid timestamp rejection; deflaked the
format-cache promotion test (django cache persists across runs).
Remove the _sd_send_ws_sync function and replace its usage with send_epg_update in the fetch_schedules_direct function. This change simplifies the code by ensuring all WebSocket updates are sent through a single function, improving maintainability and consistency in handling EPG updates.
- Fix socket leak in local IP detection using try/finally; replace 8.8.8.8
with RFC 5737 test address 203.0.113.1
- Validate ipify response with ipaddress.ip_address() before using in URL
- Push ip_lookup_complete WebSocket event when background lookup finishes,
eliminating frontend polling entirely
- Show Skeleton placeholder in sidebar while IP lookup is pending
- Replace hover-to-reveal blur with click-to-toggle for mobile support
- Fix copy button propagating click to blur toggle via stopPropagation
- Add missing city field to null-env fallback in settings store
- Add setEnvironmentFields() store action for WebSocket-driven updates
- Remove unused imports from api_views.py
Adds native catch-up/timeshift replay for Xtream Codes providers through
the same HTTPStreamReader transport pipeline as live TV.
Timeshift proxy (apps/timeshift/):
- URL cascade: 3 candidate timestamp formats per provider, per-account
format cache for fast-forward seek performance
- MPEG-TS preamble stripping (shared with HTTPStreamReader)
- Stats integration: timeshift viewers appear on /stats with TIMESHIFT badge
- Auth via hmac.compare_digest on XC password
Catchup detection — denormalized for zero-cost output queries:
- Stream.is_catchup + Stream.catchup_days populated at XC import time
- Channel.has_catchup + Channel.catchup_days + Channel.catchup_provider_stream_id
rolled up via ChannelStream post_save signal (UI path) and explicit SQL
after bulk_create (import path)
- _xc_channel_entry() reads denormalized fields instead of per-channel
custom_properties JSON introspection (eliminates N+1 queries)
- Migration 0038 backfills existing data via raw SQL
XC API enhancements:
- server_info.timezone + start/end + time_now use configured timezone
(triple consistency rule — fixes wrong-programme-plays bug)
- Dynamic has_archive flag + auto prev_days for catch-up channels
- XMLTV timestamps rewritten to local timezone for catch-up clients
HTTPStreamReader extended (apps/proxy/live_proxy/input/http_streamer.py):
- 1 MB pipe buffer via fcntl F_SETPIPE_SZ (eliminates producer/consumer
ping-pong that halved throughput)
- Pre-opened response= for URL cascade workflows
- strip_ts_preamble= for XC servers emitting PHP warnings before TS
- find_ts_sync() as shared utility
- Builds on upstream O_NONBLOCK + select() write loop
Provider stream_id lookup order:
- stream_xc() and xc_get_epg() try internal Channel.id first, fall back
to provider stream_id only when needed (avoids unconditional query on
every request)
Also includes:
- VOD provider cascade in stream_vod() — iterates all M3U relations by
priority when first provider is at capacity
- Defensive null-safety: custom_sid: None → "" in get_live_streams,
get_vod_streams, get_vod_info, get_series_info (fixes iPlayTV crash on
JSON null for string fields)
- Timeshift settings UI (timezone selector, debug toggle)
- StreamConnectionCard violet TIMESHIFT badge
- Orphan cleanup skips timeshift_* virtual channels
