This update introduces a new `is_catchup_enabled` function to determine if catch-up is allowed for users based on their custom properties and system settings. The `UserViewSet` is modified to restrict admin-managed properties, including catch-up access. Additionally, various views and tests are updated to incorporate catch-up checks, ensuring that users without access receive appropriate error responses. The frontend is enhanced with a catch-up toggle in user and system settings forms, allowing for better management of catch-up capabilities.
This update introduces a new helper function `_is_full_restart_range` to determine when a request indicates a full restart from byte 0. The existing `_should_preempt_plain_reconnect` function has been modified to utilize this new logic, ensuring that plain GET requests and open-ended byte ranges are handled correctly. Additionally, new tests have been added to validate the behavior of these functions, confirming that they correctly identify restart scenarios and preempt playback as expected.
- Introduced a new endpoint `POST /api/catchup/sessions/<session_id>/position/` for native clients to report their playhead position and pause state during catch-up sessions.
- Updated the catch-up session handling to include a `paused` flag, allowing accurate tracking of playback state without seeking the provider stream.
- Enhanced the Redis storage mechanism to accommodate the new position and pause data, ensuring real-time updates for admin stats.
- Added tests to validate the new position reporting functionality and its integration with existing catch-up features.
- Introduced an optional `duration` parameter for catch-up sessions, allowing clients to specify the programme length in minutes. This value is preferred over EPG-derived durations and includes a buffer for provider lag.
- Updated the API views and serializers to accept and process the new `duration` field.
- Enhanced the catch-up proxy implementation to utilize the client-supplied duration, improving playback accuracy.
- Added tests to validate the new duration handling and ensure proper integration with existing functionality.
- Improved multi-provider failover by preferring streams with sufficient catch-up days.
- Added new helper functions for calculating programme age and ordering catch-up streams.
- Removed deprecated XMLTV settings from the frontend and backend.
- Updated tests to validate new stream ordering logic and catch-up functionality.
Updated the catch-up timestamp normalization to reflect XC client specifications. Introduced a new mechanism to handle near-EOF duration probes, ensuring that playback statistics are preserved without reanchoring to the end of the file. Additionally, improved the handling of presentation lengths in the timeshift proxy to align with XC behavior. Updated tests to validate these changes and ensure robust playback functionality.
Clients that build catch-up requests in QUERY layout (e.g. Open-TV /
Fred TV: /streaming/timeshift.php?username=...&stream=...&start=...)
had no matching urlpattern, so the request silently fell through to
the frontend's <path:unused_path> catch-all and got served index.html
(200 OK, wrong content) instead of reaching the proxy — no error, no
log line, the client just fails to play.
The PATH-style layout (timeshift/<user>/<pass>/<stream_id>/<ts>/<dur>)
already worked; QUERY-style autodetection already existed for outgoing
provider requests (helpers.build_timeshift_url_format_a/_b) but was
never mirrored to the incoming route.
Split timeshift_proxy into a shared _timeshift_proxy_impl plus two thin
entry points (PATH-style timeshift_proxy, new QUERY-style
timeshift_proxy_query) so both incoming layouts are recognized.
Reported against the predecessor plugin as dispatcharr_timeshift#10;
reproduced identically against dispatcharr:dev (confirmed via nginx
access log: a QUERY-style request returned 200 with a response body
exactly matching the size of frontend/dist/index.html).
Updated the session handling mechanism to improve user experience during reconnects. The first request without a `session_id` now receives a `301` redirect with a minted `session_id`, while reconnects that omit `session_id` but match an existing pool entry are served immediately without a redirect. Additionally, refined playback logic for plain GET requests to restart from byte 0, aligning with provider behavior. Updated tests to reflect these changes and ensure proper session reuse and playback anchoring.
Updated the session matching logic to improve handling of fresh sessions. Introduced a new parameter, `fresh_session`, to skip adopting idle exact-media pools when a new session ID is provided. This prevents unnecessary reconnections to previously abandoned programme slots. Added tests to verify that fresh sessions correctly skip idle matches while retaining busy exact-media sessions and allowing channel hops. This change enhances the robustness of the timeshift feature and improves user experience during session transitions.
Add end-to-end catch-up support for XC clients and native apps: provider
proxy with failover and per-viewer session pooling, REST session minting
for tokenless playback URLs, catch-up admin stats, combined connection
stats, and Stats UI with dedicated cards plus websocket updates.
Includes Redis namespace consolidation under timeshift:* (dropping legacy
timeshift_ id prefixes), dedicated catch-up stop by session_id, and
cleaner channel/client metadata split for stats keys.
Closes#133
Added a helper function to close old database connections in the `timeshift_proxy` view before returning any HTTP responses. This change prevents potential database connection leaks and ensures that connections are properly managed during the request lifecycle. Updated tests to verify that connections are closed appropriately in various response scenarios.
Also stores provider_tz_name with session pool in redis to avoid an extra database call.
Self-review follow-ups on the position fix:
- When the position actually moves, drop content_length/serving_range from
the pool entry — they describe the PREVIOUS position's file, and keeping
them would feed the near-EOF/displacement heuristics another programme's
size (a metadata probe near the new file's EOF could displace live
playback, or a genuine scrub could be misjudged as a probe). The next
successful open repopulates both.
- Guard the update under the pool lock and skip it when the entry has
vanished (Redis restart/eviction) — a bare HSET would resurrect a
partial, TTL-less hash that answers every later request for that
session_id with 503.
- Align the reuse-path timezone lookup with the fresh path
(is_active=True on the default-profile filter).
Three more tests: byte state dropped on move / kept on same-position
update, vanished entry never resurrected, tz fallback to the reserved
profile when no active default exists.
Clients (TiviMate) keep the ?session_id= query when they rebuild the seek
URL with a new start timestamp, so every timestamp-jump FF/RW landed on
the reused session's STORED provider_timestamp — playback snapped back to
the position the session was created for (the rewind anchor), 100%
reproducible: two requests with different start values on one session
returned byte-identical streams.
The reuse path now always recomputes the provider timestamp from the
REQUESTED one (the provider zone is a property of the account — read from
the default profile's server_info, same as the fresh path) and moves the
pool descriptor (media_id + provider_timestamp) to the position actually
served, so fingerprint matching and same-channel displacement keep
comparing against reality. Slot continuity is unchanged.
Regression tests: reused session serves the requested timestamp (unit),
descriptor follows the seek (unit), and an end-to-end timestamp-jump with
the same session_id reaches the new position through timeshift_proxy.
Aligns catch-up with how live and VOD manage provider capacity:
- Reserve a provider profile slot (connection_pool) before every upstream
connect, walking the account's active profiles default-first when the
default is at capacity (profile_full/credential_full are transient and
never mark the account decisive). Credentials for the reserved profile
are resolved via get_transformed_credentials — the same credential
extraction live playback uses — so pool accounting and real upstream
usage always agree. All eligible streams blocked on capacity alone
returns 503 (the VOD pool-exhausted precedent).
- Release exactly once via a one-shot Redis ownership token consumed with
a transactional GET+DEL: the generator finally, the response-close
wrapper, failed failover attempts and session takeover all share it, so
no path can double-decrement and a client disconnecting before the
first chunk still releases (Django registers the iterator's close() as
a resource closer). Failures between reservation and the streaming
response owning the slot release before propagating; an ownership-token
write failure releases directly and reports transient unavailability.
- One catch-up session per user and channel: a new request (programme
jump or seek) displaces the user's previous session on that channel —
its slot is released synchronously, its stats are unregistered, and its
generator is stopped through the standard stop-key mechanism — so rapid
seeking cannot stack upstream provider connections.
- rollup_channel_catchup_fields self-heals channels left flagged
is_catchup with no remaining catch-up stream (outside the
account-scoped CTE), covering bulk removals on manual/multi-provider
channels regardless of how the link rows disappeared. A regression test
locks the ChannelStream post_delete signal firing on queryset bulk
deletes.
Backend test suite grows to 94 (slot reservation/release on every
failover outcome, profile walk, mixed capacity-vs-upstream precedence,
exception-path release, token exactly-once semantics, takeover scoping
and ordering, never-started-generator release, rollup self-heal).
Implements all four points from the latest review, plus hardening from a
pre-submission audit pass.
1. Access control: timeshift_proxy now enforces
network_access_allowed(request, "STREAMS", user) — same key and placement
as the live XC stream endpoint.
2. Catch-up failover: the proxy walks the channel's catch-up streams in
channelstream order (get_channel_catchup_streams), mirroring live
playback. Each attempt carries its own provider context: account
credentials, provider stream id, reported server_info timezone (the
UTC->provider conversion is recomputed per attempt), user-agent, and the
per-account URL-format cache. The first streamable response wins; if all
providers fail the last failure is returned.
Ban-safety is per account: a decisive auth/ban-class failure (401/403/406)
marks the account and skips its remaining streams (e.g. FHD/HD variants of
the same channel) instead of hammering a banning provider, while other
accounts — different hosts — are still tried. Streams from disabled M3U
accounts are excluded, same as live dispatch. Redirects stay enabled on
purpose (XC providers legitimately 302 to load-balanced streaming nodes);
the 3xx decisive branch is kept as defense-in-depth and documented as such.
3. apps/proxy/live_proxy/views.py restored byte-identical to upstream — the
leftover channel-id wrapper from the removed provider-stream-id fallback
is gone (zero-line diff).
4. Single remaining setting relocated: xmltv_prev_days_override now lives in
proxy_settings (backend default in get_proxy_settings, consumed by the
XMLTV prev_days resolution). The timeshift_settings group,
TIMESHIFT_DEFAULTS, get_timeshift_settings, the Settings → Timeshift form
and tab are all removed; the field appears under Settings → Proxy Settings
(0 = auto-detect, capped at 30).
Audit hardening in the same pass:
- Updated the proxy-settings defaults unit test for the new key (would have
failed CI otherwise).
- Migration backfills use schema_editor.connection instead of the global
connection (multi-database correctness).
- CHANGELOG and module docstring brought in line with the final architecture
(PATH-first cascade, failover, setting under Proxy Settings).
- Tests grown to 69 backend tests: failover success/exhaustion/skip
semantics, decisive-account skip vs soft-failure retry, per-stream
timezone conversion (different zones per provider), 406/connection-error
cascade paths, stream-limit and no-eligible-stream outcomes,
network-gate 403, server_info strict-UTC guarantee, EPG duration window
resolution, and DB-backed coverage of xc_password auth, user_level access
and the failover stream ordering (catch-up-only, active accounts,
channelstream order). The format-cache test now runs on an isolated
locmem cache.
Two coupled fixes that make catch-up play the requested programme:
1. Try the PATH catch-up form (/timeshift/.../{start}/{id}.ts) BEFORE the
timeshift.php query form. Empirical testing showed some XC providers
return the LIVE stream on the query form (HTTP 200, silently ignoring
`start`) — a valid MPEG-TS indistinguishable from a real archive, so the
cascade accepted it and the user always got live content. The PATH form
actually seeks. Candidate ordering now lives in
build_timeshift_candidate_urls() with the full rationale.
2. Strict-UTC XC API surface + single proxy-time timezone conversion:
- xc_get_epg start/end always UTC, server_info.timezone always "UTC",
time_now UTC — the timezone triple is consistently UTC.
- Removed the global default_timezone setting and the
_convert_xmltv_to_local_timezone rewrite entirely.
- The proxy converts the client's UTC timestamp to the SERVING provider's
own zone (server_info.timezone captured on account refresh, read from
the account's default profile) right before building the upstream URL —
DST-correct via ZoneInfo, no-op for UTC/unknown zones. Verified against
a real provider: it interprets the URL timestamp as its LOCAL wall
clock, so pure UTC pass-through seeks 1-2h off.
- EPG duration lookup keeps the ORIGINAL UTC timestamp (programmes are
stored in UTC) — exactly one conversion in the whole chain.
Also per review feedback:
- Removed the debug_logging toggle — verbose timeshift logging now follows
the standard logger DEBUG level (DISPATCHARR_LOG_LEVEL).
- Removed the dead default_language setting (never read anywhere; the EPG
lang field is hardcoded). TIMESHIFT_DEFAULTS is down to
xmltv_prev_days_override, and the Settings UI shows that single field
with accurate help text.
- Validate the timestamp up front (400 on malformed input) instead of
forwarding it verbatim into the upstream URL.
- Redact the upstream URL in the connection-error log (requests exceptions
embed the full URL, which carries XC credentials).
- URL-encode XC credentials in both URL builders.
- Request identity encoding upstream (the TS-sync peek reads raw bytes).
- Poll the stop key on the 5-second heartbeat cadence instead of every 100
chunks (~25 MB), so stream-limit terminations free the provider slot fast.
- New tests: proxy timestamp wiring (converted value reaches the URL
builder, original UTC value reaches the duration lookup), _redact_url,
decisive 3xx break (anti ban), invalid timestamp rejection; deflaked the
format-cache promotion test (django cache persists across runs).
Some XC servers run PHP with display_errors off, so a timestamp shape
their parser rejects comes back as a hard HTTP 500 instead of a 200 with
inline PHP warning text. The candidate-format cascade treated any 5xx as
decisive and stopped, so catch-up never reached the timestamp shape that
works on those servers and failed outright on them.
Only short-circuit on genuinely decisive, ban-sensitive statuses
(401/403/406 and 3xx redirects — a 302 is the first sign of an XC ban);
treat 5xx like 400/404 and try the next candidate shape. Add regression
tests for "500 then success" and "all candidates 500".
- timeshift_proxy(): drop the resolve_channel_by_provider_stream_id
fallback. The client only ever sends Dispatcharr's internal Channel.id,
so resolve by id and return 404 on miss. Remove the dead `channel = None`
sentinel and the now-unused import.
- Delete the now-orphaned resolve_channel_by_provider_stream_id helper and
its local Stream import from apps/channels/utils.py (zero remaining refs).
- Import django.core.cache.cache at module top-level instead of inside
_get_cached_format_index / _set_cached_format_index.
- Fix the stale dispatcharr/urls.py comment: the "duration" slot carries
Channel.id, not the provider stream_id.
- Genericize provider host references in code comments.
- Channel resolution: resolve by Channel.id first (what the XC API emits
to clients), fall back to provider stream_id for backward compatibility.
Fixes 404 on all timeshift requests from XC clients.
- Stream management: timeshift is now a first-class citizen alongside live
and VOD in the stream-limits system. get_user_active_connections()
detects type='timeshift', attempt_stream_termination() sets a Redis
stop key (same pattern as VOD), and the timeshift view calls
check_user_stream_limits() before connecting upstream. The stream
generator checks the stop key every 100 chunks. Fixes infinite retry
loops on max_connections=1 providers for live→timeshift and
timeshift→timeshift transitions.
- Streaming: replaced HTTPStreamReader thread+pipe with direct
iter_content+yield (same pattern as VOD proxy). The pipe approach
deadlocked under gevent because the reader thread's select() competed
with the greenlet's pipe read for hub scheduling. Throughput went
from ~74 B/s to 13+ MB/s.
- TS preamble: peek now strips pre-sync bytes (PHP warnings, BOM) before
prepending to the stream, preventing corrupt TS output.
- Credential safety: _redact_url() now truncates to scheme://host/...
to avoid leaking XC path-based credentials in logs.
- Tests: updated 5 tests that referenced the removed HTTPStreamReader.
21/21 timeshift tests pass.
- Migration: fixed SyntaxWarning for unescaped regex in 0038.
Adds native catch-up/timeshift replay for Xtream Codes providers through
the same HTTPStreamReader transport pipeline as live TV.
Timeshift proxy (apps/timeshift/):
- URL cascade: 3 candidate timestamp formats per provider, per-account
format cache for fast-forward seek performance
- MPEG-TS preamble stripping (shared with HTTPStreamReader)
- Stats integration: timeshift viewers appear on /stats with TIMESHIFT badge
- Auth via hmac.compare_digest on XC password
Catchup detection — denormalized for zero-cost output queries:
- Stream.is_catchup + Stream.catchup_days populated at XC import time
- Channel.has_catchup + Channel.catchup_days + Channel.catchup_provider_stream_id
rolled up via ChannelStream post_save signal (UI path) and explicit SQL
after bulk_create (import path)
- _xc_channel_entry() reads denormalized fields instead of per-channel
custom_properties JSON introspection (eliminates N+1 queries)
- Migration 0038 backfills existing data via raw SQL
XC API enhancements:
- server_info.timezone + start/end + time_now use configured timezone
(triple consistency rule — fixes wrong-programme-plays bug)
- Dynamic has_archive flag + auto prev_days for catch-up channels
- XMLTV timestamps rewritten to local timezone for catch-up clients
HTTPStreamReader extended (apps/proxy/live_proxy/input/http_streamer.py):
- 1 MB pipe buffer via fcntl F_SETPIPE_SZ (eliminates producer/consumer
ping-pong that halved throughput)
- Pre-opened response= for URL cascade workflows
- strip_ts_preamble= for XC servers emitting PHP warnings before TS
- find_ts_sync() as shared utility
- Builds on upstream O_NONBLOCK + select() write loop
Provider stream_id lookup order:
- stream_xc() and xc_get_epg() try internal Channel.id first, fall back
to provider stream_id only when needed (avoids unconditional query on
every request)
Also includes:
- VOD provider cascade in stream_vod() — iterates all M3U relations by
priority when first provider is at capacity
- Defensive null-safety: custom_sid: None → "" in get_live_streams,
get_vod_streams, get_vod_info, get_series_info (fixes iPlayTV crash on
JSON null for string fields)
- Timeshift settings UI (timezone selector, debug toggle)
- StreamConnectionCard violet TIMESHIFT badge
- Orphan cleanup skips timeshift_* virtual channels
