Add end-to-end catch-up support for XC clients and native apps: provider
proxy with failover and per-viewer session pooling, REST session minting
for tokenless playback URLs, catch-up admin stats, combined connection
stats, and Stats UI with dedicated cards plus websocket updates.
Includes Redis namespace consolidation under timeshift:* (dropping legacy
timeshift_ id prefixes), dedicated catch-up stop by session_id, and
cleaner channel/client metadata split for stats keys.
Closes#133
The Find and Replace preview did not correctly reflect the rename the sync performs, and the rename engine differed from the preview engine.
- The preview rendered the literal $1 instead of the substituted capture group, because the replacement was passed straight into the regex engine, which honors \1, not the JS-style $1 the field accepts.
- The preview compiled patterns with the regex module while the live rename used stdlib re, so patterns valid in regex but not re (for example ^*) previewed a transform the sync silently skipped.
- A rename that expanded a name past the Channel.name column length aborted the whole bulk_create sync, while the preview showed the full name.
- Convert JS-style $1 backreferences to \1 via a shared helper used by both the preview and the live rename.
- Switch the live rename from re.sub to regex.sub, matching the preview engine and the sync's own include/exclude filters, with a timeout to bound catastrophic backtracking on user patterns.
- Cap the rename result at the Channel.name column length in both paths, so an over-length result cannot abort the sync.
- Add unit, integration, and differential parity tests covering the above.
- Updated `get_streams` method in `ChannelSerializer` to use prefetched `channelstream_set`, improving performance by reducing database queries.
- Added `ChannelListIncludeStreamsQueryTests` to ensure that the number of queries remains stable as channels grow, verifying efficient stream inclusion in API responses.
- Implemented `ensure_custom_properties_dict()` to normalize custom properties across various models and serializers, addressing issues with legacy JSON-encoded strings.
- Updated M3U account and channel group models to ensure custom properties are consistently stored as dictionaries during save operations.
- Enhanced Celery task management by ensuring old DB connections are closed before and after tasks, improving reliability and preventing errors during account refresh operations. (Fixes#1338)
- Improved the handling of channel shutdown delays to ensure they reset correctly upon client reconnections, preventing premature shutdowns.
- Implemented logic to cancel pending shutdowns when clients reconnect, ensuring proper resource management and preventing client disconnect issues across uWSGI workers.
- Enhanced logging for shutdown processes and client management to provide clearer insights during channel operations.
- Added calls to `close_old_connections()` in `stream_vod()` and `build_vod_stats_data()` to prevent connection leaks during long-lived streaming responses and background stats refreshes.
- Improved connection handling in various components to ensure proper resource cleanup and prevent blocking issues.
- Improved channel teardown process to prevent lingering upstream connections and ensure proper resource cleanup.
- Enhanced client management by implementing checks for ghost clients and ensuring accurate client disconnection handling.
- Updated logging to provide clearer insights during channel initialization and teardown, including handling of unavailable channels.
- Refined stream manager behavior to manage ownership transitions more effectively and prevent blocking during shutdown.
- Improved handling of channel teardown to prevent client reconnect issues across uWSGI workers, ensuring proper resource cleanup and ownership management.
- Added functionality to clear all client entries from Redis for a channel, enhancing client management during shutdown.
- Updated logging and response mechanisms to inform clients of channel availability during teardown,.
- Enhanced stream manager behavior to ensure upstream connections are properly managed during ownership transitions.
- Replaced individual EPG program parse tasks with a centralized dispatch function to streamline guide refresh for newly assigned EPG IDs.
- Implemented batching for guide fetches when multiple EPGs are mapped, reducing redundant API calls and improving efficiency.
- Updated related utility functions to support the new fetching strategy and added tests to ensure correct behavior under various scenarios.
- Implemented a targeted guide fetch for Schedules Direct when mapping a channel, ensuring immediate data availability without redundant API calls.
- Updated EPG signals to queue guide fetches for newly mapped channels, streamlining the process and improving user experience.
- Refactored existing logic to skip unnecessary program refreshes for dummy sources and ensure proper handling of Schedules Direct data.
- Added unit tests to validate the new fetching behavior and ensure robustness of the integration.
Implements all four points from the latest review, plus hardening from a
pre-submission audit pass.
1. Access control: timeshift_proxy now enforces
network_access_allowed(request, "STREAMS", user) — same key and placement
as the live XC stream endpoint.
2. Catch-up failover: the proxy walks the channel's catch-up streams in
channelstream order (get_channel_catchup_streams), mirroring live
playback. Each attempt carries its own provider context: account
credentials, provider stream id, reported server_info timezone (the
UTC->provider conversion is recomputed per attempt), user-agent, and the
per-account URL-format cache. The first streamable response wins; if all
providers fail the last failure is returned.
Ban-safety is per account: a decisive auth/ban-class failure (401/403/406)
marks the account and skips its remaining streams (e.g. FHD/HD variants of
the same channel) instead of hammering a banning provider, while other
accounts — different hosts — are still tried. Streams from disabled M3U
accounts are excluded, same as live dispatch. Redirects stay enabled on
purpose (XC providers legitimately 302 to load-balanced streaming nodes);
the 3xx decisive branch is kept as defense-in-depth and documented as such.
3. apps/proxy/live_proxy/views.py restored byte-identical to upstream — the
leftover channel-id wrapper from the removed provider-stream-id fallback
is gone (zero-line diff).
4. Single remaining setting relocated: xmltv_prev_days_override now lives in
proxy_settings (backend default in get_proxy_settings, consumed by the
XMLTV prev_days resolution). The timeshift_settings group,
TIMESHIFT_DEFAULTS, get_timeshift_settings, the Settings → Timeshift form
and tab are all removed; the field appears under Settings → Proxy Settings
(0 = auto-detect, capped at 30).
Audit hardening in the same pass:
- Updated the proxy-settings defaults unit test for the new key (would have
failed CI otherwise).
- Migration backfills use schema_editor.connection instead of the global
connection (multi-database correctness).
- CHANGELOG and module docstring brought in line with the final architecture
(PATH-first cascade, failover, setting under Proxy Settings).
- Tests grown to 69 backend tests: failover success/exhaustion/skip
semantics, decisive-account skip vs soft-failure retry, per-stream
timezone conversion (different zones per provider), 406/connection-error
cascade paths, stream-limit and no-eligible-stream outcomes,
network-gate 403, server_info strict-UTC guarantee, EPG duration window
resolution, and DB-backed coverage of xc_password auth, user_level access
and the failover stream ordering (catch-up-only, active accounts,
channelstream order). The format-cache test now runs on an isolated
locmem cache.
- Introduced `dispatcharr_user_agent`, `dispatcharr_dvr_user_agent`, and `dispatcharr_http_headers` functions in `core.utils` to standardize User-Agent strings and HTTP headers for outbound requests.
- Updated various components, including `LogoViewSet`, `EPGSourceViewSet`, and VOD proxy views, to utilize the new header functions, ensuring consistent User-Agent usage across the application.
- Enhanced the handling of Schedules Direct API requests by including proper User-Agent headers, addressing previous API compliance issues.
- http_streamer.py: restore HTTPStreamReader to its upstream form and keep
only the find_ts_sync() addition. The response=/extra_headers=/
strip_ts_preamble= extensions had no remaining callers since the timeshift
view moved to direct iter_content streaming (delta shrinks from +235/-83
lines to +28).
- Unify the catchup_days semantics everywhere: a channel's archive depth is
MAX(catchup_days) over its CATCH-UP streams only. The SQL rollup now uses
a FILTER (WHERE s.is_catchup) aggregate and the ChannelStream signal uses
the same MAX aggregation (it previously took the first stream by order,
and the rollup aggregated over all streams including non-catchup ones).
- Migration backfill: also accept the lowercase 'true' that ->> extraction
yields for JSON booleans.
- get_channel_catchup_info(): drop the tv_archive_duration key — its only
caller never used it.
- Document why timeshift termination fails closed when Redis is unavailable
(denying the new stream is what protects the provider connection limit),
and update the stale stop-key comment (5 s cadence, not 100 chunks).
- Improved logging in the Stream model for better debugging of profile evaluations.
- Introduced a new method `_stream_assignment_is_reusable` in the Channel model to determine if existing stream assignments can be reused, enhancing efficiency.
- Updated the release logic in `release_profile_slot` to utilize stored credential keys, reducing unnecessary database lookups.
- Simplified error handling in the `get_stream_info_for_switch` function to ensure proper stream release on exceptions.
- Enhanced tests for connection pool management and error handling in the ServerGroupsTable component to improve reliability and user feedback.
- Renamed `_clear_stream_assignment_keys` to `_release_stale_stream_assignment` for clarity and updated its logic to release pool counters.
- Introduced new functions for managing credential slots during profile switches, enhancing the handling of shared connection limits across server groups.
- Removed the `max_streams` field from the `ServerGroup` model and updated related components to reflect this change, simplifying the server group management.
- Updated frontend components to integrate server group management, allowing for dynamic creation and editing of server groups.
- Enhanced error handling in stream URL generation to provide more informative feedback on connection issues.
- Added tests for stale assignment release and credential management during profile switches.
- Moved matching logic to a dedicated module for better organization and testability.
- Made single-channel auto-matching asynchronous, allowing for larger EPG libraries without hitting HTTP timeouts.
- Enhanced memory management and throughput during EPG matching, including optimizations for fuzzy matching and bulk processing.
- Fixed various reliability issues in the auto-matching process, ensuring accurate channel assignments and improved UI feedback.
- Updated API views and frontend components to reflect changes in the matching process and provide real-time notifications.
- Added tests for EPG matching functionality and name normalization.
- Single-channel and selected-channel auto-match always run, even when the channel already has EPG assigned; match-all (no channel IDs) still only processes channels without EPG.
Add support for automatically applying channel logos from EPG data during refresh. Introduce a new toggle in the UI for enabling/disabling this feature, and update the backend to handle both channel IDs and EPG source IDs for logo application. Enhance the logo application task to process large libraries efficiently in chunks, improving performance and memory usage. Update changelog to reflect these changes.
Add a new task to fetch program artwork from Schedules Direct and auto-apply logos to channels based on EPG data. Enhance the existing logic to handle poster backfill and update program records with fetched artwork. Readd the WebSocket handler to manage IP lookup events and adjust the EPGsTable component to display relevant credentials based on the source type.
The compact repack read its channels with no ORDER BY, so the pack followed PostgreSQL's physical row order. That order drifts after the
UPDATEs each repack issues, so successive syncs
packed the same channels into different numbers within the configured range. Auto-synced channel numbers reshuffled on every sync even when the provider had not changed.
- Add .order_by("id") to the _repack_inner channel query so the pack is deterministic. id order is creation order, which tracks the provider stream order used by the default "provider" sort.
- Add c.id as a secondary key to the name / tvg_id / updated_at sorts so equal values (e.g. blank tvg_id) break ties deterministically instead of churning.
- Add a deterministic regression test that forces a physical heap reorder (CLUSTER) and asserts two consecutive repacks produce identical channel numbers.
Store credential Redis keys at reserve so release works when the profile row is deleted. Return reserve failure reasons to avoid fingerprint DB queries on logging paths. Document unlimited profile bypass in pool logic and Server Group UI.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add updateSDSettings method to API client for SD logo style and settings
management. Add fetchEPGData and requeryChannels calls after EPG match
completion to ensure UI refreshes immediately. Add SD logo style picker
and cache-busting for channel logos.
- Compact numbering: resolve the source relation when an override stores channels under the target group, so hiding releases the slot, unhiding assigns one, and repack sees the channels (no more spurious RANGE_EXHAUSTED). Type-safe override match; the bulk path stays single-query on the common path.
- Channel form: keep the clear-override control available when an override's value equals the provider value (isFormFieldOverridden is now existence-aware).
- Tests: compact override hide/unhide/repack, no-override fast-path guard, repack query-scaling guard, and existence-aware override detection.
The Channel.catchup_provider_stream_id model field carries a help_text that
was not reflected in migration 0038, leaving makemigrations --check dirty.
Add the AlterField migration so model and migration state match (no schema
change — help_text only).
