- Streamlined `generate_epg()` to incrementally stream EPG data without loading the entire guide, improving memory efficiency.
- Reduced database load by deferring the fetching of large `programme_index` blobs, enhancing response times for EPG generation.
- Introduced a composite index on `(epg_id, id)` in `ProgramData` to optimize query performance during EPG exports.
- Updated tests to ensure proper functionality and performance of new features.
When an Xtream Codes provider returns no live streams on a routine refresh
(a transient upstream failure, a fetch exception, or no enabled category
matching), collect_xc_streams() returns an empty list. The refresh then fell
through to stale-marking and sync_auto_channels, which -- with nothing seen
this refresh -- deletes the account's entire auto-created channel lineup via
its per-group "no streams remaining" branch.
Guard the XC branch of _refresh_single_m3u_account_impl: on an empty result,
set the account to ERROR and return before stale-marking and auto-sync,
mirroring the standard-path empty/failed-download guards already in the
function. A transient empty fetch can no longer destroy channels.
Adds apps/m3u/tests/test_xc_empty_fetch_guard.py covering both the abort
(sync not called, channels preserved, streams not marked stale, status ERROR)
and the healthy path (non-empty fetch still runs sync).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The range-conflict warning classified each channel in the configured range as either this config's own auto-sync output or a real conflict, comparing each occupant against the source group. When a group sets a group_override, auto-sync creates its channels in the override target group, so the config's own channels failed that comparison and were misclassified as a conflict.
- Add effectiveSyncGroupId to resolve the group the sync's channels actually land in (the group_override target when set, otherwise the source group).
- Compare occupants against that effective target, so the config's own output is recognized while genuine conflicts (manual channels, channels from another account, channels in a different group, user-pinned numbers) still warn.
- Add Vitest coverage for the helper, the override case, and an over-suppression guard, plus a backend test asserting the numbers-in-range endpoint reports the override target group.
- Optimized `generate_epg()` to reduce database queries by prefetching ordered channel streams and implementing a new method for custom dummy epg name matching.
- Enhanced handling of dummy EPG sources to minimize SQL round-trips, improving performance for large guides with numerous custom-dummy channels.
- Implemented robust DB connection management in `refresh_epg_data` to handle transient errors, ensuring reliable status updates for EPG sources.
- Added retry logic for database queries and improved error handling to prevent task failures from unhandled exceptions.
- Updated EPG source status handling to ensure accurate reporting of errors and terminal states during refresh operations.
- Updated `get_streams` method in `ChannelSerializer` to use prefetched `channelstream_set`, improving performance by reducing database queries.
- Added `ChannelListIncludeStreamsQueryTests` to ensure that the number of queries remains stable as channels grow, verifying efficient stream inclusion in API responses.
- Implemented `ensure_custom_properties_dict()` to normalize custom properties across various models and serializers, addressing issues with legacy JSON-encoded strings.
- Updated M3U account and channel group models to ensure custom properties are consistently stored as dictionaries during save operations.
- Enhanced Celery task management by ensuring old DB connections are closed before and after tasks, improving reliability and preventing errors during account refresh operations. (Fixes#1338)
- Implemented `normalize_server_url()` to standardize account server URLs, ensuring that on-demand live URLs are built correctly without including API endpoints or query parameters.
- Updated `get_transformed_credentials()` and stream URL generation in `M3UMovieRelation` and `M3UEpisodeRelation` to utilize the new normalization function, improving URL handling for Xtream Codes accounts.
- Enhanced the live proxy to release geventpool DB connections more effectively across various paths.
- Implemented `close_old_connections()` in `stream_ts()`, `generate_stream_url()`, `get_stream_info_for_switch()`, `get_alternate_streams()`, `get_connections_left()`, and the cleanup process in `StreamGenerator`, ensuring that clients do not hold onto pool slots unnecessarily during streaming operations.
- Introduced `lookup_by_name_year` function to limit database scans for movies and series without TMDB/IMDB IDs, enhancing performance by scoping lookups to current batch names.
- Updated `process_movie_batch` and `process_series_batch` to utilize the new lookup function, reducing memory usage and improving efficiency.
- Registered `refresh_vod_content` and `batch_refresh_series_episodes` for post-task memory cleanup in Celery, ensuring better resource management during VOD content refreshes.
- Improved memory cleanup by invoking `gc.collect()` after batch processing in `process_m3u_batch_direct` and `collect_xc_streams`, ensuring timely release of resources.
- Added tests to verify that garbage collection is triggered appropriately after batch operations.
- Replaced the plugin listing mechanism with a more efficient handler iteration for event actions.
- Enhanced logging to provide clearer insights into the dispatching process and plugin action execution.
- Added error handling to ensure that failures in one plugin action do not block subsequent actions.
- Updated tests to cover new behavior and ensure proper handling of enabled and disabled plugins.
- Introduced a temporary staging table for efficient batch processing of EPG program inserts, reducing memory and I/O contention during updates.
- Enhanced the `parse_programs_for_source` function to stream parsed rows into the staging table before swapping them atomically into the main program data.
- Added unit tests to validate the new staging and swapping logic, ensuring existing programs are preserved during failures and that batch processing works as intended.
- Improved the handling of channel shutdown delays to ensure they reset correctly upon client reconnections, preventing premature shutdowns.
- Implemented logic to cancel pending shutdowns when clients reconnect, ensuring proper resource management and preventing client disconnect issues across uWSGI workers.
- Enhanced logging for shutdown processes and client management to provide clearer insights during channel operations.
- Added calls to `close_old_connections()` in `stream_vod()` and `build_vod_stats_data()` to prevent connection leaks during long-lived streaming responses and background stats refreshes.
- Improved connection handling in various components to ensure proper resource cleanup and prevent blocking issues.
- Improved database connection management by ensuring `close_old_connections()` is called in various methods to prevent connection leaks.
- Updated event dispatching to run asynchronously in gevent, preventing blocking during live-proxy and streaming paths.
- Added calls to `close_old_connections()` in `run_action()` and `stop_plugin()` methods to prevent connection leaks after plugin execution.
- Updated documentation to clarify connection handling for plugins, emphasizing the importance of cleanup in long-running tasks and event hooks.
- Improved channel teardown process to prevent lingering upstream connections and ensure proper resource cleanup.
- Enhanced client management by implementing checks for ghost clients and ensuring accurate client disconnection handling.
- Updated logging to provide clearer insights during channel initialization and teardown, including handling of unavailable channels.
- Refined stream manager behavior to manage ownership transitions more effectively and prevent blocking during shutdown.
- Improved handling of channel teardown to prevent client reconnect issues across uWSGI workers, ensuring proper resource cleanup and ownership management.
- Added functionality to clear all client entries from Redis for a channel, enhancing client management during shutdown.
- Updated logging and response mechanisms to inform clients of channel availability during teardown,.
- Enhanced stream manager behavior to ensure upstream connections are properly managed during ownership transitions.
- Enhanced the `stop_channel` method to prevent blocking during teardown by logging stop events asynchronously.
- Updated stream buffer behavior to ignore late writes during shutdown, improving resource management.
Aligns catch-up with how live and VOD manage provider capacity:
- Reserve a provider profile slot (connection_pool) before every upstream
connect, walking the account's active profiles default-first when the
default is at capacity (profile_full/credential_full are transient and
never mark the account decisive). Credentials for the reserved profile
are resolved via get_transformed_credentials — the same credential
extraction live playback uses — so pool accounting and real upstream
usage always agree. All eligible streams blocked on capacity alone
returns 503 (the VOD pool-exhausted precedent).
- Release exactly once via a one-shot Redis ownership token consumed with
a transactional GET+DEL: the generator finally, the response-close
wrapper, failed failover attempts and session takeover all share it, so
no path can double-decrement and a client disconnecting before the
first chunk still releases (Django registers the iterator's close() as
a resource closer). Failures between reservation and the streaming
response owning the slot release before propagating; an ownership-token
write failure releases directly and reports transient unavailability.
- One catch-up session per user and channel: a new request (programme
jump or seek) displaces the user's previous session on that channel —
its slot is released synchronously, its stats are unregistered, and its
generator is stopped through the standard stop-key mechanism — so rapid
seeking cannot stack upstream provider connections.
- rollup_channel_catchup_fields self-heals channels left flagged
is_catchup with no remaining catch-up stream (outside the
account-scoped CTE), covering bulk removals on manual/multi-provider
channels regardless of how the link rows disappeared. A regression test
locks the ChannelStream post_delete signal firing on queryset bulk
deletes.
Backend test suite grows to 94 (slot reservation/release on every
failover outcome, profile walk, mixed capacity-vs-upstream precedence,
exception-path release, token exactly-once semantics, takeover scoping
and ordering, never-started-generator release, rollup self-heal).
- Introduced a new helper function to generate user-facing error messages for RANGE_EXHAUSTED failures, ensuring the fallback range is correctly displayed.
- Updated tests to verify that the error message reflects the fallback range instead of the hidden auto_sync_channel_start, enhancing clarity for users.
- Replaced individual EPG program parse tasks with a centralized dispatch function to streamline guide refresh for newly assigned EPG IDs.
- Implemented batching for guide fetches when multiple EPGs are mapped, reducing redundant API calls and improving efficiency.
- Updated related utility functions to support the new fetching strategy and added tests to ensure correct behavior under various scenarios.
- Implemented a targeted guide fetch for Schedules Direct when mapping a channel, ensuring immediate data availability without redundant API calls.
- Updated EPG signals to queue guide fetches for newly mapped channels, streamlining the process and improving user experience.
- Refactored existing logic to skip unnecessary program refreshes for dummy sources and ensure proper handling of Schedules Direct data.
- Added unit tests to validate the new fetching behavior and ensure robustness of the integration.
Implements all four points from the latest review, plus hardening from a
pre-submission audit pass.
1. Access control: timeshift_proxy now enforces
network_access_allowed(request, "STREAMS", user) — same key and placement
as the live XC stream endpoint.
2. Catch-up failover: the proxy walks the channel's catch-up streams in
channelstream order (get_channel_catchup_streams), mirroring live
playback. Each attempt carries its own provider context: account
credentials, provider stream id, reported server_info timezone (the
UTC->provider conversion is recomputed per attempt), user-agent, and the
per-account URL-format cache. The first streamable response wins; if all
providers fail the last failure is returned.
Ban-safety is per account: a decisive auth/ban-class failure (401/403/406)
marks the account and skips its remaining streams (e.g. FHD/HD variants of
the same channel) instead of hammering a banning provider, while other
accounts — different hosts — are still tried. Streams from disabled M3U
accounts are excluded, same as live dispatch. Redirects stay enabled on
purpose (XC providers legitimately 302 to load-balanced streaming nodes);
the 3xx decisive branch is kept as defense-in-depth and documented as such.
3. apps/proxy/live_proxy/views.py restored byte-identical to upstream — the
leftover channel-id wrapper from the removed provider-stream-id fallback
is gone (zero-line diff).
4. Single remaining setting relocated: xmltv_prev_days_override now lives in
proxy_settings (backend default in get_proxy_settings, consumed by the
XMLTV prev_days resolution). The timeshift_settings group,
TIMESHIFT_DEFAULTS, get_timeshift_settings, the Settings → Timeshift form
and tab are all removed; the field appears under Settings → Proxy Settings
(0 = auto-detect, capped at 30).
Audit hardening in the same pass:
- Updated the proxy-settings defaults unit test for the new key (would have
failed CI otherwise).
- Migration backfills use schema_editor.connection instead of the global
connection (multi-database correctness).
- CHANGELOG and module docstring brought in line with the final architecture
(PATH-first cascade, failover, setting under Proxy Settings).
- Tests grown to 69 backend tests: failover success/exhaustion/skip
semantics, decisive-account skip vs soft-failure retry, per-stream
timezone conversion (different zones per provider), 406/connection-error
cascade paths, stream-limit and no-eligible-stream outcomes,
network-gate 403, server_info strict-UTC guarantee, EPG duration window
resolution, and DB-backed coverage of xc_password auth, user_level access
and the failover stream ordering (catch-up-only, active accounts,
channelstream order). The format-cache test now runs on an isolated
locmem cache.
- Added functions to compute schedule changes based on MD5 hashes and backfill missing dates for mapped stations.
- Enhanced handling of orphaned ProgramData records for unmapped EPG entries during schedule fetch.
- Introduced unit tests to validate MD5 comparison, backfill logic, and metadata fetching requirements.
- Introduced `dispatcharr_user_agent`, `dispatcharr_dvr_user_agent`, and `dispatcharr_http_headers` functions in `core.utils` to standardize User-Agent strings and HTTP headers for outbound requests.
- Updated various components, including `LogoViewSet`, `EPGSourceViewSet`, and VOD proxy views, to utilize the new header functions, ensuring consistent User-Agent usage across the application.
- Enhanced the handling of Schedules Direct API requests by including proper User-Agent headers, addressing previous API compliance issues.
- Introduced a new method `_fetch_sd_countries` in `EPGSourceViewSet` to fetch SD countries directly from the backend, eliminating the need for a separate proxy endpoint.
- Updated the `sd_lineups` action to include the fetched countries in the response.
- Removed the `getSDCountries` method from the API and adjusted the frontend to fetch countries alongside lineups, improving efficiency
- Updated related components to handle the new data structure for countries.
- http_streamer.py: restore HTTPStreamReader to its upstream form and keep
only the find_ts_sync() addition. The response=/extra_headers=/
strip_ts_preamble= extensions had no remaining callers since the timeshift
view moved to direct iter_content streaming (delta shrinks from +235/-83
lines to +28).
- Unify the catchup_days semantics everywhere: a channel's archive depth is
MAX(catchup_days) over its CATCH-UP streams only. The SQL rollup now uses
a FILTER (WHERE s.is_catchup) aggregate and the ChannelStream signal uses
the same MAX aggregation (it previously took the first stream by order,
and the rollup aggregated over all streams including non-catchup ones).
- Migration backfill: also accept the lowercase 'true' that ->> extraction
yields for JSON booleans.
- get_channel_catchup_info(): drop the tv_archive_duration key — its only
caller never used it.
- Document why timeshift termination fails closed when Redis is unavailable
(denying the new stream is what protects the provider connection limit),
and update the stale stop-key comment (5 s cadence, not 100 chunks).
Two coupled fixes that make catch-up play the requested programme:
1. Try the PATH catch-up form (/timeshift/.../{start}/{id}.ts) BEFORE the
timeshift.php query form. Empirical testing showed some XC providers
return the LIVE stream on the query form (HTTP 200, silently ignoring
`start`) — a valid MPEG-TS indistinguishable from a real archive, so the
cascade accepted it and the user always got live content. The PATH form
actually seeks. Candidate ordering now lives in
build_timeshift_candidate_urls() with the full rationale.
2. Strict-UTC XC API surface + single proxy-time timezone conversion:
- xc_get_epg start/end always UTC, server_info.timezone always "UTC",
time_now UTC — the timezone triple is consistently UTC.
- Removed the global default_timezone setting and the
_convert_xmltv_to_local_timezone rewrite entirely.
- The proxy converts the client's UTC timestamp to the SERVING provider's
own zone (server_info.timezone captured on account refresh, read from
the account's default profile) right before building the upstream URL —
DST-correct via ZoneInfo, no-op for UTC/unknown zones. Verified against
a real provider: it interprets the URL timestamp as its LOCAL wall
clock, so pure UTC pass-through seeks 1-2h off.
- EPG duration lookup keeps the ORIGINAL UTC timestamp (programmes are
stored in UTC) — exactly one conversion in the whole chain.
Also per review feedback:
- Removed the debug_logging toggle — verbose timeshift logging now follows
the standard logger DEBUG level (DISPATCHARR_LOG_LEVEL).
- Removed the dead default_language setting (never read anywhere; the EPG
lang field is hardcoded). TIMESHIFT_DEFAULTS is down to
xmltv_prev_days_override, and the Settings UI shows that single field
with accurate help text.
- Validate the timestamp up front (400 on malformed input) instead of
forwarding it verbatim into the upstream URL.
- Redact the upstream URL in the connection-error log (requests exceptions
embed the full URL, which carries XC credentials).
- URL-encode XC credentials in both URL builders.
- Request identity encoding upstream (the TS-sync peek reads raw bytes).
- Poll the stop key on the 5-second heartbeat cadence instead of every 100
chunks (~25 MB), so stream-limit terminations free the provider slot fast.
- New tests: proxy timestamp wiring (converted value reaches the URL
builder, original UTC value reaches the duration lookup), _redact_url,
decisive 3xx break (anti ban), invalid timestamp rejection; deflaked the
format-cache promotion test (django cache persists across runs).
- Removed redundant logic for releasing server group slots in `move_credential_slot_on_profile_switch` and `release_profile_slot`.
- Simplified the `_release_server_group_slot_for_profile` function by eliminating it, as its functionality was integrated into other methods.
- Improved logging in the Stream model for better debugging of profile evaluations.
- Introduced a new method `_stream_assignment_is_reusable` in the Channel model to determine if existing stream assignments can be reused, enhancing efficiency.
- Updated the release logic in `release_profile_slot` to utilize stored credential keys, reducing unnecessary database lookups.
- Simplified error handling in the `get_stream_info_for_switch` function to ensure proper stream release on exceptions.
- Enhanced tests for connection pool management and error handling in the ServerGroupsTable component to improve reliability and user feedback.
- Eliminated redundant extraction of provider usernames from stream URLs in both ChannelStatus and VOD stats data functions, simplifying the codebase.
- Updated frontend components to reflect the removal of provider username display, streamlining the UI.
- Enhanced overall readability and maintainability of the code by reducing complexity in handling stream URLs.
- Renamed `_clear_stream_assignment_keys` to `_release_stale_stream_assignment` for clarity and updated its logic to release pool counters.
- Introduced new functions for managing credential slots during profile switches, enhancing the handling of shared connection limits across server groups.
- Removed the `max_streams` field from the `ServerGroup` model and updated related components to reflect this change, simplifying the server group management.
- Updated frontend components to integrate server group management, allowing for dynamic creation and editing of server groups.
- Enhanced error handling in stream URL generation to provide more informative feedback on connection issues.
- Added tests for stale assignment release and credential management during profile switches.
- A duplicate provider number keeps one channel and falls the collider back to a free number.
- A provider number matching an existing channel falls back instead of overwriting it.
- Moved matching logic to a dedicated module for better organization and testability.
- Made single-channel auto-matching asynchronous, allowing for larger EPG libraries without hitting HTTP timeouts.
- Enhanced memory management and throughput during EPG matching, including optimizations for fuzzy matching and bulk processing.
- Fixed various reliability issues in the auto-matching process, ensuring accurate channel assignments and improved UI feedback.
- Updated API views and frontend components to reflect changes in the matching process and provide real-time notifications.
- Added tests for EPG matching functionality and name normalization.
- Single-channel and selected-channel auto-match always run, even when the channel already has EPG assigned; match-all (no channel IDs) still only processes channels without EPG.
The auto-sync overhaul added a [start, end] range and a shared numbering picker, but each mode's UI exposes only a subset of the persisted fields (Provider's "Start #" writes channel_numbering_fallback; Next Available exposes no Start/End) while the backend read auto_sync_channel_start and auto_sync_channel_end in every mode. Switching modes never resets the others, so a stale or auto-computed value silently changed numbering.
- Provider mode honors the provider-supplied number (stream_chno) verbatim; the group's Start (channel_numbering_fallback) and End bound only the fallback for streams the provider did not number. auto_sync_channel_start is no longer applied in provider mode.
- Next Available ignores End, since its UI exposes no range.
- Range enforcement (the overflow-delete) runs in fixed mode only, the one mode with a user-set [start, end] range.
- Provider mode gains the Start>End guard fixed mode already had, so an inverted fallback range cannot fail every numberless stream.
Includes backend and frontend regression tests.
