Updated the EPG generation logic to ensure that channels without a channel number do not cause crashes in the XMLTV export. Implemented a deferred assignment mechanism for channels with null numbers, allowing them to receive valid integers during processing. Added tests to verify that the XMLTV output remains stable and valid even when channels are unnumbered.
Updated the handling of channels in the live streams setup to ensure that channels without a channel number do not cause crashes. Implemented logic to assign a valid number to these channels during processing. Added tests to verify that the system correctly assigns numbers to channels with null values, ensuring stability in the live streams feature.
Enhanced the `/output/epg` cold rebuild process to prevent freezing of the gevent uWSGI worker. The `_stream_build` function now yields control to the gevent hub after processing each cached chunk, allowing other requests to be handled concurrently. This change improves responsiveness during CPU-bound XMLTV generation. Additionally, introduced a new utility function, `_cooperative_yield`, to facilitate yielding in CPU-bound loops.
- Improved `xc_get_vod_streams` and `xc_get_series` functions to utilize a new helper method for fetching distinct relations based on account priority, significantly reducing memory usage and response times for large libraries.
- Updated the handling of VOD streams to avoid unnecessary ORM instantiation, enhancing performance for endpoints dealing with extensive movie and series data.
- Added comprehensive tests to ensure the correctness of the new logic and verify that the highest priority relations are correctly selected.
- Updated the `generate_dummy_programs` function to include `export_lookback` and `export_cutoff` parameters, allowing for more precise control over the EPG data generation window.
- Added a new test case to verify that future events are correctly represented in the grid window, ensuring upcoming fillers are displayed as expected.
- Moved the `programme_index` from the `EPGSource` model to a new `EPGSourceIndex` table, ensuring that the large JSON blob is only loaded when explicitly accessed, thus improving query performance and memory efficiency.
- Updated related queries and API views to utilize the new structure, including adjustments to EPG generation and import logic to prevent unnecessary data loading.
- Enhanced memory management in the EPG grid endpoint to reduce worker RSS during response handling.
- Moved all XMLTV output logic from `apps/output/views.py` to `apps/output/epg.py`, streamlining the codebase and maintaining the existing HTTP endpoint functionality.
- Updated related tests and references to ensure proper integration with the new module structure.
- Streamlined `generate_epg()` to incrementally stream EPG data without loading the entire guide, improving memory efficiency.
- Reduced database load by deferring the fetching of large `programme_index` blobs, enhancing response times for EPG generation.
- Introduced a composite index on `(epg_id, id)` in `ProgramData` to optimize query performance during EPG exports.
- Updated tests to ensure proper functionality and performance of new features.
- Optimized `generate_epg()` to reduce database queries by prefetching ordered channel streams and implementing a new method for custom dummy epg name matching.
- Enhanced handling of dummy EPG sources to minimize SQL round-trips, improving performance for large guides with numerous custom-dummy channels.
Implements all four points from the latest review, plus hardening from a
pre-submission audit pass.
1. Access control: timeshift_proxy now enforces
network_access_allowed(request, "STREAMS", user) — same key and placement
as the live XC stream endpoint.
2. Catch-up failover: the proxy walks the channel's catch-up streams in
channelstream order (get_channel_catchup_streams), mirroring live
playback. Each attempt carries its own provider context: account
credentials, provider stream id, reported server_info timezone (the
UTC->provider conversion is recomputed per attempt), user-agent, and the
per-account URL-format cache. The first streamable response wins; if all
providers fail the last failure is returned.
Ban-safety is per account: a decisive auth/ban-class failure (401/403/406)
marks the account and skips its remaining streams (e.g. FHD/HD variants of
the same channel) instead of hammering a banning provider, while other
accounts — different hosts — are still tried. Streams from disabled M3U
accounts are excluded, same as live dispatch. Redirects stay enabled on
purpose (XC providers legitimately 302 to load-balanced streaming nodes);
the 3xx decisive branch is kept as defense-in-depth and documented as such.
3. apps/proxy/live_proxy/views.py restored byte-identical to upstream — the
leftover channel-id wrapper from the removed provider-stream-id fallback
is gone (zero-line diff).
4. Single remaining setting relocated: xmltv_prev_days_override now lives in
proxy_settings (backend default in get_proxy_settings, consumed by the
XMLTV prev_days resolution). The timeshift_settings group,
TIMESHIFT_DEFAULTS, get_timeshift_settings, the Settings → Timeshift form
and tab are all removed; the field appears under Settings → Proxy Settings
(0 = auto-detect, capped at 30).
Audit hardening in the same pass:
- Updated the proxy-settings defaults unit test for the new key (would have
failed CI otherwise).
- Migration backfills use schema_editor.connection instead of the global
connection (multi-database correctness).
- CHANGELOG and module docstring brought in line with the final architecture
(PATH-first cascade, failover, setting under Proxy Settings).
- Tests grown to 69 backend tests: failover success/exhaustion/skip
semantics, decisive-account skip vs soft-failure retry, per-stream
timezone conversion (different zones per provider), 406/connection-error
cascade paths, stream-limit and no-eligible-stream outcomes,
network-gate 403, server_info strict-UTC guarantee, EPG duration window
resolution, and DB-backed coverage of xc_password auth, user_level access
and the failover stream ordering (catch-up-only, active accounts,
channelstream order). The format-cache test now runs on an isolated
locmem cache.
Two coupled fixes that make catch-up play the requested programme:
1. Try the PATH catch-up form (/timeshift/.../{start}/{id}.ts) BEFORE the
timeshift.php query form. Empirical testing showed some XC providers
return the LIVE stream on the query form (HTTP 200, silently ignoring
`start`) — a valid MPEG-TS indistinguishable from a real archive, so the
cascade accepted it and the user always got live content. The PATH form
actually seeks. Candidate ordering now lives in
build_timeshift_candidate_urls() with the full rationale.
2. Strict-UTC XC API surface + single proxy-time timezone conversion:
- xc_get_epg start/end always UTC, server_info.timezone always "UTC",
time_now UTC — the timezone triple is consistently UTC.
- Removed the global default_timezone setting and the
_convert_xmltv_to_local_timezone rewrite entirely.
- The proxy converts the client's UTC timestamp to the SERVING provider's
own zone (server_info.timezone captured on account refresh, read from
the account's default profile) right before building the upstream URL —
DST-correct via ZoneInfo, no-op for UTC/unknown zones. Verified against
a real provider: it interprets the URL timestamp as its LOCAL wall
clock, so pure UTC pass-through seeks 1-2h off.
- EPG duration lookup keeps the ORIGINAL UTC timestamp (programmes are
stored in UTC) — exactly one conversion in the whole chain.
Also per review feedback:
- Removed the debug_logging toggle — verbose timeshift logging now follows
the standard logger DEBUG level (DISPATCHARR_LOG_LEVEL).
- Removed the dead default_language setting (never read anywhere; the EPG
lang field is hardcoded). TIMESHIFT_DEFAULTS is down to
xmltv_prev_days_override, and the Settings UI shows that single field
with accurate help text.
- Validate the timestamp up front (400 on malformed input) instead of
forwarding it verbatim into the upstream URL.
- Redact the upstream URL in the connection-error log (requests exceptions
embed the full URL, which carries XC credentials).
- URL-encode XC credentials in both URL builders.
- Request identity encoding upstream (the TS-sync peek reads raw bytes).
- Poll the stop key on the 5-second heartbeat cadence instead of every 100
chunks (~25 MB), so stream-limit terminations free the provider slot fast.
- New tests: proxy timestamp wiring (converted value reaches the URL
builder, original UTC value reaches the duration lookup), _redact_url,
decisive 3xx break (anti ban), invalid timestamp rejection; deflaked the
format-cache promotion test (django cache persists across runs).
- Precompute base URL and logo path components once per request to reduce redundant calls to `build_absolute_uri_with_port()`, enhancing performance.
- Update logo URL construction for both cached and direct logos to utilize the precomputed values, ensuring consistency and efficiency in URL handling.
- Precompute base URLs for M3U stream and logo paths to reduce redundant calls to `build_absolute_uri_with_port()`, improving performance.
- Update the generation of EPG and stream URLs to utilize the precomputed base URL, ensuring consistency and efficiency in URL handling.
- Enhance the handling of logo URLs by consolidating logic for cached and direct logo retrieval, streamlining the process and reducing overhead.
- Channel resolution: resolve by Channel.id first (what the XC API emits
to clients), fall back to provider stream_id for backward compatibility.
Fixes 404 on all timeshift requests from XC clients.
- Stream management: timeshift is now a first-class citizen alongside live
and VOD in the stream-limits system. get_user_active_connections()
detects type='timeshift', attempt_stream_termination() sets a Redis
stop key (same pattern as VOD), and the timeshift view calls
check_user_stream_limits() before connecting upstream. The stream
generator checks the stop key every 100 chunks. Fixes infinite retry
loops on max_connections=1 providers for live→timeshift and
timeshift→timeshift transitions.
- Streaming: replaced HTTPStreamReader thread+pipe with direct
iter_content+yield (same pattern as VOD proxy). The pipe approach
deadlocked under gevent because the reader thread's select() competed
with the greenlet's pipe read for hub scheduling. Throughput went
from ~74 B/s to 13+ MB/s.
- TS preamble: peek now strips pre-sync bytes (PHP warnings, BOM) before
prepending to the stream, preventing corrupt TS output.
- Credential safety: _redact_url() now truncates to scheme://host/...
to avoid leaking XC path-based credentials in logs.
- Tests: updated 5 tests that referenced the removed HTTPStreamReader.
21/21 timeshift tests pass.
- Migration: fixed SyntaxWarning for unescaped regex in 0038.
Adds native catch-up/timeshift replay for Xtream Codes providers through
the same HTTPStreamReader transport pipeline as live TV.
Timeshift proxy (apps/timeshift/):
- URL cascade: 3 candidate timestamp formats per provider, per-account
format cache for fast-forward seek performance
- MPEG-TS preamble stripping (shared with HTTPStreamReader)
- Stats integration: timeshift viewers appear on /stats with TIMESHIFT badge
- Auth via hmac.compare_digest on XC password
Catchup detection — denormalized for zero-cost output queries:
- Stream.is_catchup + Stream.catchup_days populated at XC import time
- Channel.has_catchup + Channel.catchup_days + Channel.catchup_provider_stream_id
rolled up via ChannelStream post_save signal (UI path) and explicit SQL
after bulk_create (import path)
- _xc_channel_entry() reads denormalized fields instead of per-channel
custom_properties JSON introspection (eliminates N+1 queries)
- Migration 0038 backfills existing data via raw SQL
XC API enhancements:
- server_info.timezone + start/end + time_now use configured timezone
(triple consistency rule — fixes wrong-programme-plays bug)
- Dynamic has_archive flag + auto prev_days for catch-up channels
- XMLTV timestamps rewritten to local timezone for catch-up clients
HTTPStreamReader extended (apps/proxy/live_proxy/input/http_streamer.py):
- 1 MB pipe buffer via fcntl F_SETPIPE_SZ (eliminates producer/consumer
ping-pong that halved throughput)
- Pre-opened response= for URL cascade workflows
- strip_ts_preamble= for XC servers emitting PHP warnings before TS
- find_ts_sync() as shared utility
- Builds on upstream O_NONBLOCK + select() write loop
Provider stream_id lookup order:
- stream_xc() and xc_get_epg() try internal Channel.id first, fall back
to provider stream_id only when needed (avoids unconditional query on
every request)
Also includes:
- VOD provider cascade in stream_vod() — iterates all M3U relations by
priority when first provider is at capacity
- Defensive null-safety: custom_sid: None → "" in get_live_streams,
get_vod_streams, get_vod_info, get_series_info (fixes iPlayTV crash on
JSON null for string fields)
- Timeshift settings UI (timezone selector, debug toggle)
- StreamConnectionCard violet TIMESHIFT badge
- Orphan cleanup skips timeshift_* virtual channels
The previous name suggested "hidden from a specific user account" rather than "hidden from downstream client output", which was the actual
behavior. The new name is unambiguous and matches the help_text language ("Exclude this channel from downstream client output").
Mechanical rename across the migration, model, serializer, signals, sync logic, output endpoints, frontend forms / tables / utils, tests, and CHANGELOG. 77 occurrences across 20 files.
No behavior change.
Per-field channel overrides for auto-synced channels, hide-from-output flag, range-bounded auto-numbering with re-pack, multi-stream channel safety, multi-provider shared-range merging, and an across-the-board move from per-row sync writes to bulk operations.
Migrations:
apps/channels:
0036_channeloverride_and_user_hidden,
0037_backfill_auto_created_by_null,
0038_channelgroupm3uaccount_auto_sync_channel_end,
0039_channel_channel_number_nullable
apps/m3u:
0020_m3uaccount_auto_cleanup_unused_channels
See CHANGELOG.md for the full commit log
- Performance: `xc_get_epg` now uses `select_related('epg_data__epg_source')` on all three channel fetch paths. Previously each request triggered 2 additional queries to resolve `channel.epg_data` and `channel.epg_data.epg_source`.
- Performance: `generate_m3u` now uses `prefetch_related` for streams when `?direct=true` is requested, eliminating N+1 stream queries (one per channel) on that code path.
- Performance: `EPGGridAPIView` (`apps/epg/api_views.py`) now uses `select_related('epg_data__epg_source')` on the `channels_with_custom_dummy` queryset, eliminating 2 extra queries per channel (for `epg_data` and `epg_source`) in the dummy EPG generation loop.
- `generate_epg` now uses `select_related('epg_data__epg_source')` on all EPG channel querysets
- `generate_epg` now issues a single cross-channel `ProgramData` bulk query. `.values()` returns plain dicts
- `xc_get_live_streams` no longer calls `ChannelGroup.objects.get_or_create(name="Default Group")` once per null-group channel
