Add exponential backoff retry (_db_retry helper) for transient DB errors during final metadata save, startup recovery, and initial TS proxy
connection. Add FFmpeg remux retry for transient I/O errors. Retry same proxy base before falling back to next candidate on initial connection failure.
Fix DVR card logos not displaying: channel summary API returns logo_id (integer) but frontend expected a nested logo object with cache_url.
Add getChannelLogoUrl() helper that handles both data shapes. Backend artwork prefetch and run_recording now fall back to the channel's own logo when no show-specific artwork is found, ensuring poster_logo_id is always populated.
Prevent incorrect artwork by skipping external API searches (TVMaze, iTunes) when no program title is available - channel names like
"USA A&E SD*" returned unrelated results from fuzzy matching.
Add 13 tests for retry logic (test_dvr_retry.py)
- Add reconnection loop to DVR streaming: on transient connectivity loss (ReadTimeout, ConnectionError, ChunkedEncodingError), the recording task retries the same TS proxy base up to 5 times with a 2-second delay, appending to the existing file. Counter resets on successful data receipt. TS container format tolerates brief discontinuities; MKV remux normalises timestamps.
- Extract _check_recording_cancelled() helper to consolidate repeated stop/delete DB checks across the streaming and reconnection paths.
- Fix comskip treating exit code 1 (no commercials detected) as a fatal error.
Use check=False and handle return codes explicitly: 0 = commercials found,
1 = no commercials (skipped), negative = signal, other = real error.
- Add human-readable reason messages for comskip 'skipped' notifications in
WebSocket handler.
- Remove redundant conditional Channel import in stream_generator._cleanup() that shadowed the module-level import, causing UnboundLocalError when the conditional branch did not execute.
File changes:
Backend
apps/channels/api_views.py — Added extend() action that uses queryset .update() to bypass signals, letting the running task pick up the new end_time via polling. Moved WS recording_stopped notification to synchronous (before HTTP response). Refactored destroy() to delete DB row first, send WS event immediately, defer cleanup to background thread.
apps/channels/signals.py — Added pre_save guard to skip task revocation when status is "recording". Added post_save reentrancy guard to skip artwork prefetch for internal field-only saves (custom_properties, task_id, end_time).
apps/channels/tasks.py — Added 2-second polling loop in streaming that checks for stop, deletion, and extended end_time. Added refresh_from_db() + merge before metadata save to prevent overwriting concurrent artwork prefetch. Replaced inline PeriodicTask cleanup with revoke_task()/_dvr_task_name(). Consolidated redundant DB queries in error path. Fixed TOCTOU os.path.exists + os.remove. Preserved "stopped" status in final-status logic instead of overwriting with "completed".
core/utils.py — Made send_websocket_update gevent-safe by detecting monkey-patching and offloading async_to_sync to gevent's native threadpool, fixing the event loop deadlock that caused cross-recording interference.
apps/proxy/ts_proxy/client_manager.py — Moved _trigger_stats_update to a background thread so remove_client() returns immediately.
apps/proxy/ts_proxy/stream_generator.py — Added Redis-based health check for non-owner workers in _should_send_keepalive(), so keepalives fire correctly even when stream_manager is None.
Frontend
frontend/src/WebSocket.jsx — Added 400ms debounced scheduleRecordingFetch() replacing all inline fetchRecordings() calls. Added recording_extended and recording_updated event handlers. recording_cancelled now does surgical removeRecording() by ID when available.
frontend/src/api.js — Added extendRecording(id, extraMinutes).
frontend/src/components/cards/RecordingCard.jsx — Added Extend menu (+15m/+30m/+1h) for in-progress recordings. Widened Comskip button to stopped/interrupted statuses.
frontend/src/components/forms/ProgramRecordingModal.jsx — Removed manual fetchRecordings() calls, relying on WS-driven debounced refresh.
frontend/src/components/forms/RecordingDetailsModal.jsx — Added stopped to playable statuses. Widened Comskip button. Removed manual fetchRecordings().
frontend/src/components/forms/RecurringRuleModal.jsx — Removed all manual fetchRecordings() calls, relying on WS events.
frontend/src/pages/Guide.jsx — Removed isLoading subscription that caused loading flash on debounced refresh. Removed manual fetchRecordings() after saving a series rule.
frontend/src/store/channels.jsx — Stripped isLoading/error state from fetchRecordings to prevent full-page loading indicators. Added no-op guards in removeRecording.
frontend/src/utils/cards/RecordingCardUtils.js — Added extendRecordingById wrapper.
Tests
apps/channels/tests/test_recording_extend.py (new) — 14 tests: extend endpoint validation, stacked extensions, signal bypass, pre_save guard behavior.
apps/channels/tests/test_recording_stop_cancel.py (new) — 18 tests: stop endpoint, DVR client teardown, cancel was_in_progress flag, run_recording race guards, signal reentrancy guards.
apps/channels/tests/test_ts_proxy_keepalive.py (new) — 21 tests: keepalive owner/non-owner logic, stats update error handling, client removal non-blocking, timing invariants.
apps/channels/tests/test_recording_scheduling.py — Updated mock assertion to match new _stop_dvr_clients call signature.
Summary:
Adds an Extend action for in-progress recordings that bypasses Django signals and lets the running task dynamically pick up the new end_time via its 2-second DB polling loop. Fixes cross-recording interference caused by send_websocket_update deadlocking the gevent event loop. Fixes an artwork race condition where run_recording overwrote concurrent prefetch data. Adds Redis-based keepalive health checks for non-owner workers to prevent DVR read timeouts during source transitions. Replaces manual fetchRecordings() calls throughout the frontend with a debounced WS-driven refresh. 53 new tests.
Fixes#940: Duplicate DVR task executions were caused by ClockedSchedule objects being shared across PeriodicTasks, triggering run_recording multiple
times for the same recording. Added an idempotency guard in run_recording that exits early if status is already 'recording', 'completed', or 'stopped'.
revoke_task() now deletes the PeriodicTask and orphaned ClockedSchedule on execution rather than relying on Celery revocation alone.
Fixes#454: Added POST /api/channels/recordings/{id}/stop/ action that marks the recording as stopped and terminates only the specific DVR proxy client (identified via User-Agent: Dispatcharr-DVR/recording-{id}) without disrupting
simultaneous recordings on the same or other channels. The API returns
immediately; DVR client teardown and task revocation happen in a background thread to avoid 504 timeouts on the recordings list endpoint.
Additional changes:
- destroy() now only calls _stop_dvr_clients() for in-progress recordings, preventing accidental stream termination when deleting completed recordings
- WebSocket events differentiated: recording_stopped, recording_cancelled (in-progress cancel), and recording_cancelled (delete) with was_in_progress flag so the frontend shows distinct "Recording deleted" vs "Recording cancelled" notifications
- Frontend: Stop and Cancel split into separate buttons with confirmation
modals; stopped recordings move immediately to Previously Recorded via
optimistic status bucketing; removed dead 'Stopped' badge state
- 27 new unit tests covering scheduling, idempotency, revocation, and DVR client isolation
If ProxyServer() raises during singleton construction, _instance was left
as _INITIALIZING permanently, causing all subsequent get_instance() callers
to spin in an infinite gevent.sleep() loop. Wrap construction in try/except
and reset _instance to None on failure so callers can retry.
Also replace the non-atomic setnx() + expire() pair in try_acquire_ownership()
with a single atomic SET NX EX call, consistent with the approach already used
in extend_ownership() and eliminating the race window where a crash between the
two calls could leave a key with no TTL (permanent ownership lock).
- Use atomic SET NX EX instead of separate SETNX + EXPIRE to prevent
zombie locks if the process crashes between the two calls
- Replace time.sleep() with gevent.sleep() in get_instance() spin-wait
to avoid blocking the greenlet hub
- Defer cleanup when ownership is lost but clients are still connected,
giving the new owner time to establish its stream before we tear down
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This is a micro-optimization, but prevents checking row.original.streams twice and also guards checking for stale streams unless we know we have streams. The performance hit before would have been tiny, but every bit counts.
Three interrelated bugs cause TS proxy streams to terminate prematurely
in multi-worker uWSGI deployments:
1. Double ProxyServer instantiation: ProxyConfig.ready() in apps/proxy/apps.py
calls TSProxyServer() directly, bypassing get_instance(). The subsequent
TSProxyConfig.ready() call to get_instance() creates a second instance.
Each instance starts its own cleanup thread, but only one holds channel
data — the orphaned cleanup thread cannot extend ownership.
2. Redis flushdb() on every client init: RedisClient.get_client() in
core/utils.py calls flushdb() whenever a new connection is created.
Celery autoscale workers spawning mid-stream nuke all Redis keys
including ownership, client records, and channel metadata.
3. No recovery from expired ownership: get_channel_owner() has a TOCTOU
bug (two separate GET calls in a lambda). extend_ownership() silently
fails when keys expire. Non-owner cleanup unconditionally kills streams
even when the worker holds the stream_manager.
Fixes:
- Use TSProxyServer.get_instance() in ProxyConfig.ready()
- Remove flushdb() from Redis client initialization
- Use sentinel pattern for gevent-safe singleton (threading.Lock does not
work with gevent greenlets)
- Single GET in get_channel_owner() to avoid TOCTOU race
- Re-acquire expired ownership keys in extend_ownership()
- Attempt re-acquisition before cleanup in non-owner path
Relates to #992, #980
