- Added `dispatcharr.settings_test` for isolated testing, automatically creating an empty PostgreSQL test database and ensuring transaction isolation during tests.
- Updated `manage.py` to switch to the test settings when running tests.
- Enhanced the CONTRIBUTING.md file with detailed instructions on running the backend test suite and handling Celery tasks in tests.
- Refactored test cases to use static methods for `worker_id` in `test_process_label.py` and adjusted assertions in `test_user_preferences.py` for better clarity and correctness.
- Improved `xc_get_vod_streams` and `xc_get_series` functions to utilize a new helper method for fetching distinct relations based on account priority, significantly reducing memory usage and response times for large libraries.
- Updated the handling of VOD streams to avoid unnecessary ORM instantiation, enhancing performance for endpoints dealing with extensive movie and series data.
- Added comprehensive tests to ensure the correctness of the new logic and verify that the highest priority relations are correctly selected.
- Updated the `generate_dummy_programs` function to include `export_lookback` and `export_cutoff` parameters, allowing for more precise control over the EPG data generation window.
- Added a new test case to verify that future events are correctly represented in the grid window, ensuring upcoming fillers are displayed as expected.
- Moved the `programme_index` from the `EPGSource` model to a new `EPGSourceIndex` table, ensuring that the large JSON blob is only loaded when explicitly accessed, thus improving query performance and memory efficiency.
- Updated related queries and API views to utilize the new structure, including adjustments to EPG generation and import logic to prevent unnecessary data loading.
- Enhanced memory management in the EPG grid endpoint to reduce worker RSS during response handling.
- Moved all XMLTV output logic from `apps/output/views.py` to `apps/output/epg.py`, streamlining the codebase and maintaining the existing HTTP endpoint functionality.
- Updated related tests and references to ensure proper integration with the new module structure.
- Streamlined `generate_epg()` to incrementally stream EPG data without loading the entire guide, improving memory efficiency.
- Reduced database load by deferring the fetching of large `programme_index` blobs, enhancing response times for EPG generation.
- Introduced a composite index on `(epg_id, id)` in `ProgramData` to optimize query performance during EPG exports.
- Updated tests to ensure proper functionality and performance of new features.
This PR updates the non_fqdn_pattern regex to allow underscores in non-FQDN hostnames. Such hostnames are commonly used in Docker networks, but Dispatcharr currently rejects them due to the overly strict regex.
When an Xtream Codes provider returns no live streams on a routine refresh
(a transient upstream failure, a fetch exception, or no enabled category
matching), collect_xc_streams() returns an empty list. The refresh then fell
through to stale-marking and sync_auto_channels, which -- with nothing seen
this refresh -- deletes the account's entire auto-created channel lineup via
its per-group "no streams remaining" branch.
Guard the XC branch of _refresh_single_m3u_account_impl: on an empty result,
set the account to ERROR and return before stale-marking and auto-sync,
mirroring the standard-path empty/failed-download guards already in the
function. A transient empty fetch can no longer destroy channels.
Adds apps/m3u/tests/test_xc_empty_fetch_guard.py covering both the abort
(sync not called, channels preserved, streams not marked stale, status ERROR)
and the healthy path (non-empty fetch still runs sync).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The range-conflict warning classified each channel in the configured range as either this config's own auto-sync output or a real conflict, comparing each occupant against the source group. When a group sets a group_override, auto-sync creates its channels in the override target group, so the config's own channels failed that comparison and were misclassified as a conflict.
- Add effectiveSyncGroupId to resolve the group the sync's channels actually land in (the group_override target when set, otherwise the source group).
- Compare occupants against that effective target, so the config's own output is recognized while genuine conflicts (manual channels, channels from another account, channels in a different group, user-pinned numbers) still warn.
- Add Vitest coverage for the helper, the override case, and an over-suppression guard, plus a backend test asserting the numbers-in-range endpoint reports the override target group.
- Optimized `generate_epg()` to reduce database queries by prefetching ordered channel streams and implementing a new method for custom dummy epg name matching.
- Enhanced handling of dummy EPG sources to minimize SQL round-trips, improving performance for large guides with numerous custom-dummy channels.
- Implemented robust DB connection management in `refresh_epg_data` to handle transient errors, ensuring reliable status updates for EPG sources.
- Added retry logic for database queries and improved error handling to prevent task failures from unhandled exceptions.
- Updated EPG source status handling to ensure accurate reporting of errors and terminal states during refresh operations.
- Updated `get_streams` method in `ChannelSerializer` to use prefetched `channelstream_set`, improving performance by reducing database queries.
- Added `ChannelListIncludeStreamsQueryTests` to ensure that the number of queries remains stable as channels grow, verifying efficient stream inclusion in API responses.
- Implemented `ensure_custom_properties_dict()` to normalize custom properties across various models and serializers, addressing issues with legacy JSON-encoded strings.
- Updated M3U account and channel group models to ensure custom properties are consistently stored as dictionaries during save operations.
- Enhanced Celery task management by ensuring old DB connections are closed before and after tasks, improving reliability and preventing errors during account refresh operations. (Fixes#1338)
- Implemented `normalize_server_url()` to standardize account server URLs, ensuring that on-demand live URLs are built correctly without including API endpoints or query parameters.
- Updated `get_transformed_credentials()` and stream URL generation in `M3UMovieRelation` and `M3UEpisodeRelation` to utilize the new normalization function, improving URL handling for Xtream Codes accounts.
- Enhanced the live proxy to release geventpool DB connections more effectively across various paths.
- Implemented `close_old_connections()` in `stream_ts()`, `generate_stream_url()`, `get_stream_info_for_switch()`, `get_alternate_streams()`, `get_connections_left()`, and the cleanup process in `StreamGenerator`, ensuring that clients do not hold onto pool slots unnecessarily during streaming operations.
- Added a new function `_is_uwsgi_worker` to accurately identify when the application is running within a uWSGI worker.
- Updated `get_process_role` to label processes as "uwsgi" based on the new function or command-line arguments, improving process role accuracy.
- Expanded unit tests to cover various scenarios for uWSGI role detection, ensuring robust validation of the new logic.
- Introduced `lookup_by_name_year` function to limit database scans for movies and series without TMDB/IMDB IDs, enhancing performance by scoping lookups to current batch names.
- Updated `process_movie_batch` and `process_series_batch` to utilize the new lookup function, reducing memory usage and improving efficiency.
- Registered `refresh_vod_content` and `batch_refresh_series_episodes` for post-task memory cleanup in Celery, ensuring better resource management during VOD content refreshes.
- Improved memory cleanup by invoking `gc.collect()` after batch processing in `process_m3u_batch_direct` and `collect_xc_streams`, ensuring timely release of resources.
- Added tests to verify that garbage collection is triggered appropriately after batch operations.
- Introduced a new module to determine the process role based on command-line arguments, enhancing PostgreSQL connection labeling for better monitoring.
- Updated the database connection parameters to include the application name derived from the process role.
- Added unit tests to validate the process role detection logic for different scenarios, including Celery and uWSGI.
- Replaced the plugin listing mechanism with a more efficient handler iteration for event actions.
- Enhanced logging to provide clearer insights into the dispatching process and plugin action execution.
- Added error handling to ensure that failures in one plugin action do not block subsequent actions.
- Updated tests to cover new behavior and ensure proper handling of enabled and disabled plugins.
- Introduced a temporary staging table for efficient batch processing of EPG program inserts, reducing memory and I/O contention during updates.
- Enhanced the `parse_programs_for_source` function to stream parsed rows into the staging table before swapping them atomically into the main program data.
- Added unit tests to validate the new staging and swapping logic, ensuring existing programs are preserved during failures and that batch processing works as intended.
- Improved the handling of channel shutdown delays to ensure they reset correctly upon client reconnections, preventing premature shutdowns.
- Implemented logic to cancel pending shutdowns when clients reconnect, ensuring proper resource management and preventing client disconnect issues across uWSGI workers.
- Enhanced logging for shutdown processes and client management to provide clearer insights during channel operations.
- Added calls to `close_old_connections()` in `stream_vod()` and `build_vod_stats_data()` to prevent connection leaks during long-lived streaming responses and background stats refreshes.
- Improved connection handling in various components to ensure proper resource cleanup and prevent blocking issues.
- Improved database connection management by ensuring `close_old_connections()` is called in various methods to prevent connection leaks.
- Updated event dispatching to run asynchronously in gevent, preventing blocking during live-proxy and streaming paths.
- Added calls to `close_old_connections()` in `run_action()` and `stop_plugin()` methods to prevent connection leaks after plugin execution.
- Updated documentation to clarify connection handling for plugins, emphasizing the importance of cleanup in long-running tasks and event hooks.
- Improved channel teardown process to prevent lingering upstream connections and ensure proper resource cleanup.
- Enhanced client management by implementing checks for ghost clients and ensuring accurate client disconnection handling.
- Updated logging to provide clearer insights during channel initialization and teardown, including handling of unavailable channels.
- Refined stream manager behavior to manage ownership transitions more effectively and prevent blocking during shutdown.
- Improved handling of channel teardown to prevent client reconnect issues across uWSGI workers, ensuring proper resource cleanup and ownership management.
- Added functionality to clear all client entries from Redis for a channel, enhancing client management during shutdown.
- Updated logging and response mechanisms to inform clients of channel availability during teardown,.
- Enhanced stream manager behavior to ensure upstream connections are properly managed during ownership transitions.
- Enhanced the `stop_channel` method to prevent blocking during teardown by logging stop events asynchronously.
- Updated stream buffer behavior to ignore late writes during shutdown, improving resource management.
- Introduced DATABASE_POOL_CONN_MAX_LIFETIME to manage pooled connection lifecycle, improving performance and resource management.
- Updated database engine path to use 'dispatcharr.db.backends.postgresql_psycopg3' for consistency.
- Enhanced logging configuration for geventpool connection lifecycle tracking.
Aligns catch-up with how live and VOD manage provider capacity:
- Reserve a provider profile slot (connection_pool) before every upstream
connect, walking the account's active profiles default-first when the
default is at capacity (profile_full/credential_full are transient and
never mark the account decisive). Credentials for the reserved profile
are resolved via get_transformed_credentials — the same credential
extraction live playback uses — so pool accounting and real upstream
usage always agree. All eligible streams blocked on capacity alone
returns 503 (the VOD pool-exhausted precedent).
- Release exactly once via a one-shot Redis ownership token consumed with
a transactional GET+DEL: the generator finally, the response-close
wrapper, failed failover attempts and session takeover all share it, so
no path can double-decrement and a client disconnecting before the
first chunk still releases (Django registers the iterator's close() as
a resource closer). Failures between reservation and the streaming
response owning the slot release before propagating; an ownership-token
write failure releases directly and reports transient unavailability.
- One catch-up session per user and channel: a new request (programme
jump or seek) displaces the user's previous session on that channel —
its slot is released synchronously, its stats are unregistered, and its
generator is stopped through the standard stop-key mechanism — so rapid
seeking cannot stack upstream provider connections.
- rollup_channel_catchup_fields self-heals channels left flagged
is_catchup with no remaining catch-up stream (outside the
account-scoped CTE), covering bulk removals on manual/multi-provider
channels regardless of how the link rows disappeared. A regression test
locks the ChannelStream post_delete signal firing on queryset bulk
deletes.
Backend test suite grows to 94 (slot reservation/release on every
failover outcome, profile walk, mixed capacity-vs-upstream precedence,
exception-path release, token exactly-once semantics, takeover scoping
and ordering, never-started-generator release, rollup self-heal).
