Updated the import mechanism for celery_app to avoid issues with lazy-loading, ensuring the default Celery app is correctly configured in production environments. Removed the __getattr__ function for celery_app, simplifying the module structure.
- Added `dispatcharr.settings_test` for isolated testing, automatically creating an empty PostgreSQL test database and ensuring transaction isolation during tests.
- Updated `manage.py` to switch to the test settings when running tests.
- Enhanced the CONTRIBUTING.md file with detailed instructions on running the backend test suite and handling Celery tasks in tests.
- Refactored test cases to use static methods for `worker_id` in `test_process_label.py` and adjusted assertions in `test_user_preferences.py` for better clarity and correctness.
- Implemented `ensure_custom_properties_dict()` to normalize custom properties across various models and serializers, addressing issues with legacy JSON-encoded strings.
- Updated M3U account and channel group models to ensure custom properties are consistently stored as dictionaries during save operations.
- Enhanced Celery task management by ensuring old DB connections are closed before and after tasks, improving reliability and preventing errors during account refresh operations. (Fixes#1338)
- Added a new function `_is_uwsgi_worker` to accurately identify when the application is running within a uWSGI worker.
- Updated `get_process_role` to label processes as "uwsgi" based on the new function or command-line arguments, improving process role accuracy.
- Expanded unit tests to cover various scenarios for uWSGI role detection, ensuring robust validation of the new logic.
- Introduced `lookup_by_name_year` function to limit database scans for movies and series without TMDB/IMDB IDs, enhancing performance by scoping lookups to current batch names.
- Updated `process_movie_batch` and `process_series_batch` to utilize the new lookup function, reducing memory usage and improving efficiency.
- Registered `refresh_vod_content` and `batch_refresh_series_episodes` for post-task memory cleanup in Celery, ensuring better resource management during VOD content refreshes.
- Introduced a new module to determine the process role based on command-line arguments, enhancing PostgreSQL connection labeling for better monitoring.
- Updated the database connection parameters to include the application name derived from the process role.
- Added unit tests to validate the process role detection logic for different scenarios, including Celery and uWSGI.
- Introduced a temporary staging table for efficient batch processing of EPG program inserts, reducing memory and I/O contention during updates.
- Enhanced the `parse_programs_for_source` function to stream parsed rows into the staging table before swapping them atomically into the main program data.
- Added unit tests to validate the new staging and swapping logic, ensuring existing programs are preserved during failures and that batch processing works as intended.
- Introduced DATABASE_POOL_CONN_MAX_LIFETIME to manage pooled connection lifecycle, improving performance and resource management.
- Updated database engine path to use 'dispatcharr.db.backends.postgresql_psycopg3' for consistency.
- Enhanced logging configuration for geventpool connection lifecycle tracking.
- Moved matching logic to a dedicated module for better organization and testability.
- Made single-channel auto-matching asynchronous, allowing for larger EPG libraries without hitting HTTP timeouts.
- Enhanced memory management and throughput during EPG matching, including optimizations for fuzzy matching and bulk processing.
- Fixed various reliability issues in the auto-matching process, ensuring accurate channel assignments and improved UI feedback.
- Updated API views and frontend components to reflect changes in the matching process and provide real-time notifications.
- Added tests for EPG matching functionality and name normalization.
- Single-channel and selected-channel auto-match always run, even when the channel already has EPG assigned; match-all (no channel IDs) still only processes channels without EPG.
- timeshift_proxy(): drop the resolve_channel_by_provider_stream_id
fallback. The client only ever sends Dispatcharr's internal Channel.id,
so resolve by id and return 404 on miss. Remove the dead `channel = None`
sentinel and the now-unused import.
- Delete the now-orphaned resolve_channel_by_provider_stream_id helper and
its local Stream import from apps/channels/utils.py (zero remaining refs).
- Import django.core.cache.cache at module top-level instead of inside
_get_cached_format_index / _set_cached_format_index.
- Fix the stale dispatcharr/urls.py comment: the "duration" slot carries
Channel.id, not the provider stream_id.
- Genericize provider host references in code comments.
- Switch engine to postgresql_psycopg3 (PR had wrong engine name,
and we're migrating from psycopg2 to psycopg3 as part of this work)
- Set CONN_MAX_AGE=0 (required by geventpool; PR had None)
- Reduce MAX_CONNS 20→8, REUSE_CONNS 10→3 to avoid exhausting
pg max_connections across 4 uWSGI workers + Celery
- Add pool=False to explicitly disable Django's native psycopg3 pool
Adds native catch-up/timeshift replay for Xtream Codes providers through
the same HTTPStreamReader transport pipeline as live TV.
Timeshift proxy (apps/timeshift/):
- URL cascade: 3 candidate timestamp formats per provider, per-account
format cache for fast-forward seek performance
- MPEG-TS preamble stripping (shared with HTTPStreamReader)
- Stats integration: timeshift viewers appear on /stats with TIMESHIFT badge
- Auth via hmac.compare_digest on XC password
Catchup detection — denormalized for zero-cost output queries:
- Stream.is_catchup + Stream.catchup_days populated at XC import time
- Channel.has_catchup + Channel.catchup_days + Channel.catchup_provider_stream_id
rolled up via ChannelStream post_save signal (UI path) and explicit SQL
after bulk_create (import path)
- _xc_channel_entry() reads denormalized fields instead of per-channel
custom_properties JSON introspection (eliminates N+1 queries)
- Migration 0038 backfills existing data via raw SQL
XC API enhancements:
- server_info.timezone + start/end + time_now use configured timezone
(triple consistency rule — fixes wrong-programme-plays bug)
- Dynamic has_archive flag + auto prev_days for catch-up channels
- XMLTV timestamps rewritten to local timezone for catch-up clients
HTTPStreamReader extended (apps/proxy/live_proxy/input/http_streamer.py):
- 1 MB pipe buffer via fcntl F_SETPIPE_SZ (eliminates producer/consumer
ping-pong that halved throughput)
- Pre-opened response= for URL cascade workflows
- strip_ts_preamble= for XC servers emitting PHP warnings before TS
- find_ts_sync() as shared utility
- Builds on upstream O_NONBLOCK + select() write loop
Provider stream_id lookup order:
- stream_xc() and xc_get_epg() try internal Channel.id first, fall back
to provider stream_id only when needed (avoids unconditional query on
every request)
Also includes:
- VOD provider cascade in stream_vod() — iterates all M3U relations by
priority when first provider is at capacity
- Defensive null-safety: custom_sid: None → "" in get_live_streams,
get_vod_streams, get_vod_info, get_series_info (fixes iPlayTV crash on
JSON null for string fields)
- Timeshift settings UI (timezone selector, debug toggle)
- StreamConnectionCard violet TIMESHIFT badge
- Orphan cleanup skips timeshift_* virtual channels
- Enable gevent cooperative multitasking in all uWSGI worker configs
(gevent-early-monkey-patch + import dispatcharr.gevent_patch)
- Rewrite WebSocket group sends to bypass asyncio in gevent workers:
_gevent_ws_send() replicates the channels_redis 4.x wire format
directly via synchronous Redis so send_websocket_update() and
_send_async() no longer fail silently after epoll is patched out
- Fix PostgreSQL connection exhaustion: CONN_MAX_AGE=0 + explicit
close_old_connections() in stream manager and cleanup watchdog loops
- Fix stream proxy race: register client before the connect-wait loop
so the cleanup watchdog never sees zero clients on a live channel
- Channel list/logo/profile queryset optimisations: conditional DISTINCT,
EXISTS semi-joins for filter-options, channel_count annotation to
eliminate N+1 in LogoSerializer, prefetched memberships in
ChannelProfileSerializer
- JsonResponse for channel ID list and summary endpoints
Plugins live in /data/plugins/<slug>/plugin.py, outside INSTALLED_APPS,
so celery's autodiscover_tasks() never imports them. Any plugin using
module-level @shared_task for cron-scheduled work therefore has its
task unregistered with the worker after every worker restart, until
something else (a connect/utils.trigger_event call, etc.) lazily
imports the plugin module via PluginManager.discover_plugins.
In the meantime, beat fires the periodic task on time and the worker
rejects it:
ERROR celery.worker.consumer.consumer Received unregistered task of
type 'telegram_alerts.send_daily_report'.
PeriodicTask.last_run_at advances anyway, so the failure is silent at
the default INFO log level.
Fix: hook celery's worker_ready signal in dispatcharr/celery.py to
call PluginManager.discover_plugins(sync_db=False) on every worker
boot. Importing the plugin modules runs their @shared_task decorators,
registering the tasks before beat starts firing.
Exception handling is intentionally broad: if one plugin's plugin.py
has an import error, the worker must still come up — discovery failure
should not be a fatal error for the whole worker process.
Tests in tests/test_celery_plugin_discovery.py verify:
- the handler calls discover_plugins(sync_db=False)
- the handler swallows PluginManager.get() exceptions
- the handler swallows discover_plugins() exceptions
- the handler is actually connected to the worker_ready signal
Confirmed the test fails against current dev with the exact
'cannot import name discover_plugins_on_worker_ready' that motivates
this change, and passes with the fix.
- Set `DEFAULT_PERMISSION_CLASSES` to `Authenticated` in the DRF configuration.
- Explicitly marked the HDHomeRun discovery endpoints (`DiscoverAPIView`, `LineupAPIView`, `LineupStatusAPIView`, `HDHRDeviceXMLAPIView`) and the version endpoint with `permission_classes = [AllowAny]` to document their intentionally public access now that the global default is `Authenticated`.
