From 108bd52e7fe982bdda5f070c674521b0845fb0ff Mon Sep 17 00:00:00 2001 From: SergeantPanda Date: Sun, 17 May 2026 20:23:57 -0500 Subject: [PATCH] perf: replace local cache with django-redis for improved EPG caching --- CHANGELOG.md | 2 ++ dispatcharr/settings.py | 32 +++++++++++++++++++------------- 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 651d9492..11121f3d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -82,6 +82,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **`_cleanup_local_resources` skipped `ClientManager.stop()` on channel removal.** `ProxyServer._cleanup_local_resources` deleted each channel's `ClientManager` entry with `del`, which removed it from the dict but gave the object no signal to terminate. The per-channel heartbeat greenlet inside the manager continued running until it next checked its running flag and found the channel absent from Redis. The entry is now removed with `pop()` and `stop()` is called on the captured manager before it is discarded, terminating the heartbeat immediately on channel cleanup. - **XC profile `exp_date` not updating on account refresh.** `refresh_account_profiles` saved the freshly-fetched `custom_properties` with `update_fields=['custom_properties']`, which excluded `exp_date` from the SQL `UPDATE`. The model's `save()` method parses the new expiry from `custom_properties` and assigns it to `self.exp_date`, but that value was silently dropped because the column was not listed in `update_fields`. Added `'exp_date'` to the `update_fields` list so both columns are written together. - **~25-second transcode startup delay and worker freeze when starting ffmpeg under gevent+uWSGI.** Enabling gevent cooperative multitasking (see Changed above) exposed a deadlock: `fork()` hangs indefinitely in gevent's `_before_fork` pthread_atfork handler when called from any thread while gevent is running - including from real OS threads. `subprocess.Popen`, which all three ffmpeg spawn sites used, calls `fork()` internally, stalling the uWSGI worker for ~25 seconds or freezing it entirely and blocking all other clients on that worker. - `input/manager.py`: replaced `subprocess.Popen` with `os.posix_spawn` + a minimal `_SpawnedProcess` wrapper. `os.posix_spawn` is POSIX-specified to skip pthread_atfork handlers entirely. @@ -107,6 +108,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Performance +- **EPG HTTP response cache replaced with `django-redis`.** The default Django cache backend was `LocMemCache`, an in-process memory store. With multiple uWSGI workers, each worker independently generated and cached the full EPG XML in its own heap; with 4 workers the peak memory cost was up to 4 times the size of the EPG document. The cache backend is now `django_redis.cache.RedisCache`, backed by the same Redis instance used by `channels_redis`, so a single cached EPG copy is shared across all workers. - **`AutoSyncAdvanced` and `LogoForm` are now lazy-loaded in the M3U group filter.** Both components are large and only needed when the user opens the gear modal or logo upload modal. Wrapping them in `React.lazy` + `Suspense` removes them from the initial bundle and defers their parse/execute cost until first use. — Thanks [@nick4810](https://github.com/nick4810) - **Auto-sync at scale**: the new override-aware sync flow is more capable than the prior path but the implementation choices below keep it viable on libraries with thousands of channels. — Thanks [@CodeBormen](https://github.com/CodeBormen) - **Bulk writes throughout `sync_auto_channels`.** Per-row `Channel.objects.create()` and `.save()` calls were replaced with `bulk_create()` and `bulk_update()` paths that batch the entire group's create + update sets into single round-trips. The renumber pass collects all dirty channels into one list and flushes with a single `bulk_update` at the end of the loop. `ChannelStream.order` writes were similarly consolidated into a single `bulk_update`. diff --git a/dispatcharr/settings.py b/dispatcharr/settings.py index 530b56c7..e8818108 100644 --- a/dispatcharr/settings.py +++ b/dispatcharr/settings.py @@ -116,19 +116,6 @@ DATABASE_CONN_MAX_AGE = 0 # Close after each request; gevent makes per-greenlet # Disable atomic requests for performance-sensitive views ATOMIC_REQUESTS = False -# Cache settings - add caching for EPG operations -CACHES = { - "default": { - "BACKEND": "django.core.cache.backends.locmem.LocMemCache", - "LOCATION": "dispatcharr-epg-cache", - "TIMEOUT": 3600, # 1 hour cache timeout - "OPTIONS": { - "MAX_ENTRIES": 10000, - "CULL_FREQUENCY": 3, # Purge 1/3 of entries when max is reached - }, - } -} - # Timeouts for external connections REQUESTS_TIMEOUT = 30 # Seconds for external API requests @@ -198,6 +185,25 @@ CHANNEL_LAYERS = { }, } +_django_redis_opts = { + "CLIENT_CLASS": "django_redis.client.DefaultClient", +} +if REDIS_SSL: + # rediss:// in the URL already enables SSL; pass cert paths and verify + # settings separately via CONNECTION_POOL_KWARGS. + _django_redis_opts["CONNECTION_POOL_KWARGS"] = { + k: v for k, v in REDIS_SSL_PARAMS.items() if k != "ssl" + } + +CACHES = { + "default": { + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": _channels_redis_url, + "TIMEOUT": 3600, + "OPTIONS": _django_redis_opts, + } +} + # PostgreSQL TLS configuration (defined before DATABASES for module-level access) POSTGRES_SSL = os.environ.get("POSTGRES_SSL", "false").lower() == "true" POSTGRES_SSL_MODE = os.environ.get("POSTGRES_SSL_MODE", "verify-full")