perf: replace local cache with django-redis for improved EPG caching
Some checks failed
Base Image Build / prepare (push) Has been cancelled
CI Pipeline / prepare (push) Has been cancelled
Build and Push Multi-Arch Docker Image / build-and-push (push) Has been cancelled
Base Image Build / docker (amd64, ubuntu-24.04) (push) Has been cancelled
Base Image Build / docker (arm64, ubuntu-24.04-arm) (push) Has been cancelled
Base Image Build / create-manifest (push) Has been cancelled
CI Pipeline / docker (amd64, ubuntu-24.04) (push) Has been cancelled
CI Pipeline / docker (arm64, ubuntu-24.04-arm) (push) Has been cancelled
CI Pipeline / create-manifest (push) Has been cancelled

This commit is contained in:
SergeantPanda 2026-05-17 20:23:57 -05:00
parent 28347a239b
commit 108bd52e7f
2 changed files with 21 additions and 13 deletions

View file

@ -82,6 +82,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **`_cleanup_local_resources` skipped `ClientManager.stop()` on channel removal.** `ProxyServer._cleanup_local_resources` deleted each channel's `ClientManager` entry with `del`, which removed it from the dict but gave the object no signal to terminate. The per-channel heartbeat greenlet inside the manager continued running until it next checked its running flag and found the channel absent from Redis. The entry is now removed with `pop()` and `stop()` is called on the captured manager before it is discarded, terminating the heartbeat immediately on channel cleanup.
- **XC profile `exp_date` not updating on account refresh.** `refresh_account_profiles` saved the freshly-fetched `custom_properties` with `update_fields=['custom_properties']`, which excluded `exp_date` from the SQL `UPDATE`. The model's `save()` method parses the new expiry from `custom_properties` and assigns it to `self.exp_date`, but that value was silently dropped because the column was not listed in `update_fields`. Added `'exp_date'` to the `update_fields` list so both columns are written together.
- **~25-second transcode startup delay and worker freeze when starting ffmpeg under gevent+uWSGI.** Enabling gevent cooperative multitasking (see Changed above) exposed a deadlock: `fork()` hangs indefinitely in gevent's `_before_fork` pthread_atfork handler when called from any thread while gevent is running - including from real OS threads. `subprocess.Popen`, which all three ffmpeg spawn sites used, calls `fork()` internally, stalling the uWSGI worker for ~25 seconds or freezing it entirely and blocking all other clients on that worker.
- `input/manager.py`: replaced `subprocess.Popen` with `os.posix_spawn` + a minimal `_SpawnedProcess` wrapper. `os.posix_spawn` is POSIX-specified to skip pthread_atfork handlers entirely.
@ -107,6 +108,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Performance
- **EPG HTTP response cache replaced with `django-redis`.** The default Django cache backend was `LocMemCache`, an in-process memory store. With multiple uWSGI workers, each worker independently generated and cached the full EPG XML in its own heap; with 4 workers the peak memory cost was up to 4 times the size of the EPG document. The cache backend is now `django_redis.cache.RedisCache`, backed by the same Redis instance used by `channels_redis`, so a single cached EPG copy is shared across all workers.
- **`AutoSyncAdvanced` and `LogoForm` are now lazy-loaded in the M3U group filter.** Both components are large and only needed when the user opens the gear modal or logo upload modal. Wrapping them in `React.lazy` + `Suspense` removes them from the initial bundle and defers their parse/execute cost until first use. — Thanks [@nick4810](https://github.com/nick4810)
- **Auto-sync at scale**: the new override-aware sync flow is more capable than the prior path but the implementation choices below keep it viable on libraries with thousands of channels. — Thanks [@CodeBormen](https://github.com/CodeBormen)
- **Bulk writes throughout `sync_auto_channels`.** Per-row `Channel.objects.create()` and `.save()` calls were replaced with `bulk_create()` and `bulk_update()` paths that batch the entire group's create + update sets into single round-trips. The renumber pass collects all dirty channels into one list and flushes with a single `bulk_update` at the end of the loop. `ChannelStream.order` writes were similarly consolidated into a single `bulk_update`.

View file

@ -116,19 +116,6 @@ DATABASE_CONN_MAX_AGE = 0 # Close after each request; gevent makes per-greenlet
# Disable atomic requests for performance-sensitive views
ATOMIC_REQUESTS = False
# Cache settings - add caching for EPG operations
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.locmem.LocMemCache",
"LOCATION": "dispatcharr-epg-cache",
"TIMEOUT": 3600, # 1 hour cache timeout
"OPTIONS": {
"MAX_ENTRIES": 10000,
"CULL_FREQUENCY": 3, # Purge 1/3 of entries when max is reached
},
}
}
# Timeouts for external connections
REQUESTS_TIMEOUT = 30 # Seconds for external API requests
@ -198,6 +185,25 @@ CHANNEL_LAYERS = {
},
}
_django_redis_opts = {
"CLIENT_CLASS": "django_redis.client.DefaultClient",
}
if REDIS_SSL:
# rediss:// in the URL already enables SSL; pass cert paths and verify
# settings separately via CONNECTION_POOL_KWARGS.
_django_redis_opts["CONNECTION_POOL_KWARGS"] = {
k: v for k, v in REDIS_SSL_PARAMS.items() if k != "ssl"
}
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": _channels_redis_url,
"TIMEOUT": 3600,
"OPTIONS": _django_redis_opts,
}
}
# PostgreSQL TLS configuration (defined before DATABASES for module-level access)
POSTGRES_SSL = os.environ.get("POSTGRES_SSL", "false").lower() == "true"
POSTGRES_SSL_MODE = os.environ.get("POSTGRES_SSL_MODE", "verify-full")