The custom psycopg3 gevent pool only evaluates connection age inside
get()/put(). A connection checked out by a greenlet that dies without
returning it — e.g. a live-stream / channel-switch teardown path that
never calls close_old_connections() — is never aged out. It stays counted
against MAX_CONNS but is no longer in the free queue, so once MAX_CONNS
connections leak per worker, get() blocks forever and every DB-backed
request on that worker hangs with a 60s/504 until the container is
restarted. DB-free endpoints (e.g. /api/core/version/) stay fast, which
is the tell.
Add a background reaper greenlet (one per worker, started lazily on first
get() so the gevent hub is up) that periodically closes connections which
are both expired (older than max_lifetime) and IDLE (no in-flight
transaction). The IDLE guard means an active stream or in-flight request
is never disrupted, and get() already refuses to hand out an expired
connection so a reaped connection can't be in active use. This frees the
leaked slot and lets the worker recover on its own — no restart.
Side effect: DATABASE_POOL_CONN_MAX_LIFETIME now actually reclaims leaked
connections (previously it only recycled healthy idle ones). Lower it to
shorten leak-recovery time. Reaper cadence is configurable via
DATABASE_POOL_REAP_INTERVAL (default 30s; 0 disables).
Also covers the M3U-refresh INTRANS leak family (#1338).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Introduced a new module to determine the process role based on command-line arguments, enhancing PostgreSQL connection labeling for better monitoring.
- Updated the database connection parameters to include the application name derived from the process role.
- Added unit tests to validate the process role detection logic for different scenarios, including Celery and uWSGI.
- Introduced DATABASE_POOL_CONN_MAX_LIFETIME to manage pooled connection lifecycle, improving performance and resource management.
- Updated database engine path to use 'dispatcharr.db.backends.postgresql_psycopg3' for consistency.
- Enhanced logging configuration for geventpool connection lifecycle tracking.
