perf(m3u): Enhance memory management and performance during XC stream processing

- Improved memory cleanup by invoking `gc.collect()` after batch processing in `process_m3u_batch_direct` and `collect_xc_streams`, ensuring timely release of resources.
- Added tests to verify that garbage collection is triggered appropriately after batch operations.

CHANGELOG.md

@@ -34,6 +34,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Performance
 
+- **XC live refresh releases bulk catalog data sooner during batch processing.** After filtering the single `get_all_live_streams` response, the full provider catalog list is dropped before batch DB work. `process_m3u_batch_direct` (the path XC refreshes use) now runs `gc.collect()` after each batch and clears batch slice references as thread futures complete. Large structures are still `del`'d in the refresh `finally` block before Celery's `task_postrun` runs `cleanup_memory()`.
 - **EPG programme parse now streams through PostgreSQL staging instead of holding the full catalogue in Python.** `parse_programs_for_source` writes parsed rows into a session-scoped temp table in batches (`_EPG_PARSE_BATCH_SIZE=2500`), then atomically swaps them into `ProgramData` with batched `DELETE ... RETURNING` + `INSERT` (`_EPG_SWAP_BATCH_SIZE=5000`) so Postgres never materializes the entire guide in one statement. Peak Celery memory during large XMLTV refreshes drops sharply compared with building a monolithic in-memory list before bulk insert. The byte-offset programme index build is deferred until after the swap completes so index construction no longer competes with parse for memory and I/O. `refresh_epg_data` closes its DB connection in `finally`, and `build_programme_index_task` is registered as a memory-intensive Celery task. SQLite/dev installs keep an in-memory fallback path.
 - **Pooled PostgreSQL connections now rotate on a bounded lifetime.** psycopg3 client-side cache grows on handles kept open indefinitely by `django-db-geventpool`; recycling uWSGI workers would interrupt live streams. A thin custom backend (`dispatcharr.db.backends.postgresql_psycopg3`) closes and replaces pool connections after `DATABASE_POOL_CONN_MAX_LIFETIME` seconds (default 600, env-overridable; set `0` to disable) on checkout and return while preserving warm-pool reuse within the window. (Fixes #1343)
 - **Schedules Direct refresh only fetches guide data for mapped channels.** Schedule MD5 checks and schedule downloads now target mapped lineup stations instead of the entire lineup, and schedule MD5 cache for unmapped stations is pruned each refresh. Unmapped lineup entries no longer trigger wasted schedule API calls when their MD5 changes.

apps/m3u/tasks.py

@@ -790,6 +790,7 @@ def collect_xc_streams(account_id, enabled_groups):
     """Collect all XC streams in a single API call and filter by enabled groups."""
     account = M3UAccount.objects.get(id=account_id)
     all_streams = []
+    filtered_count = 0
 
     # Create a mapping from category_id to group info for filtering
     enabled_category_ids = {}
@@ -819,7 +820,6 @@ def collect_xc_streams(account_id, enabled_groups):
             logger.info(f"Retrieved {len(all_xc_streams)} total live streams from provider")
 
             # Filter streams based on enabled categories
-            filtered_count = 0
             for stream in all_xc_streams:
                 # Fall back to a generated name if the provider returns null/empty
                 stream_name = stream.get("name") or f"{account.name} - {stream.get('stream_id', 'Unknown')}"
@@ -862,11 +862,17 @@ def collect_xc_streams(account_id, enabled_groups):
                     all_streams.append(stream_data)
                     filtered_count += 1
 
+            # Drop the full provider catalog before returning; only filtered rows are needed.
+            del all_xc_streams
+            gc.collect()
+
     except Exception as e:
         logger.error(f"Failed to fetch XC streams: {str(e)}")
         return []
 
-    logger.info(f"Filtered {filtered_count} streams from {len(enabled_category_ids)} enabled categories")
+    logger.info(
+        f"Filtered {filtered_count} streams from {len(enabled_category_ids)} enabled categories"
+    )
     return all_streams
 
 def process_xc_category_direct(account_id, batch, groups, hash_keys):
@@ -1270,6 +1276,7 @@ def process_m3u_batch_direct(account_id, batch, groups, hash_keys):
 
     # Free batch data structures (reference-counted deallocation)
     del streams_to_create, streams_to_update, stream_hashes, existing_streams
+    gc.collect()
 
     return retval
 
@@ -3338,6 +3345,8 @@ def _refresh_single_m3u_account_impl(account_id):
                     except Exception as e:
                         logger.error(f"Error in thread batch {batch_idx}: {str(e)}")
                         completed_batches += 1  # Still count it to avoid hanging
+                    finally:
+                        batches[batch_idx] = None
 
             logger.info(f"Thread-based processing completed for account {account_id}")
         else:
@@ -3449,6 +3458,8 @@ def _refresh_single_m3u_account_impl(account_id):
                         except Exception as e:
                             logger.error(f"Error in XC thread batch {batch_idx}: {str(e)}")
                             completed_batches += 1  # Still count it to avoid hanging
+                        finally:
+                            batches[batch_idx] = None
 
                 logger.info(f"XC thread-based processing completed for account {account_id}")
 

apps/m3u/tests/test_memory_cleanup.py

@@ -33,6 +33,24 @@ class ProcessM3UBatchCleanupTests(SimpleTestCase):
             process_m3u_batch_direct(1, [], {}, ["name", "url"])
             mock_connections.close_all.assert_called()
 
+    @patch("apps.m3u.tasks.Stream")
+    @patch("apps.m3u.tasks.M3UAccount")
+    def test_batch_calls_gc_collect(self, mock_account_cls, mock_stream_cls):
+        """gc.collect() must run after each batch so XC refresh threads release promptly."""
+        from apps.m3u.tasks import process_m3u_batch_direct
+
+        mock_account = MagicMock()
+        mock_account.filters.order_by.return_value = []
+        mock_account_cls.objects.get.return_value = mock_account
+        mock_stream_cls.objects.filter.return_value.select_related.return_value.only.return_value = (
+            []
+        )
+        mock_stream_cls.generate_hash_key = MagicMock(return_value="hash123")
+
+        with patch("gc.collect") as mock_gc, patch("django.db.connections"):
+            process_m3u_batch_direct(1, [], {}, ["name", "url"])
+            mock_gc.assert_called()
+
 
 class LockReleaseTests(SimpleTestCase):
     """Verify task lock is released on all exit paths."""