From 70e229a433a4046f6d0ba73737682c24775d6f11 Mon Sep 17 00:00:00 2001 From: SergeantPanda Date: Sun, 29 Mar 2026 20:43:27 -0500 Subject: [PATCH] Refactor HTML entity handling in XMLTV files: replace regex-based entity resolution with lxml's html.entities for improved accuracy and performance. Inject DOCTYPE declaration to ensure proper parsing of HTML entities, avoiding silent drops during recovery mode. --- CHANGELOG.md | 5 +- apps/epg/tasks.py | 220 +++++++++++++++++++--------------------------- 2 files changed, 90 insertions(+), 135 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e8bccb55..1f8aad67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -61,13 +61,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - M3U profile URL rewriting now uses the `regex` module instead of `re` across all URL transform code paths (`url_utils.transform_url`, `core/views.py`, `vod_proxy/_transform_url`, `tasks.get_transformed_credentials`, and the WebSocket live-preview handler in `consumers.py`). The `regex` module natively accepts JavaScript/PCRE-style named capture groups (`(?...)`) without any conversion, eliminating the root cause of patterns that matched in the frontend live preview but failed on the backend with a `re.error`. As a further improvement, `regex` also supports variable-length lookbehind assertions (e.g. `(?<=a+)`), which `re` rejects with an error; patterns using these will now work correctly on the backend as well. Replace-pattern JS tokens are still normalised before calling `regex.sub`: `$` → `\g` and `$1`/`$2`/… → `\1`/`\2`/… (Python replacement syntax). Also fixed a bug in the WebSocket preview handler where a pattern error was incorrectly returning the search pattern string as the preview output instead of the original URL. (Fixes #1005) - Web UI stream preview (`FloatingVideo`) was calling `mpegts.createPlayer()` with all `Config` options (e.g. `enableWorker`, `liveSync`, `headers`) merged into the first `MediaDataSource` argument. mpegts.js only reads `Config` from the optional second argument; unrecognised fields in the first are silently ignored. As a result all player configuration was effectively the library defaults — worker offloading was disabled, latency management had no effect, and the `Authorization: Bearer` header (required for user identification) was never sent. Fixed by splitting into the correct two-argument call. Both `liveBufferLatencyChasing` and `liveSync` have been disabled, eliminating playback-rate fluctuations that caused audible stuttering on live streams. SourceBuffer cleanup thresholds were also relaxed from 10s/5s to 120s/60s to prevent frequent SourceBuffer pauses. -- HTML named entities in XMLTV EPG files are now resolved to Unicode characters before lxml parsing. Some EPG providers (particularly French and other European sources) use HTML named entities like `é`, `î`, `ü` in channel names, program titles, and metadata. These are not valid XML entities — lxml 6.0.2 with `recover=True` silently drops them, causing characters to go missing (e.g., "Chaîne Télé" becomes "Chane Tl"). A preprocessing step in `fetch_xmltv()` now resolves HTML named entities while preserving the 5 XML-predefined entities, detecting encoding from the XML declaration (falling back to UTF-8), and processing line-by-line to a temp file before atomically replacing the original. (Closes #1095) — Thanks [@CodeBormen](https://github.com/CodeBormen) +- HTML named entities in XMLTV EPG files are now correctly preserved during lxml parsing. Some EPG providers (particularly French and other European sources) use HTML named entities like `é`, `î`, `ü` in channel names, program titles, and metadata. These are not valid XML entities — lxml 6.0.2 with `recover=True` silently drops them, causing characters to go missing (e.g., "Chaîne Télé" becomes "Chane Tl"). This is now fixed by injecting an XML `` internal subset declaring all 252 HTML 4 named entities directly into the byte stream that lxml reads, using a lightweight in-memory wrapper (`_PrependStream`) with zero disk I/O. libxml2 resolves the entities during its normal C-level parse pass — no Python-level preprocessing or temporary files are involved. The DOCTYPE block (~8 KB) is built once at module load from Python's stdlib `html.entities.name2codepoint` and reused for every parse. Files that already declare their own `` are passed through unchanged. (Closes #1095) — Thanks [@CodeBormen](https://github.com/CodeBormen) for helping with this! - Duplicate recordings created when EPG sources refresh and re-evaluate series rules (Fixes #940) — Thanks [@CodeBormen](https://github.com/CodeBormen): - **Program ID instability**: `parse_programs_for_source()` deletes and recreates all `ProgramData` rows with new auto-increment IDs on every EPG refresh. The dedup set used these IDs, so it never matched after a refresh. Deduplication now uses a stable `(tvg_id, start_time, end_time)` composite key sourced from `Recording.custom_properties.program`. - **Secondary guard using wrong times**: The DB guard compared unadjusted program times against offset-adjusted `Recording.start_time`/`end_time`, so it never matched when any DVR pre/post offset was configured. It now queries `custom_properties__program__start_time/end_time` (the original, unadjusted program times stored at recording creation). - **No concurrency guard**: Each EPG source refresh fired `evaluate_series_rules.delay()` independently. Concurrent tasks loaded the dedup set before others committed, allowing races. Evaluation is now serialized with `acquire_task_lock` (reusing the existing EPG task pattern). Gracefully degrades if Redis is unavailable — the primary and secondary dedup guards still protect. -- EPG refresh tasks (`refresh_epg_data`) were being killed mid-transaction on large EPG sources. The `soft_time_limit=1700s` introduced in v0.21.0 raised `SoftTimeLimitExceeded`, a subclass of `Exception`, which was swallowed by the existing `except Exception` handler in `parse_programs_for_source`, leaving the database in a partial state with no logged error. The underlying cause is that the HTML entity preprocessing step added in v0.21.0 adds 2–10+ minutes on multi-gigabyte EPG files, pushing total task duration past the 1700s limit for large sources. `soft_time_limit` has been removed from `refresh_epg_data` and `time_limit` raised to 14400s (4 hours) as a true last-resort ceiling; the existing `TaskLockRenewer` daemon thread continues to renew the Redis lock every 120s for legitimately long-running tasks. -- HTML entity preprocessing (`_resolve_html_entities`) now performs a fast binary pre-scan before any file read or write. The pre-scan reads the file in 4 MB chunks without text decoding and exits immediately on the first non-XML named entity found. For EPG sources that contain no HTML entities — the common case — the full line-by-line read+rewrite pass (which takes 2–10+ minutes on a 2.5 GB file) is skipped entirely. +- EPG refresh tasks (`refresh_epg_data`) were being killed mid-transaction on large EPG sources. The `soft_time_limit=1700s` introduced in v0.21.0 raised `SoftTimeLimitExceeded`, a subclass of `Exception`, which was swallowed by the existing `except Exception` handler in `parse_programs_for_source`, leaving the database in a partial state with no logged error. `soft_time_limit` has been removed from `refresh_epg_data` and `time_limit` raised to 14400s (4 hours) as a true last-resort ceiling; the existing `TaskLockRenewer` daemon thread continues to renew the Redis lock every 120s for legitimately long-running tasks. ## [0.21.1] - 2026-03-18 diff --git a/apps/epg/tasks.py b/apps/epg/tasks.py index 3126453d..0d05552f 100644 --- a/apps/epg/tasks.py +++ b/apps/epg/tasks.py @@ -1,11 +1,9 @@ # apps/epg/tasks.py -import codecs import logging import gzip -import html as html_module +import html.entities import os -import re import uuid import requests import time # Add import for tracking download progress @@ -31,138 +29,102 @@ from core.utils import acquire_task_lock, release_task_lock, TaskLockRenewer, se logger = logging.getLogger(__name__) -# Regex and helpers for resolving HTML named entities in XMLTV files. -# lxml only recognises the 5 XML-predefined entities; HTML entities like -# é are silently dropped in recovery mode. This substitution -# converts them to Unicode before the file reaches the parser. +# DOCTYPE internal subset for XMLTV files. Declares all 252 HTML 4 named +# entities so lxml/libxml2 can resolve references like é correctly +# instead of silently dropping them in recovery mode. +# The 5 XML-predefined entities (amp, lt, gt, quot, apos) are always +# recognised by the XML spec and must not be redeclared. _XML_ENTITIES = frozenset({'amp', 'lt', 'gt', 'quot', 'apos'}) -_XML_ENTITIES_BYTES = frozenset(n.encode() for n in _XML_ENTITIES) -_XML_SPECIAL_CHARS = frozenset('&<>\'"') -_NAMED_ENTITY_RE = re.compile(r'&([a-zA-Z][a-zA-Z0-9]*);') -# Binary version used by the fast pre-scan (no text decoding needed) -_BINARY_ENTITY_RE = re.compile(rb'&([a-zA-Z][a-zA-Z0-9]*);') -def _replace_html_entity(match): - """Replace a single HTML named entity, preserving XML-special ones. +def _build_html_entity_doctype() -> bytes: + """Build a DOCTYPE internal subset declaring all HTML 4 named entities.""" + lines = [b'\n'.encode('ascii')) + lines.append(b']>\n') + return b''.join(lines) - Skips lowercase XML entities (fast path) and any entity whose resolved - output contains XML-special characters that would break parsing. This - also guards against html.unescape partially matching sub-patterns - (e.g. &ersand; being split into & + ersand;). + +_HTML_ENTITY_DOCTYPE = _build_html_entity_doctype() + + +class _PrependStream: + """Wraps an open binary file and prepends a bytes prefix to its content. + + Used by _open_xmltv_file to inject a DOCTYPE entity block before the + file content reaches lxml's iterparse, with zero disk I/O. """ - original = match.group(0) - name = match.group(1) - if name in _XML_ENTITIES: - return original - resolved = html_module.unescape(original) - if resolved == original: - return original - # If the resolved output contains any XML-special character, replacing - # it would produce invalid XML (bare &, <, >, etc.) - if _XML_SPECIAL_CHARS.intersection(resolved): - return original - return resolved + + __slots__ = ('_prefix', '_prefix_pos', '_file') + + def __init__(self, prefix: bytes, file_obj): + self._prefix = prefix + self._prefix_pos = 0 + self._file = file_obj + + def read(self, size=-1): + prefix_len = len(self._prefix) + if self._prefix_pos >= prefix_len: + return self._file.read(size) + remaining = prefix_len - self._prefix_pos + if size < 0: + chunk = self._prefix[self._prefix_pos:] + self._file.read() + self._prefix_pos = prefix_len + return chunk + if size <= remaining: + chunk = self._prefix[self._prefix_pos:self._prefix_pos + size] + self._prefix_pos += size + return chunk + chunk = self._prefix[self._prefix_pos:] + self._prefix_pos = prefix_len + return chunk + self._file.read(size - remaining) + + def close(self): + self._file.close() + + def __enter__(self): + return self + + def __exit__(self, *_): + self.close() -def _detect_xml_encoding(header): - """Detect encoding from raw XML header bytes, defaulting to UTF-8. +def _open_xmltv_file(file_path: str): + """Open an XMLTV file for lxml iterparse, injecting an HTML entity DOCTYPE. - Falls back to UTF-8 if the declared encoding is not recognized by Python. + Prepends a block that declares all 252 HTML 4 named + entities so lxml/libxml2 resolves references like é correctly + instead of silently dropping them in recovery mode. This involves zero + disk I/O — the DOCTYPE is streamed in-memory before the file content. + + If the file already contains a declaration the file is returned + unchanged; a second DOCTYPE would be invalid XML. + + The caller is responsible for closing the returned object. """ - m = re.match(rb'<\?xml[^>]*encoding=["\']([^"\']+)["\']', header) - if m: - declared = m.group(1).decode('ascii') - try: - codecs.lookup(declared) - return declared - except LookupError: - logger.warning(f"Unknown encoding '{declared}', falling back to UTF-8") - return 'utf-8' - return 'utf-8' + f = open(file_path, 'rb') + start = f.read(512) + # Do not inject if the file already declares a DOCTYPE. + if b'') + if decl_end >= 0: + xml_decl = start[:decl_end + 2] + f.seek(decl_end + 2) + return _PrependStream(xml_decl + b'\n' + _HTML_ENTITY_DOCTYPE, f) - Reads in 4 MB chunks with a small overlap to catch entities that straddle - a chunk boundary. Avoids all text-decode overhead — this is effectively a - raw memory scan and exits as soon as the first HTML entity is found. - """ - CHUNK = 4 * 1024 * 1024 # 4 MB - OVERLAP = 32 # longer than any valid entity name + surrounding &; - try: - with open(file_path, 'rb') as f: - prev = b'' - while True: - chunk = f.read(CHUNK) - if not chunk: - return False - data = prev + chunk - for m in _BINARY_ENTITY_RE.finditer(data): - if m.group(1).lower() not in _XML_ENTITIES_BYTES: - return True - prev = data[-OVERLAP:] - except OSError: - return False - return False - - -def _resolve_html_entities(file_path): - """Replace HTML named entities in an XMLTV file with Unicode characters. - - Processes line-by-line to keep memory usage low, writes to a temp file, - then atomically replaces the original. Honors the encoding declared - in the XML declaration. - - This function is strictly additive — on any error the original file is - left untouched so lxml can handle it as it always did. - """ - # Fast binary pre-scan: if the file contains no non-XML named entities - # there is nothing to do and we can skip the full read+write entirely. - # For large files (e.g. 2.5 GB) this avoids significant I/O time when - # the EPG source is already clean (the common case). - if not _file_needs_entity_resolution(file_path): - logger.debug(f"No HTML entities found in {file_path}, skipping entity resolution") - return - logger.info(f"HTML entities detected in {file_path}, resolving...") - # Read the first 200 bytes to detect encoding from the XML declaration - with open(file_path, 'rb') as f: - header = f.read(200) - encoding = _detect_xml_encoding(header) - - # Capture the original mtime before modifying the file so that os.replace() - # does not update the file's mtime. The file watcher in core/tasks.py uses - # mtime to detect changes; if we let mtime advance it would trigger an - # infinite refresh loop for local file-based EPG sources. - try: - original_stat = os.stat(file_path) - except OSError: - original_stat = None - - temp_path = file_path + '.entity_tmp' - success = False - try: - with open(file_path, 'r', encoding=encoding) as src, \ - open(temp_path, 'w', encoding=encoding) as dst: - for line in src: - dst.write(_NAMED_ENTITY_RE.sub(_replace_html_entity, line)) - os.replace(temp_path, file_path) - # Restore the original mtime (and atime) so the file watcher does not - # mistake the rewrite for an external update. - if original_stat is not None: - os.utime(file_path, (original_stat.st_atime, original_stat.st_mtime)) - success = True - logger.debug(f"Resolved HTML entities in {file_path} (encoding: {encoding})") - except Exception as e: - # On any error, leave the original file untouched for lxml to - # handle with its own encoding detection and recovery mode. - # This ensures the preprocessing step never breaks a previously - # working EPG refresh. - logger.debug(f"Skipping entity resolution for {file_path}: {e}") - finally: - if not success and os.path.exists(temp_path): - os.unlink(temp_path) + # No XML declaration — insert DOCTYPE at the very start of the file. + f.seek(0) + return _PrependStream(_HTML_ENTITY_DOCTYPE, f) def validate_icon_url_fast(icon_url, max_length=None): @@ -416,9 +378,6 @@ def fetch_xmltv(source): # Send a download complete notification send_epg_update(source.id, "downloading", 100, status="success") - # Resolve HTML named entities so lxml doesn't drop them during parsing - _resolve_html_entities(source.extracted_file_path or source.file_path) - # Return True to indicate successful fetch, processing will continue with parse_channels_only return True @@ -665,9 +624,6 @@ def fetch_xmltv(source): logger.info(f"Cached EPG file saved to {source.file_path}") - # Resolve HTML named entities so lxml doesn't drop them during parsing - _resolve_html_entities(source.extracted_file_path or source.file_path) - return True except requests.exceptions.HTTPError as e: @@ -1030,7 +986,7 @@ def parse_channels_only(source): # Open the file - no need to check file type since it's always XML now logger.debug(f"Opening file for channel parsing: {file_path}") - source_file = open(file_path, 'rb') + source_file = _open_xmltv_file(file_path) if process: logger.debug(f"[parse_channels_only] Memory after opening file: {process.memory_info().rss / 1024 / 1024:.2f} MB") @@ -1413,7 +1369,7 @@ def parse_programs_for_tvg_id(epg_id): try: # Open the file directly - no need to check compression logger.debug(f"Opening file for parsing: {file_path}") - source_file = open(file_path, 'rb') + source_file = _open_xmltv_file(file_path) # Stream parse the file using lxml's iterparse program_parser = etree.iterparse(source_file, events=('end',), tag='programme', remove_blank_text=True, recover=True) @@ -1686,7 +1642,7 @@ def parse_programs_for_source(epg_source, tvg_id=None): try: logger.debug(f"Opening file for single-pass parsing: {file_path}") - source_file = open(file_path, 'rb') + source_file = _open_xmltv_file(file_path) # Stream parse the file using lxml's iterparse program_parser = etree.iterparse(source_file, events=('end',), tag='programme', remove_blank_text=True, recover=True)