Merge pull request #1158 from Dispatcharr/html-entity-with-lxml

Html-entity-with-lxml

CHANGELOG.md

@@ -61,13 +61,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - M3U profile URL rewriting now uses the `regex` module instead of `re` across all URL transform code paths (`url_utils.transform_url`, `core/views.py`, `vod_proxy/_transform_url`, `tasks.get_transformed_credentials`, and the WebSocket live-preview handler in `consumers.py`). The `regex` module natively accepts JavaScript/PCRE-style named capture groups (`(?<name>...)`) without any conversion, eliminating the root cause of patterns that matched in the frontend live preview but failed on the backend with a `re.error`. As a further improvement, `regex` also supports variable-length lookbehind assertions (e.g. `(?<=a+)`), which `re` rejects with an error; patterns using these will now work correctly on the backend as well. Replace-pattern JS tokens are still normalised before calling `regex.sub`: `$<name>` → `\g<name>` and `$1`/`$2`/… → `\1`/`\2`/… (Python replacement syntax). Also fixed a bug in the WebSocket preview handler where a pattern error was incorrectly returning the search pattern string as the preview output instead of the original URL. (Fixes #1005)
 - Web UI stream preview (`FloatingVideo`) was calling `mpegts.createPlayer()` with all `Config` options (e.g. `enableWorker`, `liveSync`, `headers`) merged into the first `MediaDataSource` argument. mpegts.js only reads `Config` from the optional second argument; unrecognised fields in the first are silently ignored. As a result all player configuration was effectively the library defaults — worker offloading was disabled, latency management had no effect, and the `Authorization: Bearer` header (required for user identification) was never sent. Fixed by splitting into the correct two-argument call. Both `liveBufferLatencyChasing` and `liveSync` have been disabled, eliminating playback-rate fluctuations that caused audible stuttering on live streams. SourceBuffer cleanup thresholds were also relaxed from 10s/5s to 120s/60s to prevent frequent SourceBuffer pauses.
-- HTML named entities in XMLTV EPG files are now resolved to Unicode characters before lxml parsing. Some EPG providers (particularly French and other European sources) use HTML named entities like `&eacute;`, `&icirc;`, `&uuml;` in channel names, program titles, and metadata. These are not valid XML entities — lxml 6.0.2 with `recover=True` silently drops them, causing characters to go missing (e.g., "Chaîne Télé" becomes "Chane Tl"). A preprocessing step in `fetch_xmltv()` now resolves HTML named entities while preserving the 5 XML-predefined entities, detecting encoding from the XML declaration (falling back to UTF-8), and processing line-by-line to a temp file before atomically replacing the original. (Closes #1095) — Thanks [@CodeBormen](https://github.com/CodeBormen)
+- HTML named entities in XMLTV EPG files are now correctly preserved during lxml parsing. Some EPG providers (particularly French and other European sources) use HTML named entities like `&eacute;`, `&icirc;`, `&uuml;` in channel names, program titles, and metadata. These are not valid XML entities — lxml 6.0.2 with `recover=True` silently drops them, causing characters to go missing (e.g., "Chaîne Télé" becomes "Chane Tl"). This is now fixed by injecting an XML `<!DOCTYPE tv [...]>` internal subset declaring all 252 HTML 4 named entities directly into the byte stream that lxml reads, using a lightweight in-memory wrapper (`_PrependStream`) with zero disk I/O. libxml2 resolves the entities during its normal C-level parse pass — no Python-level preprocessing or temporary files are involved. The DOCTYPE block (~8 KB) is built once at module load from Python's stdlib `html.entities.name2codepoint` and reused for every parse. Files that already declare their own `<!DOCTYPE>` are passed through unchanged. (Closes #1095) — Thanks [@CodeBormen](https://github.com/CodeBormen) for helping with this!
 - Duplicate recordings created when EPG sources refresh and re-evaluate series rules (Fixes #940) — Thanks [@CodeBormen](https://github.com/CodeBormen):
   - **Program ID instability**: `parse_programs_for_source()` deletes and recreates all `ProgramData` rows with new auto-increment IDs on every EPG refresh. The dedup set used these IDs, so it never matched after a refresh. Deduplication now uses a stable `(tvg_id, start_time, end_time)` composite key sourced from `Recording.custom_properties.program`.
   - **Secondary guard using wrong times**: The DB guard compared unadjusted program times against offset-adjusted `Recording.start_time`/`end_time`, so it never matched when any DVR pre/post offset was configured. It now queries `custom_properties__program__start_time/end_time` (the original, unadjusted program times stored at recording creation).
   - **No concurrency guard**: Each EPG source refresh fired `evaluate_series_rules.delay()` independently. Concurrent tasks loaded the dedup set before others committed, allowing races. Evaluation is now serialized with `acquire_task_lock` (reusing the existing EPG task pattern). Gracefully degrades if Redis is unavailable — the primary and secondary dedup guards still protect.
-- EPG refresh tasks (`refresh_epg_data`) were being killed mid-transaction on large EPG sources. The `soft_time_limit=1700s` introduced in v0.21.0 raised `SoftTimeLimitExceeded`, a subclass of `Exception`, which was swallowed by the existing `except Exception` handler in `parse_programs_for_source`, leaving the database in a partial state with no logged error. The underlying cause is that the HTML entity preprocessing step added in v0.21.0 adds 2–10+ minutes on multi-gigabyte EPG files, pushing total task duration past the 1700s limit for large sources. `soft_time_limit` has been removed from `refresh_epg_data` and `time_limit` raised to 14400s (4 hours) as a true last-resort ceiling; the existing `TaskLockRenewer` daemon thread continues to renew the Redis lock every 120s for legitimately long-running tasks.
-- HTML entity preprocessing (`_resolve_html_entities`) now performs a fast binary pre-scan before any file read or write. The pre-scan reads the file in 4 MB chunks without text decoding and exits immediately on the first non-XML named entity found. For EPG sources that contain no HTML entities — the common case — the full line-by-line read+rewrite pass (which takes 2–10+ minutes on a 2.5 GB file) is skipped entirely.
+- EPG refresh tasks (`refresh_epg_data`) were being killed mid-transaction on large EPG sources. The `soft_time_limit=1700s` introduced in v0.21.0 raised `SoftTimeLimitExceeded`, a subclass of `Exception`, which was swallowed by the existing `except Exception` handler in `parse_programs_for_source`, leaving the database in a partial state with no logged error. `soft_time_limit` has been removed from `refresh_epg_data` and `time_limit` raised to 14400s (4 hours) as a true last-resort ceiling; the existing `TaskLockRenewer` daemon thread continues to renew the Redis lock every 120s for legitimately long-running tasks.
 
 ## [0.21.1] - 2026-03-18
 

apps/epg/tasks.py

@@ -1,11 +1,9 @@
 # apps/epg/tasks.py
 
-import codecs
 import logging
 import gzip
-import html as html_module
+import html.entities
 import os
-import re
 import uuid
 import requests
 import time  # Add import for tracking download progress
@@ -31,138 +29,102 @@ from core.utils import acquire_task_lock, release_task_lock, TaskLockRenewer, se
 
 logger = logging.getLogger(__name__)
 
-# Regex and helpers for resolving HTML named entities in XMLTV files.
-# lxml only recognises the 5 XML-predefined entities; HTML entities like
-# é are silently dropped in recovery mode.  This substitution
-# converts them to Unicode before the file reaches the parser.
+# DOCTYPE internal subset for XMLTV files.  Declares all 252 HTML 4 named
+# entities so lxml/libxml2 can resolve references like é correctly
+# instead of silently dropping them in recovery mode.
+# The 5 XML-predefined entities (amp, lt, gt, quot, apos) are always
+# recognised by the XML spec and must not be redeclared.
 _XML_ENTITIES = frozenset({'amp', 'lt', 'gt', 'quot', 'apos'})
-_XML_ENTITIES_BYTES = frozenset(n.encode() for n in _XML_ENTITIES)
-_XML_SPECIAL_CHARS = frozenset('&<>\'"')
-_NAMED_ENTITY_RE = re.compile(r'&([a-zA-Z][a-zA-Z0-9]*);')
-# Binary version used by the fast pre-scan (no text decoding needed)
-_BINARY_ENTITY_RE = re.compile(rb'&([a-zA-Z][a-zA-Z0-9]*);')
 
 
-def _replace_html_entity(match):
-    """Replace a single HTML named entity, preserving XML-special ones.
+def _build_html_entity_doctype() -> bytes:
+    """Build a DOCTYPE internal subset declaring all HTML 4 named entities."""
+    lines = [b'<!DOCTYPE tv [\n']
+    for name, codepoint in sorted(html.entities.name2codepoint.items()):
+        if name not in _XML_ENTITIES:
+            # Numeric character references are always valid XML regardless of codepoint.
+            lines.append(f'<!ENTITY {name} "&#x{codepoint:X};">\n'.encode('ascii'))
+    lines.append(b']>\n')
+    return b''.join(lines)
 
-    Skips lowercase XML entities (fast path) and any entity whose resolved
-    output contains XML-special characters that would break parsing.  This
-    also guards against html.unescape partially matching sub-patterns
-    (e.g. &ampersand; being split into &amp + ersand;).
+
+_HTML_ENTITY_DOCTYPE = _build_html_entity_doctype()
+
+
+class _PrependStream:
+    """Wraps an open binary file and prepends a bytes prefix to its content.
+
+    Used by _open_xmltv_file to inject a DOCTYPE entity block before the
+    file content reaches lxml's iterparse, with zero disk I/O.
     """
-    original = match.group(0)
-    name = match.group(1)
-    if name in _XML_ENTITIES:
-        return original
-    resolved = html_module.unescape(original)
-    if resolved == original:
-        return original
-    # If the resolved output contains any XML-special character, replacing
-    # it would produce invalid XML (bare &, <, >, etc.)
-    if _XML_SPECIAL_CHARS.intersection(resolved):
-        return original
-    return resolved
+
+    __slots__ = ('_prefix', '_prefix_pos', '_file')
+
+    def __init__(self, prefix: bytes, file_obj):
+        self._prefix = prefix
+        self._prefix_pos = 0
+        self._file = file_obj
+
+    def read(self, size=-1):
+        prefix_len = len(self._prefix)
+        if self._prefix_pos >= prefix_len:
+            return self._file.read(size)
+        remaining = prefix_len - self._prefix_pos
+        if size < 0:
+            chunk = self._prefix[self._prefix_pos:] + self._file.read()
+            self._prefix_pos = prefix_len
+            return chunk
+        if size <= remaining:
+            chunk = self._prefix[self._prefix_pos:self._prefix_pos + size]
+            self._prefix_pos += size
+            return chunk
+        chunk = self._prefix[self._prefix_pos:]
+        self._prefix_pos = prefix_len
+        return chunk + self._file.read(size - remaining)
+
+    def close(self):
+        self._file.close()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *_):
+        self.close()
 
 
-def _detect_xml_encoding(header):
-    """Detect encoding from raw XML header bytes, defaulting to UTF-8.
+def _open_xmltv_file(file_path: str):
+    """Open an XMLTV file for lxml iterparse, injecting an HTML entity DOCTYPE.
 
-    Falls back to UTF-8 if the declared encoding is not recognized by Python.
+    Prepends a <!DOCTYPE tv [...]> block that declares all 252 HTML 4 named
+    entities so lxml/libxml2 resolves references like &eacute; correctly
+    instead of silently dropping them in recovery mode.  This involves zero
+    disk I/O — the DOCTYPE is streamed in-memory before the file content.
+
+    If the file already contains a <!DOCTYPE> declaration the file is returned
+    unchanged; a second DOCTYPE would be invalid XML.
+
+    The caller is responsible for closing the returned object.
     """
-    m = re.match(rb'<\?xml[^>]*encoding=["\']([^"\']+)["\']', header)
-    if m:
-        declared = m.group(1).decode('ascii')
-        try:
-            codecs.lookup(declared)
-            return declared
-        except LookupError:
-            logger.warning(f"Unknown encoding '{declared}', falling back to UTF-8")
-            return 'utf-8'
-    return 'utf-8'
+    f = open(file_path, 'rb')
+    start = f.read(512)
 
+    # Do not inject if the file already declares a DOCTYPE.
+    if b'<!DOCTYPE' in start or b'<!doctype' in start.lower():
+        f.seek(0)
+        return f
 
-def _file_needs_entity_resolution(file_path):
-    """Binary pre-scan: return True only if file contains non-XML named HTML entities.
+    # Insert the DOCTYPE after the XML declaration if one is present.
+    stripped = start.lstrip()
+    if stripped.startswith(b'<?xml'):
+        decl_end = start.find(b'?>')
+        if decl_end >= 0:
+            xml_decl = start[:decl_end + 2]
+            f.seek(decl_end + 2)
+            return _PrependStream(xml_decl + b'\n' + _HTML_ENTITY_DOCTYPE, f)
 
-    Reads in 4 MB chunks with a small overlap to catch entities that straddle
-    a chunk boundary.  Avoids all text-decode overhead — this is effectively a
-    raw memory scan and exits as soon as the first HTML entity is found.
-    """
-    CHUNK = 4 * 1024 * 1024  # 4 MB
-    OVERLAP = 32              # longer than any valid entity name + surrounding &;
-    try:
-        with open(file_path, 'rb') as f:
-            prev = b''
-            while True:
-                chunk = f.read(CHUNK)
-                if not chunk:
-                    return False
-                data = prev + chunk
-                for m in _BINARY_ENTITY_RE.finditer(data):
-                    if m.group(1).lower() not in _XML_ENTITIES_BYTES:
-                        return True
-                prev = data[-OVERLAP:]
-    except OSError:
-        return False
-    return False
-
-
-def _resolve_html_entities(file_path):
-    """Replace HTML named entities in an XMLTV file with Unicode characters.
-
-    Processes line-by-line to keep memory usage low, writes to a temp file,
-    then atomically replaces the original.  Honors the encoding declared
-    in the XML declaration.
-
-    This function is strictly additive — on any error the original file is
-    left untouched so lxml can handle it as it always did.
-    """
-    # Fast binary pre-scan: if the file contains no non-XML named entities
-    # there is nothing to do and we can skip the full read+write entirely.
-    # For large files (e.g. 2.5 GB) this avoids significant I/O time when
-    # the EPG source is already clean (the common case).
-    if not _file_needs_entity_resolution(file_path):
-        logger.debug(f"No HTML entities found in {file_path}, skipping entity resolution")
-        return
-    logger.info(f"HTML entities detected in {file_path}, resolving...")
-    # Read the first 200 bytes to detect encoding from the XML declaration
-    with open(file_path, 'rb') as f:
-        header = f.read(200)
-    encoding = _detect_xml_encoding(header)
-
-    # Capture the original mtime before modifying the file so that os.replace()
-    # does not update the file's mtime.  The file watcher in core/tasks.py uses
-    # mtime to detect changes; if we let mtime advance it would trigger an
-    # infinite refresh loop for local file-based EPG sources.
-    try:
-        original_stat = os.stat(file_path)
-    except OSError:
-        original_stat = None
-
-    temp_path = file_path + '.entity_tmp'
-    success = False
-    try:
-        with open(file_path, 'r', encoding=encoding) as src, \
-             open(temp_path, 'w', encoding=encoding) as dst:
-            for line in src:
-                dst.write(_NAMED_ENTITY_RE.sub(_replace_html_entity, line))
-        os.replace(temp_path, file_path)
-        # Restore the original mtime (and atime) so the file watcher does not
-        # mistake the rewrite for an external update.
-        if original_stat is not None:
-            os.utime(file_path, (original_stat.st_atime, original_stat.st_mtime))
-        success = True
-        logger.debug(f"Resolved HTML entities in {file_path} (encoding: {encoding})")
-    except Exception as e:
-        # On any error, leave the original file untouched for lxml to
-        # handle with its own encoding detection and recovery mode.
-        # This ensures the preprocessing step never breaks a previously
-        # working EPG refresh.
-        logger.debug(f"Skipping entity resolution for {file_path}: {e}")
-    finally:
-        if not success and os.path.exists(temp_path):
-            os.unlink(temp_path)
+    # No XML declaration — insert DOCTYPE at the very start of the file.
+    f.seek(0)
+    return _PrependStream(_HTML_ENTITY_DOCTYPE, f)
 
 
 def validate_icon_url_fast(icon_url, max_length=None):
@@ -416,9 +378,6 @@ def fetch_xmltv(source):
         # Send a download complete notification
         send_epg_update(source.id, "downloading", 100, status="success")
 
-        # Resolve HTML named entities so lxml doesn't drop them during parsing
-        _resolve_html_entities(source.extracted_file_path or source.file_path)
-
         # Return True to indicate successful fetch, processing will continue with parse_channels_only
         return True
 
@@ -665,9 +624,6 @@ def fetch_xmltv(source):
 
             logger.info(f"Cached EPG file saved to {source.file_path}")
 
-            # Resolve HTML named entities so lxml doesn't drop them during parsing
-            _resolve_html_entities(source.extracted_file_path or source.file_path)
-
             return True
 
     except requests.exceptions.HTTPError as e:
@@ -1030,7 +986,7 @@ def parse_channels_only(source):
 
             # Open the file - no need to check file type since it's always XML now
             logger.debug(f"Opening file for channel parsing: {file_path}")
-            source_file = open(file_path, 'rb')
+            source_file = _open_xmltv_file(file_path)
 
             if process:
                 logger.debug(f"[parse_channels_only] Memory after opening file: {process.memory_info().rss / 1024 / 1024:.2f} MB")
@@ -1413,7 +1369,7 @@ def parse_programs_for_tvg_id(epg_id):
         try:
             # Open the file directly - no need to check compression
             logger.debug(f"Opening file for parsing: {file_path}")
-            source_file = open(file_path, 'rb')
+            source_file = _open_xmltv_file(file_path)
 
             # Stream parse the file using lxml's iterparse
             program_parser = etree.iterparse(source_file, events=('end',), tag='programme',  remove_blank_text=True, recover=True)
@@ -1686,7 +1642,7 @@ def parse_programs_for_source(epg_source, tvg_id=None):
 
         try:
             logger.debug(f"Opening file for single-pass parsing: {file_path}")
-            source_file = open(file_path, 'rb')
+            source_file = _open_xmltv_file(file_path)
 
             # Stream parse the file using lxml's iterparse
             program_parser = etree.iterparse(source_file, events=('end',), tag='programme', remove_blank_text=True, recover=True)