Adds .xz to the set of compressed formats detect_file_format(),
extract_compressed_file(), fetch_xmltv(), and EPGSource.get_cache_file()
recognize, mirroring the existing gzip handling. Uses stdlib lzma, no new
dependency. Detection works via LZMA magic bytes (fd 37 7a 58 5a 00),
the .xz extension (including compound extensions like .xml.xz), and the
application/x-xz mimetype fallback.
Widened the magic-byte read in get_cache_file() from 4 to 6 bytes since
the xz signature is 6 bytes (the previous 4-byte read was sufficient
only for the 2-byte gzip/zip signatures it originally supported). That
widening silently broke the bare-<tv> raw-XML detection: the
fixed-length slice comparison header[:5] == b'<tv>' can never match a
5-byte slice of a 6-byte header against a 4-byte literal. Replaced the
fixed-length slice comparisons with header.startswith(...), which is
correct regardless of how many bytes are read - this also fixes the
adjacent <?xml declaration check, which was silently dead before this
change (a 4-byte read could never equal the 5-byte b'<?xml' literal
either).
FixesDispatcharr/Dispatcharr#1414
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Moved the `programme_index` from the `EPGSource` model to a new `EPGSourceIndex` table, ensuring that the large JSON blob is only loaded when explicitly accessed, thus improving query performance and memory efficiency.
- Updated related queries and API views to utilize the new structure, including adjustments to EPG generation and import logic to prevent unnecessary data loading.
- Enhanced memory management in the EPG grid endpoint to reduce worker RSS during response handling.
- Streamlined `generate_epg()` to incrementally stream EPG data without loading the entire guide, improving memory efficiency.
- Reduced database load by deferring the fetching of large `programme_index` blobs, enhancing response times for EPG generation.
- Introduced a composite index on `(epg_id, id)` in `ProgramData` to optimize query performance during EPG exports.
- Updated tests to ensure proper functionality and performance of new features.
- Fix NameError in sd_lineups_search: add dispatcharr_version import
- Fix DELETE handler: persist changesRemaining from SD response to custom_properties
- Fix lockout reset time: calculate next midnight UTC (00:00Z) instead of
rolling now+24h, matching SD's documented reset behavior
- Remove redundant explicit index from SDProgramMD5: unique_together already
creates this index in Postgres
- Rename api_key to password (RemoveField + AddField) for clarity and reusability
- Move sd_changes_remaining/sd_changes_reset_at to custom_properties JSONField
instead of nullable columns on all EPGSource rows
- Replace sd_program_md5 column on ProgramData with SDProgramMD5 relation table
keyed by epg_source + program_id, fixing the key mismatch bug in program
MD5 delta detection
- Fix total_changed counter: .items() -> .values() so len() returns date count
not tuple length
- Add missing SD lineup management endpoints to api_views.py
(sd_lineups GET/POST/DELETE, sd_lineups_search POST)
- Regenerate migration 0023_schedules_direct using Django to ensure correct
index names and schema
- Add EPGSource username field and SDScheduleMD5 model for MD5 delta sync
- Implement full Schedules Direct fetch with SHA1 auth, token caching,
20-day schedule window, and MD5-based delta sync to minimize API usage
- Add lineup manager UI allowing users to search and manage SD lineups
- Add stations-only fetch on source creation for immediate EPG matching
- Enforce 2-hour minimum refresh interval to respect SD rate limits,
with first-fetch bypass when no prior full refresh exists
- Add real-time progress updates via synchronous Redis WebSocket delivery,
bypassing gevent.spawn for reliable delivery from Celery prefork workers
- Refactor EPGsTable status cell into standalone EPGStatusCell component
with direct Zustand subscription for live progress bar updates
- Guard XMLTV parse trigger for channels linked to SD EPG sources
- Add 15 backend tests covering auth, credentials, signals, and serializers
Migrations: 0023 (username field), 0024 (api_key help_text), 0025 (SD fields + SDScheduleMD5)
This enhancement introduces a powerful custom dummy EPG system that allows users to generate EPG programs on-demand by parsing channel or stream names using configurable regex patterns.
Key Features:
- Custom Pattern Matching: Define regex patterns to extract information from channel/stream names (teams, leagues, times, dates, etc.)
- Flexible Name Source: Choose to parse either the channel name or a specific stream name (by index)
- Timezone-Aware Scheduling: Automatic DST handling using pytz timezone names (e.g., 'US/Eastern', 'Europe/London')
- Time Format Support: Parse both 12-hour (AM/PM) and 24-hour time formats
- Date Parsing: Extract dates from names with flexible month/day/year patterns
- Custom Templates: Format EPG titles and descriptions using captured groups with {placeholder} syntax
- Upcoming/Ended Customization: Define custom titles and descriptions for programs before and after scheduled events
- Live Preview: Test patterns and templates in real-time with sample input
- Smart Program Generation: Automatically creates "Upcoming" and "Ended" programs around scheduled events
Use Cases:
- Sports channels with event details in stream names (e.g., "NHL 01: Bruins VS Leafs @ 8:00PM ET")
- Movie channels with genre/title/year information
- Racing events with driver/track/series details
- Any scenario where EPG data is embedded in channel/stream naming conventions
Technical Implementation:
- Backend: Pattern matching engine with timezone conversion and program scheduling logic
- Frontend: Interactive form with validation, pattern testing, and visual group preview
- Name Source Options: Parse from channel name or selectable stream index (1-based)
- Fallback Behavior: Uses standard dummy EPG if patterns don't match
- Custom Properties: Stores all configuration in EPGSource.custom_properties JSON field
Configuration Options:
- Title Pattern: Extract primary information (required)
- Time Pattern: Extract hour/minute/AM-PM (optional)
- Date Pattern: Extract month/day/year (optional)
- Timezone: Event timezone with automatic DST support
- Program Duration: Length of generated programs in minutes
- Title Template: Format EPG title using captured groups
- Description Template: Format EPG description using captured groups
- Upcoming Title Template: Custom title for programs before event starts (optional)
- Upcoming Description Template: Custom description for programs before event starts (optional)
- Ended Title Template: Custom title for programs after event ends (optional)
- Ended Description Template: Custom description for programs after event ends (optional)
- Name Source: Channel name or stream name
- Stream Index: Which stream to use when parsing stream names (1, 2, 3, etc.)
Closes#293
