Compare commits

...

122 commits

Author SHA1 Message Date
Alex Shnitman
96f5ffe34a docs: restructure README around the wiki as companion documentation
- Move bookmarklet code and Apache/Caddy/swag reverse-proxy configs to
  the wiki; link them
- Group browser extensions, bookmarklets, iOS Shortcut, and Raycast
  under one 'Sending links to MeTube' section with shared CORS/HTTPS
  prerequisites stated once
- Move Runtime & Permissions (PUID/PGID/UMASK) to the top of the
  env-var reference; slim the CORS_ALLOWED_ORIGINS entry
- Link the new Subscriptions guide and Troubleshooting FAQ wiki pages
- State the project scope line in 'Submitting feature requests'
- Docker Compose naming, multi-arch note, trailing whitespace

24,830 -> 21,571 chars against the 25k Docker Hub limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:39:20 +03:00
Alex Shnitman
1eafecd1cc docs: add test gotchas, option checklist, and security invariants to AGENTS.md
- Test-running traps: repo-root cwd requirement, frontend-build ordering,
  the yt_dlp stub quirk in test_ytdl_utils.py
- Note that master is continuously released
- Checklist for adding a per-download option (the split_by_chapters
  pattern, including the commonly missed pieces)
- Security invariants: SSRF guard for URLs, path helpers for anything
  derived from untrusted metadata
- Conventions: OnPush/markForCheck, compact persisted state, yt-dlp
  postprocessor list ordering

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:14:33 +03:00
Alex Shnitman
8fe81dea18 docs: encode project scope boundary in AGENTS.md
Document the line between in-scope work (improving the download-time
write, surfacing yt-dlp built-ins) and out-of-scope work (post-download
tag editing, external metadata lookups, library organization), so
agent-assisted contributions can check their plans against it before
writing code. Follows the decisions on #1025, #1026/#1027, #1028, #1031.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:08:33 +03:00
Alex
f054d84108
Merge pull request #1029 from tjelite1986/readme-library-pairing
docs: document pairing MeTube with a music tagger
2026-07-17 15:00:51 +03:00
tjelite1986
fa02717e12 docs: document pairing MeTube with a music tagger
Short section pointing beets / Picard / Lidarr at AUDIO_DOWNLOAD_DIR,
as suggested in #1027.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 22:39:24 +02:00
Alex Shnitman
9ca78be199 Merge PR #1025: fill missing album-artist metadata for audio downloads
Adds _AlbumArtistPostProcessor, a yt-dlp pre_process postprocessor that
fills album_artist from the '<artist> - Topic' channel/uploader signal,
falling back to the first credited artist, when album metadata exists
but no album artist is set.

Closes #1025.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 23:03:17 +03:00
Alex Shnitman
8071611a84 upgrade dependencies 2026-07-16 23:01:23 +03:00
Your GitHub Name
220f991fae fix: prefer topic channel for album artist 2026-07-16 12:34:03 -07:00
Alex Shnitman
6d0528783c fix: block SSRF via user-submitted URLs
User-submitted URLs were passed straight to yt-dlp's generic extractor,
letting the server fetch internal endpoints (cloud metadata, loopback,
RFC1918 hosts). Add a url_guard.validate_url check at every URL ingress
(add, subscribe, and nested playlist recursion) that rejects non-http(s)
schemes and hosts resolving to non-global addresses, while leaving bare
video IDs and search prefixes untouched.
2026-07-16 21:31:51 +03:00
Your GitHub Name
c104e30451 feat: add AlbumArtistPostProcessor to fill missing album-artist metadata 2026-07-15 15:40:36 -07:00
Alex Shnitman
fdfbfed5e2 fix: prevent playlist/channel title path traversal (closes GHSA-vh67-38x4-w8pc)
Sanitize path separators and .. segments in playlist/channel titles before they are baked into yt-dlp output templates, and refuse downloads whose resolved output directory escapes DOWNLOAD_DIR.
2026-07-13 23:07:39 +03:00
Alex Shnitman
3ea4732c5d fix: harden download lifecycle, subscriptions, and UI robustness
Addresses a full-project review. Backend correctness and availability:

- ytdl: cancel() only SIGKILLs the child's process group when the child
  actually became its own group leader, so a race (or failed setpgrp)
  can no longer kill the whole server; kill the group on cancel and on
  shutdown to avoid orphaned ffmpeg children
- ytdl: dedicated ThreadPoolExecutor for download supervision so active
  downloads can't starve extract_info / live probes on the default pool
- ytdl/main/subscriptions: route fire-and-forget tasks through a
  bg_tasks helper that keeps a strong ref and logs failures
- subscriptions: run flat-playlist extraction in an executor and check
  feeds with bounded concurrency so one slow feed can't block the loop;
  set last_checked on failure so broken feeds aren't retried every 60s
- main: validate ids on /start & /delete and numeric env vars at startup;
  return 400 (not 500) on bad subscriptions/update input; serve /history
  from memory; move get_custom_dirs off the event loop; restrict t=
  stripping to YouTube hosts; drop double percent-decode in state guard
- dl_formats/ytdl: enforce requested caption format via
  FFmpegSubtitlesConvertor and strip VTT header metadata only in the
  pre-cue region so real dialogue is preserved
- ytdl: throttle progress events, dedup adds against pending, clear
  filename/size on error and reject out-of-dir trashcan deletes, pin
  fork start-method on Linux only

Frontend:

- retry deletes the done record only after a successful re-add
- surface HTTP errors for delete/start and reset the deleting flag
- ignore late 'updated' events for rows no longer in the queue
- track table rows by map key; FileSizePipe uses base-1024

Also: HTTPS-aware Docker healthcheck, dead-code removal, and shared
helpers for path-containment and yt-dlp option merging. Adds/updates
unit tests throughout (250 backend tests passing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 08:08:14 +03:00
Alex Shnitman
e2c777842e fix: honor OUTPUT_TEMPLATE for channel downloads (closes #1024)
Detect YouTube channel tabs that yt-dlp reports as playlists so channel downloads use OUTPUT_TEMPLATE_CHANNEL and its empty fallback instead of OUTPUT_TEMPLATE_PLAYLIST.
2026-07-10 09:49:59 +03:00
Alex Shnitman
c34a18de7a upgrade dependencies 2026-07-10 09:48:48 +03:00
Alex
d6bcf182c5
Merge pull request #1023 from lanthaler/no-entries-archive
Don't mark a subscription as broken just because all entries are filtered out as they have already been downloaded
2026-07-09 08:17:00 +03:00
Markus Lanthaler
ad90609c9b Don't mark a subscription as broken just because all entries are filtered out as they have already been downloaded
This happens if archive.txt is used
2026-07-08 22:54:33 +00:00
Alex
5315630ab0
Merge pull request #1021 from mvanhorn/fix/960-atomic-store-nfs-eperm
fix: fall back to a direct write when AtomicJsonStore.save cannot use a temp file (NFS EPERM)
2026-07-05 21:26:49 +03:00
Matt Van Horn
54463baf0e
fix: fsync parent dir after direct-write fallback for durability parity 2026-07-05 04:11:12 -07:00
Matt Van Horn
b00d4785ee
fix: serialize state before truncating in the direct-write fallback 2026-07-05 04:04:54 -07:00
Matt Van Horn
96e88a3555
fix: force 0600 on fallback state rewrites, not just creation 2026-07-05 04:00:59 -07:00
Matt Van Horn
49a46a7d1c
fix: create fallback state file with owner-only 0600 permissions 2026-07-05 03:57:19 -07:00
Matt Van Horn
961b54aa83
fix: make fsync best-effort so only mkstemp/replace failures fall back 2026-07-05 03:53:22 -07:00
Matt Van Horn
e0549d6c24
fix: surface real storage errors from direct-write fsync fallback 2026-07-05 03:49:15 -07:00
Matt Van Horn
f315b75bb2
fix: limit atomic-write fallback to atomic-unsupported errnos 2026-07-05 03:44:57 -07:00
Matt Van Horn
c2c129db61
fix: fall back to direct write when atomic state save hits EPERM on NFS 2026-07-05 03:42:07 -07:00
Alex
363f159a0a
Merge pull request #1020 from alexta69/copilot/fix-dockerhub-build-push
Pin pnpm version to fix Docker build failure
2026-07-05 07:51:08 +03:00
copilot-swe-agent[bot]
38c0ca22f4
Pin pnpm version in packageManager field to fix Docker build
corepack prepare pnpm --activate without a version was resolving to
12.0.0-alpha.0 (broken pre-release), causing the dockerhub-build-push
job to fail. Adding packageManager field pins it to a stable release.
2026-07-05 04:21:32 +00:00
copilot-swe-agent[bot]
24ae8f0742
Initial plan 2026-07-05 04:18:28 +00:00
AutoUpdater
0a946cc352 upgrade yt-dlp from 2026.6.9 to 2026.7.4 2026-07-05 00:28:09 +00:00
Alex Shnitman
51fd203b71 upgrade to Angular 22 2026-06-28 21:12:20 +03:00
Alex Shnitman
d136344c26 upgrade dependencies 2026-06-28 21:08:08 +03:00
dependabot[bot]
33f1412fac Bump actions/checkout from 6 to 7 in the github-actions group
Bumps the github-actions group with 1 update: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 6 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-28 21:06:58 +03:00
Alex Shnitman
ce897ee009 fix open download of cookie files 2026-06-20 09:52:34 +03:00
Alex Shnitman
dd1b4c2436 upgrade dependencies 2026-06-20 09:52:34 +03:00
Alex
8752b500d6
Merge pull request #1004 from akeeton/docker-audio-download-dir
Create AUDIO_DOWNLOAD_DIR in Docker image
2026-06-18 06:45:50 +03:00
Andrew Keeton
04b9366764 Incorporate PR feedback
Move the default assignment of AUDIO_DOWNLOAD_DIR from the Dockerfile to docker-entrypoint.sh, and change the default value from "/downloads" to $DOWNLOAD_DIR.
2026-06-17 17:17:44 -04:00
Alex Shnitman
b73e95f405 upgrade dependencies 2026-06-16 21:57:07 +03:00
Alex Shnitman
64d0d62878 fix empty PUBLIC_HOST_AUDIO_URL handling (closes #1010) 2026-06-16 21:47:07 +03:00
Alex Shnitman
37f7af0555 fix batch download (closes #1008) 2026-06-16 21:37:05 +03:00
Alex Shnitman
5aa7d033e2 review fixes 2026-06-16 21:35:07 +03:00
Alex Shnitman
5429200fba support live streams (closes #302, closes #752, closes #978) 2026-06-13 17:39:14 +03:00
Alex Shnitman
72d60ea55a upgrade dependencies 2026-06-12 12:45:38 +03:00
Alex Shnitman
a9b2e07a59 nightly release README update 2026-06-12 10:16:19 +03:00
Andrew Keeton
d157444877 Create AUDIO_DOWNLOAD_DIR in Docker image 2026-06-11 17:22:55 -04:00
AutoUpdater
e30a24ff70 upgrade yt-dlp from 2026.3.17 to 2026.6.9 2026-06-10 00:36:22 +00:00
Alex Shnitman
ee20512410 add option for following nightly yt-dlp releases (closes #999) 2026-06-06 09:42:26 +03:00
Alex Shnitman
897d52cd0d styling improvements 2026-05-30 15:35:52 +03:00
Alex Shnitman
baa72c0e94 fix pnpm upgrade to the correct package age limit 2026-05-29 14:36:05 +03:00
Alex Shnitman
66d8fa570b Merge branch 'pr-977' 2026-05-29 14:14:05 +03:00
Alex Shnitman
cf2d2dd465 review fixes 2026-05-29 14:13:47 +03:00
Alex Shnitman
0b5617e96c Merge branch 'pr-990' (iOS Web Share for completed downloads) 2026-05-29 13:25:22 +03:00
Alex Shnitman
56c0ad3b5f fix catch 2026-05-29 13:23:25 +03:00
Alex Shnitman
4478d1394e upgrade dependencies 2026-05-29 13:20:04 +03:00
Helmut
ad92607a21 fix(ui): drop redundant tooltip on share button
iOS doesn't have hover, so the tooltip only ever showed on desktop —
where the share-arrow glyph is universally recognised anyway. Aria-
label stays for screen readers.
2026-05-29 05:24:49 +02:00
Helmut
6ff364aacf feat(ui): warn before share + surface failures for large files
Web Share fails silently when iOS' share sheet refuses the payload,
typically because the file exceeds the platform's soft size limit
(~50–100 MB depending on iOS version). The previous patch logged to
the console but the user saw nothing — staring at a button that
'does nothing' is poor UX.

Adds two layers of feedback:

1. Pre-flight size check (SHARE_SIZE_WARN_BYTES = 80 MB, conservative
   relative to iOS' actual limit) with a confirm() dialog before the
   fetch. Avoids spending bandwidth pulling a 150 MB blob into the
   browser only for navigator.canShare to reject it.

2. Surfaces canShare-rejection AND share()-failure as a visible
   alert() suggesting the user fall back to the download link next
   to the share button.

Tested locally with files from 0.7 MB up to 150.7 MB: small files
share unchanged, the 150 MB file now produces a pre-flight warning
the user can dismiss, and any subsequent rejection produces a clear
alert instead of silently no-op'ing.
2026-05-29 04:56:52 +02:00
Helmut
39a8948976 feat(ui): add iOS Web Share button next to download link
Adds a share button to the completed-list action row that hands the
downloaded file off to the platform share sheet via navigator.share().
On iOS Safari/Chrome this surfaces the native Save-to-Photos / Save-to-
Files / AirDrop options for videos and images, and Files / 3rd-party
app targets for audio. On platforms without Web Share support (Desktop
Firefox/Chrome/Safari) the button hides itself; the existing download
link remains the universal fallback.

Implementation notes:
- canShareDownloads() requires both navigator.share AND navigator.canShare
  (Desktop Safari has the former without the latter; we always intend
  to share a file, not a URL)
- shareDownload() fetches the file via the existing buildDownloadLink()
  helper, wraps it in a File, then runs canShare() before share() so we
  can bail out cleanly on platforms that reject the MIME type
- AbortError (user dismisses sheet) is silenced; other errors logged
- Tooltip on the button explains the iOS behaviour briefly

Refs alexta69/metube#582 — addresses the 'add to Photos.app' request
without depending on the iOS Shortcut, which has had reliability issues
(cf #763).
2026-05-28 07:34:13 +02:00
Sean McCollum
f0348581c2 remove circle and make labels with help text have an underline 2026-05-25 00:13:12 -07:00
Sean McCollum
e2773db65a make ui more mobile mobile-friendly
Remove popovers and replace with question icons you have to click
Replace combobox in output > Download Folder with an autocomplete
2026-05-04 10:42:17 -07:00
Alex Shnitman
5d96a581b9 allow filtering out members-only videos in subscriptions (closes #971) 2026-04-28 22:02:05 +03:00
Alex Shnitman
4f83174d05 implement time-clipped downloads (closes #969, replaces #907) 2026-04-26 23:07:50 +03:00
Alex Shnitman
91ee8312bf title filter for subscriptions (closes #968) 2026-04-26 22:51:48 +03:00
dependabot[bot]
d89a5ddbe5
Bump aquasecurity/trivy-action in the github-actions group
Bumps the github-actions group with 1 update: [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action).


Updates `aquasecurity/trivy-action` from 0.35.0 to 0.36.0
- [Release notes](https://github.com/aquasecurity/trivy-action/releases)
- [Commits](https://github.com/aquasecurity/trivy-action/compare/0.35.0...v0.36.0)

---
updated-dependencies:
- dependency-name: aquasecurity/trivy-action
  dependency-version: 0.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-04-26 16:12:39 +00:00
Alex Shnitman
abb9492d21 upgrade dependencies 2026-04-21 16:20:39 +03:00
Alex Shnitman
23de9824f0 cr fixes 2026-04-21 16:13:58 +03:00
rdiaz738
0ea934c08f Updated import and fixed race condition 2026-04-20 17:24:16 -07:00
Alex Shnitman
e9f979b349 fix yt-dlp options overrides (closes #958) 2026-04-18 08:46:29 +03:00
Alex Shnitman
ab42325db5 upgrade dependencies 2026-04-16 22:30:42 +03:00
Alex Shnitman
1a32eba474 fix PUBLIC_HOST_URL without a trailing slash (closes #959) 2026-04-16 22:08:08 +03:00
Alex Shnitman
29ccc42409 don't run workflow on README changes 2026-04-13 20:49:10 +03:00
Alex Shnitman
f2d71cbe2e add more CORS details 2026-04-13 20:45:20 +03:00
Alex Shnitman
03f71fd257 fix asterisk CORS_ALLOWED_ORIGINS, mentioned in #955 2026-04-13 19:02:27 +03:00
Alex Shnitman
210c607c53 fix pnpm build 2026-04-12 23:07:22 +03:00
dependabot[bot]
381896901a
Bump softprops/action-gh-release from 2 to 3 in the github-actions group
Bumps the github-actions group with 1 update: [softprops/action-gh-release](https://github.com/softprops/action-gh-release).


Updates `softprops/action-gh-release` from 2 to 3
- [Release notes](https://github.com/softprops/action-gh-release/releases)
- [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md)
- [Commits](https://github.com/softprops/action-gh-release/compare/v2...v3)

---
updated-dependencies:
- dependency-name: softprops/action-gh-release
  dependency-version: '3'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-04-12 16:12:38 +00:00
Alex Shnitman
4330d3b6c6 fix yt-dlp options examples 2026-04-10 14:06:08 +03:00
Alex Shnitman
06c4a2c4a8 update documentation 2026-04-10 08:38:32 +03:00
Alex Shnitman
388aeb180d Merge branch 'bgervan/master' 2026-04-10 08:10:00 +03:00
Alex Shnitman
aa60420ead document CORS_ALLOWED_ORIGINS variable 2026-04-10 08:09:20 +03:00
Benjamin Gervan
a6e8617ad8 Don't mark live streams as seen 2026-04-10 06:41:45 +02:00
az10b
0072d3488a Fix permissive CORS policy that allows cross-origin attacks
The on_prepare handler unconditionally reflected the Origin request
header into Access-Control-Allow-Origin, and Socket.IO was configured
with cors_allowed_origins='*'. This allowed any website to make
authenticated cross-origin requests to all API endpoints, enabling
cross-origin download initiation, cookie overwrite, and data deletion.

Replace the blanket origin reflection with an explicit allowlist via
the CORS_ALLOWED_ORIGINS environment variable. When unset, cross-origin
requests are denied by default. Users who need cross-origin access can
set CORS_ALLOWED_ORIGINS to a comma-separated list of trusted origins.
2026-04-09 19:45:51 -05:00
Alex Shnitman
0b3645aea1 upgrade dependencies 2026-04-09 21:00:26 +03:00
Alex Shnitman
2c838e3d3d Merge branch 'dependabot/github_actions/github-actions-7530ffc9b9' of https://github.com/alexta69/metube into McSwindler/master 2026-04-09 20:59:13 +03:00
McSwindler
d38d7bd1b1 fix: handle playlists that don't supply video ids 2026-04-09 10:15:11 -05:00
dependabot[bot]
b7709d3536
Bump astral-sh/setup-uv from 6 to 7 in the github-actions group
Bumps the github-actions group with 1 update: [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv).


Updates `astral-sh/setup-uv` from 6 to 7
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](https://github.com/astral-sh/setup-uv/compare/v6...v7)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-04-05 16:12:43 +00:00
Alex
1f79883b75
Merge pull request #944 from jacinli/codex/fix-subscription-enabled-parsing
Fix string boolean parsing for subscription enabled updates
2026-04-05 10:25:46 +03:00
jacinli
373692ac65 fix: parse string boolean values when updating subscriptions 2026-04-05 14:05:59 +08:00
Alex Shnitman
54680c405c explain yt-dlp configuration in detail 2026-04-04 12:58:47 +03:00
Alex Shnitman
dd0f98d12f change option presets to be multi-select 2026-04-04 10:25:46 +03:00
Alex Shnitman
d41bdf61e2 finalize custom options (closes #563, #482, #261, #681) 2026-04-03 13:20:37 +03:00
copilot-swe-agent[bot]
a02abf5853
Keep override controls on dedicated row
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/aef158da-f919-4a3d-a5ee-b71df51c124d

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-03 09:21:44 +00:00
copilot-swe-agent[bot]
b16e597125
Fix frontend test typing for override flag
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/31b4274d-cf48-4260-b73b-633cbcd2bb09

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-03 09:07:34 +00:00
copilot-swe-agent[bot]
6e9b2dd7b3
Gate manual yt-dlp overrides behind flag
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/31b4274d-cf48-4260-b73b-633cbcd2bb09

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-03 09:05:19 +00:00
copilot-swe-agent[bot]
565a715037
feat: add per-download yt-dlp presets and overrides
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/8a3119fc-63d1-4508-a196-8c50ff248812

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-03 06:16:12 +00:00
Alex
b4d497f53d
Merge pull request #937 from alexta69/copilot/check-issue-692
Propagate missing playlist context fields (playlist_count, playlist_autonumber, etc.)
2026-04-02 10:55:00 +03:00
Alex Shnitman
0cba61c9a4 update README 2026-04-02 10:52:56 +03:00
Alex Shnitman
9858157581 Merge branch 'copilot/fix-healthcheck-failure-ipvlan' of https://github.com/alexta69/metube into copilot/check-issue-692 (closes #936) 2026-04-02 10:52:11 +03:00
copilot-swe-agent[bot]
d7eaaaa94b Add clarifying comments for n_entries and __last_playlist_index fields (closes #692)
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/b5aeb55a-3197-4a14-b8b4-96c9a67796e8

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-02 10:51:03 +03:00
copilot-swe-agent[bot]
771ba52d53
Use PORT env variable in Dockerfile HEALTHCHECK instead of hardcoded 8081
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/899e7074-fd3d-4538-8bad-8ee6804d5052

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-02 07:25:14 +00:00
copilot-swe-agent[bot]
1cc27d3f55
Initial plan 2026-04-02 07:23:14 +00:00
copilot-swe-agent[bot]
981e6c1003
Propagate missing playlist context fields (playlist_count, playlist_autonumber, n_entries, __last_playlist_index)
The playlist/channel processing loop now sets playlist_count,
playlist_autonumber, n_entries, and __last_playlist_index on each
video entry so that templates like %(playlist_autonumber)s,
%(playlist_count)s, and %(playlist_index&{} - |)s resolve correctly
instead of showing NA.

Also updates _compact_persisted_entry to preserve n_entries and
__last_playlist_index across restarts.

Fixes #692

Agent-Logs-Url: https://github.com/alexta69/metube/sessions/b5aeb55a-3197-4a14-b8b4-96c9a67796e8

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-01 19:59:32 +00:00
copilot-swe-agent[bot]
b17e1e5668
Add explanatory comment for fake STR_FORMAT_RE_TMPL key group in tests
Agent-Logs-Url: https://github.com/alexta69/metube/sessions/0ae5ff34-540f-4fc8-a81c-358fb92b7c15

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-01 19:34:09 +00:00
copilot-swe-agent[bot]
c1b5540332
Replace custom template substitution with yt-dlp's evaluate_outtmpl
Replace the hand-rolled _outtmpl_substitute_field() / _compile_outtmpl_pattern()
with a new _resolve_outtmpl_fields() that delegates to yt-dlp's
YoutubeDL.evaluate_outtmpl().  This gives playlist/channel output templates
access to yt-dlp's full template syntax: defaults (%(field|fallback)s),
conditional formatting (%(field&prefix {})s), math (%(field+N)d),
datetime formatting (%(field>%Y-%m-%d)s), and more.

Only field references whose root name matches the targeted prefix (e.g.
"playlist" or "channel") are resolved; all other references remain as
template placeholders for yt-dlp to fill during the actual download.

Agent-Logs-Url: https://github.com/alexta69/metube/sessions/0ae5ff34-540f-4fc8-a81c-358fb92b7c15

Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-04-01 19:31:27 +00:00
Alex Shnitman
483575d24a add subscriptions; change persistence file format to JSON (closes #901, #76, #113, #170, #242, #444, #503, #555, #566) 2026-04-01 14:33:24 +03:00
Alex Shnitman
84c6418f91 fix pickle (closes #814) 2026-03-21 12:42:17 +02:00
Alex Shnitman
a1f2fe3e73 implement tests 2026-03-20 13:12:31 +02:00
AutoUpdater
0bf508dbc6 upgrade yt-dlp from 2026.3.13 to 2026.3.17 2026-03-18 00:14:51 +00:00
Alex
104d547150
Update Trivy action version in workflow 2026-03-15 21:06:19 +02:00
Alex Shnitman
289133e507 upgrade dependencies 2026-03-15 20:54:46 +02:00
Alex Shnitman
7fa1fc7938 code review fixes 2026-03-15 20:53:13 +02:00
Alex Shnitman
04959a6189 upgrade dependencies 2026-03-14 12:05:04 +02:00
AutoUpdater
8b0d682b35 upgrade yt-dlp from 2026.3.3 to 2026.3.13 2026-03-14 00:13:08 +00:00
Alex Shnitman
475aeb91bf add status indicator when adding a URL 2026-03-13 19:49:18 +02:00
Alex Shnitman
5c321bfaca reoganize quality and codec selections 2026-03-13 19:47:36 +02:00
CyCl0ne
56826d33fd Add video codec selector and codec/quality columns in done list
Allow users to prefer a specific video codec (H.264, H.265, AV1, VP9)
when adding downloads. The selector filters available formats via
yt-dlp format strings, falling back to best available if the preferred
codec is not found. The completed downloads table now shows Quality
and Codec columns.
2026-03-09 08:59:01 +01:00
Alex Shnitman
3b0eaad67e Merge branch 'dependabot/github_actions/github-actions-292e5e2d7a' of https://github.com/alexta69/metube into feature/download-timestamp 2026-03-08 22:19:01 +02:00
dependabot[bot]
2a166ccf1f
Bump the github-actions group with 4 updates
Bumps the github-actions group with 4 updates: [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action), [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action), [docker/login-action](https://github.com/docker/login-action) and [docker/build-push-action](https://github.com/docker/build-push-action).


Updates `docker/setup-qemu-action` from 3 to 4
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v3...v4)

Updates `docker/setup-buildx-action` from 3 to 4
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4)

Updates `docker/login-action` from 3 to 4
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

Updates `docker/build-push-action` from 6 to 7
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v6...v7)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: docker/setup-buildx-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: docker/build-push-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-08 16:12:41 +00:00
CyCl0ne
3bbe1e8424 Add "Downloaded" timestamp column to completed downloads list
Display the completion time for each download in the done list.
The backend already stores a nanosecond timestamp on DownloadInfo;                                     this wires it up to the frontend using Angular's DatePipe.
2026-03-08 14:56:16 +01:00
Alex
a2740375be
Merge pull request #922 from alexta69/copilot/scan-open-issues-for-fixes
Fix #898, #542, #561, #504: config leak, custom-dirs fallback, NTFS path sanitization
2026-03-07 16:06:12 +02:00
copilot-swe-agent[bot]
2736425e19 Revert #504 default change: restore original playlist/channel output templates
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-03-07 14:00:29 +00:00
copilot-swe-agent[bot]
0d905c0b61 Fix issues #898, #542, #561, #504
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-03-07 07:19:05 +00:00
copilot-swe-agent[bot]
6de4a56f28 Remove DEFAULT_DOWNLOAD_FOLDER feature
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-03-07 06:37:09 +00:00
copilot-swe-agent[bot]
1f4c4df847 Implement DEFAULT_DOWNLOAD_FOLDER and CLEAR_COMPLETED_AFTER features (#875, #869)
Co-authored-by: alexta69 <7450369+alexta69@users.noreply.github.com>
2026-03-06 15:37:35 +00:00
copilot-swe-agent[bot]
d211f24e00 Initial plan 2026-03-06 15:27:55 +00:00
62 changed files with 14511 additions and 4104 deletions

View file

@ -4,40 +4,84 @@ on:
push:
branches:
- 'master'
paths-ignore:
- '**.md'
jobs:
quality-checks:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: lts/*
- name: Enable pnpm
run: corepack enable
- name: Install frontend dependencies
working-directory: ui
run: pnpm install --frozen-lockfile
- name: Run frontend lint
working-directory: ui
run: pnpm run lint
- name: Build frontend
working-directory: ui
run: pnpm run build
- name: Run frontend tests
working-directory: ui
run: pnpm exec ng test --watch=false
env:
CI: true
- name: Install uv
uses: astral-sh/setup-uv@v7
- name: Install Python dependencies
run: uv sync --frozen --group dev
- name: Run backend smoke checks
run: python -m compileall app
- name: Run backend tests
run: uv run pytest app/tests/
- name: Run Trivy filesystem scan
uses: aquasecurity/trivy-action@v0.36.0
with:
scan-type: fs
scan-ref: .
format: table
severity: CRITICAL,HIGH
dockerhub-build-push:
needs: quality-checks
runs-on: ubuntu-latest
steps:
-
name: Get current date
id: date
run: echo "::set-output name=date::$(date +'%Y.%m.%d')"
run: echo "date=$(date +'%Y.%m.%d')" >> "$GITHUB_OUTPUT"
-
name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
-
name: Set up QEMU
uses: docker/setup-qemu-action@v3
uses: docker/setup-qemu-action@v4
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
-
name: Login to DockerHub
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Login to GitHub Container Registry
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
-
name: Build and push
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
context: .
platforms: linux/amd64,linux/arm64
@ -74,7 +118,7 @@ jobs:
id: date
run: echo "date=$(date +'%Y.%m.%d')" >> $GITHUB_OUTPUT
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Get commits since last release
@ -167,7 +211,7 @@ jobs:
git push origin ":refs/tags/$TAG_NAME" || true
fi
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
uses: softprops/action-gh-release@v3
with:
tag_name: ${{ steps.date.outputs.date }}
name: Release ${{ steps.date.outputs.date }}

View file

@ -10,7 +10,7 @@ jobs:
steps:
-
name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
token: ${{ secrets.AUTOUPDATE_PAT }}
-

4
.vscode/launch.json vendored
View file

@ -13,12 +13,16 @@
"env": {
"DOWNLOAD_DIR": "${env:USERPROFILE}/Downloads",
"STATE_DIR": "${env:TEMP}",
"ALLOW_YTDL_OPTIONS_OVERRIDES": "true",
"YTDL_OPTIONS_PRESETS": "{\"sponsorblock\": {\"postprocessors\": [{\"key\": \"SponsorBlock\", \"categories\": [\"sponsor\", \"selfpromo\", \"interaction\"]}, {\"key\": \"ModifyChapters\", \"remove_sponsor_segments\": [\"sponsor\", \"selfpromo\", \"interaction\"]}]}, \"embed-subs\": {\"writesubtitles\": true, \"writeautomaticsub\": true, \"subtitleslangs\": [\"en\", \"de\"], \"postprocessors\": [{\"key\": \"FFmpegEmbedSubtitle\"}]}, \"limit-rate\": {\"ratelimit\": 5000000}}",
}
},
"osx": {
"env": {
"DOWNLOAD_DIR": "${env:HOME}/Downloads",
"STATE_DIR": "${env:TMPDIR}",
"ALLOW_YTDL_OPTIONS_OVERRIDES": "true",
"YTDL_OPTIONS_PRESETS": "{\"sponsorblock\": {\"postprocessors\": [{\"key\": \"SponsorBlock\", \"categories\": [\"sponsor\", \"selfpromo\", \"interaction\"]}, {\"key\": \"ModifyChapters\", \"remove_sponsor_segments\": [\"sponsor\", \"selfpromo\", \"interaction\"]}]}, \"embed-subs\": {\"writesubtitles\": true, \"writeautomaticsub\": true, \"subtitleslangs\": [\"en\", \"de\"], \"postprocessors\": [{\"key\": \"FFmpegEmbedSubtitle\"}]}, \"limit-rate\": {\"ratelimit\": 5000000}}",
}
},
"console": "integratedTerminal"

157
AGENTS.md Normal file
View file

@ -0,0 +1,157 @@
# Agent Guidelines
## Project scope — read this before planning a feature
MeTube's contract is: give it a URL, it runs yt-dlp well, and correct files appear.
The maintainer holds a deliberate line on what belongs inside that contract, and PRs
on the wrong side of it are declined **regardless of code quality**. Check your plan
against this line before writing any code.
**In scope — improving the write itself:**
- Features that make the file yt-dlp writes at download time come out more correct,
using only data the extractor already provides (e.g. filling a missing album-artist
tag from the extractor's own metadata).
- Surfacing functionality yt-dlp itself owns and maintains as first-class UI options
(e.g. a SponsorBlock toggle that just passes yt-dlp postprocessor params).
- Download queue, subscriptions, output templates, and UI improvements to the
download workflow.
**Out of scope — managing files after they exist:**
- Tag editors, metadata dialogs, or any workflow that rewrites files after the
download has finished. This holds even for slimmed-down versions.
- Lookups against external metadata services (iTunes, Deezer, MusicBrainz, etc.).
More broadly: any new dependency on a third-party API, or new network egress from
self-hosted instances, beyond what yt-dlp itself performs.
- Library organization: moving/renaming existing files into Artist/Album layouts,
watch-folder processing, and similar media-manager features. Dedicated tools
(beets, MusicBrainz Picard, Lidarr) do this properly; the README points users
to them.
**Corollaries that shape borderline PRs:**
- Site-specific intelligence (parsing playlist-ID prefixes, URL path conventions,
and other platform internals) is extractor work and belongs upstream in yt-dlp,
not re-implemented here — it silently breaks when the platform changes and
MeTube would own the breakage.
- Prefer enriching yt-dlp's info dict and letting its existing pipeline
(FFmpegMetadata etc.) do the writing, over adding custom per-format tag-writing
code to MeTube.
- Supplemental processing must never fail a download that otherwise succeeded:
warn and continue, don't raise.
- Keep feature scope minimal on first submission. A hardcoded sensible default
beats a configuration surface; follow-ups can add options when users actually
ask. PRs that bundle several "reasonable next steps" invite rejection of the
whole.
If a feature idea fails this test, the accepted alternative is usually a README
section documenting how to pair MeTube with the right dedicated tool.
## README.md size constraint
The README.md is synced to Docker Hub, which has a **25,000 character limit**.
Any change to README.md **must** keep the file under 25,000 characters (`wc -c README.md`).
If an addition would exceed the limit, trim existing prose elsewhere — prefer tightening verbose descriptions over removing sections.
## Tech stack
- **Backend:** Python 3.13+, aiohttp, python-socketio 5.x, yt-dlp
- **Frontend:** Angular 22, TypeScript, Bootstrap 5, SASS, ngx-socket-io
- **Package managers:** uv (Python), pnpm (frontend)
- **Container:** Multi-stage Docker (Node builder + Python runtime), multi-arch (amd64/arm64)
## Build & test commands
```bash
# Frontend (run from ui/)
pnpm install --frozen-lockfile
pnpm run lint
pnpm run build
pnpm exec ng test --watch=false
# Backend (run from repo root)
uv sync --frozen --group dev
python -m compileall app
uv run pytest app/tests/
```
All of these run in CI (`.github/workflows/main.yml`) on every push to master and must pass.
Gotchas:
- Backend tests must run **from the repo root**: `main.py` resolves the static-assets
path relative to the cwd, and several test modules import `main`. Running from
`app/` makes five test modules fail to import.
- The frontend must be **built before** running backend tests (same reason — the
assets at `ui/dist/metube/browser` must exist). The command order above is
load-bearing.
- `app/tests/test_ytdl_utils.py` stubs `yt_dlp` at import time. Run standalone,
two tests fail with `AttributeError: <module 'yt_dlp'> does not have the
attribute 'YoutubeDL'`; under the full suite the real module is imported first
and they pass. This is a known quirk, not a bug to fix in the code under test.
Every non-markdown push to master builds multi-arch Docker images and cuts a dated
release the same day. **Master is continuously released** — a PR must be
release-ready exactly as merged; there is no stabilization window for follow-up
fixes.
## Code style
Follow `.editorconfig`:
- Python: 4-space indent
- Everything else (TypeScript, YAML, JSON, HTML): 2-space indent
- UTF-8, LF line endings, trim trailing whitespace, final newline
Frontend additionally uses ESLint (`ui/eslint.config.js`) and Prettier (config in `ui/package.json`: `printWidth=100`, `singleQuote=true`).
## Project structure
```
app/main.py — HTTP server, Socket.IO events, REST API routes, Config class
app/ytdl.py — Download queue logic, yt-dlp integration
app/subscriptions.py — Channel/playlist subscription manager
app/state_store.py — JSON-based persistent storage with atomic writes
app/dl_formats.py — Video/audio codec/quality mapping
app/tests/ — pytest tests (asyncio_mode=auto)
ui/src/app/ — Angular standalone components (no NgModules)
```
## Key conventions
- Backend configuration lives in the `Config` class in `app/main.py` with env-var defaults in `_DEFAULTS`. New env vars go there.
- Real-time communication uses Socket.IO events, not REST polling.
- Frontend uses standalone Angular components with `inject()` for DI, RxJS Subjects for state, and `takeUntilDestroyed()` for cleanup.
- Frontend components use OnPush change detection: subscribe callbacks must call `cdr.markForCheck()`.
- State is persisted as JSON files via `AtomicJsonStore` in `app/state_store.py`.
- Persisted state stays compact: the completed queue deliberately drops bulky entry data (see `_compact_persisted_entry` in `app/ytdl.py`). Don't expand what gets persisted without discussion.
- Custom yt-dlp postprocessors added to `ytdl_params['postprocessors']` run in **list order** within a stage. When combining postprocessors, mirror the ordering the yt-dlp CLI would produce (e.g. sponsor-segment removal before chapter splitting).
- No pre-commit hooks — linting and tests are enforced in CI only.
## Checklist: adding a per-download option
New options on the download form (the `split_by_chapters` pattern) need **all** of
these pieces — the last three are the ones commonly missed:
1. `parse_download_options` in `app/main.py`.
2. A field on `DownloadInfo` in `app/ytdl.py`.
3. A `hasattr` backfill in `DownloadInfo.__setstate__` for old persisted records.
4. The safe-deserialization field list in `app/ytdl.py`.
5. UI form control + cookie persistence in `ui/src/app/app.ts` / `app.html`, and
the payload in `downloads.service.ts` (plus its spec).
6. The redownload path in `app.ts`, so retries carry the option.
7. If the option makes sense for unattended downloads: threading through
`app/subscriptions.py` (`SubscriptionInfo` field, serializer, add/update
routes, the enqueue call) — or a note in the PR that it's deliberately
direct-downloads-only.
## Security invariants
User input and extractor-provided metadata (titles, playlist names, URLs) are
untrusted. Use the existing guards instead of hand-rolling:
- User-submitted URLs go through the SSRF guard (see `test_url_guard.py` for the
expected behavior).
- Anything that becomes a filesystem path goes through `_is_within_directory` and
`_sanitize_path_component` in `app/ytdl.py` — including values that arrive via
yt-dlp metadata, which sites can influence.

View file

@ -1,4 +1,8 @@
FROM node:lts-alpine AS builder
# Pinned to a major version rather than the lts-alpine floating tag: that tag
# has lagged behind and resolved to a Node patch older than the Angular CLI's
# minimum supported version, breaking the build. node:22-alpine currently
# satisfies @angular/cli's >=22.22.3 requirement.
FROM node:22-alpine AS builder
WORKDIR /metube
COPY ui ./
@ -26,15 +30,12 @@ RUN sed -i 's/\r$//g' docker-entrypoint.sh && \
gosu \
curl \
tini \
file \
gdbmtool \
sqlite3 \
build-essential && \
curl -LsSf https://astral.sh/uv/install.sh | UV_INSTALL_DIR=/usr/local/bin sh && \
UV_PROJECT_ENVIRONMENT=/usr/local uv sync --frozen --no-dev --compile-bytecode && \
uv cache clean && \
rm -f /usr/local/bin/uv /usr/local/bin/uvx /usr/local/bin/uvw && \
curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh -s -- -y v2.7.2 && \
curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh -s -- -y && \
apt-get purge -y --auto-remove build-essential && \
rm -rf /var/lib/apt/lists/* && \
mkdir /.cache && chmod 777 /.cache
@ -63,11 +64,14 @@ ENV PUID=1000
ENV PGID=1000
ENV UMASK=022
ENV DOWNLOAD_DIR /downloads
ENV STATE_DIR /downloads/.metube
ENV TEMP_DIR /downloads
ENV DOWNLOAD_DIR=/downloads
ENV STATE_DIR=/downloads/.metube
ENV TEMP_DIR=/downloads
ENV PORT=8081
VOLUME /downloads
EXPOSE 8081
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
CMD case "$HTTPS" in true|True|on|1) curl -fsSk "https://localhost:${PORT}/";; *) curl -fsS "http://localhost:${PORT}/";; esac || exit 1
# Add build-time argument for version
ARG VERSION=dev

283
README.md
View file

@ -3,7 +3,12 @@
![Build Status](https://github.com/alexta69/metube/actions/workflows/main.yml/badge.svg)
![Docker Pulls](https://img.shields.io/docker/pulls/alexta69/metube.svg)
Web GUI for youtube-dl (using the [yt-dlp](https://github.com/yt-dlp/yt-dlp) fork) with playlist support. Allows you to download videos from YouTube and [dozens of other sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md).
MeTube is a self-hosted web UI for `yt-dlp`, for downloading media from YouTube and [dozens of other sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md). Docker images are multi-arch (amd64/arm64).
Key capabilities:
* Download videos, audio, captions, and thumbnails from a browser UI.
* Download playlists and channels, with configurable output and download options.
* [Subscribe](https://github.com/alexta69/metube/wiki/Subscriptions) to channels and playlists, periodically check for new items, and queue new uploads automatically.
![screenshot1](https://github.com/alexta69/metube/raw/master/screenshot.gif)
@ -13,7 +18,7 @@ Web GUI for youtube-dl (using the [yt-dlp](https://github.com/yt-dlp/yt-dlp) for
docker run -d -p 8081:8081 -v /path/to/downloads:/downloads ghcr.io/alexta69/metube
```
## 🐳 Run using docker-compose
## 🐳 Run using Docker Compose
```yaml
services:
@ -29,13 +34,26 @@ services:
## ⚙️ Configuration via environment variables
Certain values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in docker-compose.
Certain values can be set via environment variables, using the `-e` parameter on the docker command line, or the `environment:` section in Docker Compose.
### 🏠 Runtime & Permissions
* __PUID__: User under which MeTube will run. Defaults to `1000` (legacy `UID` also supported).
* __PGID__: Group under which MeTube will run. Defaults to `1000` (legacy `GID` also supported).
* __UMASK__: Umask value used by MeTube. Defaults to `022`.
* __DEFAULT_THEME__: Default theme to use for the UI, can be set to `light`, `dark`, or `auto`. Defaults to `auto`.
* __LOGLEVEL__: Log level, can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `NONE`. Defaults to `INFO`.
* __ENABLE_ACCESSLOG__: Whether to enable access log. Defaults to `false`.
### ⬇️ Download Behavior
* __MAX_CONCURRENT_DOWNLOADS__: Maximum number of simultaneous downloads allowed. For example, if set to `5`, then at most five downloads will run concurrently, and any additional downloads will wait until one of the active downloads completes. Defaults to `3`.
* __MAX_CONCURRENT_DOWNLOADS__: Maximum number of simultaneous downloads allowed. For example, if set to `5`, then at most five downloads will run concurrently, and any additional downloads will wait until one of the active downloads completes. Defaults to `3`.
* __DELETE_FILE_ON_TRASHCAN__: if `true`, downloaded files are deleted on the server, when they are trashed from the "Completed" section of the UI. Defaults to `false`.
* __DEFAULT_OPTION_PLAYLIST_ITEM_LIMIT__: Maximum number of playlist items that can be downloaded. Defaults to `0` (no limit).
* __SUBSCRIPTION_DEFAULT_CHECK_INTERVAL__: Default minutes between automatic checks for each subscription. Defaults to `60`.
* __SUBSCRIPTION_SCAN_PLAYLIST_END__: Maximum playlist/channel entries to fetch per subscription check (newest-first). Defaults to `50`.
* __SUBSCRIPTION_MAX_SEEN_IDS__: Cap on stored video IDs per subscription to limit state file growth. Defaults to `50000`.
* __CLEAR_COMPLETED_AFTER__: Number of seconds after which completed (and failed) downloads are automatically removed from the "Completed" list. Defaults to `0` (disabled).
### 📁 Storage & Directories
@ -45,7 +63,7 @@ Certain values can be set via environment variables, using the `-e` parameter on
* __CREATE_CUSTOM_DIRS__: Whether to support automatically creating directories within the __DOWNLOAD_DIR__ (or __AUDIO_DOWNLOAD_DIR__) if they do not exist. When enabled, the download directory selector supports free-text input, and the specified directory will be created recursively. Defaults to `true`.
* __CUSTOM_DIRS_EXCLUDE_REGEX__: Regular expression to exclude some custom directories from the dropdown. Empty regex disables exclusion. Defaults to `(^|/)[.@].*$`, which means directories starting with `.` or `@`.
* __DOWNLOAD_DIRS_INDEXABLE__: If `true`, the download directories (__DOWNLOAD_DIR__ and __AUDIO_DOWNLOAD_DIR__) are indexable on the web server. Defaults to `false`.
* __STATE_DIR__: Path to where the queue persistence files will be saved. Defaults to `/downloads/.metube` in the Docker image, and `.` otherwise.
* __STATE_DIR__: Path to where MeTube will store its persistent state files (`queue.json`, `pending.json`, `completed.json`, `subscriptions.json`). Defaults to `/downloads/.metube` in the Docker image, and `.` otherwise.
* __TEMP_DIR__: Path where intermediary download files will be saved. Defaults to `/downloads` in the Docker image, and `.` otherwise.
* Set this to an SSD or RAM filesystem (e.g., `tmpfs`) for better performance.
* __Note__: Using a RAM filesystem may prevent downloads from being resumed.
@ -55,10 +73,14 @@ Certain values can be set via environment variables, using the `-e` parameter on
* __OUTPUT_TEMPLATE__: The template for the filenames of the downloaded videos, formatted according to [this spec](https://github.com/yt-dlp/yt-dlp/blob/master/README.md#output-template). Defaults to `%(title)s.%(ext)s`.
* __OUTPUT_TEMPLATE_CHAPTER__: The template for the filenames of the downloaded videos when split into chapters via postprocessors. Defaults to `%(title)s - %(section_number)s %(section_title)s.%(ext)s`.
* __OUTPUT_TEMPLATE_PLAYLIST__: The template for the filenames of the downloaded videos when downloaded as a playlist. Defaults to `%(playlist_title)s/%(title)s.%(ext)s`. When empty, then `OUTPUT_TEMPLATE` is used.
* __OUTPUT_TEMPLATE_CHANNEL__: The template for the filenames of the downloaded videos when downloaded as a channel. Defaults to `%(channel)s/%(title)s.%(ext)s`. When empty, then `OUTPUT_TEMPLATE` is used.
* __YTDL_OPTIONS__: Additional options to pass to yt-dlp in JSON format. [See available options here](https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/YoutubeDL.py#L222). They roughly correspond to command-line options, though some do not have exact equivalents here. For example, `--recode-video` has to be specified via `postprocessors`. Also note that dashes are replaced with underscores. You may find [this script](https://github.com/yt-dlp/yt-dlp/blob/master/devscripts/cli_to_api.py) helpful for converting from command-line options to `YTDL_OPTIONS`.
* __YTDL_OPTIONS_FILE__: A path to a JSON file that will be loaded and used for populating `YTDL_OPTIONS` above. Please note that if both `YTDL_OPTIONS_FILE` and `YTDL_OPTIONS` are specified, the options in `YTDL_OPTIONS` take precedence. The file will be monitored for changes and reloaded automatically when changes are detected.
* __OUTPUT_TEMPLATE_PLAYLIST__: The template for the filenames of the downloaded videos when downloaded as a playlist. Defaults to `%(playlist_title)s/%(title)s.%(ext)s`. Set to empty to use `OUTPUT_TEMPLATE` instead.
* __OUTPUT_TEMPLATE_CHANNEL__: The template for the filenames of the downloaded videos when downloaded as a channel. Defaults to `%(channel)s/%(title)s.%(ext)s`. Set to empty to use `OUTPUT_TEMPLATE` instead.
* __YTDL_OPTIONS__: Additional options to pass to yt-dlp, as a JSON object. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for details, examples, and available options reference.
* __YTDL_OPTIONS_FILE__: Path to a JSON file containing yt-dlp options. Monitored and reloaded automatically on changes. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options).
* __YTDL_OPTIONS_PRESETS__: Named bundles of yt-dlp options, selectable per download in the UI. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for format and examples.
* __YTDL_OPTIONS_PRESETS_FILE__: Path to a JSON file containing presets. Monitored and reloaded automatically on changes. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options).
* __ALLOW_YTDL_OPTIONS_OVERRIDES__: Whether to show a free-text field in the UI for per-download yt-dlp option overrides. Defaults to `false`. See [Configuring yt-dlp options](#%EF%B8%8F-configuring-yt-dlp-options) for details and security considerations.
* __YTDL_NIGHTLY_UPDATE_TIME__: If set, will cause MeTube to use [nightly yt-dlp builds](https://github.com/yt-dlp/yt-dlp-nightly-builds) instead of the stable releases. Set to the time (`HH:MM`, 24-hour) when you want the daily upgrades and MeTube restart to happen. Defaults to empty (disabled).
### 🌐 Web Server & URLs
@ -70,16 +92,126 @@ Certain values can be set via environment variables, using the `-e` parameter on
* __HTTPS__: Use `https` instead of `http` (__CERTFILE__ and __KEYFILE__ required). Defaults to `false`.
* __CERTFILE__: HTTPS certificate file path.
* __KEYFILE__: HTTPS key file path.
* __CORS_ALLOWED_ORIGINS__: Comma-separated list of origins permitted to make cross-origin requests to the MeTube API; `*` allows all. When unset or empty, all cross-origin requests are denied. Required for browser extensions and bookmarklets — see [Sending links to MeTube](#-sending-links-to-metube).
* __ROBOTS_TXT__: A path to a `robots.txt` file mounted in the container.
### 🏠 Basic Setup
## 🎛️ Configuring yt-dlp options
* __PUID__: User under which MeTube will run. Defaults to `1000` (legacy `UID` also supported).
* __PGID__: Group under which MeTube will run. Defaults to `1000` (legacy `GID` also supported).
* __UMASK__: Umask value used by MeTube. Defaults to `022`.
* __DEFAULT_THEME__: Default theme to use for the UI, can be set to `light`, `dark`, or `auto`. Defaults to `auto`.
* __LOGLEVEL__: Log level, can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `NONE`. Defaults to `INFO`.
* __ENABLE_ACCESSLOG__: Whether to enable access log. Defaults to `false`.
MeTube lets you customize how [yt-dlp](https://github.com/yt-dlp/yt-dlp) behaves at three levels, from broadest to most specific:
1. **Global options** — apply to every download by default.
2. **Presets** — named bundles of options that users can pick per download from the UI.
3. **Per-download overrides** — free-form options entered in the UI for a single download.
When a download starts, these layers are combined in order. If the same option appears in more than one layer, the more specific one wins: per-download overrides beat presets, and presets beat global options.
In JSON presets and overrides, setting an option to **`null`** clears that option for that download (for example, `"download_archive": null` overrides a global archive path so the archive is not used). This follows yt-dlps usual meaning of `None` for that option.
### Option format
yt-dlp options in MeTube are expressed as JSON objects. The keys are yt-dlp API option names, which roughly correspond to command-line flags with dashes replaced by underscores. For example, the command-line flag `--write-subs` becomes `"writesubtitles": true` in JSON.
> **Tip:** Some command-line flags don't have a direct single-key equivalent — for instance, `--embed-thumbnail` and `--recode-video` must be expressed via `"postprocessors"`. A full list of available API options can be found [in the yt-dlp source](https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/YoutubeDL.py#L224), and [this conversion script](https://github.com/yt-dlp/yt-dlp/blob/master/devscripts/cli_to_api.py) can help translate command-line flags to their API equivalents.
### Global options
Global options form the baseline for every download. There are two ways to define them, and you can use either or both:
**Inline via environment variable** (`YTDL_OPTIONS`) — pass a JSON object directly:
```yaml
environment:
- 'YTDL_OPTIONS={"writesubtitles": true, "subtitleslangs": ["en", "de"], "updatetime": false, "writethumbnail": true}'
```
**Via a JSON file** (`YTDL_OPTIONS_FILE`) — mount a file into the container and point to it:
```yaml
volumes:
- /path/to/ytdl-options.json:/config/ytdl-options.json
environment:
- YTDL_OPTIONS_FILE=/config/ytdl-options.json
```
where `ytdl-options.json` contains:
```json
{
"writesubtitles": true,
"subtitleslangs": ["en", "de"],
"updatetime": false,
"writethumbnail": true
}
```
The file is monitored for changes and reloaded automatically — no container restart needed. If you use both methods and they define the same key, the **file takes precedence**.
### Presets
Presets let you define named bundles of options that appear in the web UI under **Advanced Options** as "Option Presets". Users can select one or more presets per download, making it easy to apply common option combinations without editing global settings.
Like global options, presets can be set inline or via a file:
* `YTDL_OPTIONS_PRESETS` — a JSON object where each key is a preset name and its value is a set of yt-dlp options.
* `YTDL_OPTIONS_PRESETS_FILE` — path to a JSON file containing presets, monitored and reloaded on changes.
If both are used and they define a preset with the same name, the **file's version takes precedence**.
**Example** — a presets file defining three presets:
```json
{
"sponsorblock": {
"postprocessors": [
{ "key": "SponsorBlock", "categories": ["sponsor", "selfpromo", "interaction"] },
{ "key": "ModifyChapters", "remove_sponsor_segments": ["sponsor", "selfpromo", "interaction"] }
]
},
"embed-subs": {
"writesubtitles": true,
"writeautomaticsub": true,
"subtitleslangs": ["en", "de"],
"postprocessors": [{ "key": "FFmpegEmbedSubtitle" }]
},
"limit-rate": {
"ratelimit": 5000000
}
}
```
This makes three presets available in the UI:
* **sponsorblock** — strips sponsor, self-promo, and interaction segments from videos.
* **embed-subs** — downloads English and German subtitles and embeds them into the video file.
* **limit-rate** — caps download speed to ~5 MB/s.
When multiple presets are selected for a download, they are applied in order. If two presets set the same option, the later one wins.
### Per-download overrides
For one-off tweaks, MeTube can expose a free-text JSON field in the UI ("Custom yt-dlp Options") where users type yt-dlp options that apply only to that single download. This is disabled by default:
```yaml
environment:
- ALLOW_YTDL_OPTIONS_OVERRIDES=true
```
Once enabled, the field appears under **Advanced Options**. Any options entered there take the highest priority, overriding both global options and selected presets.
> **⚠️ Security note:** Enabling this allows arbitrary yt-dlp API options to be supplied by anyone with access to the UI. Depending on the options used, this may enable arbitrary command execution inside the container. Enable only in trusted environments.
### How the layers combine
When a download starts, the final set of yt-dlp options is built in this order:
1. Start with **global options** (`YTDL_OPTIONS` / `YTDL_OPTIONS_FILE`).
2. Apply each selected **preset** in order (later presets overwrite earlier ones for conflicting keys).
3. Apply any **per-download overrides** on top (overwrite everything else for conflicting keys).
MeTube always forces its own flat-extract behaviour during the initial metadata fetch (`extract_flat`, `noplaylist`, etc.); presets cannot override those keys for that phase.
**Example:** Suppose your global options set `"writesubtitles": false`, but you select a preset that sets `"writesubtitles": true`. Subtitles will be written for that download because the preset overrides the global setting. If you additionally enter `{"writesubtitles": false}` in the per-download overrides field, that value wins and subtitles will not be written.
### Configuration cookbooks
The project's Wiki contains examples of useful configurations contributed by users of MeTube:
* [YTDL_OPTIONS Cookbook](https://github.com/alexta69/metube/wiki/YTDL_OPTIONS-Cookbook)
@ -97,62 +229,27 @@ In case you need to use your browser's cookies with MeTube, for example to downl
* After upload, the cookie indicator should show as active.
* Use **Delete Cookies** in the same section to remove uploaded cookies.
## 🔌 Browser extensions
## 🔗 Sending links to MeTube
Browser extensions allow right-clicking videos and sending them directly to MeTube. Please note that if you're on an HTTPS page, your MeTube instance must be behind an HTTPS reverse proxy (see below) for the extensions to work.
Several integrations let you send URLs to MeTube from wherever you are, instead of pasting them into the UI. The browser-based ones make cross-origin requests, so they require `CORS_ALLOWED_ORIGINS` to be set; and if you're on an HTTPS page, your MeTube instance must be served over HTTPS too (with `HTTPS=true` or behind an HTTPS reverse proxy — see below).
__Chrome:__ contributed by [Rpsl](https://github.com/rpsl). You can install it from [Google Chrome Webstore](https://chrome.google.com/webstore/detail/metube-downloader/fbmkmdnlhacefjljljlbhkodfmfkijdh) or use developer mode and install [from sources](https://github.com/Rpsl/metube-browser-extension).
__Browser extensions__ allow right-clicking videos and sending them directly to MeTube. Since extensions request from their own origin, set `CORS_ALLOWED_ORIGINS=*`.
* __Chrome:__ contributed by [Rpsl](https://github.com/rpsl) — install from the [Chrome Webstore](https://chrome.google.com/webstore/detail/metube-downloader/fbmkmdnlhacefjljljlbhkodfmfkijdh) or [from sources](https://github.com/Rpsl/metube-browser-extension).
* __Firefox:__ contributed by [nanocortex](https://github.com/nanocortex) — install from [Firefox Addons](https://addons.mozilla.org/en-US/firefox/addon/metube-downloader) or get sources [here](https://github.com/nanocortex/metube-firefox-addon).
__Firefox:__ contributed by [nanocortex](https://github.com/nanocortex). You can install it from [Firefox Addons](https://addons.mozilla.org/en-US/firefox/addon/metube-downloader) or get sources from [here](https://github.com/nanocortex/metube-firefox-addon).
__Bookmarklets__ send the currently open page to MeTube with one click. Add the origins of the sites where you use them to `CORS_ALLOWED_ORIGINS`, e.g. `https://www.youtube.com,https://www.vimeo.com`. The code (Chrome and Firefox variants, contributed by [kushfest](https://github.com/kushfest) and [shoonya75](https://github.com/shoonya75)) is in the [Bookmarklets wiki page](https://github.com/alexta69/metube/wiki/Bookmarklets).
## 📱 iOS Shortcut
__iOS Shortcut:__ [rithask](https://github.com/rithask) created an [iOS shortcut](https://www.icloud.com/shortcuts/66627a9f334c467baabdb2769763a1a6) for sending URLs to MeTube from Safari's share menu; it prompts for your instance address on first use.
[rithask](https://github.com/rithask) created an iOS shortcut to send URLs to MeTube from Safari. Enter the MeTube instance address when prompted which will be saved for later use. You can run the shortcut from Safaris share menu. The shortcut can be downloaded from [this iCloud link](https://www.icloud.com/shortcuts/66627a9f334c467baabdb2769763a1a6).
__Raycast:__ [dotvhs](https://github.com/dotvhs) has created an [extension for Raycast](https://www.raycast.com/dot/metube) for adding videos to MeTube directly from Raycast.
## 📱 iOS Compatibility
## 🎵 Pairing with a music tagger
iOS has strict requirements for video files, requiring h264 or h265 video codec and aac audio codec in MP4 container. This can sometimes be a lower quality than the best quality available. To accommodate iOS requirements, when downloading a MP4 format you can choose "Best (iOS)" to get the best quality formats as compatible as possible with iOS requirements.
MeTube deliberately stops once the file is written — tagging and library organization belong to dedicated tools. Point one at your audio download folder (`AUDIO_DOWNLOAD_DIR`):
To force all downloads to be converted to an iOS-compatible codec, insert this as an environment variable:
```yaml
environment:
- 'YTDL_OPTIONS={"format": "best", "exec": "ffmpeg -i %(filepath)q -c:v libx264 -c:a aac %(filepath)q.h264.mp4"}'
```
## 🔖 Bookmarklet
[kushfest](https://github.com/kushfest) has created a Chrome bookmarklet for sending the currently open webpage to MeTube. Please note that if you're on an HTTPS page, your MeTube instance must be configured with `HTTPS` as `true` in the environment, or be behind an HTTPS reverse proxy (see below) for the bookmarklet to work.
GitHub doesn't allow embedding JavaScript as a link, so the bookmarklet has to be created manually by copying the following code to a new bookmark you create on your bookmarks bar. Change the hostname in the URL below to point to your MeTube instance.
```javascript
javascript:!function(){xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.withCredentials=true;xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function(){if(xhr.status==200){alert("Sent to metube!")}else{alert("Send to metube failed. Check the javascript console for clues.")}}}();
```
[shoonya75](https://github.com/shoonya75) has contributed a Firefox version:
```javascript
javascript:(function(){xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function(){if(xhr.status==200){alert("Sent to metube!")}else{alert("Send to metube failed. Check the javascript console for clues.")}}})();
```
The above bookmarklets use `alert()` as a success/failure notification. The following will show a toast message instead:
Chrome:
```javascript
javascript:!function(){function notify(msg) {var sc = document.scrollingElement.scrollTop; var text = document.createElement('span');text.innerHTML=msg;var ts = text.style;ts.all = 'revert';ts.color = '#000';ts.fontFamily = 'Verdana, sans-serif';ts.fontSize = '15px';ts.backgroundColor = 'white';ts.padding = '15px';ts.border = '1px solid gainsboro';ts.boxShadow = '3px 3px 10px';ts.zIndex = '100';document.body.appendChild(text);ts.position = 'absolute'; ts.top = 50 + sc + 'px'; ts.left = (window.innerWidth / 2)-(text.offsetWidth / 2) + 'px'; setTimeout(function () { text.style.visibility = "hidden"; }, 1500);}xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function() { if(xhr.status==200){notify("Sent to metube!")}else {notify("Send to metube failed. Check the javascript console for clues.")}}}();
```
Firefox:
```javascript
javascript:(function(){function notify(msg) {var sc = document.scrollingElement.scrollTop; var text = document.createElement('span');text.innerHTML=msg;var ts = text.style;ts.all = 'revert';ts.color = '#000';ts.fontFamily = 'Verdana, sans-serif';ts.fontSize = '15px';ts.backgroundColor = 'white';ts.padding = '15px';ts.border = '1px solid gainsboro';ts.boxShadow = '3px 3px 10px';ts.zIndex = '100';document.body.appendChild(text);ts.position = 'absolute'; ts.top = 50 + sc + 'px'; ts.left = (window.innerWidth / 2)-(text.offsetWidth / 2) + 'px'; setTimeout(function () { text.style.visibility = "hidden"; }, 1500);}xhr=new XMLHttpRequest();xhr.open("POST","https://metube.domain.com/add");xhr.send(JSON.stringify({"url":document.location.href,"quality":"best"}));xhr.onload=function() { if(xhr.status==200){notify("Sent to metube!")}else {notify("Send to metube failed. Check the javascript console for clues.")}}})();
```
## ⚡ Raycast extension
[dotvhs](https://github.com/dotvhs) has created an [extension for Raycast](https://www.raycast.com/dot/metube) that allows adding videos to MeTube directly from Raycast.
* [beets](https://beets.io) — `beet import` matches tracks against MusicBrainz, fixes tags, and files them into an Artist/Album library; headless and scriptable.
* [MusicBrainz Picard](https://picard.musicbrainz.org) — GUI tagger with acoustic fingerprinting.
* [Lidarr](https://lidarr.audio) — full music library manager; add the folder as an import path.
## 🔒 HTTPS support, and running behind a reverse proxy
@ -176,13 +273,7 @@ services:
- KEYFILE=/ssl/key.pem
```
It's also possible to run MeTube behind a reverse proxy, in order to support authentication. HTTPS support can also be added in this way.
When running behind a reverse proxy which remaps the URL (i.e. serves MeTube under a subdirectory and not under root), don't forget to set the URL_PREFIX environment variable to the correct value.
If you're using the [linuxserver/swag](https://docs.linuxserver.io/general/swag) image for your reverse proxying needs (which I can heartily recommend), it already includes ready snippets for proxying MeTube both in [subfolder](https://github.com/linuxserver/reverse-proxy-confs/blob/master/metube.subfolder.conf.sample) and [subdomain](https://github.com/linuxserver/reverse-proxy-confs/blob/master/metube.subdomain.conf.sample) modes under the `nginx/proxy-confs` directory in the configuration volume. It also includes Authelia which can be used for authentication.
### 🌐 NGINX
MeTube can also run behind a reverse proxy for HTTPS termination or authentication. When serving under a subdirectory, set `URL_PREFIX` accordingly. MeTube uses WebSocket for real-time updates, so the proxy must pass the `Upgrade`/`Connection` headers, as in this NGINX example:
```nginx
location /metube/ {
@ -194,66 +285,26 @@ location /metube/ {
}
```
Note: the extra `proxy_set_header` directives are there to make WebSocket work.
### 🌐 Apache
Contributed by [PIE-yt](https://github.com/PIE-yt). Source [here](https://gist.github.com/PIE-yt/29e7116588379032427f5bd446b2cac4).
```apache
# For putting in your Apache sites site.conf
# Serves MeTube under a /metube/ subdir (http://yourdomain.com/metube/)
<Location /metube/>
ProxyPass http://localhost:8081/ retry=0 timeout=30
ProxyPassReverse http://localhost:8081/
</Location>
<Location /metube/socket.io>
RewriteEngine On
RewriteCond %{QUERY_STRING} transport=websocket [NC]
RewriteRule /(.*) ws://localhost:8081/socket.io/$1 [P,L]
ProxyPass http://localhost:8081/socket.io retry=0 timeout=30
ProxyPassReverse http://localhost:8081/socket.io
</Location>
```
### 🌐 Caddy
The following example Caddyfile gets a reverse proxy going behind [caddy](https://caddyserver.com).
```caddyfile
example.com {
route /metube/* {
uri strip_prefix metube
reverse_proxy metube:8081
}
}
```
Apache, Caddy, and [linuxserver/swag](https://docs.linuxserver.io/general/swag) (with Authelia) examples are in the [Reverse proxy configurations wiki page](https://github.com/alexta69/metube/wiki/Reverse-proxy-configurations).
## 🔄 Updating yt-dlp
The engine which powers the actual video downloads in MeTube is [yt-dlp](https://github.com/yt-dlp/yt-dlp). Since video sites regularly change their layouts, frequent updates of yt-dlp are required to keep up.
There's an automatic nightly build of MeTube which looks for a new version of yt-dlp, and if one exists, the build pulls it and publishes an updated docker image. Therefore, in order to keep up with the changes, it's recommended that you update your MeTube container regularly with the latest image.
I recommend installing and setting up [watchtower](https://github.com/nicholas-fedor/watchtower) for this purpose.
MeTube is powered by [yt-dlp](https://github.com/yt-dlp/yt-dlp), which requires frequent updates as video sites change their layouts. A new MeTube Docker image is published automatically when a new yt-dlp stable release is available, so keep your container up to date — [watchtower](https://github.com/nicholas-fedor/watchtower) works well for this. To follow yt-dlp's nightly channel instead, set `YTDL_NIGHTLY_UPDATE_TIME`.
## 🔧 Troubleshooting and submitting issues
Before asking a question or submitting an issue for MeTube, please remember that MeTube is only a UI for [yt-dlp](https://github.com/yt-dlp/yt-dlp). Any issues you might be experiencing with authentication to video websites, postprocessing, permissions, other `YTDL_OPTIONS` configurations which seem not to work, or anything else that concerns the workings of the underlying yt-dlp library, need not be opened on the MeTube project. In order to debug and troubleshoot them, it's advised to try using the yt-dlp binary directly first, bypassing the UI, and once that is working, importing the options that worked for you into `YTDL_OPTIONS`.
In order to test with the yt-dlp command directly, you can either download it and run it locally, or for a better simulation of its actual conditions, you can run it within the MeTube container itself. Assuming your MeTube container is called `metube`, run the following on your Docker host to get a shell inside the container:
MeTube is only a UI for [yt-dlp](https://github.com/yt-dlp/yt-dlp). Issues with authentication, postprocessing, permissions, or `YTDL_OPTIONS` should be debugged with yt-dlp directly first — once working, import those options into MeTube. To test inside the container:
```bash
docker exec -ti metube sh
cd /downloads
```
Once there, you can use the yt-dlp command freely.
Common issues and their fixes are collected in the [Troubleshooting FAQ](https://github.com/alexta69/metube/wiki/Troubleshooting-FAQ) on the wiki.
## 💡 Submitting feature requests
MeTube development relies on code contributions by the community. The program as it currently stands fits my own use cases, and is therefore feature-complete as far as I'm concerned. If your use cases are different and require additional features, please feel free to submit PRs that implement those features. It's advisable to create an issue first to discuss the planned implementation, because in an effort to reduce bloat, some PRs may not be accepted. However, note that opening a feature request when you don't intend to implement the feature will rarely result in the request being fulfilled.
MeTube development relies on community contributions. If you need additional features, please submit a PR. Create an issue first to discuss the implementation before writing code — MeTube's scope is deliberately narrow: it downloads well and stops once the file is written. Features that improve the download itself are welcome; post-download file management (tag editing, metadata lookups, library organization) is out of scope regardless of implementation quality — see [AGENTS.md](AGENTS.md) for the full policy. Feature requests without an accompanying PR are unlikely to be fulfilled.
## 🛠️ Building and running locally

26
app/bg_tasks.py Normal file
View file

@ -0,0 +1,26 @@
import asyncio
import logging
log = logging.getLogger("bg_tasks")
_TASKS: set[asyncio.Task] = set()
def create_task(coro, *, name: str | None = None) -> asyncio.Task:
"""create_task that keeps a strong reference and logs unexpected failures.
A bare ``asyncio.create_task(...)`` is only weakly referenced by the event
loop; if nothing else holds the returned Task, it can be garbage collected
mid-flight. Keeping a module-level strong reference (removed once the task
finishes) avoids that, and the done-callback surfaces otherwise-silent
failures.
"""
task = asyncio.get_running_loop().create_task(coro, name=name)
_TASKS.add(task)
def _done(t: asyncio.Task) -> None:
_TASKS.discard(t)
if not t.cancelled() and t.exception() is not None:
log.error("Background task %s failed", t.get_name(), exc_info=t.exception())
task.add_done_callback(_done)
return task

View file

@ -4,6 +4,28 @@ AUDIO_FORMATS = ("m4a", "mp3", "opus", "wav", "flac")
CAPTION_MODES = ("auto_only", "manual_only", "prefer_manual", "prefer_auto")
def merge_ytdl_option_layers(presets, overrides, presets_config) -> dict:
"""Overlay named presets (in order) then per-item overrides onto a fresh dict.
Does NOT include any base ``YTDL_OPTIONS`` callers layer this on top of
their own base (a per-download build adds the global base; a subscription
scan relies on ``**config.YTDL_OPTIONS`` already being present in its
params). ``presets_config`` maps a preset name to its options dict.
"""
merged: dict = {}
for name in presets or []:
merged.update(presets_config.get(name, {}))
merged.update(overrides or {})
return merged
CODEC_FILTER_MAP = {
'h264': "[vcodec~='^(h264|avc)']",
'h265': "[vcodec~='^(h265|hevc)']",
'av1': "[vcodec~='^av0?1']",
'vp9': "[vcodec~='^vp0?9']",
}
def _normalize_caption_mode(mode: str) -> str:
mode = (mode or "").strip()
return mode if mode in CAPTION_MODES else "prefer_manual"
@ -14,84 +36,92 @@ def _normalize_subtitle_language(language: str) -> str:
return language or "en"
def get_format(format: str, quality: str) -> str:
def get_format(download_type: str, codec: str, format: str, quality: str) -> str:
"""
Returns format for download
Returns yt-dlp format selector.
Args:
format (str): format selected
quality (str): quality selected
download_type (str): selected content type (video, audio, captions, thumbnail)
codec (str): selected video codec (auto, h264, h265, av1, vp9)
format (str): selected output format/profile for type
quality (str): selected quality
Raises:
Exception: unknown quality, unknown format
Exception: unknown type/format
Returns:
dl_format: Formatted download string
str: yt-dlp format selector
"""
format = format or "any"
download_type = (download_type or "video").strip().lower()
format = (format or "any").strip().lower()
codec = (codec or "auto").strip().lower()
quality = (quality or "best").strip().lower()
if format.startswith("custom:"):
# Unreachable via the HTTP API (format is validated against a fixed
# set in main.py), but legacy persisted downloads may carry a
# custom: format from before that validation existed; removing this
# would crash PersistentQueue.load() for those records.
return format[7:]
if format == "thumbnail":
# Quality is irrelevant in this case since we skip the download
if download_type == "thumbnail":
return "bestaudio/best"
if format == "captions":
# Quality is irrelevant in this case since we skip the download
if download_type == "captions":
return "bestaudio/best"
if format in AUDIO_FORMATS:
# Audio quality needs to be set post-download, set in opts
if download_type == "audio":
if format not in AUDIO_FORMATS:
raise ValueError(f"Unknown audio format {format}")
return f"bestaudio[ext={format}]/bestaudio/best"
if format in ("mp4", "any"):
if quality == "audio":
return "bestaudio/best"
# video {res} {vfmt} + audio {afmt} {res} {vfmt}
vfmt, afmt = ("[ext=mp4]", "[ext=m4a]") if format == "mp4" else ("", "")
vres = f"[height<={quality}]" if quality not in ("best", "best_ios", "worst") else ""
if download_type == "video":
if format not in ("any", "mp4", "ios"):
raise ValueError(f"Unknown video format {format}")
vfmt, afmt = ("[ext=mp4]", "[ext=m4a]") if format in ("mp4", "ios") else ("", "")
vres = f"[height<={quality}]" if quality not in ("best", "worst") else ""
vcombo = vres + vfmt
codec_filter = CODEC_FILTER_MAP.get(codec, "")
if quality == "best_ios":
# iOS has strict requirements for video files, requiring h264 or h265
# video codec and aac audio codec in MP4 container. This format string
# attempts to get the fully compatible formats first, then the h264/h265
# video codec with any M4A audio codec (because audio is faster to
# convert if needed), and falls back to getting the best available MP4
# file.
if format == "ios":
return f"bestvideo[vcodec~='^((he|a)vc|h26[45])']{vres}+bestaudio[acodec=aac]/bestvideo[vcodec~='^((he|a)vc|h26[45])']{vres}+bestaudio{afmt}/bestvideo{vcombo}+bestaudio{afmt}/best{vcombo}"
if codec_filter:
return f"bestvideo{codec_filter}{vcombo}+bestaudio{afmt}/bestvideo{vcombo}+bestaudio{afmt}/best{vcombo}"
return f"bestvideo{vcombo}+bestaudio{afmt}/best{vcombo}"
raise Exception(f"Unkown format {format}")
raise ValueError(f"Unknown download_type {download_type}")
def get_opts(
download_type: str,
_codec: str,
format: str,
quality: str,
ytdl_opts: dict,
subtitle_format: str = "srt",
subtitle_language: str = "en",
subtitle_mode: str = "prefer_manual",
) -> dict:
"""
Returns extra download options
Mostly postprocessing options
Returns extra yt-dlp options/postprocessors.
Args:
format (str): format selected
quality (str): quality of format selected (needed for some formats)
download_type (str): selected content type
codec (str): selected codec (unused currently, kept for API consistency)
format (str): selected format/profile
quality (str): selected quality
ytdl_opts (dict): current options selected
Returns:
ytdl_opts: Extra options
dict: extended options
"""
download_type = (download_type or "video").strip().lower()
format = (format or "any").strip().lower()
opts = copy.deepcopy(ytdl_opts)
postprocessors = []
if format in AUDIO_FORMATS:
if download_type == "audio":
postprocessors.append(
{
"key": "FFmpegExtractAudio",
@ -100,8 +130,7 @@ def get_opts(
}
)
# Audio formats without thumbnail
if format not in ("wav") and "writethumbnail" not in opts:
if format != "wav" and "writethumbnail" not in opts:
opts["writethumbnail"] = True
postprocessors.append(
{
@ -113,22 +142,34 @@ def get_opts(
postprocessors.append({"key": "FFmpegMetadata"})
postprocessors.append({"key": "EmbedThumbnail"})
if format == "thumbnail":
if download_type == "thumbnail":
opts["skip_download"] = True
opts["writethumbnail"] = True
postprocessors.append(
{"key": "FFmpegThumbnailsConvertor", "format": "jpg", "when": "before_dl"}
)
if format == "captions":
if download_type == "captions":
mode = _normalize_caption_mode(subtitle_mode)
language = _normalize_subtitle_language(subtitle_language)
opts["skip_download"] = True
requested_subtitle_format = (subtitle_format or "srt").lower()
# txt is a derived, non-timed format produced from SRT after download.
requested_subtitle_format = (format or "srt").lower()
if requested_subtitle_format == "txt":
requested_subtitle_format = "srt"
opts["subtitlesformat"] = requested_subtitle_format
opts["subtitlesformat"] = f"{requested_subtitle_format}/best"
if requested_subtitle_format in ("srt", "vtt"):
# subtitlesformat above is only a preference: if the extractor
# doesn't natively offer this ext (e.g. YouTube has no native srt),
# yt-dlp silently falls back to whatever it has. ffmpeg can only
# convert to srt/vtt/ass/lrc, so only guarantee the requested
# container for those; other formats stay best-effort.
postprocessors.append(
{
"key": "FFmpegSubtitlesConvertor",
"format": requested_subtitle_format,
"when": "before_dl",
}
)
if mode == "manual_only":
opts["writesubtitles"] = True
opts["writeautomaticsub"] = False

File diff suppressed because it is too large Load diff

237
app/state_store.py Normal file
View file

@ -0,0 +1,237 @@
from __future__ import annotations
import base64
import collections.abc
import errno
import json
import logging
import os
import shelve
import tempfile
import time
from datetime import datetime
from typing import Any, Optional
log = logging.getLogger("state_store")
STATE_SCHEMA_VERSION = 2
_BYTES_MARKER = "__metube_bytes__"
_DATETIME_MARKER = "__metube_datetime__"
# Errnos that signal the filesystem cannot support the temp-file + rename
# atomic-write strategy (for example an NFS-backed state dir returning EPERM on
# mkstemp). These are safe to fall back on because they mean the atomic
# mechanism is unavailable, not that the data write itself failed. Errors like
# ENOSPC/EIO are deliberately excluded so a genuine storage failure surfaces
# instead of silently truncating an existing good state file.
_ATOMIC_UNSUPPORTED_ERRNOS = frozenset(
e
for e in (
errno.EPERM,
errno.EACCES,
errno.ENOSYS,
errno.EINVAL,
getattr(errno, "EOPNOTSUPP", None),
getattr(errno, "ENOTSUP", None),
)
if e is not None
)
def to_json_compatible(value: Any) -> Any:
if value is None or isinstance(value, (bool, int, float, str)):
return value
if isinstance(value, bytes):
return {_BYTES_MARKER: base64.b64encode(value).decode("ascii")}
if isinstance(value, datetime):
return {_DATETIME_MARKER: value.isoformat()}
if isinstance(value, collections.abc.Mapping):
return {str(k): to_json_compatible(v) for k, v in value.items()}
if isinstance(value, (list, tuple, set, frozenset)):
return [to_json_compatible(v) for v in value]
if isinstance(value, collections.abc.Iterable):
return [to_json_compatible(v) for v in value]
raise TypeError(f"Value of type {type(value).__name__} is not JSON serializable")
def from_json_compatible(value: Any) -> Any:
if isinstance(value, list):
return [from_json_compatible(v) for v in value]
if isinstance(value, dict):
if set(value.keys()) == {_BYTES_MARKER}:
return base64.b64decode(value[_BYTES_MARKER].encode("ascii"))
if set(value.keys()) == {_DATETIME_MARKER}:
return datetime.fromisoformat(value[_DATETIME_MARKER])
return {k: from_json_compatible(v) for k, v in value.items()}
return value
def read_legacy_shelf(path: str) -> Optional[list[tuple[Any, Any]]]:
if not os.path.exists(path):
return None
try:
with shelve.open(path, "r") as shelf:
return list(shelf.items())
except Exception as exc:
log.warning("Could not read legacy shelf at %s: %s", path, exc)
return None
class AtomicJsonStore:
def __init__(self, path: str, *, kind: str, schema_version: int = STATE_SCHEMA_VERSION):
self.path = path
self.kind = kind
self.schema_version = schema_version
self._direct_write_fallback_warned = False
def _ensure_parent(self) -> None:
parent = os.path.dirname(self.path)
if parent and not os.path.isdir(parent):
os.makedirs(parent, exist_ok=True)
def _build_payload(self, data: dict[str, Any]) -> dict[str, Any]:
payload = {
"schema_version": self.schema_version,
"kind": self.kind,
}
payload.update(data)
return payload
def load(self) -> Optional[dict[str, Any]]:
if not os.path.exists(self.path):
return None
try:
with open(self.path, encoding="utf-8") as f:
payload = json.load(f)
if not isinstance(payload, dict):
raise ValueError("State file must contain a JSON object")
if payload.get("kind") != self.kind:
raise ValueError(
f"State file kind mismatch: expected {self.kind}, got {payload.get('kind')}"
)
return payload
except Exception as exc:
self.quarantine_invalid_file(exc)
return None
def save(self, data: dict[str, Any]) -> None:
self._ensure_parent()
payload = self._build_payload(data)
try:
self._atomic_write(payload)
except OSError as exc:
if exc.errno not in _ATOMIC_UNSUPPORTED_ERRNOS:
raise
self._warn_direct_write_fallback(exc)
self._direct_write(payload)
def _atomic_write(self, payload: dict[str, Any]) -> None:
text = self._serialize(payload)
parent = os.path.dirname(self.path) or "."
fd, tmp_path = tempfile.mkstemp(
prefix=f".{os.path.basename(self.path)}.",
suffix=".tmp",
dir=parent,
text=True,
)
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
f.write(text)
f.flush()
self._best_effort_fsync(f.fileno())
os.replace(tmp_path, self.path)
self._fsync_directory(parent)
except Exception:
try:
os.remove(tmp_path)
except OSError:
pass
raise
def _direct_write(self, payload: dict[str, Any]) -> None:
# Serialize before truncating so a serialization failure never destroys
# the existing state file (the atomic path gets this for free via its
# temp file).
text = self._serialize(payload)
# Create with 0o600 so the fallback keeps the owner-only permissions the
# atomic path gets from mkstemp; state files can contain URLs and
# per-download option overrides that must not leak on shared mounts.
fd = os.open(self.path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w", encoding="utf-8") as f:
# The 0o600 mode above only applies when the file is created; force
# it on rewrites too so an existing, broadly-permissioned state file
# is tightened to match the atomic path. Best-effort because some
# network filesystems reject chmod, and that must not re-crash save.
try:
os.fchmod(f.fileno(), 0o600)
except OSError:
pass
f.write(text)
f.flush()
self._best_effort_fsync(f.fileno())
# Make the new directory entry durable too, matching the atomic path.
self._fsync_directory(os.path.dirname(self.path) or ".")
@staticmethod
def _best_effort_fsync(fileno: int) -> None:
# Tolerate fsync being unsupported on the underlying filesystem (for
# example a network mount that returns EINVAL/ENOSYS), but let genuine
# storage failures such as ENOSPC/EIO surface so a non-durable write is
# never reported as success. An unsupported fsync must not by itself
# abandon the atomic rename path.
try:
os.fsync(fileno)
except OSError as exc:
if exc.errno not in _ATOMIC_UNSUPPORTED_ERRNOS:
raise
@staticmethod
def _serialize(payload: dict[str, Any]) -> str:
return json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n"
def _warn_direct_write_fallback(self, exc: OSError) -> None:
if self._direct_write_fallback_warned:
return
self._direct_write_fallback_warned = True
log.warning(
"Atomic state write failed for %s (%s); falling back to direct write",
self.path,
exc,
)
def quarantine_invalid_file(self, exc: Exception) -> None:
if not os.path.exists(self.path):
return
ts = time.strftime("%Y%m%d%H%M%S")
backup_path = f"{self.path}.invalid.{ts}"
try:
os.replace(self.path, backup_path)
log.warning(
"State file at %s was invalid (%s); moved it to %s",
self.path,
exc,
backup_path,
)
except OSError as move_exc:
log.warning(
"State file at %s was invalid (%s) and could not be moved aside: %s",
self.path,
exc,
move_exc,
)
@staticmethod
def _fsync_directory(path: str) -> None:
try:
flags = os.O_RDONLY
if hasattr(os, "O_DIRECTORY"):
flags |= os.O_DIRECTORY
fd = os.open(path, flags)
except OSError:
return
try:
os.fsync(fd)
except OSError:
pass
finally:
os.close(fd)

964
app/subscriptions.py Normal file
View file

@ -0,0 +1,964 @@
"""Channel/playlist subscriptions: periodic yt-dlp flat extract + queue new videos."""
from __future__ import annotations
import asyncio
import copy
import logging
import os
import re
import time
import types
import uuid
from dataclasses import dataclass, field, fields
from functools import partial
from typing import Any, Optional
import yt_dlp
import yt_dlp.networking.impersonate
import bg_tasks
from dl_formats import merge_ytdl_option_layers
from state_store import AtomicJsonStore, read_legacy_shelf
from url_guard import validate_url
log = logging.getLogger("subscriptions")
# How many subscription feeds to scan at once. Bounded so one slow/hung feed
# doesn't serialize the rest, without bursting a large subscription list at the
# extractor (which risks rate-limiting / bot detection).
_MAX_CONCURRENT_CHECKS = 4
VIDEO_ONLY_MSG = (
"This URL points to a single video, not a channel or playlist. Use Download instead."
)
_MEDIA_HINT_FIELDS = (
"duration",
"timestamp",
"release_timestamp",
"upload_date",
"view_count",
"live_status",
"availability",
)
def _impersonate_opt(ytdl_options: dict) -> dict:
opts = dict(ytdl_options)
if "impersonate" in opts:
opts["impersonate"] = yt_dlp.networking.impersonate.ImpersonateTarget.from_str(
opts["impersonate"]
)
return opts
def _build_ydl_params(
config, *, playlistend: Optional[int] = None, extra_opts: Optional[dict[str, Any]] = None
) -> dict:
params: dict[str, Any] = {
"quiet": not logging.getLogger().isEnabledFor(logging.DEBUG),
"verbose": logging.getLogger().isEnabledFor(logging.DEBUG),
"no_color": True,
"extract_flat": True,
"ignore_no_formats_error": True,
"lazy_playlist": True,
"paths": {"home": config.DOWNLOAD_DIR, "temp": config.TEMP_DIR},
**config.YTDL_OPTIONS,
**(extra_opts or {}),
}
params = _impersonate_opt(params)
if playlistend is not None and playlistend > 0:
params["playlistend"] = playlistend
return params
def _is_media_entry(entry: Any) -> bool:
if not isinstance(entry, dict):
return False
etype = str(entry.get("_type") or "")
if etype in ("playlist", "multi_video", "channel"):
return False
if entry.get("entries"):
return False
url = _entry_video_url(entry)
if not url:
return False
ie_key = str(entry.get("ie_key") or entry.get("extractor_key") or "").lower()
if any(token in ie_key for token in ("playlist", "channel", "tab")):
return any(entry.get(field) is not None for field in _MEDIA_HINT_FIELDS)
return True
def extract_flat_playlist(
config, url: str, playlistend: int, *, extra_opts: Optional[dict[str, Any]] = None, _depth: int = 0
):
"""Return (info_dict, entries_list) for playlist/channel URLs."""
params = _build_ydl_params(config, playlistend=playlistend, extra_opts=extra_opts)
with yt_dlp.YoutubeDL(params=params) as ydl:
info = ydl.extract_info(url, download=False)
if not info:
return None, []
etype = info.get("_type") or "video"
if etype == "video":
return info, []
if etype in ("playlist", "channel"):
entries = info.get("entries") or []
if isinstance(entries, types.GeneratorType):
entries = list(entries)
# Drop None placeholders from incomplete flat playlists
entries = [e for e in entries if e]
media_entries = [e for e in entries if _is_media_entry(e)]
if media_entries:
return info, media_entries
if _depth < 1:
for ent in entries[:5]:
nested_url = _entry_video_url(ent)
if not nested_url:
continue
# nested_url comes from remote playlist content; guard it too.
if validate_url(nested_url) is not None:
continue
nested_info, nested_entries = extract_flat_playlist(
config,
nested_url,
playlistend,
extra_opts=extra_opts,
_depth=_depth + 1,
)
if nested_entries:
return nested_info, nested_entries
return info, entries
if etype.startswith("url") and info.get("url"):
# Single nested URL without playlist wrapper — treat as non-subscribable
return info, []
return info, []
def _entry_video_url(entry: dict) -> Optional[str]:
return entry.get("webpage_url") or entry.get("url")
def _entry_id(entry: dict) -> Optional[str]:
eid = entry.get("id")
if eid is not None:
return str(eid)
url = _entry_video_url(entry)
return url
def _is_subscriber_only_entry(entry: dict) -> bool:
"""True when yt-dlp marks the entry as channel member-only (subscriber_only availability)."""
return str(entry.get("availability") or "") == "subscriber_only"
def coerce_optional_bool(value: Any, *, default: bool = False, field_name: str = "value") -> bool:
"""Parse optional JSON booleans for subscription settings."""
if value is None:
return default
try:
return _coerce_bool(value)
except ValueError as exc:
raise ValueError(f"{field_name} must be a boolean") from exc
@dataclass
class SubscriptionInfo:
id: str
name: str
url: str
enabled: bool = True
check_interval_minutes: int = 60
download_type: str = "video"
codec: str = "auto"
format: str = "any"
quality: str = "best"
folder: str = ""
custom_name_prefix: str = ""
auto_start: bool = True
playlist_item_limit: int = 0
split_by_chapters: bool = False
chapter_template: str = ""
subtitle_language: str = "en"
subtitle_mode: str = "prefer_manual"
ytdl_options_presets: list[str] = field(default_factory=list)
ytdl_options_overrides: dict[str, Any] = field(default_factory=dict)
title_regex: str = ""
skip_subscriber_only: bool = False
last_checked: Optional[float] = None
seen_ids: list[str] = field(default_factory=list)
error: Optional[str] = None
timestamp: float = field(default_factory=time.time)
def seen_set(self) -> set[str]:
return set(self.seen_ids)
def to_public_dict(self) -> dict:
return {
"id": self.id,
"name": self.name,
"url": self.url,
"enabled": self.enabled,
"check_interval_minutes": self.check_interval_minutes,
"download_type": self.download_type,
"codec": self.codec,
"format": self.format,
"quality": self.quality,
"folder": self.folder,
"title_regex": self.title_regex,
"skip_subscriber_only": self.skip_subscriber_only,
"last_checked": self.last_checked,
"seen_count": len(self.seen_ids),
"error": self.error,
}
def _subscription_to_record(sub: SubscriptionInfo) -> dict[str, Any]:
return {
"id": sub.id,
"name": sub.name,
"url": sub.url,
"enabled": sub.enabled,
"check_interval_minutes": sub.check_interval_minutes,
"download_type": sub.download_type,
"codec": sub.codec,
"format": sub.format,
"quality": sub.quality,
"folder": sub.folder,
"custom_name_prefix": sub.custom_name_prefix,
"auto_start": sub.auto_start,
"playlist_item_limit": sub.playlist_item_limit,
"split_by_chapters": sub.split_by_chapters,
"chapter_template": sub.chapter_template,
"subtitle_language": sub.subtitle_language,
"subtitle_mode": sub.subtitle_mode,
"ytdl_options_presets": list(sub.ytdl_options_presets),
"ytdl_options_overrides": sub.ytdl_options_overrides,
"title_regex": sub.title_regex,
"skip_subscriber_only": sub.skip_subscriber_only,
"last_checked": sub.last_checked,
"seen_ids": list(sub.seen_ids),
"error": sub.error,
}
def _normalize_subscription_record(rec: dict[str, Any]) -> dict[str, Any]:
"""Migrate legacy ytdl_options_preset (str) to ytdl_options_presets (list)."""
out = dict(rec)
if "ytdl_options_presets" not in out:
old = out.pop("ytdl_options_preset", None)
if old is None:
out["ytdl_options_presets"] = []
elif isinstance(old, list):
out["ytdl_options_presets"] = [str(x).strip() for x in old if str(x).strip()]
elif isinstance(old, str):
out["ytdl_options_presets"] = [old.strip()] if old.strip() else []
else:
out["ytdl_options_presets"] = []
else:
out.pop("ytdl_options_preset", None)
return out
def _subscription_from_record(record: Any) -> Optional[SubscriptionInfo]:
field_names = {f.name for f in fields(SubscriptionInfo)}
if isinstance(record, SubscriptionInfo):
return record
if isinstance(record, dict):
try:
normalized = _normalize_subscription_record(dict(record))
return SubscriptionInfo(**{k: v for k, v in normalized.items() if k in field_names})
except TypeError:
return None
return None
def _normalize_title_regex_value(value: Any) -> str:
if value is None:
return ""
if isinstance(value, str):
return value.strip()
return str(value).strip()
def validate_title_regex(value: Any) -> str:
"""Return stored title regex string; non-empty values must compile (re.error on failure)."""
s = _normalize_title_regex_value(value)
if s:
re.compile(s)
return s
def _coerce_bool(value: Any) -> bool:
"""Accept JSON booleans and common string forms used by API clients."""
if isinstance(value, bool):
return value
if isinstance(value, str):
lowered = value.strip().lower()
if lowered in {"true", "1", "on"}:
return True
if lowered in {"false", "0", "off"}:
return False
raise ValueError("enabled must be a boolean")
class SubscriptionNotifier:
"""Hook for Socket.IO / UI updates."""
async def subscription_added(self, sub: SubscriptionInfo) -> None:
raise NotImplementedError
async def subscription_updated(self, sub: SubscriptionInfo) -> None:
raise NotImplementedError
async def subscription_removed(self, sub_id: str) -> None:
raise NotImplementedError
async def subscriptions_all(self, subs: list[SubscriptionInfo]) -> None:
raise NotImplementedError
class SubscriptionManager:
def __init__(self, config, download_queue, notifier: SubscriptionNotifier):
self.config = config
self.dqueue = download_queue
self.notifier = notifier
pdir = config.STATE_DIR
if not os.path.isdir(pdir):
os.makedirs(pdir, exist_ok=True)
self._legacy_path = os.path.join(pdir, "subscriptions")
self._path = os.path.join(pdir, "subscriptions.json")
self._store = AtomicJsonStore(self._path, kind="subscriptions")
self._subs: dict[str, SubscriptionInfo] = {}
self._url_index: dict[str, str] = {} # normalized url -> id
self._pending_urls: set[str] = set()
self._checks_in_flight: set[str] = set() # subscription ids being checked right now
self._lock = asyncio.Lock()
self._loop_task: Optional[asyncio.Task] = None
self._load_all()
def close(self) -> None:
# No persistent shelf handle to close.
return
def _normalize_url(self, url: str) -> str:
return (url or "").strip()
def _normalize_seen_ids(self, seen_ids: list[str]) -> list[str]:
max_seen = int(getattr(self.config, "SUBSCRIPTION_MAX_SEEN_IDS", 50000))
normalized = [str(sid) for sid in dict.fromkeys(seen_ids)]
if len(normalized) > max_seen:
normalized = normalized[:max_seen]
return normalized
def _load_all(self) -> None:
payload = self._store.load()
loaded_from_legacy = False
if payload is not None:
records = payload.get("items") or []
else:
legacy_items = read_legacy_shelf(self._legacy_path)
records = [raw for _key, raw in legacy_items] if legacy_items else []
if records:
loaded_from_legacy = True
loaded_subs = self._iter_valid_subs(records)
compact_records = []
for sub in loaded_subs:
sub.seen_ids = self._normalize_seen_ids(sub.seen_ids)
self._subs[sub.id] = sub
self._url_index[self._normalize_url(sub.url)] = sub.id
compact_records.append(_subscription_to_record(sub))
if loaded_from_legacy or (
payload is not None
and (
payload.get("schema_version") != self._store.schema_version
or compact_records != records
)
):
self._store.save({"items": compact_records})
def _iter_valid_subs(self, records: list[Any]) -> list[SubscriptionInfo]:
subs: list[SubscriptionInfo] = []
for record in records:
sub = _subscription_from_record(record)
if sub is not None:
subs.append(sub)
return subs
def _save_locked(self) -> None:
self._store.save({"items": [_subscription_to_record(sub) for sub in self._subs.values()]})
def _scan_extra_opts(
self,
ytdl_options_presets: Optional[list[str]],
ytdl_options_overrides: Optional[dict[str, Any]],
) -> dict[str, Any]:
"""Merge configured presets (in order) with per-subscription overrides.
Applied on top of the global YTDL_OPTIONS when scanning a
subscription's feed, so cookies/impersonation/etc. configured via a
preset or override also take effect during the flat-playlist scan,
not just the eventual per-video download. (The global YTDL_OPTIONS base
is already spread into the scan params by ``_build_ydl_params``.)
"""
return merge_ytdl_option_layers(
ytdl_options_presets, ytdl_options_overrides, self.config.YTDL_OPTIONS_PRESETS
)
async def _queue_subscription_entries(
self,
entries: list[dict],
*,
download_type: str,
codec: str,
format: str,
quality: str,
folder: str,
custom_name_prefix: str,
playlist_item_limit: int,
auto_start: bool,
split_by_chapters: bool,
chapter_template: str,
subtitle_language: str,
subtitle_mode: str,
ytdl_options_presets: Optional[list[str]] = None,
ytdl_options_overrides: Optional[dict[str, Any]] = None,
) -> tuple[list[str], list[str]]:
queued_ids: list[str] = []
queue_errors: list[str] = []
presets = list(ytdl_options_presets or [])
for ent in entries:
eid = _entry_id(ent)
vurl = _entry_video_url(ent)
if not eid or not vurl:
continue
queue_entry = dict(ent)
if "id" not in queue_entry:
queue_entry["id"] = eid
queue_entry["_type"] = "video"
queue_entry["webpage_url"] = vurl
result = await self.dqueue.add_entry(
queue_entry,
download_type,
codec,
format,
quality,
folder or None,
custom_name_prefix,
playlist_item_limit,
auto_start,
split_by_chapters,
chapter_template or None,
subtitle_language,
subtitle_mode,
presets,
ytdl_options_overrides,
)
if isinstance(result, dict) and result.get("status") == "error":
msg = str(result.get("msg") or f"Queueing failed for {vurl}")
queue_errors.append(msg)
log.warning("Subscription queueing failed for %s: %s", vurl, msg)
continue
queued_ids.append(eid)
return queued_ids, queue_errors
def list_all(self) -> list[SubscriptionInfo]:
return list(self._subs.values())
def get(self, sub_id: str) -> Optional[SubscriptionInfo]:
return self._subs.get(sub_id)
def start_background_loop(self) -> None:
if self._loop_task is not None and not self._loop_task.done():
return
# bg_tasks.create_task already logs unexpected task failures with the name.
self._loop_task = bg_tasks.create_task(self._periodic_loop(), name="subscription_loop")
async def _periodic_loop(self) -> None:
while True:
await asyncio.sleep(60)
try:
await self.run_due_checks()
except Exception as e:
log.exception("Subscription periodic check error: %s", e)
async def run_due_checks(self) -> None:
now = time.time()
due: list[SubscriptionInfo] = []
async with self._lock:
for sub in list(self._subs.values()):
if not sub.enabled:
continue
interval_sec = max(60, int(sub.check_interval_minutes) * 60)
if sub.last_checked is None:
due.append(sub)
continue
if now - sub.last_checked < interval_sec:
continue
due.append(sub)
await self._check_many(due)
async def _check_many(self, subs: list[SubscriptionInfo]) -> None:
"""Check subscriptions with bounded concurrency so one slow feed does
not serialize the rest. Failures are isolated per subscription."""
if not subs:
return
sem = asyncio.Semaphore(_MAX_CONCURRENT_CHECKS)
async def _guarded(sub: SubscriptionInfo) -> None:
async with sem:
await self._check_one_unlocked(sub)
results = await asyncio.gather(
*(_guarded(sub) for sub in subs), return_exceptions=True
)
for sub, result in zip(subs, results):
if isinstance(result, Exception):
log.error(
"Subscription check crashed for %s: %s",
sub.name,
result,
exc_info=result,
)
async def add_subscription(
self,
url: str,
*,
check_interval_minutes: int,
download_type: str,
codec: str,
format: str,
quality: str,
folder: str,
custom_name_prefix: str,
auto_start: bool,
playlist_item_limit: int,
split_by_chapters: bool,
chapter_template: str,
subtitle_language: str,
subtitle_mode: str,
ytdl_options_presets: Optional[list[str]] = None,
ytdl_options_overrides: Optional[dict[str, Any]] = None,
title_regex: Any = None,
skip_subscriber_only: Any = None,
) -> dict:
url = self._normalize_url(url)
if not url:
return {"status": "error", "msg": "Missing URL"}
# SSRF guard: block non-http(s) schemes and internal/metadata hosts
# before yt-dlp fetches the feed. May do a DNS lookup, so run off-loop.
url_error = await asyncio.get_running_loop().run_in_executor(None, validate_url, url)
if url_error is not None:
log.warning('Rejected subscription URL "%s": %s', url, url_error)
return {"status": "error", "msg": url_error}
try:
title_regex_stored = validate_title_regex(title_regex)
except re.error as exc:
return {"status": "error", "msg": f"Invalid title_regex: {exc}"}
try:
skip_so = coerce_optional_bool(
skip_subscriber_only,
default=False,
field_name="skip_subscriber_only",
)
except ValueError as exc:
return {"status": "error", "msg": str(exc)}
async with self._lock:
if url in self._url_index or url in self._pending_urls:
return {"status": "error", "msg": "This URL is already subscribed"}
self._pending_urls.add(url)
try:
scan_first = max(int(getattr(self.config, "SUBSCRIPTION_SCAN_PLAYLIST_END", 50)), 1)
scan_extra_opts = self._scan_extra_opts(ytdl_options_presets, ytdl_options_overrides)
try:
info, entries = await asyncio.get_running_loop().run_in_executor(
None,
partial(extract_flat_playlist, self.config, url, scan_first, extra_opts=scan_extra_opts),
)
except yt_dlp.utils.YoutubeDLError as exc:
return {"status": "error", "msg": str(exc)}
if not info:
return {"status": "error", "msg": "Could not resolve URL"}
etype = info.get("_type") or "video"
if etype not in ("playlist", "channel"):
return {"status": "error", "msg": VIDEO_ONLY_MSG}
name = (
info.get("title")
or info.get("channel")
or info.get("playlist_title")
or info.get("uploader")
or url
)
seen_entries = [ent for ent in entries if _is_media_entry(ent)]
all_ids: list[str] = []
for ent in seen_entries:
if ent.get("live_status") == "is_upcoming":
continue # Don't mark scheduled streams as seen; queue them when they go live
eid = _entry_id(ent)
if eid:
all_ids.append(eid)
sub = SubscriptionInfo(
id=str(uuid.uuid4()),
name=str(name),
url=url,
enabled=True,
check_interval_minutes=max(1, int(check_interval_minutes)),
download_type=download_type,
codec=codec,
format=format,
quality=quality,
folder=folder or "",
custom_name_prefix=custom_name_prefix or "",
auto_start=bool(auto_start),
playlist_item_limit=int(playlist_item_limit),
split_by_chapters=bool(split_by_chapters),
chapter_template=chapter_template or "",
subtitle_language=subtitle_language,
subtitle_mode=subtitle_mode,
ytdl_options_presets=list(ytdl_options_presets or []),
ytdl_options_overrides=dict(ytdl_options_overrides or {}),
title_regex=title_regex_stored,
skip_subscriber_only=skip_so,
last_checked=time.time(),
seen_ids=list(dict.fromkeys(all_ids)),
error=None,
)
async with self._lock:
if url in self._url_index:
return {"status": "error", "msg": "This URL is already subscribed"}
self._subs[sub.id] = sub
self._url_index[url] = sub.id
try:
self._save_locked()
except Exception:
self._subs.pop(sub.id, None)
self._url_index.pop(url, None)
raise
await self.notifier.subscription_added(sub)
return {"status": "ok", "subscription": sub.to_public_dict()}
finally:
async with self._lock:
self._pending_urls.discard(url)
async def delete_subscriptions(self, ids: list[str]) -> dict:
removed: list[str] = []
async with self._lock:
previous_subs = self._subs.copy()
previous_index = self._url_index.copy()
for sid in ids:
sub = self._subs.pop(sid, None)
if sub:
normalized_url = self._normalize_url(sub.url)
self._url_index.pop(normalized_url, None)
removed.append(sid)
if removed:
try:
self._save_locked()
except Exception:
self._subs = previous_subs
self._url_index = previous_index
raise
for sid in removed:
await self.notifier.subscription_removed(sid)
return {"status": "ok"}
async def update_subscription(self, sub_id: str, changes: dict) -> dict:
validated_tr: Optional[str] = None
if "title_regex" in changes:
try:
validated_tr = validate_title_regex(changes["title_regex"])
except re.error as exc:
return {"status": "error", "msg": f"Invalid title_regex: {exc}"}
skip_so_set = False
validated_skip_so = False
if "skip_subscriber_only" in changes:
try:
validated_skip_so = coerce_optional_bool(
changes["skip_subscriber_only"],
field_name="skip_subscriber_only",
)
skip_so_set = True
except ValueError as exc:
return {"status": "error", "msg": str(exc)}
enabled_set = False
validated_enabled = False
if "enabled" in changes:
try:
validated_enabled = _coerce_bool(changes["enabled"])
enabled_set = True
except ValueError as exc:
return {"status": "error", "msg": str(exc)}
interval_set = False
validated_interval = 0
if "check_interval_minutes" in changes:
try:
validated_interval = max(1, int(changes["check_interval_minutes"]))
except (TypeError, ValueError):
return {"status": "error", "msg": "check_interval_minutes must be an integer"}
interval_set = True
async with self._lock:
sub = self._subs.get(sub_id)
if not sub:
return {"status": "error", "msg": "Subscription not found"}
previous = copy.deepcopy(sub)
old_enabled = sub.enabled
if enabled_set:
sub.enabled = validated_enabled
if interval_set:
sub.check_interval_minutes = validated_interval
if "name" in changes and changes["name"]:
sub.name = str(changes["name"])
if validated_tr is not None:
sub.title_regex = validated_tr
if skip_so_set:
sub.skip_subscriber_only = validated_skip_so
try:
self._save_locked()
except Exception:
self._subs[sub_id] = previous
raise
updated = sub
if "enabled" in changes and updated.enabled != old_enabled:
log.info(
"Subscription %s %s",
updated.name,
"resumed" if updated.enabled else "paused",
)
await self.notifier.subscription_updated(updated)
return {"status": "ok", "subscription": updated.to_public_dict()}
async def check_now(self, ids: Optional[list[str]] = None) -> dict:
async with self._lock:
targets = (
[self._subs[i] for i in ids if i in self._subs]
if ids
else [s for s in self._subs.values() if s.enabled]
)
log.info(
"Manual subscription check requested for %d subscription(s)",
len(targets),
)
await self._check_many(targets)
return {"status": "ok"}
async def _check_one_unlocked(self, sub: SubscriptionInfo) -> None:
sid = sub.id
# Prevent overlapping checks for the same subscription (e.g. the periodic
# loop and a manual check-now firing together), which could double-queue
# entries and drop seen_ids via a read-modify-write race.
async with self._lock:
if sid in self._checks_in_flight:
log.info("Subscription check already in progress for %s, skipping", sub.name)
return
self._checks_in_flight.add(sid)
try:
await self._check_one_inner(sub)
finally:
async with self._lock:
self._checks_in_flight.discard(sid)
async def _check_one_inner(self, sub: SubscriptionInfo) -> None:
sid = sub.id
scan = int(getattr(self.config, "SUBSCRIPTION_SCAN_PLAYLIST_END", 50))
# ytdl_options_presets/overrides are set at subscription creation and
# never mutated afterwards (update_subscription doesn't allow it), so
# reading them off `sub` here without holding the lock is safe.
scan_extra_opts = self._scan_extra_opts(sub.ytdl_options_presets, sub.ytdl_options_overrides)
log.info("Checking subscription: %s", sub.name)
try:
info, entries = await asyncio.get_running_loop().run_in_executor(
None,
partial(extract_flat_playlist, self.config, sub.url, scan, extra_opts=scan_extra_opts),
)
except yt_dlp.utils.YoutubeDLError as exc:
async with self._lock:
cur = self._subs.get(sid)
if cur:
previous = copy.deepcopy(cur)
cur.error = str(exc)
cur.last_checked = time.time()
try:
self._save_locked()
except Exception:
self._subs[sid] = previous
raise
sub = cur
log.warning("Subscription check failed for %s: %s", sub.name, exc)
await self.notifier.subscription_updated(sub)
return
entries = [ent for ent in entries if _is_media_entry(ent)]
etype = (info or {}).get("_type") or "video"
if etype == "video":
async with self._lock:
cur = self._subs.get(sid)
if cur:
previous = copy.deepcopy(cur)
cur.error = VIDEO_ONLY_MSG
cur.last_checked = time.time()
try:
self._save_locked()
except Exception:
self._subs[sid] = previous
raise
sub = cur
log.warning("Subscription %s no longer resolves to a subscribable feed", sub.name)
await self.notifier.subscription_updated(sub)
return
if not entries:
async with self._lock:
cur = self._subs.get(sid)
if cur:
previous = copy.deepcopy(cur)
cur.last_checked = time.time()
cur.error = None
try:
self._save_locked()
except Exception:
self._subs[sid] = previous
raise
sub = cur
log.warning("Subscription check finished for %s: No entries found", sub.name)
await self.notifier.subscription_updated(sub)
return
async with self._lock:
cur = self._subs.get(sid)
if not cur:
return
seen = cur.seen_set()
seen_ids_snapshot = list(cur.seen_ids)
dl_type = cur.download_type
dl_codec = cur.codec
dl_format = cur.format
dl_quality = cur.quality
dl_folder = cur.folder
dl_prefix = cur.custom_name_prefix
dl_plimit = cur.playlist_item_limit
dl_autostart = cur.auto_start
dl_split = cur.split_by_chapters
dl_chapter = cur.chapter_template
dl_sublang = cur.subtitle_language
dl_submode = cur.subtitle_mode
dl_ytdl_presets = list(cur.ytdl_options_presets)
dl_ytdl_overrides = dict(cur.ytdl_options_overrides)
dl_title_regex = cur.title_regex or ""
dl_skip_subscriber_only = bool(cur.skip_subscriber_only)
new_entries: list[dict] = []
for ent in entries:
eid = _entry_id(ent)
if not eid:
continue
# Seen entries that are currently live are deliberately re-queued:
# a stream first seen as 'upcoming' must still be captured once it
# goes live. The download queue dedups by URL while a capture is
# in flight, so this can't double-queue an active capture.
if eid in seen and ent.get("live_status") != "is_live":
continue
new_entries.append(ent)
pattern_re: Optional[re.Pattern[str]] = None
if dl_title_regex:
try:
pattern_re = re.compile(dl_title_regex)
except re.error:
log.warning(
"Invalid stored title_regex on subscription %s, ignoring filter",
sub.name,
)
queue_entries: list[dict] = []
filtered_ids: list[str] = []
for ent in new_entries:
eid = _entry_id(ent)
if pattern_re is not None:
title = str(ent.get("title") or "")
if not pattern_re.search(title):
if eid:
filtered_ids.append(eid)
continue
queue_entries.append(ent)
subscriber_filtered_ids: list[str] = []
if dl_skip_subscriber_only:
kept_entries: list[dict] = []
for ent in queue_entries:
eid = _entry_id(ent)
if _is_subscriber_only_entry(ent):
if eid:
subscriber_filtered_ids.append(eid)
continue
kept_entries.append(ent)
queue_entries = kept_entries
queued_ids, queue_errors = await self._queue_subscription_entries(
queue_entries,
download_type=dl_type,
codec=dl_codec,
format=dl_format,
quality=dl_quality,
folder=dl_folder,
custom_name_prefix=dl_prefix,
playlist_item_limit=dl_plimit,
auto_start=dl_autostart,
split_by_chapters=dl_split,
chapter_template=dl_chapter or "",
subtitle_language=dl_sublang,
subtitle_mode=dl_submode,
ytdl_options_presets=dl_ytdl_presets,
ytdl_options_overrides=dl_ytdl_overrides,
)
log.info(
"Subscription check finished for %s: %d new, %d filtered, %d subscriber_skipped, %d queued, %d failed",
sub.name,
len(new_entries),
len(filtered_ids),
len(subscriber_filtered_ids),
len(queued_ids),
len(queue_errors),
)
merged = list(
dict.fromkeys(
queued_ids + filtered_ids + subscriber_filtered_ids + seen_ids_snapshot
)
)
max_seen = int(getattr(self.config, "SUBSCRIPTION_MAX_SEEN_IDS", 50000))
if len(merged) > max_seen:
merged = merged[:max_seen]
async with self._lock:
cur = self._subs.get(sid)
if not cur:
return
previous = copy.deepcopy(cur)
cur.seen_ids = merged
cur.last_checked = time.time()
cur.error = "; ".join(queue_errors[:3]) if queue_errors else None
try:
self._save_locked()
except Exception:
self._subs[sid] = previous
raise
sub = cur
await self.notifier.subscription_updated(sub)
async def emit_all(self) -> None:
await self.notifier.subscriptions_all(self.list_all())

32
app/tests/conftest.py Normal file
View file

@ -0,0 +1,32 @@
"""Pytest configuration: set env and filesystem layout before importing ``main``."""
from __future__ import annotations
import os
import tempfile
from pathlib import Path
def _ensure_test_env() -> None:
if os.environ.get("METUBE_TEST_ENV_READY"):
return
tmp = tempfile.mkdtemp(prefix="metube-pytest-")
base = Path(tmp)
browser = base / "ui" / "dist" / "metube" / "browser"
browser.mkdir(parents=True)
(browser / "index.html").write_text("<html><body></body></html>", encoding="utf-8")
dl = base / "downloads"
st = base / "state"
dl.mkdir(parents=True)
st.mkdir(parents=True)
os.environ["DOWNLOAD_DIR"] = str(dl)
os.environ["STATE_DIR"] = str(st)
os.environ["TEMP_DIR"] = str(dl)
os.environ["YTDL_OPTIONS"] = "{}"
os.environ["YTDL_OPTIONS_FILE"] = ""
os.environ["BASE_DIR"] = str(base)
os.environ["LOGLEVEL"] = "INFO"
os.environ["METUBE_TEST_ENV_READY"] = "1"
_ensure_test_env()

430
app/tests/test_api.py Normal file
View file

@ -0,0 +1,430 @@
"""HTTP handler tests for ``main`` using mocked ``web.Request`` (no TestServer)."""
from __future__ import annotations
import json
import os
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock
from urllib.parse import quote
import pytest
from aiohttp import web
from aiohttp.test_utils import TestClient, TestServer
import main
@pytest.fixture
def mock_dqueue(monkeypatch):
d = MagicMock()
d.initialize = AsyncMock(return_value=None)
d.add = AsyncMock(return_value={"status": "ok"})
d.cancel = AsyncMock(return_value={"status": "ok"})
d.clear = AsyncMock(return_value={"status": "ok"})
d.start_pending = AsyncMock(return_value={"status": "ok"})
d.cancel_add = MagicMock()
d.queue = MagicMock()
d.done = MagicMock()
d.pending = MagicMock()
d.queue.saved_items = MagicMock(return_value=[])
d.done.saved_items = MagicMock(return_value=[])
d.pending.saved_items = MagicMock(return_value=[])
d.queue.items = MagicMock(return_value=[])
d.done.items = MagicMock(return_value=[])
d.pending.items = MagicMock(return_value=[])
d.get = MagicMock(return_value=([], []))
monkeypatch.setattr(main, "dqueue", d)
return d
def _valid_video_add_body(**kwargs):
base = {
"url": "https://example.com/watch?v=1",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
"ytdl_options_presets": [],
"ytdl_options_overrides": "",
}
base.update(kwargs)
return base
def _json_request(body: dict | None):
req = MagicMock(spec=web.Request)
req.json = AsyncMock(return_value=body)
return req
@pytest.mark.asyncio
async def test_add_ok(mock_dqueue):
req = _json_request(_valid_video_add_body())
resp = await main.add(req)
assert resp.status == 200
text = resp.text
data = json.loads(text)
assert data["status"] == "ok"
mock_dqueue.add.assert_awaited_once()
@pytest.mark.asyncio
async def test_add_passes_preset_and_overrides(mock_dqueue, monkeypatch):
monkeypatch.setattr(main.config, "YTDL_OPTIONS_PRESETS", {"Preset A": {"writesubtitles": True}})
monkeypatch.setattr(main.config, "ALLOW_YTDL_OPTIONS_OVERRIDES", True)
req = _json_request(
_valid_video_add_body(
ytdl_options_presets=["Preset A"],
ytdl_options_overrides='{"writesubtitles": true}',
)
)
resp = await main.add(req)
assert resp.status == 200
call = mock_dqueue.add.await_args
assert call is not None
assert call.args[13] == ["Preset A"]
assert call.args[14] == {"writesubtitles": True}
@pytest.mark.asyncio
async def test_add_legacy_string_preset_normalized(mock_dqueue, monkeypatch):
monkeypatch.setattr(main.config, "YTDL_OPTIONS_PRESETS", {"Legacy": {}})
body = _valid_video_add_body()
del body["ytdl_options_presets"]
body["ytdl_options_preset"] = "Legacy"
req = _json_request(body)
resp = await main.add(req)
assert resp.status == 200
call = mock_dqueue.add.await_args
assert call.args[13] == ["Legacy"]
@pytest.mark.asyncio
async def test_add_missing_url_returns_400(mock_dqueue):
req = _json_request({"download_type": "video", "quality": "best", "format": "any"})
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
mock_dqueue.add.assert_not_called()
@pytest.mark.asyncio
async def test_add_invalid_download_type(mock_dqueue):
req = _json_request(_valid_video_add_body(download_type="invalid"))
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_invalid_video_quality(mock_dqueue):
req = _json_request(_valid_video_add_body(quality="9999"))
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_invalid_subtitle_language(mock_dqueue):
req = _json_request(
{
"url": "https://example.com/v",
"download_type": "captions",
"codec": "auto",
"format": "srt",
"quality": "best",
"subtitle_language": "bad language!",
}
)
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_custom_name_prefix_path_traversal(mock_dqueue):
req = _json_request(_valid_video_add_body(custom_name_prefix="../evil"))
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_chapter_template_path_traversal(mock_dqueue):
req = _json_request(
_valid_video_add_body(
split_by_chapters=True,
chapter_template="/etc/passwd%(title)s",
)
)
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_invalid_json_body(mock_dqueue):
req = MagicMock(spec=web.Request)
req.json = AsyncMock(side_effect=json.JSONDecodeError("msg", "", 0))
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_invalid_ytdl_options_override_json(mock_dqueue):
req = _json_request(_valid_video_add_body(ytdl_options_overrides="{bad json}"))
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_rejects_ytdl_options_overrides_when_disabled(mock_dqueue):
req = _json_request(_valid_video_add_body(ytdl_options_overrides='{"exec": "rm -rf /"}'))
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_add_allows_any_ytdl_options_override_key_when_enabled(mock_dqueue, monkeypatch):
monkeypatch.setattr(main.config, "ALLOW_YTDL_OPTIONS_OVERRIDES", True)
req = _json_request(_valid_video_add_body(ytdl_options_overrides='{"exec": "echo hi"}'))
resp = await main.add(req)
assert resp.status == 200
call = mock_dqueue.add.await_args
assert call is not None
assert call.args[14] == {"exec": "echo hi"}
@pytest.mark.asyncio
async def test_add_unknown_ytdl_preset(mock_dqueue):
req = _json_request(_valid_video_add_body(ytdl_options_presets=["Missing"]))
with pytest.raises(web.HTTPBadRequest):
await main.add(req)
@pytest.mark.asyncio
async def test_delete_missing_ids(mock_dqueue):
req = _json_request({"where": "queue"})
with pytest.raises(web.HTTPBadRequest):
await main.delete(req)
@pytest.mark.asyncio
async def test_delete_queue_calls_cancel(mock_dqueue):
req = _json_request({"where": "queue", "ids": ["http://x"]})
resp = await main.delete(req)
assert resp.status == 200
mock_dqueue.cancel.assert_awaited_once_with(["http://x"])
@pytest.mark.asyncio
async def test_start_pending(mock_dqueue):
req = _json_request({"ids": ["a"]})
resp = await main.start(req)
assert resp.status == 200
mock_dqueue.start_pending.assert_awaited_once_with(["a"])
@pytest.mark.asyncio
@pytest.mark.parametrize("body", [{}, {"ids": "abc"}, {"ids": []}, {"ids": [1, 2]}])
async def test_start_rejects_malformed_ids(mock_dqueue, body):
req = _json_request(body)
with pytest.raises(web.HTTPBadRequest):
await main.start(req)
mock_dqueue.start_pending.assert_not_awaited()
@pytest.mark.asyncio
@pytest.mark.parametrize(
"body",
[
{"where": "queue"},
{"where": "queue", "ids": "abc"},
{"where": "queue", "ids": []},
{"where": "queue", "ids": [1, 2]},
],
)
async def test_delete_rejects_malformed_ids(mock_dqueue, body):
req = _json_request(body)
with pytest.raises(web.HTTPBadRequest):
await main.delete(req)
mock_dqueue.cancel.assert_not_awaited()
mock_dqueue.clear.assert_not_awaited()
@pytest.mark.asyncio
async def test_history_shape(mock_dqueue):
req = MagicMock(spec=web.Request)
resp = await main.history(req)
assert resp.status == 200
data = json.loads(resp.text)
assert set(data.keys()) == {"done", "queue", "pending"}
@pytest.mark.asyncio
async def test_history_reads_in_memory_queues_not_disk_state(mock_dqueue):
fake_queue_dl = MagicMock()
fake_queue_dl.info = {"id": "q1", "title": "Queued"}
fake_done_dl = MagicMock()
fake_done_dl.info = {"id": "d1", "title": "Done"}
fake_pending_dl = MagicMock()
fake_pending_dl.info = {"id": "p1", "title": "Pending"}
mock_dqueue.queue.items.return_value = [("q1", fake_queue_dl)]
mock_dqueue.done.items.return_value = [("d1", fake_done_dl)]
mock_dqueue.pending.items.return_value = [("p1", fake_pending_dl)]
req = MagicMock(spec=web.Request)
resp = await main.history(req)
assert resp.status == 200
data = json.loads(resp.text)
assert [item["id"] for item in data["queue"]] == ["q1"]
assert [item["id"] for item in data["done"]] == ["d1"]
assert [item["id"] for item in data["pending"]] == ["p1"]
mock_dqueue.queue.saved_items.assert_not_called()
mock_dqueue.done.saved_items.assert_not_called()
mock_dqueue.pending.saved_items.assert_not_called()
@pytest.mark.asyncio
async def test_version_json(mock_dqueue):
req = MagicMock(spec=web.Request)
resp = await main.version(req)
assert resp.status == 200
body = json.loads(resp.text)
assert "yt-dlp" in body and "version" in body
@pytest.mark.asyncio
async def test_presets_endpoint_returns_names(mock_dqueue, monkeypatch):
monkeypatch.setattr(main.config, "YTDL_OPTIONS_PRESETS", {"Preset B": {}, "Preset A": {}})
req = MagicMock(spec=web.Request)
resp = await main.presets(req)
assert resp.status == 200
assert json.loads(resp.text) == {"presets": ["Preset A", "Preset B"]}
@pytest.mark.asyncio
async def test_cookie_status(mock_dqueue):
req = MagicMock(spec=web.Request)
resp = await main.cookie_status(req)
assert resp.status == 200
data = json.loads(resp.text)
assert data.get("status") == "ok"
assert "has_cookies" in data
@pytest.mark.asyncio
async def test_options_add_cors(mock_dqueue):
req = MagicMock(spec=web.Request)
resp = await main.add_cors(req)
assert resp.status == 200
@pytest.mark.asyncio
async def test_upload_cookies_missing_field(mock_dqueue):
req = MagicMock(spec=web.Request)
reader = MagicMock()
field = MagicMock()
field.name = "wrongname"
reader.next = AsyncMock(side_effect=[field, None])
req.multipart = AsyncMock(return_value=reader)
resp = await main.upload_cookies(req)
assert resp.status == 400
@pytest.mark.asyncio
async def test_add_legacy_format_migrated(mock_dqueue):
req = _json_request({"url": "https://example.com/v", "format": "m4a", "quality": "best"})
resp = await main.add(req)
assert resp.status == 200
call = mock_dqueue.add.await_args
assert call is not None
assert call.args[1] == "audio"
@pytest.mark.asyncio
async def test_add_passes_clip_bounds_to_queue(mock_dqueue):
req = _json_request(
_valid_video_add_body(clip_start="2:26", clip_end="3:24"),
)
resp = await main.add(req)
assert resp.status == 200
call = mock_dqueue.add.await_args
assert call is not None
assert call.args[15] == pytest.approx(146.0)
assert call.args[16] == pytest.approx(204.0)
@pytest.mark.asyncio
async def test_subscribe_rejects_clip_options(mock_dqueue, monkeypatch):
monkeypatch.setattr(main.submgr, "add_subscription", AsyncMock())
req = _json_request(
{
**_valid_video_add_body(clip_start="10"),
"check_interval_minutes": 60,
}
)
with pytest.raises(web.HTTPBadRequest):
await main.subscribe(req)
main.submgr.add_subscription.assert_not_awaited()
@pytest.mark.asyncio
async def test_subscriptions_update_invalid_enabled_returns_error_not_500(mock_dqueue):
req = _json_request({"id": "nonexistent", "enabled": "maybe"})
resp = await main.subscriptions_update(req)
assert resp.status == 200
body = json.loads(resp.text)
assert body["status"] == "error"
@pytest.mark.asyncio
async def test_subscriptions_update_invalid_interval_returns_error_not_500(mock_dqueue):
req = _json_request({"id": "nonexistent", "check_interval_minutes": "abc"})
resp = await main.subscriptions_update(req)
assert resp.status == 200
body = json.loads(resp.text)
assert body["status"] == "error"
def test_is_within_state_dir_blocks_state_subtree():
state_dir = main._STATE_DIR_REAL
assert main._is_within_state_dir(state_dir)
assert main._is_within_state_dir(os.path.join(state_dir, "cookies.txt"))
assert main._is_within_state_dir(os.path.join(state_dir, "queue", "item.json"))
def test_is_within_state_dir_allows_sibling_downloads():
download_dir = os.path.realpath(main.config.DOWNLOAD_DIR)
assert not main._is_within_state_dir(os.path.join(download_dir, "video.mp4"))
assert not main._is_within_state_dir("/tmp/unrelated/video.mp4")
@pytest.mark.asyncio
async def test_download_blocks_state_dir_files(monkeypatch):
download_dir = Path(main.config.DOWNLOAD_DIR)
state_dir = download_dir / ".metube"
state_dir.mkdir(parents=True, exist_ok=True)
(state_dir / "cookies.txt").write_text("# Netscape HTTP Cookie File\n", encoding="utf-8")
(download_dir / "video.mp4").write_bytes(b"video")
# request.path is already percent-decoded by aiohttp; state_dir_guard must
# not decode it a second time, or a filename containing a literal '%'
# gets mangled into a false 404.
percent_filename = "100% done.mp4"
(download_dir / percent_filename).write_bytes(b"percent video")
monkeypatch.setattr(main.config, "STATE_DIR", str(state_dir))
monkeypatch.setattr(main, "_STATE_DIR_REAL", os.path.realpath(str(state_dir)))
try:
async with TestClient(TestServer(main.app)) as client:
blocked = await client.get("/download/.metube/cookies.txt")
assert blocked.status == 404
allowed = await client.get("/download/video.mp4")
assert allowed.status == 200
assert await allowed.read() == b"video"
percent_resp = await client.get("/download/" + quote(percent_filename))
assert percent_resp.status == 200
assert await percent_resp.read() == b"percent video"
finally:
(state_dir / "cookies.txt").unlink(missing_ok=True)
(download_dir / "video.mp4").unlink(missing_ok=True)
(download_dir / percent_filename).unlink(missing_ok=True)
state_dir.rmdir()

View file

@ -0,0 +1,55 @@
"""Tests for the ``bg_tasks.create_task`` strong-reference/logging helper."""
from __future__ import annotations
import asyncio
import logging
import pytest
import bg_tasks
@pytest.mark.asyncio
async def test_create_task_removes_itself_from_registry_on_success():
async def _ok():
return 42
task = bg_tasks.create_task(_ok(), name="ok_task")
assert task in bg_tasks._TASKS
result = await task
assert result == 42
assert task not in bg_tasks._TASKS
@pytest.mark.asyncio
async def test_create_task_logs_unhandled_exception(caplog):
async def _boom():
raise ValueError("kaboom")
with caplog.at_level(logging.ERROR, logger="bg_tasks"):
task = bg_tasks.create_task(_boom(), name="boom_task")
with pytest.raises(ValueError):
await task
# Let the done-callback (scheduled via call_soon) run.
await asyncio.sleep(0)
assert task not in bg_tasks._TASKS
assert any("boom_task" in record.message for record in caplog.records)
@pytest.mark.asyncio
async def test_create_task_does_not_log_on_cancellation(caplog):
async def _sleep_forever():
await asyncio.sleep(10)
with caplog.at_level(logging.ERROR, logger="bg_tasks"):
task = bg_tasks.create_task(_sleep_forever(), name="cancel_task")
await asyncio.sleep(0)
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
await asyncio.sleep(0)
assert task not in bg_tasks._TASKS
assert not any("cancel_task" in record.message for record in caplog.records)

231
app/tests/test_config.py Normal file
View file

@ -0,0 +1,231 @@
"""Tests for ``Config`` (env parsing, yt-dlp options, frontend_safe)."""
from __future__ import annotations
import json
import os
import tempfile
import unittest
from unittest.mock import patch
from main import Config
def _base_env(**overrides: str) -> dict[str, str]:
env = {k: str(v) for k, v in Config._DEFAULTS.items()}
env.update(overrides)
return env
class ConfigTests(unittest.TestCase):
def test_url_prefix_gets_trailing_slash(self):
with patch.dict(os.environ, _base_env(URL_PREFIX="foo"), clear=False):
c = Config()
self.assertEqual(c.URL_PREFIX, "foo/")
def test_public_host_url_gets_trailing_slash(self):
with patch.dict(
os.environ,
_base_env(PUBLIC_HOST_URL="https://ytdl.example.com"),
clear=False,
):
c = Config()
self.assertEqual(c.PUBLIC_HOST_URL, "https://ytdl.example.com/")
def test_public_host_audio_url_gets_trailing_slash(self):
with patch.dict(
os.environ,
_base_env(PUBLIC_HOST_AUDIO_URL="https://audio.example.com"),
clear=False,
):
c = Config()
self.assertEqual(c.PUBLIC_HOST_AUDIO_URL, "https://audio.example.com/")
def test_public_host_url_empty_stays_empty(self):
with patch.dict(
os.environ,
_base_env(PUBLIC_HOST_URL="", PUBLIC_HOST_AUDIO_URL=""),
clear=False,
):
c = Config()
self.assertEqual(c.PUBLIC_HOST_URL, "")
self.assertEqual(c.PUBLIC_HOST_AUDIO_URL, "")
def test_blank_audio_host_falls_back_to_audio_download_route(self):
# Regression: a present-but-blank PUBLIC_HOST_AUDIO_URL must not stay empty
# (which produced root-relative, 404ing audio links). It falls back to the
# 'audio_download/' route that serves AUDIO_DOWNLOAD_DIR.
with patch.dict(
os.environ,
_base_env(PUBLIC_HOST_URL="https://ytdl.example.com", PUBLIC_HOST_AUDIO_URL=""),
clear=False,
):
c = Config()
self.assertEqual(c.PUBLIC_HOST_URL, "https://ytdl.example.com/")
self.assertEqual(c.PUBLIC_HOST_AUDIO_URL, "audio_download/")
def test_public_host_url_already_slashed_unchanged(self):
with patch.dict(
os.environ,
_base_env(
PUBLIC_HOST_URL="https://ytdl.example.com/",
PUBLIC_HOST_AUDIO_URL="https://audio.example.com/",
),
clear=False,
):
c = Config()
self.assertEqual(c.PUBLIC_HOST_URL, "https://ytdl.example.com/")
self.assertEqual(c.PUBLIC_HOST_AUDIO_URL, "https://audio.example.com/")
def test_ytdl_options_json_loaded(self):
opts = {"quiet": True, "no_warnings": True}
with patch.dict(
os.environ,
_base_env(YTDL_OPTIONS=json.dumps(opts)),
clear=False,
):
c = Config()
self.assertEqual(c.YTDL_OPTIONS["quiet"], True)
def test_ytdl_option_presets_json_loaded(self):
presets = {"Audio extras": {"embed_thumbnail": True}}
with patch.dict(
os.environ,
_base_env(YTDL_OPTIONS_PRESETS=json.dumps(presets)),
clear=False,
):
c = Config()
self.assertEqual(c.YTDL_OPTIONS_PRESETS["Audio extras"]["embed_thumbnail"], True)
def test_invalid_ytdl_options_exits(self):
with patch.dict(os.environ, _base_env(YTDL_OPTIONS="not-json"), clear=False):
with self.assertRaises(SystemExit):
Config()
def test_invalid_boolean_env_exits(self):
with patch.dict(os.environ, _base_env(CUSTOM_DIRS="maybe"), clear=False):
with self.assertRaises(SystemExit):
Config()
def test_frontend_safe_excludes_secrets(self):
with patch.dict(os.environ, _base_env(), clear=False):
c = Config()
safe = c.frontend_safe()
self.assertNotIn("YTDL_OPTIONS", safe)
self.assertNotIn("HOST", safe)
self.assertEqual(safe["ALLOW_YTDL_OPTIONS_OVERRIDES"], False)
def test_allow_ytdl_options_overrides_boolean_loaded(self):
with patch.dict(os.environ, _base_env(ALLOW_YTDL_OPTIONS_OVERRIDES="true"), clear=False):
c = Config()
self.assertTrue(c.ALLOW_YTDL_OPTIONS_OVERRIDES)
def test_ytdl_nightly_update_time_empty_default(self):
with patch.dict(os.environ, _base_env(YTDL_NIGHTLY_UPDATE_TIME=""), clear=False):
c = Config()
self.assertEqual(c.YTDL_NIGHTLY_UPDATE_TIME, "")
def test_ytdl_nightly_update_time_valid(self):
with patch.dict(os.environ, _base_env(YTDL_NIGHTLY_UPDATE_TIME="04:00"), clear=False):
c = Config()
self.assertEqual(c.YTDL_NIGHTLY_UPDATE_TIME, "04:00")
def test_ytdl_nightly_update_time_invalid_exits(self):
for bad in ("25:00", "4am", "12:60"):
with patch.dict(os.environ, _base_env(YTDL_NIGHTLY_UPDATE_TIME=bad), clear=False):
with self.assertRaises(SystemExit):
Config()
def test_invalid_max_concurrent_downloads_exits(self):
for bad in ("0", "-1", "abc"):
with patch.dict(os.environ, _base_env(MAX_CONCURRENT_DOWNLOADS=bad), clear=False):
with self.assertRaises(SystemExit):
Config()
def test_invalid_port_exits(self):
for bad in ("0", "70000", "notaport"):
with patch.dict(os.environ, _base_env(PORT=bad), clear=False):
with self.assertRaises(SystemExit):
Config()
def test_invalid_clear_completed_after_exits(self):
for bad in ("-5", "soon"):
with patch.dict(os.environ, _base_env(CLEAR_COMPLETED_AFTER=bad), clear=False):
with self.assertRaises(SystemExit):
Config()
def test_clear_completed_after_zero_allowed(self):
with patch.dict(os.environ, _base_env(CLEAR_COMPLETED_AFTER="0"), clear=False):
c = Config()
self.assertEqual(c.CLEAR_COMPLETED_AFTER, "0")
def test_invalid_default_option_playlist_item_limit_exits(self):
for bad in ("-1", "many"):
with patch.dict(os.environ, _base_env(DEFAULT_OPTION_PLAYLIST_ITEM_LIMIT=bad), clear=False):
with self.assertRaises(SystemExit):
Config()
def test_default_option_playlist_item_limit_zero_allowed(self):
with patch.dict(os.environ, _base_env(DEFAULT_OPTION_PLAYLIST_ITEM_LIMIT="0"), clear=False):
c = Config()
self.assertEqual(c.DEFAULT_OPTION_PLAYLIST_ITEM_LIMIT, "0")
def test_invalid_subscription_default_check_interval_exits(self):
for bad in ("0", "-1", "often"):
with patch.dict(os.environ, _base_env(SUBSCRIPTION_DEFAULT_CHECK_INTERVAL=bad), clear=False):
with self.assertRaises(SystemExit):
Config()
def test_invalid_subscription_scan_playlist_end_exits(self):
for bad in ("0", "-1", "all"):
with patch.dict(os.environ, _base_env(SUBSCRIPTION_SCAN_PLAYLIST_END=bad), clear=False):
with self.assertRaises(SystemExit):
Config()
def test_invalid_subscription_max_seen_ids_exits(self):
for bad in ("0", "-1", "unlimited"):
with patch.dict(os.environ, _base_env(SUBSCRIPTION_MAX_SEEN_IDS=bad), clear=False):
with self.assertRaises(SystemExit):
Config()
def test_runtime_override_roundtrip(self):
with patch.dict(os.environ, _base_env(), clear=False):
c = Config()
c.set_runtime_override("cookiefile", "/tmp/c.txt")
self.assertEqual(c.YTDL_OPTIONS.get("cookiefile"), "/tmp/c.txt")
c.remove_runtime_override("cookiefile")
self.assertIsNone(c.YTDL_OPTIONS.get("cookiefile"))
def test_ytdl_options_file_merges(self):
with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f:
json.dump({"extractor_args": {"youtube": {"player_client": ["web"]}}}, f)
path = f.name
try:
with patch.dict(
os.environ,
_base_env(YTDL_OPTIONS="{}", YTDL_OPTIONS_FILE=path),
clear=False,
):
c = Config()
self.assertIn("extractor_args", c.YTDL_OPTIONS)
finally:
os.unlink(path)
def test_ytdl_option_presets_file_merges(self):
with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f:
json.dump({"With subtitles": {"writesubtitles": True}}, f)
path = f.name
try:
with patch.dict(
os.environ,
_base_env(YTDL_OPTIONS_PRESETS="{}", YTDL_OPTIONS_PRESETS_FILE=path),
clear=False,
):
c = Config()
self.assertIn("With subtitles", c.YTDL_OPTIONS_PRESETS)
finally:
os.unlink(path)
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,182 @@
"""Tests for ``app.dl_formats`` format selectors and yt-dlp option mapping."""
from __future__ import annotations
import copy
import unittest
from app.dl_formats import (
_normalize_caption_mode,
_normalize_subtitle_language,
get_format,
get_opts,
merge_ytdl_option_layers,
)
class DlFormatsTests(unittest.TestCase):
def test_audio_unknown_format_raises_value_error(self):
with self.assertRaises(ValueError):
get_format("audio", "auto", "invalid", "best")
def test_wav_does_not_enable_thumbnail_postprocessing(self):
opts = get_opts("audio", "auto", "wav", "best", {})
self.assertNotIn("writethumbnail", opts)
def test_mp3_enables_thumbnail_postprocessing(self):
opts = get_opts("audio", "auto", "mp3", "best", {})
self.assertTrue(opts.get("writethumbnail"))
def test_custom_format_passthrough(self):
self.assertEqual(get_format("video", "auto", "custom:bestvideo+bestaudio", "best"), "bestvideo+bestaudio")
def test_thumbnail_and_captions_format_strings(self):
self.assertEqual(get_format("thumbnail", "auto", "jpg", "best"), "bestaudio/best")
self.assertEqual(get_format("captions", "auto", "srt", "best"), "bestaudio/best")
def test_audio_formats(self):
for fmt in ("m4a", "mp3", "opus", "wav", "flac"):
with self.subTest(fmt=fmt):
self.assertIn(f"ext={fmt}", get_format("audio", "auto", fmt, "best"))
def test_video_unknown_format_raises(self):
with self.assertRaises(ValueError):
get_format("video", "auto", "mkv", "best")
def test_unknown_download_type_raises(self):
with self.assertRaises(ValueError):
get_format("unknown", "auto", "any", "best")
def test_video_any_mp4_ios_with_height_quality(self):
self.assertIn("height<=1080", get_format("video", "auto", "any", "1080"))
self.assertNotIn("height<=", get_format("video", "auto", "any", "best"))
self.assertNotIn("height<=", get_format("video", "auto", "any", "worst"))
def test_video_codec_filters(self):
self.assertIn("h264", get_format("video", "h264", "any", "best"))
self.assertIn("hevc", get_format("video", "h265", "any", "best"))
self.assertIn("av0?1", get_format("video", "av1", "any", "best"))
self.assertIn("vp0?9", get_format("video", "vp9", "any", "best"))
def test_video_mp4_includes_m4a_audio(self):
s = get_format("video", "auto", "mp4", "720")
self.assertIn("[ext=m4a]", s)
def test_video_ios_selector_contains_avc_pattern(self):
s = get_format("video", "auto", "ios", "best")
self.assertIn("h26[45]", s)
def test_get_opts_deepcopy_does_not_mutate_input(self):
base = {"postprocessors": [{"key": "Existing"}]}
orig = copy.deepcopy(base)
get_opts("audio", "auto", "mp3", "best", base)
self.assertEqual(base, orig)
def test_get_opts_audio_m4a_postprocessors(self):
opts = get_opts("audio", "auto", "m4a", "best", {})
keys = [p["key"] for p in opts["postprocessors"]]
self.assertIn("FFmpegExtractAudio", keys)
def test_get_opts_audio_mp3_quality_not_best(self):
opts = get_opts("audio", "auto", "mp3", "192", {})
ext = next(p for p in opts["postprocessors"] if p["key"] == "FFmpegExtractAudio")
self.assertEqual(ext["preferredquality"], "192")
def test_get_opts_thumbnail_skip_download(self):
opts = get_opts("thumbnail", "auto", "jpg", "best", {})
self.assertTrue(opts.get("skip_download"))
self.assertTrue(opts.get("writethumbnail"))
def test_get_opts_captions_manual_only(self):
opts = get_opts(
"captions", "auto", "vtt", "best", {}, subtitle_language="fr", subtitle_mode="manual_only"
)
self.assertTrue(opts.get("writesubtitles"))
self.assertFalse(opts.get("writeautomaticsub"))
self.assertEqual(opts["subtitleslangs"], ["fr"])
def test_get_opts_captions_auto_only(self):
opts = get_opts(
"captions", "auto", "srt", "best", {}, subtitle_language="de", subtitle_mode="auto_only"
)
self.assertFalse(opts.get("writesubtitles"))
self.assertTrue(opts.get("writeautomaticsub"))
self.assertEqual(opts["subtitleslangs"], ["de-orig", "de"])
def test_get_opts_captions_prefer_auto(self):
opts = get_opts(
"captions", "auto", "srt", "best", {}, subtitle_language="es", subtitle_mode="prefer_auto"
)
self.assertTrue(opts.get("writesubtitles"))
self.assertTrue(opts.get("writeautomaticsub"))
self.assertEqual(opts["subtitleslangs"], ["es-orig", "es"])
def test_get_opts_captions_prefer_manual_default_branch(self):
opts = get_opts(
"captions", "auto", "srt", "best", {}, subtitle_language="it", subtitle_mode="prefer_manual"
)
self.assertEqual(opts["subtitleslangs"], ["it", "it-orig"])
def test_get_opts_captions_txt_maps_to_srt_format(self):
opts = get_opts("captions", "auto", "txt", "best", {})
self.assertEqual(opts["subtitlesformat"], "srt/best")
keys = [p["key"] for p in opts["postprocessors"]]
self.assertIn("FFmpegSubtitlesConvertor", keys)
convertor = next(p for p in opts["postprocessors"] if p["key"] == "FFmpegSubtitlesConvertor")
self.assertEqual(convertor["format"], "srt")
def test_get_opts_captions_srt_guarantees_convertor(self):
opts = get_opts("captions", "auto", "srt", "best", {})
self.assertEqual(opts["subtitlesformat"], "srt/best")
keys = [p["key"] for p in opts["postprocessors"]]
self.assertIn("FFmpegSubtitlesConvertor", keys)
def test_get_opts_captions_vtt_guarantees_convertor(self):
opts = get_opts("captions", "auto", "vtt", "best", {})
self.assertEqual(opts["subtitlesformat"], "vtt/best")
keys = [p["key"] for p in opts["postprocessors"]]
self.assertIn("FFmpegSubtitlesConvertor", keys)
convertor = next(p for p in opts["postprocessors"] if p["key"] == "FFmpegSubtitlesConvertor")
self.assertEqual(convertor["format"], "vtt")
def test_get_opts_captions_ttml_has_no_convertor(self):
opts = get_opts("captions", "auto", "ttml", "best", {})
self.assertEqual(opts["subtitlesformat"], "ttml/best")
keys = [p["key"] for p in opts["postprocessors"]]
self.assertNotIn("FFmpegSubtitlesConvertor", keys)
def test_get_opts_merges_existing_postprocessors(self):
opts = get_opts("audio", "auto", "opus", "best", {"postprocessors": [{"key": "SponsorBlock"}]})
keys = [p["key"] for p in opts["postprocessors"]]
self.assertIn("SponsorBlock", keys)
self.assertIn("FFmpegExtractAudio", keys)
def test_normalize_caption_mode_invalid_defaults(self):
self.assertEqual(_normalize_caption_mode(""), "prefer_manual")
self.assertEqual(_normalize_caption_mode("not_a_mode"), "prefer_manual")
def test_normalize_subtitle_language_empty_defaults_en(self):
self.assertEqual(_normalize_subtitle_language(""), "en")
self.assertEqual(_normalize_subtitle_language(" "), "en")
class MergeYtdlOptionLayersTests(unittest.TestCase):
def test_presets_applied_in_order_then_overrides(self):
presets_config = {
"a": {"x": 1, "y": 1},
"b": {"y": 2, "z": 2},
}
merged = merge_ytdl_option_layers(["a", "b"], {"z": 3, "w": 4}, presets_config)
# b overrides a's y; explicit overrides win over presets.
self.assertEqual(merged, {"x": 1, "y": 2, "z": 3, "w": 4})
def test_no_base_options_included(self):
# The helper only produces the preset/override layer, never base opts.
self.assertEqual(merge_ytdl_option_layers(None, None, {}), {})
def test_unknown_preset_names_ignored(self):
self.assertEqual(merge_ytdl_option_layers(["missing"], {"a": 1}, {}), {"a": 1})
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,948 @@
"""Tests for ``DownloadQueue`` with mocked yt-dlp extraction."""
from __future__ import annotations
import os
import re
import tempfile
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
import time
from ytdl import Download, DownloadInfo, DownloadQueue
@pytest.fixture
def dq_env():
with tempfile.TemporaryDirectory() as tmp:
dl = os.path.join(tmp, "downloads")
st = os.path.join(tmp, "state")
os.makedirs(dl, exist_ok=True)
os.makedirs(st, exist_ok=True)
cfg = MagicMock()
cfg.STATE_DIR = st
cfg.DOWNLOAD_DIR = dl
cfg.AUDIO_DOWNLOAD_DIR = dl
cfg.TEMP_DIR = dl
cfg.MAX_CONCURRENT_DOWNLOADS = "3"
cfg.YTDL_OPTIONS = {}
cfg.YTDL_OPTIONS_PRESETS = {}
cfg.CUSTOM_DIRS = True
cfg.CREATE_CUSTOM_DIRS = True
cfg.CLEAR_COMPLETED_AFTER = "0"
cfg.DELETE_FILE_ON_TRASHCAN = False
cfg.OUTPUT_TEMPLATE = "%(title)s.%(ext)s"
cfg.OUTPUT_TEMPLATE_CHAPTER = "%(title)s.%(ext)s"
cfg.OUTPUT_TEMPLATE_PLAYLIST = ""
cfg.OUTPUT_TEMPLATE_CHANNEL = ""
yield cfg
def test_cancel_add_increments_generation(dq_env):
notifier = MagicMock()
dq = DownloadQueue(dq_env, notifier)
before = dq._add_generation
dq.cancel_add()
assert dq._add_generation == before + 1
def test_download_queue_has_dedicated_executor_sized_from_config(dq_env):
notifier = MagicMock()
dq = DownloadQueue(dq_env, notifier)
assert dq._download_executor is not None
assert dq._download_executor._max_workers == 2 * int(dq_env.MAX_CONCURRENT_DOWNLOADS) + 2
dq.close()
def test_close_cancels_running_downloads_before_shutdown(dq_env):
notifier = MagicMock()
dq = DownloadQueue(dq_env, notifier)
running = MagicMock()
running.started.return_value = True
running.running.return_value = True
idle = MagicMock()
idle.started.return_value = False
idle.running.return_value = False
dq.queue.dict["u-running"] = running
dq.queue.dict["u-idle"] = idle
dq.close()
# The active download's subprocess group is killed; the not-started one is
# left alone. Executor is shut down afterwards.
running.cancel.assert_called_once()
idle.cancel.assert_not_called()
assert dq._download_executor._shutdown
def test_get_returns_tuple_of_lists(dq_env):
notifier = MagicMock()
dq = DownloadQueue(dq_env, notifier)
q, done = dq.get()
assert q == [] and done == []
@pytest.mark.asyncio
async def test_add_single_video_goes_to_pending_when_auto_start_false(dq_env):
notifier = AsyncMock()
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
return {
"_type": "video",
"id": "vid1",
"title": "Test Video",
"url": url,
"webpage_url": url,
}
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract):
result = await dq.add(
"https://example.com/watch?v=1",
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=False,
)
assert result["status"] == "ok"
assert dq.pending.exists("https://example.com/watch?v=1")
@pytest.mark.asyncio
async def test_cancel_removes_from_pending(dq_env):
notifier = AsyncMock()
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
return {
"_type": "video",
"id": "vid1",
"title": "Test Video",
"url": url,
"webpage_url": url,
}
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract):
await dq.add(
"https://example.com/pending",
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=False,
)
url = "https://example.com/pending"
await dq.cancel([url])
assert not dq.pending.exists(url)
notifier.canceled.assert_awaited()
@pytest.mark.asyncio
async def test_cancel_before_start_marks_download_canceled(dq_env):
"""Regression test for the race condition where cancel() arrives after the
download has been placed in the queue and ``__start_download`` has been
scheduled via ``asyncio.create_task`` but has not yet executed. Without the
fix, the pending task would run ``download.start()`` despite the user
cancelling, because its ``download.canceled`` guard was never flipped."""
notifier = AsyncMock()
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
return {
"_type": "video",
"id": "vid1",
"title": "Test Video",
"url": url,
"webpage_url": url,
}
dq = DownloadQueue(dq_env, notifier)
url = "https://example.com/race"
start_mock = AsyncMock()
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract), \
patch.object(DownloadQueue, "_DownloadQueue__start_download", start_mock):
await dq.add(
url,
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=True,
)
assert dq.queue.exists(url)
download = dq.queue.get(url)
assert download.canceled is False
await dq.cancel([url])
assert not dq.queue.exists(url)
assert download.canceled is True
notifier.canceled.assert_awaited_with(url)
@pytest.mark.asyncio
async def test_start_pending_moves_to_queue(dq_env):
notifier = AsyncMock()
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
return {
"_type": "video",
"id": "vid1",
"title": "Test Video",
"url": url,
"webpage_url": url,
}
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract):
await dq.add(
"https://example.com/startme",
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=False,
)
url = "https://example.com/startme"
# Starting will spawn real download — cancel immediately before worker runs much
with patch.object(DownloadQueue, "_DownloadQueue__start_download", AsyncMock()):
await dq.start_pending([url])
assert not dq.pending.exists(url)
@pytest.mark.asyncio
async def test_add_entry_queues_single_video_without_reextracting(dq_env):
notifier = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
entry = {
"_type": "video",
"id": "vid1",
"title": "Test Video",
"url": "https://example.com/watch?v=1",
"webpage_url": "https://example.com/watch?v=1",
"playlist_index": "01",
"playlist_title": "Playlist",
}
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", side_effect=AssertionError("should not re-extract")):
result = await dq.add_entry(
entry,
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=False,
)
assert result["status"] == "ok"
assert dq.pending.exists("https://example.com/watch?v=1")
@pytest.mark.asyncio
async def test_add_entry_duplicate_while_pending_is_skipped_not_clobbered(dq_env):
notifier = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
entry = {
"_type": "video",
"id": "vid1",
"title": "Original Title",
"url": "https://example.com/watch?v=1",
"webpage_url": "https://example.com/watch?v=1",
}
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", side_effect=AssertionError("should not re-extract")):
first = await dq.add_entry(entry, "video", "auto", "any", "best", "", "", 0, auto_start=False)
assert first["status"] == "ok"
assert "msg" not in first
dupe_entry = {**entry, "title": "Different Title"}
second = await dq.add_entry(dupe_entry, "audio", "auto", "mp3", "best", "", "", 0, auto_start=False)
assert second["status"] == "ok"
assert "Already in queue" in second["msg"]
# The original pending download's options must survive untouched.
pending_dl = dq.pending.get("https://example.com/watch?v=1")
assert pending_dl.info.download_type == "video"
assert pending_dl.info.title == "Original Title"
@pytest.mark.asyncio
async def test_add_entry_duplicate_while_queued_is_skipped(dq_env):
notifier = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
entry = {
"_type": "video",
"id": "vid1",
"title": "Test Video",
"url": "https://example.com/watch?v=1",
"webpage_url": "https://example.com/watch?v=1",
}
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", side_effect=AssertionError("should not re-extract")), \
patch.object(DownloadQueue, "_DownloadQueue__start_download", new=AsyncMock()):
first = await dq.add_entry(entry, "video", "auto", "any", "best", "", "", 0, auto_start=True)
assert first["status"] == "ok"
assert dq.queue.exists("https://example.com/watch?v=1")
second = await dq.add_entry(entry, "video", "auto", "any", "best", "", "", 0, auto_start=True)
assert second["status"] == "ok"
assert "Already in queue" in second["msg"]
@pytest.mark.asyncio
async def test_channel_download_uses_output_template_when_channel_template_empty(dq_env):
"""Channel tabs reported as playlists must honor OUTPUT_TEMPLATE when OUTPUT_TEMPLATE_CHANNEL is empty."""
notifier = AsyncMock()
dq_env.OUTPUT_TEMPLATE = "%(channel)s [YT]/%(title)s.%(ext)s"
dq_env.OUTPUT_TEMPLATE_CHANNEL = ""
dq_env.OUTPUT_TEMPLATE_PLAYLIST = ""
channel_id = "UCabcd123"
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
return {
"_type": "playlist",
"id": channel_id,
"channel_id": channel_id,
"channel": "Odin",
"title": "Odin - Videos",
"entries": [
{
"id": "vid1",
"title": "Salvia Plath - Pondering",
"url": "https://example.com/watch?v=1",
"webpage_url": "https://example.com/watch?v=1",
"channel": "Odin",
"upload_date": "20130804",
},
],
}
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract):
result = await dq.add(
"https://www.youtube.com/@odin/videos",
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=False,
)
assert result["status"] == "ok"
url = "https://example.com/watch?v=1"
assert dq.pending.exists(url)
download = dq.pending.get(url)
assert download.output_template.startswith("Odin [YT]/")
assert "Odin - Videos" not in download.output_template
@pytest.mark.asyncio
async def test_playlist_download_not_treated_as_channel(dq_env):
"""Real playlists (id != channel_id) must not be promoted to channel downloads."""
notifier = AsyncMock()
dq_env.OUTPUT_TEMPLATE = "%(channel)s [YT]/%(title)s.%(ext)s"
dq_env.OUTPUT_TEMPLATE_CHANNEL = ""
dq_env.OUTPUT_TEMPLATE_PLAYLIST = "%(playlist_title)s/%(title)s.%(ext)s"
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
return {
"_type": "playlist",
"id": "PLxyz789",
"channel_id": "UCabcd123",
"channel": "Odin",
"title": "My Playlist",
"entries": [
{
"id": "vid1",
"title": "Test Video",
"url": "https://example.com/watch?v=1",
"webpage_url": "https://example.com/watch?v=1",
},
],
}
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract):
result = await dq.add(
"https://www.youtube.com/playlist?list=PLxyz789",
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=False,
)
assert result["status"] == "ok"
url = "https://example.com/watch?v=1"
assert dq.pending.exists(url)
download = dq.pending.get(url)
assert download.output_template.startswith("My Playlist/")
@pytest.mark.asyncio
async def test_add_merges_global_preset_and_override_options(dq_env):
notifier = AsyncMock()
dq_env.YTDL_OPTIONS = {"writesubtitles": False, "cookiefile": "/tmp/global.txt"}
dq_env.YTDL_OPTIONS_PRESETS = {
"Preset A": {"writesubtitles": True, "proxy": "http://preset-a"},
"Preset B": {"writesubtitles": False, "ratelimit": 1000},
}
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
return {
"_type": "video",
"id": "vid2",
"title": "Preset Video",
"url": url,
"webpage_url": url,
}
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract):
result = await dq.add(
"https://example.com/preset",
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=False,
ytdl_options_presets=["Preset A", "Preset B"],
ytdl_options_overrides={"proxy": "http://override", "embed_thumbnail": True},
)
assert result["status"] == "ok"
queued = dq.pending.get("https://example.com/preset")
assert queued.ytdl_opts["cookiefile"] == "/tmp/global.txt"
assert queued.ytdl_opts["writesubtitles"] is False
assert queued.ytdl_opts["ratelimit"] == 1000
assert queued.ytdl_opts["proxy"] == "http://override"
assert queued.ytdl_opts["embed_thumbnail"] is True
@pytest.mark.asyncio
async def test_extract_info_preset_null_download_archive_overrides_global(dq_env):
"""Preset download_archive:null must apply during extract_info (global archive otherwise wins first)."""
dq_env.YTDL_OPTIONS = {"download_archive": "/tmp/archive.txt"}
dq_env.YTDL_OPTIONS_PRESETS = {"NoArchive": {"download_archive": None}}
captured_params: list = []
class FakeYoutubeDL:
def __init__(self, params=None):
captured_params.append(params)
def extract_info(self, url, download=False):
return {
"_type": "video",
"id": "vid-archive",
"title": "Archive Test",
"url": url,
"webpage_url": url,
}
notifier = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
with patch("ytdl.yt_dlp.YoutubeDL", FakeYoutubeDL):
result = await dq.add(
"https://example.com/archive-test",
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=False,
ytdl_options_presets=["NoArchive"],
)
assert result["status"] == "ok"
assert len(captured_params) == 1
extract_params = captured_params[0]
assert extract_params.get("download_archive") is None
assert extract_params["extract_flat"] is True
assert extract_params["noplaylist"] is True
@pytest.mark.asyncio
async def test_extract_info_metube_extract_keys_win_over_preset(dq_env):
"""MeTube's flat-extract settings must not be overridden by presets."""
dq_env.YTDL_OPTIONS = {}
dq_env.YTDL_OPTIONS_PRESETS = {
"TryOverride": {"extract_flat": False, "noplaylist": False},
}
captured_params: list = []
class FakeYoutubeDL:
def __init__(self, params=None):
captured_params.append(params)
def extract_info(self, url, download=False):
return {
"_type": "video",
"id": "vid-flat",
"title": "Flat Test",
"url": url,
"webpage_url": url,
}
notifier = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
with patch("ytdl.yt_dlp.YoutubeDL", FakeYoutubeDL):
result = await dq.add(
"https://example.com/flat-test",
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=False,
ytdl_options_presets=["TryOverride"],
)
assert result["status"] == "ok"
assert captured_params[0]["extract_flat"] is True
assert captured_params[0]["noplaylist"] is True
@pytest.mark.asyncio
async def test_add_sets_clip_bounds_on_download_info(dq_env):
notifier = AsyncMock()
def fake_extract(self, url, ytdl_options_presets=None, ytdl_options_overrides=None):
return {
"_type": "video",
"id": "vid1",
"title": "Test Video",
"url": url,
"webpage_url": url,
}
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_extract):
result = await dq.add(
"https://example.com/clip",
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=False,
clip_start=10.0,
clip_end=99.5,
)
assert result["status"] == "ok"
download = dq.pending.get("https://example.com/clip")
assert download.info.clip_start == 10.0
assert download.info.clip_end == 99.5
def _upcoming_entry(url: str, *, release_timestamp: float | None = None) -> dict:
return {
"_type": "video",
"id": "live1",
"title": "Upcoming Stream",
"url": url,
"webpage_url": url,
"live_status": "is_upcoming",
"release_timestamp": release_timestamp if release_timestamp is not None else time.time() + 3600,
}
@pytest.mark.asyncio
async def test_add_upcoming_stream_scheduled_without_starting(dq_env):
notifier = AsyncMock()
url = "https://example.com/live-upcoming"
start_mock = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__start_download", start_mock):
result = await dq.add_entry(
_upcoming_entry(url),
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=True,
)
assert result["status"] == "ok"
assert dq.queue.exists(url)
download = dq.queue.get(url)
assert download.info.status == "scheduled"
assert download.info.live_status == "is_upcoming"
assert download.info.live_release_timestamp is not None
start_mock.assert_not_called()
assert url in dq._scheduled_probe_at
# The "scheduled to start at ..." message must include a UTC offset
# (a naive datetime's %z would render as an empty string here).
assert re.search(r"[+-]\d{4}$", download.info.error)
@pytest.mark.asyncio
async def test_probe_scheduled_starts_when_live(dq_env):
notifier = AsyncMock()
url = "https://example.com/live-upcoming"
start_mock = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__start_download", start_mock):
await dq.add_entry(
_upcoming_entry(url),
"video",
"auto",
"any",
"best",
"",
"",
0,
auto_start=True,
)
download = dq.queue.get(url)
def fake_probe_extract(self, probe_url, ytdl_options_presets=None, ytdl_options_overrides=None):
assert probe_url == url
return {
"_type": "video",
"id": "live1",
"title": "Live Now",
"url": url,
"webpage_url": url,
"live_status": "is_live",
"formats": [{"format_id": "22"}],
}
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", fake_probe_extract), \
patch.object(DownloadQueue, "_DownloadQueue__start_download", start_mock):
await dq._probe_scheduled_download(download)
assert url not in dq._scheduled_probe_at
assert download.info.live_status == "is_live"
assert download.info.status == "pending"
start_mock.assert_called_once_with(download)
@pytest.mark.asyncio
async def test_import_scheduled_re_registers_monitor(dq_env):
notifier = AsyncMock()
url = "https://example.com/live-restart"
release = time.time() + 7200
info = DownloadInfo(
id="live1",
title="Upcoming Stream",
url=url,
quality="best",
download_type="video",
codec="auto",
format="any",
folder="",
custom_name_prefix="",
error=None,
entry=None,
playlist_item_limit=0,
split_by_chapters=False,
chapter_template="",
live_status="is_upcoming",
live_release_timestamp=release,
)
info.status = "scheduled"
dq = DownloadQueue(dq_env, notifier)
start_mock = AsyncMock()
with patch.object(DownloadQueue, "_DownloadQueue__start_download", start_mock):
await dq._DownloadQueue__add_download(info, True)
assert dq.queue.exists(url)
assert dq.queue.get(url).info.status == "scheduled"
assert url in dq._scheduled_probe_at
start_mock.assert_not_called()
@pytest.mark.asyncio
async def test_probe_transient_error_retries_without_failing(dq_env):
"""A single probe failure must not abandon the scheduled stream."""
import ytdl
notifier = AsyncMock()
url = "https://example.com/live-transient"
start_mock = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__start_download", start_mock):
await dq.add_entry(
_upcoming_entry(url),
"video", "auto", "any", "best", "", "", 0,
auto_start=True,
)
download = dq.queue.get(url)
def boom(self, *args, **kwargs):
raise ytdl.yt_dlp.utils.YoutubeDLError("temporary network glitch")
before = time.time()
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", boom):
await dq._probe_scheduled_download(download)
# Still scheduled, still monitored, probe rescheduled into the future.
assert download.info.status == "scheduled"
assert url in dq._scheduled_probe_at
assert dq._scheduled_probe_at[url] >= before
assert dq._scheduled_probe_failures[url] == 1
notifier.completed.assert_not_called()
@pytest.mark.asyncio
async def test_probe_gives_up_after_max_failures(dq_env):
import ytdl
notifier = AsyncMock()
url = "https://example.com/live-dead"
start_mock = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__start_download", start_mock):
await dq.add_entry(
_upcoming_entry(url),
"video", "auto", "any", "best", "", "", 0,
auto_start=True,
)
download = dq.queue.get(url)
def boom(self, *args, **kwargs):
raise ytdl.yt_dlp.utils.YoutubeDLError("stream was deleted")
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", boom):
for _ in range(ytdl._LIVE_PROBE_MAX_FAILURES):
await dq._probe_scheduled_download(download)
assert url not in dq._scheduled_probe_at
assert not dq.queue.exists(url)
assert dq.done.exists(url)
assert download.info.status == "error"
notifier.completed.assert_awaited()
@pytest.mark.asyncio
async def test_probe_recovers_after_transient_then_starts(dq_env):
"""A transient failure followed by a successful live probe should start the download."""
import ytdl
notifier = AsyncMock()
url = "https://example.com/live-recover"
start_mock = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
with patch.object(DownloadQueue, "_DownloadQueue__start_download", start_mock):
await dq.add_entry(
_upcoming_entry(url),
"video", "auto", "any", "best", "", "", 0,
auto_start=True,
)
download = dq.queue.get(url)
# The scheduling placeholder error is set on add.
assert download.info.error
def boom(self, *args, **kwargs):
raise ytdl.yt_dlp.utils.YoutubeDLError("temporary glitch")
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", boom):
await dq._probe_scheduled_download(download)
assert dq._scheduled_probe_failures[url] == 1
def live_now(self, *args, **kwargs):
return {
"_type": "video", "id": "live1", "title": "Live Now",
"url": url, "webpage_url": url, "live_status": "is_live",
"formats": [{"format_id": "22"}],
}
with patch.object(DownloadQueue, "_DownloadQueue__extract_info", live_now), \
patch.object(DownloadQueue, "_DownloadQueue__start_download", start_mock):
await dq._probe_scheduled_download(download)
assert url not in dq._scheduled_probe_at
assert url not in dq._scheduled_probe_failures
assert download.info.status == "pending"
# Placeholder error/msg cleared now that a real download is starting.
assert download.info.error is None
assert download.info.msg is None
start_mock.assert_called_once_with(download)
def test_seconds_until_next_probe_none_when_empty(dq_env):
notifier = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
assert dq._seconds_until_next_probe() is None
def test_calc_download_path_allows_subfolder(dq_env):
notifier = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
path, err = dq._DownloadQueue__calc_download_path("video", "sub/dir")
assert err is None
assert os.path.realpath(path) == os.path.join(os.path.realpath(dq_env.DOWNLOAD_DIR), "sub", "dir")
def test_calc_download_path_rejects_sibling_prefix_escape(dq_env):
"""A folder resolving to a sibling sharing a name prefix must be rejected.
Regression test: ``startswith`` would have accepted ``../downloads-secret``
when the base directory is ``.../downloads``.
"""
notifier = AsyncMock()
base = os.path.realpath(dq_env.DOWNLOAD_DIR)
sibling = base + "-secret"
os.makedirs(sibling, exist_ok=True)
dq = DownloadQueue(dq_env, notifier)
escape_folder = os.path.join("..", os.path.basename(sibling), "x")
path, err = dq._DownloadQueue__calc_download_path("video", escape_folder)
assert path is None
assert err is not None and err["status"] == "error"
def test_calc_download_path_rejects_parent_escape(dq_env):
notifier = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
path, err = dq._DownloadQueue__calc_download_path("video", "../../etc")
assert path is None
assert err is not None and err["status"] == "error"
def test_download_info_to_public_dict_excludes_server_only_fields():
info = DownloadInfo(
id="vid1",
title="Test Video",
url="https://example.com/watch?v=1",
quality="best",
download_type="video",
codec="auto",
format="any",
folder="",
custom_name_prefix="",
error=None,
entry={"id": "vid1", "huge": "x" * 100000},
playlist_item_limit=0,
split_by_chapters=False,
chapter_template="",
)
info.subtitle_files = [{"filename": "a.srt", "size": 10}]
public = info.to_public_dict()
assert "entry" not in public
assert "subtitle_files" not in public
# Client-facing fields are still present.
assert public["url"] == "https://example.com/watch?v=1"
assert public["title"] == "Test Video"
assert public["status"] == "pending"
def _make_download(dq_env, *, download_type="video", status="downloading", filename=None):
info = DownloadInfo(
id="id1",
title="t",
url="http://example.com/v",
quality="best",
download_type=download_type,
codec="auto",
format="any",
folder="",
custom_name_prefix="",
error=None,
entry=None,
playlist_item_limit=0,
split_by_chapters=False,
chapter_template="",
)
info.status = status
info.filename = filename
info.size = 123 if filename else None
return Download(
dq_env.DOWNLOAD_DIR, dq_env.TEMP_DIR, "%(title)s.%(ext)s", "%(title)s.%(ext)s", "best", "any", {}, info
)
@pytest.mark.asyncio
async def test_post_download_cleanup_clears_filename_on_error(dq_env):
notifier = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
download = _make_download(dq_env, status="downloading", filename="../tmp/partial.mp4")
dq.queue.put(download)
dq._post_download_cleanup(download)
assert download.info.status == "error"
assert download.info.filename is None
assert download.info.size is None
@pytest.mark.asyncio
async def test_post_download_cleanup_keeps_captured_subtitles_on_error(dq_env):
notifier = AsyncMock()
dq = DownloadQueue(dq_env, notifier)
download = _make_download(dq_env, download_type="captions", status="downloading", filename="en.srt")
download.info.subtitle_files = [{"filename": "en.srt", "size": 42}]
dq.queue.put(download)
dq._post_download_cleanup(download)
assert download.info.status == "error"
assert download.info.filename == "en.srt"
@pytest.mark.asyncio
async def test_clear_skips_deletion_outside_download_directory(dq_env):
notifier = AsyncMock()
dq_env.DELETE_FILE_ON_TRASHCAN = True
dq = DownloadQueue(dq_env, notifier)
outside_dir = tempfile.mkdtemp()
outside_file = os.path.join(outside_dir, "outside.txt")
with open(outside_file, "w") as f:
f.write("do not delete me")
# A crafted/legacy relative filename that escapes DOWNLOAD_DIR via '..'.
escaping_filename = os.path.relpath(outside_file, dq_env.DOWNLOAD_DIR)
download = _make_download(dq_env, status="finished", filename=escaping_filename)
dq.done.put(download)
await dq.clear([download.info.url])
assert os.path.exists(outside_file)
assert not dq.done.exists(download.info.url)

View file

@ -0,0 +1,315 @@
"""Tests for pure helpers in ``main`` (legacy API migration, logging, JSON serializer)."""
from __future__ import annotations
import json
import logging
import unittest
import main
class MigrateLegacyRequestTests(unittest.TestCase):
def test_already_new_schema_unchanged(self):
post = {"download_type": "video", "codec": "h264", "format": "mp4", "quality": "1080"}
before = post.copy()
self.assertIs(main._migrate_legacy_request(post), post)
self.assertEqual(post, before)
def test_legacy_audio_m4a(self):
post = {"format": "m4a", "quality": "best"}
main._migrate_legacy_request(post)
self.assertEqual(post["download_type"], "audio")
self.assertEqual(post["codec"], "auto")
self.assertEqual(post["format"], "m4a")
def test_legacy_thumbnail(self):
post = {"format": "thumbnail", "quality": "best"}
main._migrate_legacy_request(post)
self.assertEqual(post["download_type"], "thumbnail")
self.assertEqual(post["format"], "jpg")
self.assertEqual(post["quality"], "best")
def test_legacy_captions_with_subtitle_format(self):
post = {"format": "captions", "subtitle_format": "vtt", "quality": "best"}
main._migrate_legacy_request(post)
self.assertEqual(post["download_type"], "captions")
self.assertEqual(post["format"], "vtt")
def test_legacy_video_best_ios(self):
post = {"format": "any", "quality": "best_ios", "video_codec": "auto"}
main._migrate_legacy_request(post)
self.assertEqual(post["download_type"], "video")
self.assertEqual(post["format"], "ios")
self.assertEqual(post["quality"], "best")
def test_legacy_video_quality_audio_maps_to_m4a(self):
post = {"format": "mp4", "quality": "audio", "video_codec": "h264"}
main._migrate_legacy_request(post)
self.assertEqual(post["download_type"], "audio")
self.assertEqual(post["format"], "m4a")
self.assertEqual(post["quality"], "best")
def test_legacy_video_default(self):
post = {"format": "mp4", "quality": "1080", "video_codec": "h265"}
main._migrate_legacy_request(post)
self.assertEqual(post["download_type"], "video")
self.assertEqual(post["codec"], "h265")
self.assertEqual(post["format"], "mp4")
self.assertEqual(post["quality"], "1080")
class ParseLogLevelTests(unittest.TestCase):
def test_valid_levels(self):
self.assertEqual(main.parseLogLevel("INFO"), logging.INFO)
self.assertEqual(main.parseLogLevel("debug"), logging.DEBUG)
def test_invalid_returns_none(self):
self.assertIsNone(main.parseLogLevel("not_a_level"))
self.assertIsNone(main.parseLogLevel(123))
class ObjectSerializerTests(unittest.TestCase):
def test_dict_like_object(self):
class Obj:
def __init__(self):
self.a = 1
ser = main.ObjectSerializer()
self.assertEqual(json.loads(ser.encode(Obj())), {"a": 1})
def test_generator_becomes_list(self):
ser = main.ObjectSerializer()
def gen():
yield 1
yield 2
self.assertEqual(json.loads(ser.encode(gen())), [1, 2])
def test_string_not_split_to_chars(self):
ser = main.ObjectSerializer()
self.assertEqual(json.loads(ser.encode("hello")), "hello")
class FrontendSafeTests(unittest.TestCase):
def test_only_expected_keys(self):
safe = main.config.frontend_safe()
for key in main.Config._FRONTEND_KEYS:
self.assertIn(key, safe)
self.assertNotIn("YTDL_OPTIONS", safe)
self.assertNotIn("DOWNLOAD_DIR", safe)
self.assertIn("ALLOW_YTDL_OPTIONS_OVERRIDES", safe)
class ParseYtdlOverridesTests(unittest.TestCase):
def test_empty_override_string_returns_empty_dict(self):
self.assertEqual(main._parse_ytdl_options_overrides("", enabled=False), {})
def test_rejects_non_object_json(self):
with self.assertRaises(main.web.HTTPBadRequest):
main._parse_ytdl_options_overrides('["bad"]', enabled=True)
def test_rejects_non_empty_overrides_when_disabled(self):
with self.assertRaises(main.web.HTTPBadRequest):
main._parse_ytdl_options_overrides('{"exec": "rm -rf /"}', enabled=False)
def test_allows_any_keys_when_enabled(self):
self.assertEqual(
main._parse_ytdl_options_overrides('{"exec": "rm -rf /"}', enabled=True),
{"exec": "rm -rf /"},
)
class ParseDownloadOptionsTests(unittest.TestCase):
def test_accepts_known_preset_and_overrides(self):
previous = dict(main.config.YTDL_OPTIONS_PRESETS)
previous_allow = main.config.ALLOW_YTDL_OPTIONS_OVERRIDES
main.config.YTDL_OPTIONS_PRESETS = {"With subtitles": {"writesubtitles": True}}
main.config.ALLOW_YTDL_OPTIONS_OVERRIDES = True
try:
parsed = main.parse_download_options({
"url": "https://example.com/v",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
"ytdl_options_preset": "With subtitles",
"ytdl_options_overrides": '{"writesubtitles": true}',
})
finally:
main.config.YTDL_OPTIONS_PRESETS = previous
main.config.ALLOW_YTDL_OPTIONS_OVERRIDES = previous_allow
self.assertEqual(parsed["ytdl_options_presets"], ["With subtitles"])
self.assertEqual(parsed["ytdl_options_overrides"], {"writesubtitles": True})
def test_accepts_multiple_presets_in_order(self):
previous = dict(main.config.YTDL_OPTIONS_PRESETS)
main.config.YTDL_OPTIONS_PRESETS = {
"A": {"writesubtitles": True},
"B": {"writesubtitles": False},
}
try:
parsed = main.parse_download_options({
"url": "https://example.com/v",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
"ytdl_options_presets": ["A", "B"],
})
finally:
main.config.YTDL_OPTIONS_PRESETS = previous
self.assertEqual(parsed["ytdl_options_presets"], ["A", "B"])
def test_legacy_singular_preset_string_normalized_to_list(self):
previous = dict(main.config.YTDL_OPTIONS_PRESETS)
main.config.YTDL_OPTIONS_PRESETS = {"Solo": {}}
try:
parsed = main.parse_download_options({
"url": "https://example.com/v",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
"ytdl_options_preset": "Solo",
})
finally:
main.config.YTDL_OPTIONS_PRESETS = previous
self.assertEqual(parsed["ytdl_options_presets"], ["Solo"])
def test_rejects_unknown_preset(self):
with self.assertRaises(main.web.HTTPBadRequest):
main.parse_download_options({
"url": "https://example.com/v",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
"ytdl_options_presets": ["Missing preset"],
})
def test_rejects_unknown_preset_in_list(self):
previous = dict(main.config.YTDL_OPTIONS_PRESETS)
main.config.YTDL_OPTIONS_PRESETS = {"Known": {}}
try:
with self.assertRaises(main.web.HTTPBadRequest):
main.parse_download_options({
"url": "https://example.com/v",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
"ytdl_options_presets": ["Known", "Nope"],
})
finally:
main.config.YTDL_OPTIONS_PRESETS = previous
def test_clip_start_end_seconds_and_clock(self):
parsed = main.parse_download_options({
"url": "https://example.com/watch?v=1",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
"clip_start": "2:26",
"clip_end": "3:24",
})
self.assertEqual(parsed["clip_start"], 146.0)
self.assertEqual(parsed["clip_end"], 204.0)
def test_clip_url_t_param_strips_query_and_sets_start(self):
parsed = main.parse_download_options({
"url": "https://www.youtube.com/watch?v=1&t=855s",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
})
self.assertEqual(parsed["url"], "https://www.youtube.com/watch?v=1")
self.assertEqual(parsed["clip_start"], 855.0)
self.assertIsNone(parsed["clip_end"])
def test_clip_explicit_start_wins_over_url_t(self):
parsed = main.parse_download_options({
"url": "https://www.youtube.com/watch?v=1&t=100",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
"clip_start": "50",
})
self.assertEqual(parsed["url"], "https://www.youtube.com/watch?v=1")
self.assertEqual(parsed["clip_start"], 50.0)
self.assertIsNone(parsed["clip_end"])
def test_clip_end_only_sets_start_zero_and_strips_url_t(self):
parsed = main.parse_download_options({
"url": "https://www.youtube.com/watch?v=1&t=999",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
"clip_end": "60",
})
self.assertEqual(parsed["url"], "https://www.youtube.com/watch?v=1")
self.assertEqual(parsed["clip_start"], 0.0)
self.assertEqual(parsed["clip_end"], 60.0)
def test_clip_url_t_param_ignored_on_non_youtube_host(self):
# 't' is a generic query param name; only rewrite it on YouTube hosts
# so an unrelated site's URL isn't silently mutated with a bogus clip.
parsed = main.parse_download_options({
"url": "https://example.com/watch?v=1&t=855s",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
})
self.assertEqual(parsed["url"], "https://example.com/watch?v=1&t=855s")
self.assertIsNone(parsed["clip_start"])
self.assertIsNone(parsed["clip_end"])
def test_extract_t_query_youtu_be_short_host(self):
cleaned, start = main._extract_t_query_from_url("https://youtu.be/abc123?t=90")
self.assertEqual(cleaned, "https://youtu.be/abc123")
self.assertEqual(start, 90.0)
def test_clip_rejects_end_before_start(self):
with self.assertRaises(main.web.HTTPBadRequest):
main.parse_download_options({
"url": "https://example.com/watch?v=1",
"download_type": "video",
"codec": "auto",
"format": "any",
"quality": "best",
"clip_start": "100",
"clip_end": "50",
})
def test_clip_rejected_for_captions(self):
with self.assertRaises(main.web.HTTPBadRequest):
main.parse_download_options({
"url": "https://example.com/watch?v=1",
"download_type": "captions",
"codec": "auto",
"format": "srt",
"quality": "best",
"clip_start": "1",
})
class GetCustomDirsTests(unittest.TestCase):
def test_works_without_a_running_event_loop(self):
# get_custom_dirs() used to time its cache via
# asyncio.get_running_loop().time(), which raises RuntimeError outside
# a running loop (e.g. when called from a plain executor thread). It
# must work from a synchronous context too.
result = main.get_custom_dirs()
self.assertIn("download_dir", result)
self.assertIn("audio_download_dir", result)
self.assertIn("", result["download_dir"])
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,29 @@
"""Tests for nightly yt-dlp update scheduling helpers."""
from __future__ import annotations
import unittest
from datetime import datetime
from main import seconds_until_next_daily_time
class NightlyUpdateTests(unittest.TestCase):
def test_seconds_until_later_today(self):
now = datetime(2026, 6, 4, 10, 0, 0)
delay = seconds_until_next_daily_time("15:30", now)
self.assertEqual(delay, 5 * 3600 + 30 * 60)
def test_seconds_until_wraps_to_next_day(self):
now = datetime(2026, 6, 4, 18, 0, 0)
delay = seconds_until_next_daily_time("04:00", now)
self.assertEqual(delay, 10 * 3600)
def test_seconds_until_same_minute_is_next_day(self):
now = datetime(2026, 6, 4, 4, 0, 30)
delay = seconds_until_next_daily_time("04:00", now)
self.assertAlmostEqual(delay, 24 * 3600 - 30, delta=1)
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,291 @@
"""Integration tests for ``PersistentQueue`` using the JSON state store."""
from __future__ import annotations
import json
import os
import shelve
import sys
import tempfile
import types
import unittest
from unittest.mock import patch
fake_yt_dlp = types.ModuleType("yt_dlp")
fake_networking = types.ModuleType("yt_dlp.networking")
fake_impersonate = types.ModuleType("yt_dlp.networking.impersonate")
fake_utils = types.ModuleType("yt_dlp.utils")
class _ImpersonateTarget:
@staticmethod
def from_str(value):
return value
fake_impersonate.ImpersonateTarget = _ImpersonateTarget
fake_networking.impersonate = fake_impersonate
fake_utils.STR_FORMAT_RE_TMPL = r"(?P<prefix>)%\((?P<has_key>{})\)(?P<format>[-0-9.]*{})"
fake_utils.STR_FORMAT_TYPES = "diouxXeEfFgGcrsa"
fake_yt_dlp.networking = fake_networking
fake_yt_dlp.utils = fake_utils
sys.modules.setdefault("yt_dlp", fake_yt_dlp)
sys.modules.setdefault("yt_dlp.networking", fake_networking)
sys.modules.setdefault("yt_dlp.networking.impersonate", fake_impersonate)
sys.modules.setdefault("yt_dlp.utils", fake_utils)
from ytdl import DownloadInfo, PersistentQueue
class _FakeDownload:
__slots__ = ("info",)
def __init__(self, info: DownloadInfo):
self.info = info
def _make_info(url: str = "https://example.com/v") -> DownloadInfo:
return DownloadInfo(
id="id1",
title="Title",
url=url,
quality="best",
download_type="video",
codec="auto",
format="any",
folder="",
custom_name_prefix="",
error=None,
entry=None,
playlist_item_limit=0,
split_by_chapters=False,
chapter_template="",
)
def _create_legacy_shelf(path: str, *infos: DownloadInfo) -> None:
with shelve.open(path, "c") as shelf:
for info in infos:
shelf[info.url] = info
class PersistentQueueTests(unittest.TestCase):
def test_put_get_delete_roundtrip(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue")
pq = PersistentQueue("queue", path)
dl = _FakeDownload(_make_info("http://a.example"))
pq.put(dl)
self.assertTrue(os.path.exists(path + ".json"))
self.assertTrue(pq.exists("http://a.example"))
self.assertFalse(pq.empty())
got = pq.get("http://a.example")
self.assertEqual(got.info.url, "http://a.example")
pq.delete("http://a.example")
self.assertFalse(pq.exists("http://a.example"))
def test_saved_items_sorted_by_timestamp(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue")
pq = PersistentQueue("queue", path)
a = _FakeDownload(_make_info("http://first.example"))
b = _FakeDownload(_make_info("http://second.example"))
a.info.timestamp = 100
b.info.timestamp = 200
pq.put(a)
pq.put(b)
keys = [k for k, _ in pq.saved_items()]
self.assertEqual(keys, ["http://first.example", "http://second.example"])
def test_load_restores_from_json(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue")
pq1 = PersistentQueue("queue", path)
pq1.put(_FakeDownload(_make_info("http://load.example")))
pq2 = PersistentQueue("queue", path)
pq2.load()
self.assertTrue(pq2.exists("http://load.example"))
def test_load_imports_legacy_shelve(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue")
_create_legacy_shelf(path, _make_info("http://legacy.example"))
pq = PersistentQueue("queue", path)
pq.load()
self.assertTrue(pq.exists("http://legacy.example"))
self.assertTrue(os.path.exists(path + ".json"))
def test_queue_persists_only_compact_entry_subset(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue")
pq = PersistentQueue("queue", path)
info = _make_info("http://entry.example")
info.entry = {
"playlist_index": "01",
"playlist_title": "Playlist",
"channel_index": "02",
"channel_title": "Channel",
"formats": [{"id": "huge"}],
"description": "very large payload",
}
pq.put(_FakeDownload(info))
with open(path + ".json", encoding="utf-8") as f:
payload = json.load(f)
record = payload["items"][0]["info"]
self.assertEqual(
record["entry"],
{
"playlist_index": "01",
"playlist_title": "Playlist",
"channel_index": "02",
"channel_title": "Channel",
},
)
self.assertNotIn("formats", record["entry"])
self.assertNotIn("description", record["entry"])
def test_completed_queue_does_not_persist_entry_or_transient_progress(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "completed")
pq = PersistentQueue("completed", path)
info = _make_info("http://done.example")
info.status = "finished"
info.percent = 88
info.speed = 123
info.eta = 9
info.entry = {
"playlist_index": "01",
"playlist_title": "Playlist",
"formats": [{"id": "huge"}],
}
info.filename = "done.mp4"
pq.put(_FakeDownload(info))
with open(path + ".json", encoding="utf-8") as f:
payload = json.load(f)
record = payload["items"][0]["info"]
self.assertNotIn("entry", record)
self.assertNotIn("percent", record)
self.assertNotIn("speed", record)
self.assertNotIn("eta", record)
self.assertEqual(record["filename"], "done.mp4")
def test_invalid_json_is_quarantined_and_legacy_is_imported(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue")
_create_legacy_shelf(path, _make_info("http://legacy.example"))
with open(path + ".json", "w", encoding="utf-8") as f:
f.write("{not valid json")
pq = PersistentQueue("queue", path)
pq.load()
self.assertTrue(pq.exists("http://legacy.example"))
self.assertTrue(
any(name.startswith("queue.json.invalid.") for name in os.listdir(tmp))
)
def test_loading_old_json_rewrites_to_compact_format(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue")
with open(path + ".json", "w", encoding="utf-8") as f:
json.dump(
{
"schema_version": 1,
"kind": "persistent_queue:queue",
"items": [
{
"key": "http://legacy-json.example",
"info": {
"id": "id1",
"title": "Title",
"url": "http://legacy-json.example",
"quality": "best",
"download_type": "video",
"codec": "auto",
"format": "any",
"folder": "",
"custom_name_prefix": "",
"playlist_item_limit": 0,
"split_by_chapters": False,
"chapter_template": "",
"subtitle_language": "en",
"subtitle_mode": "prefer_manual",
"status": "pending",
"timestamp": 1,
"entry": {
"playlist_index": "01",
"playlist_title": "Playlist",
"formats": [{"id": "huge"}],
},
"percent": 15,
"speed": 20,
"eta": 30,
},
}
],
},
f,
)
pq = PersistentQueue("queue", path)
pq.load()
with open(path + ".json", encoding="utf-8") as f:
payload = json.load(f)
record = payload["items"][0]["info"]
self.assertEqual(payload["schema_version"], 2)
self.assertEqual(record["entry"], {"playlist_index": "01", "playlist_title": "Playlist"})
self.assertNotIn("percent", record)
self.assertNotIn("speed", record)
self.assertNotIn("eta", record)
def test_put_rollbacks_in_memory_queue_when_state_write_fails(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue")
pq = PersistentQueue("queue", path)
dl = _FakeDownload(_make_info("http://rollback.example"))
self.assertFalse(pq.exists("http://rollback.example"))
orig_save = __import__("state_store").AtomicJsonStore.save
def bad_save(store, data):
if store.path == path + ".json":
raise OSError("simulated shelf failure")
return orig_save(store, data)
with patch("ytdl.AtomicJsonStore.save", bad_save):
with self.assertRaises(OSError):
pq.put(dl)
self.assertFalse(pq.exists("http://rollback.example"))
def test_put_rollbacks_to_previous_download_when_replace_fails(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue")
pq = PersistentQueue("queue", path)
first = _FakeDownload(_make_info("http://same.example"))
second = _FakeDownload(_make_info("http://same.example"))
second.info.title = "Replaced title"
pq.put(first)
orig_save = __import__("state_store").AtomicJsonStore.save
def bad_save(store, data):
if store.path == path + ".json":
raise OSError("simulated shelf failure")
return orig_save(store, data)
with patch("ytdl.AtomicJsonStore.save", bad_save):
with self.assertRaises(OSError):
pq.put(second)
self.assertEqual(pq.get("http://same.example").info.title, "Title")
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,183 @@
from __future__ import annotations
import os
import tempfile
import unittest
from datetime import datetime
from unittest.mock import patch
from state_store import AtomicJsonStore, from_json_compatible, to_json_compatible
class StateStoreTests(unittest.TestCase):
def test_save_and_load_roundtrip(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue.json")
store = AtomicJsonStore(path, kind="persistent_queue:queue")
store.save({"items": [{"key": "a", "info": {"title": "hello"}}]})
payload = store.load()
self.assertEqual(payload["kind"], "persistent_queue:queue")
self.assertEqual(payload["schema_version"], 2)
self.assertEqual(payload["items"][0]["info"]["title"], "hello")
def test_save_falls_back_to_direct_write_when_mkstemp_fails(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue.json")
store = AtomicJsonStore(path, kind="persistent_queue:queue")
with self.assertLogs("state_store", level="WARNING") as logs:
with patch(
"state_store.tempfile.mkstemp",
side_effect=PermissionError(1, "Operation not permitted"),
):
store.save({"items": [{"key": "a"}]})
self.assertTrue(os.path.exists(path))
self.assertTrue(any(path in message for message in logs.output))
# Fallback keeps owner-only permissions, matching the atomic path.
self.assertEqual(os.stat(path).st_mode & 0o777, 0o600)
payload = store.load()
self.assertEqual(payload["items"], [{"key": "a"}])
def test_fallback_tightens_permissions_on_existing_file(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue.json")
with open(path, "w", encoding="utf-8") as f:
f.write("{}")
os.chmod(path, 0o644)
store = AtomicJsonStore(path, kind="persistent_queue:queue")
with patch(
"state_store.tempfile.mkstemp",
side_effect=PermissionError(1, "Operation not permitted"),
):
store.save({"items": [{"key": "a"}]})
self.assertEqual(os.stat(path).st_mode & 0o777, 0o600)
self.assertEqual(store.load()["items"], [{"key": "a"}])
def test_save_falls_back_to_direct_write_when_replace_fails(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue.json")
store = AtomicJsonStore(path, kind="persistent_queue:queue")
with patch(
"state_store.os.replace",
side_effect=PermissionError(1, "Operation not permitted"),
):
store.save({"items": [{"key": "a"}]})
self.assertTrue(os.path.exists(path))
payload = store.load()
self.assertEqual(payload["items"], [{"key": "a"}])
self.assertEqual([], [name for name in os.listdir(tmp) if name.endswith(".tmp")])
def test_save_reraises_when_atomic_and_direct_write_fail(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue.json")
store = AtomicJsonStore(path, kind="persistent_queue:queue")
with patch(
"state_store.tempfile.mkstemp",
side_effect=PermissionError(1, "Operation not permitted"),
):
with patch(
"state_store.os.open",
side_effect=PermissionError(13, "Permission denied"),
):
with self.assertRaises(PermissionError) as ctx:
store.save({"items": [{"key": "a"}]})
self.assertEqual(ctx.exception.errno, 13)
self.assertFalse(os.path.exists(path))
def test_unsupported_fsync_keeps_atomic_path(self):
# fsync being unsupported (EINVAL/ENOSYS) must not by itself trigger the
# direct-write fallback; the atomic temp-file + rename path still runs.
import errno as _errno
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue.json")
store = AtomicJsonStore(path, kind="persistent_queue:queue")
with patch(
"state_store.os.fsync",
side_effect=OSError(_errno.EINVAL, "Invalid argument"),
):
with self.assertNoLogs("state_store", level="WARNING"):
store.save({"items": [{"key": "a"}]})
self.assertEqual(store.load()["items"], [{"key": "a"}])
self.assertEqual([], [name for name in os.listdir(tmp) if name.endswith(".tmp")])
def test_save_reraises_and_preserves_state_on_non_atomic_errno(self):
# A storage failure such as ENOSPC is not an "atomic unavailable"
# signal, so it must surface instead of falling back to a direct write
# that would truncate the existing good state file.
import errno as _errno
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue.json")
store = AtomicJsonStore(path, kind="persistent_queue:queue")
store.save({"items": [{"key": "good"}]})
with patch(
"state_store.tempfile.mkstemp",
side_effect=OSError(_errno.ENOSPC, "No space left on device"),
):
with self.assertRaises(OSError) as ctx:
store.save({"items": [{"key": "new"}]})
self.assertEqual(ctx.exception.errno, _errno.ENOSPC)
# Existing state is untouched.
self.assertEqual(store.load()["items"], [{"key": "good"}])
def test_serialization_failure_preserves_existing_state(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue.json")
store = AtomicJsonStore(path, kind="persistent_queue:queue")
store.save({"items": [{"key": "good"}]})
# Even on the fallback path, a non-serializable payload must raise
# before the existing good state file is touched.
with patch(
"state_store.tempfile.mkstemp",
side_effect=PermissionError(1, "Operation not permitted"),
):
with self.assertRaises(TypeError):
store.save({"items": object()})
self.assertEqual(store.load()["items"], [{"key": "good"}])
def test_invalid_file_is_quarantined(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "queue.json")
with open(path, "w", encoding="utf-8") as f:
f.write("{broken")
store = AtomicJsonStore(path, kind="persistent_queue:queue")
payload = store.load()
self.assertIsNone(payload)
self.assertTrue(
any(name.startswith("queue.json.invalid.") for name in os.listdir(tmp))
)
def test_json_compat_helpers_roundtrip_bytes_and_datetime(self):
raw = {
"payload": b"abc",
"timestamp": datetime(2024, 1, 2, 3, 4, 5),
"items": (1, 2, 3),
}
restored = from_json_compatible(to_json_compatible(raw))
self.assertEqual(restored["payload"], b"abc")
self.assertEqual(restored["timestamp"], datetime(2024, 1, 2, 3, 4, 5))
self.assertEqual(restored["items"], [1, 2, 3])
if __name__ == "__main__":
unittest.main()

File diff suppressed because it is too large Load diff

102
app/tests/test_url_guard.py Normal file
View file

@ -0,0 +1,102 @@
"""Tests for the SSRF URL guard (``url_guard.validate_url``)."""
from __future__ import annotations
import socket
import unittest
from unittest import mock
from url_guard import validate_url
def _addrinfo(*addrs, family=socket.AF_INET):
return [(family, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (addr, 0)) for addr in addrs]
class NonUrlInputTests(unittest.TestCase):
"""Bare IDs and yt-dlp search/extractor prefixes must pass untouched."""
def test_bare_video_id_allowed(self):
self.assertIsNone(validate_url("dQw4w9WgXcQ"))
def test_ytsearch_prefix_allowed(self):
self.assertIsNone(validate_url("ytsearch:some song"))
def test_empty_string_allowed(self):
self.assertIsNone(validate_url(""))
def test_non_string_rejected(self):
self.assertIsNotNone(validate_url(None))
class SchemeTests(unittest.TestCase):
def test_file_scheme_blocked(self):
self.assertIsNotNone(validate_url("file:///etc/passwd"))
def test_ftp_scheme_blocked(self):
self.assertIsNotNone(validate_url("ftp://example.com/x"))
def test_data_scheme_blocked(self):
self.assertIsNotNone(validate_url("data://text/plain;base64,AAAA"))
class HostnameBlocklistTests(unittest.TestCase):
def test_localhost_blocked_without_lookup(self):
with mock.patch("url_guard.socket.getaddrinfo") as gai:
self.assertIsNotNone(validate_url("http://localhost:8080/x"))
gai.assert_not_called()
def test_localhost_subdomain_blocked(self):
self.assertIsNotNone(validate_url("http://foo.localhost/x"))
def test_gcp_metadata_name_blocked(self):
self.assertIsNotNone(validate_url("http://metadata.google.internal/x"))
class AddressResolutionTests(unittest.TestCase):
def _validate_with_addrs(self, url, *addrs, family=socket.AF_INET):
with mock.patch("url_guard.socket.getaddrinfo", return_value=_addrinfo(*addrs, family=family)):
return validate_url(url)
def test_public_https_allowed(self):
self.assertIsNone(self._validate_with_addrs("https://youtube.com/watch?v=x", "142.250.1.1"))
def test_public_http_allowed(self):
self.assertIsNone(self._validate_with_addrs("http://example.com/x", "93.184.216.34"))
def test_link_local_metadata_blocked(self):
self.assertIsNotNone(self._validate_with_addrs("http://metadata/x", "169.254.169.254"))
def test_loopback_ipv4_blocked(self):
self.assertIsNotNone(self._validate_with_addrs("http://127.0.0.1/x", "127.0.0.1"))
def test_private_rfc1918_blocked(self):
self.assertIsNotNone(self._validate_with_addrs("http://intranet/x", "10.0.0.5"))
def test_decimal_ip_form_blocked(self):
# 2852039166 == 169.254.169.254; the OS resolver normalizes it.
self.assertIsNotNone(self._validate_with_addrs("http://2852039166/x", "169.254.169.254"))
def test_ipv6_loopback_blocked(self):
self.assertIsNotNone(
self._validate_with_addrs("http://[::1]/x", "::1", family=socket.AF_INET6)
)
def test_ipv4_mapped_ipv6_metadata_blocked(self):
self.assertIsNotNone(
self._validate_with_addrs(
"http://evil/x", "::ffff:169.254.169.254", family=socket.AF_INET6
)
)
def test_mixed_public_and_private_blocked(self):
# If any resolved address is internal, reject the whole URL.
self.assertIsNotNone(self._validate_with_addrs("http://mixed/x", "142.250.1.1", "127.0.0.1"))
def test_resolution_failure_defers_to_ytdlp(self):
with mock.patch("url_guard.socket.getaddrinfo", side_effect=socket.gaierror):
self.assertIsNone(validate_url("http://does-not-resolve.example/x"))
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,674 @@
"""Tests for pure helpers and migration logic in ``ytdl``."""
from __future__ import annotations
import pickle
import signal
import sys
import tempfile
import threading
import types
import unittest
from pathlib import Path
from unittest.mock import MagicMock, patch
fake_yt_dlp = types.ModuleType("yt_dlp")
fake_networking = types.ModuleType("yt_dlp.networking")
fake_impersonate = types.ModuleType("yt_dlp.networking.impersonate")
fake_postprocessor = types.ModuleType("yt_dlp.postprocessor")
fake_postprocessor_common = types.ModuleType("yt_dlp.postprocessor.common")
fake_utils = types.ModuleType("yt_dlp.utils")
class _ImpersonateTarget:
@staticmethod
def from_str(value):
return value
class _PostProcessor:
def __init__(self, downloader=None):
self._downloader = downloader
fake_impersonate.ImpersonateTarget = _ImpersonateTarget
fake_networking.impersonate = fake_impersonate
fake_postprocessor_common.PostProcessor = _PostProcessor
# The inner ``key`` group mirrors the real ``STR_FORMAT_RE_TMPL`` so that
# ``_OUTTMPL_FIELD_RE`` (compiled at import time) has the named group that
# ``_resolve_outtmpl_fields`` reads via ``match.group('key')``.
fake_utils.STR_FORMAT_RE_TMPL = r"(?P<prefix>)%\((?P<has_key>(?P<key>{}))\)(?P<format>[-0-9.]*{})"
fake_utils.STR_FORMAT_TYPES = "diouxXeEfFgGcrsa"
fake_yt_dlp.networking = fake_networking
fake_yt_dlp.postprocessor = fake_postprocessor
fake_yt_dlp.utils = fake_utils
sys.modules.setdefault("yt_dlp", fake_yt_dlp)
sys.modules.setdefault("yt_dlp.networking", fake_networking)
sys.modules.setdefault("yt_dlp.networking.impersonate", fake_impersonate)
sys.modules.setdefault("yt_dlp.postprocessor", fake_postprocessor)
sys.modules.setdefault("yt_dlp.postprocessor.common", fake_postprocessor_common)
sys.modules.setdefault("yt_dlp.utils", fake_utils)
from ytdl import (
Download,
DownloadInfo,
_compact_persisted_entry,
_convert_srt_to_txt_file,
_AlbumArtistPostProcessor,
_output_dir_escapes,
_resolve_outtmpl_fields,
_sanitize_entry_for_pickle,
_sanitize_path_component,
)
# Detect whether the real yt-dlp is loaded (as opposed to the minimal fake
# shim above). _resolve_outtmpl_fields needs YoutubeDL at runtime.
_has_real_ytdlp = hasattr(sys.modules.get("yt_dlp"), "YoutubeDL")
class AlbumArtistPostProcessorTests(unittest.TestCase):
def setUp(self):
self.postprocessor = _AlbumArtistPostProcessor()
def test_fills_album_artist_from_artist(self):
info = {'album': 'CrasH Talk', 'artist': 'ScHoolboy Q'}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'ScHoolboy Q')
def test_uses_main_artist_for_featured_track(self):
info = {
'album': 'CrasH Talk',
'artists': ['ScHoolboy Q · Travis Scott'],
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'ScHoolboy Q')
def test_uses_topic_channel_artist_for_joint_album(self):
info = {
'album': 'Watch the Throne',
'artists': ['JAY-Z', 'Kanye West'],
'channel': 'JAY-Z & Kanye West - Topic',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'JAY-Z & Kanye West')
def test_uses_topic_uploader_and_strips_suffix_for_compilation(self):
info = {
'album': 'Compilation',
'artist': 'Track Artist',
'channel': 'Regular Channel',
'uploader': 'Various Artists - Topic',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Various Artists')
def test_regular_channel_falls_back_to_main_artist(self):
info = {
'album': 'Album',
'artist': 'Track Artist',
'channel': 'Label Channel',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Track Artist')
def test_preserves_explicit_various_artists(self):
info = {
'album': 'Revenge of the Dreamers III',
'artist': 'J. Cole',
'album_artist': 'Various Artists',
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Various Artists')
def test_preserves_existing_album_artists_list(self):
info = {
'album': 'Album',
'artist': 'Track Artist',
'album_artists': ['Album Artist'],
}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artists'], ['Album Artist'])
self.assertNotIn('album_artist', result)
def test_uses_first_artist_when_artist_list_has_multiple_entries(self):
info = {'album': 'Album', 'artists': ['Main Artist', 'Featured Artist']}
_, result = self.postprocessor.run(info)
self.assertEqual(result['album_artist'], 'Main Artist')
def test_does_not_fill_without_album(self):
info = {'artist': 'Standalone Artist'}
_, result = self.postprocessor.run(info)
self.assertNotIn('album_artist', result)
self.assertNotIn('album_artists', result)
def test_does_not_fill_without_artist(self):
info = {'album': 'Instrumental Album'}
_, result = self.postprocessor.run(info)
self.assertNotIn('album_artist', result)
self.assertNotIn('album_artists', result)
class AlbumArtistRegistrationTests(unittest.TestCase):
def test_audio_download_registers_pre_process_postprocessor(self):
download = _make_test_download()
download.info.download_type = 'audio'
fake_ydl = MagicMock()
with patch('ytdl.yt_dlp.YoutubeDL', return_value=fake_ydl):
result = download._make_youtube_dl({'quiet': True})
self.assertIs(result, fake_ydl)
postprocessor, = fake_ydl.add_post_processor.call_args.args
self.assertIsInstance(postprocessor, _AlbumArtistPostProcessor)
self.assertEqual(fake_ydl.add_post_processor.call_args.kwargs, {'when': 'pre_process'})
def test_video_download_does_not_register_postprocessor(self):
download = _make_test_download()
fake_ydl = MagicMock()
with patch('ytdl.yt_dlp.YoutubeDL', return_value=fake_ydl):
download._make_youtube_dl({'quiet': True})
fake_ydl.add_post_processor.assert_not_called()
class SanitizePathComponentTests(unittest.TestCase):
def test_replaces_windows_invalid_chars(self):
self.assertEqual(_sanitize_path_component('a:b*c?d"e<f>g|h'), "a_b_c_d_e_f_g_h")
def test_non_string_passthrough(self):
self.assertIs(_sanitize_path_component(None), None)
self.assertEqual(_sanitize_path_component(42), 42)
def test_strips_path_separators_and_traversal(self):
result = _sanitize_path_component('../../../../etc/x')
self.assertNotIn('..', result)
self.assertNotIn('/', result)
self.assertNotIn('\\', result)
def test_strips_leading_absolute_path_separator(self):
result = _sanitize_path_component('/tmp/x')
self.assertFalse(result.startswith('/'))
self.assertFalse(result.startswith('\\'))
self.assertEqual(result, '_tmp_x')
def test_collapses_slashes_in_legitimate_titles(self):
self.assertEqual(_sanitize_path_component('AC/DC'), 'AC_DC')
def test_empty_after_strip_becomes_underscore(self):
self.assertEqual(_sanitize_path_component(' '), '_')
@unittest.skipUnless(_has_real_ytdlp, "requires real yt-dlp")
class ResolveOuttmplFieldsTests(unittest.TestCase):
"""Tests for _resolve_outtmpl_fields (delegates to yt-dlp's template engine)."""
def test_simple_playlist_substitution(self):
info = {"playlist_title": "My PL", "playlist_index": "03"}
result = _resolve_outtmpl_fields("%(playlist_title)s/%(title)s.%(ext)s", info, ("playlist",))
self.assertEqual(result, "My PL/%(title)s.%(ext)s")
def test_format_spec_int(self):
info = {"playlist_index": "3"}
result = _resolve_outtmpl_fields("%(playlist_index)02d-%(title)s", info, ("playlist",))
self.assertEqual(result, "03-%(title)s")
def test_non_targeted_fields_unchanged(self):
info = {"playlist_title": "PL"}
result = _resolve_outtmpl_fields("%(title)s/%(ext)s", info, ("playlist",))
self.assertEqual(result, "%(title)s/%(ext)s")
def test_default_value(self):
info = {"playlist_index": "1"}
result = _resolve_outtmpl_fields("%(playlist_title|Unknown)s/%(playlist_index)s", info, ("playlist",))
self.assertEqual(result, "Unknown/1")
def test_channel_prefix(self):
info = {"channel": "MyChan", "channel_index": "05"}
result = _resolve_outtmpl_fields("%(channel)s/%(channel_index)02d-%(title)s", info, ("channel",))
self.assertEqual(result, "MyChan/05-%(title)s")
def test_math_operation(self):
info = {"playlist_index": "3"}
result = _resolve_outtmpl_fields("%(playlist_index+100)d", info, ("playlist",))
self.assertEqual(result, "103")
def test_playlist_count_and_autonumber(self):
info = {
"playlist_title": "My PL",
"playlist_index": "03",
"playlist_count": 10,
"playlist_autonumber": 3,
"n_entries": 10,
"__last_playlist_index": 10,
}
result = _resolve_outtmpl_fields(
"%(playlist_title)s/%(playlist_autonumber)s of %(playlist_count)s - %(title)s.%(ext)s",
info,
("playlist",),
)
# playlist_autonumber is auto-padded by yt-dlp using __last_playlist_index
self.assertEqual(result, "My PL/03 of 10 - %(title)s.%(ext)s")
def test_conditional_playlist_index(self):
info = {
"playlist_index": "5",
"playlist_count": 10,
}
result = _resolve_outtmpl_fields(
"%(playlist_index&{} - |)s%(title)s.%(ext)s",
info,
("playlist",),
)
self.assertEqual(result, "5 - %(title)s.%(ext)s")
def test_malicious_playlist_title_cannot_escape_via_template(self):
malicious_title = '/tmp/METUBE_ARBITRARY_WRITE_POC'
entry = {
'playlist_title': malicious_title,
'playlist_index': '1',
'title': 'video',
'ext': 'mp4',
}
sanitized = {k: _sanitize_path_component(v) for k, v in entry.items()}
template = '%(playlist_title)s/%(title)s.%(ext)s'
result = _resolve_outtmpl_fields(template, sanitized, ('playlist',))
marker = result.find('%(')
literal_prefix = result[:marker] if marker != -1 else result
self.assertNotIn('..', literal_prefix)
self.assertFalse(literal_prefix.startswith('/'))
self.assertFalse(literal_prefix.startswith('\\'))
class OutputDirEscapesTests(unittest.TestCase):
def setUp(self):
self.base_dir = tempfile.mkdtemp()
def test_relative_traversal_escapes(self):
self.assertTrue(_output_dir_escapes(self.base_dir, '../../tmp/x/%(title)s.%(ext)s'))
def test_absolute_path_escapes(self):
self.assertTrue(_output_dir_escapes(self.base_dir, '/tmp/x/%(title)s.%(ext)s'))
def test_normal_playlist_dir_stays_inside(self):
self.assertFalse(_output_dir_escapes(self.base_dir, 'Playlist/%(title)s.%(ext)s'))
class SanitizeEntryForPickleTests(unittest.TestCase):
def test_nested(self):
def g():
yield 1
obj = {"a": g(), "b": [g()]}
out = _sanitize_entry_for_pickle(obj)
self.assertEqual(out, {"a": [1], "b": [[1]]})
pickle.dumps(out)
def test_plain(self):
self.assertEqual(_sanitize_entry_for_pickle(5), 5)
def test_set_converted_to_list(self):
obj = {"s": {1, 2}}
out = _sanitize_entry_for_pickle(obj)
self.assertEqual(sorted(out["s"]), [1, 2])
pickle.dumps(out)
def test_map_iterator(self):
out = _sanitize_entry_for_pickle({"m": map(int, ["1", "2"])})
self.assertEqual(out, {"m": [1, 2]})
def test_lock_replaced_with_none(self):
lock = threading.Lock()
out = _sanitize_entry_for_pickle({"k": lock})
self.assertIsNone(out["k"])
pickle.dumps(out)
def test_ordered_dict(self):
from collections import OrderedDict
od = OrderedDict([("z", 1), ("a", 2)])
out = _sanitize_entry_for_pickle(od)
self.assertEqual(out, {"z": 1, "a": 2})
def _make_test_download() -> Download:
info = DownloadInfo(
id="id1",
title="t",
url="http://example.com/v",
quality="best",
download_type="video",
codec="auto",
format="any",
folder="",
custom_name_prefix="",
error=None,
entry=None,
playlist_item_limit=0,
split_by_chapters=False,
chapter_template="",
)
return Download("/tmp", "/tmp", "%(title)s.%(ext)s", "%(title)s.%(ext)s", "best", "any", {}, info)
class ProgressThrottleTests(unittest.TestCase):
def test_downloading_ticks_are_throttled(self):
dl = _make_test_download()
forwarded = []
dl.status_queue = types.SimpleNamespace(put=forwarded.append)
hook = dl._make_progress_hook()
with patch("ytdl.time.monotonic", side_effect=[100.0, 100.1, 100.6, 100.7]):
hook({"status": "downloading", "downloaded_bytes": 1})
hook({"status": "downloading", "downloaded_bytes": 2})
hook({"status": "downloading", "downloaded_bytes": 3})
hook({"status": "downloading", "downloaded_bytes": 4})
# Only the 1st and 3rd ticks are >= 0.5s apart from the last forwarded one.
self.assertEqual(len(forwarded), 2)
def test_finished_and_error_statuses_always_forwarded(self):
dl = _make_test_download()
forwarded = []
dl.status_queue = types.SimpleNamespace(put=forwarded.append)
hook = dl._make_progress_hook()
with patch("ytdl.time.monotonic", side_effect=[200.0, 200.1]):
hook({"status": "downloading"})
hook({"status": "finished"})
hook({"status": "downloading"})
hook({"status": "error", "msg": "boom"})
statuses = [item.get("status") for item in forwarded]
self.assertIn("finished", statuses)
self.assertIn("error", statuses)
class CancelProcessGroupTests(unittest.TestCase):
def test_cancel_kills_group_when_child_is_group_leader(self):
# Child successfully ran os.setpgrp(): its pgid equals its own pid.
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321)
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", return_value=4321) as mock_getpgid, \
patch("ytdl.os.killpg") as mock_killpg:
dl.cancel()
mock_getpgid.assert_called_once_with(4321)
mock_killpg.assert_called_once_with(4321, signal.SIGKILL)
self.assertTrue(dl.canceled)
def test_cancel_does_not_killpg_parent_group_kills_child_only(self):
# Child has NOT become its own group leader yet (pgid != pid, e.g. it is
# still in the server's process group). killpg must NOT be called — that
# would SIGKILL the whole server — and we fall back to proc.kill().
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321, kill=MagicMock())
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", return_value=999), \
patch("ytdl.os.killpg") as mock_killpg:
dl.cancel()
mock_killpg.assert_not_called()
dl.proc.kill.assert_called_once()
self.assertTrue(dl.canceled)
def test_cancel_falls_back_to_proc_kill_when_getpgid_unavailable(self):
dl = _make_test_download()
dl.proc = types.SimpleNamespace(pid=4321, kill=MagicMock())
dl.status_queue = types.SimpleNamespace(put=lambda _item: None)
with patch.object(Download, "running", return_value=True), \
patch("ytdl.os.getpgid", side_effect=OSError("no such process")):
dl.cancel()
dl.proc.kill.assert_called_once()
self.assertTrue(dl.canceled)
class ConvertSrtToTxtTests(unittest.TestCase):
def test_basic_conversion(self):
srt = """1
00:00:01,000 --> 00:00:02,000
Hello <b>world</b>
2
00:00:03,000 --> 00:00:04,000
Second line
"""
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "sub.srt"
path.write_text(srt, encoding="utf-8")
txt_path = _convert_srt_to_txt_file(str(path))
self.assertIsNotNone(txt_path)
self.assertTrue(txt_path.endswith(".txt"))
content = Path(txt_path).read_text(encoding="utf-8")
self.assertIn("Hello world", content)
self.assertIn("Second line", content)
def test_vtt_input_strips_header_and_metadata(self):
# yt-dlp can fall back to VTT even when srt/txt was requested (the
# extractor may not offer a native srt track); the converter must not
# leak VTT-only header/metadata lines into the plain-text output.
vtt = """WEBVTT
Kind: captions
Language: en
NOTE
This is a note block
1
00:00:01.000 --> 00:00:02.000
Hello <b>world</b>
2
00:00:03.000 --> 00:00:04.000
Second line
"""
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "sub.vtt"
path.write_text(vtt, encoding="utf-8")
txt_path = _convert_srt_to_txt_file(str(path))
self.assertIsNotNone(txt_path)
content = Path(txt_path).read_text(encoding="utf-8")
self.assertIn("Hello world", content)
self.assertIn("Second line", content)
self.assertNotIn("WEBVTT", content)
self.assertNotIn("Kind:", content)
self.assertNotIn("Language:", content)
self.assertNotIn("This is a note block", content)
def test_vtt_standalone_header_block_is_stripped(self):
# Some VTT files put a blank line after WEBVTT, so Kind:/Language: form
# their own block. That header block (before the first timed cue) must
# still be stripped.
vtt = """WEBVTT
Kind: captions
Language: en
1
00:00:01.000 --> 00:00:02.000
Hello world
"""
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "sub.vtt"
path.write_text(vtt, encoding="utf-8")
content = Path(_convert_srt_to_txt_file(str(path))).read_text(encoding="utf-8")
self.assertIn("Hello world", content)
self.assertNotIn("Kind:", content)
self.assertNotIn("Language:", content)
def test_cue_text_starting_with_metadata_keyword_is_kept(self):
# A real caption line beginning with "Kind:"/"Language:" must NOT be
# dropped as if it were VTT header metadata.
srt = """1
00:00:01,000 --> 00:00:02,000
Kind: regards, everyone
2
00:00:03,000 --> 00:00:04,000
Language: they spoke French
"""
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "sub.srt"
path.write_text(srt, encoding="utf-8")
content = Path(_convert_srt_to_txt_file(str(path))).read_text(encoding="utf-8")
self.assertIn("Kind: regards, everyone", content)
self.assertIn("Language: they spoke French", content)
class DownloadInfoSetstateTests(unittest.TestCase):
def _base_state(self, **kwargs):
base = {
"id": "id1",
"title": "t",
"url": "http://example.com/v",
"folder": "",
"custom_name_prefix": "",
"error": None,
"entry": None,
"playlist_item_limit": 0,
"split_by_chapters": False,
"chapter_template": "",
"msg": None,
"percent": None,
"speed": None,
"eta": None,
"status": "pending",
"size": None,
"timestamp": 0,
}
base.update(kwargs)
return base
def test_migrates_old_audio_format(self):
state = self._base_state(format="m4a", quality="best")
di = DownloadInfo.__new__(DownloadInfo)
di.__setstate__(state)
self.assertEqual(di.download_type, "audio")
self.assertEqual(di.codec, "auto")
def test_migrates_thumbnail(self):
state = self._base_state(format="thumbnail", quality="best")
di = DownloadInfo.__new__(DownloadInfo)
di.__setstate__(state)
self.assertEqual(di.download_type, "thumbnail")
self.assertEqual(di.format, "jpg")
def test_migrates_captions(self):
state = self._base_state(format="captions", subtitle_format="vtt", quality="best")
di = DownloadInfo.__new__(DownloadInfo)
di.__setstate__(state)
self.assertEqual(di.download_type, "captions")
self.assertEqual(di.format, "vtt")
def test_migrates_best_ios(self):
state = self._base_state(
format="any", quality="best_ios", video_codec="auto"
)
di = DownloadInfo.__new__(DownloadInfo)
di.__setstate__(state)
self.assertEqual(di.format, "ios")
self.assertEqual(di.quality, "best")
def test_migrates_quality_audio(self):
state = self._base_state(format="mp4", quality="audio", video_codec="h264")
di = DownloadInfo.__new__(DownloadInfo)
di.__setstate__(state)
self.assertEqual(di.download_type, "audio")
self.assertEqual(di.format, "m4a")
def test_new_state_has_subtitle_files(self):
state = self._base_state(
download_type="video",
codec="auto",
format="any",
quality="best",
)
di = DownloadInfo.__new__(DownloadInfo)
di.__setstate__(state)
self.assertEqual(di.subtitle_files, [])
def test_missing_optional_fields_are_defaulted(self):
state = self._base_state(
download_type="video",
codec="auto",
format="any",
quality="best",
)
state.pop("folder")
state.pop("custom_name_prefix")
state.pop("playlist_item_limit")
state.pop("split_by_chapters")
state.pop("chapter_template")
di = DownloadInfo.__new__(DownloadInfo)
di.__setstate__(state)
self.assertEqual(di.folder, "")
self.assertEqual(di.custom_name_prefix, "")
self.assertEqual(di.playlist_item_limit, 0)
self.assertFalse(di.split_by_chapters)
self.assertEqual(di.chapter_template, "")
class CompactPersistedEntryTests(unittest.TestCase):
def test_keeps_only_playlist_and_channel_keys(self):
entry = {
"playlist_index": "01",
"playlist_title": "Playlist",
"playlist_count": 10,
"playlist_autonumber": 1,
"channel_index": "02",
"channel_title": "Channel",
"n_entries": 10,
"__last_playlist_index": 10,
"formats": [{"id": "huge"}],
"description": "big blob",
}
compact = _compact_persisted_entry(entry)
self.assertEqual(
compact,
{
"playlist_index": "01",
"playlist_title": "Playlist",
"playlist_count": 10,
"playlist_autonumber": 1,
"channel_index": "02",
"channel_title": "Channel",
"n_entries": 10,
"__last_playlist_index": 10,
},
)
def test_returns_none_when_no_restart_relevant_keys_exist(self):
self.assertIsNone(_compact_persisted_entry({"id": "x", "title": "y"}))
if __name__ == "__main__":
unittest.main()

89
app/url_guard.py Normal file
View file

@ -0,0 +1,89 @@
"""Lightweight SSRF guard for user-submitted URLs.
MeTube hands user-submitted URLs to yt-dlp, whose generic extractor will fetch
any ``http(s)`` URL. Without a guard, an attacker can make the server fetch
internal endpoints (cloud metadata services, loopback, RFC1918 hosts, etc.) and
have the response saved to the download directory and served back.
This module provides a single cheap validator applied at every URL ingress. It
intentionally does NOT attempt DNS-rebinding pinning, redirect-chain
re-validation, or validation of every media URL yt-dlp derives from remote
metadata network isolation (e.g. Docker) remains the backstop for those.
"""
import ipaddress
import logging
import socket
from urllib.parse import urlsplit
log = logging.getLogger('url_guard')
_ALLOWED_SCHEMES = ('http', 'https')
# Hostnames that must be blocked without needing a lookup. ``localhost`` and any
# subdomain of it are conventionally loopback, and the GCP metadata name is a
# well-known SSRF target that may resolve via a resolver we don't control.
_BLOCKED_HOSTNAMES = ('localhost', 'metadata.google.internal')
def _hostname_is_blocked(hostname: str) -> bool:
host = hostname.rstrip('.').lower()
for blocked in _BLOCKED_HOSTNAMES:
if host == blocked or host.endswith('.' + blocked):
return True
return False
def _address_is_global(addr: str) -> bool:
try:
ip = ipaddress.ip_address(addr)
except ValueError:
return False
# Unwrap IPv4-mapped/compatible IPv6 (e.g. ::ffff:169.254.169.254) so the
# embedded IPv4 address is judged on its own merits.
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
ip = ip.ipv4_mapped
return ip.is_global
def validate_url(url: str) -> str | None:
"""Return an error message if the URL is disallowed, else ``None``.
Inputs without a ``://`` scheme separator (bare video IDs, ``ytsearch:``
and other yt-dlp search/extractor prefixes) are allowed unchanged so that
non-URL entries keep working.
"""
if not isinstance(url, str):
return 'Invalid URL'
candidate = url.strip()
if '://' not in candidate:
# Not an absolute URL: bare video IDs, ytsearch: prefixes, etc.
return None
parts = urlsplit(candidate)
scheme = parts.scheme.lower()
if scheme not in _ALLOWED_SCHEMES:
return f'URL scheme "{parts.scheme}" is not allowed (only http and https)'
hostname = parts.hostname
if not hostname:
return 'URL is missing a host'
if _hostname_is_blocked(hostname):
return f'Refusing to fetch internal host "{hostname}"'
try:
addrinfo = socket.getaddrinfo(hostname, parts.port, proto=socket.IPPROTO_TCP)
except socket.gaierror:
# Let yt-dlp surface a normal resolution error rather than masking it.
return None
except (UnicodeError, ValueError):
return f'Invalid host "{hostname}"'
for family, _type, _proto, _canonname, sockaddr in addrinfo:
addr = sockaddr[0]
if not _address_is_global(addr):
return f'Refusing to fetch internal address "{addr}" for host "{hostname}"'
return None

File diff suppressed because it is too large Load diff

View file

@ -2,11 +2,54 @@
PUID="${UID:-$PUID}"
PGID="${GID:-$PGID}"
AUDIO_DOWNLOAD_DIR="${AUDIO_DOWNLOAD_DIR:-$DOWNLOAD_DIR}"
echo "Setting umask to ${UMASK}"
umask ${UMASK}
echo "Creating download directory (${DOWNLOAD_DIR}), state directory (${STATE_DIR}), and temp dir (${TEMP_DIR})"
mkdir -p "${DOWNLOAD_DIR}" "${STATE_DIR}" "${TEMP_DIR}"
echo "Creating download directory (${DOWNLOAD_DIR}), audio download directory (${AUDIO_DOWNLOAD_DIR}), state directory (${STATE_DIR}), and temp dir (${TEMP_DIR})"
mkdir -p "${DOWNLOAD_DIR}" "${AUDIO_DOWNLOAD_DIR}" "${STATE_DIR}" "${TEMP_DIR}"
do_upgrade() {
echo "Upgrading yt-dlp to nightly channel..."
if ! python3 -m pip --version >/dev/null 2>&1; then
echo "pip not found; attempting ensurepip"
python3 -m ensurepip --upgrade >/dev/null 2>&1 || true
fi
if ! python3 -m pip install -U --pre "yt-dlp[default,curl-cffi,deno]"; then
echo "Warning: yt-dlp nightly upgrade failed; continuing with existing installation"
return 1
fi
echo "yt-dlp nightly upgrade complete"
return 0
}
run_supervised() {
while true; do
"$@" &
child_pid=$!
trap 'kill -TERM "$child_pid" 2>/dev/null; wait "$child_pid" 2>/dev/null' TERM INT
wait "$child_pid"
exit_code=$?
trap - TERM INT
if [ "$exit_code" -eq 42 ]; then
echo "MeTube requested yt-dlp update restart (exit 42)"
do_upgrade || true
continue
fi
return "$exit_code"
done
}
nightly_enabled() {
[ -n "${YTDL_NIGHTLY_UPDATE_TIME}" ]
}
disable_nightly_for_non_root() {
if nightly_enabled; then
echo "YTDL_NIGHTLY_UPDATE_TIME is set but this container runs as a non-root user; nightly yt-dlp updates are not supported. Ignoring YTDL_NIGHTLY_UPDATE_TIME."
unset YTDL_NIGHTLY_UPDATE_TIME
fi
}
if [ `id -u` -eq 0 ] && [ `id -g` -eq 0 ]; then
if [ "${PUID}" -eq 0 ]; then
@ -14,15 +57,22 @@ if [ `id -u` -eq 0 ] && [ `id -g` -eq 0 ]; then
fi
if [ "${CHOWN_DIRS:-true}" != "false" ]; then
echo "Changing ownership of download and state directories to ${PUID}:${PGID}"
chown -R "${PUID}":"${PGID}" /app "${DOWNLOAD_DIR}" "${STATE_DIR}" "${TEMP_DIR}"
chown -R "${PUID}":"${PGID}" /app "${DOWNLOAD_DIR}" "${AUDIO_DOWNLOAD_DIR}" "${STATE_DIR}" "${TEMP_DIR}"
fi
if nightly_enabled; then
echo "YTDL_NIGHTLY_UPDATE_TIME is set to ${YTDL_NIGHTLY_UPDATE_TIME}; upgrading yt-dlp on startup"
do_upgrade || true
fi
echo "Starting BgUtils POT Provider"
gosu "${PUID}":"${PGID}" bgutil-pot server >/tmp/bgutil-pot.log 2>&1 &
echo "Running MeTube as user ${PUID}:${PGID}"
exec gosu "${PUID}":"${PGID}" python3 app/main.py
run_supervised gosu "${PUID}":"${PGID}" python3 app/main.py
exit $?
else
echo "User set by docker; running MeTube as `id -u`:`id -g`"
disable_nightly_for_non_root
echo "Starting BgUtils POT Provider"
bgutil-pot server >/tmp/bgutil-pot.log 2>&1 &
exec python3 app/main.py
run_supervised python3 app/main.py
exit $?
fi

View file

@ -15,4 +15,13 @@ dependencies = [
[dependency-groups]
dev = [
"pylint",
"pytest>=8.0",
"pytest-aiohttp>=1.0",
"pytest-asyncio>=0.24",
]
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["app/tests"]
pythonpath = [".", "app"]
addopts = "-v"

View file

@ -33,9 +33,7 @@
"node_modules/@ng-select/ng-select/themes/default.theme.css",
"src/styles.sass"
],
"scripts": [
"node_modules/bootstrap/dist/js/bootstrap.bundle.min.js"
],
"scripts": [],
"serviceWorker": "ngsw-config.json",
"browser": "src/main.ts",
"polyfills": [
@ -77,7 +75,8 @@
"buildTarget": "metube:build:production"
},
"development": {
"buildTarget": "metube:build:development"
"buildTarget": "metube:build:development",
"proxyConfig": "proxy.conf.json"
}
},
"defaultConfiguration": "development"

View file

@ -5,7 +5,7 @@
"ng": "ng",
"start": "ng serve",
"build": "ng build",
"build:watch": "ng build --watch",
"build:watch": "ng build --watch --configuration development",
"test": "ng test",
"lint": "ng lint"
},
@ -21,43 +21,44 @@
}
]
},
"packageManager": "pnpm@11.5.2",
"private": true,
"dependencies": {
"@angular/animations": "^21.2.1",
"@angular/common": "^21.2.1",
"@angular/compiler": "^21.2.1",
"@angular/core": "^21.2.1",
"@angular/forms": "^21.2.1",
"@angular/platform-browser": "^21.2.1",
"@angular/platform-browser-dynamic": "^21.2.1",
"@angular/service-worker": "^21.2.1",
"@angular/animations": "^22.0.6",
"@angular/common": "^22.0.6",
"@angular/compiler": "^22.0.6",
"@angular/core": "^22.0.6",
"@angular/forms": "^22.0.6",
"@angular/platform-browser": "^22.0.6",
"@angular/platform-browser-dynamic": "^22.0.6",
"@angular/service-worker": "^22.0.6",
"@fortawesome/angular-fontawesome": "~4.0.0",
"@fortawesome/fontawesome-svg-core": "^7.2.0",
"@fortawesome/free-brands-svg-icons": "^7.2.0",
"@fortawesome/free-regular-svg-icons": "^7.2.0",
"@fortawesome/free-solid-svg-icons": "^7.2.0",
"@ng-bootstrap/ng-bootstrap": "^20.0.0",
"@ng-select/ng-select": "^21.5.2",
"@fortawesome/fontawesome-svg-core": "^7.3.1",
"@fortawesome/free-brands-svg-icons": "^7.3.1",
"@fortawesome/free-regular-svg-icons": "^7.3.1",
"@fortawesome/free-solid-svg-icons": "^7.3.1",
"@ng-bootstrap/ng-bootstrap": "^21.0.0",
"@ng-select/ng-select": "^23.2.0",
"@popperjs/core": "^2.11.8",
"bootstrap": "^5.3.8",
"ngx-cookie-service": "^21.1.0",
"ngx-cookie-service": "^22.0.0",
"ngx-socket-io": "~4.10.0",
"rxjs": "~7.8.2",
"tslib": "^2.8.1",
"zone.js": "0.15.0"
},
"devDependencies": {
"@angular-eslint/builder": "21.1.0",
"@angular/build": "^21.2.1",
"@angular/cli": "^21.2.1",
"@angular/compiler-cli": "^21.2.1",
"@angular/localize": "^21.2.1",
"@eslint/js": "^9.39.3",
"angular-eslint": "21.1.0",
"eslint": "^9.39.3",
"@angular-eslint/builder": "22.0.0",
"@angular/build": "^22.0.7",
"@angular/cli": "^22.0.7",
"@angular/compiler-cli": "^22.0.6",
"@angular/localize": "^22.0.6",
"@eslint/js": "^9.39.5",
"angular-eslint": "22.0.0",
"eslint": "^9.39.5",
"jsdom": "^27.4.0",
"typescript": "~5.9.3",
"typescript-eslint": "8.47.0",
"vitest": "^4.0.18"
"typescript": "~6.0.3",
"typescript-eslint": "8.62.0",
"vitest": "^4.1.10"
}
}

4435
ui/pnpm-lock.yaml generated

File diff suppressed because it is too large Load diff

6
ui/pnpm-workspace.yaml Normal file
View file

@ -0,0 +1,6 @@
allowBuilds:
'@parcel/watcher': true
core-js: true
esbuild: true
lmdb: true
msgpackr-extract: true

View file

@ -1,6 +1,6 @@
import { ApplicationConfig, provideBrowserGlobalErrorListeners, isDevMode, provideZonelessChangeDetection, provideZoneChangeDetection } from '@angular/core';
import { ApplicationConfig, provideBrowserGlobalErrorListeners, isDevMode, provideZoneChangeDetection } from '@angular/core';
import { provideServiceWorker } from '@angular/service-worker';
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http';
import { provideHttpClient, withInterceptorsFromDi, withXhr } from '@angular/common/http';
export const appConfig: ApplicationConfig = {
providers: [
@ -12,6 +12,6 @@ export const appConfig: ApplicationConfig = {
// or after 30 seconds (whichever comes first).
registrationStrategy: 'registerWhenStable:30000'
}),
provideHttpClient(withInterceptorsFromDi()),
provideHttpClient(withXhr(), withInterceptorsFromDi()),
]
};

File diff suppressed because it is too large Load diff

View file

@ -1,29 +1,7 @@
.button-toggle-theme:focus, .button-toggle-theme:active
box-shadow: none
outline: 0px
.add-url-box
max-width: 960px
margin: 4rem auto
.add-url-component
margin: 0.5rem auto
.add-url-group
width: 100%
button.add-url
width: 100%
.folder-dropdown-menu
width: 500px
max-width: calc(100vw - 3rem)
.folder-dropdown-menu .input-group
display: flex
padding-left: 5px
padding-right: 5px
.metube-section-header
font-size: 1.8rem
font-weight: 300
@ -66,39 +44,11 @@ td
width: 12rem
margin-left: auto
.batch-panel
margin-top: 15px
border: 1px solid #ccc
border-radius: 8px
padding: 15px
background-color: #fff
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1)
.batch-panel-header
border-bottom: 1px solid #eee
padding-bottom: 8px
margin-bottom: 15px
h4
font-size: 1.5rem
margin: 0
.batch-panel-body
textarea.form-control
resize: vertical
.batch-status
font-size: 0.9rem
color: #555
.d-flex.my-3
margin-top: 1rem
margin-bottom: 1rem
.modal.fade.show
background-color: rgba(0, 0, 0, 0.5)
.modal-header
border-bottom: 1px solid #eee
border-bottom: 1px solid var(--bs-border-color)
.modal-body
textarea.form-control
@ -112,20 +62,12 @@ td
.spinner-border
margin-right: 0.5rem
::ng-deep .ng-select
flex: 1
.ng-select-container
min-height: 38px
.ng-value
white-space: nowrap
overflow: visible
.ng-dropdown-panel
.ng-dropdown-panel-items
max-height: 300px
.ng-option
white-space: nowrap
overflow: visible
text-overflow: ellipsis
.add-progress-btn
min-width: 9.5rem
cursor: default
.add-cancel-btn
min-width: 3.25rem
:host
display: flex
@ -240,6 +182,18 @@ main
opacity: 0.65
pointer-events: none
.settings-section-label
font-size: 0.8rem
text-transform: uppercase
letter-spacing: 0.1em
font-weight: 600
color: var(--bs-body-color)
margin-top: 1.75rem
margin-bottom: 0.75rem
&:first-child
margin-top: 0
.action-group-label
font-size: 0.7rem
text-transform: uppercase
@ -247,6 +201,12 @@ main
color: var(--bs-secondary-color)
margin-bottom: 0.4rem
.help-title
cursor: help
&:focus
outline: none
.cookie-status
font-size: 0.8rem
margin-top: 0.35rem

View file

@ -1,26 +1,135 @@
import { TestBed } from '@angular/core/testing';
import { HttpClient } from '@angular/common/http';
import { Subject, of } from 'rxjs';
import { App } from './app';
import { DownloadsService } from './services/downloads.service';
import { SubscriptionsService } from './services/subscriptions.service';
import { ToastService } from './services/toast.service';
import { CookieService } from 'ngx-cookie-service';
vi.hoisted(() => {
Object.defineProperty(window, "matchMedia", {
writable: true,
enumerable: true,
value: vi.fn().mockImplementation((query) => ({
matches: false,
media: query,
onchange: null,
addEventListener: vi.fn(),
removeEventListener: vi.fn(),
dispatchEvent: vi.fn(),
})),
});
});
class DownloadsServiceStub {
loading = false;
queue = new Map();
done = new Map();
configuration: Record<string, unknown> = { CUSTOM_DIRS: true, CREATE_CUSTOM_DIRS: true, ALLOW_YTDL_OPTIONS_OVERRIDES: false };
customDirs = { download_dir: [], audio_download_dir: [] };
queueChanged = new Subject<void>();
doneChanged = new Subject<void>();
configurationChanged = new Subject<Record<string, unknown>>();
customDirsChanged = new Subject<Record<string, string[]>>();
ytdlOptionsChanged = new Subject<Record<string, unknown>>();
updated = new Subject<void>();
getCookieStatus() {
return of({ status: 'ok', has_cookies: false });
}
getPresets() {
return of({ presets: ['Preset A'] });
}
add() {
return of({ status: 'ok' as const });
}
cancelAdd() {
return of({ status: 'ok' as const });
}
startById() {
return of({});
}
delById() {
return of({});
}
delByFilter() {
return of({});
}
startByFilter() {
return of({});
}
uploadCookies() {
return of({ status: 'ok' });
}
deleteCookies() {
return of({ status: 'ok' });
}
}
class SubscriptionsServiceStub {
subscriptions = new Map();
subscriptionsChanged = new Subject<void>();
subscribeCalls: unknown[] = [];
subscribe(payload: unknown) {
this.subscribeCalls.push(payload);
return of({ status: 'ok' as const });
}
delete() {
return of({});
}
update() {
return of({ status: 'ok' as const });
}
refreshList() {
return of([]);
}
}
class CookieServiceStub {
private cookies = new Map<string, string>();
get(name: string) {
return this.cookies.get(name) ?? '';
}
set(name: string, value: string) {
this.cookies.set(name, value);
}
check(name: string) {
return this.cookies.has(name);
}
}
describe('App', () => {
let downloads: DownloadsServiceStub;
beforeEach(async () => {
Object.defineProperty(window, 'matchMedia', {
writable: true,
enumerable: true,
value: vi.fn().mockImplementation((query: string) => ({
matches: false,
media: query,
onchange: null,
addEventListener: vi.fn(),
removeEventListener: vi.fn(),
dispatchEvent: vi.fn(),
})),
});
downloads = new DownloadsServiceStub();
await TestBed.configureTestingModule({
imports: [App],
providers: [
{ provide: DownloadsService, useValue: downloads },
{ provide: SubscriptionsService, useClass: SubscriptionsServiceStub },
{ provide: CookieService, useClass: CookieServiceStub },
{
provide: HttpClient,
useValue: {
get: vi.fn().mockReturnValue(of({ 'yt-dlp': 'test', version: 'test' })),
},
},
],
}).compileComponents();
});
@ -30,4 +139,147 @@ describe('App', () => {
expect(app).toBeTruthy();
});
it('asIsOrder returns a stable comparator value (insertion order preserved)', () => {
const fixture = TestBed.createComponent(App);
const app = fixture.componentInstance;
expect(app.asIsOrder()).toBe(0);
});
it('hides manual override input when disabled', () => {
const fixture = TestBed.createComponent(App);
fixture.componentInstance.isAdvancedOpen = true;
fixture.detectChanges();
const root = fixture.nativeElement as HTMLElement;
expect(root.querySelector('input[name="ytdlOptionsOverrides"]')).toBeNull();
const presetWrapper = root.querySelector('ng-select[name="ytdlOptionsPresets"]')?.closest('.col-12');
expect(presetWrapper?.classList.contains('col-md-6')).toBe(false);
const presetRow = root.querySelector('ng-select[name="ytdlOptionsPresets"]')?.closest('.row');
expect(presetRow?.querySelector('input[name="checkIntervalMinutes"]')).toBeNull();
});
it('shows manual override input when enabled', () => {
downloads.configuration['ALLOW_YTDL_OPTIONS_OVERRIDES'] = true;
const fixture = TestBed.createComponent(App);
fixture.componentInstance.isAdvancedOpen = true;
fixture.detectChanges();
const root = fixture.nativeElement as HTMLElement;
expect(root.querySelector('input[name="ytdlOptionsOverrides"]')).not.toBeNull();
const presetWrapper = root.querySelector('ng-select[name="ytdlOptionsPresets"]')?.closest('.col-12');
expect(presetWrapper?.classList.contains('col-md-6')).toBe(true);
const presetRow = root.querySelector('ng-select[name="ytdlOptionsPresets"]')?.closest('.row');
expect(presetRow?.querySelector('input[name="checkIntervalMinutes"]')).toBeNull();
expect(presetRow?.querySelector('input[name="ytdlOptionsOverrides"]')).not.toBeNull();
});
it('does not submit manual overrides when disabled', () => {
const fixture = TestBed.createComponent(App);
const app = fixture.componentInstance;
app.ytdlOptionsOverrides = '{"exec":"echo hi"}';
const payload = app['buildAddPayload']();
expect(payload.ytdlOptionsOverrides).toBe('');
});
it('shows waiting badge for scheduled live stream', () => {
downloads.queue.set('https://example.com/live', {
id: 'live1',
title: 'Upcoming Stream',
url: 'https://example.com/live',
download_type: 'video',
quality: 'best',
format: 'any',
folder: '',
custom_name_prefix: '',
playlist_item_limit: 0,
status: 'scheduled',
live_status: 'is_upcoming',
live_release_timestamp: Date.now() / 1000 + 3600,
msg: '',
percent: 0,
speed: 0,
eta: 0,
filename: '',
checked: false,
});
downloads.queueChanged.next();
const fixture = TestBed.createComponent(App);
fixture.detectChanges();
const root = fixture.nativeElement as HTMLElement;
expect(root.textContent).toContain('Waiting for stream');
expect(root.textContent).toContain('starts in');
});
it('includes titleRegex in subscribe payload', () => {
const fixture = TestBed.createComponent(App);
const app = fixture.componentInstance;
const subs = TestBed.inject(SubscriptionsService) as unknown as SubscriptionsServiceStub;
app.addUrl = 'https://example.com/channel';
app.titleRegex = 'EPISODE';
app.addSubscription();
expect(subs.subscribeCalls.length).toBe(1);
const payload = subs.subscribeCalls[0] as { titleRegex: string; skipSubscriberOnly: boolean };
expect(payload.titleRegex).toBe('EPISODE');
expect(payload.skipSubscriberOnly).toBe(false);
});
it('includes skipSubscriberOnly true when checked', () => {
const fixture = TestBed.createComponent(App);
const app = fixture.componentInstance;
const subs = TestBed.inject(SubscriptionsService) as unknown as SubscriptionsServiceStub;
app.addUrl = 'https://example.com/channel';
app.skipSubscriberOnly = true;
app.addSubscription();
expect(subs.subscribeCalls.length).toBe(1);
const payload = subs.subscribeCalls[0] as { skipSubscriberOnly: boolean };
expect(payload.skipSubscriberOnly).toBe(true);
});
it('omits clip fields from subscribe payload', () => {
const fixture = TestBed.createComponent(App);
const app = fixture.componentInstance;
const subs = TestBed.inject(SubscriptionsService) as unknown as SubscriptionsServiceStub;
app.addUrl = 'https://example.com/channel';
app.clipStart = '1:00';
app.clipEnd = '2:00';
app.addSubscription();
expect(subs.subscribeCalls.length).toBe(1);
const payload = subs.subscribeCalls[0] as Record<string, unknown>;
expect('clipStart' in payload).toBe(false);
expect('clipEnd' in payload).toBe(false);
});
it('buildAddPayload includes clip times', () => {
const fixture = TestBed.createComponent(App);
const app = fixture.componentInstance;
app.clipStart = '0:10';
app.clipEnd = '1:20';
const payload = app['buildAddPayload']();
expect(payload.clipStart).toBe('0:10');
expect(payload.clipEnd).toBe('1:20');
});
it('blocks subscribe with invalid title regex', () => {
const toasts = TestBed.inject(ToastService);
const errorSpy = vi.spyOn(toasts, 'error').mockImplementation(() => undefined);
const fixture = TestBed.createComponent(App);
const app = fixture.componentInstance;
const subs = TestBed.inject(SubscriptionsService) as unknown as SubscriptionsServiceStub;
app.addUrl = 'https://example.com/channel';
app.titleRegex = '[';
app.addSubscription();
expect(subs.subscribeCalls.length).toBe(0);
expect(errorSpy).toHaveBeenCalledWith('Invalid subscription title filter (regex)');
errorSpy.mockRestore();
});
});

File diff suppressed because it is too large Load diff

View file

@ -1,2 +1,3 @@
export { MasterCheckboxComponent } from './master-checkbox.component';
export { SlaveCheckboxComponent } from './slave-checkbox.component';
export { SelectAllCheckboxComponent } from './master-checkbox.component';
export { ItemCheckboxComponent } from './slave-checkbox.component';
export { ToastContainerComponent } from './toast-container.component';

View file

@ -0,0 +1,23 @@
import { TestBed } from '@angular/core/testing';
import { SelectAllCheckboxComponent } from './master-checkbox.component';
import { Checkable } from '../interfaces';
describe('SelectAllCheckboxComponent', () => {
beforeEach(async () => {
await TestBed.configureTestingModule({
imports: [SelectAllCheckboxComponent],
}).compileComponents();
});
it('clicked sets checked on all list items', () => {
const fixture = TestBed.createComponent(SelectAllCheckboxComponent);
const list = new Map<string, Checkable>();
list.set('u1', { checked: false });
fixture.componentRef.setInput('id', 'queue');
fixture.componentRef.setInput('list', list);
fixture.componentInstance.selected = true;
fixture.detectChanges();
fixture.componentInstance.clicked();
expect(list.get('u1')?.checked).toBe(true);
});
});

View file

@ -1,20 +1,23 @@
import { Component, ElementRef, viewChild, output, input } from "@angular/core";
import { Component, ElementRef, viewChild, output, input, ChangeDetectionStrategy } from "@angular/core";
import { Checkable } from "../interfaces";
import { FormsModule } from "@angular/forms";
@Component({
selector: 'app-master-checkbox',
selector: 'app-select-all-checkbox',
template: `
<div class="form-check">
<input type="checkbox" class="form-check-input" id="{{id()}}-select-all" #masterCheckbox [(ngModel)]="selected" (change)="clicked()">
<label class="form-check-label" for="{{id()}}-select-all"></label>
<input type="checkbox" class="form-check-input" id="{{id()}}-select-all" #masterCheckbox [(ngModel)]="selected" (change)="clicked()" [attr.aria-label]="'Select all ' + id() + ' items'">
<label class="form-check-label visually-hidden" for="{{id()}}-select-all">Select all</label>
</div>
`,
imports: [
// Shared Checkable objects are mutated in place; Eager preserves pre-v22 behavior.
// eslint-disable-next-line @angular-eslint/prefer-on-push-component-change-detection
changeDetection: ChangeDetectionStrategy.Eager,
imports: [
FormsModule
]
})
export class MasterCheckboxComponent {
export class SelectAllCheckboxComponent {
readonly id = input.required<string>();
readonly list = input.required<Map<string, Checkable>>();
readonly changed = output<number>();
@ -33,7 +36,7 @@ export class MasterCheckboxComponent {
return;
let checked = 0;
this.list().forEach(item => { if(item.checked) checked++ });
this.selected = checked > 0 && checked == this.list().size;
this.selected = checked > 0 && checked === this.list().size;
masterCheckbox.nativeElement.indeterminate = checked > 0 && checked < this.list().size;
this.changed.emit(checked);
}

View file

@ -0,0 +1,25 @@
import { TestBed } from '@angular/core/testing';
import { SelectAllCheckboxComponent } from './master-checkbox.component';
import { ItemCheckboxComponent } from './slave-checkbox.component';
describe('ItemCheckboxComponent', () => {
beforeEach(async () => {
await TestBed.configureTestingModule({
imports: [ItemCheckboxComponent, SelectAllCheckboxComponent],
}).compileComponents();
});
it('creates with master and checkable inputs', () => {
const masterFixture = TestBed.createComponent(SelectAllCheckboxComponent);
masterFixture.componentRef.setInput('id', 'q');
masterFixture.componentRef.setInput('list', new Map());
masterFixture.detectChanges();
const itemFixture = TestBed.createComponent(ItemCheckboxComponent);
itemFixture.componentRef.setInput('id', 'row1');
itemFixture.componentRef.setInput('master', masterFixture.componentInstance);
itemFixture.componentRef.setInput('checkable', { checked: false });
itemFixture.detectChanges();
expect(itemFixture.componentInstance).toBeTruthy();
});
});

View file

@ -1,22 +1,25 @@
import { Component, input } from '@angular/core';
import { MasterCheckboxComponent } from './master-checkbox.component';
import { Component, input, ChangeDetectionStrategy } from '@angular/core';
import { SelectAllCheckboxComponent } from './master-checkbox.component';
import { Checkable } from '../interfaces';
import { FormsModule } from '@angular/forms';
@Component({
selector: 'app-slave-checkbox',
selector: 'app-item-checkbox',
template: `
<div class="form-check">
<input type="checkbox" class="form-check-input" id="{{master().id()}}-{{id()}}-select" [(ngModel)]="checkable().checked" (change)="master().selectionChanged()">
<label class="form-check-label" for="{{master().id()}}-{{id()}}-select"></label>
<input type="checkbox" class="form-check-input" id="{{master().id()}}-{{id()}}-select" [(ngModel)]="checkable().checked" (change)="master().selectionChanged()" [attr.aria-label]="'Select item ' + id()">
<label class="form-check-label visually-hidden" for="{{master().id()}}-{{id()}}-select">Select item</label>
</div>
`,
imports: [
// Shared Checkable objects are mutated in place; Eager preserves pre-v22 behavior.
// eslint-disable-next-line @angular-eslint/prefer-on-push-component-change-detection
changeDetection: ChangeDetectionStrategy.Eager,
imports: [
FormsModule
]
})
export class SlaveCheckboxComponent {
export class ItemCheckboxComponent {
readonly id = input.required<string>();
readonly master = input.required<MasterCheckboxComponent>();
readonly master = input.required<SelectAllCheckboxComponent>();
readonly checkable = input.required<Checkable>();
}

View file

@ -0,0 +1,58 @@
import { ChangeDetectionStrategy, Component, inject } from '@angular/core';
import { FontAwesomeModule } from '@fortawesome/angular-fontawesome';
import { faCheckCircle, faTimesCircle, faInfoCircle, faXmark } from '@fortawesome/free-solid-svg-icons';
import { ToastService } from '../services/toast.service';
@Component({
selector: 'app-toast-container',
changeDetection: ChangeDetectionStrategy.OnPush,
imports: [FontAwesomeModule],
template: `
<div class="toast-container position-fixed top-0 end-0 p-3" style="z-index: 1100;" aria-live="polite" aria-atomic="true">
@for (toast of toasts.toasts(); track toast.id) {
<div class="toast show align-items-center border-0 mb-2"
[class.text-bg-danger]="toast.level === 'error'"
[class.text-bg-success]="toast.level === 'success'"
[class.text-bg-primary]="toast.level === 'info'"
role="alert" aria-live="assertive" aria-atomic="true">
<div class="d-flex">
<div class="toast-body d-flex align-items-start gap-2">
@if (toast.level === 'error') {
<fa-icon [icon]="faTimesCircle" class="mt-1" />
} @else if (toast.level === 'success') {
<fa-icon [icon]="faCheckCircle" class="mt-1" />
} @else {
<fa-icon [icon]="faInfoCircle" class="mt-1" />
}
<span style="white-space: pre-line;">{{ toast.message }}</span>
</div>
@if (!toast.actions) {
<button type="button" class="btn-close btn-close-white me-2 m-auto"
aria-label="Close" (click)="toasts.dismiss(toast.id)"></button>
}
</div>
@if (toast.actions) {
<div class="d-flex justify-content-end gap-2 px-3 pb-2">
@for (action of toast.actions; track action.label) {
<button type="button"
class="btn btn-sm"
[class.btn-light]="!action.primary"
[class.btn-outline-light]="action.primary"
(click)="toasts.respond(toast.id, action.value)">
{{ action.label }}
</button>
}
</div>
}
</div>
}
</div>
`,
})
export class ToastContainerComponent {
protected readonly toasts = inject(ToastService);
protected readonly faCheckCircle = faCheckCircle;
protected readonly faTimesCircle = faTimesCircle;
protected readonly faInfoCircle = faInfoCircle;
protected readonly faXmark = faXmark;
}

View file

@ -3,6 +3,8 @@ export interface Download {
id: string;
title: string;
url: string;
download_type: string;
codec?: string;
quality: string;
format: string;
folder: string;
@ -10,9 +12,14 @@ export interface Download {
playlist_item_limit: number;
split_by_chapters?: boolean;
chapter_template?: string;
subtitle_format?: string;
subtitle_language?: string;
subtitle_mode?: string;
ytdl_options_presets?: string[];
ytdl_options_overrides?: Record<string, unknown>;
clip_start?: number;
clip_end?: number;
live_status?: string;
live_release_timestamp?: number;
status: string;
msg: string;
percent: number;
@ -20,8 +27,9 @@ export interface Download {
eta: number;
filename: string;
checked: boolean;
timestamp?: number;
size?: number;
error?: string;
deleting?: boolean;
chapter_files?: Array<{ filename: string, size: number }>;
chapter_files?: { filename: string, size: number }[];
}

View file

@ -1,81 +1,77 @@
import { Format } from "./format";
import { Quality } from "./quality";
export interface Option {
id: string;
text: string;
}
export const Formats: Format[] = [
{
id: 'any',
text: 'Any',
qualities: [
{ id: 'best', text: 'Best' },
{ id: '2160', text: '2160p' },
{ id: '1440', text: '1440p' },
{ id: '1080', text: '1080p' },
{ id: '720', text: '720p' },
{ id: '480', text: '480p' },
{ id: '360', text: '360p' },
{ id: '240', text: '240p' },
{ id: 'worst', text: 'Worst' },
{ id: 'audio', text: 'Audio Only' },
],
},
{
id: 'mp4',
text: 'MP4',
qualities: [
{ id: 'best', text: 'Best' },
{ id: 'best_ios', text: 'Best (iOS)' },
{ id: '2160', text: '2160p' },
{ id: '1440', text: '1440p' },
{ id: '1080', text: '1080p' },
{ id: '720', text: '720p' },
{ id: '480', text: '480p' },
{ id: '360', text: '360p' },
{ id: '240', text: '240p' },
{ id: 'worst', text: 'Worst' },
],
},
{
id: 'm4a',
text: 'M4A',
qualities: [
{ id: 'best', text: 'Best' },
{ id: '192', text: '192 kbps' },
{ id: '128', text: '128 kbps' },
],
},
{
id: 'mp3',
text: 'MP3',
qualities: [
{ id: 'best', text: 'Best' },
{ id: '320', text: '320 kbps' },
{ id: '192', text: '192 kbps' },
{ id: '128', text: '128 kbps' },
],
},
{
id: 'opus',
text: 'OPUS',
qualities: [{ id: 'best', text: 'Best' }],
},
{
id: 'wav',
text: 'WAV',
qualities: [{ id: 'best', text: 'Best' }],
},
{
id: 'flac',
text: 'FLAC',
qualities: [{ id: 'best', text: 'Best' }],
},
{
id: 'thumbnail',
text: 'Thumbnail',
qualities: [{ id: 'best', text: 'Best' }],
},
{
id: 'captions',
text: 'Captions',
qualities: [{ id: 'best', text: 'Best' }],
},
export interface AudioFormatOption extends Option {
qualities: Quality[];
}
export const DOWNLOAD_TYPES: Option[] = [
{ id: "video", text: "Video" },
{ id: "audio", text: "Audio" },
{ id: "captions", text: "Captions" },
{ id: "thumbnail", text: "Thumbnail" },
];
export const VIDEO_CODECS: Option[] = [
{ id: "auto", text: "Auto" },
{ id: "h264", text: "H.264" },
{ id: "h265", text: "H.265 (HEVC)" },
{ id: "av1", text: "AV1" },
{ id: "vp9", text: "VP9" },
];
export const VIDEO_FORMATS: Option[] = [
{ id: "any", text: "Auto" },
{ id: "mp4", text: "MP4" },
{ id: "ios", text: "iOS Compatible" },
];
export const VIDEO_QUALITIES: Quality[] = [
{ id: "best", text: "Best" },
{ id: "2160", text: "2160p" },
{ id: "1440", text: "1440p" },
{ id: "1080", text: "1080p" },
{ id: "720", text: "720p" },
{ id: "480", text: "480p" },
{ id: "360", text: "360p" },
{ id: "240", text: "240p" },
{ id: "worst", text: "Worst" },
];
export const AUDIO_FORMATS: AudioFormatOption[] = [
{
id: "m4a",
text: "M4A",
qualities: [
{ id: "best", text: "Best" },
{ id: "192", text: "192 kbps" },
{ id: "128", text: "128 kbps" },
],
},
{
id: "mp3",
text: "MP3",
qualities: [
{ id: "best", text: "Best" },
{ id: "320", text: "320 kbps" },
{ id: "192", text: "192 kbps" },
{ id: "128", text: "128 kbps" },
],
},
{ id: "opus", text: "OPUS", qualities: [{ id: "best", text: "Best" }] },
{ id: "wav", text: "WAV", qualities: [{ id: "best", text: "Best" }] },
{ id: "flac", text: "FLAC", qualities: [{ id: "best", text: "Best" }] },
];
export const CAPTION_FORMATS: Option[] = [
{ id: "srt", text: "SRT" },
{ id: "txt", text: "TXT (Text only)" },
{ id: "vtt", text: "VTT" },
{ id: "ttml", text: "TTML" },
];
export const THUMBNAIL_FORMATS: Option[] = [{ id: "jpg", text: "JPG" }];

View file

@ -6,4 +6,4 @@ export * from './download';
export * from './checkable';
export * from './format';
export * from './formats';
export * from './subscription';

View file

@ -0,0 +1,17 @@
export interface SubscriptionRow {
id: string;
name: string;
url: string;
enabled: boolean;
check_interval_minutes: number;
download_type: string;
codec: string;
format: string;
quality: string;
folder: string;
title_regex?: string;
skip_subscriber_only?: boolean;
last_checked: number | null;
seen_count: number;
error: string | null;
}

View file

@ -0,0 +1,26 @@
import { EtaPipe } from './eta.pipe';
describe('EtaPipe', () => {
it('returns null for null input', () => {
const pipe = new EtaPipe();
expect(pipe.transform(null as unknown as number)).toBeNull();
});
it('formats seconds under one minute', () => {
const pipe = new EtaPipe();
expect(pipe.transform(0)).toBe('0s');
expect(pipe.transform(59)).toBe('59s');
});
it('formats minutes and seconds', () => {
const pipe = new EtaPipe();
expect(pipe.transform(60)).toBe('1m 0s');
expect(pipe.transform(90)).toBe('1m 30s');
});
it('formats hours', () => {
const pipe = new EtaPipe();
expect(pipe.transform(3600)).toBe('1h 0m 0s');
expect(pipe.transform(3661)).toBe('1h 1m 1s');
});
});

View file

@ -0,0 +1,25 @@
import { FileSizePipe } from './file-size.pipe';
describe('FileSizePipe', () => {
it('returns 0 Bytes for zero or NaN', () => {
const pipe = new FileSizePipe();
expect(pipe.transform(0)).toBe('0 Bytes');
expect(pipe.transform(Number.NaN)).toBe('0 Bytes');
});
it('formats bytes and larger units', () => {
const pipe = new FileSizePipe();
expect(pipe.transform(500)).toContain('Bytes');
expect(pipe.transform(1000)).toContain('Bytes');
expect(pipe.transform(1024)).toContain('KB');
expect(pipe.transform(1024 ** 2)).toContain('MB');
expect(pipe.transform(1024 ** 3)).toContain('GB');
});
it('handles boundaries between units', () => {
const pipe = new FileSizePipe();
expect(pipe.transform(1023)).toContain('Bytes');
expect(pipe.transform(1024)).toContain('KB');
expect(pipe.transform(1025)).toContain('KB');
});
});

View file

@ -8,9 +8,10 @@ export class FileSizePipe implements PipeTransform {
if (isNaN(value) || value === 0) return '0 Bytes';
const units = ['Bytes', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
const unitIndex = Math.floor(Math.log(value) / Math.log(1000)); // Use 1000 for common units
const k = 1024; // Matches SpeedPipe's base so file sizes and transfer speeds agree.
const unitIndex = Math.floor(Math.log(value) / Math.log(k));
const unitValue = value / Math.pow(1000, unitIndex);
const unitValue = value / Math.pow(k, unitIndex);
return `${unitValue.toFixed(2)} ${units[unitIndex]}`;
}
}

View file

@ -0,0 +1,21 @@
import { SpeedPipe } from './speed.pipe';
describe('SpeedPipe', () => {
it('returns empty string for non-positive speed values', () => {
const pipe = new SpeedPipe();
expect(pipe.transform(0)).toBe('');
expect(pipe.transform(-1)).toBe('');
});
it('formats bytes per second values', () => {
const pipe = new SpeedPipe();
expect(pipe.transform(1024)).toBe('1 KB/s');
expect(pipe.transform(1536)).toBe('1.5 KB/s');
});
it('formats MB/s and GB/s', () => {
const pipe = new SpeedPipe();
expect(pipe.transform(1024 * 1024)).toBe('1 MB/s');
expect(pipe.transform(1024 * 1024 * 1024)).toBe('1 GB/s');
});
});

View file

@ -1,43 +1,19 @@
import { Pipe, PipeTransform } from "@angular/core";
import { BehaviorSubject, throttleTime } from "rxjs";
@Pipe({
name: 'speed',
pure: false // Make the pipe impure so it can handle async updates
pure: true
})
export class SpeedPipe implements PipeTransform {
private speedSubject = new BehaviorSubject<number>(0);
private formattedSpeed = '';
constructor() {
// Throttle updates to once per second
this.speedSubject.pipe(
throttleTime(1000)
).subscribe(speed => {
// If speed is invalid or 0, return empty string
if (speed === null || speed === undefined || isNaN(speed) || speed <= 0) {
this.formattedSpeed = '';
return;
}
const k = 1024;
const dm = 2;
const sizes = ['B/s', 'KB/s', 'MB/s', 'GB/s', 'TB/s', 'PB/s', 'EB/s', 'ZB/s', 'YB/s'];
const i = Math.floor(Math.log(speed) / Math.log(k));
this.formattedSpeed = parseFloat((speed / Math.pow(k, i)).toFixed(dm)) + ' ' + sizes[i];
});
}
transform(value: number): string {
// If speed is invalid or 0, return empty string
if (value === null || value === undefined || isNaN(value) || value <= 0) {
return '';
}
// Update the speed subject
this.speedSubject.next(value);
// Return the last formatted speed
return this.formattedSpeed;
const k = 1024;
const decimals = 2;
const sizes = ['B/s', 'KB/s', 'MB/s', 'GB/s', 'TB/s', 'PB/s', 'EB/s', 'ZB/s', 'YB/s'];
const i = Math.floor(Math.log(value) / Math.log(k));
return `${parseFloat((value / Math.pow(k, i)).toFixed(decimals))} ${sizes[i]}`;
}
}

View file

@ -0,0 +1,62 @@
import { inject, Injectable } from '@angular/core';
import { DownloadsService } from './downloads.service';
import { ToastService } from './toast.service';
export type BatchUrlFilter = 'pending' | 'completed' | 'failed' | 'all';
/**
* Encapsulates collecting download URLs by status and exporting/copying them.
* Extracted from the main app component to keep it focused on view concerns.
*/
@Injectable({ providedIn: 'root' })
export class BatchUrlsService {
private downloads = inject(DownloadsService);
private toasts = inject(ToastService);
collect(filter: BatchUrlFilter): string[] {
const queueUrls = () => Array.from(this.downloads.queue.values()).map((dl) => dl.url);
const doneUrls = (status?: string) =>
Array.from(this.downloads.done.values())
.filter((dl) => status === undefined || dl.status === status)
.map((dl) => dl.url);
switch (filter) {
case 'pending':
return queueUrls();
case 'completed':
return doneUrls('finished');
case 'failed':
return doneUrls('error');
default:
return [...queueUrls(), ...doneUrls()];
}
}
export(filter: BatchUrlFilter): void {
const urls = this.collect(filter);
if (!urls.length) {
this.toasts.info('No URLs found for the selected filter.');
return;
}
const blob = new Blob([urls.join('\n')], { type: 'text/plain' });
const downloadUrl = window.URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = downloadUrl;
a.download = 'metube_urls.txt';
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
window.URL.revokeObjectURL(downloadUrl);
}
copy(filter: BatchUrlFilter): void {
const urls = this.collect(filter);
if (!urls.length) {
this.toasts.info('No URLs found for the selected filter.');
return;
}
navigator.clipboard
.writeText(urls.join('\n'))
.then(() => this.toasts.success('URLs copied to clipboard.'))
.catch(() => this.toasts.error('Failed to copy URLs.'));
}
}

View file

@ -0,0 +1,376 @@
import { TestBed } from '@angular/core/testing';
import { provideHttpClient, HttpErrorResponse } from '@angular/common/http';
import { provideHttpClientTesting, HttpTestingController } from '@angular/common/http/testing';
import { Subject } from 'rxjs';
import { DownloadsService, AddDownloadPayload } from './downloads.service';
import { MeTubeSocket } from './metube-socket.service';
import { Download } from '../interfaces';
class MeTubeSocketStub {
private subjects: Record<string, Subject<string>> = {};
fromEvent(event: string) {
if (!this.subjects[event]) {
this.subjects[event] = new Subject<string>();
}
return this.subjects[event].asObservable();
}
emit(event: string, data: string) {
if (!this.subjects[event]) {
this.subjects[event] = new Subject<string>();
}
this.subjects[event].next(data);
}
}
function basePayload(): AddDownloadPayload {
return {
url: 'https://example.com/v',
downloadType: 'video',
codec: 'auto',
quality: 'best',
format: 'any',
folder: '',
customNamePrefix: '',
playlistItemLimit: 0,
autoStart: true,
splitByChapters: false,
chapterTemplate: '',
subtitleLanguage: 'en',
subtitleMode: 'prefer_manual',
ytdlOptionsPresets: [],
ytdlOptionsOverrides: '',
clipStart: '',
clipEnd: '',
};
}
describe('DownloadsService', () => {
let socket: MeTubeSocketStub;
let httpMock: HttpTestingController;
let service: DownloadsService;
beforeEach(async () => {
socket = new MeTubeSocketStub();
await TestBed.configureTestingModule({
providers: [
DownloadsService,
provideHttpClient(),
provideHttpClientTesting(),
{ provide: MeTubeSocket, useValue: socket },
],
}).compileComponents();
service = TestBed.inject(DownloadsService);
httpMock = TestBed.inject(HttpTestingController);
});
it('add() posts snake_case fields matching backend', () => {
service.add(basePayload()).subscribe();
const req = httpMock.expectOne('add');
expect(req.request.method).toBe('POST');
expect(req.request.body).toEqual(
expect.objectContaining({
url: 'https://example.com/v',
download_type: 'video',
codec: 'auto',
quality: 'best',
format: 'any',
playlist_item_limit: 0,
auto_start: true,
split_by_chapters: false,
chapter_template: '',
subtitle_language: 'en',
subtitle_mode: 'prefer_manual',
ytdl_options_presets: [],
ytdl_options_overrides: '',
}),
);
req.flush({ status: 'ok' });
});
it('add() sends clip_start and clip_end when set', () => {
service
.add({
...basePayload(),
clipStart: '1:00',
clipEnd: '2:00',
})
.subscribe();
const req = httpMock.expectOne('add');
expect(req.request.body).toEqual(
expect.objectContaining({
clip_start: '1:00',
clip_end: '2:00',
}),
);
req.flush({ status: 'ok' });
});
it('getPresets() fetches configured preset names', () => {
service.getPresets().subscribe((result) => {
expect(result).toEqual({ presets: ['Preset A'] });
});
const req = httpMock.expectOne('presets');
expect(req.request.method).toBe('GET');
req.flush({ presets: ['Preset A'] });
});
it('cancelAdd posts to cancel-add', () => {
service.cancelAdd().subscribe();
const req = httpMock.expectOne('cancel-add');
expect(req.request.method).toBe('POST');
req.flush({ status: 'ok' });
});
it('startById posts ids', () => {
service.startById(['a', 'b']).subscribe();
const req = httpMock.expectOne('start');
expect(req.request.body).toEqual({ ids: ['a', 'b'] });
req.flush({});
});
it('delById marks items deleting and posts delete', () => {
const dl: Download = {
id: '1',
title: 't',
url: 'u1',
download_type: 'video',
quality: 'best',
format: 'any',
folder: '',
custom_name_prefix: '',
playlist_item_limit: 0,
status: 'finished',
msg: '',
percent: 0,
speed: 0,
eta: 0,
filename: '',
checked: false,
deleting: false,
};
service.queue.set('u1', dl);
service.delById('queue', ['u1']).subscribe();
expect(dl.deleting).toBe(true);
const req = httpMock.expectOne('delete');
expect(req.request.body).toEqual({ where: 'queue', ids: ['u1'] });
req.flush({});
});
it('delById resets deleting flag and emits error status on HTTP failure', () => {
const dl: Download = {
id: '1',
title: 't',
url: 'u1',
download_type: 'video',
quality: 'best',
format: 'any',
folder: '',
custom_name_prefix: '',
playlist_item_limit: 0,
status: 'finished',
msg: '',
percent: 0,
speed: 0,
eta: 0,
filename: '',
checked: false,
deleting: false,
};
service.queue.set('u1', dl);
let queueChangedCount = 0;
service.queueChanged.subscribe(() => queueChangedCount++);
let result: unknown;
let threw = false;
service.delById('queue', ['u1']).subscribe({
next: (res) => { result = res; },
error: () => { threw = true; },
});
expect(dl.deleting).toBe(true);
const req = httpMock.expectOne('delete');
req.flush({ msg: 'boom' }, { status: 500, statusText: 'Server Error' });
expect(threw).toBe(false);
expect(dl.deleting).toBe(false);
expect(queueChangedCount).toBeGreaterThan(0);
expect((result as { status: string }).status).toBe('error');
});
it('startById surfaces HTTP errors as a status object instead of throwing', () => {
let result: unknown;
let threw = false;
service.startById(['a']).subscribe({
next: (res) => { result = res; },
error: () => { threw = true; },
});
const req = httpMock.expectOne('start');
req.flush({ msg: 'nope' }, { status: 500, statusText: 'Server Error' });
expect(threw).toBe(false);
expect((result as { status: string }).status).toBe('error');
});
it('handleHTTPError extracts msg from object body', async () => {
const err = new HttpErrorResponse({
error: { msg: 'bad' },
status: 400,
});
const res = await new Promise((resolve) => {
service.handleHTTPError(err).subscribe(resolve);
});
expect((res as { status: string }).status).toBe('error');
expect((res as { msg?: string }).msg).toBe('bad');
});
it('socket all updates queue and done', () => {
const row: Download = {
id: '1',
title: 't',
url: 'u1',
download_type: 'video',
quality: 'best',
format: 'any',
folder: '',
custom_name_prefix: '',
playlist_item_limit: 0,
status: 'pending',
msg: '',
percent: 0,
speed: 0,
eta: 0,
filename: '',
checked: false,
};
const q: [string, Download][] = [['u1', row]];
const d: [string, Download][] = [];
socket.emit('all', JSON.stringify([q, d]));
expect(service.loading).toBe(false);
expect(service.queue.has('u1')).toBe(true);
});
it('socket updated preserves checked and deleting', () => {
service.queue.set('u1', {
id: '1',
title: 't',
url: 'u1',
download_type: 'video',
quality: 'best',
format: 'any',
folder: '',
custom_name_prefix: '',
playlist_item_limit: 0,
status: 'pending',
msg: '',
percent: 0,
speed: 0,
eta: 0,
filename: '',
checked: true,
deleting: true,
});
socket.emit(
'updated',
JSON.stringify({ url: 'u1', title: 't', status: 'downloading' }),
);
const updated = service.queue.get('u1');
expect(updated?.checked).toBe(true);
expect(updated?.deleting).toBe(true);
});
it('socket updated ignores events for urls not already in the queue', () => {
expect(service.queue.has('unknown-url')).toBe(false);
socket.emit(
'updated',
JSON.stringify({ url: 'unknown-url', title: 't', status: 'downloading' }),
);
expect(service.queue.has('unknown-url')).toBe(false);
});
it('socket completed moves entry to done', () => {
service.queue.set('u1', {
id: '1',
title: 't',
url: 'u1',
download_type: 'video',
quality: 'best',
format: 'any',
folder: '',
custom_name_prefix: '',
playlist_item_limit: 0,
status: 'pending',
msg: '',
percent: 0,
speed: 0,
eta: 0,
filename: '',
checked: false,
});
socket.emit('completed', JSON.stringify({ url: 'u1', title: 't', status: 'finished' }));
expect(service.queue.has('u1')).toBe(false);
expect(service.done.has('u1')).toBe(true);
});
it('socket canceled removes from queue', () => {
service.queue.set('u1', {
id: '1',
title: 't',
url: 'u1',
download_type: 'video',
quality: 'best',
format: 'any',
folder: '',
custom_name_prefix: '',
playlist_item_limit: 0,
status: 'pending',
msg: '',
percent: 0,
speed: 0,
eta: 0,
filename: '',
checked: false,
});
socket.emit('canceled', JSON.stringify('u1'));
expect(service.queue.has('u1')).toBe(false);
});
it('socket cleared removes from done', () => {
service.done.set('u1', {
id: '1',
title: 't',
url: 'u1',
download_type: 'video',
quality: 'best',
format: 'any',
folder: '',
custom_name_prefix: '',
playlist_item_limit: 0,
status: 'finished',
msg: '',
percent: 0,
speed: 0,
eta: 0,
filename: '',
checked: false,
});
socket.emit('cleared', JSON.stringify('u1'));
expect(service.done.has('u1')).toBe(false);
});
it('socket configuration updates configuration', () => {
socket.emit('configuration', JSON.stringify({ CUSTOM_DIRS: true }));
expect(service.configuration['CUSTOM_DIRS']).toBe(true);
});
it('socket custom_dirs updates customDirs', () => {
socket.emit('custom_dirs', JSON.stringify({ download_dir: [''] }));
expect(service.customDirs['download_dir']).toEqual(['']);
});
afterEach(() => {
httpMock.verify();
});
});

View file

@ -5,6 +5,26 @@ import { catchError } from 'rxjs/operators';
import { MeTubeSocket } from './metube-socket.service';
import { Download, Status, State } from '../interfaces';
import { takeUntilDestroyed } from '@angular/core/rxjs-interop';
export interface AddDownloadPayload {
url: string;
downloadType: string;
codec: string;
quality: string;
format: string;
folder: string;
customNamePrefix: string;
playlistItemLimit: number;
autoStart: boolean;
splitByChapters: boolean;
chapterTemplate: string;
subtitleLanguage: string;
subtitleMode: string;
ytdlOptionsPresets: string[];
ytdlOptionsOverrides: string;
clipStart?: string;
clipEnd?: string;
}
@Injectable({
providedIn: 'root'
})
@ -14,16 +34,15 @@ export class DownloadsService {
loading = true;
queue = new Map<string, Download>();
done = new Map<string, Download>();
queueChanged = new Subject();
doneChanged = new Subject();
customDirsChanged = new Subject();
ytdlOptionsChanged = new Subject();
configurationChanged = new Subject();
updated = new Subject();
queueChanged = new Subject<void>();
doneChanged = new Subject<void>();
customDirsChanged = new Subject<Record<string, string[]>>();
ytdlOptionsChanged = new Subject<Record<string, unknown>>();
configurationChanged = new Subject<Record<string, unknown>>();
updated = new Subject<void>();
// eslint-disable-next-line @typescript-eslint/no-explicit-any
configuration: any = {};
customDirs = {};
configuration: Record<string, unknown> = {};
customDirs: Record<string, string[]> = {};
constructor() {
this.socket.fromEvent('all')
@ -35,25 +54,31 @@ export class DownloadsService {
data[0].forEach(entry => this.queue.set(...entry));
this.done.clear();
data[1].forEach(entry => this.done.set(...entry));
this.queueChanged.next(null);
this.doneChanged.next(null);
this.queueChanged.next();
this.doneChanged.next();
});
this.socket.fromEvent('added')
.pipe(takeUntilDestroyed())
.subscribe((strdata: string) => {
const data: Download = JSON.parse(strdata);
this.queue.set(data.url, data);
this.queueChanged.next(null);
this.queueChanged.next();
});
this.socket.fromEvent('updated')
.pipe(takeUntilDestroyed())
.subscribe((strdata: string) => {
const data: Download = JSON.parse(strdata);
const dl: Download | undefined = this.queue.get(data.url);
data.checked = !!dl?.checked;
data.deleting = !!dl?.deleting;
// An 'added' event always precedes legitimate updates. If the row is
// gone (canceled/completed already processed), this update is stale —
// applying it would resurrect a ghost row until the next full refresh.
if (!dl) {
return;
}
data.checked = !!dl.checked;
data.deleting = !!dl.deleting;
this.queue.set(data.url, data);
this.updated.next(null);
this.updated.next();
});
this.socket.fromEvent('completed')
.pipe(takeUntilDestroyed())
@ -61,22 +86,22 @@ export class DownloadsService {
const data: Download = JSON.parse(strdata);
this.queue.delete(data.url);
this.done.set(data.url, data);
this.queueChanged.next(null);
this.doneChanged.next(null);
this.queueChanged.next();
this.doneChanged.next();
});
this.socket.fromEvent('canceled')
.pipe(takeUntilDestroyed())
.subscribe((strdata: string) => {
const data: string = JSON.parse(strdata);
this.queue.delete(data);
this.queueChanged.next(null);
this.queueChanged.next();
});
this.socket.fromEvent('cleared')
.pipe(takeUntilDestroyed())
.subscribe((strdata: string) => {
const data: string = JSON.parse(strdata);
this.done.delete(data);
this.doneChanged.next(null);
this.doneChanged.next();
});
this.socket.fromEvent('configuration')
.pipe(takeUntilDestroyed())
@ -103,44 +128,51 @@ export class DownloadsService {
}
handleHTTPError(error: HttpErrorResponse) {
const msg = error.error instanceof ErrorEvent ? error.error.message : error.error;
return of({status: 'error', msg: msg})
const msg = error.error instanceof ErrorEvent
? error.error.message
: (typeof error.error === 'string'
? error.error
: (error.error?.msg || error.message || 'Request failed'));
return of({ status: 'error', msg });
}
public add(
url: string,
quality: string,
format: string,
folder: string,
customNamePrefix: string,
playlistItemLimit: number,
autoStart: boolean,
splitByChapters: boolean,
chapterTemplate: string,
subtitleFormat: string,
subtitleLanguage: string,
subtitleMode: string,
) {
return this.http.post<Status>('add', {
url: url,
quality: quality,
format: format,
folder: folder,
custom_name_prefix: customNamePrefix,
playlist_item_limit: playlistItemLimit,
auto_start: autoStart,
split_by_chapters: splitByChapters,
chapter_template: chapterTemplate,
subtitle_format: subtitleFormat,
subtitle_language: subtitleLanguage,
subtitle_mode: subtitleMode
}).pipe(
public add(payload: AddDownloadPayload) {
const body: Record<string, unknown> = {
url: payload.url,
download_type: payload.downloadType,
codec: payload.codec,
quality: payload.quality,
format: payload.format,
folder: payload.folder,
custom_name_prefix: payload.customNamePrefix,
playlist_item_limit: payload.playlistItemLimit,
auto_start: payload.autoStart,
split_by_chapters: payload.splitByChapters,
chapter_template: payload.chapterTemplate,
subtitle_language: payload.subtitleLanguage,
subtitle_mode: payload.subtitleMode,
ytdl_options_presets: payload.ytdlOptionsPresets,
ytdl_options_overrides: payload.ytdlOptionsOverrides,
};
const cs = payload.clipStart?.trim();
const ce = payload.clipEnd?.trim();
if (cs) body['clip_start'] = cs;
if (ce) body['clip_end'] = ce;
return this.http.post<Status>('add', body).pipe(
catchError(this.handleHTTPError)
);
}
public getPresets() {
return this.http.get<{ presets: string[] }>('presets').pipe(
catchError(() => of({ presets: [] }))
);
}
public startById(ids: string[]) {
return this.http.post('start', {ids: ids});
return this.http.post<Status>('start', {ids: ids}).pipe(
catchError(this.handleHTTPError)
);
}
public delById(where: State, ids: string[]) {
@ -153,7 +185,22 @@ export class DownloadsService {
}
}
}
return this.http.post('delete', {where: where, ids: ids});
return this.http.post<Status>('delete', {where: where, ids: ids}).pipe(
catchError((err: HttpErrorResponse) => {
// Request failed — the rows would otherwise stay disabled forever
// with no way to retry, since nothing ever clears `deleting`.
if (map) {
for (const id of ids) {
const obj = map.get(id);
if (obj) {
obj.deleting = false;
}
}
}
(where === 'queue' ? this.queueChanged : this.doneChanged).next();
return this.handleHTTPError(err);
})
);
}
public startByFilter(where: State, filter: (dl: Download) => boolean) {
@ -167,47 +214,6 @@ export class DownloadsService {
this[where].forEach((dl: Download) => { if (filter(dl)) ids.push(dl.url) });
return this.delById(where, ids);
}
public addDownloadByUrl(url: string): Promise<{
response: Status} | {
status: string;
msg?: string;
}> {
const defaultQuality = 'best';
const defaultFormat = 'mp4';
const defaultFolder = '';
const defaultCustomNamePrefix = '';
const defaultPlaylistItemLimit = 0;
const defaultAutoStart = true;
const defaultSplitByChapters = false;
const defaultChapterTemplate = this.configuration['OUTPUT_TEMPLATE_CHAPTER'];
const defaultSubtitleFormat = 'srt';
const defaultSubtitleLanguage = 'en';
const defaultSubtitleMode = 'prefer_manual';
return new Promise((resolve, reject) => {
this.add(
url,
defaultQuality,
defaultFormat,
defaultFolder,
defaultCustomNamePrefix,
defaultPlaylistItemLimit,
defaultAutoStart,
defaultSplitByChapters,
defaultChapterTemplate,
defaultSubtitleFormat,
defaultSubtitleLanguage,
defaultSubtitleMode,
)
.subscribe({
next: (response) => resolve(response),
error: (error) => reject(error)
});
});
}
public exportQueueUrls(): string[] {
return Array.from(this.queue.values()).map(download => download.url);
}
public cancelAdd() {
return this.http.post<Status>('cancel-add', {}).pipe(
catchError(this.handleHTTPError)
@ -217,19 +223,19 @@ export class DownloadsService {
uploadCookies(file: File) {
const formData = new FormData();
formData.append('cookies', file);
return this.http.post<any>('upload-cookies', formData).pipe(
return this.http.post<{ status: string; msg?: string }>('upload-cookies', formData).pipe(
catchError(this.handleHTTPError)
);
}
deleteCookies() {
return this.http.post<any>('delete-cookies', {}).pipe(
return this.http.post<{ status: string; msg?: string }>('delete-cookies', {}).pipe(
catchError(this.handleHTTPError)
);
}
getCookieStatus() {
return this.http.get<any>('cookie-status').pipe(
return this.http.get<{ status: string; has_cookies: boolean }>('cookie-status').pipe(
catchError(this.handleHTTPError)
);
}

View file

@ -1,3 +1,4 @@
export { DownloadsService } from './downloads.service';
export { SpeedService } from './speed.service';
export { MeTubeSocket } from './metube-socket.service';
export { MeTubeSocket } from './metube-socket.service';
export { ToastService } from './toast.service';
export { BatchUrlsService } from './batch-urls.service';

View file

@ -1,39 +0,0 @@
import { Injectable } from '@angular/core';
import { BehaviorSubject, Observable, interval } from 'rxjs';
import { map } from 'rxjs/operators';
@Injectable({
providedIn: 'root'
})
export class SpeedService {
private speedBuffer = new BehaviorSubject<number[]>([]);
private readonly BUFFER_SIZE = 10; // Keep last 10 measurements (1 second at 100ms intervals)
// Observable that emits the mean speed every second
public meanSpeed$: Observable<number>;
constructor() {
// Calculate mean speed every second
this.meanSpeed$ = interval(1000).pipe(
map(() => {
const speeds = this.speedBuffer.value;
if (speeds.length === 0) return 0;
return speeds.reduce((sum, speed) => sum + speed, 0) / speeds.length;
})
);
}
// Add a new speed measurement
public addSpeedMeasurement(speed: number) {
const currentBuffer = this.speedBuffer.value;
const newBuffer = [...currentBuffer, speed].slice(-this.BUFFER_SIZE);
this.speedBuffer.next(newBuffer);
}
// Get the current mean speed
public getCurrentMeanSpeed(): number {
const speeds = this.speedBuffer.value;
if (speeds.length === 0) return 0;
return speeds.reduce((sum, speed) => sum + speed, 0) / speeds.length;
}
}

View file

@ -0,0 +1,142 @@
import { DestroyRef, inject, Injectable } from '@angular/core';
import { HttpClient, HttpErrorResponse } from '@angular/common/http';
import { of, Subject } from 'rxjs';
import { catchError, tap } from 'rxjs/operators';
import { takeUntilDestroyed } from '@angular/core/rxjs-interop';
import { MeTubeSocket } from './metube-socket.service';
import { SubscriptionRow } from '../interfaces/subscription';
import { Status } from '../interfaces';
import { AddDownloadPayload } from './downloads.service';
export interface SubscribePayload extends AddDownloadPayload {
checkIntervalMinutes: number;
titleRegex: string;
skipSubscriberOnly: boolean;
}
@Injectable({
providedIn: 'root',
})
export class SubscriptionsService {
private http = inject(HttpClient);
private socket = inject(MeTubeSocket);
private destroyRef = inject(DestroyRef);
subscriptions = new Map<string, SubscriptionRow>();
subscriptionsChanged = new Subject<void>();
private publishList(rows: SubscriptionRow[]) {
this.subscriptions.clear();
for (const row of rows) {
this.subscriptions.set(row.id, row);
}
this.subscriptionsChanged.next();
}
constructor() {
this.socket
.fromEvent('subscriptions_all')
.pipe(takeUntilDestroyed(this.destroyRef))
.subscribe((strdata: string) => {
const data: SubscriptionRow[] = JSON.parse(strdata);
this.publishList(data);
});
this.socket
.fromEvent('subscription_added')
.pipe(takeUntilDestroyed(this.destroyRef))
.subscribe((strdata: string) => {
const row: SubscriptionRow = JSON.parse(strdata);
this.subscriptions.set(row.id, row);
this.subscriptionsChanged.next();
});
this.socket
.fromEvent('subscription_updated')
.pipe(takeUntilDestroyed(this.destroyRef))
.subscribe((strdata: string) => {
const row: SubscriptionRow = JSON.parse(strdata);
this.subscriptions.set(row.id, row);
this.subscriptionsChanged.next();
});
this.socket
.fromEvent('subscription_removed')
.pipe(takeUntilDestroyed(this.destroyRef))
.subscribe((strdata: string) => {
const id: string = JSON.parse(strdata);
this.subscriptions.delete(id);
this.subscriptionsChanged.next();
});
}
handleHTTPError(error: HttpErrorResponse) {
const msg =
error.error instanceof ErrorEvent
? error.error.message
: typeof error.error === 'string'
? error.error
: error.error?.msg || error.message || 'Request failed';
return of({ status: 'error' as const, msg });
}
subscribe(payload: SubscribePayload) {
return this.http
.post<Status>('subscribe', {
url: payload.url,
download_type: payload.downloadType,
codec: payload.codec,
quality: payload.quality,
format: payload.format,
folder: payload.folder,
custom_name_prefix: payload.customNamePrefix,
playlist_item_limit: payload.playlistItemLimit,
auto_start: payload.autoStart,
split_by_chapters: payload.splitByChapters,
chapter_template: payload.chapterTemplate,
subtitle_language: payload.subtitleLanguage,
subtitle_mode: payload.subtitleMode,
ytdl_options_presets: payload.ytdlOptionsPresets,
ytdl_options_overrides: payload.ytdlOptionsOverrides,
check_interval_minutes: payload.checkIntervalMinutes,
title_regex: payload.titleRegex,
skip_subscriber_only: payload.skipSubscriberOnly,
})
.pipe(catchError((err) => this.handleHTTPError(err)));
}
delete(ids: string[]) {
return this.http.post('subscriptions/delete', { ids }).pipe(catchError((err) => this.handleHTTPError(err)));
}
update(
id: string,
changes: Partial<
Pick<
SubscriptionRow,
'enabled' | 'check_interval_minutes' | 'name' | 'title_regex' | 'skip_subscriber_only'
>
>,
) {
return this.http
.post('subscriptions/update', { id, ...changes })
.pipe(catchError((err) => this.handleHTTPError(err)));
}
checkNow(ids?: string[]) {
return this.http
.post('subscriptions/check', ids?.length ? { ids } : {})
.pipe(catchError((err) => this.handleHTTPError(err)));
}
fetchList() {
return this.http.get<SubscriptionRow[]>('subscriptions').pipe(catchError(() => of([])));
}
refreshList() {
return this.http.get<SubscriptionRow[]>('subscriptions').pipe(
tap((rows) => this.publishList(rows)),
catchError((err) => this.handleHTTPError(err)),
);
}
}

View file

@ -0,0 +1,86 @@
import { Injectable, signal } from '@angular/core';
export type ToastLevel = 'info' | 'success' | 'error';
export interface ToastAction {
label: string;
value: boolean;
primary?: boolean;
}
export interface Toast {
id: number;
level: ToastLevel;
message: string;
actions?: ToastAction[];
/** Resolver for confirm() toasts; resolved when the user picks an action or dismisses. */
_resolve?: (value: boolean) => void;
}
/**
* Lightweight non-blocking notification service. Replaces the blocking
* window.alert()/confirm() dialogs that previously littered the app component.
*/
@Injectable({ providedIn: 'root' })
export class ToastService {
private counter = 0;
readonly toasts = signal<Toast[]>([]);
info(message: string): void {
this.show('info', message, 4000);
}
success(message: string): void {
this.show('success', message, 4000);
}
error(message: string): void {
this.show('error', message, 8000);
}
/**
* Show a confirmation toast with confirm/cancel actions. Resolves true when
* confirmed, false when cancelled or auto-dismissed.
*/
confirm(message: string, confirmLabel = 'OK', cancelLabel = 'Cancel'): Promise<boolean> {
return new Promise<boolean>((resolve) => {
const id = ++this.counter;
this.toasts.update((list) => [
...list,
{
id,
level: 'info',
message,
actions: [
{ label: cancelLabel, value: false },
{ label: confirmLabel, value: true, primary: true },
],
_resolve: resolve,
},
]);
});
}
respond(id: number, value: boolean): void {
const toast = this.toasts().find((t) => t.id === id);
toast?._resolve?.(value);
this.remove(id);
}
dismiss(id: number): void {
const toast = this.toasts().find((t) => t.id === id);
// A confirm toast dismissed without an explicit choice resolves to false.
toast?._resolve?.(false);
this.remove(id);
}
private remove(id: number): void {
this.toasts.update((list) => list.filter((t) => t.id !== id));
}
private show(level: ToastLevel, message: string, autoDismissMs: number): void {
const id = ++this.counter;
this.toasts.update((list) => [...list, { id, level, message }]);
setTimeout(() => this.remove(id), autoDismissMs);
}
}

View file

@ -5,3 +5,22 @@
[data-bs-theme="dark"] &
background-color: var(--bs-dark-bg-subtle) !important
.ng-select
flex: 1
.ng-select-container
min-height: 38px
.ng-value
white-space: nowrap
overflow: visible
.ng-dropdown-panel
.ng-dropdown-panel-items
max-height: 300px
.ng-option
white-space: nowrap
overflow: visible
text-overflow: ellipsis

View file

@ -12,5 +12,13 @@
],
"exclude": [
"src/**/*.spec.ts"
]
],
"angularCompilerOptions": {
"extendedDiagnostics": {
"checks": {
"nullishCoalescingNotNullable": "suppress",
"optionalChainNotNullable": "suppress"
}
}
}
}

1037
uv.lock generated

File diff suppressed because it is too large Load diff