From 497a174a1e7213e4542c307d542a711cf790c9ca Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Tue, 19 Jul 2022 22:05:59 -0400 Subject: [PATCH 001/115] Add Matrix link to README --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 031dec4..5a79ce3 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # rffmpeg + +Chat on Matrix + + `rffmpeg` is a remote FFmpeg wrapper used to execute FFmpeg commands on a remote server via SSH. It is most useful in situations involving media servers such as Jellyfin (our reference user), where one might want to perform transcoding actions with FFmpeg on a remote machine or set of machines which can better handle transcoding, take advantage of hardware acceleration, or distribute transcodes across multiple servers for load balancing. ## Quick usage From 876a6633261c6e41cd7a5603773c40422e655b93 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Tue, 19 Jul 2022 22:06:31 -0400 Subject: [PATCH 002/115] Center logos --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 5a79ce3..4f677e6 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,10 @@ # rffmpeg +

Chat on Matrix +

`rffmpeg` is a remote FFmpeg wrapper used to execute FFmpeg commands on a remote server via SSH. It is most useful in situations involving media servers such as Jellyfin (our reference user), where one might want to perform transcoding actions with FFmpeg on a remote machine or set of machines which can better handle transcoding, take advantage of hardware acceleration, or distribute transcodes across multiple servers for load balancing. From 7cbe14e80aa5b58f8fb14d8957999ac3af814fc1 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Tue, 19 Jul 2022 22:07:19 -0400 Subject: [PATCH 003/115] Complete refactoring of rffmpeg This commit represents a complete refactoring of rffmpeg while preserving all existing functionality. Several key changes are: 1. A cleaner function tree throughout the code, hopefully making things easier to understand. 2. The use of "-t -t" to the "ssh" command to optimize behaviour when Ctrl+C is used to terminate a test process. 3. The removal of the old PID-based "state" system in favour of a ground-up SQLite-based system which can better track the current status. 4. The renaming of the binary from "rffmpeg.py" to "rffmpeg". 5. The addition of an alternate invocation and corresponding Click-based CLI interface to manage the database, accessible by calling the "rffmpeg" binary name directly instead of an "ffmpeg"/"ffprobe" alias. 6. The moving of host management out of the config file and into the database/Click CLI interface for better management capabilities. 7. The proper defaulting of the configuration; an entirely empty configuration can be specified if desired, using only default options. This new version should be functionally identical to the old version in all cases while providing the above changes. This commit also adjust the documentation to reflect the updated setup and options. --- README.md | 146 +++++----- SETUP.md | 29 +- rffmpeg | 659 +++++++++++++++++++++++++++++++++++++++++++++ rffmpeg.py | 386 -------------------------- rffmpeg.yml.sample | 68 +++-- 5 files changed, 777 insertions(+), 511 deletions(-) create mode 100755 rffmpeg delete mode 100755 rffmpeg.py diff --git a/README.md b/README.md index 4f677e6..d87ee75 100644 --- a/README.md +++ b/README.md @@ -10,109 +10,95 @@ ## Quick usage -1. Install the required Python 3 dependencies `yaml` and `subprocess` (`sudo apt install python3-yaml python3-subprocess` in Debian). +1. Install the required Python 3 dependencies: `click`, `yaml` and `subprocess` (`sudo apt install python3-click python3-yaml python3-subprocess` in Debian). 1. Create the directory `/etc/rffmpeg`. -1. Copy the `rffmpeg.yml.sample` file to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs. +1. Copy the `rffmpeg.yml.sample` file to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. -1. Install `rffmpeg.py` somewhere useful, for instance at `/usr/local/bin/rffmpeg.py`. +1. Install `rffmpeg` somewhere useful, for instance at `/usr/local/bin/rffmpeg`. -1. Create symlinks for the command names `ffmpeg` and `ffprobe` to `rffmpeg.py`, for example `sudo ln -s /usr/local/bin/rffmpeg.py /usr/local/bin/ffmpeg` and `sudo ln -s /usr/local/bin/rffmpeg.py /usr/local/bin/ffprobe`. +1. Create symlinks for the command names `ffmpeg` and `ffprobe` to `rffmpeg`, for example `sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffmpeg` and `sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffprobe`. -1. Set your media program to use `rffmpeg.py` via the symlink names created above, instead of any other `ffmpeg` binary. +1. Initialize the database and add a target host, for example `sudo rffmpeg init && rffmpeg add myhost.domain.tld`. + +1. Set your media program to use `rffmpeg` via the `ffmpeg` symlink name created above, instead of any other `ffmpeg` binary. 1. Profit! -For more detailed instructions, including what must be done to ensure data can be passed between the servers, please see [the SETUP guide](SETUP.md). +For a comprehensive installation tutorial based on a reference setup, please see [the SETUP guide](SETUP.md). -## rffmpeg options and caveats +## Important Considerations -The `rffmpeg.yml.sample` is self-documented for the most part. Some additional important information you might need is presented below. +### The rffmpeg Configuration file -### Remote hosts +The rffmpeg configuration file located at `rffmpeg.yml.sample` is an example that shows all default options. Even though this file is effectively "empty", it *must* be present at `/etc/rffmpeg/rffmpeg.yml` or at an alternative location specified by the environment variable `RFFMPEG_CONFIG`; the latter is only useful for testing, as media programs like Jellyfin provide no way to specify this. -rffmpeg supports setting multiple hosts. It keeps state in `/run/shm/rffmpeg` of all running processes, and these state files are used during rffmpeg's initialization in order to determine the optimal target host. rffmpeg will run through these hosts sequentially, choosing the one with the fewest running rffmpeg jobs. This helps distribute the transcoding load across multiple servers, and can also provide redundancy if one of the servers is offline - rffmpeg will detect if a host is unreachable and set it "bad" for the remainder of the run, thus skipping it until the process completes. +To override a default option, simply uncomment the relevant line and adjust it to suit your needs. For those using [Jellyfin](https://jellyfin.org) and following the [SETUP guide](SETUP.md), no default options will need to be changed.o -Hosts can also be assigned weights (see `rffmpeg.yml.sample` for an example) that allow the host to take on that many times the number of active processes versus weight-1 hosts. The `rffmpeg` process does a floor division of the number of active processes on a host with that host weight to determine its "weighted [process] count", which is then used instead to determine the lease-loaded host to use. Note that `rffmpeg` does not take into account actual system load, etc. when determining which host to use; it treats each running command equally regardless of how intensive it actually is. +Each option has an explanatory comment above it detailing its purpose. -#### Host lists +Since the configuration file is YAML, ensure that you do not use "Tab" characters inside of it, only spaces. -Hosts are specified as a YAML list in the relevant section of `rffmpeg.yml`, with one list entry per target. A single list entry can be specfied in one of two ways. Either a direct list value of the hostame/IP: +### Initializing Rffmpeg -``` -- myhostname.domain.tld -``` +After first installing rffmpeg, ensure you initialize the database with the `sudo rffmpeg init` command. Note that `sudo` is required here to create the required data paths, but afterwards, `rffmpeg` can be run by anyone in the configured group. -Or as a fully expanded `name:`/`weight:` pair. +Rffmpeg is a Click-based application; thus, all commands have a `-h` or `--help` flag to show usage and additional options that may be specified. -``` -- name: myhostname.domain.tld - weight: 2 -``` +### Viewing Status -The first, direct list value formatting implies `weight: 1`. Examples of both styles can be found in the same configuration. +Once installed and initialized, the status of the rffmpeg system can be viewed with the command `rffmpeg status`. This will show all configured target hosts, their states, and any active commands being run. -You can get creative with this list, especially since `rffmpeg` always checks the list in order to find the next available host. For an example of a complex setup, if you had 3 hosts, and wanted 1+2+2 processes, the following would be the default way to acheive this: +### Adding or Removing Target Hosts -``` -- name: host1 - weight: 1 -- name: host2 - weight: 2 -- name: host3 - weight: 2 -``` +To add a target host, use the command `rffmpeg add`. This command takes the optional `-w`/`--weight` flag to adjust the weight of the target host (see below). A host can be added more than once. -This would however spread processes out like this, which might work well, but might not for some usecases: +To remove a target host, use the command `rffmpeg remove`. This command takes either a target host name/IP, which affects all instances of that name, or a specific host ID. Removing an in-use target host will not terminate any running processes, though it may result in undefined behaviour within rffmpeg. Before removing a host it is best to ensure there is nothing using it. -``` -proc1: host1 -proc2: host2 -proc3: host2 -proc4: host3 -proc5: host3 -proc6: host1 -etc. -``` - -You could instead specify the hosts like this: - -``` -- host1 -- host2 -- host3 -- host2 -- host3 -``` - -Which would instead give a process spread like: - -``` -proc1: host1 -proc2: host2 -proc3: host3 -proc4: host2 -proc5: host3 -proc6: host1 -etc. -``` - -Experiment with the ordering based on your load and usecase. - -#### Localhost and fallback +### Localhost and Fallback If one of the hosts in the config file is called "localhost", rffmpeg will run locally without SSH. This can be useful if the local machine is also a powerful transcoding device. In addition, rffmpeg will fall back to "localhost" should it be unable to find any working remote hosts. This helps prevent situations where rffmpeg cannot be run due to none of the remote host(s) being available. -In both cases, note that, if hardware acceleraton is configured, it *must* be available on the local host as well, or the `ffmpeg` commands will fail. There is no easy way around this without rewriting flags, and this is currently out-of-scope for `rffmpeg`. You should always use a lowest-common-denominator approach when deciding on what additional option(s) to enable, such that any configured host can run any process. +In both cases, note that, if hardware acceleraton is configured, it *must* be available on the local host as well, or the `ffmpeg` commands will fail. There is no easy way around this without rewriting arguments, and this is currently out-of-scope for `rffmpeg`. You should always use a lowest-common-denominator approach when deciding on what additional option(s) to enable, such that any configured host can run any process. -The exact path to the local `ffmpeg` and `ffprobe` binaries can be overridden in the configuration, should their paths not match those of the remote system(s). If these options are not specified, the remote paths are used. +The exact path to the local `ffmpeg` and `ffprobe` binaries can be overridden in the configuration, should their paths not match those of the remote system(s). -### Terminating rffmpeg +### Target Host Selection -When running rffmpeg manually, *do not* exit it with `Ctrl+C`. Doing so will likely leave the `ffmpeg` process running on the remote machine. Instead, enter `q` and a newline ("Enter") into the rffmpeg process, and this will terminate the entire command cleanly. This is the method that Jellyfin uses to communicate the termination of an `ffmpeg` process. +When more than one target host is present, rffmpeg uses the following rules to select a new host: + +1. Any hosts marked `bad` are ignored until that marking is cleared. + +1. All remaining hosts are iterated through in an indetermine order (Python dictionary with root key as the host ID). For each host: + + a. If the host is not `localhost`/`127.0.0.1`, it is tested to ensure it is reachable (responds to `ffmpeg -version` over SSH). If it is not reachable, it is marked `bad` for the duration of this processes' runtime and skipped. + + b. If the host is `idle` (has no running processes), it is immediately chosen and the iteration stops. + + c. If it is active, it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role. + +1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. Rffmpeg will then begin running against this host. + +### Target Host Weights and Duplicated Target Hosts + +When adding a host to rffmpeg, a weight can be specified. Weights are used during the calculation of the fewest number of processes among hosts. The actual number of processes running on the host is floor divided (rounded down to the nearest divisible integer) by the weight to give a "weighted count", which is then used in the determination. This option allows one host to take on more processes than other nodes, as it will be chosen as the "least busy" host more often. + +For example, consider two hosts: `host1` with weight 1, and `host2` with weight 5. `host2` would have its actual number of processes floor divided by `5`, and thus any number of processes under `5` would count as `0`, any number of processes between `5` and `10` would count as `1`, and so on, resulting in `host2` being chosen over `host1` even if it had several processes. Thus, `host2` would on average handle 5x more `ffmpeg` processes than `host1` would. + +Host weighting is a fairly blunt instrument, and only becomes important when many simultaneous `ffmpeg` processes/transcodes are occurring at once. Generally leaving all hosts at weight 1 would be sufficient for most usecases. + +Furthermore, it is possible to add a host of the same name more than once in the `rffmpeg add` command. This has a very similar, but subtly different, effect from setting a higher weight. A host present in the list is more likely to be seen before another host, and thus this can further influence the desired target. + +### `bad` Hosts + +As mentioned above under [Target Host Selection](README.md#target-host-selection), a host can be marked `bad` if it does not respond to an `ffmpeg -version` command in at least 1 second. This can happen because a host is offline, unreachable, overloaded, or otherwise unresponsive. + +Once a host is marked `bad`, it will remain so for as long as the `rffmpeg` process that marked it `bad` is running. This can last from a few seconds to several tens of minutes. During this time, any new `rffmpeg` processes that start will see the host marked as `bad` and thus skip it for target selection. Once the marking `rffmpeg` process completes or is terminated, the `bad` status of that host will be cleared, allowing the next run to try it again. This strikes a balance between always retrying known-unresponsive hosts over and over, and ensuring that hosts will eventually be retried. + +If for some reason all configured hosts are marked `bad`, fallback will be engaged; see the above section [Localhost and Fallback](README.md#localhost-and-fallback) for details on what occurrs in this situation. An explicit `localhost` host entry cannot be marked `bad`. ## FAQ @@ -124,11 +110,11 @@ My virtualization setup (multiple 1U nodes with lots of live migration/failover) This depends on what "layer" you're asking at. -* Media Servers: Jellyfin is officially supported; Emby seems to work fine, with caveats (see [Issue #10](https://github.com/joshuaboniface/rffmpeg/issues/10)); no others have been tested to my knowledge -* Operating Systems (source): Debian and its derivatives (Ubuntu, Linux Mint, etc.) should all work perfectly; other Linux operating systems should work fine too as the principles are the same; MacOS should work since it has an SSH client built in; Windows might work if it has an SSH client installed -* Operating Systems (target): Any Linux system which [`jellyfin-ffmpeg`](https://github.com/jellyfin/jellyfin-ffmpeg) supports, which is currently just Debian and Ubuntu; Windows *might* work if you can get an SSH server running on it (see [Issue #17](https://github.com/joshuaboniface/rffmpeg/issues/17)) -* Install Methods for Jellyfin: Native packages/installers/archives are recommended; Docker containers can be made to work by exporting the `/config` path (see [the setup guide](SETUP.md)) but this is slightly more difficult and is not explicitly covered in the guide -* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by @BasixKOR +* Media Servers: Jellyfin is officially supported; Emby seems to work fine, with caveats (see [Issue #10](https://github.com/joshuaboniface/rffmpeg/issues/10)); no others have been tested to my knowledge. +* Operating Systems (source): Debian and its derivatives (Ubuntu, Linux Mint, etc.) should all work perfectly; other Linux operating systems should work fine too as the principles are the same; MacOS should work since it has an SSH client built in; Windows will not work as `rffmpeg` depends on some POSIX assumptions internally. +* Operating Systems (target): Any Linux system which [`jellyfin-ffmpeg`](https://github.com/jellyfin/jellyfin-ffmpeg) supports, which is currently just Debian and Ubuntu; Windows *might* work if you can get an SSH server running on it (see [Issue #17](https://github.com/joshuaboniface/rffmpeg/issues/17)). +* Install Methods for Jellyfin: Native packages/installers/archives are recommended; Docker containers can be made to work by exporting the `/config` path (see [the setup guide](SETUP.md)) but this is slightly more difficult and is not explicitly covered in the guide. +* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by @BasixKOR. ### Can `rffmpeg` mangle/alter FFMPEG arguments? @@ -136,10 +122,12 @@ Explicitly *no*. `rffmpeg` is not designed to interact with the arguments that t This has a number of side effects: - * `rffmpeg` does not know whether hardware acceleration is turned on or not (see above caveats about localhost and fallback) + * `rffmpeg` does not know whether hardware acceleration is turned on or not (see above caveats under [Localhost and Fallback](README.md#localhost-and-fallback) * `rffmpeg` does not know what media is playing or where it's outputting files to, and cannot alter these paths * `rffmpeg` cannot turn on or off special `ffmpeg` options depending on the host selected +Thus it is imperitive that you set up your entire system correctly for `rffmpeg` to work. Please see the [SETUP guide](SETUP.md) for more information. + ### Can `rffmpeg` do Wake-On-LAN or other similar options to turn on a transcode server? Right now, no. I've thought about implementing this more than once (most recently, in response to [Issue #21](https://github.com/joshuaboniface/rffmpeg/issues/21)) but ultimately I've never though this was worth the complexity and delays in spwaning that it would add to the tool. That issue does provide one example of a workaround wrapper script that could accomplish this, but I don't see it being a part of the actual tool itself. @@ -150,7 +138,7 @@ First, run though the setup guide again and make sure that everything is set up If the problem persists, please check the [closed issues](https://github.com/joshuaboniface/rffmpeg/issues?q=is%3Aissue+sort%3Aupdated-desc+is%3Aclosed) and see if it's been reported before (if it's regarding Emby and you get an "error 127", see [Issue #10](https://github.com/joshuaboniface/rffmpeg/issues/10)). -If it hasn't, please open a new issue. Ensure you: +If it hasn't, you can [ask in our chat](https://matrix.to/#/#rffmpeg:matrix.org) or open a new issue. Ensure you: 1. Use a descriptive and useful title that quickly explains the problem. @@ -166,4 +154,4 @@ Absolutely - I'm happy to take pull requests. Though please refer to the "Can `r ### Can you help me set up my server? -I'm always happy to help, though please ensure you try to follow the setup guide first. I can be found [on Matrix](https://matrix.to/#/@joshuaboniface:bonifacelabs.ca) or via email at `joshua@boniface.me`. Please note though that I may be unresponsive sometimes, though I will get back to you eventually I promise! Please don't open Issues here about setup problems; the Issue tracker is for bugs or feature requests instead. +I'm always happy to help, though please ensure you try to follow the setup guide first. Support can be found [on Matrix](https://matrix.to/#/#rffmpeg:matrix.org) or via email at `joshua@boniface.me`. Please note though that I may be unresponsive sometimes, though I will get back to you eventually I promise! Please don't open Issues here about setup problems; the Issue tracker is for bugs or feature requests instead. diff --git a/SETUP.md b/SETUP.md index 610eafb..60d63a1 100644 --- a/SETUP.md +++ b/SETUP.md @@ -1,6 +1,6 @@ # Example Setup Guide -This example setup is the one I use for `rffmpeg` with Jellyfin. It uses 2 servers: a media server running Jellyfin called `jellyfin1`, and a remote transcode server called `transcode1`. Both systems run Debian GNU/Linux, though the commands below should also work on Ubuntu. Throughout this guide I assume you are running as an unprivileged user with `sudo` privileges. +This example setup is the one I use for `rffmpeg` with Jellyfin. It uses 2 servers: a media server running Jellyfin called `jellyfin1`, and a remote transcode server called `transcode1`. Both systems run Debian GNU/Linux, though the commands below should also work on Ubuntu. Throughout this guide I assume you are running as an unprivileged user with `sudo` privileges (i.e. in the group `sudo`). Basic knowledge of Linux CLI usage is assumed. This guide is provided as a basic starting point - there are myriad possible combinations of systems, and I try to keep `rffmpeg` quite flexible. Feel free to experiment. @@ -40,18 +40,21 @@ This guide is provided as a basic starting point - there are myriad possible com jellyfin1 $ sudo -u jellyfin cp -a ${jellyfin_data_path}/.ssh/id_${keytype}.pub ${jellyfin_data_path}/.ssh/authorized_keys ``` + It is important that you do not alter the permissions under this `.ssh` directory or this can cause SSH to fail later. The SSH *must* occur as the `jellyfin` user for this to work. + 1. Scan and save the SSH host key of the transcode server(s), to avoid a prompt later: ``` jellyfin1 $ ssh-keyscan transcode1 | sudo -u jellyfin tee -a ${jellyfin_data_path}/.ssh/known_hosts ``` - * **NOTE:** Ensure you use the exact name here that you will use in `rffmpeg.yml` in the next step. If this is an FQDN (e.g. `jellyfin1.mydomain.tld`) or an IP (e.g. `192.168.0.101`) instead of a short name, use that instead in this command, or repeat it for every possible option (it doesn't hurt). + * **NOTE:** Ensure you use the exact name here that you will use in `rffmpeg`. If this is an FQDN (e.g. `jellyfin1.mydomain.tld`) or an IP (e.g. `192.168.0.101`) instead of a short name, use that instead in this command, or repeat it for every possible option (it doesn't hurt). -1. Install the required dependencies of `rffmpeg`: +1. Install the required Python3 dependencies of `rffmpeg`: ``` jellyfin1 $ sudo apt -y install python3-yaml + jellyfin1 $ sudo apt -y install python3-click jellyfin1 $ sudo apt -y install python3-subprocess ``` @@ -61,20 +64,26 @@ This guide is provided as a basic starting point - there are myriad possible com ``` jellyfin1 $ git clone https://github.com/joshuaboniface/rffmpeg # or download the files manually - jellyfin1 $ sudo cp rffmpeg/rffmpeg.py /usr/local/bin/rffmpeg.py - jellyfin1 $ sudo chmod +x /usr/local/bin/rffmpeg.py - jellyfin1 $ sudo ln -s /usr/local/bin/rffmpeg.py /usr/local/bin/ffmpeg - jellyfin1 $ sudo ln -s /usr/local/bin/rffmpeg.py /usr/local/bin/ffprobe + jellyfin1 $ sudo cp rffmpeg/rffmpeg /usr/local/bin/rffmpeg + jellyfin1 $ sudo chmod +x /usr/local/bin/rffmpeg + jellyfin1 $ sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffmpeg + jellyfin1 $ sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffprobe ``` -1. Create a directory for the `rffmpeg` configuration at `/etc/rffmpeg`, then copy `rffmpeg.yml.sample` to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs. +1. Create a directory for the `rffmpeg` configuration at `/etc/rffmpeg`, then copy `rffmpeg.yml.sample` to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. Generally, if you're following this guide exactly, you will not need to adjust anything in `rffmpeg.yml`. + ``` jellyfin1 $ sudo mkdir -p /etc/rffmpeg jellyfin1 $ sudo cp rffmpeg/rffmpeg.yml.sample /etc/rffmpeg/rffmpeg.yml - jellyfin1 $ sudo $EDITOR /etc/rffmpeg/rffmpeg.yml # edit it to suit your needs + jellyfin1 $ sudo $EDITOR /etc/rffmpeg/rffmpeg.yml # if required ``` - Generally, if you're following this guide exactly, the only part that needs to be modified is the `rffmpeg` -> `remote` -> `hosts` section, where you define the target hosts. For more detail on weights, see the main [README.md](README.md#remote-hosts). +1. Initialize `rffmpeg` (note the `sudo` command) and add at least one remote host to it. You can add multiple hosts now or later, set weights of hosts, and add a host more than once. For full details see the [main README](README.md). + + ``` + jellyfin1 $ sudo rffmpeg init --yes + jellyfin1 $ rffmpeg add --weight 1 gpu1 + ``` 1. Install the NFS kernel server. We will use NFS to export the various required directories so the transcode machine can read from and write to them. diff --git a/rffmpeg b/rffmpeg new file mode 100755 index 0000000..7dee95e --- /dev/null +++ b/rffmpeg @@ -0,0 +1,659 @@ +#!/usr/bin/env python3 + +# rffmpeg.py - Remote FFMPEG transcoding wrapper +# +# Copyright (C) 2019-2022 Joshua M. Boniface +# and Contributors. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +############################################################################### + +import click +import logging +import os +import signal +import sys +import yaml + +from contextlib import contextmanager +from grp import getgrnam +from pathlib import Path +from pwd import getpwnam +from re import search +from sqlite3 import connect as sqlite_connect +from subprocess import run + + +# Set up the logger +log = logging.getLogger("rffmpeg") + + +# Open a database connection (context manager) +@contextmanager +def dbconn(config): + """ + Open a database connection. + """ + conn = sqlite_connect(config["db_path"]) + conn.execute("PRAGMA foreign_keys = 1") + cur = conn.cursor() + yield cur + conn.commit() + conn.close() + + +def fail(msg): + """ + Output an error message and terminate. + """ + log.error(msg) + exit(1) + + +def load_config(): + """ + Parse the YAML configuration file (either /etc/rffmpeg/rffmpeg.yml or specified by the envvar + RFFMPEG_CONFIG) and return a standard dictionary of configuration values. + """ + + default_config_file = "/etc/rffmpeg/rffmpeg.yml" + config_file = os.environ.get("RFFMPEG_CONFIG", default_config_file) + + with open(config_file, "r") as cfgfh: + try: + o_config = yaml.load(cfgfh, Loader=yaml.SafeLoader) + except Exception as e: + fail("Failed to parse configuration file: {}".format(e)) + + config = dict() + + # Parse the base group ("rffmpeg") + config_base = o_config.get("rffmpeg", dict()) + if not config_base: + fail("Failed to parse configuration file top level key 'rffmpeg'.") + + # Parse the logging group ("rffmpeg" -> "logging") + config_logging = config_base.get("logging", dict()) + if config_logging is None: + config_logging = dict() + + # Parse the directories group ("rffmpeg" -> "directories") + config_directories = config_base.get("directories", dict()) + if config_directories is None: + config_directories = dict() + + # Parse the remote group ("rffmpeg" -> "remote") + config_remote = config_base.get("remote", dict()) + if config_remote is None: + config_remote = dict() + + # Parse the commands group ("rffmpeg" -> "commands") + config_commands = config_base.get("commands", dict()) + if config_commands is None: + config_commands = dict() + + # Parse the keys from the logging group + config["log_to_file"] = config_logging.get("log_to_file", True) + config["logfile"] = config_logging.get("logfile", "/var/log/jellyfin/rffmpeg.log") + + # Parse the keys from the state group + config["state_dir"] = config_directories.get("state", "/var/lib/rffmpeg") + config["persist_dir"] = config_directories.get("persist", "/run/shm") + config["dir_owner"] = config_directories.get("owner", "jellyfin") + config["dir_group"] = config_directories.get("group", "sudo") + + # Parse the keys from the remote group + config["remote_user"] = config_remote.get("user", "jellyfin") + config["remote_args"] = config_remote.get("args", ["-i", "/var/lib/jellyfin/.ssh/id_rsa"]) + config["persist_time"] = config_remote.get("persist", 300) + + # Parse the keys from the commands group + config["ssh_command"] = config_commands.get("ssh", "/usr/bin/ssh") + config["pre_commands"] = config_commands.get("pre", []) + config["ffmpeg_command"] = config_commands.get("ffmpeg", "/usr/lib/jellyfin-ffmpeg/ffmpeg") + config["ffprobe_command"] = config_commands.get("ffprobe", "/usr/lib/jellyfin-ffprobe/ffprobe") + config["fallback_ffmpeg_command"] = config_commands.get("fallback_ffmpeg", "/usr/lib/jellyfin-ffmpeg/ffmpeg") + config["fallback_ffprobe_command"] = config_commands.get("fallback_ffprobe", "/usr/lib/jellyfin-ffprobe/ffprobe") + + # Set the database path + config["db_path"]= config["state_dir"] + "/rffmpeg.db" + + # Set a list of special flags that cause different behaviour + config["special_flags"] = ["-version", "-encoders", "-decoders", "-hwaccels", "-filters", "-h"] + + # Set the current PID of this process + config["current_pid"] = os.getpid() + + return config + + +def cleanup(signum="", frame=""): + """ + Clean up this processes stored transient data. + """ + global config, p + + with dbconn(config) as cur: + cur.execute("DELETE FROM states WHERE process_id = ?", (config["current_pid"],)) + cur.execute("DELETE FROM processes WHERE process_id = ?", (config["current_pid"],)) + + +def generate_ssh_command(config, target_host): + """ + Generate an SSH command for use. + """ + ssh_command = list() + + # Add SSH component + ssh_command.append(config["ssh_command"]) + ssh_command.append("-q") + ssh_command.append("-t") + ssh_command.append("-t") + + # Set our connection details + ssh_command.extend(["-o", "ConnectTimeout=1"]) + ssh_command.extend(["-o", "ConnectionAttempts=1"]) + ssh_command.extend(["-o", "StrictHostKeyChecking=no"]) + ssh_command.extend(["-o", "UserKnownHostsFile=/dev/null"]) + + # Use SSH control persistence to keep sessions alive for subsequent commands + if config["persist_time"] > 0: + ssh_command.extend(["-o","ControlMaster=auto"]) + ssh_command.extend(["-o","ControlPath={}/ssh-%r@%h:%p".format(config["persist_dir"])]) + ssh_command.extend(["-o","ControlPersist={}".format(config["persist_time"])]) + + # Add the remote config args + for arg in config["remote_args"]: + if arg: + ssh_command.append(arg) + + # Add user+host string + ssh_command.append("{}@{}".format(config["remote_user"], target_host)) + + return ssh_command + + +def run_command(command, stdin, stdout, stderr): + """ + Execute the command using subprocess. + """ + log.info("Executing command") + p = run( + command, + shell=False, + bufsize=0, + universal_newlines=True, + stdin=stdin, + stdout=stdout, + stderr=stderr + ) + return p.returncode + + +def get_target_host(config): + """ + Determine an optimal target host via data on currently active processes and states. + """ + # Select all hosts and active processes from the database + with dbconn(config) as cur: + hosts = cur.execute("SELECT * FROM hosts").fetchall() + processes = cur.execute("SELECT * FROM processes").fetchall() + + # Generate a mapping dictionary of hosts and processes + host_mappings = dict() + for host in hosts: + hid, hostname, weight = host + + # Get the latest state + with dbconn(config) as cur: + current_state = cur.execute("SELECT * FROM states WHERE host_id = ? ORDER BY id DESC", (hid,)).fetchone() + + if not current_state: + current_state = "idle" + else: + current_state = current_state[3] + + # Create the mappings entry + host_mappings[hid] = { + "hostname": hostname, + "weight": weight, + "current_state": current_state, + "commands": [proc[2] for proc in processes if proc[1] == hid] + } + + lowest_count = 9999 + target_hid = None + target_hostname = None + # For each host in the mapping, let's determine if it is suitable + for hid, host in host_mappings.items(): + # If it's marked as bad, continue + if host["current_state"] == "bad": + continue + + # Try to connect to the host and run a very quick command to determine if it is workable + if host["hostname"] not in ["localhost", "127.0.0.1"]: + test_ssh_command = generate_ssh_command(config, host["hostname"]) + test_ffmpeg_command = [config["ffmpeg_command"], "-version"] + retcode = run_command(test_ssh_command + test_ffmpeg_command, None, None, None) + if retcode != 0: + # Mark the host as bad + with dbconn(config) as cur: + log.info("Marking host {} as bad due to retcode {}".format(host["hostname"], retcode)) + cur.execute("INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", (hid, config["current_pid"], "bad")) + continue + + # If the host state is idle, we can use it immediately + if host["current_state"] == "idle": + target_hid = hid + target_hostname = host["hostname"] + break + + # Get the modified count of the host + raw_proc_count = len(host["commands"]) + weighted_proc_count = raw_proc_count // host["weight"] + + # If this host is currently the least used, provisionally set it as the target + if weighted_proc_count < lowest_count: + lowest_count = weighted_proc_count + target_hid = hid + target_hostname = host["hostname"] + + log.info("Found optimal host '{}' (ID '{}')".format(target_hostname, target_hid)) + return target_hid, target_hostname + + +def run_local_ffmpeg(config, ffmpeg_args): + """ + Run ffmpeg locally, either because "localhost" is the target host, or because no good target + host was found by get_target_host(). + """ + rffmpeg_ffmpeg_command = list() + + # Prepare our default stdin/stdout/stderr + stdin = sys.stdin + stderr = sys.stderr + + if cmd_name == "ffprobe": + # If we're in ffprobe mode use that command and sys.stdout as stdout + rffmpeg_ffmpeg_command.append(config["fallback_ffprobe_command"]) + stdout = sys.stdout + else: + # Otherwise, we use stderr as stdout + rffmpeg_ffmpeg_command.append(config["fallback_ffmpeg_command"]) + stdout = sys.stderr + + # Check for special flags that override the default stdout + if any(item in config["special_flags"] for item in ffmpeg_args): + stdout = sys.stdout + + # Append all the passed arguments directly + for arg in ffmpeg_args: + rffmpeg_ffmpeg_command.append("{}".format(arg)) + + log.info("Local command: {}".format(" ".join(rffmpeg_ffmpeg_command))) + + with dbconn(config) as cur: + cur.execute("INSERT INTO processes (host_id, process_id, cmd) VALUES (?, ?, ?)", (0, config["current_pid"], cmd_name + " " + " ".join(ffmpeg_args))) + cur.execute("INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", (0, config["current_pid"], "active")) + + return run_command(rffmpeg_ffmpeg_command, stdin, stdout, stderr) + + +def run_remote_ffmpeg(config, target_hid, target_host, ffmpeg_args): + """ + Run ffmpeg against the remote target_host. + """ + rffmpeg_ssh_command = generate_ssh_command(config, target_host) + rffmpeg_ffmpeg_command = list() + + # Add any pre commands + for cmd in config["pre_commands"]: + if cmd: + rffmpeg_ffmpeg_command.append(cmd) + + # Prepare our default stdin/stderr + stdin = sys.stdin + stderr = sys.stderr + + if cmd_name == "ffprobe": + # If we're in ffprobe mode use that command and sys.stdout as stdout + rffmpeg_ffmpeg_command.append(config["ffprobe_command"]) + stdout = sys.stdout + else: + # Otherwise, we use stderr as stdout + rffmpeg_ffmpeg_command.append(config["ffmpeg_command"]) + stdout = sys.stderr + + # Check for special flags that override the default stdout + if any(item in config["special_flags"] for item in ffmpeg_args): + stdout = sys.stdout + + # Append all the passed arguments with requoting of any problematic characters + for arg in ffmpeg_args: + # Match bad shell characters: * ' ( ) | [ ] or whitespace + if search("[*'()|\[\]\s]", arg): + rffmpeg_ffmpeg_command.append('"{}"'.format(arg)) + else: + rffmpeg_ffmpeg_command.append("{}".format(arg)) + + log.info("Remote command: {}".format(" ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command))) + + with dbconn(config) as cur: + cur.execute("INSERT INTO processes (host_id, process_id, cmd) VALUES (?, ?, ?)", (target_hid, config["current_pid"], cmd_name + " " + " ".join(ffmpeg_args))) + cur.execute("INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", (target_hid, config["current_pid"], "active")) + + return run_command(rffmpeg_ssh_command + rffmpeg_ffmpeg_command, stdin, stdout, stderr) + + +def run_ffmpeg(config, ffmpeg_args): + """ + Entrypoint for an ffmpeg/ffprobe aliased process. + """ + signal.signal(signal.SIGTERM, cleanup) + signal.signal(signal.SIGINT, cleanup) + signal.signal(signal.SIGQUIT, cleanup) + signal.signal(signal.SIGHUP, cleanup) + + if config["log_to_file"]: + logging.basicConfig( + filename=config["logfile"], + level=logging.INFO, + format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s" + ) + else: + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s" + ) + + log.info("Starting rffmpeg with args: {}".format(" ".join(ffmpeg_args))) + + target_hid, target_hostname = get_target_host(config) + + if not target_hostname or target_hostname == "localhost": + retcode = run_local_ffmpeg(config, ffmpeg_args) + else: + retcode = run_remote_ffmpeg(config, target_hid, target_hostname, ffmpeg_args) + + cleanup() + if retcode == 0: + log.info("Finished rffmpeg with return code {}".format(retcode)) + else: + log.error("Finished rffmpeg with return code {}".format(retcode)) + exit(retcode) + + +def run_control(config): + """ + Entrypoint for the Click CLI for managing the rffmpeg system. + """ + CONTEXT_SETTINGS = dict(help_option_names=["-h", "--help"], max_content_width=120) + + @click.group(context_settings=CONTEXT_SETTINGS) + def rffmpeg_click(): + """ + rffmpeg CLI interface + """ + pass + + @click.command(name="init", short_help="Initialize the system.") + @click.option( + "-y", + "--yes", + "confirm_flag", + is_flag=True, + default=False, + help="Confirm initialization." + ) + def rffmpeg_click_init(confirm_flag): + """ + Initialize the rffmpeg system and database; this will erase all hosts and current state. + + This command should be run as "sudo" before any attempts to use rffmpeg. + """ + if os.getuid() != 0: + click.echo("Error: This command requires root privileges.") + exit(1) + + if not confirm_flag: + try: + click.confirm( + "Are you sure you want to (re)initalize the database", + prompt_suffix="? ", + abort=True + ) + except Exception: + fail("Aborting due to failed confirmation.") + + if not Path(config["state_dir"]).is_dir(): + try: + os.makedirs(config["state_dir"]) + except OSError as e: + fail("Failed to create state directory '{}': {}".format(config['state_dir'], e)) + + if Path(config["db_path"]).is_file(): + os.remove(config["db_path"]) + + try: + with dbconn(config) as cur: + cur.execute( + """CREATE TABLE hosts (id INTEGER PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1)""" + ) + cur.execute( + """CREATE TABLE processes (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)""" + ) + cur.execute( + """CREATE TABLE states (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)""" + ) + except Exception as e: + fail("Failed to create database: {}".format(e)) + + os.chown(config["state_dir"], getpwnam(config["dir_owner"]).pw_uid, getgrnam(config["dir_group"]).gr_gid) + os.chmod(config["state_dir"], 0o770) + os.chown(config["db_path"], getpwnam(config["dir_owner"]).pw_uid, getgrnam(config["dir_group"]).gr_gid) + os.chmod(config["db_path"], 0o660) + + rffmpeg_click.add_command(rffmpeg_click_init) + + @click.command(name="status", short_help="Show hosts and status.") + def rffmpeg_click_status(): + """ + Show the current status of all rffmpeg target hosts and active processes. + """ + with dbconn(config) as cur: + hosts = cur.execute("SELECT * FROM hosts").fetchall() + processes = cur.execute("SELECT * FROM processes").fetchall() + states = cur.execute("SELECT * FROM states").fetchall() + + # Determine if there are any fallback processes running + fallback_processes = list() + for process in processes: + if process[1] == 0: + fallback_processes.append(process) + + # Generate a mapping dictionary of hosts and processes + host_mappings = dict() + + if len(fallback_processes) > 0: + host_mappings[0] = { + "hostname": "localhost (fallback)", + "weight": 0, + "current_state": "fallback", + "commands": fallback_processes + } + + for host in hosts: + hid, hostname, weight = host + + # Get the latest state + with dbconn(config) as cur: + current_state = cur.execute("SELECT * FROM states WHERE host_id = ? ORDER BY id DESC", (hid,)).fetchone() + + if not current_state: + current_state = "idle" + else: + current_state = current_state[3] + + # Create the mappings entry + host_mappings[hid] = { + "hostname": hostname, + "weight": weight, + "current_state": current_state, + "commands": [proc for proc in processes if proc[1] == hid] + } + + hostname_length = 9 + hid_length = 3 + weight_length = 7 + state_length = 6 + for hid, host in host_mappings.items(): + if len(host["hostname"]) + 1 > hostname_length: + hostname_length = len(host["hostname"]) + 1 + if len(str(hid)) + 1 > hid_length: + hid_length = len(str(hid)) + 1 + if len(str(host["weight"])) + 1 > weight_length: + weight_length = len(str(host["weight"])) + 1 + if len(host["current_state"]) + 1 > state_length: + state_length = len(host["current_state"]) + 1 + + output = list() + output.append( + "{bold}{hostname: <{hostname_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}{end_bold}".format( + bold="\033[1m", + end_bold="\033[0m", + hostname="Hostname", + hostname_length=hostname_length, + hid="ID", + hid_length=hid_length, + weight="Weight", + weight_length=weight_length, + state="State", + state_length=state_length, + commands="Active Commands" + ) + ) + + for hid, host in host_mappings.items(): + if len(host["commands"]) < 1: + first_command = "N/A" + else: + first_command = "PID {}: {}".format(host["commands"][0][2], host["commands"][0][3]) + + host_entry = list() + host_entry.append( + "{hostname: <{hostname_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}".format( + hostname=host["hostname"], + hostname_length=hostname_length, + hid=hid, + hid_length=hid_length, + weight=host["weight"], + weight_length=weight_length, + state=host["current_state"], + state_length=state_length, + commands=first_command + ) + ) + + for idx, command in enumerate(host["commands"]): + if idx == 0: + continue + host_entry.append( + "{hostname: <{hostname_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}".format( + hostname="", + hostname_length=hostname_length, + hid="", + hid_length=hid_length, + weight="", + weight_length=weight_length, + state="", + state_length=state_length, + commands="PID {}: {}".format(command[2], command[3]) + ) + ) + + output.append("\n".join(host_entry)) + + click.echo("\n".join(output)) + + rffmpeg_click.add_command(rffmpeg_click_status) + + @click.command(name="add", short_help="Add a host.") + @click.option( + "-w", + "--weight", + "weight", + required=False, + default=1, + help="The weight of the host." + ) + @click.argument("host") + def rffmpeg_click_add(weight, host): + """ + Add a new host with IP or hostname HOST to the database. + """ + with dbconn(config) as cur: + cur.execute( + """INSERT INTO hosts (hostname, weight) VALUES (?, ?)""", + (host, weight) + ) + + rffmpeg_click.add_command(rffmpeg_click_add) + + @click.command(name="remove", short_help="Remove a host.") + @click.argument("host") + def rffmpeg_click_remove(host): + """ + Remove a host with internal ID or IP or hostname HOST from the database. + """ + try: + host = int(host) + field = "id" + except ValueError: + field = "hostname" + + with dbconn(config) as cur: + entry = cur.execute("SELECT * FROM hosts WHERE {} = ?".format(field), (host,)).fetchall() + if len(entry) < 1: + fail("No hosts found to delete!") + + click.echo("Deleting {} host(s):".format(len(entry))) + for h in entry: + click.echo("\tID: {}\tHostname: {}".format(h[0], h[1])) + cur.execute( + """DELETE FROM hosts WHERE id = ?""", + (h[0],) + ) + + rffmpeg_click.add_command(rffmpeg_click_remove) + + return rffmpeg_click(obj={}) + + +# Entrypoint +if __name__ == "__main__": + all_args = sys.argv + cmd_name = all_args[0] + + # Load the config + config = load_config() + + if "rffmpeg" in cmd_name: + run_control(config) + else: + if not Path(config["db_path"]).is_file(): + fail("Failed to find database '{}' - did you forget to run 'rffmpeg init'?".format(config["db_path"])) + + ffmpeg_args = all_args[1:] + run_ffmpeg(config, ffmpeg_args) diff --git a/rffmpeg.py b/rffmpeg.py deleted file mode 100755 index dee108a..0000000 --- a/rffmpeg.py +++ /dev/null @@ -1,386 +0,0 @@ -#!/usr/bin/env python3 - -# rffmpeg.py - Remote FFMPEG transcoding for Jellyfin -# -# Copyright (C) 2019-2020 Joshua M. Boniface -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . -# -############################################################################### -# -# rffmpeg works as a drop-in replacement to an existing ffmpeg binary. It is -# used to launch ffmpeg commands on a remote machine via SSH, while passing -# in any stdin from the calling environment. Its primary usecase is to enable -# a program such as Jellyfin to distribute its ffmpeg calls to remote machines -# that might be better suited to transcoding or processing ffmpeg. -# -# rffmpeg uses a configuration file, by default at `/etc/rffmpeg/rffmpeg.yml`, -# to specify a number of settings that the processes will use. This includes -# the remote system(s) to connect to, temporary directories, SSH configuration, -# and other settings. -# -############################################################################### - -############################################################################### -# Imports and helper functions -############################################################################### - -import logging -import os -import re -import signal -import subprocess -import sys - -import yaml - -log = logging.getLogger("rffmpeg") - - -############################################################################### -# Configuration parsing -############################################################################### - -# Get configuration file -default_config_file = "/etc/rffmpeg/rffmpeg.yml" -config_file = os.environ.get("RFFMPEG_CONFIG", default_config_file) - -# Parse the configuration -with open(config_file, "r") as cfgfile: - try: - o_config = yaml.load(cfgfile, Loader=yaml.BaseLoader) - except Exception as e: - log.error("ERROR: Failed to parse configuration file: %s", e) - exit(1) - -try: - config = { - "state_tempdir": o_config["rffmpeg"]["state"]["tempdir"], - "state_filename": o_config["rffmpeg"]["state"]["filename"], - "state_contents": o_config["rffmpeg"]["state"]["contents"], - "log_to_file": o_config["rffmpeg"]["logging"]["file"], - "logfile": o_config["rffmpeg"]["logging"]["logfile"], - "remote_hosts": o_config["rffmpeg"]["remote"]["hosts"], - "remote_user": o_config["rffmpeg"]["remote"]["user"], - "remote_args": o_config["rffmpeg"]["remote"]["args"], - "pre_commands": o_config["rffmpeg"]["commands"]["pre"], - "ffmpeg_command": o_config["rffmpeg"]["commands"]["ffmpeg"], - "ffprobe_command": o_config["rffmpeg"]["commands"]["ffprobe"], - } -except Exception as e: - log.error("ERROR: Failed to load configuration: %s is missing", e) - exit(1) - -# Handle the fallback configuration using get() to avoid failing -config["ssh_command"] = o_config["rffmpeg"]["commands"].get("ssh", "ssh") -config["remote_persist_time"] = int(o_config["rffmpeg"]["remote"].get("persist", 0)) -config["state_persistdir"] = o_config["rffmpeg"]["state"].get("persistdir", '/run/shm') -config["fallback_ffmpeg_command"] = o_config["rffmpeg"]["commands"].get("fallback_ffmpeg", config["ffmpeg_command"]) -config["fallback_ffprobe_command"] = o_config["rffmpeg"]["commands"].get("fallback_ffprobe", config["ffprobe_command"]) - -# Parse CLI args (ffmpeg command line) -all_args = sys.argv -cli_ffmpeg_args = all_args[1:] - -# Get PID -current_statefile = config["state_tempdir"] + "/" + config["state_filename"].format(pid=os.getpid()) - -log.info("Starting rffmpeg %s: %s", os.getpid(), " ".join(all_args)) - - -def get_target_host(): - """ - Determine the optimal target host - """ - log.info("Determining target host") - - # Ensure the state directory exists or create it - if not os.path.exists(config["state_tempdir"]): - os.makedirs(config["state_tempdir"]) - - # Check for existing state files - state_files = os.listdir(config["state_tempdir"]) - - # Read each statefile to determine which hosts are bad or in use - bad_hosts = list() - active_hosts = list() - for state_file in state_files: - with open(config["state_tempdir"] + "/" + state_file, "r") as statefile: - contents = statefile.readlines() - for line in contents: - if re.match("^badhost", line): - bad_hosts.append(line.split()[1]) - log.info("Found bad host mark from rffmpeg process %s for host '%s'", re.findall(r"[0-9]+", state_file)[0], line.split()[1]) - else: - active_hosts.append(line.split()[0]) - log.info("Found running rffmpeg process %s against host '%s'", re.findall(r"[0-9]+", state_file)[0], line.split()[0]) - - # Get the remote hosts list from the config - remote_hosts = list() - for host in config["remote_hosts"]: - if type(host) is str or host.get("name", None) is None: - host_name = host - else: - host_name = host.get("name") - - if type(host) is str or host.get("weight", None) is None: - host_weight = 1 - else: - host_weight = int(host.get("weight")) - - remote_hosts.append({ "name": host_name, "weight": host_weight, "count": 0, "weighted_count": 0, "bad": False }) - - - # Remove any bad hosts from the remote_hosts list - for bhost in bad_hosts: - for idx, rhost in enumerate(remote_hosts): - if bhost == rhost["name"]: - remote_hosts[idx]["bad"] = True - - # Find out which active hosts are in use - for idx, rhost in enumerate(remote_hosts): - # Determine process counts in active_hosts - count = 0 - for ahost in active_hosts: - if ahost == rhost["name"]: - count += 1 - remote_hosts[idx]["count"] = count - - # Reweight the host counts by floor dividing count by weight - for idx, rhost in enumerate(remote_hosts): - if rhost["bad"]: - continue - if rhost["weight"] > 1: - remote_hosts[idx]["weighted_count"] = rhost["count"] // rhost["weight"] - else: - remote_hosts[idx]["weighted_count"] = rhost["count"] - - # Select the host with the lowest weighted count (first host is parsed last) - lowest_count = 999 - target_host = None - for rhost in remote_hosts: - if rhost["bad"]: - continue - if rhost["weighted_count"] < lowest_count: - lowest_count = rhost["weighted_count"] - target_host = rhost["name"] - - if not target_host: - log.warning("Failed to find a valid target host - using local fallback instead") - target_host = "localhost" - - # Write to our state file - with open(current_statefile, "a") as statefile: - statefile.write(config["state_contents"].format(host=target_host) + "\n") - - log.info("Selected target host '%s'", target_host) - return target_host - - -def bad_host(target_host): - log.info("Setting bad host %s", target_host) - - # Rewrite the statefile, removing all instances of the target_host that were added before - with open(current_statefile, "r+") as statefile: - new_statefile = statefile.readlines() - statefile.seek(0) - for line in new_statefile: - if target_host not in line: - statefile.write(line) - statefile.truncate() - - # Add the bad host to the statefile - # This will affect this run, as well as any runs that start while this one is active; once - # this run is finished and its statefile removed, however, the host will be retried again - with open(current_statefile, "a") as statefile: - statefile.write("badhost " + config["state_contents"].format(host=target_host) + "\n") - - -def setup_remote_command(target_host): - """ - Craft the target command - """ - rffmpeg_ssh_command = list() - rffmpeg_ffmpeg_command = list() - - # Add SSH component - rffmpeg_ssh_command.append(config["ssh_command"]) - rffmpeg_ssh_command.append("-q") - - # Set our connection timeouts, in case one of several remote machines is offline - rffmpeg_ssh_command.extend([ "-o", "ConnectTimeout=1" ]) - rffmpeg_ssh_command.extend([ "-o", "ConnectionAttempts=1" ]) - rffmpeg_ssh_command.extend([ "-o", "StrictHostKeyChecking=no" ]) - rffmpeg_ssh_command.extend([ "-o", "UserKnownHostsFile=/dev/null" ]) - - # Use SSH control persistence to keep sessions alive for subsequent commands - persist_time = config["remote_persist_time"] - if persist_time > 0: - rffmpeg_ssh_command.extend([ "-o", "ControlMaster=auto" ]) - rffmpeg_ssh_command.extend([ "-o", "ControlPath={}/ssh-%r@%h:%p".format(config["state_persistdir"]) ]) - rffmpeg_ssh_command.extend([ "-o", "ControlPersist={}".format(persist_time) ]) - - for arg in config["remote_args"]: - if arg: - rffmpeg_ssh_command.append(arg) - - # Add user+host string - rffmpeg_ssh_command.append("{}@{}".format(config["remote_user"], target_host)) - log.info("Running as %s@%s", config["remote_user"], target_host) - - # Add any pre command - for cmd in config["pre_commands"]: - if cmd: - rffmpeg_ffmpeg_command.append(cmd) - - # Prepare our default stdin/stdout/stderr (normally, stdout to stderr) - stdin = sys.stdin - stdout = sys.stderr - stderr = sys.stderr - - # Verify if we're in ffmpeg or ffprobe mode - if "ffprobe" in all_args[0]: - rffmpeg_ffmpeg_command.append(config["ffprobe_command"]) - stdout = sys.stdout - else: - rffmpeg_ffmpeg_command.append(config["ffmpeg_command"]) - - # Determine if version, encorders, or decoders is an argument; if so, we output stdout to stdout - # Weird workaround for something Jellyfin requires... - specials = ["-version", "-encoders", "-decoders", "-hwaccels", "-filters", "-h"] - if any(item in specials for item in cli_ffmpeg_args): - stdout = sys.stdout - - # Parse and re-quote any problematic arguments - for arg in cli_ffmpeg_args: - # Match bad shell characters: * ' ( ) whitespace - if re.search("[*'()\s|\[\]]", arg): - rffmpeg_ffmpeg_command.append('"{}"'.format(arg)) - else: - rffmpeg_ffmpeg_command.append("{}".format(arg)) - - return rffmpeg_ssh_command, rffmpeg_ffmpeg_command, stdin, stdout, stderr - - -def run_command(rffmpeg_ssh_command, rffmpeg_ffmpeg_command, stdin, stdout, stderr): - """ - Execute the command using subprocess - """ - rffmpeg_command = rffmpeg_ssh_command + rffmpeg_ffmpeg_command - p = subprocess.run( - rffmpeg_command, shell=False, bufsize=0, universal_newlines=True, stdin=stdin, stderr=stderr, stdout=stdout - ) - returncode = p.returncode - - return returncode - - -def run_local_ffmpeg(): - """ - Fallback call to local ffmpeg - """ - rffmpeg_ffmpeg_command = list() - - # Prepare our default stdin/stdout/stderr (normally, stdout to stderr) - stdin = sys.stdin - stdout = sys.stderr - stderr = sys.stderr - - # Verify if we're in ffmpeg or ffprobe mode - if "ffprobe" in all_args[0]: - rffmpeg_ffmpeg_command.append(config["fallback_ffprobe_command"]) - stdout = sys.stdout - else: - rffmpeg_ffmpeg_command.append(config["fallback_ffmpeg_command"]) - - # Determine if version, encorders, or decoders is an argument; if so, we output stdout to stdout - # Weird workaround for something Jellyfin requires... - specials = ["-version", "-encoders", "-decoders", "-hwaccels", "-filters", "-h"] - if any(item in specials for item in cli_ffmpeg_args): - stdout = sys.stdout - - # Parse and re-quote any problematic arguments - for arg in cli_ffmpeg_args: - rffmpeg_ffmpeg_command.append("{}".format(arg)) - - log.info("Local command: %s", " ".join(rffmpeg_ffmpeg_command)) - - return run_command([], rffmpeg_ffmpeg_command, stdin, stdout, stderr) - - -def run_remote_ffmpeg(target_host): - rffmpeg_ssh_command, rffmpeg_ffmpeg_command, stdin, stdout, stderr = setup_remote_command(target_host) - log.info("Remote command: %s '%s'", " ".join(rffmpeg_ssh_command), " ".join(rffmpeg_ffmpeg_command)) - - return run_command(rffmpeg_ssh_command, rffmpeg_ffmpeg_command, stdin, stdout, stderr) - - -def cleanup(signum="", frame=""): - # Remove the current statefile - try: - os.remove(current_statefile) - except FileNotFoundError: - pass - - -def main(): - signal.signal(signal.SIGTERM, cleanup) - signal.signal(signal.SIGINT, cleanup) - signal.signal(signal.SIGQUIT, cleanup) - signal.signal(signal.SIGHUP, cleanup) - - log_to_file = config.get("log_to_file", False) - if log_to_file: - logfile = config.get("logfile") - logging.basicConfig( - filename=logfile, level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s" - ) - else: - logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s") - - log.info("Starting rffmpeg PID %s", os.getpid()) - - # Main process loop; executes until the ffmpeg command actually runs on a reachable host - returncode = 1 - while True: - target_host = get_target_host() - if target_host == "localhost": - returncode = run_local_ffmpeg() - break - else: - returncode = run_remote_ffmpeg(target_host) - - # A returncode of 255 means that the SSH process failed; - # ffmpeg does not throw this return code (https://ffmpeg.org/pipermail/ffmpeg-user/2013-July/016245.html) - if returncode == 255: - log.info( - "SSH failed to host %s with retcode %s: marking this host as bad and retrying", - target_host, - returncode, - ) - bad_host(target_host) - else: - # The SSH succeeded, so we can abort the loop - break - - cleanup() - if returncode == 0: - log.info("Finished rffmpeg PID %s with return code %s", os.getpid(), returncode) - else: - log.error("Finished rffmpeg PID %s with return code %s", os.getpid(), returncode) - exit(returncode) - - -if __name__ == "__main__": - main() diff --git a/rffmpeg.yml.sample b/rffmpeg.yml.sample index 6dea92b..d8673ba 100644 --- a/rffmpeg.yml.sample +++ b/rffmpeg.yml.sample @@ -1,66 +1,62 @@ --- -# Example configuration file for rffmpeg +# Configuration file for rffmpeg +# # Copy this sample to /etc/rffmpeg/rffmpeg.yml and replace the various attributes # with the values for your environment. For more details please see the README. +# +# Any commented value represents the default. Uncomment and alter as required. rffmpeg: - # rffmpeg state configuration - YOU SHOULD NOT ALTER THESE - state: - # Temporary directory to store state - tempdir: "/run/shm/rffmpeg" - - # Filename format for state instance files - filename: "instance_{pid}.pid" - - # Contents of the state instance file - contents: "{host}" - - # Temporary directory to store SSH persistence sockets - persistdir: "/run/shm" - # Logging configuration logging: # Enable or disable file logging - file: true + #log_to_file: true # Log messages to this file - ensure the user running rffmpeg can write to it - logfile: "/var/log/jellyfin/rffmpeg.log" + #logfile: "/var/log/jellyfin/rffmpeg.log" + + # Directory configuration + directories: + # Persistent directory to store state database + #state: "/var/lib/rffmpeg" + + # Temporary directory to store SSH persistence sockets + #persist: "/run/shm" + + # The user who should own the state directory and database (the user who normally runs rffmpeg commands) + #owner: jellyfin + + # The group who should own the state directory and database (an administrative group) + # Use this group to control who is able to run commands like "rrffmpeg status". + #group: sudo # Remote (SSH) configuration remote: - # A YAML list of remote hosts to connect to; either direct list or name/weight supported - hosts: - - localhost - - name: gpu1 - weight: 2 # Relative to any non-weighted hosts which have weight 1 - # The remote SSH user to connect as - user: jellyfin + #user: jellyfin # How long to persist SSH sessions (0 to disable) - persist: 300 + #persist: 300 # A YAML list of additional SSH arguments (e.g. private keys), # one line per space-separated argument element. - args: - - "-i" - - "/var/lib/jellyfin/.ssh/id_rsa" - + #args: + # - "-i" + # - "/var/lib/jellyfin/id_rsa" # Remote command configuration commands: - # By default rffmpeg uses $PATH to find the "ssh" program; use this option to set a full path - # to an SSH binary if you want to override the default. - ssh: "ssh" + # The path (either full or in $PATH) to the SSH binary. + #ssh: "/usr/bin/ssh" # A YAML list of prefixes to the ffmpeg command (e.g. sudo, nice, etc.), # one line per space-separated command element. - pre: - - "" + #pre: + # - "" # The (remote) ffmpeg and ffprobe command binary paths - ffmpeg: "/usr/lib/jellyfin-ffmpeg/ffmpeg" - ffprobe: "/usr/lib/jellyfin-ffmpeg/ffprobe" + #ffmpeg: "/usr/lib/jellyfin-ffmpeg/ffmpeg" + #ffprobe: "/usr/lib/jellyfin-ffmpeg/ffprobe" # An optional local fallback ffmpeg and ffprobe, if you wish this to be different from the above paths #fallback_ffmpeg: "/usr/lib/jellyfin-ffmpeg/ffmpeg" From d276cde92e974082d10312440a4a48deb1f76656 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Tue, 19 Jul 2022 23:59:58 -0400 Subject: [PATCH 004/115] Try to fix banner link --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index d87ee75..a9297d4 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,7 @@ # rffmpeg

-Chat on Matrix - +[Chat on Matrix](https://matrix.to/#/#rffmpeg:matrix.org)

`rffmpeg` is a remote FFmpeg wrapper used to execute FFmpeg commands on a remote server via SSH. It is most useful in situations involving media servers such as Jellyfin (our reference user), where one might want to perform transcoding actions with FFmpeg on a remote machine or set of machines which can better handle transcoding, take advantage of hardware acceleration, or distribute transcodes across multiple servers for load balancing. From 5159e52c3d9e22d3992bc3503d9326cda1702c2f Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 00:00:14 -0400 Subject: [PATCH 005/115] Revert "Try to fix banner link" This reverts commit d276cde92e974082d10312440a4a48deb1f76656. --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a9297d4..d87ee75 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,9 @@ # rffmpeg

](https://matrix.to/#/#rffmpeg:matrix.org) + +Chat on Matrix +

`rffmpeg` is a remote FFmpeg wrapper used to execute FFmpeg commands on a remote server via SSH. It is most useful in situations involving media servers such as Jellyfin (our reference user), where one might want to perform transcoding actions with FFmpeg on a remote machine or set of machines which can better handle transcoding, take advantage of hardware acceleration, or distribute transcodes across multiple servers for load balancing. From 9d26e8c055536345633ee43480af74f37123130a Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 00:00:55 -0400 Subject: [PATCH 006/115] Fix broken tag in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d87ee75..0d1bd4f 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # rffmpeg -

Chat on Matrix From d65d93a7653e7a874e3fb7ac08c72623d3cab9e4 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 00:02:26 -0400 Subject: [PATCH 007/115] Fix some formatting in README --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0d1bd4f..03ad074 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ For a comprehensive installation tutorial based on a reference setup, please see The rffmpeg configuration file located at `rffmpeg.yml.sample` is an example that shows all default options. Even though this file is effectively "empty", it *must* be present at `/etc/rffmpeg/rffmpeg.yml` or at an alternative location specified by the environment variable `RFFMPEG_CONFIG`; the latter is only useful for testing, as media programs like Jellyfin provide no way to specify this. -To override a default option, simply uncomment the relevant line and adjust it to suit your needs. For those using [Jellyfin](https://jellyfin.org) and following the [SETUP guide](SETUP.md), no default options will need to be changed.o +To override a default option, simply uncomment the relevant line and adjust it to suit your needs. For those using [Jellyfin](https://jellyfin.org) and following the [SETUP guide](SETUP.md), no default options will need to be changed. Each option has an explanatory comment above it detailing its purpose. @@ -42,7 +42,7 @@ Since the configuration file is YAML, ensure that you do not use "Tab" character ### Initializing Rffmpeg -After first installing rffmpeg, ensure you initialize the database with the `sudo rffmpeg init` command. Note that `sudo` is required here to create the required data paths, but afterwards, `rffmpeg` can be run by anyone in the configured group. +After first installing rffmpeg, ensure you initialize the database with the `sudo rffmpeg init` command. Note that `sudo` is required here to create the required data paths, but afterwards, `rffmpeg` can be run by anyone in the configured group (by default the `sudo` group). Rffmpeg is a Click-based application; thus, all commands have a `-h` or `--help` flag to show usage and additional options that may be specified. From cc5b1d469b8bfcbb3d6504808165343072b6a3fd Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 00:19:50 -0400 Subject: [PATCH 008/115] Add log viewing support Adds two methods to view the log; the first is the entire log in a pager, and the second is following any new messages (like 'tail -f -0' in Linux). --- rffmpeg | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/rffmpeg b/rffmpeg index 7dee95e..de6e1cc 100755 --- a/rffmpeg +++ b/rffmpeg @@ -34,6 +34,7 @@ from pwd import getpwnam from re import search from sqlite3 import connect as sqlite_connect from subprocess import run +from time import sleep # Set up the logger @@ -638,6 +639,37 @@ def run_control(config): rffmpeg_click.add_command(rffmpeg_click_remove) + @click.command(name="log", short_help="View the rffmpeg log.") + @click.option( + "-f", + "--follow", + "follow_flag", + is_flag=True, + default=False, + help="Follow new log entries instead of seeing current log entries." + ) + def rffmpeg_click_log(follow_flag): + """ + View the rffmpeg log file. + """ + if follow_flag: + with open(config["logfile"]) as file_: + # Go to the end of file + file_.seek(0,2) + while True: + curr_position = file_.tell() + line = file_.readline() + if not line: + file_.seek(curr_position) + sleep(0.1) + else: + click.echo(line, nl=False) + else: + with open(config["logfile"], "r") as logfh: + click.echo_via_pager(logfh.readlines()) + + rffmpeg_click.add_command(rffmpeg_click_log) + return rffmpeg_click(obj={}) From d9a18d7137fa8b171ba3f267622b40617dfb4be9 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 00:24:17 -0400 Subject: [PATCH 009/115] Adjust quoting of name and add logfile section --- README.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 03ad074..d8d5c63 100644 --- a/README.md +++ b/README.md @@ -30,9 +30,9 @@ For a comprehensive installation tutorial based on a reference setup, please see ## Important Considerations -### The rffmpeg Configuration file +### The `rffmpeg` Configuration file -The rffmpeg configuration file located at `rffmpeg.yml.sample` is an example that shows all default options. Even though this file is effectively "empty", it *must* be present at `/etc/rffmpeg/rffmpeg.yml` or at an alternative location specified by the environment variable `RFFMPEG_CONFIG`; the latter is only useful for testing, as media programs like Jellyfin provide no way to specify this. +The `rffmpeg` configuration file located at `rffmpeg.yml.sample` is an example that shows all default options. Even though this file is effectively "empty", it *must* be present at `/etc/rffmpeg/rffmpeg.yml` or at an alternative location specified by the environment variable `RFFMPEG_CONFIG`; the latter is only useful for testing, as media programs like Jellyfin provide no way to specify this. To override a default option, simply uncomment the relevant line and adjust it to suit your needs. For those using [Jellyfin](https://jellyfin.org) and following the [SETUP guide](SETUP.md), no default options will need to be changed. @@ -42,25 +42,29 @@ Since the configuration file is YAML, ensure that you do not use "Tab" character ### Initializing Rffmpeg -After first installing rffmpeg, ensure you initialize the database with the `sudo rffmpeg init` command. Note that `sudo` is required here to create the required data paths, but afterwards, `rffmpeg` can be run by anyone in the configured group (by default the `sudo` group). +After first installing `rffmpeg`, ensure you initialize the database with the `sudo rffmpeg init` command. Note that `sudo` is required here to create the required data paths, but afterwards, `rffmpeg` can be run by anyone in the configured group (by default the `sudo` group). Rffmpeg is a Click-based application; thus, all commands have a `-h` or `--help` flag to show usage and additional options that may be specified. ### Viewing Status -Once installed and initialized, the status of the rffmpeg system can be viewed with the command `rffmpeg status`. This will show all configured target hosts, their states, and any active commands being run. +Once installed and initialized, the status of the `rffmpeg` system can be viewed with the command `rffmpeg status`. This will show all configured target hosts, their states, and any active commands being run. ### Adding or Removing Target Hosts To add a target host, use the command `rffmpeg add`. This command takes the optional `-w`/`--weight` flag to adjust the weight of the target host (see below). A host can be added more than once. -To remove a target host, use the command `rffmpeg remove`. This command takes either a target host name/IP, which affects all instances of that name, or a specific host ID. Removing an in-use target host will not terminate any running processes, though it may result in undefined behaviour within rffmpeg. Before removing a host it is best to ensure there is nothing using it. +To remove a target host, use the command `rffmpeg remove`. This command takes either a target host name/IP, which affects all instances of that name, or a specific host ID. Removing an in-use target host will not terminate any running processes, though it may result in undefined behaviour within `rffmpeg`. Before removing a host it is best to ensure there is nothing using it. + +### Viewing the Logfile + +The `rffmpeg` CLI offers a convenient way to view the log file. Use `rffmpeg log` to view the entire logfile in the current pager (usually `less`), or use `rffmpeg log -f` to follow any new log entries after that point (like `tail -0 -f`). ### Localhost and Fallback -If one of the hosts in the config file is called "localhost", rffmpeg will run locally without SSH. This can be useful if the local machine is also a powerful transcoding device. +If one of the hosts in the config file is called "localhost", `rffmpeg` will run locally without SSH. This can be useful if the local machine is also a powerful transcoding device. -In addition, rffmpeg will fall back to "localhost" should it be unable to find any working remote hosts. This helps prevent situations where rffmpeg cannot be run due to none of the remote host(s) being available. +In addition, `rffmpeg` will fall back to "localhost" should it be unable to find any working remote hosts. This helps prevent situations where `rffmpeg` cannot be run due to none of the remote host(s) being available. In both cases, note that, if hardware acceleraton is configured, it *must* be available on the local host as well, or the `ffmpeg` commands will fail. There is no easy way around this without rewriting arguments, and this is currently out-of-scope for `rffmpeg`. You should always use a lowest-common-denominator approach when deciding on what additional option(s) to enable, such that any configured host can run any process. @@ -68,7 +72,7 @@ The exact path to the local `ffmpeg` and `ffprobe` binaries can be overridden in ### Target Host Selection -When more than one target host is present, rffmpeg uses the following rules to select a new host: +When more than one target host is present, `rffmpeg` uses the following rules to select a new host: 1. Any hosts marked `bad` are ignored until that marking is cleared. @@ -84,7 +88,7 @@ When more than one target host is present, rffmpeg uses the following rules to s ### Target Host Weights and Duplicated Target Hosts -When adding a host to rffmpeg, a weight can be specified. Weights are used during the calculation of the fewest number of processes among hosts. The actual number of processes running on the host is floor divided (rounded down to the nearest divisible integer) by the weight to give a "weighted count", which is then used in the determination. This option allows one host to take on more processes than other nodes, as it will be chosen as the "least busy" host more often. +When adding a host to `rffmpeg`, a weight can be specified. Weights are used during the calculation of the fewest number of processes among hosts. The actual number of processes running on the host is floor divided (rounded down to the nearest divisible integer) by the weight to give a "weighted count", which is then used in the determination. This option allows one host to take on more processes than other nodes, as it will be chosen as the "least busy" host more often. For example, consider two hosts: `host1` with weight 1, and `host2` with weight 5. `host2` would have its actual number of processes floor divided by `5`, and thus any number of processes under `5` would count as `0`, any number of processes between `5` and `10` would count as `1`, and so on, resulting in `host2` being chosen over `host1` even if it had several processes. Thus, `host2` would on average handle 5x more `ffmpeg` processes than `host1` would. @@ -102,7 +106,7 @@ If for some reason all configured hosts are marked `bad`, fallback will be engag ## FAQ -### Why did you make rffmpeg? +### Why did you make `rffmpeg`? My virtualization setup (multiple 1U nodes with lots of live migration/failover) didn't lend itself well to passing a GPU into my Jellyfin VM, but I wanted to offload transcoding because doing 4K HEVC transcodes with a CPU performs horribly. I happened to have another machine (my "base" remote headless desktop/gaming server) which had a GPU, so I wanted to find a way to offload the transcoding to it. I came up with `rffmpeg` as a simple wrapper to the `ffmpeg` and `ffprobe` calls that Jellyfin (and Emby, and likely other media servers too) makes which would run them on that host instead. After finding it quite useful myself, I released it publicly as GPLv3 software so that others may benefit as well! From e3cf4e62305a4c2f91ef3ce1b5d7ac3445c96b75 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 00:27:26 -0400 Subject: [PATCH 010/115] Fix a few more instances --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index d8d5c63..131d774 100644 --- a/README.md +++ b/README.md @@ -40,11 +40,11 @@ Each option has an explanatory comment above it detailing its purpose. Since the configuration file is YAML, ensure that you do not use "Tab" characters inside of it, only spaces. -### Initializing Rffmpeg +### Initializing `rffmpeg` After first installing `rffmpeg`, ensure you initialize the database with the `sudo rffmpeg init` command. Note that `sudo` is required here to create the required data paths, but afterwards, `rffmpeg` can be run by anyone in the configured group (by default the `sudo` group). -Rffmpeg is a Click-based application; thus, all commands have a `-h` or `--help` flag to show usage and additional options that may be specified. +`rffmpeg` is a Click-based application; thus, all commands have a `-h` or `--help` flag to show usage and additional options that may be specified. ### Viewing Status @@ -84,7 +84,7 @@ When more than one target host is present, `rffmpeg` uses the following rules to c. If it is active, it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role. -1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. Rffmpeg will then begin running against this host. +1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. ### Target Host Weights and Duplicated Target Hosts From afd97a0729d1ec0cd3401324f8d7bf2fdcc24407 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 00:32:09 -0400 Subject: [PATCH 011/115] Mention when the rules take effect --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 131d774..7b2dc22 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ The exact path to the local `ffmpeg` and `ffprobe` binaries can be overridden in ### Target Host Selection -When more than one target host is present, `rffmpeg` uses the following rules to select a new host: +When more than one target host is present, `rffmpeg` uses the following rules to select a new host. These rules are evaluated each time a new `rffmpeg` alias process is spawned. 1. Any hosts marked `bad` are ignored until that marking is cleared. From 08ac5dbd8a72394e8f9aa8a67ea548d041649de7 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 00:32:34 -0400 Subject: [PATCH 012/115] Clarify the ignored line --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7b2dc22..75fed3c 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ The exact path to the local `ffmpeg` and `ffprobe` binaries can be overridden in When more than one target host is present, `rffmpeg` uses the following rules to select a new host. These rules are evaluated each time a new `rffmpeg` alias process is spawned. -1. Any hosts marked `bad` are ignored until that marking is cleared. +1. Any hosts marked `bad` are ignored. 1. All remaining hosts are iterated through in an indetermine order (Python dictionary with root key as the host ID). For each host: From 192e2eebaf13c398046c26ad56a5b75c902e9c84 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 00:34:03 -0400 Subject: [PATCH 013/115] Further formatting tweak --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 75fed3c..366fc2d 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ When more than one target host is present, `rffmpeg` uses the following rules to b. If the host is `idle` (has no running processes), it is immediately chosen and the iteration stops. - c. If it is active, it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role. + c. If the host is `active`, it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role. 1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. From 04b8794e66bdd85be5aea782f2723ce54a697bb3 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 00:36:59 -0400 Subject: [PATCH 014/115] Clarify active status further --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 366fc2d..c872a4e 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ When more than one target host is present, `rffmpeg` uses the following rules to b. If the host is `idle` (has no running processes), it is immediately chosen and the iteration stops. - c. If the host is `active`, it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role. + c. If the host is `active` (has at least one running process), it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role. 1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. From a993836d6090fdef8ca24038e4750cade8cf9574 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 02:31:09 -0400 Subject: [PATCH 015/115] Format code with Black --- rffmpeg | 176 +++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 122 insertions(+), 54 deletions(-) diff --git a/rffmpeg b/rffmpeg index de6e1cc..fd17c07 100755 --- a/rffmpeg +++ b/rffmpeg @@ -117,22 +117,39 @@ def load_config(): # Parse the keys from the remote group config["remote_user"] = config_remote.get("user", "jellyfin") - config["remote_args"] = config_remote.get("args", ["-i", "/var/lib/jellyfin/.ssh/id_rsa"]) + config["remote_args"] = config_remote.get( + "args", ["-i", "/var/lib/jellyfin/.ssh/id_rsa"] + ) config["persist_time"] = config_remote.get("persist", 300) # Parse the keys from the commands group config["ssh_command"] = config_commands.get("ssh", "/usr/bin/ssh") config["pre_commands"] = config_commands.get("pre", []) - config["ffmpeg_command"] = config_commands.get("ffmpeg", "/usr/lib/jellyfin-ffmpeg/ffmpeg") - config["ffprobe_command"] = config_commands.get("ffprobe", "/usr/lib/jellyfin-ffprobe/ffprobe") - config["fallback_ffmpeg_command"] = config_commands.get("fallback_ffmpeg", "/usr/lib/jellyfin-ffmpeg/ffmpeg") - config["fallback_ffprobe_command"] = config_commands.get("fallback_ffprobe", "/usr/lib/jellyfin-ffprobe/ffprobe") + config["ffmpeg_command"] = config_commands.get( + "ffmpeg", "/usr/lib/jellyfin-ffmpeg/ffmpeg" + ) + config["ffprobe_command"] = config_commands.get( + "ffprobe", "/usr/lib/jellyfin-ffprobe/ffprobe" + ) + config["fallback_ffmpeg_command"] = config_commands.get( + "fallback_ffmpeg", "/usr/lib/jellyfin-ffmpeg/ffmpeg" + ) + config["fallback_ffprobe_command"] = config_commands.get( + "fallback_ffprobe", "/usr/lib/jellyfin-ffprobe/ffprobe" + ) # Set the database path - config["db_path"]= config["state_dir"] + "/rffmpeg.db" + config["db_path"] = config["state_dir"] + "/rffmpeg.db" # Set a list of special flags that cause different behaviour - config["special_flags"] = ["-version", "-encoders", "-decoders", "-hwaccels", "-filters", "-h"] + config["special_flags"] = [ + "-version", + "-encoders", + "-decoders", + "-hwaccels", + "-filters", + "-h", + ] # Set the current PID of this process config["current_pid"] = os.getpid() @@ -148,7 +165,9 @@ def cleanup(signum="", frame=""): with dbconn(config) as cur: cur.execute("DELETE FROM states WHERE process_id = ?", (config["current_pid"],)) - cur.execute("DELETE FROM processes WHERE process_id = ?", (config["current_pid"],)) + cur.execute( + "DELETE FROM processes WHERE process_id = ?", (config["current_pid"],) + ) def generate_ssh_command(config, target_host): @@ -171,9 +190,11 @@ def generate_ssh_command(config, target_host): # Use SSH control persistence to keep sessions alive for subsequent commands if config["persist_time"] > 0: - ssh_command.extend(["-o","ControlMaster=auto"]) - ssh_command.extend(["-o","ControlPath={}/ssh-%r@%h:%p".format(config["persist_dir"])]) - ssh_command.extend(["-o","ControlPersist={}".format(config["persist_time"])]) + ssh_command.extend(["-o", "ControlMaster=auto"]) + ssh_command.extend( + ["-o", "ControlPath={}/ssh-%r@%h:%p".format(config["persist_dir"])] + ) + ssh_command.extend(["-o", "ControlPersist={}".format(config["persist_time"])]) # Add the remote config args for arg in config["remote_args"]: @@ -198,7 +219,7 @@ def run_command(command, stdin, stdout, stderr): universal_newlines=True, stdin=stdin, stdout=stdout, - stderr=stderr + stderr=stderr, ) return p.returncode @@ -219,7 +240,9 @@ def get_target_host(config): # Get the latest state with dbconn(config) as cur: - current_state = cur.execute("SELECT * FROM states WHERE host_id = ? ORDER BY id DESC", (hid,)).fetchone() + current_state = cur.execute( + "SELECT * FROM states WHERE host_id = ? ORDER BY id DESC", (hid,) + ).fetchone() if not current_state: current_state = "idle" @@ -231,7 +254,7 @@ def get_target_host(config): "hostname": hostname, "weight": weight, "current_state": current_state, - "commands": [proc[2] for proc in processes if proc[1] == hid] + "commands": [proc[2] for proc in processes if proc[1] == hid], } lowest_count = 9999 @@ -247,12 +270,21 @@ def get_target_host(config): if host["hostname"] not in ["localhost", "127.0.0.1"]: test_ssh_command = generate_ssh_command(config, host["hostname"]) test_ffmpeg_command = [config["ffmpeg_command"], "-version"] - retcode = run_command(test_ssh_command + test_ffmpeg_command, None, None, None) + retcode = run_command( + test_ssh_command + test_ffmpeg_command, None, None, None + ) if retcode != 0: # Mark the host as bad with dbconn(config) as cur: - log.info("Marking host {} as bad due to retcode {}".format(host["hostname"], retcode)) - cur.execute("INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", (hid, config["current_pid"], "bad")) + log.info( + "Marking host {} as bad due to retcode {}".format( + host["hostname"], retcode + ) + ) + cur.execute( + "INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", + (hid, config["current_pid"], "bad"), + ) continue # If the host state is idle, we can use it immediately @@ -294,7 +326,7 @@ def run_local_ffmpeg(config, ffmpeg_args): # Otherwise, we use stderr as stdout rffmpeg_ffmpeg_command.append(config["fallback_ffmpeg_command"]) stdout = sys.stderr - + # Check for special flags that override the default stdout if any(item in config["special_flags"] for item in ffmpeg_args): stdout = sys.stdout @@ -306,8 +338,14 @@ def run_local_ffmpeg(config, ffmpeg_args): log.info("Local command: {}".format(" ".join(rffmpeg_ffmpeg_command))) with dbconn(config) as cur: - cur.execute("INSERT INTO processes (host_id, process_id, cmd) VALUES (?, ?, ?)", (0, config["current_pid"], cmd_name + " " + " ".join(ffmpeg_args))) - cur.execute("INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", (0, config["current_pid"], "active")) + cur.execute( + "INSERT INTO processes (host_id, process_id, cmd) VALUES (?, ?, ?)", + (0, config["current_pid"], cmd_name + " " + " ".join(ffmpeg_args)), + ) + cur.execute( + "INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", + (0, config["current_pid"], "active"), + ) return run_command(rffmpeg_ffmpeg_command, stdin, stdout, stderr) @@ -336,7 +374,7 @@ def run_remote_ffmpeg(config, target_hid, target_host, ffmpeg_args): # Otherwise, we use stderr as stdout rffmpeg_ffmpeg_command.append(config["ffmpeg_command"]) stdout = sys.stderr - + # Check for special flags that override the default stdout if any(item in config["special_flags"] for item in ffmpeg_args): stdout = sys.stdout @@ -349,13 +387,25 @@ def run_remote_ffmpeg(config, target_hid, target_host, ffmpeg_args): else: rffmpeg_ffmpeg_command.append("{}".format(arg)) - log.info("Remote command: {}".format(" ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command))) + log.info( + "Remote command: {}".format( + " ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command) + ) + ) with dbconn(config) as cur: - cur.execute("INSERT INTO processes (host_id, process_id, cmd) VALUES (?, ?, ?)", (target_hid, config["current_pid"], cmd_name + " " + " ".join(ffmpeg_args))) - cur.execute("INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", (target_hid, config["current_pid"], "active")) + cur.execute( + "INSERT INTO processes (host_id, process_id, cmd) VALUES (?, ?, ?)", + (target_hid, config["current_pid"], cmd_name + " " + " ".join(ffmpeg_args)), + ) + cur.execute( + "INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", + (target_hid, config["current_pid"], "active"), + ) - return run_command(rffmpeg_ssh_command + rffmpeg_ffmpeg_command, stdin, stdout, stderr) + return run_command( + rffmpeg_ssh_command + rffmpeg_ffmpeg_command, stdin, stdout, stderr + ) def run_ffmpeg(config, ffmpeg_args): @@ -371,12 +421,12 @@ def run_ffmpeg(config, ffmpeg_args): logging.basicConfig( filename=config["logfile"], level=logging.INFO, - format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s" + format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s", ) else: logging.basicConfig( level=logging.INFO, - format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s" + format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s", ) log.info("Starting rffmpeg with args: {}".format(" ".join(ffmpeg_args))) @@ -416,7 +466,7 @@ def run_control(config): "confirm_flag", is_flag=True, default=False, - help="Confirm initialization." + help="Confirm initialization.", ) def rffmpeg_click_init(confirm_flag): """ @@ -433,7 +483,7 @@ def run_control(config): click.confirm( "Are you sure you want to (re)initalize the database", prompt_suffix="? ", - abort=True + abort=True, ) except Exception: fail("Aborting due to failed confirmation.") @@ -442,7 +492,11 @@ def run_control(config): try: os.makedirs(config["state_dir"]) except OSError as e: - fail("Failed to create state directory '{}': {}".format(config['state_dir'], e)) + fail( + "Failed to create state directory '{}': {}".format( + config["state_dir"], e + ) + ) if Path(config["db_path"]).is_file(): os.remove(config["db_path"]) @@ -461,9 +515,17 @@ def run_control(config): except Exception as e: fail("Failed to create database: {}".format(e)) - os.chown(config["state_dir"], getpwnam(config["dir_owner"]).pw_uid, getgrnam(config["dir_group"]).gr_gid) + os.chown( + config["state_dir"], + getpwnam(config["dir_owner"]).pw_uid, + getgrnam(config["dir_group"]).gr_gid, + ) os.chmod(config["state_dir"], 0o770) - os.chown(config["db_path"], getpwnam(config["dir_owner"]).pw_uid, getgrnam(config["dir_group"]).gr_gid) + os.chown( + config["db_path"], + getpwnam(config["dir_owner"]).pw_uid, + getgrnam(config["dir_group"]).gr_gid, + ) os.chmod(config["db_path"], 0o660) rffmpeg_click.add_command(rffmpeg_click_init) @@ -492,27 +554,29 @@ def run_control(config): "hostname": "localhost (fallback)", "weight": 0, "current_state": "fallback", - "commands": fallback_processes + "commands": fallback_processes, } for host in hosts: hid, hostname, weight = host - + # Get the latest state with dbconn(config) as cur: - current_state = cur.execute("SELECT * FROM states WHERE host_id = ? ORDER BY id DESC", (hid,)).fetchone() - + current_state = cur.execute( + "SELECT * FROM states WHERE host_id = ? ORDER BY id DESC", (hid,) + ).fetchone() + if not current_state: current_state = "idle" else: current_state = current_state[3] - + # Create the mappings entry host_mappings[hid] = { "hostname": hostname, "weight": weight, "current_state": current_state, - "commands": [proc for proc in processes if proc[1] == hid] + "commands": [proc for proc in processes if proc[1] == hid], } hostname_length = 9 @@ -528,7 +592,7 @@ def run_control(config): weight_length = len(str(host["weight"])) + 1 if len(host["current_state"]) + 1 > state_length: state_length = len(host["current_state"]) + 1 - + output = list() output.append( "{bold}{hostname: <{hostname_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}{end_bold}".format( @@ -542,7 +606,7 @@ def run_control(config): weight_length=weight_length, state="State", state_length=state_length, - commands="Active Commands" + commands="Active Commands", ) ) @@ -550,7 +614,9 @@ def run_control(config): if len(host["commands"]) < 1: first_command = "N/A" else: - first_command = "PID {}: {}".format(host["commands"][0][2], host["commands"][0][3]) + first_command = "PID {}: {}".format( + host["commands"][0][2], host["commands"][0][3] + ) host_entry = list() host_entry.append( @@ -563,7 +629,7 @@ def run_control(config): weight_length=weight_length, state=host["current_state"], state_length=state_length, - commands=first_command + commands=first_command, ) ) @@ -580,7 +646,7 @@ def run_control(config): weight_length=weight_length, state="", state_length=state_length, - commands="PID {}: {}".format(command[2], command[3]) + commands="PID {}: {}".format(command[2], command[3]), ) ) @@ -597,7 +663,7 @@ def run_control(config): "weight", required=False, default=1, - help="The weight of the host." + help="The weight of the host.", ) @click.argument("host") def rffmpeg_click_add(weight, host): @@ -606,8 +672,7 @@ def run_control(config): """ with dbconn(config) as cur: cur.execute( - """INSERT INTO hosts (hostname, weight) VALUES (?, ?)""", - (host, weight) + """INSERT INTO hosts (hostname, weight) VALUES (?, ?)""", (host, weight) ) rffmpeg_click.add_command(rffmpeg_click_add) @@ -625,17 +690,16 @@ def run_control(config): field = "hostname" with dbconn(config) as cur: - entry = cur.execute("SELECT * FROM hosts WHERE {} = ?".format(field), (host,)).fetchall() + entry = cur.execute( + "SELECT * FROM hosts WHERE {} = ?".format(field), (host,) + ).fetchall() if len(entry) < 1: fail("No hosts found to delete!") click.echo("Deleting {} host(s):".format(len(entry))) for h in entry: click.echo("\tID: {}\tHostname: {}".format(h[0], h[1])) - cur.execute( - """DELETE FROM hosts WHERE id = ?""", - (h[0],) - ) + cur.execute("""DELETE FROM hosts WHERE id = ?""", (h[0],)) rffmpeg_click.add_command(rffmpeg_click_remove) @@ -646,7 +710,7 @@ def run_control(config): "follow_flag", is_flag=True, default=False, - help="Follow new log entries instead of seeing current log entries." + help="Follow new log entries instead of seeing current log entries.", ) def rffmpeg_click_log(follow_flag): """ @@ -655,7 +719,7 @@ def run_control(config): if follow_flag: with open(config["logfile"]) as file_: # Go to the end of file - file_.seek(0,2) + file_.seek(0, 2) while True: curr_position = file_.tell() line = file_.readline() @@ -685,7 +749,11 @@ if __name__ == "__main__": run_control(config) else: if not Path(config["db_path"]).is_file(): - fail("Failed to find database '{}' - did you forget to run 'rffmpeg init'?".format(config["db_path"])) + fail( + "Failed to find database '{}' - did you forget to run 'rffmpeg init'?".format( + config["db_path"] + ) + ) ffmpeg_args = all_args[1:] run_ffmpeg(config, ffmpeg_args) From ce5059959c14b210d62a7386a859df0825422046 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 02:36:00 -0400 Subject: [PATCH 016/115] Add shields.io badges --- README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.md b/README.md index c872a4e..ac6c70a 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,17 @@ # rffmpeg

+License: GPLv3+ +Code Style: Black Chat on Matrix + +Support me on Patreon + + +Support me on GitHub +

`rffmpeg` is a remote FFmpeg wrapper used to execute FFmpeg commands on a remote server via SSH. It is most useful in situations involving media servers such as Jellyfin (our reference user), where one might want to perform transcoding actions with FFmpeg on a remote machine or set of machines which can better handle transcoding, take advantage of hardware acceleration, or distribute transcodes across multiple servers for load balancing. From b49c13890fbf808e74a237b3e839ae884c1a9d16 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 02:43:45 -0400 Subject: [PATCH 017/115] Standardize comment formatting --- rffmpeg.yml.sample | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/rffmpeg.yml.sample b/rffmpeg.yml.sample index d8673ba..faf8c9f 100644 --- a/rffmpeg.yml.sample +++ b/rffmpeg.yml.sample @@ -9,55 +9,59 @@ rffmpeg: # Logging configuration logging: - # Enable or disable file logging + # Enable or disable file logging. #log_to_file: true - # Log messages to this file - ensure the user running rffmpeg can write to it + # Log messages to this file. + # Ensure the user running rffmpeg can write to this directory. #logfile: "/var/log/jellyfin/rffmpeg.log" # Directory configuration directories: - # Persistent directory to store state database + # Persistent directory to store state database. #state: "/var/lib/rffmpeg" - # Temporary directory to store SSH persistence sockets + # Temporary directory to store SSH persistence sockets. #persist: "/run/shm" - # The user who should own the state directory and database (the user who normally runs rffmpeg commands) + # The user who should own the state directory and database. + # This should normally be the user who normally runs rffmpeg commands (i.e. the media + # server service user). #owner: jellyfin - # The group who should own the state directory and database (an administrative group) - # Use this group to control who is able to run commands like "rrffmpeg status". + # The group who should own the state directory and database (an administrative group). + # Use this group to control who is able to run "rffmpeg" management commands; users in + # this group will have unlimited access to the tool to add/remove hosts, view status, etc. #group: sudo # Remote (SSH) configuration remote: - # The remote SSH user to connect as + # The remote SSH user to connect as. #user: jellyfin - # How long to persist SSH sessions (0 to disable) + # How long to persist SSH sessions; 0 to disable SSH persistence. #persist: 300 - # A YAML list of additional SSH arguments (e.g. private keys), - # one line per space-separated argument element. + # A YAML list of additional SSH arguments (e.g. private keys). + # One entry line per space-separated argument element. #args: # - "-i" # - "/var/lib/jellyfin/id_rsa" # Remote command configuration commands: - # The path (either full or in $PATH) to the SSH binary. + # The path (either full or in $PATH) to the default SSH binary. #ssh: "/usr/bin/ssh" - # A YAML list of prefixes to the ffmpeg command (e.g. sudo, nice, etc.), - # one line per space-separated command element. + # A YAML list of prefixes to the ffmpeg command (e.g. sudo, nice, etc.). + # One entry line per space-separated command element. #pre: # - "" - # The (remote) ffmpeg and ffprobe command binary paths + # The (remote) ffmpeg and ffprobe command binary paths. #ffmpeg: "/usr/lib/jellyfin-ffmpeg/ffmpeg" #ffprobe: "/usr/lib/jellyfin-ffmpeg/ffprobe" - # An optional local fallback ffmpeg and ffprobe, if you wish this to be different from the above paths + # Optional local fallback ffmpeg and ffprobe binary paths, if different from the above. #fallback_ffmpeg: "/usr/lib/jellyfin-ffmpeg/ffmpeg" #fallback_ffprobe: "/usr/lib/jellyfin-ffmpeg/ffprobe" From 2a0c74ad30b41f04c75eebdeff79a14c374f4b82 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 02:46:17 -0400 Subject: [PATCH 018/115] Adjust wording in target host selection --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ac6c70a..446428f 100644 --- a/README.md +++ b/README.md @@ -80,7 +80,7 @@ The exact path to the local `ffmpeg` and `ffprobe` binaries can be overridden in ### Target Host Selection -When more than one target host is present, `rffmpeg` uses the following rules to select a new host. These rules are evaluated each time a new `rffmpeg` alias process is spawned. +When more than one target host is present, `rffmpeg` uses the following rules to select a target host. These rules are evaluated each time a new `rffmpeg` alias process is spawned based on the current state (actively running processes, etc.). 1. Any hosts marked `bad` are ignored. From 9009e3161c612b55c9e93957020afbb5919811c5 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 02:48:34 -0400 Subject: [PATCH 019/115] Clarify performance reasons for not weighting --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 446428f..c7199ff 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,7 @@ When adding a host to `rffmpeg`, a weight can be specified. Weights are used dur For example, consider two hosts: `host1` with weight 1, and `host2` with weight 5. `host2` would have its actual number of processes floor divided by `5`, and thus any number of processes under `5` would count as `0`, any number of processes between `5` and `10` would count as `1`, and so on, resulting in `host2` being chosen over `host1` even if it had several processes. Thus, `host2` would on average handle 5x more `ffmpeg` processes than `host1` would. -Host weighting is a fairly blunt instrument, and only becomes important when many simultaneous `ffmpeg` processes/transcodes are occurring at once. Generally leaving all hosts at weight 1 would be sufficient for most usecases. +Host weighting is a fairly blunt instrument, and only becomes important when many simultaneous `ffmpeg` processes/transcodes are occurring at once, and where the target hosts have very different performance profiles. Generally leaving all hosts at weight 1 would be sufficient for most usecases. Furthermore, it is possible to add a host of the same name more than once in the `rffmpeg add` command. This has a very similar, but subtly different, effect from setting a higher weight. A host present in the list is more likely to be seen before another host, and thus this can further influence the desired target. From 6385254fbf120c5f2cb0eeaa3702cf579dc41b76 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 02:49:54 -0400 Subject: [PATCH 020/115] Clarify why retrying is bad --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c7199ff..b8cb690 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ Furthermore, it is possible to add a host of the same name more than once in the As mentioned above under [Target Host Selection](README.md#target-host-selection), a host can be marked `bad` if it does not respond to an `ffmpeg -version` command in at least 1 second. This can happen because a host is offline, unreachable, overloaded, or otherwise unresponsive. -Once a host is marked `bad`, it will remain so for as long as the `rffmpeg` process that marked it `bad` is running. This can last from a few seconds to several tens of minutes. During this time, any new `rffmpeg` processes that start will see the host marked as `bad` and thus skip it for target selection. Once the marking `rffmpeg` process completes or is terminated, the `bad` status of that host will be cleared, allowing the next run to try it again. This strikes a balance between always retrying known-unresponsive hosts over and over, and ensuring that hosts will eventually be retried. +Once a host is marked `bad`, it will remain so for as long as the `rffmpeg` process that marked it `bad` is running. This can last from a few seconds to several tens of minutes. During this time, any new `rffmpeg` processes that start will see the host marked as `bad` and thus skip it for target selection. Once the marking `rffmpeg` process completes or is terminated, the `bad` status of that host will be cleared, allowing the next run to try it again. This strikes a balance between always retrying known-unresponsive hosts over and over (and thus delaying process startup), and ensuring that hosts will eventually be retried. If for some reason all configured hosts are marked `bad`, fallback will be engaged; see the above section [Localhost and Fallback](README.md#localhost-and-fallback) for details on what occurrs in this situation. An explicit `localhost` host entry cannot be marked `bad`. From b1a7f86128f393f611bb906f029ab9edae162e84 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 02:54:03 -0400 Subject: [PATCH 021/115] Fix formatting inconsistencies --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index b8cb690..9879e8b 100644 --- a/README.md +++ b/README.md @@ -134,9 +134,9 @@ Explicitly *no*. `rffmpeg` is not designed to interact with the arguments that t This has a number of side effects: - * `rffmpeg` does not know whether hardware acceleration is turned on or not (see above caveats under [Localhost and Fallback](README.md#localhost-and-fallback) - * `rffmpeg` does not know what media is playing or where it's outputting files to, and cannot alter these paths - * `rffmpeg` cannot turn on or off special `ffmpeg` options depending on the host selected + * `rffmpeg` does not know whether hardware acceleration is turned on or not (see above caveats under [Localhost and Fallback](README.md#localhost-and-fallback)). + * `rffmpeg` does not know what media is playing or where it's outputting files to, and cannot alter these paths. + * `rffmpeg` cannot turn on or off special `ffmpeg` options depending on the host selected. Thus it is imperitive that you set up your entire system correctly for `rffmpeg` to work. Please see the [SETUP guide](SETUP.md) for more information. From 575af44a93ade25c8c5f460f2eea136301058164 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 02:55:40 -0400 Subject: [PATCH 022/115] Additional formatting tweaks --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9879e8b..22b5ba4 100644 --- a/README.md +++ b/README.md @@ -160,9 +160,9 @@ If it hasn't, you can [ask in our chat](https://matrix.to/#/#rffmpeg:matrix.org) I will probably ask clarifying questions as required; please be prepared to run test commands, etc. as requested and paste the output. -### I found a bug/flaw and fixed or, or made a feature improvement; can I share it? +### I found a bug/flaw and fixed it, or made a feature improvement; can I share it? -Absolutely - I'm happy to take pull requests. Though please refer to the "Can `rffmpeg` mangle/alter FFMPEG arguments?" entry above; unless it's really good work with a very explicitly defined limitation, I probably don't want to go down that route, but I'm more than willing to look at what you've done and consider it on its merits. +Absolutely - I'm happy to take pull requests for just about any bugfix or improvement. There is one exception: please refer to the "Can `rffmpeg` mangle/alter FFMPEG arguments?" entry above; unless it's really good work with a very explicitly defined limitation, I probably don't want to go down that route, but I'm more than willing to look at what you've done and consider it on its merits. ### Can you help me set up my server? From 3291f5edf3187d3efb9477a6a55a7cbe4d70e074 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 02:56:03 -0400 Subject: [PATCH 023/115] Add comment about setup.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 22b5ba4..d910cce 100644 --- a/README.md +++ b/README.md @@ -166,4 +166,4 @@ Absolutely - I'm happy to take pull requests for just about any bugfix or improv ### Can you help me set up my server? -I'm always happy to help, though please ensure you try to follow the setup guide first. Support can be found [on Matrix](https://matrix.to/#/#rffmpeg:matrix.org) or via email at `joshua@boniface.me`. Please note though that I may be unresponsive sometimes, though I will get back to you eventually I promise! Please don't open Issues here about setup problems; the Issue tracker is for bugs or feature requests instead. +I'm always happy to help, though please ensure you try to follow the setup guide first - that's why I wrote it! Support can be found [on Matrix](https://matrix.to/#/#rffmpeg:matrix.org) or via email at `joshua@boniface.me`. Please note though that I may be unresponsive sometimes, though I will get back to you eventually I promise! Please don't open Issues here about setup problems; the Issue tracker is for bugs or feature requests instead. From 1006583ffff18efd1cb7de62de1d4142da393b37 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 03:07:48 -0400 Subject: [PATCH 024/115] Reformat and refactor setup guide further --- README.md | 12 +++++----- SETUP.md | 66 ++++++++++++++++++++++++++++++++++--------------------- 2 files changed, 47 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index d910cce..6046626 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ If one of the hosts in the config file is called "localhost", `rffmpeg` will run In addition, `rffmpeg` will fall back to "localhost" should it be unable to find any working remote hosts. This helps prevent situations where `rffmpeg` cannot be run due to none of the remote host(s) being available. -In both cases, note that, if hardware acceleraton is configured, it *must* be available on the local host as well, or the `ffmpeg` commands will fail. There is no easy way around this without rewriting arguments, and this is currently out-of-scope for `rffmpeg`. You should always use a lowest-common-denominator approach when deciding on what additional option(s) to enable, such that any configured host can run any process. +In both cases, note that, if hardware acceleration is configured, it *must* be available on the local host as well, or the `ffmpeg` commands will fail. There is no easy way around this without rewriting arguments, and this is currently out-of-scope for `rffmpeg`. You should always use a lowest-common-denominator approach when deciding on what additional option(s) to enable, such that any configured host can run any process. The exact path to the local `ffmpeg` and `ffprobe` binaries can be overridden in the configuration, should their paths not match those of the remote system(s). @@ -84,7 +84,7 @@ When more than one target host is present, `rffmpeg` uses the following rules to 1. Any hosts marked `bad` are ignored. -1. All remaining hosts are iterated through in an indetermine order (Python dictionary with root key as the host ID). For each host: +1. All remaining hosts are iterated through in an indeterminate order (Python dictionary with root key as the host ID). For each host: a. If the host is not `localhost`/`127.0.0.1`, it is tested to ensure it is reachable (responds to `ffmpeg -version` over SSH). If it is not reachable, it is marked `bad` for the duration of this processes' runtime and skipped. @@ -100,7 +100,7 @@ When adding a host to `rffmpeg`, a weight can be specified. Weights are used dur For example, consider two hosts: `host1` with weight 1, and `host2` with weight 5. `host2` would have its actual number of processes floor divided by `5`, and thus any number of processes under `5` would count as `0`, any number of processes between `5` and `10` would count as `1`, and so on, resulting in `host2` being chosen over `host1` even if it had several processes. Thus, `host2` would on average handle 5x more `ffmpeg` processes than `host1` would. -Host weighting is a fairly blunt instrument, and only becomes important when many simultaneous `ffmpeg` processes/transcodes are occurring at once, and where the target hosts have very different performance profiles. Generally leaving all hosts at weight 1 would be sufficient for most usecases. +Host weighting is a fairly blunt instrument, and only becomes important when many simultaneous `ffmpeg` processes/transcodes are occurring at once, and where the target hosts have very different performance profiles. Generally leaving all hosts at weight 1 would be sufficient for most use-cases. Furthermore, it is possible to add a host of the same name more than once in the `rffmpeg add` command. This has a very similar, but subtly different, effect from setting a higher weight. A host present in the list is more likely to be seen before another host, and thus this can further influence the desired target. @@ -110,7 +110,7 @@ As mentioned above under [Target Host Selection](README.md#target-host-selection Once a host is marked `bad`, it will remain so for as long as the `rffmpeg` process that marked it `bad` is running. This can last from a few seconds to several tens of minutes. During this time, any new `rffmpeg` processes that start will see the host marked as `bad` and thus skip it for target selection. Once the marking `rffmpeg` process completes or is terminated, the `bad` status of that host will be cleared, allowing the next run to try it again. This strikes a balance between always retrying known-unresponsive hosts over and over (and thus delaying process startup), and ensuring that hosts will eventually be retried. -If for some reason all configured hosts are marked `bad`, fallback will be engaged; see the above section [Localhost and Fallback](README.md#localhost-and-fallback) for details on what occurrs in this situation. An explicit `localhost` host entry cannot be marked `bad`. +If for some reason all configured hosts are marked `bad`, fallback will be engaged; see the above section [Localhost and Fallback](README.md#localhost-and-fallback) for details on what occurs in this situation. An explicit `localhost` host entry cannot be marked `bad`. ## FAQ @@ -138,11 +138,11 @@ This has a number of side effects: * `rffmpeg` does not know what media is playing or where it's outputting files to, and cannot alter these paths. * `rffmpeg` cannot turn on or off special `ffmpeg` options depending on the host selected. -Thus it is imperitive that you set up your entire system correctly for `rffmpeg` to work. Please see the [SETUP guide](SETUP.md) for more information. +Thus it is imperative that you set up your entire system correctly for `rffmpeg` to work. Please see the [SETUP guide](SETUP.md) for more information. ### Can `rffmpeg` do Wake-On-LAN or other similar options to turn on a transcode server? -Right now, no. I've thought about implementing this more than once (most recently, in response to [Issue #21](https://github.com/joshuaboniface/rffmpeg/issues/21)) but ultimately I've never though this was worth the complexity and delays in spwaning that it would add to the tool. That issue does provide one example of a workaround wrapper script that could accomplish this, but I don't see it being a part of the actual tool itself. +Right now, no. I've thought about implementing this more than once (most recently, in response to [Issue #21](https://github.com/joshuaboniface/rffmpeg/issues/21)) but ultimately I've never though this was worth the complexity and delays in spawning that it would add to the tool. That issue does provide one example of a workaround wrapper script that could accomplish this, but I don't see it being a part of the actual tool itself. ### I'm getting an error, help! diff --git a/SETUP.md b/SETUP.md index 60d63a1..5ab352a 100644 --- a/SETUP.md +++ b/SETUP.md @@ -6,6 +6,8 @@ This guide is provided as a basic starting point - there are myriad possible com ## Set up the media server (`jellyfin1`) +### Basic Setup + 1. Install Jellyfin (or similar FFMPEG-using media server) on your machine. This guide assumes you're using native `.deb` packages. 1. Make note of the Jellyfin service user's details, specifically the UID and any groups (and GIDs) it is a member of; this will be needed later on. @@ -50,6 +52,8 @@ This guide is provided as a basic starting point - there are myriad possible com * **NOTE:** Ensure you use the exact name here that you will use in `rffmpeg`. If this is an FQDN (e.g. `jellyfin1.mydomain.tld`) or an IP (e.g. `192.168.0.101`) instead of a short name, use that instead in this command, or repeat it for every possible option (it doesn't hurt). +### `rffmpeg` Setup + 1. Install the required Python3 dependencies of `rffmpeg`: ``` @@ -58,9 +62,9 @@ This guide is provided as a basic starting point - there are myriad possible com jellyfin1 $ sudo apt -y install python3-subprocess ``` - Note: On some Ubuntu versions, `python3-subprocess` does not exist, and should instead be part of the Python standard library. Skip installing this package if it can't be found. + * **NOTE:** On some Ubuntu versions, `python3-subprocess` does not exist, and should instead be part of the Python standard library. Skip installing this package if it can't be found. -1. Clone the `rffmpeg` repository somewhere ont he system, then install the `rffmpeg` binary, make it executable, and prepare symlinks for the command names `ffmpeg` and `ffprobe` to it. I recommend storing these in `/usr/local/bin` for simplicity. +1. Clone the `rffmpeg` repository somewhere onto the system, then install the `rffmpeg` binary, make it executable, and prepare symlinks for the command names `ffmpeg` and `ffprobe` to it. I recommend storing these in `/usr/local/bin` for simplicity and so that they are present on the default `$PATH` for most users. ``` jellyfin1 $ git clone https://github.com/joshuaboniface/rffmpeg # or download the files manually @@ -78,13 +82,17 @@ This guide is provided as a basic starting point - there are myriad possible com jellyfin1 $ sudo $EDITOR /etc/rffmpeg/rffmpeg.yml # if required ``` -1. Initialize `rffmpeg` (note the `sudo` command) and add at least one remote host to it. You can add multiple hosts now or later, set weights of hosts, and add a host more than once. For full details see the [main README](README.md). +1. Initialize `rffmpeg` (note the `sudo` command) and add at least one remote host to it. You can add multiple hosts now or later, set weights of hosts, and add a host more than once. For full details see the [main README](README.md) or run `rffmpeg --help` to view the CLI help menu. ``` jellyfin1 $ sudo rffmpeg init --yes jellyfin1 $ rffmpeg add --weight 1 gpu1 ``` +### NFS Setup + +* **WARNING:** This guide assumes your hosts are on the same private local network. It is not recommended to run NFS over the Internet as it is unencrypted and bandwidth-intensive. Consider other remote filesystems like SSHFS in such cases as these will offer greater privacy and robustness. + 1. Install the NFS kernel server. We will use NFS to export the various required directories so the transcode machine can read from and write to them. ``` @@ -128,6 +136,14 @@ This guide is provided as a basic starting point - there are myriad possible com 1. Install the `jellyfin-ffmpeg` (Jellyfin <= 10.7.7) or `jellyfin-ffmpeg5` (Jellyfin >= 10.8.0) package; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just `jellyfin-ffmpeg[5]`. + ``` + transcode1 $ sudo apt -y install curl gnupg + transcode1 $ curl -fsSL https://repo.jellyfin.org/ubuntu/jellyfin_team.gpg.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/jellyfin.gpg + transcode1 $ echo "deb [arch=$( dpkg --print-architecture )] https://repo.jellyfin.org/$( awk -F'=' '/^ID=/{ print $NF }' /etc/os-release ) $( awk -F'=' '/^VERSION_CODENAME=/{ print $NF }' /etc/os-release ) main" | sudo tee /etc/apt/sources.list.d/jellyfin.list + transcode1 $ sudo apt update + transcode1 $ sudo apt install jellyfin-ffmpeg5 # or jellyfin-ffmpeg with Jellyfin <= 10.7.7 + ``` + 1. Install the NFS client utilities: ``` @@ -152,7 +168,7 @@ This guide is provided as a basic starting point - there are myriad possible com transcode1 $ sudo chattr +i ${jellyfin_data_path} ``` - * Don't worry about permissions here; the mount will set those. + * **NOTE:** Don't worry about permissions here; the mount will set those. 1. Create the NFS client mount. There are two main ways to do this: @@ -165,31 +181,31 @@ This guide is provided as a basic starting point - there are myriad possible com * Use a SystemD `mount` unit, which is a newer way of doing mounts with SystemD. I personally prefer this method as I find it easier to set up automatically, but this is up to preference. An example based on mine would be: - ``` - transcode1 $ cat /etc/systemd/system/var-lib-jellyfin.mount - [Unit] - Description = NFS volume for Jellyfin data directory - Requires = network-online.target - After = network-online.target + ``` + transcode1 $ cat /etc/systemd/system/var-lib-jellyfin.mount + [Unit] + Description = NFS volume for Jellyfin data directory + Requires = network-online.target + After = network-online.target - [Mount] - type = nfs - What = jellyfin1:/var/lib/jellyfin - Where = /var/lib/jellyfin - Options = _netdev,sync,vers=3 + [Mount] + type = nfs + What = jellyfin1:/var/lib/jellyfin + Where = /var/lib/jellyfin + Options = _netdev,sync,vers=3 - [Install] - WantedBy = remote-fs.target - ``` + [Install] + WantedBy = remote-fs.target + ``` - Once the unit file is created, you can then reload the unit list and mount it: + Once the unit file is created, you can then reload the unit list and mount it: - ``` - transcode1 $ sudo systemctl daemon-reload - transcode1 $ sudo systemctl start var-lib-jellyfin.mount - ``` + ``` + transcode1 $ sudo systemctl daemon-reload + transcode1 $ sudo systemctl start var-lib-jellyfin.mount + ``` - Note that mount units are fairly "new" and can be a bit finicky, be sure to read the SystemD documentation if you get stuck! Generally for new users, I'd recommend the `/etc/fstab` method instead. + Note that mount units are fairly "new" and can be a bit finicky, be sure to read the SystemD documentation if you get stuck! Generally for new users, I'd recommend the `/etc/fstab` method instead. 1. Mount your media directories in the same location(s) as on the media server. If you exported them via NFS from your media server, use the process above only for those directories instead. @@ -229,7 +245,7 @@ As long as these steps work, all further steps should as well. 1. Try to play a movie that requires transcoding, and verify that everything is working as expected. -## NOTE for NVEnv/NVDec HWA +## NOTE for NVEnv/NVDec Hardware Acceleration If you are using NVEnv/NVDec, it's probably a good idea to symlink the `.nv` folder inside the Jellyfin user's homedir (i.e. `/var/lib/jellyfin/.nv`) to somewhere outside of the NFS volume on both sides. For example: From f071e4544d4270b4fa8865d89c09a8532dd9e35a Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 20 Jul 2022 03:18:41 -0400 Subject: [PATCH 025/115] Add another formatting tweak --- SETUP.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/SETUP.md b/SETUP.md index 5ab352a..826435e 100644 --- a/SETUP.md +++ b/SETUP.md @@ -174,10 +174,10 @@ This guide is provided as a basic starting point - there are myriad possible com * Use the traditional `/etc/fstab` by adding a new entry like so, replacing the paths and hostname as required, and then mounting it: - ``` - transcode1 $ echo "jellyfin1:${jellyfin_data_path} ${jellyfin_data_path} nfs defaults,vers=3,sync" | sudo tee -a /etc/fstab - transcode1 $ sudo mount ${jellyfin_data_path} - ``` + ``` + transcode1 $ echo "jellyfin1:${jellyfin_data_path} ${jellyfin_data_path} nfs defaults,vers=3,sync" | sudo tee -a /etc/fstab + transcode1 $ sudo mount ${jellyfin_data_path} + ``` * Use a SystemD `mount` unit, which is a newer way of doing mounts with SystemD. I personally prefer this method as I find it easier to set up automatically, but this is up to preference. An example based on mine would be: From 64b0da5edcd845a1292f6db226a2b2851809a664 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 22 Jul 2022 02:30:21 -0400 Subject: [PATCH 026/115] Update wording around localhost --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 6046626..a4f22d2 100644 --- a/README.md +++ b/README.md @@ -70,11 +70,11 @@ The `rffmpeg` CLI offers a convenient way to view the log file. Use `rffmpeg log ### Localhost and Fallback -If one of the hosts in the config file is called "localhost", `rffmpeg` will run locally without SSH. This can be useful if the local machine is also a powerful transcoding device. +If one of the configured target hosts is called `localhost` or `127.0.0.1`, `rffmpeg` will run the `ffmpeg`/`ffprobe` commands locally without SSH. This can be useful if the local machine is also a powerful transcoding device, but you still want to offload some transcoding jobs to other machines. -In addition, `rffmpeg` will fall back to "localhost" should it be unable to find any working remote hosts. This helps prevent situations where `rffmpeg` cannot be run due to none of the remote host(s) being available. +In addition, `rffmpeg` will fall back to `localhost` automatically, even if it is not explicitly configured, should it be unable to find any working remote hosts. This helps prevent situations where `rffmpeg` cannot be run due to none of the remote host(s) being available. -In both cases, note that, if hardware acceleration is configured, it *must* be available on the local host as well, or the `ffmpeg` commands will fail. There is no easy way around this without rewriting arguments, and this is currently out-of-scope for `rffmpeg`. You should always use a lowest-common-denominator approach when deciding on what additional option(s) to enable, such that any configured host can run any process. +In both cases, note that, if hardware acceleration is configured, it *must* be available on the local host as well, or the `ffmpeg` commands will fail. There is no easy way around this without rewriting arguments, and this is currently out-of-scope for `rffmpeg`. You should always use a lowest-common-denominator approach when deciding on what additional option(s) to enable, such that any configured host can run any process, or accept that fallback will not work if all remote hosts are unavailable. The exact path to the local `ffmpeg` and `ffprobe` binaries can be overridden in the configuration, should their paths not match those of the remote system(s). From fadfff34f22f72a962411851a05d9f5c4e262b1b Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 22 Jul 2022 02:33:06 -0400 Subject: [PATCH 027/115] Reference previous section in state guide --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a4f22d2..32343e8 100644 --- a/README.md +++ b/README.md @@ -92,7 +92,7 @@ When more than one target host is present, `rffmpeg` uses the following rules to c. If the host is `active` (has at least one running process), it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role. -1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. +1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. If no valid hosts were found, `localhost`is used (see section [Localhost and Fallback](README.md#localhost-and-fallback) above). ### Target Host Weights and Duplicated Target Hosts From 200726768f18893e1028444c7abb6a9f48435a85 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 22 Jul 2022 02:33:56 -0400 Subject: [PATCH 028/115] Fix incorrect grammar --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 32343e8..2b5b910 100644 --- a/README.md +++ b/README.md @@ -92,7 +92,7 @@ When more than one target host is present, `rffmpeg` uses the following rules to c. If the host is `active` (has at least one running process), it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role. -1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. If no valid hosts were found, `localhost`is used (see section [Localhost and Fallback](README.md#localhost-and-fallback) above). +1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. If no valid target host was found, `localhost`is used (see section [Localhost and Fallback](README.md#localhost-and-fallback) above). ### Target Host Weights and Duplicated Target Hosts From 79733f7261de4751f99e3578cfe24a7ebd9ef88f Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 22 Jul 2022 02:34:23 -0400 Subject: [PATCH 029/115] Fix missing space --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2b5b910..7d6c7d3 100644 --- a/README.md +++ b/README.md @@ -92,7 +92,7 @@ When more than one target host is present, `rffmpeg` uses the following rules to c. If the host is `active` (has at least one running process), it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role. -1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. If no valid target host was found, `localhost`is used (see section [Localhost and Fallback](README.md#localhost-and-fallback) above). +1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. If no valid target host was found, `localhost` is used (see section [Localhost and Fallback](README.md#localhost-and-fallback) above). ### Target Host Weights and Duplicated Target Hosts From 600e6912cc400802009df37131f415f4011d84e5 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 22 Jul 2022 02:37:49 -0400 Subject: [PATCH 030/115] Reword the weighting section --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7d6c7d3..37f2dc2 100644 --- a/README.md +++ b/README.md @@ -100,9 +100,9 @@ When adding a host to `rffmpeg`, a weight can be specified. Weights are used dur For example, consider two hosts: `host1` with weight 1, and `host2` with weight 5. `host2` would have its actual number of processes floor divided by `5`, and thus any number of processes under `5` would count as `0`, any number of processes between `5` and `10` would count as `1`, and so on, resulting in `host2` being chosen over `host1` even if it had several processes. Thus, `host2` would on average handle 5x more `ffmpeg` processes than `host1` would. -Host weighting is a fairly blunt instrument, and only becomes important when many simultaneous `ffmpeg` processes/transcodes are occurring at once, and where the target hosts have very different performance profiles. Generally leaving all hosts at weight 1 would be sufficient for most use-cases. +Host weighting is a fairly blunt instrument, and only becomes important when many simultaneous `ffmpeg` processes/transcodes are occurring at once across at least 2 remote hosts, and where the target hosts have significantly different performance profiles. Generally leaving all hosts at weight 1 would be sufficient for most use-cases. -Furthermore, it is possible to add a host of the same name more than once in the `rffmpeg add` command. This has a very similar, but subtly different, effect from setting a higher weight. A host present in the list is more likely to be seen before another host, and thus this can further influence the desired target. +Furthermore, it is possible to add a host of the same name more than once in the `rffmpeg add` command. This is functionally equivalent to setting the host with a higher weight, but may have some subtle effects on host selection beyond what weight alone can do; this is probably not worthwhile but is left in for the option. ### `bad` Hosts From ed73089fb7a77bc9b70768c3d4a2b72f1a60b36e Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 22 Jul 2022 02:40:01 -0400 Subject: [PATCH 031/115] Reword bad host section --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 37f2dc2..33ee775 100644 --- a/README.md +++ b/README.md @@ -106,9 +106,9 @@ Furthermore, it is possible to add a host of the same name more than once in the ### `bad` Hosts -As mentioned above under [Target Host Selection](README.md#target-host-selection), a host can be marked `bad` if it does not respond to an `ffmpeg -version` command in at least 1 second. This can happen because a host is offline, unreachable, overloaded, or otherwise unresponsive. +As mentioned above under [Target Host Selection](README.md#target-host-selection), a host can be marked `bad` if it does not respond to an `ffmpeg -version` command in at least 1 second if it is due to be checked as a target for a new `rffmpeg` alias process. This can happen because a host is offline, unreachable, overloaded, or otherwise unresponsive. -Once a host is marked `bad`, it will remain so for as long as the `rffmpeg` process that marked it `bad` is running. This can last from a few seconds to several tens of minutes. During this time, any new `rffmpeg` processes that start will see the host marked as `bad` and thus skip it for target selection. Once the marking `rffmpeg` process completes or is terminated, the `bad` status of that host will be cleared, allowing the next run to try it again. This strikes a balance between always retrying known-unresponsive hosts over and over (and thus delaying process startup), and ensuring that hosts will eventually be retried. +Once a host is marked `bad`, it will remain so for as long as the `rffmpeg` process that marked it `bad` is running. This can last anywhere from a few seconds (library scan processes, image extraction) to several tens of minutes (a long video transcode). During this time, any new `rffmpeg` processes that start will see that the host is marked as `bad` and thus skip it for target selection. Once the marking `rffmpeg` process completes or is terminated, the `bad` status of that host will be cleared, allowing the next run to try it again. This strikes a balance between always retrying known-unresponsive hosts over and over (and thus delaying process startup), and ensuring that hosts will eventually be retried. If for some reason all configured hosts are marked `bad`, fallback will be engaged; see the above section [Localhost and Fallback](README.md#localhost-and-fallback) for details on what occurs in this situation. An explicit `localhost` host entry cannot be marked `bad`. From 8f76a5fb24c976c0059b85968ba301f8489703cd Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 22 Jul 2022 02:51:27 -0400 Subject: [PATCH 032/115] Mention style of verbatim commands --- SETUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SETUP.md b/SETUP.md index 826435e..c20017f 100644 --- a/SETUP.md +++ b/SETUP.md @@ -1,6 +1,6 @@ # Example Setup Guide -This example setup is the one I use for `rffmpeg` with Jellyfin. It uses 2 servers: a media server running Jellyfin called `jellyfin1`, and a remote transcode server called `transcode1`. Both systems run Debian GNU/Linux, though the commands below should also work on Ubuntu. Throughout this guide I assume you are running as an unprivileged user with `sudo` privileges (i.e. in the group `sudo`). Basic knowledge of Linux CLI usage is assumed. +This example setup is the one I use for `rffmpeg` with Jellyfin. It uses 2 servers: a media server running Jellyfin called `jellyfin1`, and a remote transcode server called `transcode1`. Both systems run Debian GNU/Linux, though the commands below should also work on Ubuntu. Throughout this guide I assume you are running as an unprivileged user with `sudo` privileges (i.e. in the group `sudo`). Basic knowledge of Linux CLI usage is assumed. Whenever a verbatim command is specified, it will be prefixed by the relevant host to run it on (either `jellyfin1` or `transcode1`) and then a `$` prompt indicator. This guide is provided as a basic starting point - there are myriad possible combinations of systems, and I try to keep `rffmpeg` quite flexible. Feel free to experiment. From 8d1a8dcb77a3c232987fcb709c4a580818d8bb5d Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 22 Jul 2022 02:52:14 -0400 Subject: [PATCH 033/115] Mention output style --- SETUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SETUP.md b/SETUP.md index c20017f..e9bbf5d 100644 --- a/SETUP.md +++ b/SETUP.md @@ -1,6 +1,6 @@ # Example Setup Guide -This example setup is the one I use for `rffmpeg` with Jellyfin. It uses 2 servers: a media server running Jellyfin called `jellyfin1`, and a remote transcode server called `transcode1`. Both systems run Debian GNU/Linux, though the commands below should also work on Ubuntu. Throughout this guide I assume you are running as an unprivileged user with `sudo` privileges (i.e. in the group `sudo`). Basic knowledge of Linux CLI usage is assumed. Whenever a verbatim command is specified, it will be prefixed by the relevant host to run it on (either `jellyfin1` or `transcode1`) and then a `$` prompt indicator. +This example setup is the one I use for `rffmpeg` with Jellyfin. It uses 2 servers: a media server running Jellyfin called `jellyfin1`, and a remote transcode server called `transcode1`. Both systems run Debian GNU/Linux, though the commands below should also work on Ubuntu. Throughout this guide I assume you are running as an unprivileged user with `sudo` privileges (i.e. in the group `sudo`). Basic knowledge of Linux CLI usage is assumed. Whenever a verbatim command is specified, it will be prefixed by the relevant host to run it on (either `jellyfin1` or `transcode1`) and then a `$` prompt indicator. Any command output is usually not shown unless it is relevant. This guide is provided as a basic starting point - there are myriad possible combinations of systems, and I try to keep `rffmpeg` quite flexible. Feel free to experiment. From e84651138af97ff14e7fe6c73dd6a0a79560cc21 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 22 Jul 2022 02:56:07 -0400 Subject: [PATCH 034/115] Clarify wording of media export --- SETUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SETUP.md b/SETUP.md index e9bbf5d..07ae712 100644 --- a/SETUP.md +++ b/SETUP.md @@ -105,7 +105,7 @@ This guide is provided as a basic starting point - there are myriad possible com * Note the security options of NFS. It will limit mounts to the IP addresses specified. If your home network is secure, you can use the entire network, e.g. `192.168.0.0/24`, but I would recommend determining the exact IP of your transcode server(s) and use them explicitly, e.g. for this example `192.168.0.101` and `192.168.0.102`. * The `sync` option is very important here. Jellyfin (and presumably Emby) determines that the next chunk is ready by waiting on inotifies in this directory (I think). Thus, we'd want the client to always do an `fsync` call after every write or the server might miss chunks which results in poor playback performance. * For the above reason, it's also very important that you export *from* the Jellyfin server and not from the transcode server. - * If your media is local to the Jellyfin server (and not otherwise mounted via a remote filesystems like NFS, Samba, CephFS, etc.), also add an export for it as well. + * If your media is local to the Jellyfin server (and not already mountable on the transcode host via a remote filesystems like NFS, Samba, CephFS, etc.), also add an export for it as well. An example `/etc/exports` file would look like this: From 0100731b549ff03c2e9a86a2b515928936feb939 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 22 Jul 2022 02:57:08 -0400 Subject: [PATCH 035/115] Add media to example exports --- SETUP.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/SETUP.md b/SETUP.md index 07ae712..2d70627 100644 --- a/SETUP.md +++ b/SETUP.md @@ -116,7 +116,9 @@ This guide is provided as a basic starting point - there are myriad possible com # Other examples removed # jellyfin_data_path first host second host, etc. - /var/lib/jellyfin 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102(rw,sync,no_subtree_check,no_root_squash) + /var/lib/jellyfin 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102/32(rw,sync,no_subtree_check,no_root_squash) + # Local media path if required + /srv/mymedia 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102/32(rw,sync,no_subtree_check,no_root_squash) ``` 1. Reload the exports file and ensure the NFS server is properly exporting it now: From 53290d73441631ea93a5efa4b55898118be8fc36 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Sun, 24 Jul 2022 12:27:11 -0400 Subject: [PATCH 036/115] Ensure nonetype list args become empty lists Prevents a potential bug like in #23 --- rffmpeg | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rffmpeg b/rffmpeg index fd17c07..2dd80a2 100755 --- a/rffmpeg +++ b/rffmpeg @@ -120,11 +120,15 @@ def load_config(): config["remote_args"] = config_remote.get( "args", ["-i", "/var/lib/jellyfin/.ssh/id_rsa"] ) + if config["remote_args"] is None: + config["remote_args"] = [] config["persist_time"] = config_remote.get("persist", 300) # Parse the keys from the commands group config["ssh_command"] = config_commands.get("ssh", "/usr/bin/ssh") config["pre_commands"] = config_commands.get("pre", []) + if config["pre_commands"] is None: + config["pre_commands"] = [] config["ffmpeg_command"] = config_commands.get( "ffmpeg", "/usr/lib/jellyfin-ffmpeg/ffmpeg" ) From 57cb6f8b67bd86336d02a1eb4b81eb065703f88a Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Sun, 24 Jul 2022 12:28:14 -0400 Subject: [PATCH 037/115] Fix formatting error --- SETUP.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/SETUP.md b/SETUP.md index 2d70627..3bbb77e 100644 --- a/SETUP.md +++ b/SETUP.md @@ -33,7 +33,7 @@ This guide is provided as a basic starting point - there are myriad possible com * `.ssh/`: This doesn't exist yet but will after the next step. 1. Create an SSH keypair to use for `rffmpeg`'s login to the remote server. For ease of use with the following steps, use the Jellyfin service user (`jellyfin`) to create the keypair and store it under its home directory (the Jellyfin data path above). I use `rsa` here but you can substitute `ed25519` instead (avoid `dsa` and `ecdsa` for reasons I won't get into here). Once done, copy the public key to `authorized_keys` which will be used to authenticate the key later. - + ``` jellyfin1 $ sudo -u jellyfin mkdir ${jellyfin_data_path}/.ssh jellyfin1 $ sudo chmod 700 ${jellyfin_data_path}/.ssh @@ -116,9 +116,9 @@ This guide is provided as a basic starting point - there are myriad possible com # Other examples removed # jellyfin_data_path first host second host, etc. - /var/lib/jellyfin 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102/32(rw,sync,no_subtree_check,no_root_squash) + /var/lib/jellyfin 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102/32(rw,sync,no_subtree_check,no_root_squash) # Local media path if required - /srv/mymedia 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102/32(rw,sync,no_subtree_check,no_root_squash) + /srv/mymedia 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102/32(rw,sync,no_subtree_check,no_root_squash) ``` 1. Reload the exports file and ensure the NFS server is properly exporting it now: From fdbf0b8d911f5ff48f0b824d9bf93bebe1104611 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Sun, 24 Jul 2022 15:03:58 -0400 Subject: [PATCH 038/115] Add command output when host marked bad Allows easy replication of the failed command. --- rffmpeg | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index 2dd80a2..44359ce 100755 --- a/rffmpeg +++ b/rffmpeg @@ -280,11 +280,16 @@ def get_target_host(config): if retcode != 0: # Mark the host as bad with dbconn(config) as cur: - log.info( + log.warning( "Marking host {} as bad due to retcode {}".format( host["hostname"], retcode ) ) + log.info( + "SSH command was: {}".format( + " ".join(test_ssh_command + test_ffmpeg_command) + ) + ) cur.execute( "INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", (hid, config["current_pid"], "bad"), From 7c20076a7c4cc251510d6f10a35ad1f456be7cd8 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Sun, 24 Jul 2022 15:08:06 -0400 Subject: [PATCH 039/115] Move logging and continue out of contextmanager --- rffmpeg | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/rffmpeg b/rffmpeg index 44359ce..7e78446 100755 --- a/rffmpeg +++ b/rffmpeg @@ -279,22 +279,22 @@ def get_target_host(config): ) if retcode != 0: # Mark the host as bad + log.warning( + "Marking host {} as bad due to retcode {}".format( + host["hostname"], retcode + ) + ) + log.info( + "SSH command was: {}".format( + " ".join(test_ssh_command + test_ffmpeg_command) + ) + ) with dbconn(config) as cur: - log.warning( - "Marking host {} as bad due to retcode {}".format( - host["hostname"], retcode - ) - ) - log.info( - "SSH command was: {}".format( - " ".join(test_ssh_command + test_ffmpeg_command) - ) - ) cur.execute( "INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", (hid, config["current_pid"], "bad"), ) - continue + continue # If the host state is idle, we can use it immediately if host["current_state"] == "idle": From ba07bb803c3134cb3141e82c894727c0848d7014 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 25 Jul 2022 13:43:22 -0400 Subject: [PATCH 040/115] Solve incorrect ffprobe invocation Fixes #24 --- rffmpeg | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/rffmpeg b/rffmpeg index 7e78446..7eeab44 100755 --- a/rffmpeg +++ b/rffmpeg @@ -133,13 +133,13 @@ def load_config(): "ffmpeg", "/usr/lib/jellyfin-ffmpeg/ffmpeg" ) config["ffprobe_command"] = config_commands.get( - "ffprobe", "/usr/lib/jellyfin-ffprobe/ffprobe" + "ffprobe", "/usr/lib/jellyfin-ffmpeg/ffprobe" ) config["fallback_ffmpeg_command"] = config_commands.get( "fallback_ffmpeg", "/usr/lib/jellyfin-ffmpeg/ffmpeg" ) config["fallback_ffprobe_command"] = config_commands.get( - "fallback_ffprobe", "/usr/lib/jellyfin-ffprobe/ffprobe" + "fallback_ffprobe", "/usr/lib/jellyfin-ffmpeg/ffprobe" ) # Set the database path @@ -327,7 +327,7 @@ def run_local_ffmpeg(config, ffmpeg_args): stdin = sys.stdin stderr = sys.stderr - if cmd_name == "ffprobe": + if "ffprobe" in cmd_name: # If we're in ffprobe mode use that command and sys.stdout as stdout rffmpeg_ffmpeg_command.append(config["fallback_ffprobe_command"]) stdout = sys.stdout @@ -375,7 +375,7 @@ def run_remote_ffmpeg(config, target_hid, target_host, ffmpeg_args): stdin = sys.stdin stderr = sys.stderr - if cmd_name == "ffprobe": + if "ffprobe" in cmd_name: # If we're in ffprobe mode use that command and sys.stdout as stdout rffmpeg_ffmpeg_command.append(config["ffprobe_command"]) stdout = sys.stdout From 33734720a8996b3aeafa6d0de73cc0ec180a61a0 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 25 Jul 2022 13:50:25 -0400 Subject: [PATCH 041/115] Add more detailed output to SSH test Will help debug failed SSH attempts more conveniently. --- rffmpeg | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/rffmpeg b/rffmpeg index 7eeab44..b4bcef0 100755 --- a/rffmpeg +++ b/rffmpeg @@ -33,7 +33,7 @@ from pathlib import Path from pwd import getpwnam from re import search from sqlite3 import connect as sqlite_connect -from subprocess import run +from subprocess import run, PIPE from time import sleep @@ -211,6 +211,23 @@ def generate_ssh_command(config, target_host): return ssh_command +def run_test_command(command): + """ + Execute the test command using subprocess. + """ + log.info("Executing command") + p = run( + command, + shell=False, + bufsize=0, + universal_newlines=True, + stdin=PIPE, + stdout=PIPE, + stderr=PIPE, + ) + return p + + def run_command(command, stdin, stdout, stderr): """ Execute the command using subprocess. @@ -274,10 +291,8 @@ def get_target_host(config): if host["hostname"] not in ["localhost", "127.0.0.1"]: test_ssh_command = generate_ssh_command(config, host["hostname"]) test_ffmpeg_command = [config["ffmpeg_command"], "-version"] - retcode = run_command( - test_ssh_command + test_ffmpeg_command, None, None, None - ) - if retcode != 0: + ret = run_test_command(test_ssh_command + test_ffmpeg_command) + if ret.returncode != 0: # Mark the host as bad log.warning( "Marking host {} as bad due to retcode {}".format( @@ -285,10 +300,16 @@ def get_target_host(config): ) ) log.info( - "SSH command was: {}".format( + "SSH test command was: {}".format( " ".join(test_ssh_command + test_ffmpeg_command) ) ) + log.info( + "SSH test command stdout: {}".format(ret.stdout.decode("utf8")) + ) + log.info( + "SSH test command stderr: {}".format(ret.stderr.decode("utf8")) + ) with dbconn(config) as cur: cur.execute( "INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", From fa3ea4bed024008ed1e82579c801614f5c363543 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 25 Jul 2022 14:02:22 -0400 Subject: [PATCH 042/115] Add debug log support and additional debug logs Also fix decode bug from previous commit and re-unify test command function. --- rffmpeg | 76 +++++++++++++++++++++++----------------------- rffmpeg.yml.sample | 3 ++ 2 files changed, 41 insertions(+), 38 deletions(-) diff --git a/rffmpeg b/rffmpeg index b4bcef0..0e29177 100755 --- a/rffmpeg +++ b/rffmpeg @@ -108,6 +108,7 @@ def load_config(): # Parse the keys from the logging group config["log_to_file"] = config_logging.get("log_to_file", True) config["logfile"] = config_logging.get("logfile", "/var/log/jellyfin/rffmpeg.log") + config["logdebug"] = config_logging.get("debug", False) # Parse the keys from the state group config["state_dir"] = config_directories.get("state", "/var/lib/rffmpeg") @@ -211,28 +212,10 @@ def generate_ssh_command(config, target_host): return ssh_command -def run_test_command(command): - """ - Execute the test command using subprocess. - """ - log.info("Executing command") - p = run( - command, - shell=False, - bufsize=0, - universal_newlines=True, - stdin=PIPE, - stdout=PIPE, - stderr=PIPE, - ) - return p - - def run_command(command, stdin, stdout, stderr): """ Execute the command using subprocess. """ - log.info("Executing command") p = run( command, shell=False, @@ -242,13 +225,14 @@ def run_command(command, stdin, stdout, stderr): stdout=stdout, stderr=stderr, ) - return p.returncode + return p def get_target_host(config): """ Determine an optimal target host via data on currently active processes and states. """ + log.debug("Determining optimal target host") # Select all hosts and active processes from the database with dbconn(config) as cur: hosts = cur.execute("SELECT * FROM hosts").fetchall() @@ -267,14 +251,17 @@ def get_target_host(config): if not current_state: current_state = "idle" + marking_pid = 'N/A' else: current_state = current_state[3] + marking_pid = current_state[2] # Create the mappings entry host_mappings[hid] = { "hostname": hostname, "weight": weight, "current_state": current_state, + "marking_pid": marking_pid, "commands": [proc[2] for proc in processes if proc[1] == hid], } @@ -283,32 +270,35 @@ def get_target_host(config): target_hostname = None # For each host in the mapping, let's determine if it is suitable for hid, host in host_mappings.items(): + log.debug("Trying host ID {} '{}'".format(hid, host["hostname"])) # If it's marked as bad, continue if host["current_state"] == "bad": + log.debug("Host previously marked bad by PID {}".format(host['marking_pid'])) continue # Try to connect to the host and run a very quick command to determine if it is workable if host["hostname"] not in ["localhost", "127.0.0.1"]: + log.debug("Running SSH test") test_ssh_command = generate_ssh_command(config, host["hostname"]) test_ffmpeg_command = [config["ffmpeg_command"], "-version"] - ret = run_test_command(test_ssh_command + test_ffmpeg_command) + ret = run_command(test_ssh_command + test_ffmpeg_command, PIPE, PIPE, PIPE) if ret.returncode != 0: # Mark the host as bad log.warning( "Marking host {} as bad due to retcode {}".format( - host["hostname"], retcode + host["hostname"], ret.returncode ) ) - log.info( + log.debug( "SSH test command was: {}".format( " ".join(test_ssh_command + test_ffmpeg_command) ) ) - log.info( - "SSH test command stdout: {}".format(ret.stdout.decode("utf8")) + log.debug( + "SSH test command stdout: {}".format(ret.stdout) ) - log.info( - "SSH test command stderr: {}".format(ret.stderr.decode("utf8")) + log.debug( + "SSH test command stderr: {}".format(ret.stderr) ) with dbconn(config) as cur: cur.execute( @@ -316,11 +306,13 @@ def get_target_host(config): (hid, config["current_pid"], "bad"), ) continue + log.debug("SSH test succeeded with retcode {}".format(ret.returncode)) # If the host state is idle, we can use it immediately if host["current_state"] == "idle": target_hid = hid target_hostname = host["hostname"] + log.debug("Selecting host as idle") break # Get the modified count of the host @@ -332,8 +324,9 @@ def get_target_host(config): lowest_count = weighted_proc_count target_hid = hid target_hostname = host["hostname"] + log.debug("Selecting host as current lowest proc count (raw count: {}, weighted count: {})".format(raw_proc_count, weighted_proc_count)) - log.info("Found optimal host '{}' (ID '{}')".format(target_hostname, target_hid)) + log.debug("Found optimal host ID {} '{}'".format(target_hid, target_hostname)) return target_hid, target_hostname @@ -365,7 +358,8 @@ def run_local_ffmpeg(config, ffmpeg_args): for arg in ffmpeg_args: rffmpeg_ffmpeg_command.append("{}".format(arg)) - log.info("Local command: {}".format(" ".join(rffmpeg_ffmpeg_command))) + log.info("Running command on host 'localhost'") + log.debug("Local command: {}".format(" ".join(rffmpeg_ffmpeg_command))) with dbconn(config) as cur: cur.execute( @@ -417,7 +411,8 @@ def run_remote_ffmpeg(config, target_hid, target_host, ffmpeg_args): else: rffmpeg_ffmpeg_command.append("{}".format(arg)) - log.info( + log.info("Running command on host '{}'".format(target_host)) + log.debug( "Remote command: {}".format( " ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command) ) @@ -447,33 +442,38 @@ def run_ffmpeg(config, ffmpeg_args): signal.signal(signal.SIGQUIT, cleanup) signal.signal(signal.SIGHUP, cleanup) + if config["logdebug"] is True: + logging_level = logging.DEBUG + else: + logging_level = logging.INFO + if config["log_to_file"]: logging.basicConfig( filename=config["logfile"], - level=logging.INFO, + level=logging_level, format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s", ) else: logging.basicConfig( - level=logging.INFO, + level=logging_level, format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s", ) - log.info("Starting rffmpeg with args: {}".format(" ".join(ffmpeg_args))) + log.info("Starting rffmpeg as {} with args: {}".format(cmd_name, " ".join(ffmpeg_args))) target_hid, target_hostname = get_target_host(config) if not target_hostname or target_hostname == "localhost": - retcode = run_local_ffmpeg(config, ffmpeg_args) + ret = run_local_ffmpeg(config, ffmpeg_args) else: - retcode = run_remote_ffmpeg(config, target_hid, target_hostname, ffmpeg_args) + ret = run_remote_ffmpeg(config, target_hid, target_hostname, ffmpeg_args) cleanup() - if retcode == 0: - log.info("Finished rffmpeg with return code {}".format(retcode)) + if ret.returncode == 0: + log.info("Finished rffmpeg with return code {}".format(ret.returncode)) else: - log.error("Finished rffmpeg with return code {}".format(retcode)) - exit(retcode) + log.error("Finished rffmpeg with return code {}".format(ret.returncode)) + exit(ret.returncode) def run_control(config): diff --git a/rffmpeg.yml.sample b/rffmpeg.yml.sample index faf8c9f..d5b4602 100644 --- a/rffmpeg.yml.sample +++ b/rffmpeg.yml.sample @@ -16,6 +16,9 @@ rffmpeg: # Ensure the user running rffmpeg can write to this directory. #logfile: "/var/log/jellyfin/rffmpeg.log" + # Show debugging messages + #debug: false + # Directory configuration directories: # Persistent directory to store state database. From d07869f7586d4131ba7f5ab575fb17a007fb2790 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 25 Jul 2022 14:21:28 -0400 Subject: [PATCH 043/115] Properly capture test SSH output --- rffmpeg | 1 + 1 file changed, 1 insertion(+) diff --git a/rffmpeg b/rffmpeg index 0e29177..b7243be 100755 --- a/rffmpeg +++ b/rffmpeg @@ -280,6 +280,7 @@ def get_target_host(config): if host["hostname"] not in ["localhost", "127.0.0.1"]: log.debug("Running SSH test") test_ssh_command = generate_ssh_command(config, host["hostname"]) + test_ssh_command.remove("-q") test_ffmpeg_command = [config["ffmpeg_command"], "-version"] ret = run_command(test_ssh_command + test_ffmpeg_command, PIPE, PIPE, PIPE) if ret.returncode != 0: From d2dec7e323c6d4870152f8ec8e43f4c91150dcb6 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 25 Jul 2022 14:23:21 -0400 Subject: [PATCH 044/115] Add additional output messages --- rffmpeg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index b7243be..870ea1d 100755 --- a/rffmpeg +++ b/rffmpeg @@ -519,6 +519,8 @@ def run_control(config): except Exception: fail("Aborting due to failed confirmation.") + click.echo("Initializing database") + if not Path(config["state_dir"]).is_dir(): try: os.makedirs(config["state_dir"]) @@ -701,6 +703,7 @@ def run_control(config): """ Add a new host with IP or hostname HOST to the database. """ + click.echo("Adding new host '{}'".format(host)) with dbconn(config) as cur: cur.execute( """INSERT INTO hosts (hostname, weight) VALUES (?, ?)""", (host, weight) @@ -727,7 +730,7 @@ def run_control(config): if len(entry) < 1: fail("No hosts found to delete!") - click.echo("Deleting {} host(s):".format(len(entry))) + click.echo("Removing {} host(s):".format(len(entry))) for h in entry: click.echo("\tID: {}\tHostname: {}".format(h[0], h[1])) cur.execute("""DELETE FROM hosts WHERE id = ?""", (h[0],)) From f75efa2dcc88979556a30ea228efd2365ecd585c Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 25 Jul 2022 14:26:58 -0400 Subject: [PATCH 045/115] Add note about debug logging when reqing help --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 33ee775..a3a1bb0 100644 --- a/README.md +++ b/README.md @@ -44,6 +44,8 @@ The `rffmpeg` configuration file located at `rffmpeg.yml.sample` is an example t To override a default option, simply uncomment the relevant line and adjust it to suit your needs. For those using [Jellyfin](https://jellyfin.org) and following the [SETUP guide](SETUP.md), no default options will need to be changed. +**NOTE:** If you are running into problems with `rffmpeg`, please adjust `logging` -> `debug` to `true` to obtain more detailed logs before requesting help. + Each option has an explanatory comment above it detailing its purpose. Since the configuration file is YAML, ensure that you do not use "Tab" characters inside of it, only spaces. From cdc38a4220579af6c629d878ca9098f3488377b3 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 25 Jul 2022 14:30:55 -0400 Subject: [PATCH 046/115] Add additional detail to help section --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a3a1bb0..da18533 100644 --- a/README.md +++ b/README.md @@ -154,11 +154,13 @@ If the problem persists, please check the [closed issues](https://github.com/jos If it hasn't, you can [ask in our chat](https://matrix.to/#/#rffmpeg:matrix.org) or open a new issue. Ensure you: -1. Use a descriptive and useful title that quickly explains the problem. +1. Enable debug logging in `rffmpeg.yml` (`logging` -> `debug` to `true`) and re-run any failing or incorrect command(s) to obtain debug-level logs for analysis. -1. Clearly explain in the body of the issue your setup, what is going wrong, and what you expect should be happening. Don't fret if English isn't your first language or anything like that, as long as you are trying to be clear that's what counts! +1. For issues, use a descriptive and useful title that quickly explains the problem. -1. Include your `rffmpeg.log` and Jellyfin/Emby `ffmpeg-transcode-*.txt` logs. +1. Clearly explain (in the body of the issue or in your chat message) your setup, what is going wrong, and what you expect should be happening. Don't fret if English isn't your first language or anything like that, as long as you are trying to be clear that's what counts! + +1. Include your `rffmpeg.log` and Jellyfin/Emby transcode logs as these are absolutely critical in determining what is going on. Use triple-backticks ("```") to enclose logs inline, both in chat and in issues. I will probably ask clarifying questions as required; please be prepared to run test commands, etc. as requested and paste the output. From 8055ee13c258307bff91bf0a60e7b6b3e63f8cd9 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 25 Jul 2022 14:36:05 -0400 Subject: [PATCH 047/115] Try fixing section links in README --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index da18533..88e64ed 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ When more than one target host is present, `rffmpeg` uses the following rules to c. If the host is `active` (has at least one running process), it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role. -1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. If no valid target host was found, `localhost` is used (see section [Localhost and Fallback](README.md#localhost-and-fallback) above). +1. Once all hosts have been iterated through, at least one host should have been chosen: either the first `idle` host, or the host with the fewest number of active processes. `rffmpeg` will then begin running against this host. If no valid target host was found, `localhost` is used (see section [Localhost and Fallback](#localhost-and-fallback) above). ### Target Host Weights and Duplicated Target Hosts @@ -108,11 +108,11 @@ Furthermore, it is possible to add a host of the same name more than once in the ### `bad` Hosts -As mentioned above under [Target Host Selection](README.md#target-host-selection), a host can be marked `bad` if it does not respond to an `ffmpeg -version` command in at least 1 second if it is due to be checked as a target for a new `rffmpeg` alias process. This can happen because a host is offline, unreachable, overloaded, or otherwise unresponsive. +As mentioned above under [Target Host Selection](#target-host-selection), a host can be marked `bad` if it does not respond to an `ffmpeg -version` command in at least 1 second if it is due to be checked as a target for a new `rffmpeg` alias process. This can happen because a host is offline, unreachable, overloaded, or otherwise unresponsive. Once a host is marked `bad`, it will remain so for as long as the `rffmpeg` process that marked it `bad` is running. This can last anywhere from a few seconds (library scan processes, image extraction) to several tens of minutes (a long video transcode). During this time, any new `rffmpeg` processes that start will see that the host is marked as `bad` and thus skip it for target selection. Once the marking `rffmpeg` process completes or is terminated, the `bad` status of that host will be cleared, allowing the next run to try it again. This strikes a balance between always retrying known-unresponsive hosts over and over (and thus delaying process startup), and ensuring that hosts will eventually be retried. -If for some reason all configured hosts are marked `bad`, fallback will be engaged; see the above section [Localhost and Fallback](README.md#localhost-and-fallback) for details on what occurs in this situation. An explicit `localhost` host entry cannot be marked `bad`. +If for some reason all configured hosts are marked `bad`, fallback will be engaged; see the above section [Localhost and Fallback](#localhost-and-fallback) for details on what occurs in this situation. An explicit `localhost` host entry cannot be marked `bad`. ## FAQ @@ -136,7 +136,7 @@ Explicitly *no*. `rffmpeg` is not designed to interact with the arguments that t This has a number of side effects: - * `rffmpeg` does not know whether hardware acceleration is turned on or not (see above caveats under [Localhost and Fallback](README.md#localhost-and-fallback)). + * `rffmpeg` does not know whether hardware acceleration is turned on or not (see above caveats under [Localhost and Fallback](#localhost-and-fallback)). * `rffmpeg` does not know what media is playing or where it's outputting files to, and cannot alter these paths. * `rffmpeg` cannot turn on or off special `ffmpeg` options depending on the host selected. From b8b62d3865bda1cc0809def3a07225262d774117 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 25 Jul 2022 14:45:21 -0400 Subject: [PATCH 048/115] Reference Shadowghost's docker containers --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 88e64ed..471784a 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# rffmpeg +

License: GPLv3+ @@ -127,8 +127,8 @@ This depends on what "layer" you're asking at. * Media Servers: Jellyfin is officially supported; Emby seems to work fine, with caveats (see [Issue #10](https://github.com/joshuaboniface/rffmpeg/issues/10)); no others have been tested to my knowledge. * Operating Systems (source): Debian and its derivatives (Ubuntu, Linux Mint, etc.) should all work perfectly; other Linux operating systems should work fine too as the principles are the same; MacOS should work since it has an SSH client built in; Windows will not work as `rffmpeg` depends on some POSIX assumptions internally. * Operating Systems (target): Any Linux system which [`jellyfin-ffmpeg`](https://github.com/jellyfin/jellyfin-ffmpeg) supports, which is currently just Debian and Ubuntu; Windows *might* work if you can get an SSH server running on it (see [Issue #17](https://github.com/joshuaboniface/rffmpeg/issues/17)). -* Install Methods for Jellyfin: Native packages/installers/archives are recommended; Docker containers can be made to work by exporting the `/config` path (see [the setup guide](SETUP.md)) but this is slightly more difficult and is not explicitly covered in the guide. -* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by @BasixKOR. +* Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost). +* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by [@BasixKOR](https://github.com/BasixKOR). ### Can `rffmpeg` mangle/alter FFMPEG arguments? From 6458bc85b758b696073adb52af57cfa27eac0621 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Tue, 26 Jul 2022 01:16:52 -0400 Subject: [PATCH 049/115] Add blurb about paths in Docker --- README.md | 2 +- SETUP.md | 7 +++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 471784a..ddb3350 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ 1. Profit! -For a comprehensive installation tutorial based on a reference setup, please see [the SETUP guide](SETUP.md). +`rffmpeg` does require a little bit more configuration to work properly however. For a comprehensive installation tutorial based on a reference setup, please see [the SETUP guide](SETUP.md). ## Important Considerations diff --git a/SETUP.md b/SETUP.md index 3bbb77e..ca510f5 100644 --- a/SETUP.md +++ b/SETUP.md @@ -28,10 +28,13 @@ This guide is provided as a basic starting point - there are myriad possible com The important subdirectories for `rffmpeg`'s operation are: - * `transcodes/`: used to store on-the-fly transcoding files, and configurable separately in Jellyfin but with `rffmpeg` I recommend leaving it at the default location under the data path. - * `data/subtitles/`: used to store on-the-fly extracted subtitles so that they can be reused later. + * `transcodes/`: Used to store on-the-fly transcoding files, and configurable separately in Jellyfin but with `rffmpeg` I recommend leaving it at the default location under the data path. + * `data/subtitles/`: Used to store on-the-fly extracted subtitles so that they can be reused later. + * `cache/`: Used to store cached extracted data. * `.ssh/`: This doesn't exist yet but will after the next step. + **NOTE:** On Docker, these directories are different. The main data directory (our `jellyfin_data_path`) is `/config`, and the cache directory is separate at `/cache`. Both must be exported and mounted on targets for proper operation. + 1. Create an SSH keypair to use for `rffmpeg`'s login to the remote server. For ease of use with the following steps, use the Jellyfin service user (`jellyfin`) to create the keypair and store it under its home directory (the Jellyfin data path above). I use `rsa` here but you can substitute `ed25519` instead (avoid `dsa` and `ecdsa` for reasons I won't get into here). Once done, copy the public key to `authorized_keys` which will be used to authenticate the key later. ``` From 883d4333685af606af1685264f1b481fc2635951 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Wed, 27 Jul 2022 17:32:51 -0400 Subject: [PATCH 050/115] Remove second (forced) pseudo-terminal invocation This seems to cause JSON breakage on certain platforms with certain SSH binaries. Ctrl+C still works with only normal pseudoterminal invocation when running from a normal (real) shell, so this isn't needed. --- rffmpeg | 1 - 1 file changed, 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index 870ea1d..1a42450 100755 --- a/rffmpeg +++ b/rffmpeg @@ -185,7 +185,6 @@ def generate_ssh_command(config, target_host): ssh_command.append(config["ssh_command"]) ssh_command.append("-q") ssh_command.append("-t") - ssh_command.append("-t") # Set our connection details ssh_command.extend(["-o", "ConnectTimeout=1"]) From cf5e9daa85801e1f3d0d41415f7dd0764f3d067f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Thu, 17 Nov 2022 15:15:39 +0100 Subject: [PATCH 051/115] Server name field server_name field added for easier naming of the servers and to allow support for hcloud-rffmpeg script --- rffmpeg | 83 +++++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 57 insertions(+), 26 deletions(-) diff --git a/rffmpeg b/rffmpeg index 1a42450..26f1567 100755 --- a/rffmpeg +++ b/rffmpeg @@ -240,7 +240,7 @@ def get_target_host(config): # Generate a mapping dictionary of hosts and processes host_mappings = dict() for host in hosts: - hid, hostname, weight = host + hid, hostname, weight, server_name = host # Get the latest state with dbconn(config) as cur: @@ -259,6 +259,7 @@ def get_target_host(config): host_mappings[hid] = { "hostname": hostname, "weight": weight, + "server_name": server_name, "current_state": current_state, "marking_pid": marking_pid, "commands": [proc[2] for proc in processes if proc[1] == hid], @@ -285,8 +286,8 @@ def get_target_host(config): if ret.returncode != 0: # Mark the host as bad log.warning( - "Marking host {} as bad due to retcode {}".format( - host["hostname"], ret.returncode + "Marking host {} - {} as bad due to retcode {}".format( + host["server_name"], host["hostname"], ret.returncode ) ) log.debug( @@ -312,6 +313,7 @@ def get_target_host(config): if host["current_state"] == "idle": target_hid = hid target_hostname = host["hostname"] + target_server_name = host["server_name"] log.debug("Selecting host as idle") break @@ -326,8 +328,8 @@ def get_target_host(config): target_hostname = host["hostname"] log.debug("Selecting host as current lowest proc count (raw count: {}, weighted count: {})".format(raw_proc_count, weighted_proc_count)) - log.debug("Found optimal host ID {} '{}'".format(target_hid, target_hostname)) - return target_hid, target_hostname + log.debug("Found optimal host {} ID {} '{}'".format(target_server_name, target_hid, target_hostname)) + return target_server_name, target_hid, target_hostname def run_local_ffmpeg(config, ffmpeg_args): @@ -374,7 +376,7 @@ def run_local_ffmpeg(config, ffmpeg_args): return run_command(rffmpeg_ffmpeg_command, stdin, stdout, stderr) -def run_remote_ffmpeg(config, target_hid, target_host, ffmpeg_args): +def run_remote_ffmpeg(config, target_hid, target_host, target_server_name, ffmpeg_args): """ Run ffmpeg against the remote target_host. """ @@ -411,7 +413,7 @@ def run_remote_ffmpeg(config, target_hid, target_host, ffmpeg_args): else: rffmpeg_ffmpeg_command.append("{}".format(arg)) - log.info("Running command on host '{}'".format(target_host)) + log.info("Running command on {} host '{}'".format(target_server_name, target_host)) log.debug( "Remote command: {}".format( " ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command) @@ -461,12 +463,12 @@ def run_ffmpeg(config, ffmpeg_args): log.info("Starting rffmpeg as {} with args: {}".format(cmd_name, " ".join(ffmpeg_args))) - target_hid, target_hostname = get_target_host(config) + target_hid, target_hostname, target_server_name = get_target_host(config) if not target_hostname or target_hostname == "localhost": ret = run_local_ffmpeg(config, ffmpeg_args) else: - ret = run_remote_ffmpeg(config, target_hid, target_hostname, ffmpeg_args) + ret = run_remote_ffmpeg(config, target_hid, target_hostname, target_server_name, ffmpeg_args) cleanup() if ret.returncode == 0: @@ -536,13 +538,13 @@ def run_control(config): try: with dbconn(config) as cur: cur.execute( - """CREATE TABLE hosts (id INTEGER PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1)""" + "CREATE TABLE hosts (id INTEGER PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, server_name TEXT NOT NULL)" ) cur.execute( - """CREATE TABLE processes (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)""" + "CREATE TABLE processes (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)" ) cur.execute( - """CREATE TABLE states (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)""" + "CREATE TABLE states (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)" ) except Exception as e: fail("Failed to create database: {}".format(e)) @@ -590,7 +592,7 @@ def run_control(config): } for host in hosts: - hid, hostname, weight = host + hid, hostname, weight, server_name = host # Get the latest state with dbconn(config) as cur: @@ -607,17 +609,21 @@ def run_control(config): host_mappings[hid] = { "hostname": hostname, "weight": weight, + "server_name": server_name, "current_state": current_state, "commands": [proc for proc in processes if proc[1] == hid], } hostname_length = 9 + server_name_length = 11 hid_length = 3 weight_length = 7 state_length = 6 for hid, host in host_mappings.items(): if len(host["hostname"]) + 1 > hostname_length: hostname_length = len(host["hostname"]) + 1 + if len(host["server_name"]) + 1 > server_name_length: + server_name_length = len(host["server_name"]) + 1 if len(str(hid)) + 1 > hid_length: hid_length = len(str(hid)) + 1 if len(str(host["weight"])) + 1 > weight_length: @@ -627,11 +633,13 @@ def run_control(config): output = list() output.append( - "{bold}{hostname: <{hostname_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}{end_bold}".format( + "{bold}{hostname: <{hostname_length}} {server_name: <{server_name_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}{end_bold}".format( bold="\033[1m", end_bold="\033[0m", hostname="Hostname", hostname_length=hostname_length, + server_name="Server name", + server_name_length=server_name_length, hid="ID", hid_length=hid_length, weight="Weight", @@ -652,9 +660,11 @@ def run_control(config): host_entry = list() host_entry.append( - "{hostname: <{hostname_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}".format( + "{hostname: <{hostname_length}} {server_name: <{server_name_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}".format( hostname=host["hostname"], hostname_length=hostname_length, + server_name=host["server_name"], + server_name_length=server_name_length, hid=hid, hid_length=hid_length, weight=host["weight"], @@ -669,9 +679,11 @@ def run_control(config): if idx == 0: continue host_entry.append( - "{hostname: <{hostname_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}".format( + "{hostname: <{hostname_length}} {server_name: <{server_name_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}".format( hostname="", hostname_length=hostname_length, + server_name="", + server_name_length=server_name_length, hid="", hid_length=hid_length, weight="", @@ -697,15 +709,25 @@ def run_control(config): default=1, help="The weight of the host.", ) + @click.option( + "-n", + "--name", + "name", + required=False, + default="manual", + help="The name of the host.", + ) @click.argument("host") - def rffmpeg_click_add(weight, host): + def rffmpeg_click_add(weight, name, host): """ Add a new host with IP or hostname HOST to the database. """ - click.echo("Adding new host '{}'".format(host)) + click.echo("Adding new {} host '{}'".format(name, host)) + if name == "manual": + name = host with dbconn(config) as cur: cur.execute( - """INSERT INTO hosts (hostname, weight) VALUES (?, ?)""", (host, weight) + "INSERT INTO hosts (hostname, weight, server_name) VALUES (?, ?, ?)", (host, weight, name) ) rffmpeg_click.add_command(rffmpeg_click_add) @@ -714,25 +736,34 @@ def run_control(config): @click.argument("host") def rffmpeg_click_remove(host): """ - Remove a host with internal ID or IP or hostname HOST from the database. + Remove a host with internal ID or IP or hostname or server_name HOST from the database. """ try: host = int(host) field = "id" except ValueError: - field = "hostname" + field = "server_name" + fieldAlt = "hostname" with dbconn(config) as cur: entry = cur.execute( "SELECT * FROM hosts WHERE {} = ?".format(field), (host,) ).fetchall() if len(entry) < 1: - fail("No hosts found to delete!") + entry = cur.execute( + "SELECT * FROM hosts WHERE {} = ?".format(fieldAlt), (host,) + ).fetchall() + if len(entry) < 1: + fail("No hosts found to delete!") - click.echo("Removing {} host(s):".format(len(entry))) - for h in entry: - click.echo("\tID: {}\tHostname: {}".format(h[0], h[1])) - cur.execute("""DELETE FROM hosts WHERE id = ?""", (h[0],)) + if len(entry) == 1: + click.echo("Removing {} host:".format(len(entry))) + else: + click.echo("Removing {} hosts:".format(len(entry))) + for host in entry: + hid, hostname, weight, server_name = host + click.echo("\tID: {}\tHostname: {}\tServer name: {}".format(hid, hostname, server_name)) + cur.execute("DELETE FROM hosts WHERE id = ?", (hid,)) rffmpeg_click.add_command(rffmpeg_click_remove) From b439fc248e4b9fea0759e2fec31bb02e64e5901c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Thu, 17 Nov 2022 15:40:04 +0100 Subject: [PATCH 052/115] Added more docker integrations and Cloud For both the jellyfin server with optional intro-skipper web-ui integration and jellyfin node to act as an ffmpeg transcode target. Also added HCloud script in a new section for Cloud integration. --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ddb3350..25b02bd 100644 --- a/README.md +++ b/README.md @@ -127,8 +127,9 @@ This depends on what "layer" you're asking at. * Media Servers: Jellyfin is officially supported; Emby seems to work fine, with caveats (see [Issue #10](https://github.com/joshuaboniface/rffmpeg/issues/10)); no others have been tested to my knowledge. * Operating Systems (source): Debian and its derivatives (Ubuntu, Linux Mint, etc.) should all work perfectly; other Linux operating systems should work fine too as the principles are the same; MacOS should work since it has an SSH client built in; Windows will not work as `rffmpeg` depends on some POSIX assumptions internally. * Operating Systems (target): Any Linux system which [`jellyfin-ffmpeg`](https://github.com/jellyfin/jellyfin-ffmpeg) supports, which is currently just Debian and Ubuntu; Windows *might* work if you can get an SSH server running on it (see [Issue #17](https://github.com/joshuaboniface/rffmpeg/issues/17)). -* Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost). -* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by [@BasixKOR](https://github.com/BasixKOR). +* Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg-server) and [Jellyfin intro skipper](https://github.com/aleksasiriski/jellyfin-intro-skipper-rffmpeg-server) by [@aleksasiriski](https://github.com/aleksasiriski). +* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by [@BasixKOR](https://github.com/BasixKOR) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg-node) by [@aleksasiriski](https://github.com/aleksasiriski). +* Cloud: [HCloud Rffmpeg](https://github.com/aleksasiriski/hcloud-rffmpeg) script made to read rffmpeg database and spin up more transcode nodes in Hetzner Cloud by [@aleksasiriski](https://github.com/aleksasiriski). ### Can `rffmpeg` mangle/alter FFMPEG arguments? From b20595638a73cda7e14f81e12844ebcd8d223f20 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Thu, 17 Nov 2022 09:42:13 -0500 Subject: [PATCH 053/115] Clean up variable names --- rffmpeg | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/rffmpeg b/rffmpeg index 26f1567..05c3703 100755 --- a/rffmpeg +++ b/rffmpeg @@ -175,7 +175,7 @@ def cleanup(signum="", frame=""): ) -def generate_ssh_command(config, target_host): +def generate_ssh_command(config, target_hostname): """ Generate an SSH command for use. """ @@ -206,7 +206,7 @@ def generate_ssh_command(config, target_host): ssh_command.append(arg) # Add user+host string - ssh_command.append("{}@{}".format(config["remote_user"], target_host)) + ssh_command.append("{}@{}".format(config["remote_user"], target_hostname)) return ssh_command @@ -329,7 +329,7 @@ def get_target_host(config): log.debug("Selecting host as current lowest proc count (raw count: {}, weighted count: {})".format(raw_proc_count, weighted_proc_count)) log.debug("Found optimal host {} ID {} '{}'".format(target_server_name, target_hid, target_hostname)) - return target_server_name, target_hid, target_hostname + return target_hid, target_server_name, target_hostname def run_local_ffmpeg(config, ffmpeg_args): @@ -376,11 +376,11 @@ def run_local_ffmpeg(config, ffmpeg_args): return run_command(rffmpeg_ffmpeg_command, stdin, stdout, stderr) -def run_remote_ffmpeg(config, target_hid, target_host, target_server_name, ffmpeg_args): +def run_remote_ffmpeg(config, target_hid, target_hostname, target_server_name, ffmpeg_args): """ - Run ffmpeg against the remote target_host. + Run ffmpeg against the remote target_hostname. """ - rffmpeg_ssh_command = generate_ssh_command(config, target_host) + rffmpeg_ssh_command = generate_ssh_command(config, target_hostname) rffmpeg_ffmpeg_command = list() # Add any pre commands @@ -413,7 +413,7 @@ def run_remote_ffmpeg(config, target_hid, target_host, target_server_name, ffmpe else: rffmpeg_ffmpeg_command.append("{}".format(arg)) - log.info("Running command on {} host '{}'".format(target_server_name, target_host)) + log.info("Running command on host {} '{}'".format(target_server_name, target_hostname)) log.debug( "Remote command: {}".format( " ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command) @@ -463,7 +463,7 @@ def run_ffmpeg(config, ffmpeg_args): log.info("Starting rffmpeg as {} with args: {}".format(cmd_name, " ".join(ffmpeg_args))) - target_hid, target_hostname, target_server_name = get_target_host(config) + target_hid, target_server_name, target_hostname = get_target_host(config) if not target_hostname or target_hostname == "localhost": ret = run_local_ffmpeg(config, ffmpeg_args) From c60983dd17bad6a81c1e0e70bf175caf4c7bd787 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Thu, 17 Nov 2022 10:00:35 -0500 Subject: [PATCH 054/115] Clean up new field variable and add migration References #36 --- rffmpeg | 87 ++++++++++++++++++++++++++++++++++----------------------- 1 file changed, 52 insertions(+), 35 deletions(-) diff --git a/rffmpeg b/rffmpeg index 05c3703..6123f22 100755 --- a/rffmpeg +++ b/rffmpeg @@ -240,7 +240,7 @@ def get_target_host(config): # Generate a mapping dictionary of hosts and processes host_mappings = dict() for host in hosts: - hid, hostname, weight, server_name = host + hid, hostname, weight, servername = host # Get the latest state with dbconn(config) as cur: @@ -259,7 +259,7 @@ def get_target_host(config): host_mappings[hid] = { "hostname": hostname, "weight": weight, - "server_name": server_name, + "servername": servername, "current_state": current_state, "marking_pid": marking_pid, "commands": [proc[2] for proc in processes if proc[1] == hid], @@ -286,8 +286,8 @@ def get_target_host(config): if ret.returncode != 0: # Mark the host as bad log.warning( - "Marking host {} - {} as bad due to retcode {}".format( - host["server_name"], host["hostname"], ret.returncode + "Marking host {} ({}) as bad due to retcode {}".format( + host["hostname"], host["servername"], ret.returncode ) ) log.debug( @@ -313,7 +313,7 @@ def get_target_host(config): if host["current_state"] == "idle": target_hid = hid target_hostname = host["hostname"] - target_server_name = host["server_name"] + target_servername = host["servername"] log.debug("Selecting host as idle") break @@ -328,8 +328,8 @@ def get_target_host(config): target_hostname = host["hostname"] log.debug("Selecting host as current lowest proc count (raw count: {}, weighted count: {})".format(raw_proc_count, weighted_proc_count)) - log.debug("Found optimal host {} ID {} '{}'".format(target_server_name, target_hid, target_hostname)) - return target_hid, target_server_name, target_hostname + log.debug("Found optimal host ID {} '{}' ({})".format(target_hid, target_hostname, target_servername)) + return target_hid, target_hostname, target_servername def run_local_ffmpeg(config, ffmpeg_args): @@ -376,7 +376,7 @@ def run_local_ffmpeg(config, ffmpeg_args): return run_command(rffmpeg_ffmpeg_command, stdin, stdout, stderr) -def run_remote_ffmpeg(config, target_hid, target_hostname, target_server_name, ffmpeg_args): +def run_remote_ffmpeg(config, target_hid, target_hostname, target_servername, ffmpeg_args): """ Run ffmpeg against the remote target_hostname. """ @@ -413,7 +413,7 @@ def run_remote_ffmpeg(config, target_hid, target_hostname, target_server_name, f else: rffmpeg_ffmpeg_command.append("{}".format(arg)) - log.info("Running command on host {} '{}'".format(target_server_name, target_hostname)) + log.info("Running command on host '{}' ({})".format(target_hostname, target_servername)) log.debug( "Remote command: {}".format( " ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command) @@ -463,12 +463,12 @@ def run_ffmpeg(config, ffmpeg_args): log.info("Starting rffmpeg as {} with args: {}".format(cmd_name, " ".join(ffmpeg_args))) - target_hid, target_server_name, target_hostname = get_target_host(config) + target_hid, target_hostname, target_servername = get_target_host(config) if not target_hostname or target_hostname == "localhost": ret = run_local_ffmpeg(config, ffmpeg_args) else: - ret = run_remote_ffmpeg(config, target_hid, target_hostname, target_server_name, ffmpeg_args) + ret = run_remote_ffmpeg(config, target_hid, target_hostname, target_servername, ffmpeg_args) cleanup() if ret.returncode == 0: @@ -489,7 +489,24 @@ def run_control(config): """ rffmpeg CLI interface """ - pass + # List all DB migrations here + did_alter_0001AddServername = False + # Check conditions for migrations + with dbconn(config) as cur: + # Migration for new servername (PR #36) + if cur.execute("SELECT COUNT(*) AS CNTREC FROM pragma_table_info('hosts') WHERE name='servername'").fetchone()[0] == 0: + cur.execute( + "ALTER TABLE hosts ADD servername TEXT NOT NULL DEFAULT 'invalid'" + ) + did_alter_0001AddServername = True + # Migration for new servername (PR #36) + if did_alter_0001AddServername: + with dbconn(config) as cur: + for host in cur.execute("SELECT * FROM hosts").fetchall(): + if host[3] == 'invalid': + cur.execute( + "UPDATE hosts SET servername = ? WHERE hostname = ?", (host[1], host[1]) + ) @click.command(name="init", short_help="Initialize the system.") @click.option( @@ -538,7 +555,7 @@ def run_control(config): try: with dbconn(config) as cur: cur.execute( - "CREATE TABLE hosts (id INTEGER PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, server_name TEXT NOT NULL)" + "CREATE TABLE hosts (id INTEGER PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)" ) cur.execute( "CREATE TABLE processes (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)" @@ -592,7 +609,7 @@ def run_control(config): } for host in hosts: - hid, hostname, weight, server_name = host + hid, hostname, weight, servername = host # Get the latest state with dbconn(config) as cur: @@ -608,22 +625,22 @@ def run_control(config): # Create the mappings entry host_mappings[hid] = { "hostname": hostname, + "servername": servername, "weight": weight, - "server_name": server_name, "current_state": current_state, "commands": [proc for proc in processes if proc[1] == hid], } hostname_length = 9 - server_name_length = 11 + servername_length = 11 hid_length = 3 weight_length = 7 state_length = 6 for hid, host in host_mappings.items(): if len(host["hostname"]) + 1 > hostname_length: hostname_length = len(host["hostname"]) + 1 - if len(host["server_name"]) + 1 > server_name_length: - server_name_length = len(host["server_name"]) + 1 + if len(host["servername"]) + 1 > servername_length: + servername_length = len(host["servername"]) + 1 if len(str(hid)) + 1 > hid_length: hid_length = len(str(hid)) + 1 if len(str(host["weight"])) + 1 > weight_length: @@ -633,13 +650,13 @@ def run_control(config): output = list() output.append( - "{bold}{hostname: <{hostname_length}} {server_name: <{server_name_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}{end_bold}".format( + "{bold}{hostname: <{hostname_length}} {servername: <{servername_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}{end_bold}".format( bold="\033[1m", end_bold="\033[0m", hostname="Hostname", hostname_length=hostname_length, - server_name="Server name", - server_name_length=server_name_length, + servername="Servername", + servername_length=servername_length, hid="ID", hid_length=hid_length, weight="Weight", @@ -660,11 +677,11 @@ def run_control(config): host_entry = list() host_entry.append( - "{hostname: <{hostname_length}} {server_name: <{server_name_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}".format( + "{hostname: <{hostname_length}} {servername: <{servername_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}".format( hostname=host["hostname"], hostname_length=hostname_length, - server_name=host["server_name"], - server_name_length=server_name_length, + servername=host["servername"], + servername_length=servername_length, hid=hid, hid_length=hid_length, weight=host["weight"], @@ -679,11 +696,11 @@ def run_control(config): if idx == 0: continue host_entry.append( - "{hostname: <{hostname_length}} {server_name: <{server_name_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}".format( + "{hostname: <{hostname_length}} {servername: <{servername_length}} {hid: <{hid_length}} {weight: <{weight_length}} {state: <{state_length}} {commands}".format( hostname="", hostname_length=hostname_length, - server_name="", - server_name_length=server_name_length, + servername="", + servername_length=servername_length, hid="", hid_length=hid_length, weight="", @@ -714,7 +731,7 @@ def run_control(config): "--name", "name", required=False, - default="manual", + default=None, help="The name of the host.", ) @click.argument("host") @@ -722,12 +739,12 @@ def run_control(config): """ Add a new host with IP or hostname HOST to the database. """ - click.echo("Adding new {} host '{}'".format(name, host)) - if name == "manual": + if name is None: name = host + click.echo("Adding new host '{}' ({})".format(host, name)) with dbconn(config) as cur: cur.execute( - "INSERT INTO hosts (hostname, weight, server_name) VALUES (?, ?, ?)", (host, weight, name) + "INSERT INTO hosts (hostname, weight, servername) VALUES (?, ?, ?)", (hostname, weight, name) ) rffmpeg_click.add_command(rffmpeg_click_add) @@ -736,13 +753,13 @@ def run_control(config): @click.argument("host") def rffmpeg_click_remove(host): """ - Remove a host with internal ID or IP or hostname or server_name HOST from the database. + Remove a host with internal ID or IP or hostname or servername HOST from the database. """ try: host = int(host) field = "id" except ValueError: - field = "server_name" + field = "servername" fieldAlt = "hostname" with dbconn(config) as cur: @@ -761,8 +778,8 @@ def run_control(config): else: click.echo("Removing {} hosts:".format(len(entry))) for host in entry: - hid, hostname, weight, server_name = host - click.echo("\tID: {}\tHostname: {}\tServer name: {}".format(hid, hostname, server_name)) + hid, hostname, weight, servername = host + click.echo("\tID: {}\tHostname: {}\tServername: {}".format(hid, hostname, servername)) cur.execute("DELETE FROM hosts WHERE id = ?", (hid,)) rffmpeg_click.add_command(rffmpeg_click_remove) From a3d5e655909b168da07d727dd71113889cf9c944 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Thu, 17 Nov 2022 10:12:34 -0500 Subject: [PATCH 055/115] Ensure cleanup on SIGKILL --- rffmpeg | 1 + 1 file changed, 1 insertion(+) diff --git a/rffmpeg b/rffmpeg index 6123f22..52e5c45 100755 --- a/rffmpeg +++ b/rffmpeg @@ -442,6 +442,7 @@ def run_ffmpeg(config, ffmpeg_args): signal.signal(signal.SIGTERM, cleanup) signal.signal(signal.SIGINT, cleanup) signal.signal(signal.SIGQUIT, cleanup) + signal.signal(signal.SIGKILL, cleanup) signal.signal(signal.SIGHUP, cleanup) if config["logdebug"] is True: From d2d136539f3da86636a07f2bc8fa98b03c1cfd94 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Thu, 17 Nov 2022 10:14:03 -0500 Subject: [PATCH 056/115] Fix bug in insert variable name --- rffmpeg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index 52e5c45..b00099b 100755 --- a/rffmpeg +++ b/rffmpeg @@ -745,7 +745,7 @@ def run_control(config): click.echo("Adding new host '{}' ({})".format(host, name)) with dbconn(config) as cur: cur.execute( - "INSERT INTO hosts (hostname, weight, servername) VALUES (?, ?, ?)", (hostname, weight, name) + "INSERT INTO hosts (hostname, weight, servername) VALUES (?, ?, ?)", (host, weight, name) ) rffmpeg_click.add_command(rffmpeg_click_add) From 6ea8b238160b0005cdef8b29d71d79ccdc8ae4ed Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Thu, 17 Nov 2022 10:23:57 -0500 Subject: [PATCH 057/115] Reference difference of name from host --- rffmpeg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index b00099b..3c08572 100755 --- a/rffmpeg +++ b/rffmpeg @@ -733,7 +733,7 @@ def run_control(config): "name", required=False, default=None, - help="The name of the host.", + help="The name of the host (if different from the HOST).", ) @click.argument("host") def rffmpeg_click_add(weight, name, host): From 6a6c9c4f4a0e2c1c19360ec3f0e0c9e69586d2e6 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Thu, 17 Nov 2022 10:48:32 -0500 Subject: [PATCH 058/115] Add check for database existing before migrations Fixes #38 --- rffmpeg | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rffmpeg b/rffmpeg index 3c08572..56f88ee 100755 --- a/rffmpeg +++ b/rffmpeg @@ -490,6 +490,9 @@ def run_control(config): """ rffmpeg CLI interface """ + if not Path(config["state_dir"]).is_dir() or not Path(config["db_path"]).is_file(): + return + # List all DB migrations here did_alter_0001AddServername = False # Check conditions for migrations From e6990076d9b7c2422b8d5ac9c4cdf16bc73fa3f3 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Thu, 17 Nov 2022 11:10:40 -0500 Subject: [PATCH 059/115] Revert "Ensure cleanup on SIGKILL" This reverts commit a3d5e655909b168da07d727dd71113889cf9c944. Doesn't work as per ttps://stackoverflow.com/questions/64282634 --- rffmpeg | 1 - 1 file changed, 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index 56f88ee..1bf3c8a 100755 --- a/rffmpeg +++ b/rffmpeg @@ -442,7 +442,6 @@ def run_ffmpeg(config, ffmpeg_args): signal.signal(signal.SIGTERM, cleanup) signal.signal(signal.SIGINT, cleanup) signal.signal(signal.SIGQUIT, cleanup) - signal.signal(signal.SIGKILL, cleanup) signal.signal(signal.SIGHUP, cleanup) if config["logdebug"] is True: From ca12fdae019e4a3f39959820c4ba1ac42c6f3303 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Thu, 17 Nov 2022 12:00:25 -0500 Subject: [PATCH 060/115] Ensure target_servername is always set Addresses #39 --- rffmpeg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rffmpeg b/rffmpeg index 1bf3c8a..e71c752 100755 --- a/rffmpeg +++ b/rffmpeg @@ -268,6 +268,7 @@ def get_target_host(config): lowest_count = 9999 target_hid = None target_hostname = None + target_servername = None # For each host in the mapping, let's determine if it is suitable for hid, host in host_mappings.items(): log.debug("Trying host ID {} '{}'".format(hid, host["hostname"])) @@ -326,6 +327,7 @@ def get_target_host(config): lowest_count = weighted_proc_count target_hid = hid target_hostname = host["hostname"] + target_servername = host["servername"] log.debug("Selecting host as current lowest proc count (raw count: {}, weighted count: {})".format(raw_proc_count, weighted_proc_count)) log.debug("Found optimal host ID {} '{}' ({})".format(target_hid, target_hostname, target_servername)) From 35743e10ebffe5e2788514ea6132a597eb6afdad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Thu, 17 Nov 2022 18:52:12 +0100 Subject: [PATCH 061/115] Status command fix When there are processes running on localhost (fallback), can't use status command because it expects servername field which is missing for locahost (fallback) --- rffmpeg | 1 + 1 file changed, 1 insertion(+) diff --git a/rffmpeg b/rffmpeg index e71c752..e06f6be 100755 --- a/rffmpeg +++ b/rffmpeg @@ -608,6 +608,7 @@ def run_control(config): if len(fallback_processes) > 0: host_mappings[0] = { "hostname": "localhost (fallback)", + "servername": "localhost (fallback)", "weight": 0, "current_state": "fallback", "commands": fallback_processes, From 61dfaf4b01041b03d2238bf95297f8369868951c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Sat, 3 Dec 2022 11:50:32 +0100 Subject: [PATCH 062/115] LinuxServer Docker Mod Asked integration in #26 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 25b02bd..8f45d1b 100644 --- a/README.md +++ b/README.md @@ -127,7 +127,7 @@ This depends on what "layer" you're asking at. * Media Servers: Jellyfin is officially supported; Emby seems to work fine, with caveats (see [Issue #10](https://github.com/joshuaboniface/rffmpeg/issues/10)); no others have been tested to my knowledge. * Operating Systems (source): Debian and its derivatives (Ubuntu, Linux Mint, etc.) should all work perfectly; other Linux operating systems should work fine too as the principles are the same; MacOS should work since it has an SSH client built in; Windows will not work as `rffmpeg` depends on some POSIX assumptions internally. * Operating Systems (target): Any Linux system which [`jellyfin-ffmpeg`](https://github.com/jellyfin/jellyfin-ffmpeg) supports, which is currently just Debian and Ubuntu; Windows *might* work if you can get an SSH server running on it (see [Issue #17](https://github.com/joshuaboniface/rffmpeg/issues/17)). -* Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg-server) and [Jellyfin intro skipper](https://github.com/aleksasiriski/jellyfin-intro-skipper-rffmpeg-server) by [@aleksasiriski](https://github.com/aleksasiriski). +* Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg-server) and [Jellyfin intro skipper](https://github.com/aleksasiriski/jellyfin-intro-skipper-rffmpeg-server) by [@aleksasiriski](https://github.com/aleksasiriski). In addition to these special docker images you can use linuxserver's image with [this mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg). * Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by [@BasixKOR](https://github.com/BasixKOR) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg-node) by [@aleksasiriski](https://github.com/aleksasiriski). * Cloud: [HCloud Rffmpeg](https://github.com/aleksasiriski/hcloud-rffmpeg) script made to read rffmpeg database and spin up more transcode nodes in Hetzner Cloud by [@aleksasiriski](https://github.com/aleksasiriski). From d3a2a4df41e3fa159319b70c825611e22c552ae6 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Thu, 15 Dec 2022 21:44:46 -0500 Subject: [PATCH 063/115] Remove extra global ref --- rffmpeg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index e06f6be..60d769b 100755 --- a/rffmpeg +++ b/rffmpeg @@ -166,7 +166,7 @@ def cleanup(signum="", frame=""): """ Clean up this processes stored transient data. """ - global config, p + global config with dbconn(config) as cur: cur.execute("DELETE FROM states WHERE process_id = ?", (config["current_pid"],)) From c57facbdc27e7a096dcd056a95d3d0c692543090 Mon Sep 17 00:00:00 2001 From: Pablo Yaniz Date: Thu, 22 Dec 2022 20:34:50 -0600 Subject: [PATCH 064/115] added optional date to log file --- rffmpeg | 5 +++++ rffmpeg.yml.sample | 5 ++++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index 60d769b..ec7dc9e 100755 --- a/rffmpeg +++ b/rffmpeg @@ -108,6 +108,11 @@ def load_config(): # Parse the keys from the logging group config["log_to_file"] = config_logging.get("log_to_file", True) config["logfile"] = config_logging.get("logfile", "/var/log/jellyfin/rffmpeg.log") + config["logfiledated"] = config_logging.get("logfiledated", False) + if config["logfiledated"] is True: + config["logfile"] = config_logging.get("logfile", "/var/log/jellyfin/"+ (datetime.today()).strftime('%Y%m%d') +"_rffmpeg.log") + else: + config["logfile"] = config_logging.get("logfile", "/var/log/jellyfin/rffmpeg.log") config["logdebug"] = config_logging.get("debug", False) # Parse the keys from the state group diff --git a/rffmpeg.yml.sample b/rffmpeg.yml.sample index d5b4602..21aaf8b 100644 --- a/rffmpeg.yml.sample +++ b/rffmpeg.yml.sample @@ -15,7 +15,10 @@ rffmpeg: # Log messages to this file. # Ensure the user running rffmpeg can write to this directory. #logfile: "/var/log/jellyfin/rffmpeg.log" - + + # You can add the date to the logfile if you prefer to have one file per day. + #logfiledated: false + # Show debugging messages #debug: false From 9b2ab0c59a64e53fa8f88e5c9d21418922eebe2d Mon Sep 17 00:00:00 2001 From: Pablo Yaniz Date: Thu, 22 Dec 2022 21:05:54 -0600 Subject: [PATCH 065/115] added dated log functionality --- rffmpeg | 1 + 1 file changed, 1 insertion(+) diff --git a/rffmpeg b/rffmpeg index ec7dc9e..bf1f2b5 100755 --- a/rffmpeg +++ b/rffmpeg @@ -35,6 +35,7 @@ from re import search from sqlite3 import connect as sqlite_connect from subprocess import run, PIPE from time import sleep +from datetime import datetime # Set up the logger From 45f43353f5db7dc72b2785f0ffe2445b888f4b4e Mon Sep 17 00:00:00 2001 From: Pablo Yaniz Date: Fri, 23 Dec 2022 11:33:28 -0600 Subject: [PATCH 066/115] Reviewed functionality and implemented quality of code changes --- rffmpeg | 9 ++++----- rffmpeg.yml.sample | 9 +++++++-- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/rffmpeg b/rffmpeg index bf1f2b5..5010282 100755 --- a/rffmpeg +++ b/rffmpeg @@ -109,11 +109,10 @@ def load_config(): # Parse the keys from the logging group config["log_to_file"] = config_logging.get("log_to_file", True) config["logfile"] = config_logging.get("logfile", "/var/log/jellyfin/rffmpeg.log") - config["logfiledated"] = config_logging.get("logfiledated", False) - if config["logfiledated"] is True: - config["logfile"] = config_logging.get("logfile", "/var/log/jellyfin/"+ (datetime.today()).strftime('%Y%m%d') +"_rffmpeg.log") - else: - config["logfile"] = config_logging.get("logfile", "/var/log/jellyfin/rffmpeg.log") + config["datedlogfiles"] = config_logging.get("datedlogfiles", False) + if config["datedlogfiles"] is True: + config["datedlogdir"] = config_logging.get("datedlogdir", "/var/log/jellyfin") + config["logfile"] = f"{config['datedlogdir']}/{datetime.today().strftime('%Y%m%d')}_rffmpeg.log" config["logdebug"] = config_logging.get("debug", False) # Parse the keys from the state group diff --git a/rffmpeg.yml.sample b/rffmpeg.yml.sample index 21aaf8b..840c048 100644 --- a/rffmpeg.yml.sample +++ b/rffmpeg.yml.sample @@ -16,9 +16,14 @@ rffmpeg: # Ensure the user running rffmpeg can write to this directory. #logfile: "/var/log/jellyfin/rffmpeg.log" - # You can add the date to the logfile if you prefer to have one file per day. - #logfiledated: false + # Use a Jellyfin-logging compatible dated log format, e.g. "20221223_rffmpeg.log" + # Supersedes the "logfile" directive above + #datedlogfiles: false + # Use this base directory for Jellyfin-logging compatible dated log files if you enable "datedlogfiles" + # Set this to your Jellyfin logging directory if it differs from the default + #datedlogdir: /var/log/jellyfin + # Show debugging messages #debug: false From c6e475671dca6c49385271e70102bc51b98e754d Mon Sep 17 00:00:00 2001 From: Pablo Yaniz Date: Fri, 23 Dec 2022 15:37:24 -0600 Subject: [PATCH 067/115] added quotes to file --- rffmpeg.yml.sample | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rffmpeg.yml.sample b/rffmpeg.yml.sample index 840c048..bccc49d 100644 --- a/rffmpeg.yml.sample +++ b/rffmpeg.yml.sample @@ -22,7 +22,7 @@ rffmpeg: # Use this base directory for Jellyfin-logging compatible dated log files if you enable "datedlogfiles" # Set this to your Jellyfin logging directory if it differs from the default - #datedlogdir: /var/log/jellyfin + #datedlogdir: "/var/log/jellyfin" # Show debugging messages #debug: false From b9930e6a455be0f33879c2906f6bd09971a20f41 Mon Sep 17 00:00:00 2001 From: Pablo Yaniz Date: Fri, 23 Dec 2022 15:42:17 -0600 Subject: [PATCH 068/115] -q --- rffmpeg | 2 +- rffmpeg.yml.sample | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/rffmpeg b/rffmpeg index 5010282..e532379 100755 --- a/rffmpeg +++ b/rffmpeg @@ -112,7 +112,7 @@ def load_config(): config["datedlogfiles"] = config_logging.get("datedlogfiles", False) if config["datedlogfiles"] is True: config["datedlogdir"] = config_logging.get("datedlogdir", "/var/log/jellyfin") - config["logfile"] = f"{config['datedlogdir']}/{datetime.today().strftime('%Y%m%d')}_rffmpeg.log" + config["logfile"] = config['datedlogdir'] + "/" + datetime.today().strftime('%Y%m%d') + "_rffmpeg.log" config["logdebug"] = config_logging.get("debug", False) # Parse the keys from the state group diff --git a/rffmpeg.yml.sample b/rffmpeg.yml.sample index bccc49d..ab1b389 100644 --- a/rffmpeg.yml.sample +++ b/rffmpeg.yml.sample @@ -22,7 +22,7 @@ rffmpeg: # Use this base directory for Jellyfin-logging compatible dated log files if you enable "datedlogfiles" # Set this to your Jellyfin logging directory if it differs from the default - #datedlogdir: "/var/log/jellyfin" + #datedlogdir: "/var/log/jellyfin/" # Show debugging messages #debug: false From 55d72e73c0a0fccdfa60a515ccf3b90e443fd522 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Sat, 31 Dec 2022 01:35:52 -0500 Subject: [PATCH 069/115] Clarify why sync and transcodes need a real fs --- SETUP.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/SETUP.md b/SETUP.md index ca510f5..b5aefbe 100644 --- a/SETUP.md +++ b/SETUP.md @@ -106,9 +106,8 @@ This guide is provided as a basic starting point - there are myriad possible com * Always export the `${jellyfin_data_path}` in full. Advanced users might be able to export the required subdirectories individually, but I find this to be not worth the hassle. * Note the security options of NFS. It will limit mounts to the IP addresses specified. If your home network is secure, you can use the entire network, e.g. `192.168.0.0/24`, but I would recommend determining the exact IP of your transcode server(s) and use them explicitly, e.g. for this example `192.168.0.101` and `192.168.0.102`. - * The `sync` option is very important here. Jellyfin (and presumably Emby) determines that the next chunk is ready by waiting on inotifies in this directory (I think). Thus, we'd want the client to always do an `fsync` call after every write or the server might miss chunks which results in poor playback performance. - * For the above reason, it's also very important that you export *from* the Jellyfin server and not from the transcode server. - * If your media is local to the Jellyfin server (and not already mountable on the transcode host via a remote filesystems like NFS, Samba, CephFS, etc.), also add an export for it as well. + * It's quite important to both set `sync` on the transcode host(s), and ensure that the `transcodes` directory is on a real filesystem on the Jellyfin system (i.e. is not a remote filesystem mount itself) which is then exported to the clients. If this is not the case, playback will be very slow to start. I believe the reason for this is that Jellyfin (and presumably Emby) listen for an `inotify` on the directory to know that the playlist is ready for consumption, but I have not confirmed this, and NFS *et al.* do not properly support this. + * If your media is local to the Jellyfin server (and not already mountable on the transcode host(s) via a remote filesystems like NFS, Samba, CephFS, etc.), also add an export for it as well. An example `/etc/exports` file would look like this: From 6f9971bdeabe914f9cce4803ac4620b4513acb20 Mon Sep 17 00:00:00 2001 From: gitdeath <59674699+gitdeath@users.noreply.github.com> Date: Sat, 31 Dec 2022 12:07:57 -0600 Subject: [PATCH 070/115] Update SETUP.md Added a bullet on `actimeo` parameter - I didn't want to edit your original content in the two previous bullets --- SETUP.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/SETUP.md b/SETUP.md index b5aefbe..635c6d9 100644 --- a/SETUP.md +++ b/SETUP.md @@ -109,6 +109,8 @@ This guide is provided as a basic starting point - there are myriad possible com * It's quite important to both set `sync` on the transcode host(s), and ensure that the `transcodes` directory is on a real filesystem on the Jellyfin system (i.e. is not a remote filesystem mount itself) which is then exported to the clients. If this is not the case, playback will be very slow to start. I believe the reason for this is that Jellyfin (and presumably Emby) listen for an `inotify` on the directory to know that the playlist is ready for consumption, but I have not confirmed this, and NFS *et al.* do not properly support this. * If your media is local to the Jellyfin server (and not already mountable on the transcode host(s) via a remote filesystems like NFS, Samba, CephFS, etc.), also add an export for it as well. + * If your `transcodes` directory is external to Jellyfin, such as a NAS, then you may experience delays of ~15-60s starting content as NFS uses a file attribute cache that in most applications greatly increases performance, however for this usecase it causes a delay in Jellyfin seeing the `.ts` files. The solution for this is to reduce the NFS cache time by adding `actimeo=1` to your mount command (or fstab), which will set the NFS file attribute cache to 1 second (reducing the NFS delay to ~1-2 seconds.) It is not recommended to use the `noac` flag, which would reduce the NFS delay to ~0, but at the cost of negatively impacting other NFS performance. To verify your mount added the `actimeo=1` parameter correcly `cat /proc/mounts`, which will show `acregmin=1,acregmax=1,acdirmin=1,acdirmax=1` as parameters for your `transcodes` mount. + An example `/etc/exports` file would look like this: ``` From 33cb41924140e4766cd7725f7a9bfdcff6248943 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Thu, 12 Jan 2023 00:27:09 +0100 Subject: [PATCH 071/115] Update links to my images --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8f45d1b..7537c84 100644 --- a/README.md +++ b/README.md @@ -127,8 +127,8 @@ This depends on what "layer" you're asking at. * Media Servers: Jellyfin is officially supported; Emby seems to work fine, with caveats (see [Issue #10](https://github.com/joshuaboniface/rffmpeg/issues/10)); no others have been tested to my knowledge. * Operating Systems (source): Debian and its derivatives (Ubuntu, Linux Mint, etc.) should all work perfectly; other Linux operating systems should work fine too as the principles are the same; MacOS should work since it has an SSH client built in; Windows will not work as `rffmpeg` depends on some POSIX assumptions internally. * Operating Systems (target): Any Linux system which [`jellyfin-ffmpeg`](https://github.com/jellyfin/jellyfin-ffmpeg) supports, which is currently just Debian and Ubuntu; Windows *might* work if you can get an SSH server running on it (see [Issue #17](https://github.com/joshuaboniface/rffmpeg/issues/17)). -* Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg-server) and [Jellyfin intro skipper](https://github.com/aleksasiriski/jellyfin-intro-skipper-rffmpeg-server) by [@aleksasiriski](https://github.com/aleksasiriski). In addition to these special docker images you can use linuxserver's image with [this mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg). -* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by [@BasixKOR](https://github.com/BasixKOR) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg-node) by [@aleksasiriski](https://github.com/aleksasiriski). +* Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg) by [@aleksasiriski](https://github.com/aleksasiriski). In addition to these special docker images you can use linuxserver's image with [this mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg). +* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by [@BasixKOR](https://github.com/BasixKOR) as well as [another](https://github.com/aleksasiriski/rffmpeg-node) by [@aleksasiriski](https://github.com/aleksasiriski). * Cloud: [HCloud Rffmpeg](https://github.com/aleksasiriski/hcloud-rffmpeg) script made to read rffmpeg database and spin up more transcode nodes in Hetzner Cloud by [@aleksasiriski](https://github.com/aleksasiriski). ### Can `rffmpeg` mangle/alter FFMPEG arguments? From 7201d3da5a57e747d82ac06771b24d282766f5af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Thu, 12 Jan 2023 22:23:39 +0100 Subject: [PATCH 072/115] Updated integrations 1) Fixed link to my `rffmpeg-worker`. 2) Moved my `rffmpeg-worker` infront of BasixKOR's `rffmpeg-docker` since it uses an old and non working `panubo/sshd` image. 3) Added example yaml files for Kubernetes. 4) Added LinuxServer mod to WoL section. --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 7537c84..4b5f5bf 100644 --- a/README.md +++ b/README.md @@ -128,8 +128,9 @@ This depends on what "layer" you're asking at. * Operating Systems (source): Debian and its derivatives (Ubuntu, Linux Mint, etc.) should all work perfectly; other Linux operating systems should work fine too as the principles are the same; MacOS should work since it has an SSH client built in; Windows will not work as `rffmpeg` depends on some POSIX assumptions internally. * Operating Systems (target): Any Linux system which [`jellyfin-ffmpeg`](https://github.com/jellyfin/jellyfin-ffmpeg) supports, which is currently just Debian and Ubuntu; Windows *might* work if you can get an SSH server running on it (see [Issue #17](https://github.com/joshuaboniface/rffmpeg/issues/17)). * Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg) by [@aleksasiriski](https://github.com/aleksasiriski). In addition to these special docker images you can use linuxserver's image with [this mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg). -* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by [@BasixKOR](https://github.com/BasixKOR) as well as [another](https://github.com/aleksasiriski/rffmpeg-node) by [@aleksasiriski](https://github.com/aleksasiriski). -* Cloud: [HCloud Rffmpeg](https://github.com/aleksasiriski/hcloud-rffmpeg) script made to read rffmpeg database and spin up more transcode nodes in Hetzner Cloud by [@aleksasiriski](https://github.com/aleksasiriski). +* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/aleksasiriski/rffmpeg-worker) has been created by [@aleksasiriski](https://github.com/aleksasiriski) as well as [another](https://github.com/BasixKOR/rffmpeg-docker) by [@BasixKOR](https://github.com/BasixKOR). +* Cloud: [HCloud Rffmpeg](https://github.com/aleksasiriski/hcloud-rffmpeg) script made to read rffmpeg database and spin up more transcode nodes in Hetzner Cloud. +* Kubernetes: A short guide and example yaml files are available [here](https://github.com/aleksasiriski/rffmpeg-worker/tree/main/Kubernetes). ### Can `rffmpeg` mangle/alter FFMPEG arguments? @@ -145,7 +146,7 @@ Thus it is imperative that you set up your entire system correctly for `rffmpeg` ### Can `rffmpeg` do Wake-On-LAN or other similar options to turn on a transcode server? -Right now, no. I've thought about implementing this more than once (most recently, in response to [Issue #21](https://github.com/joshuaboniface/rffmpeg/issues/21)) but ultimately I've never though this was worth the complexity and delays in spawning that it would add to the tool. That issue does provide one example of a workaround wrapper script that could accomplish this, but I don't see it being a part of the actual tool itself. +Right now, not officially. You can use the linuxserver.io [docker mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg). I've thought about implementing this more than once (most recently, in response to [Issue #21](https://github.com/joshuaboniface/rffmpeg/issues/21)) but ultimately I've never though this was worth the complexity and delays in spawning that it would add to the tool. That issue does provide one example of a workaround wrapper script that could accomplish this, but I don't see it being a part of the actual tool itself. ### I'm getting an error, help! From 4e21253509a6c82e3107117141732d3fafbffac5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Fri, 13 Jan 2023 18:33:12 +0100 Subject: [PATCH 073/115] Postgresql Added optional support for Postgresql. SQLite is still supported and the default. --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 7537c84..9f02858 100644 --- a/README.md +++ b/README.md @@ -18,21 +18,21 @@ ## Quick usage -1. Install the required Python 3 dependencies: `click`, `yaml` and `subprocess` (`sudo apt install python3-click python3-yaml python3-subprocess` in Debian). +1. Install the required Python 3 dependencies: `click`, `yaml` and `subprocess` (`sudo apt install python3-click python3-yaml python3-subprocess` in Debian) and optionally install `psycopg2` with `sudo apt install python3-psycopg2` for Postgresql support. -1. Create the directory `/etc/rffmpeg`. +2. Create the directory `/etc/rffmpeg`. -1. Copy the `rffmpeg.yml.sample` file to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. +3. Copy the `rffmpeg.yml.sample` file to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. -1. Install `rffmpeg` somewhere useful, for instance at `/usr/local/bin/rffmpeg`. +4. Install `rffmpeg` somewhere useful, for instance at `/usr/local/bin/rffmpeg`. -1. Create symlinks for the command names `ffmpeg` and `ffprobe` to `rffmpeg`, for example `sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffmpeg` and `sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffprobe`. +5. Create symlinks for the command names `ffmpeg` and `ffprobe` to `rffmpeg`, for example `sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffmpeg` and `sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffprobe`. -1. Initialize the database and add a target host, for example `sudo rffmpeg init && rffmpeg add myhost.domain.tld`. +6. Initialize the database and add a target host, for example `sudo rffmpeg init && rffmpeg add myhost.domain.tld`. -1. Set your media program to use `rffmpeg` via the `ffmpeg` symlink name created above, instead of any other `ffmpeg` binary. +7. Set your media program to use `rffmpeg` via the `ffmpeg` symlink name created above, instead of any other `ffmpeg` binary. -1. Profit! +8. Profit! `rffmpeg` does require a little bit more configuration to work properly however. For a comprehensive installation tutorial based on a reference setup, please see [the SETUP guide](SETUP.md). From 59456e502054c68cedf59682a8523215b0c1e9b0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Fri, 13 Jan 2023 18:34:17 +0100 Subject: [PATCH 074/115] Postgresql Added optional support for Postgresql. SQLite is still supported and the default. --- rffmpeg | 70 +++++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 53 insertions(+), 17 deletions(-) diff --git a/rffmpeg b/rffmpeg index e532379..315f1c8 100755 --- a/rffmpeg +++ b/rffmpeg @@ -42,18 +42,50 @@ from datetime import datetime log = logging.getLogger("rffmpeg") +# Use Postgresql if specified, otherwise use SQLite +DB_TYPE = "SQLITE" +POSTGRES_HOST = os.getenv("POSTGRES_HOST") +POSTGRES_DB = os.getenv("POSTGRES_DB") +POSTGRES_USER = os.getenv("POSTGRES_USER") +POSTGRES_PASS = os.getenv("POSTGRES_PASS") +if POSTGRES_HOST != None and POSTGRES_DB != None and POSTGRES_USER != None: + DB_TYPE = "POSTGRES" + from psycopg2 import connect as postgres_connect + + # Open a database connection (context manager) @contextmanager def dbconn(config): """ Open a database connection. """ - conn = sqlite_connect(config["db_path"]) - conn.execute("PRAGMA foreign_keys = 1") - cur = conn.cursor() - yield cur - conn.commit() + if DB_TYPE == "SQLITE": + if not Path(config["db_path"]).is_file(): + fail( + "Failed to find database '{}' - did you forget to run 'rffmpeg init' or add all env vars for Postgresql?".format( + config["db_path"] + ) + ) + log.debug("Using SQLite as database.") + conn = sqlite_connect(config["db_path"]) + conn.execute("PRAGMA foreign_keys = 1") + cur = conn.cursor() + yield cur + conn.commit() + elif DB_TYPE == "POSTGRES": + try: + log.debug("Using Postgresql as database. Connecting...") + conn = postgres_connect() + cur = conn.cursor() + cur.execute('SELECT version()') + db_version = cur.fetchone() + log.debug("Connected to Postgresql version {}".format(db_version)) + yield cur + conn.commit() + except (Exception, psycopg2.DatabaseError) as error: + log.error(error) conn.close() + log.debug("Database connection closed.") def fail(msg): @@ -73,11 +105,22 @@ def load_config(): default_config_file = "/etc/rffmpeg/rffmpeg.yml" config_file = os.environ.get("RFFMPEG_CONFIG", default_config_file) - with open(config_file, "r") as cfgfh: - try: - o_config = yaml.load(cfgfh, Loader=yaml.SafeLoader) - except Exception as e: - fail("Failed to parse configuration file: {}".format(e)) + if not Path(config_file).is_file(): + log.info("No config found in {}. Using default settings.".format(config_file)) + o_config = { + "rffmpeg": { + "logging": {}, + "directories": {}, + "remote": {}, + "commands": {} + } + } + else: + with open(config_file, "r") as cfgfh: + try: + o_config = yaml.load(cfgfh, Loader=yaml.SafeLoader) + except Exception as e: + fail("Failed to parse configuration file: {}".format(e)) config = dict() @@ -839,13 +882,6 @@ if __name__ == "__main__": if "rffmpeg" in cmd_name: run_control(config) - else: - if not Path(config["db_path"]).is_file(): - fail( - "Failed to find database '{}' - did you forget to run 'rffmpeg init'?".format( - config["db_path"] - ) - ) ffmpeg_args = all_args[1:] run_ffmpeg(config, ffmpeg_args) From 933cfac6759e6593d9d3c2bc9fb97b86ea2287a6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Fri, 13 Jan 2023 18:57:17 +0100 Subject: [PATCH 075/115] Fix README --- README.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 9f02858..89999ae 100644 --- a/README.md +++ b/README.md @@ -20,19 +20,19 @@ 1. Install the required Python 3 dependencies: `click`, `yaml` and `subprocess` (`sudo apt install python3-click python3-yaml python3-subprocess` in Debian) and optionally install `psycopg2` with `sudo apt install python3-psycopg2` for Postgresql support. -2. Create the directory `/etc/rffmpeg`. +1. Create the directory `/etc/rffmpeg`. -3. Copy the `rffmpeg.yml.sample` file to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. +1. Copy the `rffmpeg.yml.sample` file to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. -4. Install `rffmpeg` somewhere useful, for instance at `/usr/local/bin/rffmpeg`. +1. Install `rffmpeg` somewhere useful, for instance at `/usr/local/bin/rffmpeg`. -5. Create symlinks for the command names `ffmpeg` and `ffprobe` to `rffmpeg`, for example `sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffmpeg` and `sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffprobe`. +1. Create symlinks for the command names `ffmpeg` and `ffprobe` to `rffmpeg`, for example `sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffmpeg` and `sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffprobe`. -6. Initialize the database and add a target host, for example `sudo rffmpeg init && rffmpeg add myhost.domain.tld`. +1. Initialize the database and add a target host, for example `sudo rffmpeg init && rffmpeg add myhost.domain.tld`. -7. Set your media program to use `rffmpeg` via the `ffmpeg` symlink name created above, instead of any other `ffmpeg` binary. +1. Set your media program to use `rffmpeg` via the `ffmpeg` symlink name created above, instead of any other `ffmpeg` binary. -8. Profit! +1. Profit! `rffmpeg` does require a little bit more configuration to work properly however. For a comprehensive installation tutorial based on a reference setup, please see [the SETUP guide](SETUP.md). @@ -128,8 +128,9 @@ This depends on what "layer" you're asking at. * Operating Systems (source): Debian and its derivatives (Ubuntu, Linux Mint, etc.) should all work perfectly; other Linux operating systems should work fine too as the principles are the same; MacOS should work since it has an SSH client built in; Windows will not work as `rffmpeg` depends on some POSIX assumptions internally. * Operating Systems (target): Any Linux system which [`jellyfin-ffmpeg`](https://github.com/jellyfin/jellyfin-ffmpeg) supports, which is currently just Debian and Ubuntu; Windows *might* work if you can get an SSH server running on it (see [Issue #17](https://github.com/joshuaboniface/rffmpeg/issues/17)). * Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg) by [@aleksasiriski](https://github.com/aleksasiriski). In addition to these special docker images you can use linuxserver's image with [this mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg). -* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/BasixKOR/rffmpeg-docker) has been created by [@BasixKOR](https://github.com/BasixKOR) as well as [another](https://github.com/aleksasiriski/rffmpeg-node) by [@aleksasiriski](https://github.com/aleksasiriski). -* Cloud: [HCloud Rffmpeg](https://github.com/aleksasiriski/hcloud-rffmpeg) script made to read rffmpeg database and spin up more transcode nodes in Hetzner Cloud by [@aleksasiriski](https://github.com/aleksasiriski). +* Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/aleksasiriski/rffmpeg-worker) has been created by [@aleksasiriski](https://github.com/aleksasiriski) as well as [another](https://github.com/BasixKOR/rffmpeg-docker) by [@BasixKOR](https://github.com/BasixKOR). +* Cloud: [HCloud Rffmpeg](https://github.com/aleksasiriski/hcloud-rffmpeg) script made to read rffmpeg database and spin up more transcode nodes in Hetzner Cloud. +* Kubernetes: A short guide and example yaml files are available [here](https://github.com/aleksasiriski/rffmpeg-worker/tree/main/Kubernetes). ### Can `rffmpeg` mangle/alter FFMPEG arguments? @@ -145,7 +146,7 @@ Thus it is imperative that you set up your entire system correctly for `rffmpeg` ### Can `rffmpeg` do Wake-On-LAN or other similar options to turn on a transcode server? -Right now, no. I've thought about implementing this more than once (most recently, in response to [Issue #21](https://github.com/joshuaboniface/rffmpeg/issues/21)) but ultimately I've never though this was worth the complexity and delays in spawning that it would add to the tool. That issue does provide one example of a workaround wrapper script that could accomplish this, but I don't see it being a part of the actual tool itself. +Right now, not officially. You can use the linuxserver.io [docker mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg). I've thought about implementing this more than once (most recently, in response to [Issue #21](https://github.com/joshuaboniface/rffmpeg/issues/21)) but ultimately I've never though this was worth the complexity and delays in spawning that it would add to the tool. That issue does provide one example of a workaround wrapper script that could accomplish this, but I don't see it being a part of the actual tool itself. ### I'm getting an error, help! From 37871b38d91feb550a54ef481ee5631d31552b23 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 13 Jan 2023 13:14:57 -0500 Subject: [PATCH 076/115] Adjust README to reflect #50 This PR added default top-level configuration values, so a configuration file is no longer strictly required except to override defaults. Adjust the wording of the relevant README sections to reflect this. --- README.md | 8 ++++---- SETUP.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 89999ae..81bbacd 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ 1. Create the directory `/etc/rffmpeg`. -1. Copy the `rffmpeg.yml.sample` file to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. +1. Optionally, copy the `rffmpeg.yml.sample` file to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs. 1. Install `rffmpeg` somewhere useful, for instance at `/usr/local/bin/rffmpeg`. @@ -40,11 +40,11 @@ ### The `rffmpeg` Configuration file -The `rffmpeg` configuration file located at `rffmpeg.yml.sample` is an example that shows all default options. Even though this file is effectively "empty", it *must* be present at `/etc/rffmpeg/rffmpeg.yml` or at an alternative location specified by the environment variable `RFFMPEG_CONFIG`; the latter is only useful for testing, as media programs like Jellyfin provide no way to specify this. +`rffmpeg` will look at `/etc/rffmpeg/rffmpeg.yml` (or a path specified by the `RFFMPEG_CONFIG` environment variable) for a configuration file. If it doesn't find one, defaults will be used instead. You can use this file to override many configurable default values to better fit your environment. The defaults should be sensible for anyone using [Jellyfin](https://jellyfin.org) and following the [SETUP guide](SETUP.md). -To override a default option, simply uncomment the relevant line and adjust it to suit your needs. For those using [Jellyfin](https://jellyfin.org) and following the [SETUP guide](SETUP.md), no default options will need to be changed. +The example configuration file at `rffmpeg.yml.sample` shows all available options; this file can be copied as-is to the above location and edited to suit your needs; simply uncomment any lines you want to change. Note that if you do specify a file, you *must* ensure that all top-level categories are present or it will error out. -**NOTE:** If you are running into problems with `rffmpeg`, please adjust `logging` -> `debug` to `true` to obtain more detailed logs before requesting help. +**NOTE:** If you are running into problems with `rffmpeg`, you must use the config file to adjust `logging` -> `debug` to `true` to obtain more detailed logs before requesting help. Each option has an explanatory comment above it detailing its purpose. diff --git a/SETUP.md b/SETUP.md index 635c6d9..d129a0c 100644 --- a/SETUP.md +++ b/SETUP.md @@ -77,7 +77,7 @@ This guide is provided as a basic starting point - there are myriad possible com jellyfin1 $ sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffprobe ``` -1. Create a directory for the `rffmpeg` configuration at `/etc/rffmpeg`, then copy `rffmpeg.yml.sample` to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. Generally, if you're following this guide exactly, you will not need to adjust anything in `rffmpeg.yml`. +1. Optional: Create a directory for the `rffmpeg` configuration at `/etc/rffmpeg`, then copy `rffmpeg.yml.sample` to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. Generally, if you're following this guide exactly, you will not need to install this file or adjust anything in in it. If you do require help though, I require debug logging to be enabled via the configuration file, so it's probably best to get this out of the way when installing `rffmpeg`: ``` jellyfin1 $ sudo mkdir -p /etc/rffmpeg From dd03313458d30bcdd0108fdba9c08d1666173b56 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 13 Jan 2023 13:48:11 -0500 Subject: [PATCH 077/115] Fix missing else removed in #50 --- rffmpeg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index 315f1c8..18d6f40 100755 --- a/rffmpeg +++ b/rffmpeg @@ -882,6 +882,6 @@ if __name__ == "__main__": if "rffmpeg" in cmd_name: run_control(config) - + else: ffmpeg_args = all_args[1:] run_ffmpeg(config, ffmpeg_args) From 3a6a4523613e9ed65430816e914667c6cb29891c Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 13 Jan 2023 13:57:02 -0500 Subject: [PATCH 078/115] Add state/process clear command Helps simplify things in situations where a process terminates unexpectedly and doesn't clean itself up. This can result in stale process entries being left in the database. This command provides a convenient way to clear such stuck process or state values, either for the entire rffmpeg system, or for a particular host. --- rffmpeg | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/rffmpeg b/rffmpeg index 18d6f40..987f3e1 100755 --- a/rffmpeg +++ b/rffmpeg @@ -869,6 +869,55 @@ def run_control(config): rffmpeg_click.add_command(rffmpeg_click_log) + @click.command(name="clear", short_help="Clear processes and states.") + @click.argument("host", required=False, default=None) + def rffmpeg_click_log(host): + """ + Clear all active process and states from the database, optionally limited to a host with internal ID or IP or hostname or servername HOST. + + This command is designed to assist in rare error cases whereby stuck process states are present in the database, and should be used sparingly. This will not affect running processes negatively, though rffmpeg will no longer see them as active. It is recommended to run this command only when you are sure that no processes are actually running. + """ + with dbconn(config) as cur: + if host is not None: + try: + host = int(host) + field = "id" + except ValueError: + field = "servername" + fieldAlt = "hostname" + + entry = cur.execute( + "SELECT id FROM hosts WHERE {} = ?".format(field), (host,) + ).fetchall() + if len(entry) < 1: + entry = cur.execute( + "SELECT id FROM hosts WHERE {} = ?".format(fieldAlt), (host,) + ).fetchall() + if len(entry) < 1: + fail("Host not found!") + if len(entry) > 1: + fail("Multiple hosts found, please be more specific!") + host_id = entry[0][0] + + click.echo("Clearing all active processes and states for host ID '{}'".format(host_id)) + processes = cur.execute( + "SELECT id FROM processes WHERE host_id = ?", (host_id,) + ).fetchall() + states = cur.execute( + "SELECT id FROM states WHERE host_id = ?", (host_id,) + ).fetchall() + + for process in processes: + cur.execute("DELETE FROM processes WHERE id = ?", process) + for state in states: + cur.execute("DELETE FROM states WHERE id = ?", state) + else: + click.echo("Clearing all active processes and states") + cur.execute("DELETE FROM processes") + cur.execute("DELETE FROM states") + + rffmpeg_click.add_command(rffmpeg_click_log) + return rffmpeg_click(obj={}) From 51831feae4ac29fe7039b194e45885713d0ba973 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 13 Jan 2023 13:58:58 -0500 Subject: [PATCH 079/115] Simplify wording for multiple fields --- rffmpeg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rffmpeg b/rffmpeg index 987f3e1..4f8a53c 100755 --- a/rffmpeg +++ b/rffmpeg @@ -807,7 +807,7 @@ def run_control(config): @click.argument("host") def rffmpeg_click_remove(host): """ - Remove a host with internal ID or IP or hostname or servername HOST from the database. + Remove a host with internal ID/IP/hostname/servername HOST from the database. """ try: host = int(host) @@ -873,7 +873,7 @@ def run_control(config): @click.argument("host", required=False, default=None) def rffmpeg_click_log(host): """ - Clear all active process and states from the database, optionally limited to a host with internal ID or IP or hostname or servername HOST. + Clear all active process and states from the database, optionally limited to a host with internal ID/IP/hostname/servername HOST. This command is designed to assist in rare error cases whereby stuck process states are present in the database, and should be used sparingly. This will not affect running processes negatively, though rffmpeg will no longer see them as active. It is recommended to run this command only when you are sure that no processes are actually running. """ From 23ef83b20d3f7a8677a0acc8e1bb48a80db97574 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Fri, 13 Jan 2023 20:51:20 +0100 Subject: [PATCH 080/115] Fixed Postgres SQLite uses `?` for vars in it's SQL and Postgres uses `%s`, I hope this it the only difference... Also, for some reason Postgres doesn't like fetch in single command as it errors out with: ``` 'NoneType' object has no attribute 'fetchall' ``` So I divided every execute from every fetch. --- rffmpeg | 202 ++++++++++++++++++++++++++------------------------------ 1 file changed, 94 insertions(+), 108 deletions(-) diff --git a/rffmpeg b/rffmpeg index 4f8a53c..668cf3c 100755 --- a/rffmpeg +++ b/rffmpeg @@ -44,13 +44,25 @@ log = logging.getLogger("rffmpeg") # Use Postgresql if specified, otherwise use SQLite DB_TYPE = "SQLITE" -POSTGRES_HOST = os.getenv("POSTGRES_HOST") -POSTGRES_DB = os.getenv("POSTGRES_DB") -POSTGRES_USER = os.getenv("POSTGRES_USER") -POSTGRES_PASS = os.getenv("POSTGRES_PASS") -if POSTGRES_HOST != None and POSTGRES_DB != None and POSTGRES_USER != None: +SQL_VAR_SIGN="?" +POSTGRES_DB = os.environ.get("RFFMPEG_POSTGRES_DB", "rffmpeg") +POSTGRES_USER = os.environ.get("RFFMPEG_POSTGRES_USER") +POSTGRES_PASS = os.environ.get("RFFMPEG_POSTGRES_PASS") +POSTGRES_PORT = os.environ.get("RFFMPEG_POSTGRES_PORT", "5432") +POSTGRES_HOST = os.environ.get("RFFMPEG_POSTGRES_HOST", "localhost") + +if POSTGRES_DB != None and POSTGRES_USER != None: DB_TYPE = "POSTGRES" + SQL_VAR_SIGN="%s" + POSTGRES_CREDENTIALS = { + "dbname": POSTGRES_DB, + "user": POSTGRES_USER, + "password": POSTGRES_PASS, + "port": int(POSTGRES_PORT), + "host": POSTGRES_HOST + } from psycopg2 import connect as postgres_connect + from psycopg2 import DatabaseError as postgres_error # Open a database connection (context manager) @@ -72,20 +84,26 @@ def dbconn(config): cur = conn.cursor() yield cur conn.commit() + conn.close() + log.debug("SQLite connection closed.") elif DB_TYPE == "POSTGRES": + conn = None try: log.debug("Using Postgresql as database. Connecting...") - conn = postgres_connect() + conn = postgres_connect(**POSTGRES_CREDENTIALS) cur = conn.cursor() cur.execute('SELECT version()') db_version = cur.fetchone() log.debug("Connected to Postgresql version {}".format(db_version)) yield cur conn.commit() - except (Exception, psycopg2.DatabaseError) as error: + except (Exception, postgres_error) as error: + print(error) log.error(error) - conn.close() - log.debug("Database connection closed.") + finally: + if conn is not None: + conn.close() + log.debug("Postgresql connection closed.") def fail(msg): @@ -161,11 +179,11 @@ def load_config(): # Parse the keys from the state group config["state_dir"] = config_directories.get("state", "/var/lib/rffmpeg") config["persist_dir"] = config_directories.get("persist", "/run/shm") - config["dir_owner"] = config_directories.get("owner", "jellyfin") - config["dir_group"] = config_directories.get("group", "sudo") + config["dir_owner"] = config_directories.get("owner", "root") + config["dir_group"] = config_directories.get("group", "root") # Parse the keys from the remote group - config["remote_user"] = config_remote.get("user", "jellyfin") + config["remote_user"] = config_remote.get("user", "root") config["remote_args"] = config_remote.get( "args", ["-i", "/var/lib/jellyfin/.ssh/id_rsa"] ) @@ -217,10 +235,8 @@ def cleanup(signum="", frame=""): global config with dbconn(config) as cur: - cur.execute("DELETE FROM states WHERE process_id = ?", (config["current_pid"],)) - cur.execute( - "DELETE FROM processes WHERE process_id = ?", (config["current_pid"],) - ) + cur.execute(f"DELETE FROM states WHERE process_id = {SQL_VAR_SIGN}", (config["current_pid"],)) + cur.execute(f"DELETE FROM processes WHERE process_id = {SQL_VAR_SIGN}", (config["current_pid"],)) def generate_ssh_command(config, target_hostname): @@ -282,8 +298,10 @@ def get_target_host(config): log.debug("Determining optimal target host") # Select all hosts and active processes from the database with dbconn(config) as cur: - hosts = cur.execute("SELECT * FROM hosts").fetchall() - processes = cur.execute("SELECT * FROM processes").fetchall() + cur.execute("SELECT * FROM hosts") + hosts = cur.fetchall() + cur.execute("SELECT * FROM processes") + processes = cur.fetchall() # Generate a mapping dictionary of hosts and processes host_mappings = dict() @@ -292,9 +310,8 @@ def get_target_host(config): # Get the latest state with dbconn(config) as cur: - current_state = cur.execute( - "SELECT * FROM states WHERE host_id = ? ORDER BY id DESC", (hid,) - ).fetchone() + cur.execute(f"SELECT * FROM states WHERE host_id = {SQL_VAR_SIGN} ORDER BY id DESC", (hid,)) + current_state = cur.fetchone() if not current_state: current_state = "idle" @@ -352,7 +369,7 @@ def get_target_host(config): ) with dbconn(config) as cur: cur.execute( - "INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", + f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", (hid, config["current_pid"], "bad"), ) continue @@ -415,11 +432,11 @@ def run_local_ffmpeg(config, ffmpeg_args): with dbconn(config) as cur: cur.execute( - "INSERT INTO processes (host_id, process_id, cmd) VALUES (?, ?, ?)", + f"INSERT INTO processes (host_id, process_id, cmd) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", (0, config["current_pid"], cmd_name + " " + " ".join(ffmpeg_args)), ) cur.execute( - "INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", + f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", (0, config["current_pid"], "active"), ) @@ -472,11 +489,11 @@ def run_remote_ffmpeg(config, target_hid, target_hostname, target_servername, ff with dbconn(config) as cur: cur.execute( - "INSERT INTO processes (host_id, process_id, cmd) VALUES (?, ?, ?)", + f"INSERT INTO processes (host_id, process_id, cmd) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", (target_hid, config["current_pid"], cmd_name + " " + " ".join(ffmpeg_args)), ) cur.execute( - "INSERT INTO states (host_id, process_id, state) VALUES (?, ?, ?)", + f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", (target_hid, config["current_pid"], "active"), ) @@ -547,7 +564,8 @@ def run_control(config): # Check conditions for migrations with dbconn(config) as cur: # Migration for new servername (PR #36) - if cur.execute("SELECT COUNT(*) AS CNTREC FROM pragma_table_info('hosts') WHERE name='servername'").fetchone()[0] == 0: + cur.execute("SELECT COUNT(*) AS CNTREC FROM pragma_table_info('hosts') WHERE name='servername'") + if cur.fetchone()[0] == 0: cur.execute( "ALTER TABLE hosts ADD servername TEXT NOT NULL DEFAULT 'invalid'" ) @@ -555,10 +573,11 @@ def run_control(config): # Migration for new servername (PR #36) if did_alter_0001AddServername: with dbconn(config) as cur: - for host in cur.execute("SELECT * FROM hosts").fetchall(): + cur.execute("SELECT * FROM hosts") + for host in cur.fetchall(): if host[3] == 'invalid': cur.execute( - "UPDATE hosts SET servername = ? WHERE hostname = ?", (host[1], host[1]) + f"UPDATE hosts SET servername = {SQL_VAR_SIGN} WHERE hostname = {SQL_VAR_SIGN}", (host[1], host[1]) ) @click.command(name="init", short_help="Initialize the system.") @@ -602,20 +621,37 @@ def run_control(config): ) ) - if Path(config["db_path"]).is_file(): - os.remove(config["db_path"]) - try: with dbconn(config) as cur: cur.execute( - "CREATE TABLE hosts (id INTEGER PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)" + "DROP TABLE IF EXISTS hosts" ) cur.execute( - "CREATE TABLE processes (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)" + "DROP TABLE IF EXISTS processes" ) cur.execute( - "CREATE TABLE states (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)" + "DROP TABLE IF EXISTS states" ) + if DB_TYPE == "SQLITE": + cur.execute( + "CREATE TABLE hosts (id INTEGER PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)" + ) + cur.execute( + "CREATE TABLE processes (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)" + ) + cur.execute( + "CREATE TABLE states (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)" + ) + elif DB_TYPE == "POSTGRES": + cur.execute( + "CREATE TABLE hosts (id SERIAL PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)" + ) + cur.execute( + "CREATE TABLE processes (id SERIAL PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)" + ) + cur.execute( + "CREATE TABLE states (id SERIAL PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)" + ) except Exception as e: fail("Failed to create database: {}".format(e)) @@ -625,12 +661,13 @@ def run_control(config): getgrnam(config["dir_group"]).gr_gid, ) os.chmod(config["state_dir"], 0o770) - os.chown( - config["db_path"], - getpwnam(config["dir_owner"]).pw_uid, - getgrnam(config["dir_group"]).gr_gid, - ) - os.chmod(config["db_path"], 0o660) + if DB_TYPE == "SQLITE": + os.chown( + config["db_path"], + getpwnam(config["dir_owner"]).pw_uid, + getgrnam(config["dir_group"]).gr_gid, + ) + os.chmod(config["db_path"], 0o660) rffmpeg_click.add_command(rffmpeg_click_init) @@ -640,9 +677,12 @@ def run_control(config): Show the current status of all rffmpeg target hosts and active processes. """ with dbconn(config) as cur: - hosts = cur.execute("SELECT * FROM hosts").fetchall() - processes = cur.execute("SELECT * FROM processes").fetchall() - states = cur.execute("SELECT * FROM states").fetchall() + cur.execute("SELECT * FROM hosts") + hosts = cur.fetchall() + cur.execute("SELECT * FROM processes") + processes = cur.fetchall() + cur.execute("SELECT * FROM states") + states = cur.fetchall() # Determine if there are any fallback processes running fallback_processes = list() @@ -667,9 +707,8 @@ def run_control(config): # Get the latest state with dbconn(config) as cur: - current_state = cur.execute( - "SELECT * FROM states WHERE host_id = ? ORDER BY id DESC", (hid,) - ).fetchone() + cur.execute(f"SELECT * FROM states WHERE host_id = {SQL_VAR_SIGN} ORDER BY id DESC", (hid,)) + current_state = cur.fetchone() if not current_state: current_state = "idle" @@ -797,9 +836,7 @@ def run_control(config): name = host click.echo("Adding new host '{}' ({})".format(host, name)) with dbconn(config) as cur: - cur.execute( - "INSERT INTO hosts (hostname, weight, servername) VALUES (?, ?, ?)", (host, weight, name) - ) + cur.execute(f"INSERT INTO hosts (hostname, weight, servername) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", (host, weight, name)) rffmpeg_click.add_command(rffmpeg_click_add) @@ -807,7 +844,7 @@ def run_control(config): @click.argument("host") def rffmpeg_click_remove(host): """ - Remove a host with internal ID/IP/hostname/servername HOST from the database. + Remove a host with internal ID or IP or hostname or servername HOST from the database. """ try: host = int(host) @@ -817,13 +854,11 @@ def run_control(config): fieldAlt = "hostname" with dbconn(config) as cur: - entry = cur.execute( - "SELECT * FROM hosts WHERE {} = ?".format(field), (host,) - ).fetchall() + cur.execute(f"SELECT * FROM hosts WHERE {field} = {SQL_VAR_SIGN}", (host,)) + entry = cur.fetchall() if len(entry) < 1: - entry = cur.execute( - "SELECT * FROM hosts WHERE {} = ?".format(fieldAlt), (host,) - ).fetchall() + cur.execute(f"SELECT * FROM hosts WHERE {fieldAlt} = {SQL_VAR_SIGN}", (host,)) + entry = cur.fetchall() if len(entry) < 1: fail("No hosts found to delete!") @@ -834,7 +869,7 @@ def run_control(config): for host in entry: hid, hostname, weight, servername = host click.echo("\tID: {}\tHostname: {}\tServername: {}".format(hid, hostname, servername)) - cur.execute("DELETE FROM hosts WHERE id = ?", (hid,)) + cur.execute(f"DELETE FROM hosts WHERE id = {SQL_VAR_SIGN}", (hid,)) rffmpeg_click.add_command(rffmpeg_click_remove) @@ -869,55 +904,6 @@ def run_control(config): rffmpeg_click.add_command(rffmpeg_click_log) - @click.command(name="clear", short_help="Clear processes and states.") - @click.argument("host", required=False, default=None) - def rffmpeg_click_log(host): - """ - Clear all active process and states from the database, optionally limited to a host with internal ID/IP/hostname/servername HOST. - - This command is designed to assist in rare error cases whereby stuck process states are present in the database, and should be used sparingly. This will not affect running processes negatively, though rffmpeg will no longer see them as active. It is recommended to run this command only when you are sure that no processes are actually running. - """ - with dbconn(config) as cur: - if host is not None: - try: - host = int(host) - field = "id" - except ValueError: - field = "servername" - fieldAlt = "hostname" - - entry = cur.execute( - "SELECT id FROM hosts WHERE {} = ?".format(field), (host,) - ).fetchall() - if len(entry) < 1: - entry = cur.execute( - "SELECT id FROM hosts WHERE {} = ?".format(fieldAlt), (host,) - ).fetchall() - if len(entry) < 1: - fail("Host not found!") - if len(entry) > 1: - fail("Multiple hosts found, please be more specific!") - host_id = entry[0][0] - - click.echo("Clearing all active processes and states for host ID '{}'".format(host_id)) - processes = cur.execute( - "SELECT id FROM processes WHERE host_id = ?", (host_id,) - ).fetchall() - states = cur.execute( - "SELECT id FROM states WHERE host_id = ?", (host_id,) - ).fetchall() - - for process in processes: - cur.execute("DELETE FROM processes WHERE id = ?", process) - for state in states: - cur.execute("DELETE FROM states WHERE id = ?", state) - else: - click.echo("Clearing all active processes and states") - cur.execute("DELETE FROM processes") - cur.execute("DELETE FROM states") - - rffmpeg_click.add_command(rffmpeg_click_log) - return rffmpeg_click(obj={}) @@ -931,6 +917,6 @@ if __name__ == "__main__": if "rffmpeg" in cmd_name: run_control(config) - else: + ffmpeg_args = all_args[1:] run_ffmpeg(config, ffmpeg_args) From 7963c0713d8999fe6c787beefb36604604033bf2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Fri, 13 Jan 2023 20:57:26 +0100 Subject: [PATCH 081/115] Fixed Postgres and updated to master --- rffmpeg | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 50 insertions(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index 668cf3c..4638dd3 100755 --- a/rffmpeg +++ b/rffmpeg @@ -844,7 +844,7 @@ def run_control(config): @click.argument("host") def rffmpeg_click_remove(host): """ - Remove a host with internal ID or IP or hostname or servername HOST from the database. + Remove a host with internal ID/IP/hostname/servername HOST from the database. """ try: host = int(host) @@ -904,6 +904,54 @@ def run_control(config): rffmpeg_click.add_command(rffmpeg_click_log) + @click.command(name="clear", short_help="Clear processes and states.") + @click.argument("host", required=False, default=None) + def rffmpeg_click_log(host): + """ + Clear all active process and states from the database, optionally limited to a host with internal ID/IP/hostname/servername HOST. + This command is designed to assist in rare error cases whereby stuck process states are present in the database, and should be used sparingly. This will not affect running processes negatively, though rffmpeg will no longer see them as active. It is recommended to run this command only when you are sure that no processes are actually running. + """ + with dbconn(config) as cur: + if host is not None: + try: + host = int(host) + field = "id" + except ValueError: + field = "servername" + fieldAlt = "hostname" + + entry = cur.execute( + "SELECT id FROM hosts WHERE {} = ?".format(field), (host,) + ).fetchall() + if len(entry) < 1: + entry = cur.execute( + "SELECT id FROM hosts WHERE {} = ?".format(fieldAlt), (host,) + ).fetchall() + if len(entry) < 1: + fail("Host not found!") + if len(entry) > 1: + fail("Multiple hosts found, please be more specific!") + host_id = entry[0][0] + + click.echo("Clearing all active processes and states for host ID '{}'".format(host_id)) + processes = cur.execute( + "SELECT id FROM processes WHERE host_id = ?", (host_id,) + ).fetchall() + states = cur.execute( + "SELECT id FROM states WHERE host_id = ?", (host_id,) + ).fetchall() + + for process in processes: + cur.execute("DELETE FROM processes WHERE id = ?", process) + for state in states: + cur.execute("DELETE FROM states WHERE id = ?", state) + else: + click.echo("Clearing all active processes and states") + cur.execute("DELETE FROM processes") + cur.execute("DELETE FROM states") + + rffmpeg_click.add_command(rffmpeg_click_log) + return rffmpeg_click(obj={}) @@ -918,5 +966,6 @@ if __name__ == "__main__": if "rffmpeg" in cmd_name: run_control(config) + else: ffmpeg_args = all_args[1:] run_ffmpeg(config, ffmpeg_args) From 94e4402dd2a01820de088208d95380c8f75f7111 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Fri, 13 Jan 2023 21:00:07 +0100 Subject: [PATCH 082/115] Reverted root to jellyfin user I needed root for testing --- rffmpeg | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rffmpeg b/rffmpeg index 4638dd3..80bf951 100755 --- a/rffmpeg +++ b/rffmpeg @@ -179,11 +179,11 @@ def load_config(): # Parse the keys from the state group config["state_dir"] = config_directories.get("state", "/var/lib/rffmpeg") config["persist_dir"] = config_directories.get("persist", "/run/shm") - config["dir_owner"] = config_directories.get("owner", "root") - config["dir_group"] = config_directories.get("group", "root") + config["dir_owner"] = config_directories.get("owner", "jellyfin") + config["dir_group"] = config_directories.get("group", "sudo") # Parse the keys from the remote group - config["remote_user"] = config_remote.get("user", "root") + config["remote_user"] = config_remote.get("user", "jellyfin") config["remote_args"] = config_remote.get( "args", ["-i", "/var/lib/jellyfin/.ssh/id_rsa"] ) From bfa55a9370436ed262a2364913ce10ebe8177e8a Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 13 Jan 2023 15:03:43 -0500 Subject: [PATCH 083/115] Re-add erroneously removed newline --- rffmpeg | 1 + 1 file changed, 1 insertion(+) diff --git a/rffmpeg b/rffmpeg index 80bf951..14ac503 100755 --- a/rffmpeg +++ b/rffmpeg @@ -909,6 +909,7 @@ def run_control(config): def rffmpeg_click_log(host): """ Clear all active process and states from the database, optionally limited to a host with internal ID/IP/hostname/servername HOST. + This command is designed to assist in rare error cases whereby stuck process states are present in the database, and should be used sparingly. This will not affect running processes negatively, though rffmpeg will no longer see them as active. It is recommended to run this command only when you are sure that no processes are actually running. """ with dbconn(config) as cur: From 76c18202615774022939a0c2a3d27972cb3e062b Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 13 Jan 2023 15:05:35 -0500 Subject: [PATCH 084/115] Reformat with Black --- rffmpeg | 133 ++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 86 insertions(+), 47 deletions(-) diff --git a/rffmpeg b/rffmpeg index 14ac503..0d3cfe2 100755 --- a/rffmpeg +++ b/rffmpeg @@ -44,7 +44,7 @@ log = logging.getLogger("rffmpeg") # Use Postgresql if specified, otherwise use SQLite DB_TYPE = "SQLITE" -SQL_VAR_SIGN="?" +SQL_VAR_SIGN = "?" POSTGRES_DB = os.environ.get("RFFMPEG_POSTGRES_DB", "rffmpeg") POSTGRES_USER = os.environ.get("RFFMPEG_POSTGRES_USER") POSTGRES_PASS = os.environ.get("RFFMPEG_POSTGRES_PASS") @@ -53,13 +53,13 @@ POSTGRES_HOST = os.environ.get("RFFMPEG_POSTGRES_HOST", "localhost") if POSTGRES_DB != None and POSTGRES_USER != None: DB_TYPE = "POSTGRES" - SQL_VAR_SIGN="%s" + SQL_VAR_SIGN = "%s" POSTGRES_CREDENTIALS = { "dbname": POSTGRES_DB, "user": POSTGRES_USER, "password": POSTGRES_PASS, "port": int(POSTGRES_PORT), - "host": POSTGRES_HOST + "host": POSTGRES_HOST, } from psycopg2 import connect as postgres_connect from psycopg2 import DatabaseError as postgres_error @@ -92,7 +92,7 @@ def dbconn(config): log.debug("Using Postgresql as database. Connecting...") conn = postgres_connect(**POSTGRES_CREDENTIALS) cur = conn.cursor() - cur.execute('SELECT version()') + cur.execute("SELECT version()") db_version = cur.fetchone() log.debug("Connected to Postgresql version {}".format(db_version)) yield cur @@ -126,19 +126,14 @@ def load_config(): if not Path(config_file).is_file(): log.info("No config found in {}. Using default settings.".format(config_file)) o_config = { - "rffmpeg": { - "logging": {}, - "directories": {}, - "remote": {}, - "commands": {} - } + "rffmpeg": {"logging": {}, "directories": {}, "remote": {}, "commands": {}} } else: with open(config_file, "r") as cfgfh: try: o_config = yaml.load(cfgfh, Loader=yaml.SafeLoader) except Exception as e: - fail("Failed to parse configuration file: {}".format(e)) + fail("Failed to parse configuration file: {}".format(e)) config = dict() @@ -173,7 +168,12 @@ def load_config(): config["datedlogfiles"] = config_logging.get("datedlogfiles", False) if config["datedlogfiles"] is True: config["datedlogdir"] = config_logging.get("datedlogdir", "/var/log/jellyfin") - config["logfile"] = config['datedlogdir'] + "/" + datetime.today().strftime('%Y%m%d') + "_rffmpeg.log" + config["logfile"] = ( + config["datedlogdir"] + + "/" + + datetime.today().strftime("%Y%m%d") + + "_rffmpeg.log" + ) config["logdebug"] = config_logging.get("debug", False) # Parse the keys from the state group @@ -235,8 +235,14 @@ def cleanup(signum="", frame=""): global config with dbconn(config) as cur: - cur.execute(f"DELETE FROM states WHERE process_id = {SQL_VAR_SIGN}", (config["current_pid"],)) - cur.execute(f"DELETE FROM processes WHERE process_id = {SQL_VAR_SIGN}", (config["current_pid"],)) + cur.execute( + f"DELETE FROM states WHERE process_id = {SQL_VAR_SIGN}", + (config["current_pid"],), + ) + cur.execute( + f"DELETE FROM processes WHERE process_id = {SQL_VAR_SIGN}", + (config["current_pid"],), + ) def generate_ssh_command(config, target_hostname): @@ -310,12 +316,15 @@ def get_target_host(config): # Get the latest state with dbconn(config) as cur: - cur.execute(f"SELECT * FROM states WHERE host_id = {SQL_VAR_SIGN} ORDER BY id DESC", (hid,)) + cur.execute( + f"SELECT * FROM states WHERE host_id = {SQL_VAR_SIGN} ORDER BY id DESC", + (hid,), + ) current_state = cur.fetchone() if not current_state: current_state = "idle" - marking_pid = 'N/A' + marking_pid = "N/A" else: current_state = current_state[3] marking_pid = current_state[2] @@ -339,7 +348,9 @@ def get_target_host(config): log.debug("Trying host ID {} '{}'".format(hid, host["hostname"])) # If it's marked as bad, continue if host["current_state"] == "bad": - log.debug("Host previously marked bad by PID {}".format(host['marking_pid'])) + log.debug( + "Host previously marked bad by PID {}".format(host["marking_pid"]) + ) continue # Try to connect to the host and run a very quick command to determine if it is workable @@ -361,12 +372,8 @@ def get_target_host(config): " ".join(test_ssh_command + test_ffmpeg_command) ) ) - log.debug( - "SSH test command stdout: {}".format(ret.stdout) - ) - log.debug( - "SSH test command stderr: {}".format(ret.stderr) - ) + log.debug("SSH test command stdout: {}".format(ret.stdout)) + log.debug("SSH test command stderr: {}".format(ret.stderr)) with dbconn(config) as cur: cur.execute( f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", @@ -393,9 +400,17 @@ def get_target_host(config): target_hid = hid target_hostname = host["hostname"] target_servername = host["servername"] - log.debug("Selecting host as current lowest proc count (raw count: {}, weighted count: {})".format(raw_proc_count, weighted_proc_count)) + log.debug( + "Selecting host as current lowest proc count (raw count: {}, weighted count: {})".format( + raw_proc_count, weighted_proc_count + ) + ) - log.debug("Found optimal host ID {} '{}' ({})".format(target_hid, target_hostname, target_servername)) + log.debug( + "Found optimal host ID {} '{}' ({})".format( + target_hid, target_hostname, target_servername + ) + ) return target_hid, target_hostname, target_servername @@ -443,7 +458,9 @@ def run_local_ffmpeg(config, ffmpeg_args): return run_command(rffmpeg_ffmpeg_command, stdin, stdout, stderr) -def run_remote_ffmpeg(config, target_hid, target_hostname, target_servername, ffmpeg_args): +def run_remote_ffmpeg( + config, target_hid, target_hostname, target_servername, ffmpeg_args +): """ Run ffmpeg against the remote target_hostname. """ @@ -480,7 +497,9 @@ def run_remote_ffmpeg(config, target_hid, target_hostname, target_servername, ff else: rffmpeg_ffmpeg_command.append("{}".format(arg)) - log.info("Running command on host '{}' ({})".format(target_hostname, target_servername)) + log.info( + "Running command on host '{}' ({})".format(target_hostname, target_servername) + ) log.debug( "Remote command: {}".format( " ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command) @@ -528,14 +547,18 @@ def run_ffmpeg(config, ffmpeg_args): format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s", ) - log.info("Starting rffmpeg as {} with args: {}".format(cmd_name, " ".join(ffmpeg_args))) + log.info( + "Starting rffmpeg as {} with args: {}".format(cmd_name, " ".join(ffmpeg_args)) + ) target_hid, target_hostname, target_servername = get_target_host(config) if not target_hostname or target_hostname == "localhost": ret = run_local_ffmpeg(config, ffmpeg_args) else: - ret = run_remote_ffmpeg(config, target_hid, target_hostname, target_servername, ffmpeg_args) + ret = run_remote_ffmpeg( + config, target_hid, target_hostname, target_servername, ffmpeg_args + ) cleanup() if ret.returncode == 0: @@ -556,7 +579,10 @@ def run_control(config): """ rffmpeg CLI interface """ - if not Path(config["state_dir"]).is_dir() or not Path(config["db_path"]).is_file(): + if ( + not Path(config["state_dir"]).is_dir() + or not Path(config["db_path"]).is_file() + ): return # List all DB migrations here @@ -564,7 +590,9 @@ def run_control(config): # Check conditions for migrations with dbconn(config) as cur: # Migration for new servername (PR #36) - cur.execute("SELECT COUNT(*) AS CNTREC FROM pragma_table_info('hosts') WHERE name='servername'") + cur.execute( + "SELECT COUNT(*) AS CNTREC FROM pragma_table_info('hosts') WHERE name='servername'" + ) if cur.fetchone()[0] == 0: cur.execute( "ALTER TABLE hosts ADD servername TEXT NOT NULL DEFAULT 'invalid'" @@ -575,9 +603,10 @@ def run_control(config): with dbconn(config) as cur: cur.execute("SELECT * FROM hosts") for host in cur.fetchall(): - if host[3] == 'invalid': + if host[3] == "invalid": cur.execute( - f"UPDATE hosts SET servername = {SQL_VAR_SIGN} WHERE hostname = {SQL_VAR_SIGN}", (host[1], host[1]) + f"UPDATE hosts SET servername = {SQL_VAR_SIGN} WHERE hostname = {SQL_VAR_SIGN}", + (host[1], host[1]), ) @click.command(name="init", short_help="Initialize the system.") @@ -623,15 +652,9 @@ def run_control(config): try: with dbconn(config) as cur: - cur.execute( - "DROP TABLE IF EXISTS hosts" - ) - cur.execute( - "DROP TABLE IF EXISTS processes" - ) - cur.execute( - "DROP TABLE IF EXISTS states" - ) + cur.execute("DROP TABLE IF EXISTS hosts") + cur.execute("DROP TABLE IF EXISTS processes") + cur.execute("DROP TABLE IF EXISTS states") if DB_TYPE == "SQLITE": cur.execute( "CREATE TABLE hosts (id INTEGER PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)" @@ -707,7 +730,10 @@ def run_control(config): # Get the latest state with dbconn(config) as cur: - cur.execute(f"SELECT * FROM states WHERE host_id = {SQL_VAR_SIGN} ORDER BY id DESC", (hid,)) + cur.execute( + f"SELECT * FROM states WHERE host_id = {SQL_VAR_SIGN} ORDER BY id DESC", + (hid,), + ) current_state = cur.fetchone() if not current_state: @@ -836,7 +862,10 @@ def run_control(config): name = host click.echo("Adding new host '{}' ({})".format(host, name)) with dbconn(config) as cur: - cur.execute(f"INSERT INTO hosts (hostname, weight, servername) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", (host, weight, name)) + cur.execute( + f"INSERT INTO hosts (hostname, weight, servername) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", + (host, weight, name), + ) rffmpeg_click.add_command(rffmpeg_click_add) @@ -857,7 +886,9 @@ def run_control(config): cur.execute(f"SELECT * FROM hosts WHERE {field} = {SQL_VAR_SIGN}", (host,)) entry = cur.fetchall() if len(entry) < 1: - cur.execute(f"SELECT * FROM hosts WHERE {fieldAlt} = {SQL_VAR_SIGN}", (host,)) + cur.execute( + f"SELECT * FROM hosts WHERE {fieldAlt} = {SQL_VAR_SIGN}", (host,) + ) entry = cur.fetchall() if len(entry) < 1: fail("No hosts found to delete!") @@ -868,7 +899,11 @@ def run_control(config): click.echo("Removing {} hosts:".format(len(entry))) for host in entry: hid, hostname, weight, servername = host - click.echo("\tID: {}\tHostname: {}\tServername: {}".format(hid, hostname, servername)) + click.echo( + "\tID: {}\tHostname: {}\tServername: {}".format( + hid, hostname, servername + ) + ) cur.execute(f"DELETE FROM hosts WHERE id = {SQL_VAR_SIGN}", (hid,)) rffmpeg_click.add_command(rffmpeg_click_remove) @@ -934,7 +969,11 @@ def run_control(config): fail("Multiple hosts found, please be more specific!") host_id = entry[0][0] - click.echo("Clearing all active processes and states for host ID '{}'".format(host_id)) + click.echo( + "Clearing all active processes and states for host ID '{}'".format( + host_id + ) + ) processes = cur.execute( "SELECT id FROM processes WHERE host_id = ?", (host_id,) ).fetchall() From d4edaf6ab646c1da84e9851507c2e06b40c7b577 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Fri, 13 Jan 2023 21:15:22 +0100 Subject: [PATCH 085/115] Removed duplicate code Switch to var instead of IF ELSE for creating tables. --- rffmpeg | 31 +++++++++++-------------------- 1 file changed, 11 insertions(+), 20 deletions(-) diff --git a/rffmpeg b/rffmpeg index 0d3cfe2..c39b4b9 100755 --- a/rffmpeg +++ b/rffmpeg @@ -45,6 +45,7 @@ log = logging.getLogger("rffmpeg") # Use Postgresql if specified, otherwise use SQLite DB_TYPE = "SQLITE" SQL_VAR_SIGN = "?" +SQL_PRIMARY_KEY="INTEGER" POSTGRES_DB = os.environ.get("RFFMPEG_POSTGRES_DB", "rffmpeg") POSTGRES_USER = os.environ.get("RFFMPEG_POSTGRES_USER") POSTGRES_PASS = os.environ.get("RFFMPEG_POSTGRES_PASS") @@ -54,6 +55,7 @@ POSTGRES_HOST = os.environ.get("RFFMPEG_POSTGRES_HOST", "localhost") if POSTGRES_DB != None and POSTGRES_USER != None: DB_TYPE = "POSTGRES" SQL_VAR_SIGN = "%s" + SQL_PRIMARY_KEY="SERIAL" POSTGRES_CREDENTIALS = { "dbname": POSTGRES_DB, "user": POSTGRES_USER, @@ -655,26 +657,15 @@ def run_control(config): cur.execute("DROP TABLE IF EXISTS hosts") cur.execute("DROP TABLE IF EXISTS processes") cur.execute("DROP TABLE IF EXISTS states") - if DB_TYPE == "SQLITE": - cur.execute( - "CREATE TABLE hosts (id INTEGER PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)" - ) - cur.execute( - "CREATE TABLE processes (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)" - ) - cur.execute( - "CREATE TABLE states (id INTEGER PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)" - ) - elif DB_TYPE == "POSTGRES": - cur.execute( - "CREATE TABLE hosts (id SERIAL PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)" - ) - cur.execute( - "CREATE TABLE processes (id SERIAL PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)" - ) - cur.execute( - "CREATE TABLE states (id SERIAL PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)" - ) + cur.execute( + f"CREATE TABLE hosts (id {SQL_PRIMARY_KEY} PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)" + ) + cur.execute( + f"CREATE TABLE processes (id {SQL_PRIMARY_KEY} PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)" + ) + cur.execute( + f"CREATE TABLE states (id {SQL_PRIMARY_KEY} PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)" + ) except Exception as e: fail("Failed to create database: {}".format(e)) From fefaf7a71d847c7b9998a36547038f3ce023e449 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Fri, 13 Jan 2023 21:42:42 +0100 Subject: [PATCH 086/115] Fixed `rffmpeg init` for SQLite Moving the check from main to `dbconn` resulted in that check being triggered when running `rffmpeg init`. --- rffmpeg | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/rffmpeg b/rffmpeg index c39b4b9..96ce66a 100755 --- a/rffmpeg +++ b/rffmpeg @@ -48,11 +48,11 @@ SQL_VAR_SIGN = "?" SQL_PRIMARY_KEY="INTEGER" POSTGRES_DB = os.environ.get("RFFMPEG_POSTGRES_DB", "rffmpeg") POSTGRES_USER = os.environ.get("RFFMPEG_POSTGRES_USER") -POSTGRES_PASS = os.environ.get("RFFMPEG_POSTGRES_PASS") +POSTGRES_PASS = os.environ.get("RFFMPEG_POSTGRES_PASS", "") POSTGRES_PORT = os.environ.get("RFFMPEG_POSTGRES_PORT", "5432") POSTGRES_HOST = os.environ.get("RFFMPEG_POSTGRES_HOST", "localhost") -if POSTGRES_DB != None and POSTGRES_USER != None: +if POSTGRES_USER != None: DB_TYPE = "POSTGRES" SQL_VAR_SIGN = "%s" SQL_PRIMARY_KEY="SERIAL" @@ -69,12 +69,12 @@ if POSTGRES_DB != None and POSTGRES_USER != None: # Open a database connection (context manager) @contextmanager -def dbconn(config): +def dbconn(config, init = False): """ Open a database connection. """ if DB_TYPE == "SQLITE": - if not Path(config["db_path"]).is_file(): + if not init and not Path(config["db_path"]).is_file(): fail( "Failed to find database '{}' - did you forget to run 'rffmpeg init' or add all env vars for Postgresql?".format( config["db_path"] @@ -653,7 +653,7 @@ def run_control(config): ) try: - with dbconn(config) as cur: + with dbconn(config, True) as cur: cur.execute("DROP TABLE IF EXISTS hosts") cur.execute("DROP TABLE IF EXISTS processes") cur.execute("DROP TABLE IF EXISTS states") From cd1a2a7c5fabc2123a8821409dff5a8a70d07d25 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Sat, 14 Jan 2023 00:16:42 +0100 Subject: [PATCH 087/115] Fixed the new `clear` command and switched to f Switched almost every `.format()` with new `f""` way of inserting vars inside strings since it's much easier to read the code. Three instances of `.format()` were left because it made sense to use it there over `f""`. --- rffmpeg | 148 ++++++++++++++++++-------------------------------------- 1 file changed, 47 insertions(+), 101 deletions(-) diff --git a/rffmpeg b/rffmpeg index 96ce66a..1efb6c6 100755 --- a/rffmpeg +++ b/rffmpeg @@ -75,11 +75,7 @@ def dbconn(config, init = False): """ if DB_TYPE == "SQLITE": if not init and not Path(config["db_path"]).is_file(): - fail( - "Failed to find database '{}' - did you forget to run 'rffmpeg init' or add all env vars for Postgresql?".format( - config["db_path"] - ) - ) + fail(f"Failed to find database '{config["db_path"]}' - did you forget to run 'rffmpeg init' or add all env vars for Postgresql?") log.debug("Using SQLite as database.") conn = sqlite_connect(config["db_path"]) conn.execute("PRAGMA foreign_keys = 1") @@ -96,7 +92,7 @@ def dbconn(config, init = False): cur = conn.cursor() cur.execute("SELECT version()") db_version = cur.fetchone() - log.debug("Connected to Postgresql version {}".format(db_version)) + log.debug(f"Connected to Postgresql version {db_version}") yield cur conn.commit() except (Exception, postgres_error) as error: @@ -126,7 +122,7 @@ def load_config(): config_file = os.environ.get("RFFMPEG_CONFIG", default_config_file) if not Path(config_file).is_file(): - log.info("No config found in {}. Using default settings.".format(config_file)) + log.info(f"No config found in {config_file}. Using default settings.") o_config = { "rffmpeg": {"logging": {}, "directories": {}, "remote": {}, "commands": {}} } @@ -135,7 +131,7 @@ def load_config(): try: o_config = yaml.load(cfgfh, Loader=yaml.SafeLoader) except Exception as e: - fail("Failed to parse configuration file: {}".format(e)) + fail(f"Failed to parse configuration file: {e}") config = dict() @@ -268,9 +264,9 @@ def generate_ssh_command(config, target_hostname): if config["persist_time"] > 0: ssh_command.extend(["-o", "ControlMaster=auto"]) ssh_command.extend( - ["-o", "ControlPath={}/ssh-%r@%h:%p".format(config["persist_dir"])] + ["-o", f"ControlPath={config["persist_dir"]}/ssh-%r@%h:%p"] ) - ssh_command.extend(["-o", "ControlPersist={}".format(config["persist_time"])]) + ssh_command.extend(["-o", f"ControlPersist={config["persist_time"]}"]) # Add the remote config args for arg in config["remote_args"]: @@ -278,7 +274,7 @@ def generate_ssh_command(config, target_hostname): ssh_command.append(arg) # Add user+host string - ssh_command.append("{}@{}".format(config["remote_user"], target_hostname)) + ssh_command.append(f"{config["remote_user"]}@{target_hostname}") return ssh_command @@ -347,12 +343,10 @@ def get_target_host(config): target_servername = None # For each host in the mapping, let's determine if it is suitable for hid, host in host_mappings.items(): - log.debug("Trying host ID {} '{}'".format(hid, host["hostname"])) + log.debug(f"Trying host ID {hid} '{host["hostname"]}'") # If it's marked as bad, continue if host["current_state"] == "bad": - log.debug( - "Host previously marked bad by PID {}".format(host["marking_pid"]) - ) + log.debug(f"Host previously marked bad by PID {host["marking_pid"]}") continue # Try to connect to the host and run a very quick command to determine if it is workable @@ -364,25 +358,17 @@ def get_target_host(config): ret = run_command(test_ssh_command + test_ffmpeg_command, PIPE, PIPE, PIPE) if ret.returncode != 0: # Mark the host as bad - log.warning( - "Marking host {} ({}) as bad due to retcode {}".format( - host["hostname"], host["servername"], ret.returncode - ) - ) - log.debug( - "SSH test command was: {}".format( - " ".join(test_ssh_command + test_ffmpeg_command) - ) - ) - log.debug("SSH test command stdout: {}".format(ret.stdout)) - log.debug("SSH test command stderr: {}".format(ret.stderr)) + log.warning(f"Marking host {host["hostname"]} ({host["servername"]}) as bad due to retcode {ret.returncode}") + log.debug(f"SSH test command was: {" ".join(test_ssh_command + test_ffmpeg_command)}") + log.debug(f"SSH test command stdout: {ret.stdout}") + log.debug(f"SSH test command stderr: {ret.stderr}") with dbconn(config) as cur: cur.execute( f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", (hid, config["current_pid"], "bad"), ) continue - log.debug("SSH test succeeded with retcode {}".format(ret.returncode)) + log.debug(f"SSH test succeeded with retcode {ret.returncode}") # If the host state is idle, we can use it immediately if host["current_state"] == "idle": @@ -402,17 +388,9 @@ def get_target_host(config): target_hid = hid target_hostname = host["hostname"] target_servername = host["servername"] - log.debug( - "Selecting host as current lowest proc count (raw count: {}, weighted count: {})".format( - raw_proc_count, weighted_proc_count - ) - ) + log.debug(f"Selecting host as current lowest proc count (raw count: {raw_proc_count}, weighted count: {weighted_proc_count})") - log.debug( - "Found optimal host ID {} '{}' ({})".format( - target_hid, target_hostname, target_servername - ) - ) + log.debug(f"Found optimal host ID {target_hid} '{target_hostname}' ({target_servername})") return target_hid, target_hostname, target_servername @@ -442,10 +420,10 @@ def run_local_ffmpeg(config, ffmpeg_args): # Append all the passed arguments directly for arg in ffmpeg_args: - rffmpeg_ffmpeg_command.append("{}".format(arg)) + rffmpeg_ffmpeg_command.append(f"{arg}") log.info("Running command on host 'localhost'") - log.debug("Local command: {}".format(" ".join(rffmpeg_ffmpeg_command))) + log.debug(f"Local command: {" ".join(rffmpeg_ffmpeg_command)}") with dbconn(config) as cur: cur.execute( @@ -495,18 +473,12 @@ def run_remote_ffmpeg( for arg in ffmpeg_args: # Match bad shell characters: * ' ( ) | [ ] or whitespace if search("[*'()|\[\]\s]", arg): - rffmpeg_ffmpeg_command.append('"{}"'.format(arg)) + rffmpeg_ffmpeg_command.append(f'"{arg}"') else: - rffmpeg_ffmpeg_command.append("{}".format(arg)) + rffmpeg_ffmpeg_command.append(f"{arg}") - log.info( - "Running command on host '{}' ({})".format(target_hostname, target_servername) - ) - log.debug( - "Remote command: {}".format( - " ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command) - ) - ) + log.info(f"Running command on host '{target_hostname}' ({target_servername})") + log.debug(f"Remote command: {" ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command)}") with dbconn(config) as cur: cur.execute( @@ -549,9 +521,7 @@ def run_ffmpeg(config, ffmpeg_args): format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s", ) - log.info( - "Starting rffmpeg as {} with args: {}".format(cmd_name, " ".join(ffmpeg_args)) - ) + log.info(f"Starting rffmpeg as {cmd_name} with args: {" ".join(ffmpeg_args)}") target_hid, target_hostname, target_servername = get_target_host(config) @@ -564,9 +534,9 @@ def run_ffmpeg(config, ffmpeg_args): cleanup() if ret.returncode == 0: - log.info("Finished rffmpeg with return code {}".format(ret.returncode)) + log.info(f"Finished rffmpeg with return code {ret.returncode}") else: - log.error("Finished rffmpeg with return code {}".format(ret.returncode)) + log.error(f"Finished rffmpeg with return code {ret.returncode}") exit(ret.returncode) @@ -646,28 +616,18 @@ def run_control(config): try: os.makedirs(config["state_dir"]) except OSError as e: - fail( - "Failed to create state directory '{}': {}".format( - config["state_dir"], e - ) - ) + fail(f"Failed to create state directory '{config["state_dir"]}': {e}") try: with dbconn(config, True) as cur: cur.execute("DROP TABLE IF EXISTS hosts") cur.execute("DROP TABLE IF EXISTS processes") cur.execute("DROP TABLE IF EXISTS states") - cur.execute( - f"CREATE TABLE hosts (id {SQL_PRIMARY_KEY} PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)" - ) - cur.execute( - f"CREATE TABLE processes (id {SQL_PRIMARY_KEY} PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)" - ) - cur.execute( - f"CREATE TABLE states (id {SQL_PRIMARY_KEY} PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)" - ) + cur.execute(f"CREATE TABLE hosts (id {SQL_PRIMARY_KEY} PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)") + cur.execute(f"CREATE TABLE processes (id {SQL_PRIMARY_KEY} PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)") + cur.execute(f"CREATE TABLE states (id {SQL_PRIMARY_KEY} PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)") except Exception as e: - fail("Failed to create database: {}".format(e)) + fail(f"Failed to create database: {e}") os.chown( config["state_dir"], @@ -781,9 +741,7 @@ def run_control(config): if len(host["commands"]) < 1: first_command = "N/A" else: - first_command = "PID {}: {}".format( - host["commands"][0][2], host["commands"][0][3] - ) + first_command = f"PID {host["commands"][0][2]}: {host["commands"][0][3]}" host_entry = list() host_entry.append( @@ -817,7 +775,7 @@ def run_control(config): weight_length=weight_length, state="", state_length=state_length, - commands="PID {}: {}".format(command[2], command[3]), + commands=f"PID {command[2]}: {command[3]}", ) ) @@ -851,7 +809,7 @@ def run_control(config): """ if name is None: name = host - click.echo("Adding new host '{}' ({})".format(host, name)) + click.echo(f"Adding new host '{host}' ({name})") with dbconn(config) as cur: cur.execute( f"INSERT INTO hosts (hostname, weight, servername) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", @@ -885,16 +843,12 @@ def run_control(config): fail("No hosts found to delete!") if len(entry) == 1: - click.echo("Removing {} host:".format(len(entry))) + click.echo(f"Removing {len(entry)} host:") else: - click.echo("Removing {} hosts:".format(len(entry))) + click.echo(f"Removing {len(entry)} hosts:") for host in entry: hid, hostname, weight, servername = host - click.echo( - "\tID: {}\tHostname: {}\tServername: {}".format( - hid, hostname, servername - ) - ) + click.echo(f"\tID: {hid}\tHostname: {hostname}\tServername: {servername}") cur.execute(f"DELETE FROM hosts WHERE id = {SQL_VAR_SIGN}", (hid,)) rffmpeg_click.add_command(rffmpeg_click_remove) @@ -947,35 +901,27 @@ def run_control(config): field = "servername" fieldAlt = "hostname" - entry = cur.execute( - "SELECT id FROM hosts WHERE {} = ?".format(field), (host,) - ).fetchall() + cur.execute(f"SELECT id FROM hosts WHERE {field} = {SQL_VAR_SIGN}", (host,)) + entry = cur.fetchall() if len(entry) < 1: - entry = cur.execute( - "SELECT id FROM hosts WHERE {} = ?".format(fieldAlt), (host,) - ).fetchall() + cur.execute(f"SELECT id FROM hosts WHERE {fieldAlt} = {SQL_VAR_SIGN}", (host,)) + entry = cur.fetchall() if len(entry) < 1: fail("Host not found!") if len(entry) > 1: fail("Multiple hosts found, please be more specific!") host_id = entry[0][0] - click.echo( - "Clearing all active processes and states for host ID '{}'".format( - host_id - ) - ) - processes = cur.execute( - "SELECT id FROM processes WHERE host_id = ?", (host_id,) - ).fetchall() - states = cur.execute( - "SELECT id FROM states WHERE host_id = ?", (host_id,) - ).fetchall() + click.echo(f"Clearing all active processes and states for host ID '{host_id}'") + cur.execute(f"SELECT id FROM processes WHERE host_id = {SQL_VAR_SIGN}", (host_id,)) + processes = cur.fetchall() + cur.execute(f"SELECT id FROM states WHERE host_id = {SQL_VAR_SIGN}", (host_id,)) + states = cur.fetchall() for process in processes: - cur.execute("DELETE FROM processes WHERE id = ?", process) + cur.execute(f"DELETE FROM processes WHERE id = {SQL_VAR_SIGN}", process) for state in states: - cur.execute("DELETE FROM states WHERE id = ?", state) + cur.execute(f"DELETE FROM states WHERE id = {SQL_VAR_SIGN}", state) else: click.echo("Clearing all active processes and states") cur.execute("DELETE FROM processes") From a64f17a8fcac166ecbdf619fc554ffc4eb70d9ea Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Sat, 14 Jan 2023 14:30:20 +0100 Subject: [PATCH 088/115] Fixed f"" Changed every ```config["sth"] -> config['sth']``` ```" " -> ' '``` inside f"" since it clashes with "" --- rffmpeg | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/rffmpeg b/rffmpeg index 1efb6c6..1e7c862 100755 --- a/rffmpeg +++ b/rffmpeg @@ -75,7 +75,7 @@ def dbconn(config, init = False): """ if DB_TYPE == "SQLITE": if not init and not Path(config["db_path"]).is_file(): - fail(f"Failed to find database '{config["db_path"]}' - did you forget to run 'rffmpeg init' or add all env vars for Postgresql?") + fail(f"Failed to find database '{config['db_path']}' - did you forget to run 'rffmpeg init' or add all env vars for Postgresql?") log.debug("Using SQLite as database.") conn = sqlite_connect(config["db_path"]) conn.execute("PRAGMA foreign_keys = 1") @@ -264,9 +264,9 @@ def generate_ssh_command(config, target_hostname): if config["persist_time"] > 0: ssh_command.extend(["-o", "ControlMaster=auto"]) ssh_command.extend( - ["-o", f"ControlPath={config["persist_dir"]}/ssh-%r@%h:%p"] + ["-o", f"ControlPath={config['persist_dir']}/ssh-%r@%h:%p"] ) - ssh_command.extend(["-o", f"ControlPersist={config["persist_time"]}"]) + ssh_command.extend(["-o", f"ControlPersist={config['persist_time']}"]) # Add the remote config args for arg in config["remote_args"]: @@ -274,7 +274,7 @@ def generate_ssh_command(config, target_hostname): ssh_command.append(arg) # Add user+host string - ssh_command.append(f"{config["remote_user"]}@{target_hostname}") + ssh_command.append(f"{config['remote_user']}@{target_hostname}") return ssh_command @@ -343,10 +343,10 @@ def get_target_host(config): target_servername = None # For each host in the mapping, let's determine if it is suitable for hid, host in host_mappings.items(): - log.debug(f"Trying host ID {hid} '{host["hostname"]}'") + log.debug(f"Trying host ID {hid} '{host['hostname']}'") # If it's marked as bad, continue if host["current_state"] == "bad": - log.debug(f"Host previously marked bad by PID {host["marking_pid"]}") + log.debug(f"Host previously marked bad by PID {host['marking_pid']}") continue # Try to connect to the host and run a very quick command to determine if it is workable @@ -358,8 +358,8 @@ def get_target_host(config): ret = run_command(test_ssh_command + test_ffmpeg_command, PIPE, PIPE, PIPE) if ret.returncode != 0: # Mark the host as bad - log.warning(f"Marking host {host["hostname"]} ({host["servername"]}) as bad due to retcode {ret.returncode}") - log.debug(f"SSH test command was: {" ".join(test_ssh_command + test_ffmpeg_command)}") + log.warning(f"Marking host {host['hostname']} ({host['servername']}) as bad due to retcode {ret.returncode}") + log.debug(f"SSH test command was: {' '.join(test_ssh_command + test_ffmpeg_command)}") log.debug(f"SSH test command stdout: {ret.stdout}") log.debug(f"SSH test command stderr: {ret.stderr}") with dbconn(config) as cur: @@ -423,12 +423,12 @@ def run_local_ffmpeg(config, ffmpeg_args): rffmpeg_ffmpeg_command.append(f"{arg}") log.info("Running command on host 'localhost'") - log.debug(f"Local command: {" ".join(rffmpeg_ffmpeg_command)}") + log.debug(f"Local command: {' '.join(rffmpeg_ffmpeg_command)}") with dbconn(config) as cur: cur.execute( f"INSERT INTO processes (host_id, process_id, cmd) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", - (0, config["current_pid"], cmd_name + " " + " ".join(ffmpeg_args)), + (0, config["current_pid"], cmd_name + ' ' + ' '.join(ffmpeg_args)), ) cur.execute( f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", @@ -478,12 +478,12 @@ def run_remote_ffmpeg( rffmpeg_ffmpeg_command.append(f"{arg}") log.info(f"Running command on host '{target_hostname}' ({target_servername})") - log.debug(f"Remote command: {" ".join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command)}") + log.debug(f"Remote command: {' '.join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command)}") with dbconn(config) as cur: cur.execute( f"INSERT INTO processes (host_id, process_id, cmd) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", - (target_hid, config["current_pid"], cmd_name + " " + " ".join(ffmpeg_args)), + (target_hid, config["current_pid"], cmd_name + ' ' + ' '.join(ffmpeg_args)), ) cur.execute( f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", @@ -521,7 +521,7 @@ def run_ffmpeg(config, ffmpeg_args): format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s", ) - log.info(f"Starting rffmpeg as {cmd_name} with args: {" ".join(ffmpeg_args)}") + log.info(f"Starting rffmpeg as {cmd_name} with args: {' '.join(ffmpeg_args)}") target_hid, target_hostname, target_servername = get_target_host(config) @@ -616,7 +616,7 @@ def run_control(config): try: os.makedirs(config["state_dir"]) except OSError as e: - fail(f"Failed to create state directory '{config["state_dir"]}': {e}") + fail(f"Failed to create state directory '{config['state_dir']}': {e}") try: with dbconn(config, True) as cur: @@ -741,7 +741,7 @@ def run_control(config): if len(host["commands"]) < 1: first_command = "N/A" else: - first_command = f"PID {host["commands"][0][2]}: {host["commands"][0][3]}" + first_command = f"PID {host['commands'][0][2]}: {host['commands'][0][3]}" host_entry = list() host_entry.append( From 5fe09d2800a9967477a588a0794af371655594d5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Sun, 22 Jan 2023 18:33:33 +0100 Subject: [PATCH 089/115] special_flags Added `-muxers` and `-fp_format` to special flags and option to add more in the config without overriding the default ones. #56 --- rffmpeg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index 1e7c862..db28164 100755 --- a/rffmpeg +++ b/rffmpeg @@ -210,6 +210,7 @@ def load_config(): # Set the database path config["db_path"] = config["state_dir"] + "/rffmpeg.db" + # Set a list of special flags that cause different behaviour config["special_flags"] = [ "-version", @@ -218,7 +219,9 @@ def load_config(): "-hwaccels", "-filters", "-h", - ] + "-muxers", + "-fp_format", + ] + config_commands.get("special_flags", []) # Set the current PID of this process config["current_pid"] = os.getpid() From 4586971a0f4df6bf47e90f8927ec7db65861831d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Sun, 22 Jan 2023 18:35:23 +0100 Subject: [PATCH 090/115] special_flags --- rffmpeg.yml.sample | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/rffmpeg.yml.sample b/rffmpeg.yml.sample index ab1b389..5cafdc7 100644 --- a/rffmpeg.yml.sample +++ b/rffmpeg.yml.sample @@ -76,3 +76,8 @@ rffmpeg: # Optional local fallback ffmpeg and ffprobe binary paths, if different from the above. #fallback_ffmpeg: "/usr/lib/jellyfin-ffmpeg/ffmpeg" #fallback_ffprobe: "/usr/lib/jellyfin-ffmpeg/ffprobe" + + # Optional additions to special flags that output to stdout instead of stderr. This isn't an override. + #special_flags: + # - "-muxers" + # - "-fp_format" From 8e487abbb4442b84fb2a84da57a4bd05fb4f5276 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Tue, 21 Feb 2023 18:42:30 +0100 Subject: [PATCH 091/115] adding datetime to hosts --- rffmpeg | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/rffmpeg b/rffmpeg index db28164..19741e0 100755 --- a/rffmpeg +++ b/rffmpeg @@ -46,6 +46,7 @@ log = logging.getLogger("rffmpeg") DB_TYPE = "SQLITE" SQL_VAR_SIGN = "?" SQL_PRIMARY_KEY="INTEGER" +SQL_DATE_TIME="DATETIME" POSTGRES_DB = os.environ.get("RFFMPEG_POSTGRES_DB", "rffmpeg") POSTGRES_USER = os.environ.get("RFFMPEG_POSTGRES_USER") POSTGRES_PASS = os.environ.get("RFFMPEG_POSTGRES_PASS", "") @@ -56,6 +57,7 @@ if POSTGRES_USER != None: DB_TYPE = "POSTGRES" SQL_VAR_SIGN = "%s" SQL_PRIMARY_KEY="SERIAL" + SQL_DATE_TIME="TIMESTAMP" POSTGRES_CREDENTIALS = { "dbname": POSTGRES_DB, "user": POSTGRES_USER, @@ -313,7 +315,7 @@ def get_target_host(config): # Generate a mapping dictionary of hosts and processes host_mappings = dict() for host in hosts: - hid, hostname, weight, servername = host + hid, servername, hostname, weight, created = host # Get the latest state with dbconn(config) as cur: @@ -626,7 +628,7 @@ def run_control(config): cur.execute("DROP TABLE IF EXISTS hosts") cur.execute("DROP TABLE IF EXISTS processes") cur.execute("DROP TABLE IF EXISTS states") - cur.execute(f"CREATE TABLE hosts (id {SQL_PRIMARY_KEY} PRIMARY KEY, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, servername TEXT NOT NULL)") + cur.execute(f"CREATE TABLE hosts (id {SQL_PRIMARY_KEY} PRIMARY KEY, servername TEXT NOT NULL UNIQUE, hostname TEXT NOT NULL, weight INTEGER DEFAULT 1, created {SQL_DATE_TIME} NOT NULL)") cur.execute(f"CREATE TABLE processes (id {SQL_PRIMARY_KEY} PRIMARY KEY, host_id INTEGER, process_id INTEGER, cmd TEXT)") cur.execute(f"CREATE TABLE states (id {SQL_PRIMARY_KEY} PRIMARY KEY, host_id INTEGER, process_id INTEGER, state TEXT)") except Exception as e: @@ -680,7 +682,7 @@ def run_control(config): } for host in hosts: - hid, hostname, weight, servername = host + hid, servername, hostname, weight, created = host # Get the latest state with dbconn(config) as cur: @@ -810,13 +812,14 @@ def run_control(config): """ Add a new host with IP or hostname HOST to the database. """ + created = datetime.now() if name is None: name = host click.echo(f"Adding new host '{host}' ({name})") with dbconn(config) as cur: cur.execute( - f"INSERT INTO hosts (hostname, weight, servername) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", - (host, weight, name), + f"INSERT INTO hosts (hostname, weight, servername) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", + (host, weight, name, created), ) rffmpeg_click.add_command(rffmpeg_click_add) @@ -850,7 +853,7 @@ def run_control(config): else: click.echo(f"Removing {len(entry)} hosts:") for host in entry: - hid, hostname, weight, servername = host + hid, servername, hostname, weight, created = host click.echo(f"\tID: {hid}\tHostname: {hostname}\tServername: {servername}") cur.execute(f"DELETE FROM hosts WHERE id = {SQL_VAR_SIGN}", (hid,)) From db59781fdd1868f91b35d0b8b68efc563e837037 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Tue, 21 Feb 2023 21:10:19 +0100 Subject: [PATCH 092/115] forgot add command --- rffmpeg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rffmpeg b/rffmpeg index 19741e0..a9f11a0 100755 --- a/rffmpeg +++ b/rffmpeg @@ -818,8 +818,8 @@ def run_control(config): click.echo(f"Adding new host '{host}' ({name})") with dbconn(config) as cur: cur.execute( - f"INSERT INTO hosts (hostname, weight, servername) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", - (host, weight, name, created), + f"INSERT INTO hosts (servername, hostname, weight, created) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", + (name, host, weight, created), ) rffmpeg_click.add_command(rffmpeg_click_add) From b50e87f6c7acb6df9b5b4a747e2c7b2740b5cc41 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Tue, 21 Feb 2023 21:19:02 +0100 Subject: [PATCH 093/115] more logical field access --- rffmpeg | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/rffmpeg b/rffmpeg index a9f11a0..0645f9b 100755 --- a/rffmpeg +++ b/rffmpeg @@ -329,8 +329,8 @@ def get_target_host(config): current_state = "idle" marking_pid = "N/A" else: - current_state = current_state[3] - marking_pid = current_state[2] + current_state = current_state["state"] + marking_pid = current_state["process_id"] # Create the mappings entry host_mappings[hid] = { @@ -580,10 +580,10 @@ def run_control(config): with dbconn(config) as cur: cur.execute("SELECT * FROM hosts") for host in cur.fetchall(): - if host[3] == "invalid": + if host["servername"] == "invalid": cur.execute( f"UPDATE hosts SET servername = {SQL_VAR_SIGN} WHERE hostname = {SQL_VAR_SIGN}", - (host[1], host[1]), + (host["hostname"], host["hostname"]), ) @click.command(name="init", short_help="Initialize the system.") @@ -666,7 +666,7 @@ def run_control(config): # Determine if there are any fallback processes running fallback_processes = list() for process in processes: - if process[1] == 0: + if process["host_id"] == 0: fallback_processes.append(process) # Generate a mapping dictionary of hosts and processes @@ -695,7 +695,7 @@ def run_control(config): if not current_state: current_state = "idle" else: - current_state = current_state[3] + current_state = current_state["state"] # Create the mappings entry host_mappings[hid] = { From 88b572d27c8d58f32214df07ae4a0f7251946366 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Wed, 22 Feb 2023 00:01:38 +0100 Subject: [PATCH 094/115] fixed fields --- rffmpeg | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/rffmpeg b/rffmpeg index 0645f9b..43e9af5 100755 --- a/rffmpeg +++ b/rffmpeg @@ -329,8 +329,9 @@ def get_target_host(config): current_state = "idle" marking_pid = "N/A" else: - current_state = current_state["state"] - marking_pid = current_state["process_id"] + sid, host_id, process_id, state = current_state + current_state = state + marking_pid = process_id # Create the mappings entry host_mappings[hid] = { @@ -580,10 +581,11 @@ def run_control(config): with dbconn(config) as cur: cur.execute("SELECT * FROM hosts") for host in cur.fetchall(): - if host["servername"] == "invalid": + hid, servername, hostname, weight, created = host + if servername == "invalid": cur.execute( f"UPDATE hosts SET servername = {SQL_VAR_SIGN} WHERE hostname = {SQL_VAR_SIGN}", - (host["hostname"], host["hostname"]), + (hostname, hostname), ) @click.command(name="init", short_help="Initialize the system.") @@ -666,7 +668,8 @@ def run_control(config): # Determine if there are any fallback processes running fallback_processes = list() for process in processes: - if process["host_id"] == 0: + pid, host_id, process_id, cmd = process + if host_id == 0: fallback_processes.append(process) # Generate a mapping dictionary of hosts and processes @@ -695,7 +698,8 @@ def run_control(config): if not current_state: current_state = "idle" else: - current_state = current_state["state"] + sid, host_id, process_id, state = current_state + current_state = state # Create the mappings entry host_mappings[hid] = { From d064e7fb961e78024d1c20451b8b80f89fa6f5c1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Aleksa=20Siri=C5=A1ki?= <31509435+aleksasiriski@users.noreply.github.com> Date: Wed, 8 Mar 2023 23:06:09 +0100 Subject: [PATCH 095/115] rffmpeg-go --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 81bbacd..e8a21fd 100644 --- a/README.md +++ b/README.md @@ -127,9 +127,9 @@ This depends on what "layer" you're asking at. * Media Servers: Jellyfin is officially supported; Emby seems to work fine, with caveats (see [Issue #10](https://github.com/joshuaboniface/rffmpeg/issues/10)); no others have been tested to my knowledge. * Operating Systems (source): Debian and its derivatives (Ubuntu, Linux Mint, etc.) should all work perfectly; other Linux operating systems should work fine too as the principles are the same; MacOS should work since it has an SSH client built in; Windows will not work as `rffmpeg` depends on some POSIX assumptions internally. * Operating Systems (target): Any Linux system which [`jellyfin-ffmpeg`](https://github.com/jellyfin/jellyfin-ffmpeg) supports, which is currently just Debian and Ubuntu; Windows *might* work if you can get an SSH server running on it (see [Issue #17](https://github.com/joshuaboniface/rffmpeg/issues/17)). -* Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost) as well as [another](https://github.com/aleksasiriski/jellyfin-rffmpeg) by [@aleksasiriski](https://github.com/aleksasiriski). In addition to these special docker images you can use linuxserver's image with [this mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg). +* Install Methods for Jellyfin: Native packages/installers/archives are recommended; a set of [Jellyfin Docker containers integrating `rffmpeg`](https://github.com/Shadowghost/jellyfin-rffmpeg) has been created by [@Shadowghost](https://github.com/Shadowghost). In addition to this special docker image you can use linuxserver's image with [this mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg). * Install Methods for `rffmpeg`: Direct installation is recommended; a [Docker container to act as an ffmpeg transcode target](https://github.com/aleksasiriski/rffmpeg-worker) has been created by [@aleksasiriski](https://github.com/aleksasiriski) as well as [another](https://github.com/BasixKOR/rffmpeg-docker) by [@BasixKOR](https://github.com/BasixKOR). -* Cloud: [HCloud Rffmpeg](https://github.com/aleksasiriski/hcloud-rffmpeg) script made to read rffmpeg database and spin up more transcode nodes in Hetzner Cloud. +* OUTDATED Cloud: [HCloud Rffmpeg](https://github.com/aleksasiriski/hcloud-rffmpeg) script made to read rffmpeg database and spin up more transcode nodes in Hetzner Cloud. * Kubernetes: A short guide and example yaml files are available [here](https://github.com/aleksasiriski/rffmpeg-worker/tree/main/Kubernetes). ### Can `rffmpeg` mangle/alter FFMPEG arguments? @@ -173,3 +173,7 @@ Absolutely - I'm happy to take pull requests for just about any bugfix or improv ### Can you help me set up my server? I'm always happy to help, though please ensure you try to follow the setup guide first - that's why I wrote it! Support can be found [on Matrix](https://matrix.to/#/#rffmpeg:matrix.org) or via email at `joshua@boniface.me`. Please note though that I may be unresponsive sometimes, though I will get back to you eventually I promise! Please don't open Issues here about setup problems; the Issue tracker is for bugs or feature requests instead. + +### `rffmpeg-go` - forked project + +There's also a [fork of this script written in Go](https://github.com/aleksasiriski/rffmpeg-go) with semver tags and binaries available, as well as docker images for both the [script](https://github.com/aleksasiriski/rffmpeg-go/pkgs/container/rffmpeg-go) and [Jellyfin](https://github.com/aleksasiriski/jellyfin-rffmpeg). From 196bbeee4d5459732ec69dec72e56515b8eaa766 Mon Sep 17 00:00:00 2001 From: Sim0nW0lf <31454341+Sim0nW0lf@users.noreply.github.com> Date: Thu, 6 Apr 2023 06:05:42 +0200 Subject: [PATCH 096/115] allow multiple ssh users Different users on multiple servers were not possible because you had to set a specific user. I changed that so that setting no user in rffmpeg.yml and configuring the ssh config with a user works instead. If you have configured a specific user, this won't break your setup and disables the feature. --- rffmpeg | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index 43e9af5..88070b7 100755 --- a/rffmpeg +++ b/rffmpeg @@ -183,7 +183,7 @@ def load_config(): config["dir_group"] = config_directories.get("group", "sudo") # Parse the keys from the remote group - config["remote_user"] = config_remote.get("user", "jellyfin") + config["remote_user"] = config_remote.get("user", "") config["remote_args"] = config_remote.get( "args", ["-i", "/var/lib/jellyfin/.ssh/id_rsa"] ) @@ -360,6 +360,7 @@ def get_target_host(config): log.debug("Running SSH test") test_ssh_command = generate_ssh_command(config, host["hostname"]) test_ssh_command.remove("-q") + test_ssh_command = [arg.replace('@', '', 1) if arg.startswith('@') else arg for arg in test_ssh_command] test_ffmpeg_command = [config["ffmpeg_command"], "-version"] ret = run_command(test_ssh_command + test_ffmpeg_command, PIPE, PIPE, PIPE) if ret.returncode != 0: @@ -451,6 +452,7 @@ def run_remote_ffmpeg( Run ffmpeg against the remote target_hostname. """ rffmpeg_ssh_command = generate_ssh_command(config, target_hostname) + rffmpeg_ssh_command = [arg.replace('@', '', 1) if arg.startswith('@') else arg for arg in rffmpeg_ssh_command] rffmpeg_ffmpeg_command = list() # Add any pre commands From 3a3a3ae738470eb5aa4001307769bc23198ceef2 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 22 May 2023 20:27:10 -0400 Subject: [PATCH 097/115] Add escaping of $ characters in arguments Needed if a filename contains $. Could be added to later but for now hardcoded. --- rffmpeg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rffmpeg b/rffmpeg index 43e9af5..fcbfa2f 100755 --- a/rffmpeg +++ b/rffmpeg @@ -477,6 +477,8 @@ def run_remote_ffmpeg( # Append all the passed arguments with requoting of any problematic characters for arg in ffmpeg_args: + # Escape $ characters + arg = arg.replace('$', '\\$') # Match bad shell characters: * ' ( ) | [ ] or whitespace if search("[*'()|\[\]\s]", arg): rffmpeg_ffmpeg_command.append(f'"{arg}"') From 32c2c3de0d9920b6a1ab90d12c7219c128f60edf Mon Sep 17 00:00:00 2001 From: Jendrik Weise Date: Fri, 23 Jun 2023 02:13:13 +0200 Subject: [PATCH 098/115] Add option to run command directly Usable with the "run" subcommand. Factored out now reused code. --- rffmpeg | 208 ++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 136 insertions(+), 72 deletions(-) diff --git a/rffmpeg b/rffmpeg index fcbfa2f..ee530c9 100755 --- a/rffmpeg +++ b/rffmpeg @@ -400,48 +400,104 @@ def get_target_host(config): return target_hid, target_hostname, target_servername -def run_local_ffmpeg(config, ffmpeg_args): +def run_local_command(config, command, command_args, stderr_as_stdout = False, mapped_cmd = None): """ - Run ffmpeg locally, either because "localhost" is the target host, or because no good target + Run command locally, either because "localhost" is the target host, or because no good target host was found by get_target_host(). """ - rffmpeg_ffmpeg_command = list() + rffmpeg_command = [mapped_cmd or command] # Prepare our default stdin/stdout/stderr stdin = sys.stdin stderr = sys.stderr - if "ffprobe" in cmd_name: - # If we're in ffprobe mode use that command and sys.stdout as stdout - rffmpeg_ffmpeg_command.append(config["fallback_ffprobe_command"]) - stdout = sys.stdout - else: - # Otherwise, we use stderr as stdout - rffmpeg_ffmpeg_command.append(config["fallback_ffmpeg_command"]) + if stderr_as_stdout: stdout = sys.stderr - - # Check for special flags that override the default stdout - if any(item in config["special_flags"] for item in ffmpeg_args): + else: stdout = sys.stdout # Append all the passed arguments directly - for arg in ffmpeg_args: - rffmpeg_ffmpeg_command.append(f"{arg}") + for arg in command_args: + rffmpeg_command.append(f"{arg}") log.info("Running command on host 'localhost'") - log.debug(f"Local command: {' '.join(rffmpeg_ffmpeg_command)}") + log.debug(f"Local command: {' '.join(rffmpeg_command)}") with dbconn(config) as cur: cur.execute( f"INSERT INTO processes (host_id, process_id, cmd) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", - (0, config["current_pid"], cmd_name + ' ' + ' '.join(ffmpeg_args)), + (0, config["current_pid"], command + ' ' + ' '.join(command_args)), ) cur.execute( f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", (0, config["current_pid"], "active"), ) - return run_command(rffmpeg_ffmpeg_command, stdin, stdout, stderr) + return run_command(rffmpeg_command, stdin, stdout, stderr) + + +def run_local_ffmpeg(config, ffmpeg_args): + """ + Run ffmpeg locally, either because "localhost" is the target host, or because no good target + host was found by get_target_host(). + """ + if "ffprobe" in cmd_name: + return run_local_command(config, cmd_name, ffmpeg_args, mapped_cmd=config["fallback_ffprobe_command"]) + else: + return run_local_command(config, cmd_name, ffmpeg_args, stderr_as_stdout=not any(item in config["special_flags"] for item in ffmpeg_args), mapped_cmd=config["fallback_ffmpeg_command"]) + + +def run_remote_command( + config, target_hid, target_hostname, target_servername, command, command_args, stderr_as_stdout = False, mapped_cmd = None, pre_commands = [] +): + """ + Run command against the remote target_hostname. + """ + rffmpeg_ssh_command = generate_ssh_command(config, target_hostname) + rffmpeg_command = list() + + # Add any pre commands + for cmd in pre_commands: + if cmd: + rffmpeg_command.append(cmd) + + rffmpeg_command.append(mapped_cmd or command) + + # Prepare our default stdin/stderr + stdin = sys.stdin + stderr = sys.stderr + + if stderr_as_stdout: + stdout = sys.stderr + else: + stdout = sys.stdout + + # Append all the passed arguments with requoting of any problematic characters + for arg in command_args: + # Escape $ characters + arg = arg.replace('$', '\\$') + # Match bad shell characters: * ' ( ) | [ ] or whitespace + if search("[*'()|\[\]\s]", arg): + rffmpeg_command.append(f'"{arg}"') + else: + rffmpeg_command.append(f"{arg}") + + log.info(f"Running command on host '{target_hostname}' ({target_servername})") + log.debug(f"Remote command: {' '.join(rffmpeg_ssh_command + rffmpeg_command)}") + + with dbconn(config) as cur: + cur.execute( + f"INSERT INTO processes (host_id, process_id, cmd) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", + (target_hid, config["current_pid"], command + ' ' + ' '.join(command_args)), + ) + cur.execute( + f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", + (target_hid, config["current_pid"], "active"), + ) + + return run_command( + rffmpeg_ssh_command + rffmpeg_command, stdin, stdout, stderr + ) def run_remote_ffmpeg( @@ -450,68 +506,18 @@ def run_remote_ffmpeg( """ Run ffmpeg against the remote target_hostname. """ - rffmpeg_ssh_command = generate_ssh_command(config, target_hostname) - rffmpeg_ffmpeg_command = list() - - # Add any pre commands - for cmd in config["pre_commands"]: - if cmd: - rffmpeg_ffmpeg_command.append(cmd) - - # Prepare our default stdin/stderr - stdin = sys.stdin - stderr = sys.stderr - if "ffprobe" in cmd_name: # If we're in ffprobe mode use that command and sys.stdout as stdout - rffmpeg_ffmpeg_command.append(config["ffprobe_command"]) - stdout = sys.stdout + return run_remote_command(config, target_hid, target_hostname, target_servername, cmd_name, ffmpeg_args, mapped_cmd=config["ffprobe_command"], pre_commands=config["pre_commands"]) else: # Otherwise, we use stderr as stdout - rffmpeg_ffmpeg_command.append(config["ffmpeg_command"]) - stdout = sys.stderr - - # Check for special flags that override the default stdout - if any(item in config["special_flags"] for item in ffmpeg_args): - stdout = sys.stdout - - # Append all the passed arguments with requoting of any problematic characters - for arg in ffmpeg_args: - # Escape $ characters - arg = arg.replace('$', '\\$') - # Match bad shell characters: * ' ( ) | [ ] or whitespace - if search("[*'()|\[\]\s]", arg): - rffmpeg_ffmpeg_command.append(f'"{arg}"') - else: - rffmpeg_ffmpeg_command.append(f"{arg}") - - log.info(f"Running command on host '{target_hostname}' ({target_servername})") - log.debug(f"Remote command: {' '.join(rffmpeg_ssh_command + rffmpeg_ffmpeg_command)}") - - with dbconn(config) as cur: - cur.execute( - f"INSERT INTO processes (host_id, process_id, cmd) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", - (target_hid, config["current_pid"], cmd_name + ' ' + ' '.join(ffmpeg_args)), - ) - cur.execute( - f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", - (target_hid, config["current_pid"], "active"), - ) - - return run_command( - rffmpeg_ssh_command + rffmpeg_ffmpeg_command, stdin, stdout, stderr - ) + return run_remote_command(config, target_hid, target_hostname, target_servername, cmd_name, ffmpeg_args, stderr_as_stdout=not any(item in config["special_flags"] for item in ffmpeg_args), mapped_cmd=config["ffmpeg_command"], pre_commands=config["pre_commands"]) -def run_ffmpeg(config, ffmpeg_args): +def setup_logging(config): """ - Entrypoint for an ffmpeg/ffprobe aliased process. + Set up logging. """ - signal.signal(signal.SIGTERM, cleanup) - signal.signal(signal.SIGINT, cleanup) - signal.signal(signal.SIGQUIT, cleanup) - signal.signal(signal.SIGHUP, cleanup) - if config["logdebug"] is True: logging_level = logging.DEBUG else: @@ -529,6 +535,22 @@ def run_ffmpeg(config, ffmpeg_args): format="%(asctime)s - %(name)s[%(process)s] - %(levelname)s - %(message)s", ) + +def hook_signals(): + signal.signal(signal.SIGTERM, cleanup) + signal.signal(signal.SIGINT, cleanup) + signal.signal(signal.SIGQUIT, cleanup) + signal.signal(signal.SIGHUP, cleanup) + + +def run_ffmpeg(config, ffmpeg_args): + """ + Entrypoint for an ffmpeg/ffprobe aliased process. + """ + hook_signals() + + setup_logging(config) + log.info(f"Starting rffmpeg as {cmd_name} with args: {' '.join(ffmpeg_args)}") target_hid, target_hostname, target_servername = get_target_host(config) @@ -865,6 +887,48 @@ def run_control(config): rffmpeg_click.add_command(rffmpeg_click_remove) + @click.command(name="run", short_help="Run a command.", context_settings={ + "ignore_unknown_options": True + }) + @click.option("--stderr-as-stdout", "stderr_as_stdout", is_flag=True, default=False, help="Use stderr as stdout for the command.") + @click.argument('full_command', nargs=-1, type=click.UNPROCESSED) + def rffmpeg_click_run(stderr_as_stdout, full_command): + """ + Run a command on the optimal host. + """ + hook_signals() + + setup_logging(config) + + command = full_command[0] + command_args = full_command[1:] + + log.info(f"Starting rffmpeg as {command} with args: {' '.join(command_args)}") + + target_hid, target_hostname, target_servername = get_target_host(config) + + if not target_hostname or target_hostname == "localhost": + ret = run_local_command(config, command, command_args) + else: + ret = run_remote_command( + config, + target_hid, + target_hostname, + target_servername, + command, + command_args, + stderr_as_stdout=stderr_as_stdout + ) + + cleanup() + if ret.returncode == 0: + log.info(f"Finished rffmpeg with return code {ret.returncode}") + else: + log.error(f"Finished rffmpeg with return code {ret.returncode}") + exit(ret.returncode) + + rffmpeg_click.add_command(rffmpeg_click_run) + @click.command(name="log", short_help="View the rffmpeg log.") @click.option( "-f", From f6734839f966ddc783ba6d9659823adfcb062def Mon Sep 17 00:00:00 2001 From: Jendrik Weise Date: Thu, 17 Aug 2023 02:21:52 +0200 Subject: [PATCH 099/115] Improve argument quoting using shlex.quote --- rffmpeg | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/rffmpeg b/rffmpeg index ee530c9..376caac 100755 --- a/rffmpeg +++ b/rffmpeg @@ -25,6 +25,7 @@ import logging import os import signal import sys +import shlex import yaml from contextlib import contextmanager @@ -471,16 +472,8 @@ def run_remote_command( stdout = sys.stderr else: stdout = sys.stdout - - # Append all the passed arguments with requoting of any problematic characters - for arg in command_args: - # Escape $ characters - arg = arg.replace('$', '\\$') - # Match bad shell characters: * ' ( ) | [ ] or whitespace - if search("[*'()|\[\]\s]", arg): - rffmpeg_command.append(f'"{arg}"') - else: - rffmpeg_command.append(f"{arg}") + + rffmpeg_command.extend(map(shlex.quote, command_args)) log.info(f"Running command on host '{target_hostname}' ({target_servername})") log.debug(f"Remote command: {' '.join(rffmpeg_ssh_command + rffmpeg_command)}") From 40a2fc562f33d5d54734d77f4456adc17c5a713c Mon Sep 17 00:00:00 2001 From: Adrian Campos Garrido Date: Thu, 30 May 2024 10:29:19 +0200 Subject: [PATCH 100/115] Upgrade documentation to add new 10.9.3 version I just test (debian OS) with jellyfin-ffmpeg6 and is working correct so you can update to latest packages --- SETUP.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SETUP.md b/SETUP.md index d129a0c..77a09e6 100644 --- a/SETUP.md +++ b/SETUP.md @@ -140,14 +140,14 @@ This guide is provided as a basic starting point - there are myriad possible com * **NOTE:** Make sure you understand the caveats of using hardware transcoding with `rffmpeg` from the main README if you do decide to go this route. -1. Install the `jellyfin-ffmpeg` (Jellyfin <= 10.7.7) or `jellyfin-ffmpeg5` (Jellyfin >= 10.8.0) package; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just `jellyfin-ffmpeg[5]`. +1. Install the `jellyfin-ffmpeg` (Jellyfin <= 10.7.7), `jellyfin-ffmpeg5` (Jellyfin >= 10.8.0) or `jellyfin-ffmpeg6` (Jellyfin >= 10.9.3) package; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just `jellyfin-ffmpeg[5,6]`. ``` transcode1 $ sudo apt -y install curl gnupg transcode1 $ curl -fsSL https://repo.jellyfin.org/ubuntu/jellyfin_team.gpg.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/jellyfin.gpg transcode1 $ echo "deb [arch=$( dpkg --print-architecture )] https://repo.jellyfin.org/$( awk -F'=' '/^ID=/{ print $NF }' /etc/os-release ) $( awk -F'=' '/^VERSION_CODENAME=/{ print $NF }' /etc/os-release ) main" | sudo tee /etc/apt/sources.list.d/jellyfin.list transcode1 $ sudo apt update - transcode1 $ sudo apt install jellyfin-ffmpeg5 # or jellyfin-ffmpeg with Jellyfin <= 10.7.7 + transcode1 $ sudo apt install jellyfin-ffmpeg6 # or or jellyfin-ffmpeg5 with Jellyfin <= 10.8.0, jellyfin-ffmpeg with Jellyfin <= 10.7.7 ``` 1. Install the NFS client utilities: From f95ccd2fbfeaf63b1b111512064fbc70cc12dcda Mon Sep 17 00:00:00 2001 From: Adrian Campos Garrido Date: Thu, 30 May 2024 19:27:12 +0200 Subject: [PATCH 101/115] Add requested changes from author Add requested changes from author --- SETUP.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SETUP.md b/SETUP.md index 77a09e6..3d5c8bc 100644 --- a/SETUP.md +++ b/SETUP.md @@ -140,14 +140,14 @@ This guide is provided as a basic starting point - there are myriad possible com * **NOTE:** Make sure you understand the caveats of using hardware transcoding with `rffmpeg` from the main README if you do decide to go this route. -1. Install the `jellyfin-ffmpeg` (Jellyfin <= 10.7.7), `jellyfin-ffmpeg5` (Jellyfin >= 10.8.0) or `jellyfin-ffmpeg6` (Jellyfin >= 10.9.3) package; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just `jellyfin-ffmpeg[5,6]`. +1. Install the `jellyfin-ffmpeg` (Jellyfin <= 10.7.7), `jellyfin-ffmpeg5` (Jellyfin >= 10.8.13) or `jellyfin-ffmpeg6` (Jellyfin >= 10.9.0) package; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just `jellyfin-ffmpeg[5,6]`. ``` transcode1 $ sudo apt -y install curl gnupg transcode1 $ curl -fsSL https://repo.jellyfin.org/ubuntu/jellyfin_team.gpg.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/jellyfin.gpg transcode1 $ echo "deb [arch=$( dpkg --print-architecture )] https://repo.jellyfin.org/$( awk -F'=' '/^ID=/{ print $NF }' /etc/os-release ) $( awk -F'=' '/^VERSION_CODENAME=/{ print $NF }' /etc/os-release ) main" | sudo tee /etc/apt/sources.list.d/jellyfin.list transcode1 $ sudo apt update - transcode1 $ sudo apt install jellyfin-ffmpeg6 # or or jellyfin-ffmpeg5 with Jellyfin <= 10.8.0, jellyfin-ffmpeg with Jellyfin <= 10.7.7 + transcode1 $ sudo apt install jellyfin-ffmpeg6 # or jellyfin-ffmpeg5 with Jellyfin <= 10.8.13, jellyfin-ffmpeg with Jellyfin <= 10.7.7 ``` 1. Install the NFS client utilities: From d5285014a0a91ad009f5acfc22c4ecadf30cb6c8 Mon Sep 17 00:00:00 2001 From: Adrian Campos Garrido Date: Thu, 30 May 2024 19:30:39 +0200 Subject: [PATCH 102/115] Commit with requested changes Commit with requested changes --- SETUP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SETUP.md b/SETUP.md index 3d5c8bc..ce504cd 100644 --- a/SETUP.md +++ b/SETUP.md @@ -140,7 +140,7 @@ This guide is provided as a basic starting point - there are myriad possible com * **NOTE:** Make sure you understand the caveats of using hardware transcoding with `rffmpeg` from the main README if you do decide to go this route. -1. Install the `jellyfin-ffmpeg` (Jellyfin <= 10.7.7), `jellyfin-ffmpeg5` (Jellyfin >= 10.8.13) or `jellyfin-ffmpeg6` (Jellyfin >= 10.9.0) package; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just `jellyfin-ffmpeg[5,6]`. +1. Install the `jellyfin-ffmpeg` (Jellyfin <= 10.7.7), `jellyfin-ffmpeg5` (Jellyfin >= 10.8.0) or `jellyfin-ffmpeg6` (Jellyfin >= 10.9.0) package; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just `jellyfin-ffmpeg[5,6]`. ``` transcode1 $ sudo apt -y install curl gnupg From 7e535419d6691b79aa608181326c64f77552fdec Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Tue, 14 May 2024 13:38:57 -0400 Subject: [PATCH 103/115] Add debug to cleanup --- rffmpeg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rffmpeg b/rffmpeg index 74a6534..46fa525 100755 --- a/rffmpeg +++ b/rffmpeg @@ -238,6 +238,8 @@ def cleanup(signum="", frame=""): """ global config + log.debug(f"Cleaning up after signal {signum} PID {config['current_pid']}") + with dbconn(config) as cur: cur.execute( f"DELETE FROM states WHERE process_id = {SQL_VAR_SIGN}", From b043ed4db4f0974c94decc348c8132a4f22190c2 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 5 Jul 2024 17:13:33 -0400 Subject: [PATCH 104/115] Add no-root flag to initialization Allows rootless operation in cases where the configured state and database paths are writable by the initializing user, bypassing both the root check and disabling the chmod/chown later in the initialization. --- rffmpeg | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/rffmpeg b/rffmpeg index 46fa525..6d6703a 100755 --- a/rffmpeg +++ b/rffmpeg @@ -611,6 +611,13 @@ def run_control(config): ) @click.command(name="init", short_help="Initialize the system.") + @click.option( + "--no-root", + "no_root_flag", + is_flag=True, + default=False, + help="Bypass root check and permission adjustments; if destinations aren't writable, init will fail.", + ) @click.option( "-y", "--yes", @@ -619,13 +626,15 @@ def run_control(config): default=False, help="Confirm initialization.", ) - def rffmpeg_click_init(confirm_flag): + def rffmpeg_click_init(no_root_flag, confirm_flag): """ Initialize the rffmpeg system and database; this will erase all hosts and current state. - This command should be run as "sudo" before any attempts to use rffmpeg. + By default this command should be run as "sudo" before any attempts to use rffmpeg. If you + do not require root permissions to write to your configured state and database paths, you + may pass the "--no-root" option. """ - if os.getuid() != 0: + if not no_root_flag and os.getuid() != 0: click.echo("Error: This command requires root privileges.") exit(1) @@ -658,19 +667,21 @@ def run_control(config): except Exception as e: fail(f"Failed to create database: {e}") - os.chown( - config["state_dir"], - getpwnam(config["dir_owner"]).pw_uid, - getgrnam(config["dir_group"]).gr_gid, - ) - os.chmod(config["state_dir"], 0o770) - if DB_TYPE == "SQLITE": + if not no_root_flag: os.chown( - config["db_path"], + config["state_dir"], getpwnam(config["dir_owner"]).pw_uid, getgrnam(config["dir_group"]).gr_gid, ) - os.chmod(config["db_path"], 0o660) + os.chmod(config["state_dir"], 0o770) + + if DB_TYPE == "SQLITE": + os.chown( + config["db_path"], + getpwnam(config["dir_owner"]).pw_uid, + getgrnam(config["dir_group"]).gr_gid, + ) + os.chmod(config["db_path"], 0o660) rffmpeg_click.add_command(rffmpeg_click_init) From 6c581c0bf4cf670179ed40146c068e7fdf4c1bd4 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 5 Jul 2024 17:46:23 -0400 Subject: [PATCH 105/115] Revamp several sections of README --- README.md | 50 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index e8a21fd..26f973f 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ `rffmpeg` does require a little bit more configuration to work properly however. For a comprehensive installation tutorial based on a reference setup, please see [the SETUP guide](SETUP.md). -## Important Considerations +## Setup and Usage ### The `rffmpeg` Configuration file @@ -50,25 +50,31 @@ Each option has an explanatory comment above it detailing its purpose. Since the configuration file is YAML, ensure that you do not use "Tab" characters inside of it, only spaces. +### CLI interface to `rffmpeg` + +`rffmpeg` is a [Click](https://click.palletsprojects.com)-based application; thus, all commands have a `-h` or `--help` flag to show usage and additional options that may be specified. + ### Initializing `rffmpeg` -After first installing `rffmpeg`, ensure you initialize the database with the `sudo rffmpeg init` command. Note that `sudo` is required here to create the required data paths, but afterwards, `rffmpeg` can be run by anyone in the configured group (by default the `sudo` group). +After first installing `rffmpeg`, you must initialize the database with the `rffmpeg init` command. -`rffmpeg` is a Click-based application; thus, all commands have a `-h` or `--help` flag to show usage and additional options that may be specified. +Note that by default, `sudo`/root privilege is required for this command to create the required data paths, but afterwards, `rffmpeg` can be run by anyone in the configured group (by default the `sudo` group). You can bypass the `sudo` requirement with the `--no-root` command, for example when running in a rootless container; this will require the running user to have write permissions to the state and database parent directories, and will not perform any permissions modifications on the resulting files. ### Viewing Status -Once installed and initialized, the status of the `rffmpeg` system can be viewed with the command `rffmpeg status`. This will show all configured target hosts, their states, and any active commands being run. +Once installed and initialized, you can see the status of the `rffmpeg` system with the `rffmpeg status` command. This will show all configured target hosts, their states, and any active commands being run. ### Adding or Removing Target Hosts -To add a target host, use the command `rffmpeg add`. This command takes the optional `-w`/`--weight` flag to adjust the weight of the target host (see below). A host can be added more than once. +To add a target host, use the `rffmpeg add` command. You must add at least one target host for `rffmpeg` to be useful. This command takes the optional `-w`/`--weight` flag to adjust the weight of the target host (see below). A host can also be added more than once for a pseudo-weight, but this is an advanced usage. -To remove a target host, use the command `rffmpeg remove`. This command takes either a target host name/IP, which affects all instances of that name, or a specific host ID. Removing an in-use target host will not terminate any running processes, though it may result in undefined behaviour within `rffmpeg`. Before removing a host it is best to ensure there is nothing using it. +To remove a target host, use the `rffmpeg remove` command. This command takes either a target host name/IP, which affects all instances of that name, or a specific host ID. Removing an in-use target host will not terminate any running processes, though it may result in undefined behaviour within `rffmpeg`. Before removing a host it is best to ensure there is nothing using it. ### Viewing the Logfile -The `rffmpeg` CLI offers a convenient way to view the log file. Use `rffmpeg log` to view the entire logfile in the current pager (usually `less`), or use `rffmpeg log -f` to follow any new log entries after that point (like `tail -0 -f`). +The `rffmpeg` CLI offers a convenient way to view the log file. Use `rffmpeg log` to view the entire logfile in the default pager (usually `less`), or use `rffmpeg log -f` to follow any new log entries after that point (like `tail -0 -f`). + +## Important Considerations ### Localhost and Fallback @@ -76,10 +82,16 @@ If one of the configured target hosts is called `localhost` or `127.0.0.1`, `rff In addition, `rffmpeg` will fall back to `localhost` automatically, even if it is not explicitly configured, should it be unable to find any working remote hosts. This helps prevent situations where `rffmpeg` cannot be run due to none of the remote host(s) being available. -In both cases, note that, if hardware acceleration is configured, it *must* be available on the local host as well, or the `ffmpeg` commands will fail. There is no easy way around this without rewriting arguments, and this is currently out-of-scope for `rffmpeg`. You should always use a lowest-common-denominator approach when deciding on what additional option(s) to enable, such that any configured host can run any process, or accept that fallback will not work if all remote hosts are unavailable. - The exact path to the local `ffmpeg` and `ffprobe` binaries can be overridden in the configuration, should their paths not match those of the remote system(s). +### Hardware Acceleration + +Note that if hardware acceleration is configured in the calling application, **the exact same hardware acceleration modes must be available on all configured hosts, and, for fallback to work, the local host as well**, or the `ffmpeg` commands will fail. + +This is an explicit requirement, and there is no easy way around this without rewriting the passed arguments, which is explicitly out-of-scope for `rffmpeg` (see the FAQ entry below about mangling arguments). + +You should always use a lowest-common-denominator approach when deciding what hardware acceleration option(s) to enable, such that any configured host can run any process, or accept that fallback will not work if all remote hosts are unavailable. + ### Target Host Selection When more than one target host is present, `rffmpeg` uses the following rules to select a target host. These rules are evaluated each time a new `rffmpeg` alias process is spawned based on the current state (actively running processes, etc.). @@ -118,7 +130,7 @@ If for some reason all configured hosts are marked `bad`, fallback will be engag ### Why did you make `rffmpeg`? -My virtualization setup (multiple 1U nodes with lots of live migration/failover) didn't lend itself well to passing a GPU into my Jellyfin VM, but I wanted to offload transcoding because doing 4K HEVC transcodes with a CPU performs horribly. I happened to have another machine (my "base" remote headless desktop/gaming server) which had a GPU, so I wanted to find a way to offload the transcoding to it. I came up with `rffmpeg` as a simple wrapper to the `ffmpeg` and `ffprobe` calls that Jellyfin (and Emby, and likely other media servers too) makes which would run them on that host instead. After finding it quite useful myself, I released it publicly as GPLv3 software so that others may benefit as well! +My virtualization setup (multiple 1U nodes with lots of live migration/failover) didn't lend itself well to passing a GPU into my Jellyfin VM, but I wanted to offload transcoding because doing 4K HEVC transcodes with a CPU performs horribly. I happened to have another machine (my "base" remote headless desktop/gaming server) which had a GPU, so I wanted to find a way to offload the transcoding to it. I came up with `rffmpeg` as a simple wrapper to the `ffmpeg` and `ffprobe` calls that Jellyfin (and Emby, and likely other media servers too) makes which would run them on that host instead. After finding it quite useful myself, I released it publicly as GPLv3 software so that others may benefit as well! It has since received a lot of feedback and feature requests from the community, leading to the tool as it exists today. ### What supports `rffmpeg`? @@ -134,19 +146,23 @@ This depends on what "layer" you're asking at. ### Can `rffmpeg` mangle/alter FFMPEG arguments? -Explicitly *no*. `rffmpeg` is not designed to interact with the arguments that the media server passes to `ffmpeg`/`ffprobe` at all, nor will it. This is an explicit design decision due to the massive complexity of FFMpeg - to do this, I would need to create a mapping of just about every possible FFMpeg argument, what it means, and when to turn it on or off, which is way out of scope. +Explicitly *no*. `rffmpeg` is not designed to interact with the arguments that the media server passes to `ffmpeg`/`ffprobe` at all, nor will it. -This has a number of side effects: +This is an explicit design decision due to the massive complexity of FFmpeg. FFmpeg has a very large number of possible arguments, many of which are position-dependent or dependent on other arguments elsewhere in the chain. To implement argument mangling, we would need to be aware of every possible FFmpeg argument, exactly how each argument maps to each other argument, and be able to dynamically parse and update arguments based on this. As should hopefully be quite obvious, this is a massive undertaking and not something that I have any desire to implement or manage in such a (relatively) simple utility. - * `rffmpeg` does not know whether hardware acceleration is turned on or not (see above caveats under [Localhost and Fallback](#localhost-and-fallback)). - * `rffmpeg` does not know what media is playing or where it's outputting files to, and cannot alter these paths. - * `rffmpeg` cannot turn on or off special `ffmpeg` options depending on the host selected. +This has a number of effects: -Thus it is imperative that you set up your entire system correctly for `rffmpeg` to work. Please see the [SETUP guide](SETUP.md) for more information. + * `rffmpeg` cannot adjust any `ffmpeg` options based on the host selected. + * `rffmpeg` does not know whether hardware acceleration is turned on or not (see above caveats under [Hardware Acceleration](#hardware-acceleration)), or what type(s) of hardware acceleration are active. + * `rffmpeg` does not know what media file(s) is is handling or where it's outputting files to, and cannot alter these paths. + +Thus it is imperative that you set up your entire system correctly for `rffmpeg` to work using a "least-common-denominator" approach as required. Please see the [SETUP guide](SETUP.md) for more information. ### Can `rffmpeg` do Wake-On-LAN or other similar options to turn on a transcode server? -Right now, not officially. You can use the linuxserver.io [docker mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg). I've thought about implementing this more than once (most recently, in response to [Issue #21](https://github.com/joshuaboniface/rffmpeg/issues/21)) but ultimately I've never though this was worth the complexity and delays in spawning that it would add to the tool. That issue does provide one example of a workaround wrapper script that could accomplish this, but I don't see it being a part of the actual tool itself. +Explicitly *no*, though the linuxserver.io [docker mod](https://github.com/linuxserver/docker-mods/tree/jellyfin-rffmpeg) does support this. + +I've thought about implementing this more than once (most recently, in response to [Issue #21](https://github.com/joshuaboniface/rffmpeg/issues/21)) but ultimately I do not believe this is worth the complexity and delays it would introduce when spawning processes. That issue does provide one example of a workaround wrapper script that could accomplish this, but I do not plan for it to be a part of `rffmpeg` itself. ### I'm getting an error, help! From b4b0950d1e1a48d18dec1b9dbc3d17f9c534d4b7 Mon Sep 17 00:00:00 2001 From: Till Fricke Date: Mon, 29 Jul 2024 11:04:02 +0200 Subject: [PATCH 106/115] fix localhost always being saved as target_hid = 0 --- rffmpeg | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/rffmpeg b/rffmpeg index 6d6703a..07f65f5 100755 --- a/rffmpeg +++ b/rffmpeg @@ -404,7 +404,7 @@ def get_target_host(config): return target_hid, target_hostname, target_servername -def run_local_command(config, command, command_args, stderr_as_stdout = False, mapped_cmd = None): +def run_local_command(config, target_hid, command, command_args, stderr_as_stdout = False, mapped_cmd = None): """ Run command locally, either because "localhost" is the target host, or because no good target host was found by get_target_host(). @@ -427,28 +427,32 @@ def run_local_command(config, command, command_args, stderr_as_stdout = False, m log.info("Running command on host 'localhost'") log.debug(f"Local command: {' '.join(rffmpeg_command)}") + if target_hid is None: + target_hid = 0 + log.debug(f"Could not find any host, using fallback 'localhost'") + with dbconn(config) as cur: cur.execute( f"INSERT INTO processes (host_id, process_id, cmd) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", - (0, config["current_pid"], command + ' ' + ' '.join(command_args)), + (target_hid, config["current_pid"], command + ' ' + ' '.join(command_args)), ) cur.execute( f"INSERT INTO states (host_id, process_id, state) VALUES ({SQL_VAR_SIGN}, {SQL_VAR_SIGN}, {SQL_VAR_SIGN})", - (0, config["current_pid"], "active"), + (target_hid, config["current_pid"], "active"), ) return run_command(rffmpeg_command, stdin, stdout, stderr) -def run_local_ffmpeg(config, ffmpeg_args): +def run_local_ffmpeg(config, target_hid, ffmpeg_args): """ Run ffmpeg locally, either because "localhost" is the target host, or because no good target host was found by get_target_host(). """ if "ffprobe" in cmd_name: - return run_local_command(config, cmd_name, ffmpeg_args, mapped_cmd=config["fallback_ffprobe_command"]) + return run_local_command(config, target_hid, cmd_name, ffmpeg_args, mapped_cmd=config["fallback_ffprobe_command"]) else: - return run_local_command(config, cmd_name, ffmpeg_args, stderr_as_stdout=not any(item in config["special_flags"] for item in ffmpeg_args), mapped_cmd=config["fallback_ffmpeg_command"]) + return run_local_command(config, target_hid, cmd_name, ffmpeg_args, stderr_as_stdout=not any(item in config["special_flags"] for item in ffmpeg_args), mapped_cmd=config["fallback_ffmpeg_command"]) def run_remote_command( @@ -554,7 +558,7 @@ def run_ffmpeg(config, ffmpeg_args): target_hid, target_hostname, target_servername = get_target_host(config) if not target_hostname or target_hostname == "localhost": - ret = run_local_ffmpeg(config, ffmpeg_args) + ret = run_local_ffmpeg(config, target_hid, ffmpeg_args) else: ret = run_remote_ffmpeg( config, target_hid, target_hostname, target_servername, ffmpeg_args @@ -917,7 +921,7 @@ def run_control(config): target_hid, target_hostname, target_servername = get_target_host(config) if not target_hostname or target_hostname == "localhost": - ret = run_local_command(config, command, command_args) + ret = run_local_command(config, target_hid, command, command_args) else: ret = run_remote_command( config, From 16ddb2bdeeb773284560b06517b3735f540709b1 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 5 Jul 2024 18:18:35 -0400 Subject: [PATCH 107/115] Wrap command runs in a finally block Ensures that cleanup happens even if something else weird happens, which tends to happen to my systems a lot leaving stuck processes. --- rffmpeg | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/rffmpeg b/rffmpeg index 07f65f5..b459860 100755 --- a/rffmpeg +++ b/rffmpeg @@ -557,19 +557,23 @@ def run_ffmpeg(config, ffmpeg_args): target_hid, target_hostname, target_servername = get_target_host(config) - if not target_hostname or target_hostname == "localhost": - ret = run_local_ffmpeg(config, target_hid, ffmpeg_args) - else: - ret = run_remote_ffmpeg( - config, target_hid, target_hostname, target_servername, ffmpeg_args - ) - - cleanup() - if ret.returncode == 0: - log.info(f"Finished rffmpeg with return code {ret.returncode}") - else: - log.error(f"Finished rffmpeg with return code {ret.returncode}") - exit(ret.returncode) + try: + returncode = 200 + if not target_hostname or target_hostname == "localhost": + ret = run_local_ffmpeg(config, ffmpeg_args) + returncode = ret.returncode + else: + ret = run_remote_ffmpeg( + config, target_hid, target_hostname, target_servername, ffmpeg_args + ) + returncode = ret.returncode + finally: + cleanup() + if returncode == 0: + log.info(f"Finished rffmpeg with return code {ret.returncode}") + else: + log.error(f"Finished rffmpeg with return code {ret.returncode}") + exit(returncode) def run_control(config): From 6117e0d81d6ceb9fb5a788ee351eafd82a52bccb Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 16 Aug 2024 09:43:13 -0400 Subject: [PATCH 108/115] Update SETUP guide for newer versions Fixes a few bugs, and includes setup details for more recent versions of Jellyfin that do not allow setting the FFmpeg path from the WebUI. --- SETUP.md | 56 +++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 37 insertions(+), 19 deletions(-) diff --git a/SETUP.md b/SETUP.md index ce504cd..7fd310d 100644 --- a/SETUP.md +++ b/SETUP.md @@ -85,16 +85,16 @@ This guide is provided as a basic starting point - there are myriad possible com jellyfin1 $ sudo $EDITOR /etc/rffmpeg/rffmpeg.yml # if required ``` -1. Initialize `rffmpeg` (note the `sudo` command) and add at least one remote host to it. You can add multiple hosts now or later, set weights of hosts, and add a host more than once. For full details see the [main README](README.md) or run `rffmpeg --help` to view the CLI help menu. +1. Initialize `rffmpeg` (note the `sudo` command) and add at the target host to it. You can add other hosts now or later, and set weights of hosts, if required; for full details see the [main README](README.md) or run `rffmpeg --help` to view the CLI help menu. ``` jellyfin1 $ sudo rffmpeg init --yes - jellyfin1 $ rffmpeg add --weight 1 gpu1 + jellyfin1 $ rffmpeg add --weight 1 transcode1 ``` ### NFS Setup -* **WARNING:** This guide assumes your hosts are on the same private local network. It is not recommended to run NFS over the Internet as it is unencrypted and bandwidth-intensive. Consider other remote filesystems like SSHFS in such cases as these will offer greater privacy and robustness. +* **WARNING:** This guide assumes your hosts are on the same private local network. It is not recommended to run NFS over the Internet as it is unencrypted, and any rffmpeg connection will be very bandwidth-intensive. If you must have both systems in separate networks, consider other remote filesystems like SSHFS in such cases as these will offer greater privacy and robustness. 1. Install the NFS kernel server. We will use NFS to export the various required directories so the transcode machine can read from and write to them. @@ -106,11 +106,9 @@ This guide is provided as a basic starting point - there are myriad possible com * Always export the `${jellyfin_data_path}` in full. Advanced users might be able to export the required subdirectories individually, but I find this to be not worth the hassle. * Note the security options of NFS. It will limit mounts to the IP addresses specified. If your home network is secure, you can use the entire network, e.g. `192.168.0.0/24`, but I would recommend determining the exact IP of your transcode server(s) and use them explicitly, e.g. for this example `192.168.0.101` and `192.168.0.102`. - * It's quite important to both set `sync` on the transcode host(s), and ensure that the `transcodes` directory is on a real filesystem on the Jellyfin system (i.e. is not a remote filesystem mount itself) which is then exported to the clients. If this is not the case, playback will be very slow to start. I believe the reason for this is that Jellyfin (and presumably Emby) listen for an `inotify` on the directory to know that the playlist is ready for consumption, but I have not confirmed this, and NFS *et al.* do not properly support this. + * If your `transcodes` directory is not on a **native Linux filesystem** (i.e. external to Jellyfin, such as on a NAS exported by NFS, SMB, etc.), then you may experience delays of ~15-60s when playback starts. This is because NFS uses a file attribute cache that in most applications greatly increases performance, however for this usecase it causes a delay in Jellyfin seeing the `.ts` files. The solution for this is to reduce the NFS cache time by adding `sync` and `actimeo=1` to your NFS mount(s) (command or fstab), which will set the NFS file attribute cache to 1 second (reducing the NFS delay to ~1-2 seconds). This time can be further reduced to 0 by setting the `noac` option, but this is not normally recommended because it will negatively impact the performance other NFS applications. Verify that your mount added the `actimeo=1` parameter correcly by checking `mount` or `cat /proc/mounts`, which will show `sync,acregmin=1,acregmax=1,acdirmin=1,acdirmax=1` as parameters for your `transcodes` mount. * If your media is local to the Jellyfin server (and not already mountable on the transcode host(s) via a remote filesystems like NFS, Samba, CephFS, etc.), also add an export for it as well. - * If your `transcodes` directory is external to Jellyfin, such as a NAS, then you may experience delays of ~15-60s starting content as NFS uses a file attribute cache that in most applications greatly increases performance, however for this usecase it causes a delay in Jellyfin seeing the `.ts` files. The solution for this is to reduce the NFS cache time by adding `actimeo=1` to your mount command (or fstab), which will set the NFS file attribute cache to 1 second (reducing the NFS delay to ~1-2 seconds.) It is not recommended to use the `noac` flag, which would reduce the NFS delay to ~0, but at the cost of negatively impacting other NFS performance. To verify your mount added the `actimeo=1` parameter correcly `cat /proc/mounts`, which will show `acregmin=1,acregmax=1,acdirmin=1,acdirmax=1` as parameters for your `transcodes` mount. - An example `/etc/exports` file would look like this: ``` @@ -138,16 +136,18 @@ This guide is provided as a basic starting point - there are myriad possible com 1. Install and configure anything you need for hardware transcoding, if applicable. For example GPU drivers if using a GPU for transcoding. - * **NOTE:** Make sure you understand the caveats of using hardware transcoding with `rffmpeg` from the main README if you do decide to go this route. + * **NOTE:** Make sure you understand the caveats of using hardware transcoding with `rffmpeg` from [the main README](README.md#hardware-acceleration). -1. Install the `jellyfin-ffmpeg` (Jellyfin <= 10.7.7), `jellyfin-ffmpeg5` (Jellyfin >= 10.8.0) or `jellyfin-ffmpeg6` (Jellyfin >= 10.9.0) package; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just `jellyfin-ffmpeg[5,6]`. +1. Install the correct `jellyfin-ffmpeg` package for your version of Jellyfin; check which version is installed on your `jellyfin1` system with `dpkg -l | grep jellyfin-ffmpeg`, then install that version on this host too; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just the `jellyfin-ffmpeg` of the required version. ``` + jellyfin1 $ dpkg -l | grep jellyfin-ffmpeg + ii jellyfin-ffmpeg6 6.0.1-8-bookworm amd64 Tools for transcoding, streaming and playing of multimedia files transcode1 $ sudo apt -y install curl gnupg transcode1 $ curl -fsSL https://repo.jellyfin.org/ubuntu/jellyfin_team.gpg.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/jellyfin.gpg transcode1 $ echo "deb [arch=$( dpkg --print-architecture )] https://repo.jellyfin.org/$( awk -F'=' '/^ID=/{ print $NF }' /etc/os-release ) $( awk -F'=' '/^VERSION_CODENAME=/{ print $NF }' /etc/os-release ) main" | sudo tee /etc/apt/sources.list.d/jellyfin.list transcode1 $ sudo apt update - transcode1 $ sudo apt install jellyfin-ffmpeg6 # or jellyfin-ffmpeg5 with Jellyfin <= 10.8.13, jellyfin-ffmpeg with Jellyfin <= 10.7.7 + transcode1 $ sudo apt install jellyfin-ffmpeg6 ``` 1. Install the NFS client utilities: @@ -165,7 +165,7 @@ This guide is provided as a basic starting point - there are myriad possible com * **NOTE:** For some hardware acceleration, you might need to add this user to additional groups. For example `--groups video,render`. - * **NOTE:** The UID and GIDs here are dynamic; on the `jellyfin1` machine, they would have been created at install time with the next available ID in the range 100-199 (at least in Debian/Ubuntu). However, this means that the exact UID of your Jellyfin service user might not be available on your transcode server, depending on what packages are installed and in what order. If there is a conflict, you must adjust user IDs on one side or the other so that they match on both machines. You can use `sudo usermod` to change a user's ID if required. + * **NOTE:** The UID and GIDs here are dynamic; on the `jellyfin1` machine, they would have been selected automatically at install time with the next available ID in the range 100-199 (at least in Debian/Ubuntu). However, this means that the exact UID of your Jellyfin service user might not be available on your transcode server, depending on what packages are installed and in what order. If there is a conflict, you must adjust user IDs on one side or the other so that they match on both machines. You can use `sudo usermod` to change a user's ID if required. 1. Create the Jellyfin data directory at the same location as on the media server, and set it immutable so that it won't be written to if the NFS mount goes down: @@ -208,12 +208,14 @@ This guide is provided as a basic starting point - there are myriad possible com ``` transcode1 $ sudo systemctl daemon-reload - transcode1 $ sudo systemctl start var-lib-jellyfin.mount + transcode1 $ sudo systemctl enable --now var-lib-jellyfin.mount ``` Note that mount units are fairly "new" and can be a bit finicky, be sure to read the SystemD documentation if you get stuck! Generally for new users, I'd recommend the `/etc/fstab` method instead. -1. Mount your media directories in the same location(s) as on the media server. If you exported them via NFS from your media server, use the process above only for those directories instead. + **NOTE:** Don't forget about `actimeo=1` here if you need it! + +1. Mount your media directories in the **same location(s)** as on the media server. If you exported them via NFS from your media server, use the process above only for those directories instead. ## Test the setup @@ -237,23 +239,39 @@ This guide is provided as a basic starting point - there are myriad possible com [...] ``` -As long as these steps work, all further steps should as well. +As long as these steps work, all further steps should as well. If one of these *doesn't* work, double-check all previous steps and confirm that everything is set up right. -## Configure Jellyfin +## Configure Jellyfin to use `rffmpeg` -1. In the Hamburger Menu -> Administration -> Dashboard, navigate to Playback. +**NOTE**: With Jellyfin 10.8.13 and newer, the ability to configure the `ffmpeg` path has been removed from the WebUI due to major security concerns. You must follow this method to change it. + +1. On the `jellyfin1` system, edit `/etc/default/jellyfin`: + + ``` + jellyfin1 $ sudo $EDITOR /etc/default/jellyfin + ``` + +1. Change the value of `JELLYFIN_FFMPEG_OPT` to be `--ffmpeg=/usr/local/bin/ffmpeg` (the `rffmpeg` alias name `ffmpeg` in whatever path you installed `rffmpeg` to). + +1. Save the file and restart Jellyfin: + + ``` + jellyfin1 $ sudo systemctl restart jellyfin + ``` + +If you wish to use hardware transcoding, you must also enable it in Jellyfin's WebUI: + +1. Navigate to Hamburger Menu -> Administration -> Dashboard, navigate to Playback. 1. Configure any hardware acceleration you require and have set up on the remote server(s). -1. Under "FFmpeg path:", enter `/usr/local/bin/ffmpeg`. - 1. Save the settings. -1. Try to play a movie that requires transcoding, and verify that everything is working as expected. +Now, run `rffmpeg log -f` on the `jellyfin1` machine and try to play a video that requires transcoding. You should see `rffmpeg` spawn a process on the `jellyfin1` machine, which then begins running the `ffmpeg` process on the `transcode1` machine, writing data to the configured paths, and playback should begin normally. If anything doesn't work, double-check all previous steps and confirm that everything is set up right. ## NOTE for NVEnv/NVDec Hardware Acceleration -If you are using NVEnv/NVDec, it's probably a good idea to symlink the `.nv` folder inside the Jellyfin user's homedir (i.e. `/var/lib/jellyfin/.nv`) to somewhere outside of the NFS volume on both sides. For example: +If you are using NVEnv/NVDec, you will need to symlink the `.nv` folder inside the Jellyfin user's homedir (i.e. `/var/lib/jellyfin/.nv`) to somewhere outside of the NFS volume on both the Jellyfin and transcoding hosts. For example: ``` jellyfin1 $ sudo mv /var/lib/jellyfin/.nv /var/lib/nvidia-cache # or "sudo mkdir /var/lib/nvidia-cache" and "sudo chown jellyfin /var/lib/nvidia-cache" if it does not yet exist From cfe816a377dcf061223d075fb74c081387caf4cf Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Fri, 16 Aug 2024 09:47:24 -0400 Subject: [PATCH 109/115] Add check for log file existence --- rffmpeg | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rffmpeg b/rffmpeg index b459860..d5c4a17 100755 --- a/rffmpeg +++ b/rffmpeg @@ -960,6 +960,10 @@ def run_control(config): View the rffmpeg log file. """ if follow_flag: + if not os.path.exists(config["logfile"]): + click.echo("Failed to find log file; have you initialized rffmpeg?") + exit(1) + with open(config["logfile"]) as file_: # Go to the end of file file_.seek(0, 2) From 2fa9c57b235e5d1c362a9750bfdd2c80ae96469d Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 19 Aug 2024 14:20:39 -0400 Subject: [PATCH 110/115] Ensure HID is passed to run_local_ffmpeg Fixes #87 --- rffmpeg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rffmpeg b/rffmpeg index d5c4a17..7f1bd8c 100755 --- a/rffmpeg +++ b/rffmpeg @@ -560,7 +560,7 @@ def run_ffmpeg(config, ffmpeg_args): try: returncode = 200 if not target_hostname or target_hostname == "localhost": - ret = run_local_ffmpeg(config, ffmpeg_args) + ret = run_local_ffmpeg(config, None, ffmpeg_args) returncode = ret.returncode else: ret = run_remote_ffmpeg( From da27620522c2011644aef54258fa71e3439d197e Mon Sep 17 00:00:00 2001 From: Toddneal Stallworth Date: Wed, 9 Oct 2024 22:08:15 -0700 Subject: [PATCH 111/115] fixed spelling mistakes --- SETUP.md | 2 +- rffmpeg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/SETUP.md b/SETUP.md index 7fd310d..936bfc6 100644 --- a/SETUP.md +++ b/SETUP.md @@ -106,7 +106,7 @@ This guide is provided as a basic starting point - there are myriad possible com * Always export the `${jellyfin_data_path}` in full. Advanced users might be able to export the required subdirectories individually, but I find this to be not worth the hassle. * Note the security options of NFS. It will limit mounts to the IP addresses specified. If your home network is secure, you can use the entire network, e.g. `192.168.0.0/24`, but I would recommend determining the exact IP of your transcode server(s) and use them explicitly, e.g. for this example `192.168.0.101` and `192.168.0.102`. - * If your `transcodes` directory is not on a **native Linux filesystem** (i.e. external to Jellyfin, such as on a NAS exported by NFS, SMB, etc.), then you may experience delays of ~15-60s when playback starts. This is because NFS uses a file attribute cache that in most applications greatly increases performance, however for this usecase it causes a delay in Jellyfin seeing the `.ts` files. The solution for this is to reduce the NFS cache time by adding `sync` and `actimeo=1` to your NFS mount(s) (command or fstab), which will set the NFS file attribute cache to 1 second (reducing the NFS delay to ~1-2 seconds). This time can be further reduced to 0 by setting the `noac` option, but this is not normally recommended because it will negatively impact the performance other NFS applications. Verify that your mount added the `actimeo=1` parameter correcly by checking `mount` or `cat /proc/mounts`, which will show `sync,acregmin=1,acregmax=1,acdirmin=1,acdirmax=1` as parameters for your `transcodes` mount. + * If your `transcodes` directory is not on a **native Linux filesystem** (i.e. external to Jellyfin, such as on a NAS exported by NFS, SMB, etc.), then you may experience delays of ~15-60s when playback starts. This is because NFS uses a file attribute cache that in most applications greatly increases performance, however for this usecase it causes a delay in Jellyfin seeing the `.ts` files. The solution for this is to reduce the NFS cache time by adding `sync` and `actimeo=1` to your NFS mount(s) (command or fstab), which will set the NFS file attribute cache to 1 second (reducing the NFS delay to ~1-2 seconds). This time can be further reduced to 0 by setting the `noac` option, but this is not normally recommended because it will negatively impact the performance other NFS applications. Verify that your mount added the `actimeo=1` parameter correctly by checking `mount` or `cat /proc/mounts`, which will show `sync,acregmin=1,acregmax=1,acdirmin=1,acdirmax=1` as parameters for your `transcodes` mount. * If your media is local to the Jellyfin server (and not already mountable on the transcode host(s) via a remote filesystems like NFS, Samba, CephFS, etc.), also add an export for it as well. An example `/etc/exports` file would look like this: diff --git a/rffmpeg b/rffmpeg index 7f1bd8c..5610a3e 100755 --- a/rffmpeg +++ b/rffmpeg @@ -649,7 +649,7 @@ def run_control(config): if not confirm_flag: try: click.confirm( - "Are you sure you want to (re)initalize the database", + "Are you sure you want to (re)initialize the database", prompt_suffix="? ", abort=True, ) From f867610e2ef607230a56e45918cacf8706e7f93f Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 28 Oct 2024 02:58:15 -0400 Subject: [PATCH 112/115] Add notes for Jellyfin 10.10.x tempdir --- README.md | 2 ++ SETUP.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/README.md b/README.md index 26f973f..d2936cb 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,8 @@ `rffmpeg` does require a little bit more configuration to work properly however. For a comprehensive installation tutorial based on a reference setup, please see [the SETUP guide](SETUP.md). +**NOTE** Jellyfin 10.10.x and newer require an additional `TMPDIR` environment variable set to somewhere exported to the remote machine, or these paths will not work properly. Edit your Jellyfin startup/service configuration to set that. See the setup guide for more details. + ## Setup and Usage ### The `rffmpeg` Configuration file diff --git a/SETUP.md b/SETUP.md index 7fd310d..cf81556 100644 --- a/SETUP.md +++ b/SETUP.md @@ -35,6 +35,8 @@ This guide is provided as a basic starting point - there are myriad possible com **NOTE:** On Docker, these directories are different. The main data directory (our `jellyfin_data_path`) is `/config`, and the cache directory is separate at `/cache`. Both must be exported and mounted on targets for proper operation. + **NOTE:** On Jellyfin 10.10.x and newer, temporary transient files were moved into the system temporary storage path (on Linux, usually `/tmp`). This will break rffmpeg for certain tasks that use these files, for instance trickplay generation. To restore the previous behaviour, ensure you set the `TMPDIR` environment variable for your Jellyfin service to a path under the data path above, for example `/var/lib/jellyfin/temp`, and create this directory with correct ownership and permissions. + 1. Create an SSH keypair to use for `rffmpeg`'s login to the remote server. For ease of use with the following steps, use the Jellyfin service user (`jellyfin`) to create the keypair and store it under its home directory (the Jellyfin data path above). I use `rsa` here but you can substitute `ed25519` instead (avoid `dsa` and `ecdsa` for reasons I won't get into here). Once done, copy the public key to `authorized_keys` which will be used to authenticate the key later. ``` From df5214cc9d5eb843558752e30a0eee909d1fb99a Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Mon, 28 Oct 2024 03:13:12 -0400 Subject: [PATCH 113/115] Add cache path to setup guide Closes #88 --- SETUP.md | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/SETUP.md b/SETUP.md index ca6ce45..e93b70c 100644 --- a/SETUP.md +++ b/SETUP.md @@ -23,15 +23,17 @@ This guide is provided as a basic starting point - there are myriad possible com ``` jellyfin1 $ export jellyfin_data_path="/var/lib/jellyfin" + jellyfin1 $ export jellyfin_cache_path="/var/lib/jellyfin" transcode1 $ export jellyfin_data_path="/var/lib/jellyfin" + transcode1 $ export jellyfin_cache_path="/var/lib/jellyfin" ``` The important subdirectories for `rffmpeg`'s operation are: - * `transcodes/`: Used to store on-the-fly transcoding files, and configurable separately in Jellyfin but with `rffmpeg` I recommend leaving it at the default location under the data path. - * `data/subtitles/`: Used to store on-the-fly extracted subtitles so that they can be reused later. - * `cache/`: Used to store cached extracted data. - * `.ssh/`: This doesn't exist yet but will after the next step. + * `$jellyfin_cache_path/`: Used to store cached extracted data. + * `$jellyfin_cache_path/transcodes/`: Used to store on-the-fly transcoding files, and configurable separately in Jellyfin but with `rffmpeg` I recommend leaving it at the default location under the cache path. + * `$jellyfin_data_path/data/subtitles/`: Used to store on-the-fly extracted subtitles so that they can be reused later. + * `$jellyfin_data_path/.ssh/`: This doesn't exist yet but will after the next step. **NOTE:** On Docker, these directories are different. The main data directory (our `jellyfin_data_path`) is `/config`, and the cache directory is separate at `/cache`. Both must be exported and mounted on targets for proper operation. @@ -121,6 +123,8 @@ This guide is provided as a basic starting point - there are myriad possible com # jellyfin_data_path first host second host, etc. /var/lib/jellyfin 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102/32(rw,sync,no_subtree_check,no_root_squash) + # jellyfin_cache_path first host second host, etc. + /var/cache/jellyfin 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102/32(rw,sync,no_subtree_check,no_root_squash) # Local media path if required /srv/mymedia 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102/32(rw,sync,no_subtree_check,no_root_squash) ``` @@ -132,6 +136,8 @@ This guide is provided as a basic starting point - there are myriad possible com jellyfin1 $ sudo exportfs /var/lib/jellyfin 192.168.0.101/32 /var/lib/jellyfin 192.168.0.102/32 + /var/cache/jellyfin 192.168.0.101/32 + /var/cache/jellyfin 192.168.0.102/32 ``` ## Set up the transcode server (`transcode1`) @@ -184,7 +190,9 @@ This guide is provided as a basic starting point - there are myriad possible com ``` transcode1 $ echo "jellyfin1:${jellyfin_data_path} ${jellyfin_data_path} nfs defaults,vers=3,sync" | sudo tee -a /etc/fstab + transcode1 $ echo "jellyfin1:${jellyfin_cache_path} ${jellyfin_cache_path} nfs defaults,vers=3,sync" | sudo tee -a /etc/fstab transcode1 $ sudo mount ${jellyfin_data_path} + transcode1 $ sudo mount ${jellyfin_cache_path} ``` * Use a SystemD `mount` unit, which is a newer way of doing mounts with SystemD. I personally prefer this method as I find it easier to set up automatically, but this is up to preference. An example based on mine would be: @@ -206,11 +214,29 @@ This guide is provided as a basic starting point - there are myriad possible com WantedBy = remote-fs.target ``` + ``` + transcode1 $ cat /etc/systemd/system/var-cache-jellyfin.mount + [Unit] + Description = NFS volume for Jellyfin cache directory + Requires = network-online.target + After = network-online.target + + [Mount] + type = nfs + What = jellyfin1:/var/cache/jellyfin + Where = /var/cache/jellyfin + Options = _netdev,sync,vers=3 + + [Install] + WantedBy = remote-fs.target + ``` + Once the unit file is created, you can then reload the unit list and mount it: ``` transcode1 $ sudo systemctl daemon-reload transcode1 $ sudo systemctl enable --now var-lib-jellyfin.mount + transcode1 $ sudo systemctl enable --now var-cache-jellyfin.mount ``` Note that mount units are fairly "new" and can be a bit finicky, be sure to read the SystemD documentation if you get stuck! Generally for new users, I'd recommend the `/etc/fstab` method instead. @@ -255,6 +281,8 @@ As long as these steps work, all further steps should as well. If one of these * 1. Change the value of `JELLYFIN_FFMPEG_OPT` to be `--ffmpeg=/usr/local/bin/ffmpeg` (the `rffmpeg` alias name `ffmpeg` in whatever path you installed `rffmpeg` to). +1. On Jellyfin 10.10.x or newer, add `TMPDIR=$jellyfin_cache_path/temp`, for instance `TMPDIR=/var/cache/jellyfin/temp`, to ensure this is properly synchronized over the network. + 1. Save the file and restart Jellyfin: ``` From 5994d0e7a062ef8c032c09ccf339e141c311abf7 Mon Sep 17 00:00:00 2001 From: Juha Leivo Date: Mon, 3 Nov 2025 16:16:16 +0200 Subject: [PATCH 114/115] Feat/ssh hardening SSH hardening configuration - Access for jellyfin user will be limited to jellyfin1 server only - Commands that jellyfin user can run will be limited to ffmpeg only - Commands run by jellyfin user will be logged - (optional) Logs stored in separate log file tweaks on documentation - docs folder - updated SETUP.MD to be copy paste friendly - added HARDENING.md --- README.md | 4 +- docs/HARDENING | 92 +++++++++++++ SETUP.md => docs/SETUP.md | 214 ++++++++++++++++------------- hardening/10-jellyfin-limits.conf | 13 ++ hardening/limited-wrapper-log.conf | 3 + hardening/limited-wrapper.sh | 87 ++++++++++++ 6 files changed, 314 insertions(+), 99 deletions(-) create mode 100644 docs/HARDENING rename SETUP.md => docs/SETUP.md (72%) create mode 100644 hardening/10-jellyfin-limits.conf create mode 100644 hardening/limited-wrapper-log.conf create mode 100755 hardening/limited-wrapper.sh diff --git a/README.md b/README.md index d2936cb..2c0fddb 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ 1. Profit! -`rffmpeg` does require a little bit more configuration to work properly however. For a comprehensive installation tutorial based on a reference setup, please see [the SETUP guide](SETUP.md). +`rffmpeg` does require a little bit more configuration to work properly however. For a comprehensive installation tutorial based on a reference setup, please see [the SETUP guide](docs/SETUP.md). **NOTE** Jellyfin 10.10.x and newer require an additional `TMPDIR` environment variable set to somewhere exported to the remote machine, or these paths will not work properly. Edit your Jellyfin startup/service configuration to set that. See the setup guide for more details. @@ -194,4 +194,6 @@ I'm always happy to help, though please ensure you try to follow the setup guide ### `rffmpeg-go` - forked project +NOTICE: project was archived in Oct 27, 2024. + There's also a [fork of this script written in Go](https://github.com/aleksasiriski/rffmpeg-go) with semver tags and binaries available, as well as docker images for both the [script](https://github.com/aleksasiriski/rffmpeg-go/pkgs/container/rffmpeg-go) and [Jellyfin](https://github.com/aleksasiriski/jellyfin-rffmpeg). diff --git a/docs/HARDENING b/docs/HARDENING new file mode 100644 index 0000000..9043391 --- /dev/null +++ b/docs/HARDENING @@ -0,0 +1,92 @@ + +*NOTICE* Do not do these tasks until you have a verified working solution + +These were tested and validated on Ubuntu 24.04 LTS, 2025-11-03 + +# Hardening + +- Access for jellyfin user will be limited to jellyfin1 server only +- Commands that jellyfin user can run will be limited to ffmpeg only +- Commands run by jellyfin user will be logged + - (optional) Logs stored in separate log file + +## Prerequisites + +- static IP on the jellyfin1 server + +## Configure SSH server + +SSH server configuration is formed out of two files + +1. `10-jellyfin-limits.conf` - SSH config +2. `limited-wrapper.sh` - a script to limit what commands can be run + +### 10-jellyfin-limits.conf + +This config file does few things +- allows only jellyfin user to SSH from jellyfin server +- limits jellyfin user login options to be only from jellyfin server +- limits the commands jellyfin user can run to `limited-wrapper.sh` + +1. Copy `10-jellyfin-limits.conf` to `/etc/ssh/sshd_config.d` +2. Update the IP of the jellyfin server to the file +3. Restart ssh + ```bash + sudo systemctl restart ssh + ``` + +### limited-wrapper.sh + +This file analyses what commands are being run over SSH and limits them +to the ones we defined. + +1. Update the ALLOWED list to match your `ffmpeg` file locations in the script +2. Copy the script to `/usr/local/bin/limited-wrapper.sh` and allow only root to modify it + ```bash + sudo chwon root:root /usr/local/bin/limited-wrapper.sh &&\ + sudo chmod 755 /usr/local/bin/limited-wrapper.sh + ``` +### Test configuration + +1. Login to your jellyfin1 server and run + ```bash + sudo -u jellyfin ssh jellyfin@transcode1 /usr/bin/ffmpeg + ``` + command should succeed and print out ffmpeg info + +2. Run a command that should fail + + ```bash + sudo -u jellyfin ssh jellyfin@transcode1 uname -a + ``` + command should fail and you should see `ERROR: command not allowed.` + + +### Troubleshooting + +#### Permission denied (publickey) + +1. check your auth.log +you should see the IP you are connecting from, make sure it is the same as in your `10-jellyfin-limits.conf` -file. + +## Logging + +All commands run by the jellyfin user are logged to standard syslog (via logger). They can be extracted to their own file. + +### rsyslog config + +File `limited-wrapper-log.conf` creates a rsyslog config to redirect the log entries to a separate file + +1. Update the `limited-wrapper-log.conf` file with the log file name you want. Default is `/var/log/jellyfin_commands.log` +2. Copy the file to /etc/rsyslog.d/ +3. Correct the file rights + ```bash + sudo chown root:root /etc/rsyslog.d/limited-wrapper-log.conf &&\ + sudo chmod 644 /etc/rsyslog.d/limited-wrapper-log.conf + ``` +4. Create the log file + ```bash + sudo touch /var/log/jellyfin_commands.log &&\ + sudo chown syslog:adm /var/log/jellyfin_commands.log &&\ + sudo chmod 664 /var/log/jellyfin_commands.log + ``` \ No newline at end of file diff --git a/SETUP.md b/docs/SETUP.md similarity index 72% rename from SETUP.md rename to docs/SETUP.md index e93b70c..fde0566 100644 --- a/SETUP.md +++ b/docs/SETUP.md @@ -12,20 +12,20 @@ This guide is provided as a basic starting point - there are myriad possible com 1. Make note of the Jellyfin service user's details, specifically the UID and any groups (and GIDs) it is a member of; this will be needed later on. - ``` - jellyfin1 $ id jellyfin - uid=110(jellyfin) gid=117(jellyfin) groups=117(jellyfin) +#### jellyfin1 + ```bash + id jellyfin + # should output + # uid=110(jellyfin) gid=117(jellyfin) groups=117(jellyfin) ``` 1. Make note of the Jellyfin data path; this will be needed later on. By default when using native OS packages, this is `/var/lib/jellyfin`. If you choose to move this directory, do so now (I personally use `/srv/jellyfin` but this guide will assume the default). To make life easier below, you can store this in a variable that I will reference frequently later: - ``` - jellyfin1 $ export jellyfin_data_path="/var/lib/jellyfin" - jellyfin1 $ export jellyfin_cache_path="/var/lib/jellyfin" - transcode1 $ export jellyfin_data_path="/var/lib/jellyfin" - transcode1 $ export jellyfin_cache_path="/var/lib/jellyfin" + ```bash + export jellyfin_data_path="/var/lib/jellyfin" + export jellyfin_cache_path="/var/cache/jellyfin" ``` The important subdirectories for `rffmpeg`'s operation are: @@ -41,20 +41,20 @@ This guide is provided as a basic starting point - there are myriad possible com 1. Create an SSH keypair to use for `rffmpeg`'s login to the remote server. For ease of use with the following steps, use the Jellyfin service user (`jellyfin`) to create the keypair and store it under its home directory (the Jellyfin data path above). I use `rsa` here but you can substitute `ed25519` instead (avoid `dsa` and `ecdsa` for reasons I won't get into here). Once done, copy the public key to `authorized_keys` which will be used to authenticate the key later. - ``` - jellyfin1 $ sudo -u jellyfin mkdir ${jellyfin_data_path}/.ssh - jellyfin1 $ sudo chmod 700 ${jellyfin_data_path}/.ssh - jellyfin1 $ export keytype="rsa" - jellyfin1 $ sudo -u jellyfin ssh-keygen -t ${keytype} -f ${jellyfin_data_path}/.ssh/id_${keytype} - jellyfin1 $ sudo -u jellyfin cp -a ${jellyfin_data_path}/.ssh/id_${keytype}.pub ${jellyfin_data_path}/.ssh/authorized_keys + ```bash + export keytype="rsa" &&\ + sudo -u jellyfin mkdir ${jellyfin_data_path}/.ssh &&\ + sudo chmod 700 ${jellyfin_data_path}/.ssh &&\ + sudo -u jellyfin ssh-keygen -t ${keytype} -f ${jellyfin_data_path}/.ssh/id_${keytype} &&\ + sudo -u jellyfin cp -a ${jellyfin_data_path}/.ssh/id_${keytype}.pub ${jellyfin_data_path}/.ssh/authorized_keys ``` It is important that you do not alter the permissions under this `.ssh` directory or this can cause SSH to fail later. The SSH *must* occur as the `jellyfin` user for this to work. 1. Scan and save the SSH host key of the transcode server(s), to avoid a prompt later: - ``` - jellyfin1 $ ssh-keyscan transcode1 | sudo -u jellyfin tee -a ${jellyfin_data_path}/.ssh/known_hosts + ```bash + ssh-keyscan transcode1 | sudo -u jellyfin tee -a ${jellyfin_data_path}/.ssh/known_hosts ``` * **NOTE:** Ensure you use the exact name here that you will use in `rffmpeg`. If this is an FQDN (e.g. `jellyfin1.mydomain.tld`) or an IP (e.g. `192.168.0.101`) instead of a short name, use that instead in this command, or repeat it for every possible option (it doesn't hurt). @@ -63,37 +63,35 @@ This guide is provided as a basic starting point - there are myriad possible com 1. Install the required Python3 dependencies of `rffmpeg`: - ``` - jellyfin1 $ sudo apt -y install python3-yaml - jellyfin1 $ sudo apt -y install python3-click - jellyfin1 $ sudo apt -y install python3-subprocess + ```bash + sudo apt -y install python3-yaml python3-click python3-subprocess ``` * **NOTE:** On some Ubuntu versions, `python3-subprocess` does not exist, and should instead be part of the Python standard library. Skip installing this package if it can't be found. -1. Clone the `rffmpeg` repository somewhere onto the system, then install the `rffmpeg` binary, make it executable, and prepare symlinks for the command names `ffmpeg` and `ffprobe` to it. I recommend storing these in `/usr/local/bin` for simplicity and so that they are present on the default `$PATH` for most users. +2. Clone the `rffmpeg` repository somewhere onto the system, then install the `rffmpeg` binary, make it executable, and prepare symlinks for the command names `ffmpeg` and `ffprobe` to it. I recommend storing these in `/usr/local/bin` for simplicity and so that they are present on the default `$PATH` for most users. - ``` - jellyfin1 $ git clone https://github.com/joshuaboniface/rffmpeg # or download the files manually - jellyfin1 $ sudo cp rffmpeg/rffmpeg /usr/local/bin/rffmpeg - jellyfin1 $ sudo chmod +x /usr/local/bin/rffmpeg - jellyfin1 $ sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffmpeg - jellyfin1 $ sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffprobe + ```bash + git clone https://github.com/joshuaboniface/rffmpeg # or download the files manually + sudo cp rffmpeg/rffmpeg /usr/local/bin/rffmpeg &&\ + sudo chmod +x /usr/local/bin/rffmpeg &&\ + sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffmpeg &&\ + sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffprobe ``` -1. Optional: Create a directory for the `rffmpeg` configuration at `/etc/rffmpeg`, then copy `rffmpeg.yml.sample` to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. Generally, if you're following this guide exactly, you will not need to install this file or adjust anything in in it. If you do require help though, I require debug logging to be enabled via the configuration file, so it's probably best to get this out of the way when installing `rffmpeg`: +3. Optional: Create a directory for the `rffmpeg` configuration at `/etc/rffmpeg`, then copy `rffmpeg.yml.sample` to `/etc/rffmpeg/rffmpeg.yml` and edit it to suit your needs if required. Generally, if you're following this guide exactly, you will not need to install this file or adjust anything in in it. If you do require help though, I require debug logging to be enabled via the configuration file, so it's probably best to get this out of the way when installing `rffmpeg`: - ``` - jellyfin1 $ sudo mkdir -p /etc/rffmpeg - jellyfin1 $ sudo cp rffmpeg/rffmpeg.yml.sample /etc/rffmpeg/rffmpeg.yml - jellyfin1 $ sudo $EDITOR /etc/rffmpeg/rffmpeg.yml # if required + ```bash + sudo mkdir -p /etc/rffmpeg &&\ + sudo cp rffmpeg/rffmpeg.yml.sample /etc/rffmpeg/rffmpeg.yml &&\ + sudo $EDITOR /etc/rffmpeg/rffmpeg.yml # if required ``` -1. Initialize `rffmpeg` (note the `sudo` command) and add at the target host to it. You can add other hosts now or later, and set weights of hosts, if required; for full details see the [main README](README.md) or run `rffmpeg --help` to view the CLI help menu. +5. Initialize `rffmpeg` (note the `sudo` command) and add at the target host to it. You can add other hosts now or later, and set weights of hosts, if required; for full details see the [main README](../README.md) or run `rffmpeg --help` to view the CLI help menu. - ``` - jellyfin1 $ sudo rffmpeg init --yes - jellyfin1 $ rffmpeg add --weight 1 transcode1 + ```bash + sudo rffmpeg init --yes &&\ + rffmpeg add --weight 1 transcode1 ``` ### NFS Setup @@ -102,11 +100,11 @@ This guide is provided as a basic starting point - there are myriad possible com 1. Install the NFS kernel server. We will use NFS to export the various required directories so the transcode machine can read from and write to them. - ``` - jellyfin1 $ sudo apt -y install nfs-kernel-server + ```bash + sudo apt -y install nfs-kernel-server ``` -1. Create an `/etc/exports` configuration. What to put here can vary a lot, but here are some important points: +2. Create an `/etc/exports` configuration. What to put here can vary a lot, but here are some important points: * Always export the `${jellyfin_data_path}` in full. Advanced users might be able to export the required subdirectories individually, but I find this to be not worth the hassle. * Note the security options of NFS. It will limit mounts to the IP addresses specified. If your home network is secure, you can use the entire network, e.g. `192.168.0.0/24`, but I would recommend determining the exact IP of your transcode server(s) and use them explicitly, e.g. for this example `192.168.0.101` and `192.168.0.102`. @@ -115,7 +113,7 @@ This guide is provided as a basic starting point - there are myriad possible com An example `/etc/exports` file would look like this: - ``` + ```text # /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5). # @@ -129,11 +127,14 @@ This guide is provided as a basic starting point - there are myriad possible com /srv/mymedia 192.168.0.101/32(rw,sync,no_subtree_check,no_root_squash) 192.168.0.102/32(rw,sync,no_subtree_check,no_root_squash) ``` -1. Reload the exports file and ensure the NFS server is properly exporting it now: +3. Reload the exports file and ensure the NFS server is properly exporting it now: + ```bash + sudo exportfs -arfv + sudo exportfs ``` - jellyfin1 $ sudo exportfs -arfv - jellyfin1 $ sudo exportfs + should output something like + ```text /var/lib/jellyfin 192.168.0.101/32 /var/lib/jellyfin 192.168.0.102/32 /var/cache/jellyfin 192.168.0.101/32 @@ -142,57 +143,70 @@ This guide is provided as a basic starting point - there are myriad possible com ## Set up the transcode server (`transcode1`) +setup the temporary convenience variables + +```bash +export jellyfin_data_path="/var/lib/jellyfin" +export jellyfin_cache_path="/var/cache/jellyfin" +``` + 1. Install and configure anything you need for hardware transcoding, if applicable. For example GPU drivers if using a GPU for transcoding. - * **NOTE:** Make sure you understand the caveats of using hardware transcoding with `rffmpeg` from [the main README](README.md#hardware-acceleration). + * **NOTE:** Make sure you understand the caveats of using hardware transcoding with `rffmpeg` from [the main README](../README.md#hardware-acceleration). -1. Install the correct `jellyfin-ffmpeg` package for your version of Jellyfin; check which version is installed on your `jellyfin1` system with `dpkg -l | grep jellyfin-ffmpeg`, then install that version on this host too; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just the `jellyfin-ffmpeg` of the required version. +2. Install the correct `jellyfin-ffmpeg` package for your version of Jellyfin; check which version is installed on your `jellyfin1` system with `dpkg -l | grep jellyfin-ffmpeg`, then install that version on this host too; follow the same steps as you would to install Jellyfin on the media server, only don't install `jellyfin` (and `jellyfin-server`/`jellyfin-web`) itself, just the `jellyfin-ffmpeg` of the required version. + in jellyfin1 + ```bash + dpkg -l | grep jellyfin-ffmpeg + # ii jellyfin-ffmpeg6 6.0.1-8-bookworm amd64 Tools for transcoding, streaming and playing of multimedia files ``` - jellyfin1 $ dpkg -l | grep jellyfin-ffmpeg - ii jellyfin-ffmpeg6 6.0.1-8-bookworm amd64 Tools for transcoding, streaming and playing of multimedia files - transcode1 $ sudo apt -y install curl gnupg - transcode1 $ curl -fsSL https://repo.jellyfin.org/ubuntu/jellyfin_team.gpg.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/jellyfin.gpg - transcode1 $ echo "deb [arch=$( dpkg --print-architecture )] https://repo.jellyfin.org/$( awk -F'=' '/^ID=/{ print $NF }' /etc/os-release ) $( awk -F'=' '/^VERSION_CODENAME=/{ print $NF }' /etc/os-release ) main" | sudo tee /etc/apt/sources.list.d/jellyfin.list - transcode1 $ sudo apt update - transcode1 $ sudo apt install jellyfin-ffmpeg6 + in transcode1 + ```bash + sudo apt -y install curl gnupg &&\ + curl -fsSL https://repo.jellyfin.org/ubuntu/jellyfin_team.gpg.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/jellyfin.gpg &&\ + echo "deb [arch=$( dpkg --print-architecture )] https://repo.jellyfin.org/$( awk -F'=' '/^ID=/{ print $NF }' /etc/os-release ) $( awk -F'=' '/^VERSION_CODENAME=/{ print $NF }' /etc/os-release ) main" | sudo tee /etc/apt/sources.list.d/jellyfin.list &&\ + sudo apt update &&\ + sudo apt install -y jellyfin-ffmpeg6 ``` -1. Install the NFS client utilities: +3. Install the NFS client utilities: - ``` - transcode1 $ sudo apt install -y nfs-common + ```bash + sudo apt install -y nfs-common ``` -1. Create the Jellyfin service user and its default group; ensure you use the exact same UID and GID values you found in the beginning of the last section and adjust the example here to match yours: +4. Create the Jellyfin service user and its default group; ensure you use the exact same UID and GID values you found in the beginning of the last section and adjust the example here to match yours: - ``` - transcode1 $ sudo groupadd --gid 117 jellyfin - transcode1 $ sudo useradd --uid 110 --gid jellyfin --shell /bin/bash --no-create-home --home-dir ${jellyfin_data_path} jellyfin + ```bash + sudo groupadd --gid 117 jellyfin &&\ + sudo useradd --uid 110 --gid jellyfin --shell /bin/bash --no-create-home --home-dir ${jellyfin_data_path} jellyfin ``` * **NOTE:** For some hardware acceleration, you might need to add this user to additional groups. For example `--groups video,render`. * **NOTE:** The UID and GIDs here are dynamic; on the `jellyfin1` machine, they would have been selected automatically at install time with the next available ID in the range 100-199 (at least in Debian/Ubuntu). However, this means that the exact UID of your Jellyfin service user might not be available on your transcode server, depending on what packages are installed and in what order. If there is a conflict, you must adjust user IDs on one side or the other so that they match on both machines. You can use `sudo usermod` to change a user's ID if required. -1. Create the Jellyfin data directory at the same location as on the media server, and set it immutable so that it won't be written to if the NFS mount goes down: +5. Create the Jellyfin directories at the same location as on the media server, and set it immutable so that it won't be written to if the NFS mount goes down: - ``` - transcode1 $ sudo mkdir ${jellyfin_data_path} - transcode1 $ sudo chattr +i ${jellyfin_data_path} + ```bash + for file in ${jellyfin_data_path} ${jellyfin_cache_path}; do + sudo mkdir ${file} &&\ + sudo chattr +i ${file} + done ``` * **NOTE:** Don't worry about permissions here; the mount will set those. -1. Create the NFS client mount. There are two main ways to do this: +6. Create the NFS client mount. There are two main ways to do this: * Use the traditional `/etc/fstab` by adding a new entry like so, replacing the paths and hostname as required, and then mounting it: - ``` - transcode1 $ echo "jellyfin1:${jellyfin_data_path} ${jellyfin_data_path} nfs defaults,vers=3,sync" | sudo tee -a /etc/fstab - transcode1 $ echo "jellyfin1:${jellyfin_cache_path} ${jellyfin_cache_path} nfs defaults,vers=3,sync" | sudo tee -a /etc/fstab - transcode1 $ sudo mount ${jellyfin_data_path} - transcode1 $ sudo mount ${jellyfin_cache_path} + ```bash + echo "jellyfin1:${jellyfin_data_path} ${jellyfin_data_path} nfs defaults,vers=3,sync" | sudo tee -a /etc/fstab &&\ + echo "jellyfin1:${jellyfin_cache_path} ${jellyfin_cache_path} nfs defaults,vers=3,sync" | sudo tee -a /etc/fstab &&\ + sudo mount ${jellyfin_data_path} &&\ + sudo mount ${jellyfin_cache_path} ``` * Use a SystemD `mount` unit, which is a newer way of doing mounts with SystemD. I personally prefer this method as I find it easier to set up automatically, but this is up to preference. An example based on mine would be: @@ -233,38 +247,38 @@ This guide is provided as a basic starting point - there are myriad possible com Once the unit file is created, you can then reload the unit list and mount it: - ``` - transcode1 $ sudo systemctl daemon-reload - transcode1 $ sudo systemctl enable --now var-lib-jellyfin.mount - transcode1 $ sudo systemctl enable --now var-cache-jellyfin.mount + ```bash + sudo systemctl daemon-reload &&\ + sudo systemctl enable --now var-lib-jellyfin.mount &&\ + sudo systemctl enable --now var-cache-jellyfin.mount ``` Note that mount units are fairly "new" and can be a bit finicky, be sure to read the SystemD documentation if you get stuck! Generally for new users, I'd recommend the `/etc/fstab` method instead. **NOTE:** Don't forget about `actimeo=1` here if you need it! -1. Mount your media directories in the **same location(s)** as on the media server. If you exported them via NFS from your media server, use the process above only for those directories instead. +7. Mount your media directories in the **same location(s)** as on the media server. If you exported them via NFS from your media server, use the process above only for those directories instead. ## Test the setup 1. On the media server, verify that SSH as the Jellyfin service user is working as expected to each transcoding server: - ``` - jellyfin1 $ sudo -u jellyfin ssh -i ${jellyfin_data_path}/.ssh/id_rsa jellyfin@transcode1 uname -a - Linux transcode1 [...] + ```bash + sudo -u jellyfin ssh -i ${jellyfin_data_path}/.ssh/id_rsa jellyfin@transcode1 uname -a + # Linux transcode1 [...] ``` 1. Validate that `rffmpeg` itself is working by calling its `ffmpeg` and `ffprobe` aliases with the `-version` option: - ``` - jellyfin1 $ sudo -u jellyfin /usr/local/bin/ffmpeg -version - ffmpeg version 5.0.1-Jellyfin Copyright (c) 2000-2022 the FFmpeg developers - built with gcc 10 (Debian 10.2.1-6) - [...] - jellyfin1 $ sudo -u jellyfin /usr/local/bin/ffprobe -version - ffprobe version 5.0.1-Jellyfin Copyright (c) 2007-2022 the FFmpeg developers - built with gcc 10 (Debian 10.2.1-6) - [...] + ```bash + sudo -u jellyfin /usr/local/bin/ffmpeg -version + # ffmpeg version 5.0.1-Jellyfin Copyright (c) 2000-2022 the FFmpeg developers + # built with gcc 10 (Debian 10.2.1-6) + # [...] + sudo -u jellyfin /usr/local/bin/ffprobe -version + # ffprobe version 5.0.1-Jellyfin Copyright (c) 2007-2022 the FFmpeg developers + # built with gcc 10 (Debian 10.2.1-6) + # [...] ``` As long as these steps work, all further steps should as well. If one of these *doesn't* work, double-check all previous steps and confirm that everything is set up right. @@ -275,8 +289,8 @@ As long as these steps work, all further steps should as well. If one of these * 1. On the `jellyfin1` system, edit `/etc/default/jellyfin`: - ``` - jellyfin1 $ sudo $EDITOR /etc/default/jellyfin + ```bash + sudo $EDITOR /etc/default/jellyfin ``` 1. Change the value of `JELLYFIN_FFMPEG_OPT` to be `--ffmpeg=/usr/local/bin/ffmpeg` (the `rffmpeg` alias name `ffmpeg` in whatever path you installed `rffmpeg` to). @@ -285,8 +299,8 @@ As long as these steps work, all further steps should as well. If one of these * 1. Save the file and restart Jellyfin: - ``` - jellyfin1 $ sudo systemctl restart jellyfin + ```bash + sudo systemctl restart jellyfin ``` If you wish to use hardware transcoding, you must also enable it in Jellyfin's WebUI: @@ -303,15 +317,19 @@ Now, run `rffmpeg log -f` on the `jellyfin1` machine and try to play a video tha If you are using NVEnv/NVDec, you will need to symlink the `.nv` folder inside the Jellyfin user's homedir (i.e. `/var/lib/jellyfin/.nv`) to somewhere outside of the NFS volume on both the Jellyfin and transcoding hosts. For example: +on jellyfin1 + ```bash + sudo mv /var/lib/jellyfin/.nv /var/lib/nvidia-cache # or "sudo mkdir /var/lib/nvidia-cache" and "sudo chown jellyfin /var/lib/nvidia-cache" if it does not yet exist + sudo ln -s /var/lib/nvidia-cache /var/lib/jellyfin/.nv ``` - jellyfin1 $ sudo mv /var/lib/jellyfin/.nv /var/lib/nvidia-cache # or "sudo mkdir /var/lib/nvidia-cache" and "sudo chown jellyfin /var/lib/nvidia-cache" if it does not yet exist - jellyfin1 $ sudo ln -s /var/lib/nvidia-cache /var/lib/jellyfin/.nv - transcode1 $ sudo mkdir /var/lib/nvidia-cache - transcode1 $ sudo chown jellyfin /var/lib/nvidia-cache - transcode1 $ ls -alh /var/lib/jellyfin - [...] - lrwxrwxrwx 1 root root 17 Jun 11 15:51 .nv -> /var/lib/nvidia-cache - [...] + on transcode1 + ```bash + sudo mkdir /var/lib/nvidia-cache + sudo chown jellyfin /var/lib/nvidia-cache + ls -alh /var/lib/jellyfin + #[...] + #lrwxrwxrwx 1 root root 17 Jun 11 15:51 .nv -> /var/lib/nvidia-cache + #[...] ``` Be sure to adjust these paths to match your Jellyfin setup. The name of the target doesn't matter too much, as long as `.nv` inside the homedir is symlinked to it and it is owned by the `jellyfin` service user. diff --git a/hardening/10-jellyfin-limits.conf b/hardening/10-jellyfin-limits.conf new file mode 100644 index 0000000..06254e9 --- /dev/null +++ b/hardening/10-jellyfin-limits.conf @@ -0,0 +1,13 @@ +# Limit jellyfin access +# IPJELLYFIN is our Jellyfin server + +Match Address IPJELLYFIN + AllowUsers jellyfin@IPJELLYFIN + +Match User jellyfin, Address IPJELLYFIN + AllowUsers jellyfin@IPJELLYFIN + ForceCommand /usr/local/bin/limited-wrapper.sh + PermitTTY no + X11Forwarding no + AllowAgentForwarding no + AllowTcpForwarding no \ No newline at end of file diff --git a/hardening/limited-wrapper-log.conf b/hardening/limited-wrapper-log.conf new file mode 100644 index 0000000..ca19f51 --- /dev/null +++ b/hardening/limited-wrapper-log.conf @@ -0,0 +1,3 @@ +# Match the tag *including* the trailing colon +:syslogtag, startswith, "limited-wrapper.sh" /var/log/jellyfin_commands.log +& stop \ No newline at end of file diff --git a/hardening/limited-wrapper.sh b/hardening/limited-wrapper.sh new file mode 100755 index 0000000..26cb603 --- /dev/null +++ b/hardening/limited-wrapper.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +set -euo pipefail # safer defaults + +# Author: Juha Leivo +# Version: 1.1.0 +# Date: 2025-11-03 +# +# Prevent unauthorized SSH command execution by allowing only a limited set of binaries. +# +# History +# 1.0.0 - 2025-11-02, initial version +# 1.1.0 - 2025-11-03, moved to use logging 1.0.0 + +# Function to log messages both to TTY and to a logfile in syslog format +# Ref logging.sh version 1.0.0 +log_msg() { + local level="$1" + shift + # Concatenate all arguments into a single string + local msg="$*" + + # Map level to syslog priority + local prio="notice" + case "$level" in + INFO) prio="info" ;; + WARN) prio="warning" ;; + ERROR) prio="err" ;; + DEBUG) prio="debug" ;; + *) prio="notice" + msg="$level $msg" ;; + esac + + if [ -t 1 ]; then + # Interactive TTY: print plain message without level prefix + echo "$msg" + else + # Non‑interactive: send to syslog + logger -p user.$prio -t "$(basename "$0")" "$level $msg" + fi +} + +log_debug() { log_msg DEBUG "$@"; } +log_info() { log_msg INFO "$@"; } +log_warn() { log_msg WARN "$@"; } +log_error() { log_msg ERROR "$@"; } +# ------------------------------------------------------------------ +# Whitelist of absolute paths to allowed binaries +ALLOWED=( + /usr/bin/ffmpeg + /usr/bin/ffprobe + /usr/local/bin/ffmpeg + /usr/local/bin/ffprobe + /usr/lib/jellyfin-ffmpeg/ffmpeg + /usr/lib/jellyfin-ffmpeg/ffprobe +) + +# ------------------------------------------------------------------ +REQ_CMD="${SSH_ORIGINAL_COMMAND:-}" +if [[ -z "$REQ_CMD" ]]; then + echo "You may run only: ${ALLOWED[*]}" + exit 0 +fi + +# Split the command into an array preserving quoting +read -r -a ARGS <<<"$REQ_CMD" +BIN="${ARGS[0]}" + +# Resolve symlinks if possible +if command -v realpath >/dev/null; then + BIN=$(realpath -m "$BIN") +else + BIN=$(readlink -f "$BIN" 2>/dev/null || echo "$BIN") +fi + +log_debug "Checking for bin $BIN" + +# Whitelist check +for ok in "${ALLOWED[@]}"; do + if [[ "$BIN" == "$ok" ]]; then + log_info "Running command $REQ_CMD" + eval "exec $REQ_CMD" + fi +done + +log_error "Not allowed $REQ_CMD" +echo "ERROR: command not allowed." # For SSH to show the error on client +exit 1 From f5681397a7c6c449e691725b77b6ebc9813251bf Mon Sep 17 00:00:00 2001 From: Juha Leivo Date: Mon, 3 Nov 2025 21:33:26 +0200 Subject: [PATCH 115/115] feat: Added python version of the wrapper - GPT-OSS:120b did the converson of the script from bash to python3 - Updated rsyslog definition to work with either bash or python -file - updated SSH config to use the python version --- docs/HARDENING | 12 +-- hardening/10-jellyfin-limits.conf | 2 +- hardening/limited-wrapper-log.conf | 2 +- hardening/limited-wrapper.py | 152 +++++++++++++++++++++++++++++ 4 files changed, 160 insertions(+), 8 deletions(-) create mode 100644 hardening/limited-wrapper.py diff --git a/docs/HARDENING b/docs/HARDENING index 9043391..ad7d24d 100644 --- a/docs/HARDENING +++ b/docs/HARDENING @@ -19,14 +19,14 @@ These were tested and validated on Ubuntu 24.04 LTS, 2025-11-03 SSH server configuration is formed out of two files 1. `10-jellyfin-limits.conf` - SSH config -2. `limited-wrapper.sh` - a script to limit what commands can be run +2. `limited-wrapper.sh` or `limited-wrapper.py` - a script to limit what commands can be run ### 10-jellyfin-limits.conf This config file does few things - allows only jellyfin user to SSH from jellyfin server - limits jellyfin user login options to be only from jellyfin server -- limits the commands jellyfin user can run to `limited-wrapper.sh` +- limits the commands jellyfin user can run to `limited-wrapper.py` 1. Copy `10-jellyfin-limits.conf` to `/etc/ssh/sshd_config.d` 2. Update the IP of the jellyfin server to the file @@ -35,16 +35,16 @@ This config file does few things sudo systemctl restart ssh ``` -### limited-wrapper.sh +### limited-wrapper.sh and limited-wrapper.py This file analyses what commands are being run over SSH and limits them to the ones we defined. 1. Update the ALLOWED list to match your `ffmpeg` file locations in the script -2. Copy the script to `/usr/local/bin/limited-wrapper.sh` and allow only root to modify it +2. Copy the script to `/usr/local/bin/limited-wrapper.py` and allow only root to modify it ```bash - sudo chwon root:root /usr/local/bin/limited-wrapper.sh &&\ - sudo chmod 755 /usr/local/bin/limited-wrapper.sh + sudo chwon root:root /usr/local/bin/limited-wrapper.py &&\ + sudo chmod 755 /usr/local/bin/limited-wrapper.py ``` ### Test configuration diff --git a/hardening/10-jellyfin-limits.conf b/hardening/10-jellyfin-limits.conf index 06254e9..f431dea 100644 --- a/hardening/10-jellyfin-limits.conf +++ b/hardening/10-jellyfin-limits.conf @@ -6,7 +6,7 @@ Match Address IPJELLYFIN Match User jellyfin, Address IPJELLYFIN AllowUsers jellyfin@IPJELLYFIN - ForceCommand /usr/local/bin/limited-wrapper.sh + ForceCommand /usr/local/bin/limited-wrapper.py PermitTTY no X11Forwarding no AllowAgentForwarding no diff --git a/hardening/limited-wrapper-log.conf b/hardening/limited-wrapper-log.conf index ca19f51..ff812e2 100644 --- a/hardening/limited-wrapper-log.conf +++ b/hardening/limited-wrapper-log.conf @@ -1,3 +1,3 @@ # Match the tag *including* the trailing colon -:syslogtag, startswith, "limited-wrapper.sh" /var/log/jellyfin_commands.log +:syslogtag, startswith, "limited-wrapper" /var/log/jellyfin_commands.log & stop \ No newline at end of file diff --git a/hardening/limited-wrapper.py b/hardening/limited-wrapper.py new file mode 100644 index 0000000..5e89c97 --- /dev/null +++ b/hardening/limited-wrapper.py @@ -0,0 +1,152 @@ +#!/usr/bin/env python3 +"""limited-wrapper.py + +Author: GPT-OSS:120b +Version: 1.1.0 +Date: 2025-11-03 + +Python 3 implementation of the limited-wrapper.sh script. +It restricts SSH command execution to a whitelist of allowed binaries +and logs activity either to the console (interactive) or to syslog. + +History + 1.0.0 - 2025-11-03, initial version + +""" + +import os +import sys +import shlex +import logging +import logging.handlers +from typing import List + +# --------------------------------------------------------------------------- +# Logging utilities +# --------------------------------------------------------------------------- + +def _setup_logger() -> logging.Logger: + logger = logging.getLogger("limited-wrapper.py") + logger.setLevel(logging.DEBUG) # Capture all levels; handlers will filter + # Ensure no duplicate handlers if the module is reloaded + logger.handlers.clear() + + if sys.stdout.isatty(): + # Interactive TTY – simple console output without timestamp or level prefix + console = logging.StreamHandler(sys.stdout) + console.setLevel(logging.INFO) + console.setFormatter(logging.Formatter("%(message)s")) + logger.addHandler(console) + else: + # Non‑interactive – forward to syslog. Let syslog generate its own timestamp, + # hostname, and program identifier (the logger name). No extra formatter is + # needed to avoid adding the PID or duplicate timestamps. + try: + syslog = logging.handlers.SysLogHandler(address="/dev/log") + except OSError: + # Fallback for systems without /dev/log (e.g., macOS) + syslog = logging.handlers.SysLogHandler(address=("localhost", 514)) + syslog.setLevel(logging.DEBUG) + # Prefix with logger name (script tag) to match original format + syslog.setFormatter(logging.Formatter("%(name)s: %(message)s")) + logger.addHandler(syslog) + return logger + +_logger = _setup_logger() + + +def log_msg(level: str, *msg: str) -> None: + """Log a message with an explicit level prefix. + + The original Bash implementation prefixed the log line with the level + (e.g. ``DEBUG`` or ``INFO``) before sending it to syslog. To preserve that + format we construct ``full_msg = f"{level.upper()} {text}"`` and log the + resulting string. This ensures syslog entries look like: + ``limited-wrapper.sh: DEBUG `` while interactive console output + remains readable. + """ + text = " ".join(msg) + level = level.upper() + full_msg = f"{level} {text}" + if level == "DEBUG": + _logger.debug(full_msg) + elif level == "INFO": + _logger.info(full_msg) + elif level in ("WARN", "WARNING"): + _logger.warning(full_msg) + elif level == "ERROR": + _logger.error(full_msg) + else: + _logger.info(full_msg) + + +def log_debug(*msg: str) -> None: + log_msg("DEBUG", *msg) + + +def log_info(*msg: str) -> None: + log_msg("INFO", *msg) + + +def log_warn(*msg: str) -> None: + log_msg("WARN", *msg) + + +def log_error(*msg: str) -> None: + log_msg("ERROR", *msg) + +# --------------------------------------------------------------------------- +# Whitelist of absolute paths to allowed binaries +# --------------------------------------------------------------------------- +ALLOWED: List[str] = [ + "/usr/bin/ffmpeg", + "/usr/bin/ffprobe", + "/usr/local/bin/ffmpeg", + "/usr/local/bin/ffprobe", + "/usr/lib/jellyfin-ffmpeg/ffmpeg", + "/usr/lib/jellyfin-ffmpeg/ffprobe", +] + + +def main() -> None: + req_cmd = os.getenv("SSH_ORIGINAL_COMMAND", "") + if not req_cmd: + # No command supplied – show the whitelist and exit successfully + print("You may run only: " + " ".join(ALLOWED)) + sys.exit(0) + + # Parse the command string respecting shell quoting (handles spaces in arguments) + # Using shlex.split provides proper handling of quoted arguments, unlike the + # original bash script which split on whitespace only. + try: + args = shlex.split(req_cmd, posix=True) + except ValueError as e: + log_error(f"Failed to parse SSH_ORIGINAL_COMMAND: {e}") + print("ERROR: could not parse command.") + sys.exit(1) + + if not args: + log_error("Empty command after parsing.") + print("ERROR: empty command.") + sys.exit(1) + + bin_path = os.path.realpath(args[0]) + log_debug(f"Checking for bin {bin_path}") + + if bin_path in ALLOWED: + log_info(f"Running command {req_cmd}") + # Ensure the argument list uses the resolved binary path as argv[0] + args[0] = bin_path + # Replace the current process with the requested command without PATH lookup + os.execv(bin_path, args) + # execv only returns on failure + log_error(f"Failed to exec {req_cmd}") + sys.exit(1) + else: + log_error(f"Not allowed {req_cmd}") + print("ERROR: command not allowed.") + sys.exit(1) + + +if __name__ == "__main__": + main()