From 1cf6de3140f152244a06f69731d1e3bc7773e023 Mon Sep 17 00:00:00 2001 From: Andrei Vagin Date: Sat, 7 Mar 2026 22:46:01 +0000 Subject: [PATCH] docs: update Service Descriptors documentation - Detail the top-down allocation strategy using RLIMIT_NOFILE - Explain per-process isolation (service_fd_id) for shared FD tables - Document the relocation mechanism (F_DUPFD_CLOEXEC, dup3) - Describe the 'sfds_protected' flag and safety invariants - List common Service FD types (LOG, IMG, RPC, TRANSPORT, etc.) Signed-off-by: Andrei Vagin --- .../under-the-hood/service-descriptors.md | 38 ++++++++++++++++--- 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/Documentation/under-the-hood/service-descriptors.md b/Documentation/under-the-hood/service-descriptors.md index 8f7d06fb4..a2ddbbdc6 100644 --- a/Documentation/under-the-hood/service-descriptors.md +++ b/Documentation/under-the-hood/service-descriptors.md @@ -1,11 +1,37 @@ -# Service Descriptors +# Service Descriptors (Service FDs) -Service descriptors are file descriptors (FDs) used by CRIU to facilitate the checkpoint and restore processes. +During dump and restore operations, CRIU requires numerous internal file descriptors (FDs) to manage logs, images, RPC communication, and transport sockets. Because the application being checkpointed or restored may use any arbitrary FD number, CRIU must ensure its internal descriptors never conflict with those of the application. To achieve this, CRIU uses the **Service FD Engine**. -Only files that are used frequently by CRIU and are difficult to obtain by other means should be maintained as service FDs. Because these FDs cause the file table to grow, they can result in higher memory usage after restoration compared to the initial dump. Other files used during restoration should generally be placed in the `fdstore` instead. Note that not all members of the `sfd_type` enum in `criu/include/servicefd.h` are required to be present at all times; some may be moved to the `fdstore` when appropriate. +## The Protected Range -## Restore Process +CRIU avoids FD collisions by placing its internal descriptors in a "protected" range at the very top of the process's file descriptor table. -CRIU attempts to assign service FDs to the lowest possible numbers to minimize memory overhead. The `service_fd_base` variable tracks the highest service FD number and is determined by `choose_service_fd_base()` based on the maximum file descriptor used by the task. +* **Lifting Limits**: Upon startup, CRIU attempts to lift its `RLIMIT_NOFILE` resource limit (using `rlimit_unlimit_nofile()`) to a very high value (typically 1,048,576 or higher). +* **Top-Down Allocation**: Service FDs are allocated starting from the maximum allowed FD number and working downwards. This strategy places them as far as possible from the range typically used by applications (which usually start from 0 and work upwards). -Because standard files opened during restoration may have descriptors higher than `service_fd_base`, certain sections of code are marked where service FDs must not be modified. These areas are protected via the `sfds_protected` variable; if an unauthorized modification to a service FD is detected in these sections, the restoration process is aborted. +## Service FD Engine Mechanisms + +The engine (`criu/servicefd.c`) provides a robust abstraction for managing these descriptors through several key techniques: + +### 1. Per-Process Isolation in Shared Tables +In scenarios where multiple processes share the same file descriptor table (e.g., threads or processes created with `CLONE_FILES`), CRIU assigns a unique `service_fd_id` to each task. The engine uses this ID to offset the service FD range, ensuring that even tasks sharing an FD table have distinct, non-overlapping slots for their internal CRIU descriptors. + +### 2. Descriptor Relocation +When CRIU opens a file for its own use (such as an image file or the log), the kernel initially assigns it the lowest available FD number (e.g., FD 3). CRIU then uses `fcntl(F_DUPFD_CLOEXEC)` or `dup3()` to "move" that descriptor to its designated high-range slot and immediately closes the original low-numbered descriptor. + +### 3. Protection Flags and Safety +During critical phases of restoration—specifically when the application's FDs are being "planted" into their final numeric slots—CRIU sets a global `sfds_protected` flag. While this flag is set, the service FD engine is "locked." Any attempt by the code to modify or close a service descriptor will trigger an immediate safety crash (BUG), preventing accidental corruption of the restoration state. + +## Common Service FD Types + +The engine manages various types of descriptors, each with a specific role: +* **LOG_FD**: The descriptor for the main CRIU log file. +* **IMG_FD**: The descriptor used for accessing image files. +* **RPC_SK**: The socket used for RPC communication with external management tools. +* **TRANSPORT_FD**: Sockets used to "send" and "receive" FDs between processes via `SCM_RIGHTS`. +* **PROC_FD**: A stable handle to the `/proc` filesystem. +* **CGROUP_YARD**: A descriptor for the temporary directory used during cgroup restoration. + +## See also +* [Dumping File Descriptors](dumping-files.md) +* [Descriptor Assignment](how-to-assign-needed-file-descriptor-to-a-file.md)