mirror of
https://github.com/checkpoint-restore/criu.git
synced 2026-07-17 16:47:50 +00:00
docs: update Checkpoint/Restore architecture documentation
- Clarify freezing mechanisms (PTRACE_INTERRUPT, Freezer CGroup) - Detail the parasite injection and bootstrap process - Explain the role of the restorer blob as a PIE and its conflict avoidance - Document the final transition via sigreturn Signed-off-by: Andrei Vagin <avagin@google.com>
This commit is contained in:
Andrei Vagin
parent
76c5e865c3
commit
ec36774aea
1 changed files with 38 additions and 49 deletions
|
|
@ -1,68 +1,57 @@
|
|||
# Checkpoint/Restore
|
||||
# Checkpoint and Restore Architecture
|
||||
|
||||
This page describes the overall design of the Checkpoint and Restore mechanisms in CRIU.
|
||||
This page describes the high-level design and internal mechanics of the Checkpoint and Restore processes in CRIU.
|
||||
|
||||
## Checkpoint
|
||||
|
||||
The checkpoint procedure relies heavily on the **/proc** filesystem, which is the primary source of information for CRIU.
|
||||
The information gathered from `/proc` includes:
|
||||
The checkpoint procedure captures the full state of a process tree. It combines information from the Linux kernel's `/proc` filesystem with data extracted directly from the processes' address space.
|
||||
|
||||
- File descriptor information (via **/proc/$pid/fd** and **/proc/$pid/fdinfo**).
|
||||
- Pipe parameters.
|
||||
- Memory maps (via **/proc/$pid/maps** and **/proc/$pid/map_files/**).
|
||||
- And more.
|
||||
### 1. Freezing the Process Tree
|
||||
CRIU begins by identifying the process group leader (via the `--tree` option) and recursively collecting all threads and children. To ensure a consistent snapshot, the entire tree must be "frozen."
|
||||
|
||||
The process dumper (hereafter referred to as the "dumper") performs the following steps during the checkpoint stage.
|
||||
* **ptrace**: CRIU uses `PTRACE_SEIZE` followed by `PTRACE_INTERRUPT` to stop tasks without delivering signals that could be visible to the application.
|
||||
* **Freezer CGroup**: Alternatively, the [Freezer CGroup](freezing-the-tree.md) can be used to freeze all tasks in a single operation.
|
||||
|
||||
### Collecting and Freezing the Process Tree
|
||||
The PID of the process group leader is obtained from the command line (`--tree` option). Using this PID, the dumper traverses the **/proc/$pid/task/** directory to collect threads and **/proc/$pid/task/$tid/children** to gather child processes recursively. During this traversal, tasks are stopped using the `ptrace` `PTRACE_SEIZE` command.
|
||||
### 2. Resource Collection (External State)
|
||||
CRIU gathers state that the kernel exposes via `/proc`:
|
||||
* **File Descriptors**: Parsed from `/proc/$pid/fdinfo` (which includes positions and flags).
|
||||
* **Memory Maps**: Captured from `/proc/$pid/smaps` and `/proc/$pid/map_files`.
|
||||
* **Core State**: Task statistics and basic identifiers from `/proc/$pid/stat`.
|
||||
|
||||
*See also: [Freezing the tree](freezing-the-tree.md)*
|
||||
### 3. Parasite Injection (Internal State)
|
||||
Some state (like memory contents and specific credentials) can only be captured from within the process. CRIU uses a technique called **parasite injection**:
|
||||
1. **Infection**: CRIU uses `ptrace` to inject a small bit of code into the task's instruction stream (at the current `CS:IP`).
|
||||
2. **Bootstrap**: This code executes an `mmap` syscall to allocate space for the full **parasite blob**.
|
||||
3. **Execution**: The parasite code runs as a daemon inside the task, communicating with the CRIU coordinator via a Unix socket to dump memory pages and other internal metadata.
|
||||
|
||||
### Collecting and Dumping Task Resources
|
||||
In this step, CRIU reads all available information about the collected tasks and writes it to dump files. Resources are obtained as follows:
|
||||
### 4. Cleanup
|
||||
Once the state is captured, CRIU uses `ptrace` to remove the parasite code and restore the original instructions. The processes are then either resumed or killed, depending on the command-line options.
|
||||
|
||||
1. Virtual Memory Areas (VMAs) are parsed from **/proc/$pid/smaps**, and mapped files are identified via **/proc/$pid/map_files** links.
|
||||
1. File descriptor numbers are retrieved from **/proc/$pid/fd**.
|
||||
1. Core task parameters, such as registers, are dumped using the `ptrace` interface and by parsing the **/proc/$pid/stat** entry.
|
||||
|
||||
CRIU then injects [parasite code](parasite-code.md) into the task via the `ptrace` interface. This occurs in two stages: first, a few bytes are injected at the task's current `CS:IP` to execute an `mmap` syscall. Once `ptrace` runs this syscall, enough memory is allocated for the full parasite code. The parasite code is then copied into the dumpee's address space, and `CS:IP` is updated to point to it.
|
||||
|
||||
From the parasite's context, CRIU gathers additional information, such as:
|
||||
1. Credentials
|
||||
1. Memory contents
|
||||
|
||||
### Cleanup
|
||||
|
||||
Once all resources (such as memory pages, which can only be written from within the dumpee's address space) have been dumped, CRIU uses the `ptrace` facility again to "cure" the dumpee by removing the parasite code and restoring the original instructions. CRIU then detaches from the tasks, allowing them to resume operation.
|
||||
---
|
||||
|
||||
## Restore
|
||||
|
||||
The restoration procedure (the "restorer") involves CRIU morphing itself into the tasks it is restoring. This process consists of four high-level steps:
|
||||
The restore procedure is essentially the reverse of a checkpoint. CRIU "morphs" itself into the process tree it is restoring through a multi-stage process.
|
||||
|
||||
### Resolving Shared Resources
|
||||
### 1. Resolve Shared Resources
|
||||
CRIU analyzes the image files to identify resources shared between processes (e.g., shared memory segments, pipes, or inherited file descriptors). It determines which process will "create" the resource and how others will "inherit" it.
|
||||
|
||||
CRIU reads the image files to identify which processes share specific resources. Shared resources are typically restored by one process; others either inherit them during the second stage (e.g., sessions) or obtain them through other means. Examples include shared files sent via UNIX sockets with `SCM_RIGHTS` messages or shared memory areas restored via `memfd` file descriptors.
|
||||
### 2. Fork the Process Tree
|
||||
CRIU calls `fork()` repeatedly to recreate the original process hierarchy. To restore specific PIDs, it uses the `ns_last_pid` interface or the `clone3` system call. At this stage, only process leaders are created; threads are restored later.
|
||||
|
||||
### Forking the Process Tree
|
||||
### 3. Restore Basic Resources
|
||||
Each process in the new tree begins restoring its environment:
|
||||
* **Namespaces**: Joins or creates Network, Mount, UTS, and IPC namespaces.
|
||||
* **Files and Sockets**: Reopens file descriptors and recreates network sockets.
|
||||
* **Memory Prep**: Maps anonymous memory regions and fills them with data from the images.
|
||||
|
||||
CRIU calls `fork()` repeatedly to recreate the process tree. Note that threads are not restored at this stage, but rather in the fourth step.
|
||||
### 4. The Restorer Context
|
||||
To restore the final memory layout, CRIU must unmap its own code and data. This requires a **restorer blob**:
|
||||
* **Self-Contained**: The blob is a Position-Independent Executable (PIE) that contains all necessary logic to perform the final `mmap` and `munmap` calls.
|
||||
* **Non-Conflicting**: It is mapped into a "hole" in the task's address space that does not conflict with either CRIU's current mappings or the task's original mappings.
|
||||
* **Final Transition**: The process jumps into the restorer blob, which unmaps CRIU, maps the final memory regions, restores timers and credentials, and recreates any additional threads.
|
||||
|
||||
### Restoring Basic Task Resources
|
||||
### 5. Sigreturn
|
||||
The very last step of the restorer is to call `sigreturn`. CRIU prepares a special signal frame on the stack that contains the original register state (including the instruction pointer) of the process at the time of the checkpoint. The `sigreturn` syscall tells the kernel to load this state and resume execution of the application code.
|
||||
|
||||
In this stage, CRIU restores most resources, excluding:
|
||||
|
||||
1. Exact memory mapping locations
|
||||
1. Timers
|
||||
1. Credentials
|
||||
1. Threads
|
||||
|
||||
The restoration of these four resource types is delayed until the final stage for the reasons described below. During this stage, CRIU opens files, prepares [namespaces](namespaces.md), maps and populates private memory areas, creates sockets, and performs operations like `chdir()` and `chroot()`.
|
||||
|
||||
### Switching to the Restorer Context, Finalizing Restoration, and Resuming
|
||||
|
||||
The restorer blob is necessary because as CRIU morphs into the target process, it must unmap its own memory and map the target's. Some code must remain in memory to perform these `munmap` and `mmap` operations. The restorer blob is a small piece of code positioned so that it does not intersect with either CRIU's or the target's memory mappings. At the end of the third stage, CRIU jumps into this blob to restore the memory maps.
|
||||
|
||||
In this context, timers are restored (to prevent them from firing prematurely), credentials are set (allowing privileged operations like `fork_with_pid`), and threads are recreated to avoid issues with memory layout changes.
|
||||
|
||||
*See also: [restorer context](restorer-context.md), [tree after restore](tree-after-restore.md).*
|
||||
*See also: [Restorer Context](restorer-context.md), [Tree After Restore](tree-after-restore.md)*
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue