From f3b293fd850eddc0b328be2a822076af1cff6f1f Mon Sep 17 00:00:00 2001 From: Andrei Vagin Date: Tue, 23 Jun 2026 16:57:03 +0000 Subject: [PATCH] docs: Clarify memory consistency and performance in pre-dump documentation Correct the documentation regarding `vmsplice` and `SPLICE_F_GIFT` behavior. `SPLICE_F_GIFT` does not enforce Copy-on-Write (COW) when the user-space process modifies the gifted memory. Instead, it is a zero-copy mechanism where post-resume modifications can lead to inconsistent intermediate dumps. Explain how CRIU handles this inconsistency via its iterative design, relying on the soft-dirty tracker to catch post-resume modifications and re-dump them in subsequent iterations, ensuring final consistency. Also emphasize that the `splice` scheme relies heavily on `vmsplice` performance to minimize freeze time, making the migration almost invisible to the process. Signed-off-by: Andrei Vagin --- .../optimizing-pre-dump-algorithm.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/Documentation/under-the-hood/optimizing-pre-dump-algorithm.md b/Documentation/under-the-hood/optimizing-pre-dump-algorithm.md index 3ac61dcd4..c822d178c 100644 --- a/Documentation/under-the-hood/optimizing-pre-dump-algorithm.md +++ b/Documentation/under-the-hood/optimizing-pre-dump-algorithm.md @@ -5,9 +5,9 @@ Pre-dumping is the process of capturing dirty memory pages while an application ## Traditional vs. Optimized Pre-dump ### The `read` Mode (Traditional) -In this mode, CRIU uses the `process_vm_readv()` system call to read memory from the target process. +In this mode, CRIU uses the `process_vm_readv` system call to read memory from the target process. * **Workflow**: Tasks are briefly frozen to identify dirty pages and reset the soft-dirty bit, then resumed. CRIU then reads the pages from the running process's address space. -* **Challenge**: Reading memory while a process is running can lead to minor inconsistencies if the process modifies a page *while* it is being read. Furthermore, `process_vm_readv()` requires the target process to be alive and its memory mappings to remain stable during the read. +* **Challenge**: Reading memory while a process is running can lead to minor inconsistencies if the process modifies a page *while* it is being read (see [Memory Consistency](#memory-consistency) below). Furthermore, `process_vm_readv` requires the target process to be alive and its memory mappings to remain stable during the read. ### The `splice` Mode (Optimized & Default) The `splice` mode (enabled via `--pre-dump-mode=splice`) uses a zero-copy "gift" mechanism to further reduce freeze time and improve reliability. @@ -19,8 +19,17 @@ The `splice` mode (enabled via `--pre-dump-mode=splice`) uses a zero-copy "gift" 4. **Parallel Draining**: While the tasks are running, the main CRIU process "drains" the data from the pipes and writes it to the image files or sends it to the page server. #### Why `splice` is Better: -* **Consistency via COW**: The `SPLICE_F_GIFT` flag ensures that if the process modifies a "gifted" page after resuming, the kernel performs a **Copy-on-Write (COW)**. The pipe buffer continues to hold the *original* version of the page as it existed at the moment of the `vmsplice()` call, ensuring a perfectly consistent snapshot of that page. -* **Minimized Downtime**: The "freeze" duration is reduced to just the time needed for the parasite to execute the `vmsplice()` system calls, rather than the time needed to transfer gigabytes of memory data over the network or to disk. +* **Minimized Downtime**: The "freeze" duration is reduced to just the time needed for the parasite to execute the `vmsplice()` system calls, rather than the time needed to transfer memory data over the network or to disk. This scheme relies entirely on `vmsplice()` being extremely fast. Because the target process is frozen during these calls, minimizing this duration is critical to the primary goal of pre-dumping: reducing process downtime during live migration and making the migration process almost invisible to the application. +* **Zero-Copy Transfer**: By gifting pages directly from the target process to the pipe, `splice` mode avoids copying memory to CRIU user-space buffers (unlike `read` mode which uses `process_vm_readv` to copy data). While this zero-copy mechanism does not use COW (meaning intermediate dumps can be inconsistent if pages are modified after resume), CRIU's iterative design handles this inconsistency (see below) while maximizing transfer performance. + +## Memory Consistency + +Because the target process is resumed while CRIU is still writing the memory data (in both `read` and `splice` modes), intermediate pre-dump images may contain inconsistent memory states. + +This inconsistency is expected and handled by CRIU's iterative design: +1. **Tracking Changes**: CRIU uses the kernel's soft-dirty tracker to monitor memory writes after the process is resumed. +2. **Subsequent Dumps**: Any page modified after it has been pre-dumped is marked dirty again and will be captured in the next pre-dump iteration or during the final dump. +3. **Restoring Consistency**: During restore, CRIU applies the dump images in sequence (from oldest pre-dump to the final dump). The final dump is taken while the process is fully frozen, ensuring that the final state of all memory pages is consistent. ## Usage