- Detail the challenges of finding the 'watchee' path
- Explain the use of open_by_handle_at() and Irmap
- Explicitly document that pending events are dropped with a warning
- Explain how spurious events are generated during restore (ghost files)
- Add details for Fanotify inode and mount marks
Signed-off-by: Andrei Vagin <avagin@google.com>
- Explain the PTRACE_SEIZE and PTRACE_INTERRUPT sequence
- Detail the transparency of ptrace-stop (TRAP_STOP)
- Document cgroup v1 and v2 freezer mechanisms
- Mention kernel kludges for v1 freezer unreliability
- Clarify the relationship between freezer and ptrace
Signed-off-by: Andrei Vagin <avagin@google.com>
- Formalize TASK_ALIVE, TASK_STOPPED, and TASK_DEAD states
- Explain the rationale for default behaviors in dump/restore
- Mention pre-dump enforcement of the Running state
- Document the use of --leave-stopped for debugging
- Add instructions for resuming trees via SIGCONT and pstree_cont.py
Signed-off-by: Andrei Vagin <avagin@google.com>
- Formalize the Master and Slave descriptor concepts
- Describe the 'open()' state machine and early FD distribution via SCM_RIGHTS
- Document the inter-process synchronization (set_fds_event, futexes)
- List key dependencies (TTYs, Unix Sockets, Epoll)
- Add notes on Service FDs and restoration ordering
Signed-off-by: Andrei Vagin <avagin@google.com>
- Detail the Linux file object hierarchy (Inode, Dentry, File)
- Explain the SCM_RIGHTS mechanism for retrieving local FD copies
- Describe the gen_id and kcmp optimization for shared file detection
- Clarify the two-tier image storage structure (fdinfo vs specialized images)
Signed-off-by: Andrei Vagin <avagin@google.com>
- Formalize the architectural comparison (userspace vs. kernel integration)
- Highlight the dangers of DMTCP's fake PID virtualization
- Explain CRIU's usage of ns_last_pid and clone3 for real PID restoration
- Improve overall technical clarity and structure
Signed-off-by: Andrei Vagin <avagin@google.com>
- Explain the identification of COW candidates by comparing parent/child VMAs
- Describe the pre-mapping strategy before fork to leverage kernel sharing
- Detail the content verification and manual COW triggering
- Document the use of madvise(MADV_DONTNEED) for final memory layout accuracy
- Clarify current limitations regarding reparenting and VMA movement
Signed-off-by: Andrei Vagin <avagin@google.com>
- Embed DMTCP description and characteristics
- Update CRIU supported architectures (s390, MIPS, RISC-V, etc.)
- Refine the comparison table for accuracy and modern features
- Add more context for BLCR, PinPlay, and Legacy OpenVZ
Signed-off-by: Andrei Vagin <avagin@google.com>
- Document the use of 'compel hgen' for header generation
- Update the example header format to include structured relocations
- Describe the 'parasite_blob_desc' setup functions
- Refine the build procedure steps
Signed-off-by: Andrei Vagin <avagin@google.com>
- Clarify freezing mechanisms (PTRACE_INTERRUPT, Freezer CGroup)
- Detail the parasite injection and bootstrap process
- Explain the role of the restorer blob as a PIE and its conflict avoidance
- Document the final transition via sigreturn
Signed-off-by: Andrei Vagin <avagin@google.com>
- Explain the core problem of TCP 4-tuple mismatch
- Describe solutions for listening, in-flight, and established sockets
- Document the UPDATE_INETSK plugin hook for programmatic IP remapping
- Add a summary table of options and flags
Signed-off-by: Andrei Vagin <avagin@google.com>
- Document full CGroup v2 support and properties
- Explain CGroup namespace (CLONE_NEWCGROUP) handling
- Clarify the 'soft mode' default and other restoration strategies
- Detail the root mount requirement for bind-mounted subgroups
Signed-off-by: Andrei Vagin <avagin@google.com>
- Explain metadata collection from /proc and BPF syscall
- Describe data serialization using batch operations
- Add details about frozen maps handling
- Clarify current limitations regarding map_extra and BTF
Signed-off-by: Andrei Vagin <avagin@google.com>
- Explain how CRIU restores AIO context IDs and ring buffers
- Describe the tail synchronization technique using dummy /dev/null requests
- Clarify the lack of support for in-flight events and its implications
Signed-off-by: Andrei Vagin <avagin@google.com>
- Replace FIXME with a detailed description of the current approach
- Explain architecture detection using PTRACE_GETREGSET
- Describe the restoration process via sigreturn and mode switching
- Update vsyscall handling details
- Clarify the status of x32 support and TIF_IA32 removal
Signed-off-by: Andrei Vagin <avagin@google.com>
The README currently uses an external link to criu.org for the embedded
CRIU logo. Loading this URL when viewing the README on GitHub sometimes
fails with "Error Fetching Resource". Using a local copy of the logo
fixes this issue.
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The syntax of the inherit-fd functionality for unix socket and pipe
includes a colon.
Fixes: 0df3f79fc0 ("criu(8): fix --inherit-fd description")
Fixes: c37324b6d0 ("crtools: describe the inherit-fd option")
Signed-off-by: Mark Polyakov <mark@thundercompute.com>
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
This functionality (#2527) is being reverted and excluded from this
release due to issue #2812.
It will be included in a subsequent release once all associated issues
are resolved.
Signed-off-by: Andrei Vagin <avagin@google.com>
This patch updates the dependencies section of the AMDGPU plugin man
page to reflect that the plugin has been merged upstream and to fix a
formatting issue.
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The amdgpu plugin would create a memory buffer at the size
of the largest VRAM bo (buffer object). On some systems, VRAM
size exceeds RAM size, so the largest bo might be larger than
the available memory.
Add an environment variable KFD_MAX_BUFFER_SIZE, which caps the
size of this buffer. By default, it is set to 0, and has no
effect. When active, any bo larger than its value will be
saved to/restored from file in multiple passes.
Signed-off-by: David Francis <David.Francis@amd.com>
This commit removes the checks for the Python 2 binary in the makefile
and makes sure that ZDTM tests always use python3. Since support for
Python 2 has been dropped, these checks are no longer needed.
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
New 'query-ext-files' action for `criu dump` is sent after
freezing the process tree. This allows to defer gathering
the external file list when the process tree is in a stable
state and avoids race with the process creating and deleting
files.
Change-Id: Iae32149dc3992dea086f513ada52cf6863beaa1f
Signed-off-by: Michał Mirosław <emmir@google.com>
Make it possible to skip network lock to enable uses that break connections
anyway to work without iptables/nftables being present.
Signed-off-by: Michał Mirosław <emmir@google.com>
By default, the file name 'amdgpu_plugin.txt' is used also as the name
for the corresponding man page (`man amdgpu_plugin`). However, when
this man page is installed system-wide it would be more appropriate
to have a prefix 'criu-' (e.g., `man criu-amdgpu-plugin`).
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The --ghost-fiemap option was introduced with #1963.
It enables an optimized algorithm based on fiemap ioctl that can reduce
the number of syscalls used to checkpoint highly sparse ghost files. This
option is enabled by default. It can be disabled with --no-ghost-fiemap
when using SEEK_HOLE/SEEK_DATA is preferred. In addition, an automatic
fallback to SEEK_HOLE/SEEK_DATA is used for filesystems that do not
supporting fiemap.
Co-authored-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
This adds the non-root section and information about the parameter
--unprivileged to the man page.
Co-authored-by: Anna Singleton <annabeths111@gmail.com>
Signed-off-by: Adrian Reber <areber@redhat.com>
Signed-off-by: Anna Singleton <annabeths111@gmail.com>
A file's r/w/x changing between checkpoint and restore does
not necessarily imply that something is wrong. For example,
if a process opens a file having perms rw- for reading and
we change the perms to r--, the process can be restored and
will function as expected.
Therefore, this patch adds an option
--skip-file-rwx-check
to disable this check on restore. File validation is unaffected
and should still function as expected with respect to the content
of files.
Signed-off-by: Younes Manton <ymanton@ca.ibm.com>
Add SIGTSTP signal dump and restore. Add a corresponding field
in the image, save it only if a task is in the stopped state.
Restore task state by sending desired stop signal if it is present
in the image. Fallback to SIGSTOP if it's absent.
Signed-off-by: Yuriy Vasiliev <yuriy.vasiliev@openvz.org>
Modifications to support criu image streamer when using amdgpu_plugin.
When running with criu image streamer, fseek/lseek is not available so
we store the file size in the first 8-bytes of the actual file.
Signed-off-by: David Yat Sin <david.yatsin@amd.com>
Store BO contents directly to file (1 per GPU) instead of using
protobuf.
Bug Fix:
Fixes an issue where we could not handle BOs bigger than 4GB because
protobuf has an internal limit of 4GB for the Bytes structure.
Performance Improvements:
This significantly reduces CR duration on multi-GPU systems as it allows
reading and writing to disk in parallel. During checkpoint, instead of
waiting for all the BO contents to be read from the one protobuf file,
we can now start writing the BO contents as soon as the first BO is read
from disk. During restore, we can start writing BO contents to disk
after the first BO from VRAM. This also reduces the peak amount of
system memory used as we only need to keep 1 BO content in memory per
GPU at a time instead of all the BO contents.
Signed-off-by: David Yat Sin <david.yatsin@amd.com>
Add optional parameters to override default behavior during restore.
These parameters are passed in as environment variables before executing
CRIU.
List of parameters:
KFD_FW_VER_CHECK - disable firmware version check
KFD_SDMA_FW_VER_CHECK - disable SDMA firmware version check
KFD_CACHES_COUNT_CHECK - disable caches count check
KFD_NUM_GWS_CHECK - disable num_gws check
KFD_VRAM_SIZE_CHECK - disable VRAM size check
KFD_NUMA_CHECK - preserve NUMA regions
KFD_CAPABILITY_CHECK - disable capability check
Signed-off-by: David Yat Sin <david.yatsin@amd.com>
The device topology on the restore node can be different from the
topology on the checkpointed node. The GPUs on the restore node may
have different gpu_ids, minor number. or some GPUs may have different
properties as checkpointed node. During restore, the CRIU plugin
determines the target GPUs to avoid restore failures caused by trying
to restore a process on a gpu that is different.
Signed-off-by: David Yat Sin <david.yatsin@amd.com>
To support Checkpoint Restore with AMDGPUs for ROCm workloads, introduce
a new plugin to assist CRIU with the help of AMD KFD kernel driver. This
initial commit just provides the basic framework to build up further
capabilities. Like CRIU, the amdgpu plugin also uses protobuf to
serialize
and save the amdkfd data which is mostly VRAM contents with some
metadata.
We generate a data file "amdgpu-kfd-<id>.img" during the dump stage. On restore
this file is read and extracted to re-create various types of buffer
objects that belonged to the previously checkpointed process. Upon
restore the mmap page offset within a device file might change so we use
the new hook to update and adjust the mmap offsets for newly created
target process. This is needed for sys_mmap call in pie restorer phase.
Support for queues and events is added in future patches of this series.
With the current implementation (amdgpu_plugin), we support:
- Only compute workloads such (Non Gfx) are supported
- GPU visible inside a container
- AMD GPU Gfx 9 Family
- Pytorch Benchmarks such as BERT Base
amdgpu plugin dependes on libdrm and libdrm_amdgpu which are typically
installed with libdrm-dev package. We build amdgpu_plugin only when the
dependencies are met on the target system and when user intends to
install the amdgpu plugin and not by default with criu build.
Suggested-by: Felix Kuehling <felix.kuehling@amd.com>
Co-authored-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: Rajneesh Bhardwaj <rajneesh.bhardwaj@amd.com>
The --timeout option was introduced in [1] to prevent criu dump from
being able to hang indefinitely and allow users to adjust the time limit
in seconds for collecting tasks during the dump operation.
[1] d0ff730
Signed-off-by: Radostin Stoyanov <radostin@redhat.com>
The expected behavior of --tcp-close option when dumpping is to close
all established tcp connections including connection that is once
established but now closed. This adds an explicit description about
that behavior.
Signed-off-by: Bui Quang Minh <minhquangbui99@gmail.com>
Support for external net namespaces has been introduced with
commit c2b21fbf (criu: add support for external net namespaces).
Signed-off-by: Radostin Stoyanov <radostin@redhat.com>
Python 2 has been deprecated since January 1, 2020 and linux distributions
already support Python 3. Thus, to simplify maintenance and packaging
we could support criu-ns as Python 3 only.
v2: Add a message for criu-ns installation
Signed-off-by: Radostin Stoyanov <radostin@redhat.com>
This adds the option to choose the networking locking method.
CRIU currently uses iptables-restore cli for network locking/unlocking
but nftables support will be added later.
There have been reports from users that iptables-restore fails in some
way and an nftables based approach using libnftables could avoid this
external dependency.
v2: remove dependency details in man page for --network-lock.
v3: remove --network-lock from restore section in docs because it is
automatically detected from the inventory image now.
v4: add message that --network-lock will be ignored during restore
and value from dump will be used.
v5: run make indent
Signed-off-by: Zeyad Yasser <zeyady98@gmail.com>
