Private anonymous VMA contents have to be copied inside the PIE
restorer after mappings are placed at their final addresses.
Replace the synchronous preadv loop with a bounded Linux native AIO
window that submits page-read iovecs, reaps completions, and
resubmits short reads.
Keep auto-dedup punching on completed byte ranges and tear down the
AIO context on both success and error paths. The pages fd is
prepared for O_DIRECT before entering the restorer.
VMA page reads are page-aligned by construction: the destination
buffers are page-aligned, the iovec lengths are page multiples, and
the file offsets are page-aligned. O_DIRECT completions are
therefore always a whole number of pages (short reads only occur at
MAX_RW_COUNT or at EOF on page-multiple files), so resubmitting a
short read keeps its offset and buffers aligned without masking the
completion length.
Assisted-by: Claude:claude-sonnet-4-6
Assisted-by: Claude:claude-opus-4-7
Assisted-by: Codex:gpt-5
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Shared-memory restore currently issues page reads one pagemap entry
at a time. Use the existing PR_ASYNC path to enqueue shmem and
memfd-shmem reads, then flush each batch with one async-queue sync
pass.
Restore shmem, sysv-shmem and memfd-shmem content through a single
flat do_restore_shmem_content() loop that owns its page_read, and
map the memfd the same way as anonymous shmem so both share one
restore path instead of an extra temporary mapping.
Shmem segments are mapped via mmap() (page-aligned addresses) and
the pagemap offsets are PAGE_SIZE multiples, so the batched reads
are page-aligned and need no special O_DIRECT alignment handling.
Assisted-by: Claude:claude-sonnet-4-6
Assisted-by: Claude:claude-opus-4-7
Assisted-by: Codex:gpt-5
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Native AIO page restore needs the pages image fd to support direct
I/O, but some filesystems accept O_DIRECT with fcntl() and reject
the first read with EINVAL.
Enable O_DIRECT after the image headers are read, probe with an
aligned read, and fall back to buffered sequential I/O when the
probe reports EINVAL. Cache the result in page_read so normal page
reads do not need a flag lookup per request.
The streamer fd (opts.stream) is excluded from the probe because
its read loop uses arbitrary user buffers that are not aligned for
O_DIRECT.
PAGE_SIZE is not a compile-time constant on aarch64 (sysconf-backed
runtime value to support 4K/16K/64K kernels), so the probe buffer
is allocated with posix_memalign() instead of a stack array with
__attribute__((aligned(PAGE_SIZE))).
The single-page COW comparison buffer in restore_priv_vma_content()
is likewise allocated with posix_memalign() so direct reads into it
are page-aligned; this lets the local read path drop its per-read
bounce buffer.
Assisted-by: Claude:claude-sonnet-4-6
Assisted-by: Claude:claude-opus-4-7
Assisted-by: Codex:gpt-5
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
The restorer already has io_setup, io_submit, and io_getevents
syscall wrappers for all supported architectures. Native AIO restore
also needs io_destroy to release the AIO context before leaving the
restorer path.
Add the missing io_destroy syscall entries next to the existing
native AIO syscalls.
Assisted-by: Claude:claude-sonnet-4-6
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
The alpine-test CI job runs all ~483 zdtm tests sequentially three
times (normal, mntns-compat-mode, criu-config), followed by many
non-shardable tests. This dominates overall CI wait time. With only
2 jobs running in parallel (GCC and CLANG) the alpine tests take
around 30 minutes.
Use the existing --test-shard-index and --test-shard-count flags
already built into test/zdtm.py to split the zdtm test suite across
four parallel runners (shards 0-3). A fifth shard runs all
non-shardable tests (lazy pages, fault injection, test/others/*,
rootless, compel, plugins, etc.) independently and in parallel with
the zdtm shards. This increases parallelism from 2 to 10 jobs and
reduces the alpine test wall-clock time from ~30 to ~10 minutes.
Changes:
- run-ci-tests.sh: Build SHARD_OPTS from ZDTM_SHARD_INDEX/COUNT
env vars and pass them to zdtm.py. Extract all non-shardable
tests into a run_non_shardable_tests() function. Dispatch based
on shard index: 0-3 run zdtm slices, 4 runs non-shardable
tests, unset runs everything sequentially (preserving existing
behavior). Validate that ZDTM_SHARD_INDEX is set when
ZDTM_SHARD_COUNT is set.
- Makefile: Pass ZDTM_SHARD_INDEX and ZDTM_SHARD_COUNT into the
container when set. Split long container run command across
multiple lines for readability.
- ci.yml: Add shard: [0, 1, 2, 3, 4] to the alpine-test matrix,
producing 10 jobs (2 compilers x 5 shards). Job labels now show
descriptive shard names (e.g. "zdtm 1/4", "non-zdtm") instead
of raw indices.
When sharding is not configured the script behaves identically to
before, so other CI jobs (aarch64, compat, gcov, etc.) are
unaffected.
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Adrian Reber <areber@redhat.com>
Starting with Linux 7.1, the kernel enforces strict read-only field
protection for the rseq V2 ABI. Userspace is no longer allowed to
modify kernel-managed fields (cpu_id_start, cpu_id, node_id, mm_cid)
while rseq is registered. Violations are detected on the next context
switch and the offending process is killed with SIGSEGV.
The rseq01 test was zeroing the entire glibc-registered rseq area
with memset() before calling test_init(), which internally calls
fork(). This corrupted the read-only fields while the glibc rseq
registration was still active, causing the kernel to send SIGSEGV
during the fork.
Fix this by calling unregister_old_rseq() before the memset in both
main() and thread_routine(), so the kernel is no longer monitoring
those fields when they are zeroed. The subsequent register_thread()
call re-registers rseq with the clean area.
Link: 7f00232152
Assisted-by: Claude Code (https://claude.ai/code):claude-opus-4-6
Signed-off-by: Adrian Reber <areber@redhat.com>
Commit
1961c5e0de ("plugins/amdgpu: Cleanup env variable handling")
converted some error logs into warnings under the rationale of failures
not being fatal, but the execution continues with the default values.
Convert one more such error to a warning, as pointed out by Copilot that
makes things more consistent.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Copilot reports a possible false positive error on strtoul conversion due
errno not being reset on success. Fix it by clearing errnor prior to the
call.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Copilot reports %d is correct for boolean and %zx for size_t, fix it.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Fixes: 1961c5e0de ("plugins/amdgpu: Cleanup env variable handling")
A bunch of globals defined at the top of amdgpu_plugin.c only needs local
scope so make it so.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
It is a bit tidier to make the helpers return the value instead of passing
in the pointer, so lets make them take the default and return the overall
result, allowing the call sites to consolidate into a single line.
While at it we make the helpers static since they are not used outside the
file and demote the parsing errors to warning log messages since the
plugin does continue operating.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
In restorer_get_vma_hint, if a VMA from self_vma_list or tgt_vma_list
overlaps with the current candidate region (prev_vma_end + vma_len), we
advance prev_vma_end and move to the next VMA in the list. However, we
must then restart the check from the beginning of the while loop to
ensure that the updated candidate region doesn't overlap with any other
VMAs.
Fixes#3018
Fixes: 2c9ee0abd2 ("restore: update prev_vma_end after processing the last vma in a list")
Signed-off-by: Andrei Vagin <avagin@google.com>
mmaps done just for reserving address space something like
void *addr = mmap(NULL, SIZE, PROT_NONE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
Ensure that these do not have ac flag set in /proc/pid/smaps.
Signed-off-by: Bhavik Sachdev <b.sachdev1904@gmail.com>
An application might call mmap with PROT_NONE to reserve a large amount
of the virtual address space. Something like:
void *addr = mmap(NULL, SIZE, PROT_NONE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
During restore, when we have to premap private vmas (ie pieok is false),
CRIU tries to mmap this large PROT_NONE and fails.
If a vma has never been written to, it does not have "ac" flag set. Use
this fact to safely premap vmas that were done for address reservation.
Suggested-by: Andrei Vagin <avagin@gmail.com>
Signed-off-by: Bhavik Sachdev <b.sachdev1904@gmail.com>
Here is actually two issues:
First, prev_vma_end isn't updated properly after processing the last vma
in a list.
Second, a dummy end_vma isn't fully initialized and vma_area_is can
access uninitilized data.
Signed-off-by: Andrei Vagin <avagin@google.com>
Linking on arm64 using lld fails with:
LINK criu/pie/parasite.built-in.o
ld: error: discarding .shstrtab section is not allowed
keep the missing sections to enhance portability
Fixes: #2978
Signed-off-by: Pepper Gray <hello@peppergray.xyz>
building on a system without gcc fails with error:
make[1]: gcc: No such file or directory
use system default cc to enhance portability
fixes: #2976
Signed-off-by: Pepper Gray <hello@peppergray.xyz>
Add tests for pre-dump, dump, and restore workflow using the loop
workload, a shell-job test with PTY setup, and a service test that
exercises the full workflow through a criu service daemon.
Pre-dump tests are skipped on architectures without soft-dirty page
tracking (e.g. aarch64). Only dump/restore are tested on these cases.
To help with CI debugging, the CRIU log files are shown on test failure.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Add a reference C client that uses libcriu to checkpoint and restore
processes. Set LOCAL=0 to build against the system-installed libcriu
instead of the source tree.
Status messages use fprintf(stderr) so they are not delayed by stdout
buffering when running without a terminal (e.g. in CI).
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Starting with Linux kernel commit 7b735ef81286 ("rtnetlink: add
missing netlink_ns_capable() check for peer netns"), creating a
veth pair with a peer in a different network namespace requires
CAP_NET_ADMIN in the peer namespace as well:
rtnetlink: add missing netlink_ns_capable() check for peer netns
rtnl_newlink() lacks a CAP_NET_ADMIN capability check on the peer
network namespace when creating paired devices (veth, vxcan,
netkit). This allows an unprivileged user with a user namespace
to create interfaces in arbitrary network namespaces, including
init_net.
Add a netlink_ns_capable() check for CAP_NET_ADMIN in the peer
namespace before allowing device creation to proceed.
Link: 7b735ef812
When CRIU restores a veth in user namespace mode, it sends the
RTM_NEWLINK request from a netlink socket inside the child user
namespace. The veth peer is placed into the root network namespace
via IFLA_NET_NS_FD, but the child user namespace does not have
CAP_NET_ADMIN in the root namespace, so the new kernel check
rejects the request with EPERM.
Fix this by routing the veth creation through usernsd when the
peer lives in an external (host) namespace specified via the
--external veth[name] restore option. usernsd runs with real
root privileges in the init user namespace, so it passes the
capability check in both namespaces.
The usernsd path is only used for external veth mappings, not for
child-to-child namespace veths (has_peer_nsid), because in that
case both namespaces share the same user namespace and already
have CAP_NET_ADMIN in each other.
The approach mirrors restore_one_macvlan(), which already solves
the same CAP_NET_ADMIN-in-both-namespaces problem for macvlan
devices. A new veth_link_info_userns() builds the RTM_NEWLINK
request without IFLA_NET_NS_FD for the peer -- since usernsd
sends the request from the root network namespace, the peer
naturally stays there. userns_restore_one_link() then adds a
top-level IFLA_NET_NS_FD to move the main device into the child
namespace.
Assisted-by: Claude Code (claude-opus-4-6):claude-opus-4-6@default
Signed-off-by: Adrian Reber <areber@redhat.com>
Protobuf 7.x (shipped in Arch Linux) removed the `label` instance
attribute from `FieldDescriptor`. Accessing `field.label` now
raises `AttributeError`, which breaks all CRIU tests at the
`check_pages_counts` step when pycriu tries to convert protobuf
stats images to dicts:
File "test/pycriu/images/pb2dict.py", line 345, in pb2dict
if field.label == FD.LABEL_REPEATED:
AttributeError: 'FieldDescriptor' object has no attribute 'label'
Protobuf 7.x replaced `field.label` with boolean properties
`field.is_repeated` and `field.is_required`. These properties
do not exist in protobuf 3.x/4.x.
Add a `_is_repeated()` compatibility helper that checks for the
`label` attribute first (protobuf <= 5.x) and falls back to
`is_repeated` (protobuf >= 7.x), keeping pycriu working across
both old and new protobuf versions.
Assisted-by: Claude Code (claude-opus-4-6):claude-opus-4-6@default
Signed-off-by: Adrian Reber <areber@redhat.com>
Starting with Linux 7.0, the rseq feature size has grown to 33 bytes
and AT_RSEQ_ALIGN has increased to 64. This causes sizeof(struct rseq)
from the uapi header to be 64 (33 bytes padded to aligned(32)), while
the actual registration size used by glibc (__rseq_size) is 33.
The rseq01 test was using sizeof(struct rseq) as the registration
size and a test-local __rseq_abi variable (with only 32-byte
alignment from the uapi header) as the rseq area. Both are
wrong on kernel 7.0:
- The kernel now checks alignment against __alignof__(struct rseq)
which is 64 internally; the 32-byte-aligned test variable may
not satisfy this.
- sizeof(struct rseq) = 64 does not match the feature size of 33
that glibc registered with, so re-registration with a different
size fails.
Fix by:
- Using __rseq_size (the feature size reported by glibc) as the
registration size instead of sizeof(struct rseq).
- Using the glibc-provided rseq area (at thread_pointer() +
__rseq_offset) which is allocated with proper AT_RSEQ_ALIGN
alignment, instead of a test-local variable.
Both fall back to the previous behavior when glibc rseq support
is not available.
This mirrors the glibc fix:
https://sourceware.org/cgit/glibc/commit/?id=67f303b47dc584f204e3f2441b9832082415eebc
Assisted-by: Claude Code (claude-opus-4-6):claude-opus-4-6@default
Signed-off-by: Adrian Reber <areber@redhat.com>
Kernel 7.1 removed IPPROTO_UDPLITE support. Add a checkskip
script that probes for a UDPLITE socket and skips the test
with EPROTONOSUPPORT instead of failing.
Assisted-by: Claude Code (claude-opus-4-6):claude-opus-4-6@default
Signed-off-by: Adrian Reber <areber@redhat.com>
Refactor lima.sh to extract shared helpers (_common_setup and
_common_test) and add a second variant that uses the
@kernel-vanilla/next COPR repository. Convert the Lima CI job
to a matrix with both stable and next variants so we catch
upstream kernel changes (such as the UDPLITE removal in 7.1)
early.
Assisted-by: Claude Code (claude-opus-4-6):claude-opus-4-6@default
Signed-off-by: Adrian Reber <areber@redhat.com>
Kernel 7.1 removed IPPROTO_UDPLITE support. When the UDPLITE diag
module is absent, the netlink request returns -ENOENT, which CRIU
was treating as a fatal error causing the entire dump to fail.
Apply the same treatment as IPPROTO_RAW: return success from the
error callback so the dump can proceed. If a UDPLITE socket is
actually encountered, it will fail at lookup time with a clear
error rather than failing the entire dump preemptively.
Fixes: https://github.com/checkpoint-restore/criu/issues/3002
Assisted-by: Claude Code (claude-opus-4-6):claude-opus-4-6@default
Signed-off-by: Adrian Reber <areber@redhat.com>
Mostly after calling libdrm functions pr_perror is wrong because calling
convention there is to return zero or negative error code - errno is not
guaranteed to be valid. So convert those to pr_err with manually calling
strerror to log user friendly messages. Same conversion for handling
errors from some plugin internal functions.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com> # v1
Cc: Radostin Stoyanov <rstoyanov@fedoraproject.org>
---
v2:
* Handle newly added pr_perror after amdgpu_device_initialize().
* Add newlines.
v3:
* Add some newlines to pr_err.
There is no need to initialize and de-initialize libdrm only to use the
duplicated fd for calls to drmPrimeFDToHandle(). From the kernel's
perspective it is exactly the same file so operation is identical to just
using the passed in file descriptor.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Consolidate KFD and DRM VMA handling path to the same (existing) helper.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Only try to restore the dmabuf fd if the render node restore failed due
image not present. Other errors means the file is a render node (image is
present) so we can return an error straight away.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Currently the code contains multiple places where callers expect the errno
to be valid (implied by calling pr_perror to log file IO errors) after
calling open_img_file(), read_fp() and write_fp() helpers. Problem there
is that those helpers can destroy the errno courtesy of themselves
emitting log messages via fprintf.
Furthermore, the callers then sometimes invent the errors to return back
to the caller, or even outside of the plugin.
On top of that there is no benefit to buffered file IO given the plugin
reads and writes in buffer object size chunks, so it it preferrable to
just go direct and avoid any possibility of extra copying to and from libc
temporary buffers.
So lets just convert it all in one swoop to POSIX IO and make sure correct
errnos are always propagated to the caller (including to CRIU core).
Hopefully this removes all instances of incorrect pr_perror log messages.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com> # v1
---
v2:
* Handle short reads and writes.
* Add O_TRUNC.
v3:
* Handle EINTR and EAGAIN.
* Set errno on len == 0.
* Use %zu for size_t printing.
v4:
* Drop shadowed ret.
* Drop needless casts.
Use the recently added image load helper in one more function which
contains the exactly same flow of open-alloc-read.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Given that dump and restore of KFD vs DRM forks into completely parallel
flows inside the respective plugin callbacks, lets split it out by
extracting into helpers. This de-clutters the code flow and makes it
obvious which part is done where.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Currently the plugin is added to the inventory only if there were amdgpu
mapped vmas in the target process. This makes restoring a simple process
with only an open DRM fd and some buffer objects not work.
Fix it by adding the plugin to the inventory when there were open DRM
files as well.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Check SDMA operation type once at the beginning of the copy helper and fix
one incorrect usage of pr_perror while at it.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Open_drm_render_device() is a little bit confused. It can log confusing
errors using errno which hasn't been set, or it has been overwritten. It
can also inspect errno after it had reset it creating effectively
unreachable code. And finally it sometimes returns the negative errno, and
sometimes a plain -1. Fix it all up and make it consistent, while at the
same time correcting two calls site incorrectly using pr_perror(), while
the function does not guarantee a valid errno.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Error logging for just a single error condition will eat the errno so lets
just remove it. All callers use pr_perror() and rely on the helper not
munching it up so no information is lost by removing the internal log.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
We remove some function entry and exit markers, some interim messages
which provide no real value, and only log virtual address allocation
failures on the failure path. For the last one we also fix incorrect
usage of pr_perror.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
The extra debug build log helper does not seem very useful and some of
them do not even compile. Lets just remove it and replace with the
standard pr_debug. In case of too much noise, we can later re-evaluate
to remove some of the not very useful log messages.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Instead of open coding the same ioctl twice we can put it in a loop from
which we break out once we have allocated enough space for all objects.
While at it we add error handling for the memory allocation.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
---
v2:
* s/for(;;)/while(1)/ (David)
Both CRIU and plugin are built with 64-bit off_t and most of the parallel
bo restore path uses uint64_t for storing offsets.
Lets consolidate by changing int to off_t, and by doing so remove any
possibility that the call to fseeko can overflow the return type.
While at it, add error checking to both seek operations.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com> # v1
---
v2:
- Correct offset type as well.
Shallower indentation makes a heavily indented function hopefully a bit
more readable.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Consolidate to one fclose() call with the extra bonus the inconsequential
file descriptor leak is eliminated.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Close the directory when probing images during restore init fails.
While at it, consolidate the identical code between the DRM and KFD images
into a common helper.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Currently the failure to save content of a buffer object is only logged,
while the checkpointing will continue unaware due the error code being
"eaten" inside sdma_copy_bo() cleanup path, together with the caller not
checking the return value to begin with.
Fix it by preserving and returning the original error code inside
sdma_copy_bo(), and abort the checkpointing process in case of a failure.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Currently failure to open files logs an unitialized stack string as the
failed path. Lets just log the basename and drop the unused stack buffer.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
Size needs to be initialized before opening the file for write, otherwise
garbage is written into the file.
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-By: David Francis <David.Francis@amd.com>
