End-to-end restore tests do not isolate codec behavior or worker-pool
decisions, which makes boundary failures difficult to diagnose.
Test per-page and region round trips for zero, repeating, pseudorandom,
and single-byte inputs across several acceleration values and region
sizes. Verify zero detection, deterministic raw fallback, and page-image
offset alignment above 4 GiB.
Exercise serial and parallel LZ4 decoding and zero filling,
deterministic 0/1/N thread selection, effective-capacity gating,
work-based automatic width, growth of a persistent pool, initialization
of the shared batch budget, and return of the CPU budget. Also exercise
nonblocking acquisition and release of the second encoded input slot.
Verify that serial decoding skips caller work, while a parallel batch
runs it once after dispatch. Link the tests only when LZ4 is available.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Existing memory tests do not deliberately produce every compression
storage form or cross compression-specific batch and transport
boundaries. Regressions in zero, LZ4, raw, and mixed handling could
therefore go unnoticed.
Add neutral pages_content workloads and use them through symlinked
per-page and region-compressed variants. The same contents now run
uncompressed in ordinary sweeps, while NO_LZ4 builds skip only the
compressed variants.
Exercise compressible, incompressible, zero, mixed, file-backed, COW,
and exceptional mappings. Keep a large private zero range over nonzero
file contents so automatic restore exercises parallel zero filling when
the host has at least two available CPUs, and cannot pass by leaving an
untouched zero mapping. Size the compressible mapping at 33 MiB so
restore handles one full 32 MiB batch followed by 1 MiB of useful work.
This crosses the batch boundary with enough work to exercise read/decode
overlap. Cover pre-dump, deduplication, lazy pages, stream, direct I/O,
page server, RPC, region boundaries, incremental parent chains, and
repeated batches beyond 32 MiB. Verify restored contents in every case.
Pass both syntaxes for restore-only decompression options through the
ZDTM RPC runner. Exercise automatic concurrency in per-page and region
variants; the remaining variants retain the serial default.
Suggested-by: Andrei Vagin <avagin@google.com>
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The compression paths have no supported user or RPC controls, and
clients cannot discover whether CRIU was built with LZ4.
Expose per-page and region compression, LZ4 acceleration, and worker
concurrency for LZ4 decoding and eligible large zero fills through the
CLI, configuration files, RPC, and libcriu. Add compression to the
feature-check API and document each interface.
Define --decompress-threads=0 as automatic worker concurrency and one as
the serial default with zero-fill workers disabled. Values above one
bound aggregate worker concurrency. Automatic mode derives a useful
width from CPU affinity and the work in each batch. Reduce explicit
requests above the available CPU count with a warning.
Document the independent two-slot encoded-input budget. A local restore
overlaps its next read only when it can reserve the second slot without
waiting; otherwise it keeps synchronous I/O.
Use the compression mode as the source of truth and validate every
related value. An explicit RPC OFF clears ambient compression settings
and rejects conflicting fields. Reject combinations unsupported by page
server, stream, TLS, and image-format paths, and cover option and RPC
precedence in the tests.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Compression changes the pages-image payload layout. Older CRIU versions
ignore optional pagemap metadata and would interpret packed bytes as
ordinary pages.
Introduce image version V1.2 for compressed page payloads and reject it
when CRIU lacks LZ4 support. Validate compression and version
combinations when reading inventories. Propagate V1.2 through parent
chains, including an uncompressed child of a compressed parent, while
independent uncompressed images retain the previous version.
Reject an incremental dump when its configured parent inventory is
unreadable or unsupported instead of risking a version downgrade. Keep
accepting legacy parents without dump uptime when pidfds can detect PID
reuse.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The PIE restorer cannot link against LZ4. Decode mappings containing
LZ4 payloads through CRIU's premap path, while retaining delayed PIE
restore for raw and zero ranges when a worker pool cannot help.
Classify image ranges as LZ4, raw, or zero. Decode LZ4 blocks while CRIU
still runs in its normal address space, read aligned raw ranges directly
into their final VMAs, and clear zero ranges in place. Omit compression
metadata for entries containing only raw blocks so they retain the
ordinary restore path.
Share one encoded-read context across each incremental parent chain and
reuse its worker pool for bounded batches, COW reads, streamed images,
and page-server chunks. Use the same jobs to fill zero runs of at least
1 MiB when two blocks and two CPUs are available. Zero-only ranges that
cannot benefit retain the PIE path, while zero blocks in an existing
encoded batch remain inline without effective parallel capacity. Hold
input buffers only while encoded work is active, and join every worker
before child restore or the PIE handoff.
Account calling restore threads and workers against a shared CPU budget.
A requested width of zero selects automatic concurrency, one keeps LZ4
decoding serial and disables zero-fill workers, and larger values cap
aggregate worker concurrency. Bound the active width by CPU affinity and
useful work. Release encoded-work leases before ordinary reads so idle
contexts cannot block ready workers.
Local payload-heavy restores otherwise alternate between reading an
encoded batch and decoding it. This leaves either storage or worker CPUs
idle. When a second encoded-input lease is immediately available,
dispatch the current batch to the pool and use the already-accounted
calling thread to read the next adjacent payload. Keep synchronous reads
for serial restore, streams, direct I/O, small batches, and contended
leases.
Select direct AIO from the ranges actually delayed for PIE. Batch
inherited COW comparisons and streamed payload reads, and align
page-server chunks to compression regions. Validate metadata and offset
arithmetic before allocation or I/O, roll back partially enqueued work,
and propagate worker, I/O, and deferred deduplication failures before
remapping.
On a 256 MiB pseudorandom workload with the worker limit set to eight
and three measured runs, median restore time was 114.5 ms uncompressed,
113.4 ms with LZ4 per page, and 116.5 ms with 64 KiB regions. The
previous compressed restore path took about 227 ms in either mode.
With automatic concurrency and 256 KiB regions, parallel zero filling
reduced median restore time over five runs from 103.1 ms to 31.7 ms for
zero-filled memory and from 89.3 ms to 60.0 ms for mixed memory.
On 512 MiB TEXT and ELF workloads with automatic concurrency and 256 KiB
regions, five-run median restore time fell from 117.0 ms to 110.7 ms and
from 115.4 ms to 109.5 ms, respectively.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
A coalesced compressed read can allocate buffers approaching the size
of a large checkpoint and issue readahead across large, unrelated gaps.
Limit each asynchronous batch to 32 MiB of decoded pages. Split
readahead ranges when adjacent entries are more than 1 MiB apart or
their span would exceed 256 MiB. This bounds compressed input, metadata,
the decoded destination, and each individual readahead request.
Keep a split logical read transactional. If a later chunk fails, remove
newly queued requests and restore any pre-existing tail that an earlier
chunk extended.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Incremental restore can request alternating pages from one compressed
parent region. Each partial request currently reads and decompresses the
complete region again, multiplying both I/O and CPU work.
Cache the most recently decoded region in each page reader, keyed by its
virtual address and decoded size. Keep the cache across cursor advances
and resets, while duplicate readers start with independent empty caches.
A reader's pages image is immutable, so this key remains valid.
A 512 MiB mapping with 1 MiB regions was pre-dumped and then modified on
alternating pages. Across five measured restores after one warmup, the
median fell from 18.318 seconds to 0.359 seconds, a 51.1x speedup, while
every restored checksum matched.
Suggested-by: Andrei Vagin <avagin@google.com>
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Existing pagemap readers derive each pages-image offset from a fixed
PAGE_SIZE payload per present page. Packed compressed blocks make that
calculation invalid and must be reconstructed before reaching restore
destinations.
Track block and intra-region cursors while advancing, skipping, and
reading pagemap entries. Validate block counts, decoded sizes, totals,
and aligned-payload offsets before I/O. Read per-page and region images
locally, preserve ordinary parent entries, and support sequential
per-page image streams.
Carry block metadata with queued iovecs, coalesce only compatible
regions, read each packed batch once, and rebuild its destinations block
by block. Allow exactly IOV_MAX destinations and roll back iovec changes
if appending metadata fails. Decompress complete regions directly into
a contiguous destination and use scratch space for partial or split
regions. Copy the metadata into restorer memory for the later PIE decode
path.
Keep compressed images on buffered I/O, avoid hole punching packed
payloads, handle short reads, and keep compressed local reads out of PIE
until that path can decode them.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
struct page_read combines image cursors, parent traversal state, and
callbacks for several restore paths. Its terse field comments make
changes to that state difficult to review safely.
Describe each callback and cursor before compressed readers extend the
structure. Make no functional change.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Present pages are currently written as fixed PAGE_SIZE payloads. To
store packed compressed data, page transfer must choose a representation
for each block and record its size before publishing the pagemap entry.
Write zero, raw-fallback, and LZ4 blocks for individual pages or fixed
regions. Delay present pagemap entries until their block metadata is
complete, and keep region entries within one VMA so restore never splits
an LZ4 block at a VMA boundary.
Align entries whose first stored block is raw so direct restore does not
pay for mismatched file and memory page phases. Track padding in the
pages-image offset and keep variable-length compressed output on
buffered I/O.
Add a per-page compressed page-server command that sends each block size
with its payload. Validate its flags and ranges, preserve auto-dedup
ordering, and store the received representation without recompressing
it. Handle short pipe, file, and socket I/O throughout the new path.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Page-server cleanup can run after partial setup and more than once.
Partially initialized pipe arrays can still contain fd 0, while a
retained destination ID can close descriptors again after their numbers
have been reused. Setup failures and completed sessions can also leave
transfer pipes open.
Initialize pipe descriptors to -1 and clear the destination ID before
closing its image. Check pipe initialization and size queries, and close
transfer pipes when setup fails or a session completes.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The page-image writer and restore reader need a shared compression
policy before user-facing interfaces can select it.
Add internal options for the compression mode, region size, LZ4
acceleration, and worker concurrency for LZ4 decoding and large zero
fills. This gives later writer and reader changes one process-wide
source of configuration while leaving CLI and RPC exposure to a
separate change.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Compressed page writers and readers need one definition of the block
format and its size limits. Duplicating those rules at each call site
would make zero, raw-fallback, and LZ4 handling easy to diverge.
Add common LZ4 helpers, zero-page detection, and shared bounds and
storage thresholds for per-page and region users. Region helpers encode
zero blocks as size zero, fall back to raw bytes when compression saves
too little, and decode all three representations. Build the
implementation only when CONFIG_LZ4 is enabled.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Present pagemap entries currently imply one PAGE_SIZE payload per page.
That is not enough to locate packed zero, raw, and LZ4 blocks or to
describe region compression.
Record the compression mode and region size in the inventory. Add
packed per-block sizes, their total, and the region width to pagemap
entries, plus PE_PAYLOAD_ALIGNED for entries whose payload starts after
padding. Keep the fields optional so ordinary and older images retain
their existing representation.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Memory page compression requires liblz4, but making it mandatory would
prevent CRIU from building on systems that do not provide it.
Detect liblz4 with pkg-config and enable CONFIG_LZ4 only when it is
available. Add NO_LZ4=1 as an explicit opt-out and register the
development package in the supported distribution dependency lists.
Install liblz4 for the per-commit build, then run a separate NO_LZ4
build and unit test to keep the optional configuration usable.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The kernel for the github actions 26.04 images fixed the kernel to no
longer have the bug that lead to the `socket-tcbuf*` failures.
Signed-off-by: Adrian Reber <areber@redhat.com>
During CI preparation, tests running inside user namespaces or
unprivileged hooks require path traversal execute permissions from
the root directory down to the current working directory.
Signed-off-by: Andrei Vagin <avagin@google.com>
When checkpointing tasks confined by stacked AppArmor profiles inside an
AppArmor namespace, CRIU compiles a permissive parasite profile using
apparmor_parser and writes it to the namespace policy replacement
interface
(/sys/kernel/security/apparmor/policy/namespaces/<ns>/.replace).
Previously, get_suspend_policy() invoked apparmor_parser with flags
(-QWL) requesting cache file generation and read the resulting cache
file. On modern AppArmor 4.x kernels, cache files include cache metadata
headers that cause the kernel policy unpacker to reject the binary blob
with EPROTONOSUPPORT (-93, "invalid profile format"). Replace the cache
mechanism by invoking apparmor_parser with the -o flag to output a clean
binary policy stream directly.
Furthermore, when writing a replacement profile blob to an AppArmor
.replace interface, successful writes return the total size of the
profile load data structure consumed or updated in the kernel (e.g.,
5233 bytes for stacked profiles), rather than the exact byte length of
the input payload (e.g., 727 bytes for a parasite profile). Checking if
(n != len) erroneously treats a successful replacement write as a
failure. Fix policy write validation to check if (n < 0).
Across Linux kernel versions, AppArmor enforces different requirements
for policy replacement blobs written to a namespace .replace interface:
- Older kernels (AppArmor 3.x / early 4.x on Ubuntu 22.04) expect blobs
compiled without namespace headers (-n <ns>); specifying -n causes
the kernel to load the replacement into a nested sub-namespace
(<ns>//<ns>), leaving the active profile unchanged and causing socket
permission failures during dump.
- Modern kernels (AppArmor 4.x on Ubuntu 26.04) reject blobs compiled
without namespace headers with EPROTONOSUPPORT (-93, "invalid profile
format").
Update get_suspend_policy() and write_aa_policy() to implement runtime
fallback: first try compiling without -n <ns> and writing directly to
the namespace interface for older kernels. If writing fails (n < 0),
retry compilation passing -n <ns> and write to the root interface to
satisfy modern kernels.
In addition, when restoring tasks confined by profiles inside an
AppArmor namespace on AppArmor 4.x kernels, the host root unconfined
profile remains stacked with the target namespace profile when viewed
from the root namespace (e.g., "unconfined//&:ns:profile"). Update
checkprofile() in the apparmor_stacking test to recognize and allow the
"unconfined//&" prefix when verifying restored profiles.
Assisted-by: Gemini
Signed-off-by: Andrei Vagin <avagin@google.com>
In file included from compel/plugins/std/infect.c:14:
compel/include/uapi/compel/asm/sigframe.h:17:8: error: redefinition of 'gcs_context'
17 | struct gcs_context {
| ^
/usr/include/aarch64-linux-gnu/asm/sigcontext.h:188:8: note: previous definition is here
188 | struct gcs_context {
| ^
Signed-off-by: Andrei Vagin <avagin@google.com>
Add an --image-io-mode passthrough to zdtm.py, set on both dump and restore
like --stream and translated into the RPC request like --pre-dump-mode. Run
maps04 with --image-io-mode=direct and maps00 with --image-io-mode=direct --rpc
in run-ci-tests.sh so the O_DIRECT dump/restore paths and the RPC handler are
exercised.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Complete --image-io-mode=direct on the dump side: for a local, non-streaming
dump, set O_DIRECT on the pages image fd in open_page_xfer() so the existing
splice() in write_pages_loc() performs direct I/O. Otherwise the dump leaves the
written pages dirty in the cache, so a same-host direct restore still has to
flush them and a large checkpoint pollutes the dump host's cache.
splice() to a file goes through iter_file_splice_write() -> ->write_iter, which
honours O_DIRECT on filesystems that support it (ext4, xfs). Where it is not
supported splice() fails with EFAULT (or EINVAL); write_pages_loc() clears
O_DIRECT, warns, and retries the remainder buffered. Only whole pages are
transferred, so the file offset stays block-aligned across the switch and the
bytes written match the buffered path, so images restore in either mode. This
keeps the transfer zero-copy: no bounce buffer and no kernel-version check.
The page server and image streaming stay buffered.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Since #3022 restore reads the pages image with O_DIRECT unconditionally. That
helps a cold or cross-host restore, but on a same-host restore it bypasses the
still-warm cache and reads from disk, which made maps04 much slower (#3053).
Add an --image-io-mode option to select the pages-image I/O method: writeback
(default) keeps the buffered behaviour, direct uses O_DIRECT. It applies to both
dump and restore and can be set on the command line or over RPC.
Gate the restore side at the probe's call sites: open_page_read_at() and the
restore vma-io builder in mem.c switch the pages fd to O_DIRECT only when direct
mode was requested, so writeback keeps reading through the host page cache. The
probe itself no longer inspects the mode; as it now runs only on an explicit
request, its buffered-fallback log becomes a warning. The dump side follows in
the next commit.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Remove the second p1.stdout.read() call which was a no-op
since p1.stdout was already fully consumed by the read() on
line 38. The result was immediately overwritten by the
p2.stdout.read() that follows.
Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Adrian Reber <areber@redhat.com>
Remove C-style pre-initialization of mnt_id to -1. The
variable is always assigned inside the loop, and the for/else
construct raises an exception if it is never set.
Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Adrian Reber <areber@redhat.com>
Remove C-style pre-initializations of variables that are always
set before use. Python does not require variables to be declared
before assignment, so sentinel values like `pid = -1`,
`status = -1`, and `stats_written = -1` are unnecessary when
the variable is unconditionally assigned in the following code.
Also drop the unused capture of __dump_process.wait() return
value in fini().
Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Adrian Reber <areber@redhat.com>
Add a regex-based ignore list to grep_errors() so that known
harmless log messages do not trigger "ERROR OVER" output. This
reduces noise when running tests with ns/uns flavors where
"Address already assigned" messages are expected and benign.
This reduces CI output by at least 14400 lines (8*450*2*2):
8 lines per occurrence, ~450 tests, 2 flavors (ns and uns),
each running with and without --mntns-compat-mode.
Assisted-by: Claude Code:claude-opus-4-6
Signed-off-by: Adrian Reber <areber@redhat.com>
Use ExitStack to ensure the conditionally-opened file for
FILE_SHARED/FILE_PRIVATE VMAs is always closed, even if an
exception occurs during seek or page iteration.
Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Adrian Reber <areber@redhat.com>
Use 'with' statement instead of manual open/read/close to
ensure the file is closed if read() raises an exception.
Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Adrian Reber <areber@redhat.com>
Use 'with' statement for opening /proc fdinfo to ensure the
file handle is closed after iteration.
Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Adrian Reber <areber@redhat.com>
Use 'with' statements for both input and output file handles
to ensure they are closed on exceptions.
Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Adrian Reber <areber@redhat.com>
Close the fd from os.open("/dev/null") after dup2, matching
the pattern used for stdout/stderr a few lines above. Also
convert __dump_meta() and dump() to use 'with' statements
for all file opens.
Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Adrian Reber <areber@redhat.com>
Use 'with' statement instead of open().read() to ensure the
file handle is properly closed.
Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Adrian Reber <areber@redhat.com>
Convert inf(), outf(), and dinf() to context managers and update
all call sites to use 'with' statements. For inf() and outf(),
only close the handle when it is a real file, not sys.stdin or
sys.stdout.
Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Adrian Reber <areber@redhat.com>
The service socket now defaults to 0600 (owner-only). Since the
test runs the setuid criu binary as a non-root user (uid 1000),
the socket ends up owned by root. Chown the socket to the test
user after service startup so it can connect, mirroring what an
administrator would do in production.
Assisted-by: Claude Code (claude-opus-4-6):claude-opus-4-6@default
Generated with Claude Code (https://claude.ai/code)
Signed-off-by: Adrian Reber <areber@redhat.com>
Change the service socket permissions from 0666 (world-accessible)
to 0600 (owner-only) to prevent unauthorized local users from
connecting to the CRIU service socket.
Print a message informing the administrator that the socket is
restricted and that chmod should be used to widen access if needed.
Assisted-by: Claude Code (claude-opus-4-6):claude-opus-4-6@default
Generated with Claude Code (https://claude.ai/code)
Signed-off-by: Adrian Reber <areber@redhat.com>
Third-party GitHub Actions referenced by tags can change without a CRIU
patch if the upstream action repository moves or compromises a tag. That
is especially sensitive for actions that receive CI credentials such as
CODECOV_TOKEN or GITHUB_TOKEN.
Pin the remaining non-GitHub actions used by regular workflows to the
full commit IDs currently behind their version tags. Keep the version
tag in a comment next to each SHA so reviewers can see the intended
release and future updates remain explicit.
Leave GitHub-owned actions on version tags to match the existing
linux-next workflow policy, which already pins non-GitHub actions while
using tags for actions/* refs. This keeps the hardening focused on
third-party supply chain risk without forcing the repository-wide SHA
policy that would also require pinning GitHub-authored actions.
Fixes: #3068
Assisted-by: Codex:gpt-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Replace 'from .images import *' with an explicit import list to
prevent namespace pollution, as the images module does not define
'__all__'.
Assisted-by: Claude Code (claude-opus-4-6@default)
Signed-off-by: Adrian Reber <areber@redhat.com>
ctrl_dir_and_opt() builds comma-separated controller directory and
mount option strings by repeatedly appending via snprintf() into
stack-allocated buffers. The offsets (doff, ooff) are advanced by
the snprintf() return value, but snprintf() returns the number of
bytes that *would* have been written on truncation, not the number
actually written. When a controller name is long enough to cause
truncation, the offset advances past the buffer end. On the next
iteration, the remaining-length calculation (e.g. ds - doff) goes
negative and is implicitly cast to a large size_t, disabling the
truncation guard entirely. Subsequent snprintf() calls and the
trailing-comma removal then write out of bounds on the stack
(CWE-121: Stack-based Buffer Overflow).
A crafted cgroup.img with oversized cnames entries triggers this
during restore in any code path that calls ctrl_dir_and_opt():
prepare_cgns(), move_in_cgroup(), prepare_cgroup_properties(),
prepare_cgroup_sfd(), and cgroupd().
Fix this by checking every snprintf() call for truncation before
advancing the offset. Each append now verifies that remaining
space is positive and that snprintf() did not return a value
greater than or equal to the remaining space. On any overflow
the function returns -1 and all five callers now propagate this
error. Also change the type of `off` in prepare_cgroup_properties()
from unsigned int to int so the -1 return is not silently converted
to a large positive value.
This vulnerability was identified and analyzed by an AI security
analysis tool.
Assisted-by: claude-opus-4-6
Signed-off-by: Adrian Reber <areber@redhat.com>
A not-accountable VMA cannot hold restored pages. If such a VMA is
matched into the COW chain, a child can inherit a parent's PROT_NONE
reservation as the target for its restored memory.
Reject not-accountable VMAs when preparing COW relationships. This makes
the child restore its pages through a separate writable premap, while the
parent's PROT_NONE reservation can still be mapped without write access.
Signed-off-by: Andrei Vagin <avagin@google.com>
Signed-off-by: Bhavik Sachdev <b.sachdev1904@gmail.com>
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The asyncd worker threads added in this series call pr_*() concurrently
on restore, but vprint_on_level() formatted every line into a shared
static buffer, so their output could interleave and garble log lines.
Build each line on the stack instead. Only the small read-only pid
prefix ("%6d: ") stays in process state -- written once per task in
log_init_by_pid() before any asyncd worker starts. Emitted bytes are
unchanged.
Assisted-by: Claude Code:claude-opus-4-8
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
When restoring an anonymous shared memory VMA, CRIU uses a memfd if the
kernel supports it and otherwise falls back to an anonymous shared mapping
opened via map_files. Route both through the async fill daemon so the
content is filled in parallel with the rest of restore.
The map_files fallback fd of an anonymous shared mapping must not be
ftruncate()d, while a memfd must. Introduce restore_shmem_fd_content(),
which takes a truncate flag, and keep restore_memfd_shmem_content() as a
truncate=true wrapper. open_shmem() offloads the fill to the daemon and
passes truncate only for the memfd case.
As with memfd inodes, the content is filled inline when restoring from a
stream, where the daemon's out-of-band reads are not possible.
Co-developed-by: Dan Feigin <dfeigin@nvidia.com>
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Signed-off-by: Andrei Vagin <avagin@google.com>
memfds are restored as regular files, so their content can be filled in
parallel at the file level. Filling memfd data is the slowest part of
restore, and since the other restore steps do not depend on it, it can be
done asynchronously by a helper daemon.
Each restoring task creates its memfd and hands the descriptor to an
asynchronous daemon (asyncd) that fills the content (mmap + pread) from a
pool of worker threads. The daemon mirrors the usernsd design: tasks ship
work over a SEQPACKET socket via SCM_RIGHTS, and the content fill runs
out-of-band while the tasks keep restoring.
The daemon's worker threads must not hold a TID that the restorer later
needs. The restorer recreates each application thread at its original TID
via clone3(set_tid); if a daemon worker still occupies that TID the clone
fails with EEXIST ("Unable to create a thread: -17"). To guarantee the
daemon is gone before any application thread is cloned, a new restore
stage CR_STATE_PRE_RESTORER is introduced between CR_STATE_FORKING and
CR_STATE_RESTORE. The root task starts the daemon and switches all tasks
to PRE_RESTORER, during which the tasks ship their fills. The root task
then drains and reaps the daemon (stop_asyncd) before switching to
CR_STATE_RESTORE, where threads are cloned. Because the daemon is reaped
first, its worker TIDs are free by the time clone3(set_tid) runs.
Keeping the daemon inside the restored task tree (rather than forking it
from the coordinator) keeps the content fill charged to the restored
container's memory cgroup instead of to criu.
The fill-thread pool is sized to min(online_cpus, 16). stop_asyncd() fails
the restore if the daemon exits abnormally, so a failed content fill is
never silently lost, and it is idempotent so the restore paths can call it
unconditionally.
criu-image-streamer serves the image in a single sequential pass, which is
incompatible with the daemon's out-of-band reads, so the daemon is not
started when restoring from a stream and the content is filled inline
instead.
Co-developed-by: Dan Feigin <dfeigin@nvidia.com>
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Signed-off-by: Andrei Vagin <avagin@google.com>
Parallel memfd restore opens page images from multiple pthread workers
later in this series. Those paths can use bfd helpers, whose global
buffer pool was previously single-threaded.
Protect the shared buffer list around allocation, return, and buffer
extension paths. Add pthread linkage for the new restore-time pthread
users.
Assisted-by: Claude:claude-sonnet-4-6
Assisted-by: Codex:gpt-5
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Add a memfd mmap test that keeps many independent memfds alive across
checkpoint and restore, with shared and private mappings for each one.
Verify restored fd flags and offsets, mapping devices, shared mapping
contents, private COW contents, and post-restore shared/private write
semantics.
Assisted-by: Codex:gpt-5
Assisted-by: Claude Code:claude-opus-4-8
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Correct the documentation regarding `vmsplice` and `SPLICE_F_GIFT`
behavior. `SPLICE_F_GIFT` does not enforce Copy-on-Write (COW) when the
user-space process modifies the gifted memory. Instead, it is a
zero-copy mechanism where post-resume modifications can lead to
inconsistent intermediate dumps.
Explain how CRIU handles this inconsistency via its iterative design,
relying on the soft-dirty tracker to catch post-resume modifications and
re-dump them in subsequent iterations, ensuring final consistency.
Also emphasize that the `splice` scheme relies heavily on `vmsplice`
performance to minimize freeze time, making the migration almost
invisible to the process.
Signed-off-by: Andrei Vagin <avagin@google.com>
