This would help to understand restore logs of unix sockets.
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
(cherry picked from commit ae1bc673960ce7679527a7fb13cb5dcf8aaf5890)
Signed-off-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
Printing info message when we go to nested fds will make easier to
understand logs of unix sockets dumping.
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
(cherry picked from commit ae7cee5ef93efba90466bbf67d0e0367a05cc830)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
This allows us to catch long standing bug in vzcriu when scm-s were not
dumped properly - scm-s from second packet contained both scm-s from
first and second packet. This happenes in case there several SCM_RIGHTS
packets and at least one SCM_CREDENTIALS packets were sent to unix
socket at the same time.
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
(cherry picked from commit 65ea504a97506355c0a5a4d9612a4827a58d6631)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
When we have several SCM_RIGHTS packets and at least one SCM_CREDENTIALS
sent simultaneousely into same unix socket we see that SCM_RIGHTS are
dumped wrong. It happens because before this patch SkPacketEntry had
been reused when dumping consequent packets, thus when dumping
SCM_RIGHTS for non-first packet it's pe.scm also contained SCM_RIGHTS
for previous packet.
To fix this let's not reuse SkPacketEntry. Also while on it let's split
dump_sk_queue_packet() - separate function for dumping one packet, to
add more readability to the code. Also drop meaningless "ret = -E2BIG"
and staff. Note that after this change pe.ucred is also not reused as it
probably should've always been.
Fixes: e4956b0b1 ("sk-queue: Add support of ucreds in unix sockets")
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
(cherry picked from commit 6832ccd499c1bef9bc220cdf42202c468fe5a37c)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
Reproducer for the next problem:
When unix socket contains SCM with SCM_CREDENTIALS
cmsg_type we have to postpone saving this packet to
CR_FD_SK_QUEUES image because we need to have information
about process which creds was transfered in this message.
But if this SkPacketEntry already contains data from
SCM_RIGHTS messages (file descriptors was transfered
through unix socket) we will lose this data because of
incorrect queue_packet_entry function implementation.
This problem tends to inconsistent CRIU dump and on
restore we will get:
Error (criu/files.c:230): Empty list on file desc id 13(5)
Error (criu/files.c:232): BUG at criu/files.c:232
Reviewed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Alexander Mikhalitsyn <alexander.mikhalitsyn@virtuozzo.com>
(cherry picked from commit 6aa13217721db06b7cbd873e615580f0a455d5a7)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
If SkPacketEntry contains both SCM_CREDENTIALS and SCM_RIGHTS message
then only SCM_RIGHTS will be put into socket queue.
Reviewed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Alexander Mikhalitsyn <alexander.mikhalitsyn@virtuozzo.com>
(cherry picked from commit 1aa1b1cb430e39271e7fbd44bd8609198a7f3a20)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
When unix socket contains SCM with SCM_CREDENTIALS
cmsg_type we have to postpone saving this packet to
CR_FD_SK_QUEUES image because we need to have information
about process which creds was transfered in this message.
But if this SkPacketEntry already contains data from
SCM_RIGHTS messages (file descriptors was transfered
through unix socket) we will lose this data because of
incorrect queue_packet_entry function implementation.
This problem tends to inconsistent CRIU dump and on
restore we will get:
Error (criu/files.c:230): Empty list on file desc id 13(5)
Error (criu/files.c:232): BUG at criu/files.c:232
Reviewed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Alexander Mikhalitsyn <alexander.mikhalitsyn@virtuozzo.com>
(cherry picked from commit e6a9ce32ad14d18ff3160e5339e4582885d740fe)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
The queue may have a dead process' data in such case we rather
should ignore the packet.
Signed-off-by: Cyrill Gorcunov <gorcunov@virtuozzo.com>
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
(cherry picked from commit f854da2284c2c0163055322804aa616b17a34c28)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
For debug sake.
Signed-off-by: Cyrill Gorcunov <gorcunov@virtuozzo.com>
(cherry picked from commit 318009a0ddf7150e7967f2dc52752cbb469cacd2)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
A process may send creds message and get exited
so the message will be detected on checkpoint
but we won't be able to queue it back because
process is already dead and such sending is
forbidden by kernel. So just ignore such
packets.
Signed-off-by: Cyrill Gorcunov <gorcunov@virtuozzo.com>
(cherry picked from commit e0e73bcbb8e07e60dbfd4e23cebe2f477a930704)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
Checkpoint and restore ucreds on unix sockets are not
that trivial: the @pid returned by the kernel upon
message fetching is mapped to current user namespace
so it is not the same as it would be received inside
container. So we need to map real pid returned into
virtual seen from inside, but we know virtual pids
only when complete process tree is built.
Thus what we do -- queue such packets into own
queue and when process tree is build walk over
and transform former pids into virtual ones.
Signed-off-by: Cyrill Gorcunov <gorcunov@virtuozzo.com>
(cherry picked from commit 8f3061c360381894a97edca2ada85466f95e1d88)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
A socket may support receiving timestamps from the
kernel (see net/socket.c:__sock_recv_timestamp
in the kernel), so simply pass up this data to
userspace.
Signed-off-by: Cyrill Gorcunov <gorcunov@virtuozzo.com>
(cherry picked from commit 1ecd757bf23831d1574e1dca0566dbe26e0fcb49)
Ahmed Elaidy:
- ported as a standalone commit (was previously squashed into the
unix-sockets ucred support); only the SOL_SOCKET timestamp handling
is kept, the netlink pktinfo part of the original is not ported here.
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
When error happened in dump_sk_creds or dumping
in dump_sk_queue passed fine we can leak xmalloc'ed
SkUcredEntry. Fix it.
Note it's not critical anyhow since CRIU will end up
execution in both cases and OS release memory
automatically but still.
Signed-off-by: Cyrill Gorcunov <gorcunov@virtuozzo.com>
(cherry picked from commit 969a2aba0331ff940501205077766859453b9a35)
Signed-off-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
ucred.pid is zero for kernel messages and in this case it is
impossible to restore it.
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@virtuozzo.com>
(cherry picked from commit 70948c510699aa8a2d28fc1669ad25dd81a6b102)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
We need to know a type of unsupported control messages.
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@virtuozzo.com>
(cherry picked from commit 4ab24ee8840b4d879147442f4c65bb7eac77a8bb)
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
ucred in a netlink message contains the same pid and
it doesn't metter from which pidns it is read.
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@virtuozzo.com>
(cherry picked from commit 66f8f7b2ae51a2cef4044df478389137735e1d6e)
Ahmed Elaidy:
- dropped the ENOBUFS bits (SK_QUEUE_TRACK_ENOBUFS and the int *val
out-parameter): they belong to "netlink: save and generate ENOBUFS
at restore", which needs a non-upstream kernel extension and is
intentionally not ported.
- SK_QUEUE_DUMP_ADDR / the int flags signature come from the preceding
"sk-queue: use flags in dump_sk_queue" cherry-pick rather than being
glued in here.
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
(cherry picked from commit 8b83405614703211fee27f90f08450cba080a44c)
Ahmed Elaidy:
- adapted onto the upstream 2-arg dump_sk_queue(): upstream has no
netlink queue dump, so there was no bool dump_addr argument to
convert; introduce int flags on the 2-arg base and update the only
caller (unix sockets). No sk-netlink.c change for the same reason.
Signed-off-by: Ahmed Elaidy <elaidya225@gmail.com>
Reviewed-by: Alexander Mikhalitsyn <aleksandr.mikhalitsyn@futurfusion.io>
The benchmark and its five data generators otherwise remain standalone
tooling, so normal CI would not notice broken integrity checks,
statistics accounting, or compression configuration.
Add a bounded smoke run for zero, mixed, pseudorandom, text, and
ELF-derived mappings. Compare uncompressed, per-page LZ4, and 64 KiB
region images with one warmup and one measured iteration, print the
results, and fail on checksum, accounting, or configuration errors.
Run the helper tests first, then build the pycriu bindings needed by the
live benchmark. Leave the host page cache untouched and enforce a
five-minute timeout. Gate the run on compression support.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Compression can fall back to raw storage for every block in an entry.
The writer then omits compression metadata so restore can use ordinary
I/O, but that fast path also depends on a page-aligned payload.
Add per-page and region tests with an incompressible mapping. Inspect
the pagemap for an aligned 56-page entry without compression metadata,
while requiring another genuine LZ4 block to prove compression was
active. Restore through direct image I/O, require native AIO when
O_DIRECT is available, and verify the workload contents.
Run the test in CI when page compression is available.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Successive layers in an incremental dump can use different compression
modes. Restore must interpret each layer through its own inventory and
pagemap metadata instead of applying the final image mode to the parent
chain.
Add chains with two pre-dumps and a final dump for plain-to-page,
page-to-plain, plain-to-region, and region-to-plain transitions. Toggle
a page between layers so every image carries real payload, check the
exact inventory version and compression settings, and restore the ZDTM
workload to verify its memory contents.
Run the test in CI when both page compression and memory dirty tracking
are available.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Benchmark output can look credible even when command construction,
configuration recovery, archive inspection, accounting, or cleanup has
drifted. Full Podman and model-serving trials are too expensive for
focused checks of those helpers.
Add fast tests that load both serving frontends against their shared
implementation while keeping their mutable state separate. Cover CRIU
option isolation, explicit automatic decompression, port-derived URLs,
inventory validation, runc.conf locking and crash recovery, metadata
preservation, secret redaction, container failure detection, response
integrity, signal handling, and cleanup.
Also verify deterministic workload generation, report normalization,
cache-control failures, and the pagemap evidence required by the
region-cache benchmark. Add the compression benchmark tree to ruff and
provide a dedicated unit target.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Serving engines have different memory layouts and restore behavior, so
SGLang results do not establish the cost or correctness of compression
for vLLM.
Add a vLLM frontend to the shared Podman benchmark. Support CPU and GPU
images and the supported server entrypoint forms, compare uncompressed,
per-page LZ4, and region LZ4 images, and verify deterministic inference
after restore. Report archive, checkpoint, restore, and post-restore
request costs, and add a bounded CPU test runner.
Reuse the shared runc.conf transaction, CRIU configuration isolation,
inventory validation, secret redaction, failure diagnostics, and
cleanup semantics so both serving benchmarks follow the same
methodology.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Process-only measurements do not show how compressed pages affect a
containerized inference server, where Podman configuration, model
startup, and service readiness are part of the restore path. SGLang
also needs content validation so a quick restore cannot hide damaged
state.
Add a CPU and GPU SGLang benchmark that checkpoints through Podman.
Compare uncompressed, per-page LZ4, and region LZ4 images. Send the same
deterministic chat request before and after restore, compare response
hashes, measure archive and operation costs, and verify the compression
mode and region size recorded in the exported inventory. Derive the
local serving URL from the selected port unless one is supplied.
Podman passes CRIU options through the host-wide runc.conf. Serialize
updates with a canonical lock and crash-recovery journal, preserve file
metadata and xattrs, refuse to overwrite external changes, and restore
the original configuration on success and during error or signal
cleanup. Isolate CRIU from ambient defaults, keep access tokens out of
arguments and diagnostics, detect stopped containers during health
checks, and preserve failed artifacts. Add a bounded CPU test runner.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Incremental restore can request alternating pages from one compressed
parent region. Without reuse, each partial request decompresses the same
block again, but a timing result is meaningful only if the images really
create that access pattern.
Add an A/B benchmark that takes a compressed pre-dump, dirties
alternating pages, and compares restore with two CRIU binaries. Publish
the workload mapping address and inspect both pagemaps before restore.
Require the requested region size and a genuine LZ4 parent block.
Require alternating present and parent pages plus at least two parent
slices from the same block.
Record that evidence with each trial, verify the restored mapping with
SHA-256, alternate binary order, and separate warmups from measured
iterations. Emit both machine-readable data and a Markdown summary.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Compression effectiveness depends on page contents and block size,
while dump and restore costs can move independently of image size. The
series needs a repeatable baseline that measures those trade-offs and
rejects a fast result if restored memory is wrong.
Add a root-only benchmark for zero, mixed, pseudorandom, text, and
ELF-derived mappings. Compare uncompressed images with per-page LZ4 and
a sweep of region sizes, rotate configuration order after a warmup, and
report median storage, latency, and throughput from required CRIU
statistics. Verify every restored mapping with SHA-256 and preserve
artifacts when a trial fails.
Bound workload startup and cleanup, handle termination signals, and
make cache dropping an explicit, checked part of the methodology. Leave
CRIU's compressed-page worker width unchanged unless requested,
preserve an explicit automatic value of zero, and generate deterministic
random input on Python versions before Random.randbytes(). Support JSON
output for longer external runs.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
CRIT can edit image metadata but cannot convert the matching page
payloads as one consistent image set. Changing either side alone can
leave pagemap, inventory, and payload data incompatible.
Add compress and decompress commands for checkpoint directories. Match
CRIU's block encoding, raw threshold, alignment, image version, parent
compatibility, and validation rules. Keep hugetlb and external-plugin
ranges raw because CRIU cannot premap them generically, and add the
Python LZ4 binding to supported dependencies.
Stage every output before replacing any image. Preserve ownership,
permissions, timestamps, xattrs, ACLs, and security metadata; create
exclusive hard-link backups; synchronize directory changes; and defer
terminating signals until the transaction has a definite result. Roll
back the complete image set after a failure and retain the recoverable
source if rollback or cleanup also fails.
Reject symlinks, non-regular images, unsupported versions, inconsistent
compression metadata, unknown modes, and truncated payloads. Recheck
source identities before replacement so a changed pathname is not
overwritten. Cover page and region images, parent chains, exceptional
mappings, metadata preservation, signals, races, and rollback failures.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
End-to-end restore tests do not isolate codec behavior or worker-pool
decisions, which makes boundary failures difficult to diagnose.
Test per-page and region round trips for zero, repeating, pseudorandom,
and single-byte inputs across several acceleration values and region
sizes. Verify zero detection, deterministic raw fallback, and page-image
offset alignment above 4 GiB.
Exercise serial and parallel LZ4 decoding and zero filling,
deterministic 0/1/N thread selection, effective-capacity gating,
work-based automatic width, growth of a persistent pool, initialization
of the shared batch budget, and return of the CPU budget. Also exercise
nonblocking acquisition and release of the second encoded input slot.
Verify that serial decoding skips caller work, while a parallel batch
runs it once after dispatch. Link the tests only when LZ4 is available.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Existing memory tests do not deliberately produce every compression
storage form or cross compression-specific batch and transport
boundaries. Regressions in zero, LZ4, raw, and mixed handling could
therefore go unnoticed.
Add neutral pages_content workloads and use them through symlinked
per-page and region-compressed variants. The same contents now run
uncompressed in ordinary sweeps, while NO_LZ4 builds skip only the
compressed variants.
Exercise compressible, incompressible, zero, mixed, file-backed, COW,
and exceptional mappings. Keep a large private zero range over nonzero
file contents so automatic restore exercises parallel zero filling when
the host has at least two available CPUs, and cannot pass by leaving an
untouched zero mapping. Size the compressible mapping at 33 MiB so
restore handles one full 32 MiB batch followed by 1 MiB of useful work.
This crosses the batch boundary with enough work to exercise read/decode
overlap. Cover pre-dump, deduplication, lazy pages, stream, direct I/O,
page server, RPC, region boundaries, incremental parent chains, and
repeated batches beyond 32 MiB. Verify restored contents in every case.
Pass both syntaxes for restore-only decompression options through the
ZDTM RPC runner. Exercise automatic concurrency in per-page and region
variants; the remaining variants retain the serial default.
Suggested-by: Andrei Vagin <avagin@google.com>
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The compression paths have no supported user or RPC controls, and
clients cannot discover whether CRIU was built with LZ4.
Expose per-page and region compression, LZ4 acceleration, and worker
concurrency for LZ4 decoding and eligible large zero fills through the
CLI, configuration files, RPC, and libcriu. Add compression to the
feature-check API and document each interface.
Define --decompress-threads=0 as automatic worker concurrency and one as
the serial default with zero-fill workers disabled. Values above one
bound aggregate worker concurrency. Automatic mode derives a useful
width from CPU affinity and the work in each batch. Reduce explicit
requests above the available CPU count with a warning.
Document the independent two-slot encoded-input budget. A local restore
overlaps its next read only when it can reserve the second slot without
waiting; otherwise it keeps synchronous I/O.
Use the compression mode as the source of truth and validate every
related value. An explicit RPC OFF clears ambient compression settings
and rejects conflicting fields. Reject combinations unsupported by page
server, stream, TLS, and image-format paths, and cover option and RPC
precedence in the tests.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Compression changes the pages-image payload layout. Older CRIU versions
ignore optional pagemap metadata and would interpret packed bytes as
ordinary pages.
Introduce image version V1.2 for compressed page payloads and reject it
when CRIU lacks LZ4 support. Validate compression and version
combinations when reading inventories. Propagate V1.2 through parent
chains, including an uncompressed child of a compressed parent, while
independent uncompressed images retain the previous version.
Reject an incremental dump when its configured parent inventory is
unreadable or unsupported instead of risking a version downgrade. Keep
accepting legacy parents without dump uptime when pidfds can detect PID
reuse.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The PIE restorer cannot link against LZ4. Decode mappings containing
LZ4 payloads through CRIU's premap path, while retaining delayed PIE
restore for raw and zero ranges when a worker pool cannot help.
Classify image ranges as LZ4, raw, or zero. Decode LZ4 blocks while CRIU
still runs in its normal address space, read aligned raw ranges directly
into their final VMAs, and clear zero ranges in place. Omit compression
metadata for entries containing only raw blocks so they retain the
ordinary restore path.
Share one encoded-read context across each incremental parent chain and
reuse its worker pool for bounded batches, COW reads, streamed images,
and page-server chunks. Use the same jobs to fill zero runs of at least
1 MiB when two blocks and two CPUs are available. Zero-only ranges that
cannot benefit retain the PIE path, while zero blocks in an existing
encoded batch remain inline without effective parallel capacity. Hold
input buffers only while encoded work is active, and join every worker
before child restore or the PIE handoff.
Account calling restore threads and workers against a shared CPU budget.
A requested width of zero selects automatic concurrency, one keeps LZ4
decoding serial and disables zero-fill workers, and larger values cap
aggregate worker concurrency. Bound the active width by CPU affinity and
useful work. Release encoded-work leases before ordinary reads so idle
contexts cannot block ready workers.
Local payload-heavy restores otherwise alternate between reading an
encoded batch and decoding it. This leaves either storage or worker CPUs
idle. When a second encoded-input lease is immediately available,
dispatch the current batch to the pool and use the already-accounted
calling thread to read the next adjacent payload. Keep synchronous reads
for serial restore, streams, direct I/O, small batches, and contended
leases.
Select direct AIO from the ranges actually delayed for PIE. Batch
inherited COW comparisons and streamed payload reads, and align
page-server chunks to compression regions. Validate metadata and offset
arithmetic before allocation or I/O, roll back partially enqueued work,
and propagate worker, I/O, and deferred deduplication failures before
remapping.
On a 256 MiB pseudorandom workload with the worker limit set to eight
and three measured runs, median restore time was 114.5 ms uncompressed,
113.4 ms with LZ4 per page, and 116.5 ms with 64 KiB regions. The
previous compressed restore path took about 227 ms in either mode.
With automatic concurrency and 256 KiB regions, parallel zero filling
reduced median restore time over five runs from 103.1 ms to 31.7 ms for
zero-filled memory and from 89.3 ms to 60.0 ms for mixed memory.
On 512 MiB TEXT and ELF workloads with automatic concurrency and 256 KiB
regions, five-run median restore time fell from 117.0 ms to 110.7 ms and
from 115.4 ms to 109.5 ms, respectively.
Assisted-by: Codex:GPT-5
Assisted-by: Claude:claude-fable-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
A coalesced compressed read can allocate buffers approaching the size
of a large checkpoint and issue readahead across large, unrelated gaps.
Limit each asynchronous batch to 32 MiB of decoded pages. Split
readahead ranges when adjacent entries are more than 1 MiB apart or
their span would exceed 256 MiB. This bounds compressed input, metadata,
the decoded destination, and each individual readahead request.
Keep a split logical read transactional. If a later chunk fails, remove
newly queued requests and restore any pre-existing tail that an earlier
chunk extended.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Incremental restore can request alternating pages from one compressed
parent region. Each partial request currently reads and decompresses the
complete region again, multiplying both I/O and CPU work.
Cache the most recently decoded region in each page reader, keyed by its
virtual address and decoded size. Keep the cache across cursor advances
and resets, while duplicate readers start with independent empty caches.
A reader's pages image is immutable, so this key remains valid.
A 512 MiB mapping with 1 MiB regions was pre-dumped and then modified on
alternating pages. Across five measured restores after one warmup, the
median fell from 18.318 seconds to 0.359 seconds, a 51.1x speedup, while
every restored checksum matched.
Suggested-by: Andrei Vagin <avagin@google.com>
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Existing pagemap readers derive each pages-image offset from a fixed
PAGE_SIZE payload per present page. Packed compressed blocks make that
calculation invalid and must be reconstructed before reaching restore
destinations.
Track block and intra-region cursors while advancing, skipping, and
reading pagemap entries. Validate block counts, decoded sizes, totals,
and aligned-payload offsets before I/O. Read per-page and region images
locally, preserve ordinary parent entries, and support sequential
per-page image streams.
Carry block metadata with queued iovecs, coalesce only compatible
regions, read each packed batch once, and rebuild its destinations block
by block. Allow exactly IOV_MAX destinations and roll back iovec changes
if appending metadata fails. Decompress complete regions directly into
a contiguous destination and use scratch space for partial or split
regions. Copy the metadata into restorer memory for the later PIE decode
path.
Keep compressed images on buffered I/O, avoid hole punching packed
payloads, handle short reads, and keep compressed local reads out of PIE
until that path can decode them.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
struct page_read combines image cursors, parent traversal state, and
callbacks for several restore paths. Its terse field comments make
changes to that state difficult to review safely.
Describe each callback and cursor before compressed readers extend the
structure. Make no functional change.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Present pages are currently written as fixed PAGE_SIZE payloads. To
store packed compressed data, page transfer must choose a representation
for each block and record its size before publishing the pagemap entry.
Write zero, raw-fallback, and LZ4 blocks for individual pages or fixed
regions. Delay present pagemap entries until their block metadata is
complete, and keep region entries within one VMA so restore never splits
an LZ4 block at a VMA boundary.
Align entries whose first stored block is raw so direct restore does not
pay for mismatched file and memory page phases. Track padding in the
pages-image offset and keep variable-length compressed output on
buffered I/O.
Add a per-page compressed page-server command that sends each block size
with its payload. Validate its flags and ranges, preserve auto-dedup
ordering, and store the received representation without recompressing
it. Handle short pipe, file, and socket I/O throughout the new path.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Page-server cleanup can run after partial setup and more than once.
Partially initialized pipe arrays can still contain fd 0, while a
retained destination ID can close descriptors again after their numbers
have been reused. Setup failures and completed sessions can also leave
transfer pipes open.
Initialize pipe descriptors to -1 and clear the destination ID before
closing its image. Check pipe initialization and size queries, and close
transfer pipes when setup fails or a session completes.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The page-image writer and restore reader need a shared compression
policy before user-facing interfaces can select it.
Add internal options for the compression mode, region size, LZ4
acceleration, and worker concurrency for LZ4 decoding and large zero
fills. This gives later writer and reader changes one process-wide
source of configuration while leaving CLI and RPC exposure to a
separate change.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Compressed page writers and readers need one definition of the block
format and its size limits. Duplicating those rules at each call site
would make zero, raw-fallback, and LZ4 handling easy to diverge.
Add common LZ4 helpers, zero-page detection, and shared bounds and
storage thresholds for per-page and region users. Region helpers encode
zero blocks as size zero, fall back to raw bytes when compression saves
too little, and decode all three representations. Build the
implementation only when CONFIG_LZ4 is enabled.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Present pagemap entries currently imply one PAGE_SIZE payload per page.
That is not enough to locate packed zero, raw, and LZ4 blocks or to
describe region compression.
Record the compression mode and region size in the inventory. Add
packed per-block sizes, their total, and the region width to pagemap
entries, plus PE_PAYLOAD_ALIGNED for entries whose payload starts after
padding. Keep the fields optional so ordinary and older images retain
their existing representation.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Memory page compression requires liblz4, but making it mandatory would
prevent CRIU from building on systems that do not provide it.
Detect liblz4 with pkg-config and enable CONFIG_LZ4 only when it is
available. Add NO_LZ4=1 as an explicit opt-out and register the
development package in the supported distribution dependency lists.
Install liblz4 for the per-commit build, then run a separate NO_LZ4
build and unit test to keep the optional configuration usable.
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The kernel for the github actions 26.04 images fixed the kernel to no
longer have the bug that lead to the `socket-tcbuf*` failures.
Signed-off-by: Adrian Reber <areber@redhat.com>
During CI preparation, tests running inside user namespaces or
unprivileged hooks require path traversal execute permissions from
the root directory down to the current working directory.
Signed-off-by: Andrei Vagin <avagin@google.com>
When checkpointing tasks confined by stacked AppArmor profiles inside an
AppArmor namespace, CRIU compiles a permissive parasite profile using
apparmor_parser and writes it to the namespace policy replacement
interface
(/sys/kernel/security/apparmor/policy/namespaces/<ns>/.replace).
Previously, get_suspend_policy() invoked apparmor_parser with flags
(-QWL) requesting cache file generation and read the resulting cache
file. On modern AppArmor 4.x kernels, cache files include cache metadata
headers that cause the kernel policy unpacker to reject the binary blob
with EPROTONOSUPPORT (-93, "invalid profile format"). Replace the cache
mechanism by invoking apparmor_parser with the -o flag to output a clean
binary policy stream directly.
Furthermore, when writing a replacement profile blob to an AppArmor
.replace interface, successful writes return the total size of the
profile load data structure consumed or updated in the kernel (e.g.,
5233 bytes for stacked profiles), rather than the exact byte length of
the input payload (e.g., 727 bytes for a parasite profile). Checking if
(n != len) erroneously treats a successful replacement write as a
failure. Fix policy write validation to check if (n < 0).
Across Linux kernel versions, AppArmor enforces different requirements
for policy replacement blobs written to a namespace .replace interface:
- Older kernels (AppArmor 3.x / early 4.x on Ubuntu 22.04) expect blobs
compiled without namespace headers (-n <ns>); specifying -n causes
the kernel to load the replacement into a nested sub-namespace
(<ns>//<ns>), leaving the active profile unchanged and causing socket
permission failures during dump.
- Modern kernels (AppArmor 4.x on Ubuntu 26.04) reject blobs compiled
without namespace headers with EPROTONOSUPPORT (-93, "invalid profile
format").
Update get_suspend_policy() and write_aa_policy() to implement runtime
fallback: first try compiling without -n <ns> and writing directly to
the namespace interface for older kernels. If writing fails (n < 0),
retry compilation passing -n <ns> and write to the root interface to
satisfy modern kernels.
In addition, when restoring tasks confined by profiles inside an
AppArmor namespace on AppArmor 4.x kernels, the host root unconfined
profile remains stacked with the target namespace profile when viewed
from the root namespace (e.g., "unconfined//&:ns:profile"). Update
checkprofile() in the apparmor_stacking test to recognize and allow the
"unconfined//&" prefix when verifying restored profiles.
Assisted-by: Gemini
Signed-off-by: Andrei Vagin <avagin@google.com>
In file included from compel/plugins/std/infect.c:14:
compel/include/uapi/compel/asm/sigframe.h:17:8: error: redefinition of 'gcs_context'
17 | struct gcs_context {
| ^
/usr/include/aarch64-linux-gnu/asm/sigcontext.h:188:8: note: previous definition is here
188 | struct gcs_context {
| ^
Signed-off-by: Andrei Vagin <avagin@google.com>
Add an --image-io-mode passthrough to zdtm.py, set on both dump and restore
like --stream and translated into the RPC request like --pre-dump-mode. Run
maps04 with --image-io-mode=direct and maps00 with --image-io-mode=direct --rpc
in run-ci-tests.sh so the O_DIRECT dump/restore paths and the RPC handler are
exercised.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Complete --image-io-mode=direct on the dump side: for a local, non-streaming
dump, set O_DIRECT on the pages image fd in open_page_xfer() so the existing
splice() in write_pages_loc() performs direct I/O. Otherwise the dump leaves the
written pages dirty in the cache, so a same-host direct restore still has to
flush them and a large checkpoint pollutes the dump host's cache.
splice() to a file goes through iter_file_splice_write() -> ->write_iter, which
honours O_DIRECT on filesystems that support it (ext4, xfs). Where it is not
supported splice() fails with EFAULT (or EINVAL); write_pages_loc() clears
O_DIRECT, warns, and retries the remainder buffered. Only whole pages are
transferred, so the file offset stays block-aligned across the switch and the
bytes written match the buffered path, so images restore in either mode. This
keeps the transfer zero-copy: no bounce buffer and no kernel-version check.
The page server and image streaming stay buffered.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
