struct parasite_ctl, parasite_thread_ctl, and plain_regs_struct
all contain user_fpregs_struct_t (typedef for struct xsave_struct),
which is declared with __aligned(64). When these structs are
allocated on the heap with xmalloc/xzalloc (i.e. malloc/calloc),
the allocator only guarantees 16-byte alignment on x86_64.
The compiler, seeing the __aligned(64) attribute on the struct
type, may emit aligned memory instructions (e.g. movaps, vmovdqa)
for struct copies, assuming the memory is properly aligned. When
the heap pointer is not 64-byte aligned, these instructions trigger
a General Protection Fault (#GP).
This was observed as a crash in save_regs_plain() at infect.c:1314
(prs->regs = *r) on CentOS Stream 10 under QEMU/KVM, where the
compiler generated aligned vector instructions for the struct copy.
The crash did not reproduce on bare metal with a different compiler
version that happened to emit unaligned instructions.
Add xmemalign() wrapper around posix_memalign() to xmalloc.h and
use it for all three allocation sites:
- compel_prepare_noctx(): struct parasite_ctl
- compel_prepare_thread(): struct parasite_thread_ctl
- compel_prepare(): struct plain_regs_struct
Memory from posix_memalign() can be freed with free(), so no
changes to cleanup paths are needed.
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Adrian Reber <areber@redhat.com>
Update the state machine to track entry and exit stages for all
syscalls, not just the target one. This allows skipping
ptrace_get_regs() calls on the exit stage of syscalls.
Signed-off-by: Andrei Vagin <avagin@google.com>
compel_stop_on_syscall is only ever called for a single task. Simplify
its implementation by removing the 'tasks' parameter and the associated
logic for handling multiple tasks. It now takes a 'pid' directly and
waits for that specific task to exit the required syscall.
Signed-off-by: Andrei Vagin <avagin@google.com>
Hardware breakpoints were originally intended to speed up the resume process
by stopping the process at a specific point in the pie code. However, it
turned out that they don't provide a significant speedup and, in some cases,
can even slow it down. This is especially critical for hosts with a large
number of CPUs.
Hardware Breakpoint Restore Performance Benchmark
==================================================
System: Linux 6.17.0-1007-gcp x86_64
CPU: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz
Virt: google
CRIU: Version: 4.2
Iterations per data point: 5
Threads With BP (us) Without BP (us) Diff (%)
-------- ------------ --------------- --------
>>> Benchmarking with 1 thread(s)...
1 391 326 19.9%
>>> Benchmarking with 10 thread(s)...
10 1098 695 58.0%
>>> Benchmarking with 50 thread(s)...
50 3772 2344 60.9%
>>> Benchmarking with 100 thread(s)...
100 6740 4504 49.6%
>>> Benchmarking with 500 thread(s)...
500 31382 19982 57.1%
>>> Benchmarking with 1000 thread(s)...
1000 58617 40568 44.5%
Notes:
'With BP' = hardware breakpoints enabled (current default)
'Without BP' = CRIU_FAULT=130 (FI_NO_BREAKPOINTS, uses PTRACE_SYSCALL)
Positive diff% means breakpoints are slower
Signed-off-by: Andrei Vagin <avagin@google.com>
Splitting the ptrace interrupt and wait loops allows us to send the
interrupt to all tasks and threads first, so they can start stopping in
parallel while we're still sending interrupts to the remaining ones.
This should improve restore performance for large process trees.
The redundant parse_threads calls in catch_tasks are removed as threads
are already parsed in attach_to_tasks which is called earlier.
Signed-off-by: Andrei Vagin <avagin@google.com>
When CRIU traces a large number of tasks, calling wait4(-1, ...) can
be slow because the kernel has to iterate over many tasks to find one
that has changed state. This overhead becomes significant when
thousands of tasks are involved. The time complexity of the old approach
was O(n^2) because we called wait4(-1, ...) N times, and each call
took O(N) time in the kernel.
This patch introduces compel_stop_tasks_on_syscall which takes an
array of PIDs and waits for each one specifically. This avoids the
O(N) search in the kernel's do_wait implementation and significantly
improves performance during the restore phase when many tasks are
being resumed and stopped at the rt_sigreturn syscall.
Some parallelism still exists in this approach. The loop waits for each
process to stop on a syscall, verifies it, and then lets it continue
running. While we move to wait for the next process in the array, the
previously processed ones continue their execution in parallel.
The performance profile of the bottleneck showed:
- 70.45% entry_SYSCALL_64
- do_syscall_64
- 68.29% __do_sys_wait4
- kernel_wait4
- 68.28% do_wait
- 68.26% __do_wait
- 65.69% wait_consider_task
8.71% _raw_spin_lock_irq
2.39% _raw_spin_unlock_irq
Signed-off-by: Andrei Vagin <avagin@google.com>
After the CRIU process saves the parasite code for the target thread in
the shared mmap, it is necessary to call __clear_cache before the target
thread executes the code.
Without this step, the target thread may not see the correct code to
execute, which can result in a SIGILL signal.
For the specific arm64 case. this is important so that the newly copied
code is flushed from d-cache to RAM, so that the target thread sees the
new code.
The change is based on commit 6be10a2 by @fu.lin and on input received
from @adrianreber.
[ avagin: tweak code comment ]
Signed-off-by: Ignacio Moreno Gonzalez <Ignacio.MorenoGonzalez@kuka.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
The stack test incorrectly assumed the page immediately
following the stack pointer could never be changed. This doesn't work,
because this page can be a part of another mapping.
This commit introduces a dedicated "stack redzone," a small guard region
directly after the stack. The stack test is modified to specifically
check for corruption within this redzone.
Signed-off-by: Andrei Vagin <avagin@gmail.com>
PAC stands for Pointer Authentication Code. Each process has 5 PAC keys
and a mask of enabled keys. All this properties have to be C/R-ed.
As they are per-process protperties, we can save/restore them just for
one thread.
Signed-off-by: Andrei Vagin <avagin@google.com>
criu_run_id will be used in upcoming changes to create and remove
network rules for network locking. Instead of trying to come up with
a way to create unique IDs, just use an existing library.
libuuid should be installed on most systems as it is indirectly required
by systemd (via libmount).
Signed-off-by: Adrian Reber <areber@redhat.com>
1. os auto assignment vma addr maybe conflict with vma in gpu living migrate scene;
2. so, we should give choice to user;
Signed-off-by: haozi007 <liuhao27@huawei.com>
When calling sigreturn with CET enabled, the kernel verifies that the
shadow stack has proper address of sa_restorer and a "restore token".
Normally, they pushed to the shadow stack when signal processing is
started.
Since compel calls sigreturn directly, the shadow stack should be
updated to match the kernel expectations for sigreturn invocation.
Add parasite_setup_shstk() that sets up the shadow stack with the
address of __export_parasite_head_start as sa_restorer and with the
required restore token.
Signed-off-by: Mike Rapoport (IBM) <rppt@kernel.org>
All architectures create on-stack structure for floating point save area
in compel_get_task_regs() if the caller passes NULL rather than a valid
pointer.
The only place that calls compel_get_task_regs() with NULL for floating
point save area is parasite_start_daemon() and it is simpler to define
this strucuture on stack of parasite_start_daemon().
The availability of floating point save data is required in
parasite_start_daemon() to detect shadow stack presence early during
parasite infection and will be used in later patches.
Signed-off-by: Mike Rapoport (IBM) <rppt@kernel.org>
Starting the daemon is the first time we run code in the victim
using the parasite stack.
It's useful for testing to be able to infect the victim without starting
the daemon so that we can inspect the victim's state, set up stack
guards, and so on before stack-related corruption can happen.
Add compel_infect_no_daemon() to infect the victim but not start the
daemon and compel_start_daemon() to start the daemon after the victim
is infected.
Add compel_get_stack() to get the victim's main and thread parasite
stacks.
Signed-off-by: Younes Manton <ymanton@ca.ibm.com>
Breakpoints are used to stop as close as possible to a target system call.
First, we don't need it after this point.
Second, PTRACE_CONT can't pass through a breakpoint on arm64.
Signed-off-by: Andrei Vagin <avagin@gmail.com>
When delivering system call traps, set bit 7 in the signal number (i.e.,
deliver SIGTRAP|0x80). This makes it easy for the tracer to distinguish
normal traps from those caused by a system call.
Signed-off-by: Andrei Vagin <avagin@gmail.com>
Add SIGTSTP signal dump and restore. Add a corresponding field
in the image, save it only if a task is in the stopped state.
Restore task state by sending desired stop signal if it is present
in the image. Fallback to SIGSTOP if it's absent.
Signed-off-by: Yuriy Vasiliev <yuriy.vasiliev@openvz.org>
CRIU has a few places where it creates unix sockets and their names have to be
unique for each criu run.
Fixes: #1798
Signed-off-by: Andrei Vagin <avagin@google.com>
This is the workaround for #1429.
The parasite code contains instructions that trigger SIGTRAP to stop at
certain points. In such cases, the kernel sends a force SIGTRAP that
can't be ignore and if it is blocked, the kernel resets its signal
handler to a default one and unblocks it. It means that if we want to
save the origin signal handle
Signed-off-by: Andrei Vagin <avagin@gmail.com>
My editor (vim) auto-removes whitespace at EOL for *.c and *.h files,
and I think it makes sense to have a separate commit for this, rather
than littering other commits with such changes.
To make sure this won't pile up again, add a line to Makefile under
the linter target to check for such things (so CI will fail).
This is all whitespace except an addition to Makefile.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Arch-dependend way to restore extended registers set.
Use it straight-away to restore per-thread registers.
Signed-off-by: Dmitry Safonov <dima@arista.com>
Extended registers set for task is restored with rt_sigreturn() through
prepared sigframe. For threads it's currently lost.
Preserve it inside thread context to restore on thread curing.
Signed-off-by: Dmitry Safonov <dima@arista.com>
To minimize things done in parasite, PTRACE_GET_THREAD_AREA can be
used to get remote tls. That also removes an additional compat stack
(de)allocation in the parasite (also asm-coded syscall).
In order to use PTRACE_GET_THREAD_AREA, the dumpee should be stopped.
So, let's move this from criu to compel to non-seized state and put tls
into thread info on x86.
Signed-off-by: Dmitry Safonov <dima@arista.com>
We've seen addresses in parasite.built-in.o precalculated by linker but
in some unexpected manner:
readelf -WS criu/pie/parasite.built-in.o
Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
[ 1] .text PROGBITS 0000000000000000 000040 00400a 00 AX 0 0 16
[87] .data PROGBITS 0000000000000000 005000 000068 00 WA 0 0 4096
[88] .rodata PROGBITS 0000000000000080 005080 001016 00 A 0 0 32
(Notes: All other sections does not have SHF_ALLOC or are of size 0, so I
skip them. Need to add "-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1" to
CFLAGS to reproduce.)
Section 88 has address 0x80 in elf file but in our "consequent"
addresses precalculation algorithm it should be at 0x5080:
addr(.text) == 0x0
addr(.data) == 0x400a + (0x1000 - 0x400a % 0x1000) + 0x68 == 0x5068
addr(.rodata) == 0x5068 + (0x20 - 0x5068 % 0x20) == 0x5080
Probably the linker advises us to move 4096 aligned section to the
beginning to save some space, but it's just a guess.
So probably we should be ready to "non-consequent" alignments
precalculated and just override them.
https://github.com/checkpoint-restore/criu/issues/1301
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Using scan-build there is a warning about
infect.c:231:17: warning: The left operand of '!=' is a garbage value
if (ss->state != 'Z') {
which is a false positive as every process will have a 'Status' field,
but initializing the structure makes the clang analyzer silent.
Signed-off-by: Adrian Reber <areber@redhat.com>
Some kernels have W^X mitigation, which means they won't execute memory
blocks if that memory block is also writable or ever was writable. This
patch enables CRIU to run on such kernels.
1. Align .data section to a page.
2. mmap a memory block for parasite as RX.
3. mprotect everything after .text as RW.
Signed-off-by: Michał Cłapiński <mclapinski@google.com>
GNU ld precalculates this information but lld does not. With this
change, handle-elf.c calculates those addresses on its own.
When calculating addresses sections with SHF_ALLOC bit are put one after
another, respecting their alignment requirements. This matches the way
how the blob is constructed by copying section contents.
Signed-off-by: Wojciech Marczenko <marczenko@google.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
Previously, the GOT table was using the same memory location as the
args region, leading to difficult to debug memory corruption bugs.
We allocate the GOT table between the parasite blob and the args region.
The reason this is a good placement is:
1) Putting it after the args region is possible but a bit combersome as
the args region has a variable size
2) The cr-restore.c code maps the parasite code without the args region,
as it does not do RPC.
Another option is to rely on the linker to generate a GOT section, but I
failed to do so despite my best attempts.
Signed-off-by: Nicolas Viennot <Nicolas.Viennot@twosigma.com>
We don't need to push 0 on the stack. This seems to be a remnant of the
initial commit of 2011 that had a `pushq $0`.
The 16 bytes %rsp alignment was added with commit 2a0cea29 in 2012.
This is no longer necessary as we already guarantee that %rsp is 16
bytes aligned. A BUG_ON() is added to enforce this guarantee.
Signed-off-by: Nicolas Viennot <Nicolas.Viennot@twosigma.com>
Previously, __export_parasite_cmd was located in parasite-head.S, and
__export_parasite_args located exactly at the end of the parasite blob.
This is not ideal for various reasons:
1) These two variables work together. It would be preferrable to have
them in the same location
2) This prevent us from allocating another section betweeen the parasite
blob and the args area. We'll need this to allocate a GOT table
This commit changes the allocation of these symbols from assembly/linker
script to a C file.
Moreover, the assembly entry points that invoke parasite_service()
prepares arguments with hand crafted assembly. This is unecessary.
This commit rewrite this logic with regular C code.
Note: if it wasn't for the x86 compat mode, we could remove all
parasite-head.S files and directly jump to parasite_service() via
ptrace. An int3 architecture specific equivalent could be called at the
end of parasite_service() with an inline asm statement.
Signed-off-by: Nicolas Viennot <Nicolas.Viennot@twosigma.com>
It is unnecessary and potentially confusing for understanding the memory
layout requirement of the parasite blob.
Signed-off-by: Nicolas Viennot <Nicolas.Viennot@twosigma.com>
It removes the potential confusion when it comes to virtual address vs
offsets. Further, doing so makes naming more consistent with the rest of
the parasite_blob_desc struct.
Signed-off-by: Nicolas Viennot <Nicolas.Viennot@twosigma.com>
compel_relocs_apply() was taking arguments mostly from the struct
parasite_blob_desc. Instead of passing all the arguments, we pass a
pointer to the struct itself.
This makes the code safer, as cr-restore.c calls compel_relocs_apply().
It previously needed to poke into what can be considered private
variables of the restorer-pie.h file.
To allow the parasite_blob_desc struct to be populated without a
parasite_ctl struct, we expand the compel API to export a
parasite_setup_c_header_desc() in the generated pie.h.
Signed-off-by: Nicolas Viennot <Nicolas.Viennot@twosigma.com>
COMMON symbols are emitted for global variable that are not initialized
in shared libraries.
They typically end up in the .bss section when linking an executable.
Signed-off-by: Nicolas Viennot <Nicolas.Viennot@twosigma.com>
The file only includes other headers (which may be not needed).
If we aim for one-include-for-compel, we could instead paste all
subheaders into "compel.h".
Rather, I think it's worth to migrate to more fine-grained compel
headers than follow the strategy 'one header to rule them all'.
Further, the header creates problems for cross-compilation: it's
included in files, those are used by host-compel. Which rightfully
confuses compiler/linker as host's definitions for fpu regs/other
platform details get drained into host's compel.
Signed-off-by: Dmitry Safonov <dima@arista.com>
Unknown state means that the task in the end may be not in wanted state.
Return err code.
Signed-off-by: Dmitry Safonov <dima@arista.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
As a preparation for __must_check on compel_syscall(), check it on
close() too - maybe not as useful as with other syscalls, but why not.
Signed-off-by: Dmitry Safonov <dima@arista.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
Also, don't use the magic -2 => return errno on failure.
Signed-off-by: Dmitry Safonov <dima@arista.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
>From man ptrace:
> On error, all requests return -1, and errno is set appropriately.
> Since the value returned by a successful PTRACE_PEEK* request may be
> -1, the caller must clear errno before the call, and then check
> it afterward to determine whether or not an error occurred.
FWIW: if ptrace_peek_area() is called with (errno != 0) it may
false-fail if the data is (-1).
Signed-off-by: Dmitry Safonov <dima@arista.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
