criu/criu
Dan Feigin 5141fbcca8 pie: restore VMA pages with native AIO
Private anonymous VMA contents have to be copied inside the PIE
restorer after mappings are placed at their final addresses.
Replace the synchronous preadv loop with a bounded Linux native AIO
window that submits page-read iovecs, reaps completions, and
resubmits short reads.

Keep auto-dedup punching on completed byte ranges and tear down the
AIO context on both success and error paths. The pages fd is
prepared for O_DIRECT before entering the restorer.

VMA page reads are page-aligned by construction: the destination
buffers are page-aligned, the iovec lengths are page multiples, and
the file offsets are page-aligned. O_DIRECT completions are
therefore always a whole number of pages (short reads only occur at
MAX_RW_COUNT or at EOF on page-multiple files), so resubmitting a
short read keeps its offset and buffers aligned without masking the
completion length.

Assisted-by: Claude:claude-sonnet-4-6
Assisted-by: Claude:claude-opus-4-7
Assisted-by: Codex:gpt-5
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
2026-06-01 14:25:13 -07:00
..
arch compel: remove hardware breakpoint usage 2026-03-08 23:01:54 -07:00
include pagemap/mem: probe O_DIRECT for page reads 2026-06-01 14:25:13 -07:00
pie pie: restore VMA pages with native AIO 2026-06-01 14:25:13 -07:00
unittest bfd: Support dynamic buffer resizing 2026-04-01 16:47:00 +00:00
action-scripts.c rpc: Support gathering external file list after freezing process tree. 2023-10-22 13:29:25 -07:00
aio.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
apparmor.c apparmor: get_suspend_policy must return NULL in error cases 2024-09-11 16:02:11 -07:00
autofs.c criu: use proper format-specified to accommodate time_t 64-bit change 2024-09-11 16:02:11 -07:00
bfd.c bfd: fix partial write handling in bwritev() 2026-04-02 11:05:42 -07:00
bitmap.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
bpfmap.c bpf: update deprecated API 2022-04-28 17:53:52 -07:00
cgroup-props.c cgroup-v2: Checkpoint and restore some global properties 2023-04-15 21:17:21 -07:00
cgroup.c criu: fix internal representation of cgroups hierarchical structure 2025-03-21 12:40:31 -07:00
clone-noasan.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
config.c vma: introduce --allow-uprobes option 2025-11-05 15:41:34 -08:00
cr-check.c compel: remove hardware breakpoint usage 2026-03-08 23:01:54 -07:00
cr-dedup.c pagemap: print page regions in the format start - end 2025-11-02 07:48:23 -08:00
cr-dump.c criu: collect and dump timer_slack_ns per thread 2026-03-19 21:35:31 -07:00
cr-errno.c build: Move everything criu related into criu directory 2016-02-15 15:32:45 +03:00
cr-restore.c restore: add missing continue in restorer_get_vma_hint 2026-05-18 17:03:22 +01:00
cr-service.c cr-service: fix address handling of requests 2026-03-28 10:04:14 +00:00
crtools.c doc: update pipe/socket examples for --inherit-fd 2025-11-16 15:56:26 +00:00
eventfd.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
eventpoll.c Fix some codespell warnings 2022-04-28 17:53:52 -07:00
external.c configs: fix used after free cases 2021-09-03 10:31:00 -07:00
fault-injection.c seize: enable support for frozen containers 2025-03-21 12:40:31 -07:00
fdstore.c criu: use libuuid for criu_run_id generation 2025-03-21 12:40:31 -07:00
fifo.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
file-ids.c criu/files: Don't cache fd ids for device files 2022-04-28 17:53:52 -07:00
file-lock.c file-lock: fix type confusion in broken lease restore 2026-03-18 08:08:59 -07:00
files-ext.c files-ext: Allow plugin files to retry 2025-11-14 18:31:36 +00:00
files-reg.c files-reg: warn when external file has no inherit-fd mapping on restore 2026-04-20 08:28:36 -07:00
files.c files: fork helpers without CLONE_FILES | CLONE_FS 2025-11-02 07:48:23 -08:00
filesystems.c criu: Support binfmt_misc sandboxing 2026-02-17 17:07:29 +00:00
fsnotify.c fsnotify: Fix mnt_id type to avoid undefined behavior with -1 sentinel 2026-02-18 10:02:18 -08:00
hugetlb.c hugetlb: don't dump anonymous private hugetlb mapping using memfd approach 2023-04-15 21:17:21 -07:00
image-desc.c criu: Remove legacy binfmt_misc handling code 2026-02-17 17:07:29 +00:00
image.c cr-service: refactor images/workdir setup 2025-11-05 15:41:34 -08:00
img-streamer.c action-scripts: Add pre-stream hook 2023-10-22 13:29:25 -07:00
ipc_ns.c ipc_sysctl: Prioritize restoring IPC variables using non usernsd approach 2023-04-15 21:17:21 -07:00
irmap.c irmap: duplicate string in irmap_scan_path_add 2024-09-11 16:02:11 -07:00
kcmp-ids.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
kerndat.c compel: remove hardware breakpoint usage 2026-03-08 23:01:54 -07:00
libnetlink.c build: libnfnetlink: Remove nla_get_s32(). 2023-10-22 13:29:25 -07:00
log.c log: use sizeof(*hdr) instead of sizeof(hdr) 2025-11-13 08:40:46 -08:00
lsm.c lsm: use attr/apparmor/current to get apparmor label 2025-11-02 07:48:23 -08:00
Makefile bfd: Support dynamic buffer resizing 2026-04-01 16:47:00 +00:00
Makefile.crtools criu: Support C/R of pidfds 2025-03-21 12:40:31 -07:00
Makefile.packages criu: use libuuid for criu_run_id generation 2025-03-21 12:40:31 -07:00
mem.c pagemap/mem: probe O_DIRECT for page reads 2026-06-01 14:25:13 -07:00
memfd.c memfd: don't reopen file descriptors for memory mappings 2023-10-22 13:29:25 -07:00
mount-v2.c criu: Remove legacy binfmt_misc handling code 2026-02-17 17:07:29 +00:00
mount.c criu: Remove legacy binfmt_misc handling code 2026-02-17 17:07:29 +00:00
namespaces.c criu: fix double-open of userns image in --stream mode 2026-03-04 23:41:08 -08:00
net.c net: Route veth restore through usernsd for userns mode 2026-04-22 12:56:36 +01:00
netfilter.c net: remember the name of the lock chain (nftables) 2025-03-21 12:40:31 -07:00
page-pipe.c pagemap: print page regions in the format start - end 2025-11-02 07:48:23 -08:00
page-xfer.c page-xfer: Fix page_read resource leak in page_pipe_from_pagemap() 2026-02-14 12:52:26 -08:00
pagemap-cache.c pagemap-cache: stop filling cache on VMA_AREA_GUARD 2026-03-05 10:33:37 +01:00
pagemap.c shmem: batch page reads with PR_ASYNC 2026-06-01 14:25:13 -07:00
parasite-syscall.c compel: remove hardware breakpoint usage 2026-03-08 23:01:54 -07:00
path.c Fix some codespell warnings 2022-04-28 17:53:52 -07:00
pidfd-store.c criu: use libuuid for criu_run_id generation 2025-03-21 12:40:31 -07:00
pidfd.c pidfd: add missing include 2025-03-21 12:40:31 -07:00
pie-util-vdso-elf32.c vdso: fill compat symtab in CRIU 2017-03-15 00:09:52 +03:00
pie-util-vdso.c build: Move everything criu related into criu directory 2016-02-15 15:32:45 +03:00
pie-util.c build: Move everything criu related into criu directory 2016-02-15 15:32:45 +03:00
pipes.c pipes: restore pipe ownership to fix /proc/self/fd access 2026-03-25 21:09:15 -07:00
plugin.c Add UPDATE_INETSK hook for inet address rewrite 2026-02-22 15:15:38 -08:00
proc_parse.c mem: don't PROT_WRITE on reservation mmaps 2026-05-07 11:45:34 -07:00
protobuf-desc.c criu: Support C/R of pidfds 2025-03-21 12:40:31 -07:00
protobuf.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
pstree.c criu: Fix garbage write to /proc files 2026-03-30 10:55:04 +02:00
rbtree.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
rst-malloc.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
seccomp.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
seize.c pstree: Add has_children function 2025-11-02 07:48:22 -08:00
servicefd.c plugin/amdgpu: Add handling for amdgpu drm buffer objects 2025-11-14 18:31:36 +00:00
setproctitle.c string: define wrapers __setproctitle and __setproctitle_init to hide bsd headers 2023-04-15 21:17:21 -07:00
shmem.c shmem: batch page reads with PR_ASYNC 2026-06-01 14:25:13 -07:00
sigact.c criu: move sigact dump/restore code into sigact.c 2024-09-11 16:02:11 -07:00
sigframe.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
signalfd.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
sk-inet.c sk-inet: remove trailing whitespace 2026-02-23 18:13:37 +00:00
sk-netlink.c criu/sockets: Restrict SO_PASSCRED and SO_PASSSEC to supported families 2025-11-02 07:48:23 -08:00
sk-packet.c criu/sockets: Restrict SO_PASSCRED and SO_PASSSEC to supported families 2025-11-02 07:48:23 -08:00
sk-queue.c sk-queue: increase CMSG_MAX_SIZE to handle all SCM types 2026-03-23 17:38:57 -07:00
sk-tcp.c sk-tcp: Move TCP socket options from SkOptsEntry to TcpOptsEntry 2024-09-11 16:02:11 -07:00
sk-unix.c unix: fix dangling pointers in icon hash on error path 2026-02-23 23:05:33 -08:00
sockets.c sockets: Treat UDPLITE as optional in collect_err() 2026-04-22 08:22:50 +01:00
stats.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
string.c string: use our own __strlcpy and __strlcat to remove bsd headers 2023-04-15 21:17:21 -07:00
sysctl.c ipc_sysctl: Prioritize restoring IPC variables using non usernsd approach 2023-04-15 21:17:21 -07:00
sysfs_parse.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
timens.c criu: use proper format-specified to accommodate time_t 64-bit change 2024-09-11 16:02:11 -07:00
timer.c restore: use the new kernel interface to restore timers 2025-11-02 07:42:55 -08:00
timerfd.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
tls.c Fix some codespell warnings 2022-04-28 17:53:52 -07:00
tty.c tty: fix compiler error 2025-11-28 09:18:59 +00:00
tun.c tun: don't parse buffers that have not been filled with data 2023-10-22 13:29:25 -07:00
uffd.c pagemap: use unsigned long for page counts 2025-11-02 07:48:23 -08:00
util.c util: Make close_safe() reset fd to -1 even on close() failure 2025-12-29 10:00:35 +00:00
uts_ns.c util: Include int.h (compilation fix) 2016-10-24 16:00:16 +03:00
vdso-compat.c Run 'make indent' on all C files 2021-09-03 10:31:00 -07:00
vdso.c criu/{mem, vdso, cr-restore}: introduce VMA_AREA_GUARD fake VMAs 2025-11-02 07:48:23 -08:00