d7fa8f3fea
memfds are restored as regular files, so their content can be filled in
parallel at the file level. Filling memfd data is the slowest part of
restore, and since the other restore steps do not depend on it, it can be
done asynchronously by a helper daemon.
Each restoring task creates its memfd and hands the descriptor to an
asynchronous daemon (asyncd) that fills the content (mmap + pread) from a
pool of worker threads. The daemon mirrors the usernsd design: tasks ship
work over a SEQPACKET socket via SCM_RIGHTS, and the content fill runs
out-of-band while the tasks keep restoring.
The daemon's worker threads must not hold a TID that the restorer later
needs. The restorer recreates each application thread at its original TID
via clone3(set_tid); if a daemon worker still occupies that TID the clone
fails with EEXIST ("Unable to create a thread: -17"). To guarantee the
daemon is gone before any application thread is cloned, a new restore
stage CR_STATE_PRE_RESTORER is introduced between CR_STATE_FORKING and
CR_STATE_RESTORE. The root task starts the daemon and switches all tasks
to PRE_RESTORER, during which the tasks ship their fills. The root task
then drains and reaps the daemon (stop_asyncd) before switching to
CR_STATE_RESTORE, where threads are cloned. Because the daemon is reaped
first, its worker TIDs are free by the time clone3(set_tid) runs.
Keeping the daemon inside the restored task tree (rather than forking it
from the coordinator) keeps the content fill charged to the restored
container's memory cgroup instead of to criu.
The fill-thread pool is sized to min(online_cpus, 16). stop_asyncd() fails
the restore if the daemon exits abnormally, so a failed content fill is
never silently lost, and it is idempotent so the restore paths can call it
unconditionally.
criu-image-streamer serves the image in a single sequential pass, which is
incompatible with the daemon's out-of-band reads, so the daemon is not
started when restoring from a stream and the content is filled inline
instead.
Co-developed-by: Dan Feigin <dfeigin@nvidia.com>
Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Signed-off-by: Andrei Vagin <avagin@google.com>
|
||
|---|---|---|
| .circleci | ||
| .github | ||
| compel | ||
| contrib | ||
| coredump | ||
| crit | ||
| criu | ||
| Documentation | ||
| images | ||
| include | ||
| lib | ||
| plugins | ||
| scripts | ||
| soccr | ||
| test | ||
| .clang-format | ||
| .codespellrc | ||
| .gitignore | ||
| .lgtm.yml | ||
| .mailmap | ||
| CLAUDE.md | ||
| CONTRIBUTING.md | ||
| COPYING | ||
| CREDITS | ||
| docs | ||
| flake.lock | ||
| flake.nix | ||
| GEMINI.md | ||
| INSTALL.md | ||
| MAINTAINERS | ||
| MAINTAINERS_GUIDE.md | ||
| Makefile | ||
| Makefile.compel | ||
| Makefile.config | ||
| Makefile.install | ||
| Makefile.versions | ||
| README.md | ||
CRIU -- A project to implement checkpoint/restore functionality for Linux
CRIU (stands for Checkpoint and Restore in Userspace) is a utility to checkpoint/restore Linux tasks.
Using this tool, you can freeze a running application (or part of it) and checkpoint it to a hard drive as a collection of files. You can then use the files to restore and run the application from the point it was frozen at. The distinctive feature of the CRIU project is that it is mainly implemented in user space. There are some more projects doing C/R for Linux, and so far CRIU appears to be the most feature-rich and up-to-date with the kernel.
CRIU project is (almost) the never-ending story, because we have to always keep up with the Linux kernel supporting checkpoint and restore for all the features it provides. Thus we're looking for contributors of all kinds -- feedback, bug reports, testing, coding, writing, etc. Please refer to CONTRIBUTING.md if you would like to get involved.
The project started as the way to do live migration for OpenVZ Linux containers, but later grew to more sophisticated and flexible tool. It is currently used by (integrated into) OpenVZ, LXC/LXD, Docker, and other software, project gets tremendous help from the community, and its packages are included into many Linux distributions.
The project home is at http://criu.org. This wiki contains all the knowledge base for CRIU we have. Pages worth starting with are:
- Installation instructions
- A simple example of usage
- Examples of more advanced usage
- Troubleshooting can be hard, some help can be found here, here and here
Checkpoint and restore of simple loop process
Advanced features
As main usage for CRIU is live migration, there's a library for it called P.Haul. Also the project exposes two cool core features as standalone libraries. These are libcompel for parasite code injection and libsoccr for TCP connections checkpoint-restore.
Live migration
True live migration using CRIU is possible, but doing all the steps by hands might be complicated. The phaul sub-project provides a Go library that encapsulates most of the complexity. This library and the Go bindings for CRIU are stored in the go-criu repository.
Parasite code injection
In order to get state of the running process CRIU needs to make this process execute some code, that would fetch the required information. To make this happen without killing the application itself, CRIU uses the parasite code injection technique, which is also available as a standalone library called libcompel.
TCP sockets checkpoint-restore
One of the CRIU features is the ability to save and restore state of a TCP socket without breaking the connection. This functionality is considered to be useful by itself, and we have it available as the libsoccr library.
Licence
The project is licensed under GPLv2 (though files sitting in the lib/ directory are LGPLv2.1).
All files in the images/ directory are licensed under the Expat license (so-called MIT). See the images/LICENSE file.