If we build tags for our repo:
[criu]$ make tags
GEN tags
And then run codespell, we get an error:
[criu]$ codespell
./tags:3755: struc ==> struct
Let's exclude tags file from codespell search, this would add usability
to `make lint`.
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
As our pr_* functions are complex and can call different system calls
inside before actual printing (e.g. gettimeofday for timestamps) actual
errno at the time of printing may be changed.
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Previousely "make indent" checked all files in criu source directory for
codding style flaws. We have several problems with it:
- clang-format default format sometimes changes in new versions of the
package and we need to reformat all our code base each time it happens
- on different systems we may have different versions of clang-format
and on latest criu-dev "make indent" may be still unhappy on your system
- when we want to update clang-format rules ourselves we need to update
all our code base each time
- sometimes clang-format rules are not fitting all our cases, (e.g.: an
option IndentGotoLabels works nice for simple C code, but is a no go for
assembler and C macros) and putting "clang-format off" everywhere is a
mess
- sometimes we intentionally want to break clang-format rules (e.g.:
we want to put function arguments on a new line separating them
"logically" not "mechanically" following 120-char rule like clang-format
does).
This adds a BASE option for "make indent" where all commits in range
BASE..HEAD would be checked with git-clang-format for codding style
flaws. For instance when developing on top of criu-dev, one can use
"make BASE=origin/criu-dev indent" to check all their commits for
compliance with the clang-format rules. Default base is HEAD~1 to make
last commit checked when "make indent" is called. The closest thing to
the old behaviour would then be "make indent BASE=init", note that only
commited files would be checked.
Extra options to git-clang-format may be passed through OPTS variable.
Also reuse "make indent" in github lint workflow.
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Name collision with an abandoned project named 'crit' in pypi causes pip
to show crit (CRiu Image Tool) as outdated. This patch updates crit to
use the same version and license as criu.
Fixes#1878
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
To support Checkpoint Restore with AMDGPUs for ROCm workloads, introduce
a new plugin to assist CRIU with the help of AMD KFD kernel driver. This
initial commit just provides the basic framework to build up further
capabilities. Like CRIU, the amdgpu plugin also uses protobuf to
serialize
and save the amdkfd data which is mostly VRAM contents with some
metadata.
We generate a data file "amdgpu-kfd-<id>.img" during the dump stage. On restore
this file is read and extracted to re-create various types of buffer
objects that belonged to the previously checkpointed process. Upon
restore the mmap page offset within a device file might change so we use
the new hook to update and adjust the mmap offsets for newly created
target process. This is needed for sys_mmap call in pie restorer phase.
Support for queues and events is added in future patches of this series.
With the current implementation (amdgpu_plugin), we support:
- Only compute workloads such (Non Gfx) are supported
- GPU visible inside a container
- AMD GPU Gfx 9 Family
- Pytorch Benchmarks such as BERT Base
amdgpu plugin dependes on libdrm and libdrm_amdgpu which are typically
installed with libdrm-dev package. We build amdgpu_plugin only when the
dependencies are met on the target system and when user intends to
install the amdgpu plugin and not by default with criu build.
Suggested-by: Felix Kuehling <felix.kuehling@amd.com>
Co-authored-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: Rajneesh Bhardwaj <rajneesh.bhardwaj@amd.com>
Starting with gcc-11, Debian's armhf compiler no longer builds with
a default -mfpu= option. Instead it enables the FPU via an extension
to the -march flag (--with-arch=armv7-a+fp). criu's Makefile explicitly
passes its own -march=armv7-a setting, which overrides the +fp default,
so we end up with no FPU:
cc1: error: '-mfloat-abi=hard': selected architecture lacks an FPU
Signed-off-by: Radostin Stoyanov <radostin@redhat.com>
This is just a placeholder dummy plugin and will be replaced by a proper
plugin that implements support for AMD GPU devices. This just
facilitates the initial pull request and CI build test trigger for early
code review of CRIU specific changes. Future PRs will bring in more
support for amdgpu_plugin to enable CRIU with AMD ROCm.
Signed-off-by: Rajneesh Bhardwaj <rajneesh.bhardwaj@amd.com>
This is another attempt to introduce a tool to format CRIU's source
code. This time it is based on clang-format.
The .clang-format file is taken from the linux kernel git tree (5.13).
I removed all comments from lines which state that it requires at least
clang-format 4 or 5. For this resulting file at least clang-format 11
is required. See scripts/fetch-clang-format.sh for all the changes
done to the Linux kernel .clang-format file.
Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Signed-off-by: Adrian Reber <areber@redhat.com>
This adds a test run to ensure known (but fixed) configuration file
parser errors are not crashing CRIU anymore.
Based on missing test code coverage this script also tests code paths of
the option handling which have not been tested until now.
Signed-off-by: Adrian Reber <areber@redhat.com>
Unlike pr_perror, pr_err and other macros do not append \n
to the message being printed, so the caller needs to take care of it.
Sometimes it was not done, so let's add this manually.
To make sure it won't happen again, add a line to Makefile under the
linter target to check for such missing \n. NOTE this check is only
done for part of such cases (where the pr_* statement fits in one line
and there's no comment after), but it's better than nothing.
Add comments after pr_msg and pr_info statements where we deliberately
don't add \n, so that the above check ignores them.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
My editor (vim) auto-removes whitespace at EOL for *.c and *.h files,
and I think it makes sense to have a separate commit for this, rather
than littering other commits with such changes.
To make sure this won't pile up again, add a line to Makefile under
the linter target to check for such things (so CI will fail).
This is all whitespace except an addition to Makefile.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
On my system (shellcheck v0.7.1) make lint shows a few warnings about
needing to quote variables.
Fix those.
PS I am not sure why those are not shown by GHA CI, I assume there is
different shellcheck version used. Add shellcheck -- version to the
appropriate Makefile target to avoid confusion.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In many cases developers forget that pr_perror and fail macros
are a bit special, in particular:
1. they already show errno;
2. they already append \n to the message.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Running zdtm tests does not require input and therefore it is not
necessary to use -it. This change also allows to run the test in CI
where it currently fails with:
the input device is not a TTY
make: *** [Makefile:388: docker-test] Error 1
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
All zdtm tests pass on Fedora 33 for `make docker-build && make docker-test`
with devicemapper storage driver.
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The test/zdtm_mount_cgroups script fails with 'permission denied'
when running tests with private cgroup namespace.
Using the host network namespace allows us to test criu as if
it is running on the host, sharing iptables rules etc.
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
This fixes the others/crit test to work again and extends it to make
sure all possible input and output options are correctly handled by
crit.
Signed-off-by: Adrian Reber <areber@redhat.com>
CRIU is already using multiple CI systems and not just Travis. This
renames all Travis related things to 'ci' to show it is actually
independent of Travis.
Just a simple rename.
Signed-off-by: Adrian Reber <areber@redhat.com>
Shellcheck (https://github.com/koalaman/shellcheck) can identify common
errors in shell scripts. This initial integration of shellcheck only
checks the scripts in the 'scripts/' folder. This commit fixes (or
disables) all reports of shellcheck to ensure this part starts error
free. I am not convinced this is really necessary as most changes do not
seem to be necessary for their circumstances. On the other hand it
probably does not hurt to use a checker to avoid unnecessary errors.
Signed-off-by: Adrian Reber <areber@redhat.com>
Source files modified:
* Makefile.config - Checks whether libbpf is installed on the system.
If so, we add -lbpf to LIBS_FEATURES, -DCONFIG_HAS_LIBBPF to
FEATURE_DEFINES and set CONFIG_HAS_LIBBPF. This allows us to check for
the presence of libbpf before compiling or executing BPF c/r code and
ZDTM tests.
* Makefile - Set CONFIG_HAS_LIBBPF to clean all files.
Signed-off-by: Abhishek Vijeev <abhishek.vijeev@gmail.com>
Include warnings that the kernel uses during compilation:
-Wstrict-prototypes: enforces full declaration of functions.
Previously, when declaring extern void func(), one can call func(123)
and have no compilation error. This is dangerous. The correct declaration
is extern void func(void).
Signed-off-by: Nicolas Viennot <Nicolas.Viennot@twosigma.com>
[Generated a commit message from the pull request]
Signed-off-by: Dmitry Safonov <dima@arista.com>
Include warnings that the kernel uses during compilation:
-Wdeclaration-after-statement: enforces having variables declared at the top of scopes
Signed-off-by: Nicolas Viennot <Nicolas.Viennot@twosigma.com>
[Generated a commit message from the pull request]
Signed-off-by: Dmitry Safonov <dima@arista.com>
Instead of doing additional `uname -m` - use provided $(SUBARCH) to detect
what architecture flavour the build should produce the result for.
Fixes two things:
- zdtm make now correctly supplies $(USERCFLAGS)
- subtly fixes cross compilation by providing a way to specify $(SUBARCH)
Signed-off-by: Dmitry Safonov <dima@arista.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
SRCARCH is always equal ARCH. There are no rules when to use one or
another and architectures may forget to set one of them up.
No need for a second variable meaning the same and confusing people.
Remove it completely.
Self-correction [after some debug]: SRCARCH was different in one place:
zdtm Makefile by some unintentional mistake:
> ifeq ($(ARCH),arm64)
> ARCH ?= aarch64
> SRCARCH ?= aarch64
> endif
That meant to be "ARCH := aarch64" because "?=" would never work inside
that ifeq. Fix up this part of mess too.
Signed-off-by: Dmitry Safonov <dima@arista.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
For CRIU's compile only tests for armv7hf on Travis we are using
'setarch linux32' which returns armv8l on Travis aarch64.
This adds a path in the Makefile to treat armv8l just as armv7hf during
compile. This enables us to run armv7hf compile tests on Travis aarch64
hardware. Much faster. Maybe not entirely correct, but probably good
enough for compile testing in an armv7hf container.
Signed-off-by: Adrian Reber <areber@redhat.com>
This commit adds Transport Layer Security (TLS) support for remote
page-server connections.
The following command-line options are introduced with this commit:
--tls-cacert FILE Trust certificates signed only by this CA
--tls-cacrl FILE CA certificate revocation list
--tls-cert FILE TLS certificate
--tls-key FILE TLS private key
--tls Use TLS to secure remote connections
The default PKI locations are:
CA certificate /etc/pki/CA/cacert.pem
CA revocation list /etc/pki/CA/cacrl.pem
Client/server certificate /etc/pki/criu/cert.pem
Client/server private key /etc/pki/criu/private/key.pem
The files cacert.pem and cacrl.pem are optional. If they are not
present, and not explicitly specified with a command-line option,
CRIU will use only the system's trusted CAs to verify the remote
peer's identity. This implies that if a CA certificate is specified
using "--tls-cacert" only this CA will be used for verification.
If CA certificate (cacert.pem) is not present, certificate revocation
list (cacrl.pem) will be ignored.
Both (client and server) sides require a private key and certificate.
When the "--tls" option is specified, a TLS handshake (key exchange)
will be performed immediately after the remote TCP connection has been
accepted.
X.509 certificates can be generated as follows:
-------------------------%<-------------------------
# Generate CA key and certificate
echo -ne "ca\ncert_signing_key" > temp
certtool --generate-privkey > cakey.pem
certtool --generate-self-signed \
--template temp \
--load-privkey cakey.pem \
--outfile cacert.pem
# Generate server key and certificate
echo -ne "cn=$HOSTNAME\nencryption_key\nsigning_key" > temp
certtool --generate-privkey > key.pem
certtool --generate-certificate \
--template temp \
--load-privkey key.pem \
--load-ca-certificate cacert.pem \
--load-ca-privkey cakey.pem \
--outfile cert.pem
rm temp
mkdir -p /etc/pki/CA
mkdir -p /etc/pki/criu/private
mv cacert.pem /etc/pki/CA/
mv cert.pem /etc/pki/criu/
mv key.pem /etc/pki/criu/private
-------------------------%<-------------------------
Usage Example:
Page-server:
[src]# criu page-server -D <PATH> --port <PORT> --tls
[dst]# criu dump --page-server --address <SRC> --port <PORT> \
-t <PID> -D <PATH> --tls
Lazy migration:
[src]# criu dump --lazy-pages --port <PORT> -t <PID> -D <PATH> --tls
[dst]# criu lazy-pages --page-server --address <SRC> --port <PORT> \
-D <PATH> --tls
[dst]# criu restore -D <PATH> --lazy-pages
Signed-off-by: Radostin Stoyanov <rstoyanov1@gmail.com>
Vigorously remove the config ifdef.
The config option *never* had any excuse to exist:
- for x86 we were grand
- for ppc64/arm64 patches to support mremap() on vdso were long ago
accepted, but regardless - it's not possible to disable CONFIG_VDSO for
those platforms in kernel
- for s390 - patches were mainstreamed not that long ago, but it's not
possible to disable the kernel config
- for arm32 it's possible to disable the kernel config, but kernel
returns to userspace historically through sigpage, not vdso.
That's the only platform that criu disallows to have CONFIG_VDSO=y in
kernel, but that's just meaningles. A kernel patch for sigpage mremap()
has gone into v4.13: commit 280e87e98c09 ("ARM: 8683/1: ARM32: Support
mremap() for sigpage/vDSO").
So, removing the config was long-lived item on my TODO list that bligted
arm32 users and made changes to vdso more complex by all "needed"
iffdeferry. Get rid of it with fire.
Fixes: #446
Signed-off-by: Dmitry Safonov <dima@arista.com>
__ASSEMBLY__ is used to guard C-related code in headers from
asm-compatible defines. We actually want every .S file to be assembled
with -D__ASSEMBLY__ not to burst with C in asm file.
Move __ASSEMBLY__ from all local asflags to top Makefile's AFLAGS.
Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com>
Signed-off-by: Dmitry Safonov <dima@arista.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
I don't want to see CFLAGS redefined per-architecture in PIE makefiles
in couple of places. Clean it up.
The only expected per-arch ifdeffery should be object files.
Also add a big comment about -marm vs -mthumb[2]
Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com>
Signed-off-by: Dmitry Safonov <dima@arista.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
This test checks the following things:
* Does configuration file parsing work at all.
* Does the parser detect wrong options.
* Does the configuration file work via RPC.
* Do the configuration file options not overwrite the RPC settings in
the default setup.
* Is it possible to tell CRIU to prefer the configuration file via RPC.
Signed-off-by: Adrian Reber <areber@redhat.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
With this last commit of the crit with python3 series it is possible to
either use python2 or python3 with CRIU.
Now the basic build system functionality (make and make install) are
python2/python3 aware. zdtm.py and criu-coredump are still python2, but
as they are not part of 'make install' those parts have not yet been
ported from python2 to python3.
Signed-off-by: Adrian Reber <areber@redhat.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
We shouldn't set MAKEFLAGS by the following reasons:
1. User may want to specify some make parameter (e.g., `-d` for debug)
2. We lose parallel build. No `-j` is passed to submake and it looks
like, gnu/make will not deal with parallel recursive make if
$(MAKEFLAGS) is unset back.
Easy to verify: Add `sleep 3` to build rule in Makefile.inc and
you'll find only one sleep process at a time. After the patch
if you specify say `-j5` to make - you'll have 5 sleep processes.
Reverts: commit e9beed7bb3 ("build: zdtm -- Add implicit rules into
zdtm building").
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Signed-off-by: Dmitry Safonov <dima@arista.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Sometimes we see errors like this:
criu/cr-restore.gcda:Merge mismatch for function 106
It proabably means that this gcda file was corrupted. According to the
gcc man page, the -fprofile-update=atomic should fix this problem.
v2: this options appered in gcc7, so we need to install it.
Reported-by: Mr Travis CI
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
config.h is a generated file with "build-features" defines.
We use it for several purposes:
o to check that compiler can do it's job
o to complement user-visible API between distributions
o to add compile-time options from .config global file
It's used in criu and soccr, but compel also needs such thing.
Previously, soccr has a link to config.h in criu includes,
but it would be much cleaner to move it to other headers,
that are shared between sub-projects into include/common.
Reported-by: Adrian Reber <areber@redhat.com>
Signed-off-by: Dmitry Safonov <dsafonov@virtuozzo.com>
Tested-by: Adrian Reber <areber@redhat.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
