Currently, adding a package which is required either for development or testing
requires it to be added in multiple places due to many duplicated Dockerfiles
and installation scripts. This makes it difficult to ensure that all scripts
are updated appropriately and can lead to some places being missed.
This patch consolidates the list of dependencies and adds installation
scripts for each package-manager used in our CI (apk, apt, dnf, pacman).
This change also replaces the `debian/dev-packages.lst` as this subfolder
conflicts with the Ubuntu/Debian packing scripts used for CRIU:
https://github.com/rst0git/criu-deb-packages
This patch also removes the CentOS 8 build scripts as it is EOL
and the container registry is no longer available.
Signed-off-by: Shashank Balaji <shashank.mahadasyam@sony.com>
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
We are dropping support for generating JUnit XML reports in zdtm.py as we've
migrated testing infrastructure entirely to `GitHub Actions` and other
third-party test runners.
This package has been removed from some distribution repositories (e.g.,
Fedora), making it simpler to remove the dependency than to force installation
via pip.
Signed-off-by: Andrei Vagin <avagin@google.com>
criu_run_id will be used in upcoming changes to create and remove
network rules for network locking. Instead of trying to come up with
a way to create unique IDs, just use an existing library.
libuuid should be installed on most systems as it is indirectly required
by systemd (via libmount).
Signed-off-by: Adrian Reber <areber@redhat.com>
Ruff (https://github.com/astral-sh/ruff) is a Python linter
written in Rust, designed to replace Flake8. It is significantly
faster and actively maintained.
In addition to replacing flake8 with ruff, this patch also
creates separate makefile targets for ruff, shellcheck and
codespell, so that they can be tested independently.
RUFF_FLAGS can be used to specify options such as '--fix'.
Example:
make lint
make ruff RUFF_FLAGS=--fix
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
This patch adds the `libdrm-dev` package to the list of CRIU
dependencies installed in CI to build CRIU with amdgpu plugin.
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
CAP_CHECKPOINT_RESTORE does not give access to /proc/$pid/map_files in
user namespaces. In order to test that CRIU in unprivileged mode can
dump and restore anonymous shared memory pages we will run the maps00
tests in a user namespace.
Signed-off-by: Younes Manton <ymanton@ca.ibm.com>
The python3 package in Alpine has recently been updated to install
symbolic link for /usr/bin/python.
https://git.alpinelinux.org/aports/commit/main/python3?id=d91da210b1614eb75517d59b7f348fee01699f35
This causes the following error in CI:
Step 10/11 : RUN ln -s /usr/bin/python3 /usr/bin/python
---> Running in a5a94be9dc93
ln: failed to create symbolic link '/usr/bin/python': File exists
The command '/bin/sh -c ln -s /usr/bin/python3 /usr/bin/python' returned a non-zero code: 1
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The version of ps in Alpine image by default is very limited.
It is based on the one from busybox and doesn't support options
such as '-p'.
Signed-off-by: Radostin Stoyanov <radostin@redhat.com>
The ENV1 variable was first introduced with commit
7290de5 (travis: enable ccache for docker/qemu builds)
and it is not used anymore.
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
ccache was set up in Travis to speed up compilation by re-using the
.ccache directory from previous CI runs. As we are no longer using
Travis we can remove all CI related ccache setup.
Signed-off-by: Adrian Reber <areber@redhat.com>
With the latest version of the alpine container image it seems that
alpine changed a few package names. This adapts the alpine container
to solve the travis failures.
Signed-off-by: Adrian Reber <areber@redhat.com>
Now that Python 2 has officially reached its end of life also switch the
Alpine based test to Python 3.
Signed-off-by: Adrian Reber <areber@redhat.com>
After Centos-8 nft used instead of iptables. But we had never supported nft rules in
CRIU, and after c/r all rules are flushed.
Co-developed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Alexander Mikhalitsyn <alexander.mikhalitsyn@virtuozzo.com>
Signed-off-by: Alexander Mikhalitsyn <alexander@mihalicyn.com>
Signed-off-by: Dmitry Safonov <dima@arista.com>
With the newly introduced aarch64 at Travis it is possible for the CRIU
test-cases to switch to aarch64.
Travis uses unprivileged LXD containers on aarch64 which blocks many of
the kernel interfaces CRIU needs. So for now this only tests building
CRIU natively on aarch64 instead of using the Docker+QEMU combination.
All tests based on Docker are not working on aarch64 is there currently
seems to be a problem with Docker on aarch64. Maybe because of the
nesting of Docker in LXD.
Signed-off-by: Adrian Reber <areber@redhat.com>
The final release of asciidoc was on Sep 29, 2017 and the development
is continued under asciidoctor.
With commit 0493724 (Documentation: Allow to use asciidoctor for
formatting man pages) was added support for this tool by introducing
USE_ASCIIDOCTOR.
However, using asciidoctor by default might be a better option. With
this change CRIU will use asciidoctor if installed. Otherwise, it will
fallback to asciidoc.
Signed-off-by: Radostin Stoyanov <rstoyanov1@gmail.com>
Install sudo, create test user with ID 1000, install bash,
fix pidfile creation and pidfile chmod.
v2:
* use sleep to give the criu daemon some time to start up
Signed-off-by: Adrian Reber <areber@redhat.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
For python2/python3 compatibility install additional (python2-future)
and different (python2-ipaddress) packages during test.
Also switch Fedora tests to explicitly install versioned python packages
(python2-* instead of python-*).
Signed-off-by: Adrian Reber <areber@redhat.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Commit 299e4b4 ("travis: enable ccache for docker/qemu builds") combined
multiple RUN statements in Dockerfiles into a single one, which is good
for performance (as there is an intermediate cache created after each
statement). Unfortunately, it partially did so by combining statements
with use of semicolons, meaning we are ignoring non-zero exit codes.
As a result, failure from make are ignored and Travis builds are all
green.
To fix, replace all occurences of semicolon with &&.
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
As we compile-test non-x86_64 architectures under qemu emulation,
it works pretty slow.
Dmitry Safonov suggested, and Andrey Vagin initially implemented
supporting ccache for such builds. This patch is based heavily
on Andrey's work -- all the bugs added are purely mine though.
Performance results: in an ideal environment (two builds of the same
code, one with cold (empty) ccache, another with the hot one)
I saw compile time improvements of 4x to 5x, and total test run time
improvement up to 2x to 2.5x. In layman terms, the complete test run
that was taking more than 50 minutes now takes about 25!
Notes on handling .ccache directory:
1. Before running docker build, .ccache directory (saved in between
runs by Travis) is moved to criu source code root, from where it
is copied by docker together with criu sources.
2. In Dockerfile, .ccache gets moved to /tmp, and CCACHE_DIR
is set accordingly.
3. After running docker build, .ccache is copied out from docker
container back to the host (from where it is saved by Travis).
Ccache envorinment notes:
1. CCACHE_NOCOMPRESS is needed because otherwise tons of time is spent
on compression/decompression (big performance hit under qemu).
2. CCACHE_CPP2 is required with clang, see detailed explanation at
http://petereisentraut.blogspot.com/2011/09/ccache-and-clang-part-2.html
The logic of setting CCACHE_CPP2 in Dockerfile is somewhat fancy;
unfortunately I was not able to come up with a simpler approach.
Misc:
1. Travis runs "ccache -s" after the build is completed. A call to
"ccache -s" is called to Dockerfile before make, so one can
compare before/after numbers.
2. make invocations are surrounded by "date" calls so one can get the
compilation times.
Changes in v2:
- consolidate Dockerfile statements (ENV, RUN)
- single object make test is no longer commented out
- simplify "make mrproper" test
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Reviewed-by: Dmitry Safonov <dsafonov@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
"zdtm.py run -a" doesn't work on Alpine:
find: unrecognized: -executable
So run zdtm/static/env00 for now to be sure that
it is not broken at all.
travis-ci: success for travis: execute tests on Alpine
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
We see the "No space left on device" error on the current version,
so let's try to use a newer version of Alpine, maybe this error
has been fixed there.
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
It will be used to generate tcp packets for restoring half-closed sockets.
travis-ci: success for series starting with [01/21] build: install libnet-dev
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
With this, one can do a CRUI build via QEMU+Docker using clang
as a compiler, for example:
make -C scripts/build alpine-clang
make -C scripts/build ppc64le-clang
For alpine, default clang version is used. For others, a version has
to be specified explicitly, as there is no package called "clang" in
Ubuntu Trusty (which is used as a base for most arches). Since
clang-3.8 appears to be available from Trusty, and this is the current
stable version of clang, let's use it.
travis-ci: success for QEMU+Dockerfile improvements + clang build
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Acked-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
I just spent some considerable time trying to figure out the source
of errors like this:
make -C scripts/build ppc64
...
DEP images/autofs.pb-c.d
In file included from images/autofs.pb-c.c:9:0:
images/autofs.pb-c.h:7:35: fatal error: protobuf-c/protobuf-c.h:
No such file or directory
#include <protobuf-c/protobuf-c.h>
^
compilation terminated.
It appears that we copy the complete tree to a docker container, and it
is not cleaned properly before the build. In particular, images/*.pb-c.{c,h}
are not removed/rebuilt. The problem is not noticeable until version of
protobuf-c differs between the host and container, and they use different
include paths (/usr/include/google/protobuf-c/ vs /usr/include/protobuf-c).
As I don't want anyone else to spend so much time figuring this out,
here is the patch.
travis-ci: success for QEMU+Dockerfile improvements + clang build
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Acked-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
git is needed as the top-level Makefile calls it. Strictly speaking
git is not required to build CRIU, but it's better to have it than
to see some error messages.
coreutils is needed for nproc utility which is called from Dockerfile:
make -j $(nproc)
Before this patch, it was equivalent to "make -j" (plus an error
message) which is not the best thing to do.
travis-ci: success for QEMU+Dockerfile improvements + clang build
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Acked-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
