summary refs log tree commit diff stats
path: root/tests/Makefile.include (follow)
Commit message (Collapse)AuthorAgeFilesLines
* tests/functional: Re-activate the check-venv targetGustavo Romero2025-10-071-2/+2
| | | | | | | | | | | | | | Add check-venv target as a dependency for the functional tests. This causes Python modules listed in pythondeps.toml, under the testdeps group, to be installed when 'make check-functional{-<ARCH>}' is executed to prepare and run the functional tests. Suggested-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org> Message-ID: <20251003141820.85278-2-gustavo.romero@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
* tests: Fix "make check-functional" for targets without thorough testsThomas Huth2025-09-241-0/+3
| | | | | | | | | | | | | | | | If QEMU gets configured for a single target that does not have any thorough functional tests, "make check-functional" currently fails with the error message "No rule to make target 'check-func'". This happens because "check-func" only gets defined for thorough tests (quick ones get added to "check-func-quick" instead). The same problem can happen with the quick tests for targets that do not have any functional test at all. To fix it, simply make sure that the targets are always available in the Makefile. Reported-by: Peter Maydell <peter.maydell@linaro.org> Closes: https://gitlab.com/qemu-project/qemu/-/issues/3119 Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20250918125154.126072-1-thuth@redhat.com>
* tracetool: add test suite for tracetool with reference outputDaniel P. Berrangé2025-09-161-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When reviewing tracetool patches it is often very unclear what the expected output will be for the generated backends. Compounding this is that a default build will only enable the 'log' trace backend, so developers won't see generated code for other backends without making a special effort. Some backends are also platform specific, so can't be enabled in QEMU builds, even though tracetool could generate the code. To address this, introduce a test suite for tracetool which is conceptually similar to the qapi-schema test. It is a simple python program that runs tracetool and compares the actual output to historical reference output kept in git. The test directly emits TAP format logs for ease of integration with meson. This can be run with make check-tracetool to make it easier for developers changing generated output, the sample expected content can be auto-recreated QEMU_TEST_REGENERATE=1 make check-tracetool and the changes reviewed and added to the commit. This will also assist reviewers interpreting the change. Developers are reminded of this in the test output on failure: $ make check-tracetool 1/6 qemu:tracetool / dtrace OK 0.14s 2/6 qemu:tracetool / ftrace FAIL 0.06s exit status 1 ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――― 1..2 ok 1 - ftrace.c # not ok 1 - ftrace.h (set QEMU_TEST_REGENERATE=1 to recreate reference output if tracetool generator was intentionally changed) ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――― 3/6 qemu:tracetool / log OK 0.06s 4/6 qemu:tracetool / simple OK 0.06s 5/6 qemu:tracetool / syslog OK 0.06s 6/6 qemu:tracetool / ust OK 0.11s Summary of Failures: 2/6 qemu:tracetool / ftrace FAIL 0.06s exit status 1 Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-id: 20250916081638.764020-6-berrange@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
* Remove the remainders of the Avocado testsThomas Huth2025-04-231-56/+4
| | | | | | | | | | | Now that all Avocado tests have been converted to or been replaced by other functional tests, we can delete the remainders of the Avocado tests from the QEMU source tree. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20250414113031.151105-16-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: enable pre-emptive caching of assetsDaniel P. Berrangé2024-09-041-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Many tests need to access assets stored on remote sites. We don't want to download these during test execution when run by meson, since this risks hitting test timeouts when data transfers are slow. Add support for pre-emptive caching of assets by setting the env var QEMU_TEST_PRECACHE to point to a timestamp file. When this is set, instead of running the test, the assets will be downloaded and saved to the cache, then the timestamp file created. A meson custom target is created as a dependency of each test suite to trigger the pre-emptive caching logic before the test runs. When run in caching mode, it will locate assets by looking for class level variables with a name prefix "ASSET_", and type "Asset". At the ninja level ninja test --suite functional will speculatively download any assets that are not already cached, so it is advisable to set a timeout multiplier. QEMU_TEST_NO_DOWNLOAD=1 ninja test --suite functional will fail the test if a required asset is not already cached ninja precache-functional will download and cache all assets required by the functional tests At the make level, precaching is always done by make check-functional Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Tested-by: Richard Henderson <richard.henderson@linaro.org> [thuth: Remove the duplicated "path = os.path.basename(...)" line] Message-ID: <20240830133841.142644-16-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: Prepare the meson build system for the functional testsThomas Huth2024-09-041-0/+11
| | | | | | | | | | | | | | | | | | Provide a meson.build file for the upcoming python-based functional tests, and add some wrapper glue targets to the tests/Makefile.include file. We are going to use two "speed" modes for the functional tests: The "quick" tests can be run at any time (i.e. also during "make check"), while the "thorough" tests should only be run when running a "make check-functional" test run (since these tests might download additional assets from the internet). The changes to the meson.build files are partly based on an earlier patch by Ani Sinha. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20240830133841.142644-12-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/Makefile.include: Increase the level of indentation in the help textThomas Huth2024-09-041-15/+15
| | | | | | | | | | | | The next patch is going to add some entries that need more space between the command and the help text, so let's increase the indentation here first. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-ID: <20240830133841.142644-11-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* Bump avocado to 103.0Cleber Rosa2024-09-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | This bumps Avocado to latest the LTS release. An LTS release is one that can receive bugfixes and guarantees stability for a much longer period and has incremental minor releases made. Even though the 103.0 LTS release is pretty a rewrite of Avocado when compared to 88.1, the behavior of all existing tests under tests/avocado has been extensively tested no regression in behavior was found. To keep behavior of jobs as close as possible with previous version, this version bump keeps the execution serial (maximum of one task at a time being run). Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html Signed-off-by: Cleber Rosa <crosa@redhat.com> Tested-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> Message-ID: <20240806173119.582857-2-crosa@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20240830133841.142644-5-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/avocado: Allow overwriting AVOCADO_SHOW env variablePhilippe Mathieu-Daudé2024-07-221-1/+1
| | | | | | | | | | | | | The 'app' level logging is useful, but sometimes we want more, for example QEMU leverages the 'console' logging. Allow overwriting AVOCADO_SHOW from environment, i.e.: $ make check-avocado AVOCADO_SHOW='app,console' Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20240719180211.48073-1-philmd@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
* meson: Drop the .fa library suffixPaolo Bonzini2024-07-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The non-standard .fa library suffix breaks the link source de-duplication done by Meson so drop it. The lack of link source de-duplication causes AddressSanitizer to complain ODR violations, and makes GNU ld abort when combined with clang's LTO. Fortunately, the non-standard suffix is not necessary anymore for two reasons. First, the non-standard suffix was necessary for fork-fuzzing. Meson wraps all standard-suffixed libraries with --start-group and --end-group. This made a fork-fuzz.ld linker script wrapped as well and broke builds. Commit d2e6f9272d33 ("fuzz: remove fork-fuzzing scaffolding") dropped fork-fuzzing so we can now restore the standard suffix. Second, the libraries are not even built anymore, because it is possible to just use the object files directly via extract_all_objects(). The occurences of the suffix were detected and removed by performing a tree-wide search with 'fa' and .fa (note the quotes and dot). Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com> Message-ID: <20240524-xkb-v4-4-2de564e5c859@daynix.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests: respect --enable/--disable-download for AvocadoPaolo Bonzini2023-11-241-1/+1
| | | | | | | Pass the content of $mkvenv_flags (which is either "--online" or empty) down to tests/Makefile.include. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* Makefile: build plugins before running TCG testsPaolo Bonzini2023-10-021-1/+1
| | | | | | | | | | | Add back test-plugins and, after making sure it is always defined, do so unconditionally. Reported-by: Alex Bennée <alex.bennee@linaro.org> Fixes: 2c13c574418 ("configure, meson: move --enable-plugins to meson", 2023-09-07) Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Tested-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* configure, meson: move --enable-plugins to mesonPaolo Bonzini2023-09-071-1/+1
| | | | | | | | | While the option still needs to be parsed in the configure script (it's needed by tests/tcg, and also to decide about recursing into contrib/plugins), passing it to Meson can be done with -D instead of using config-host.mak. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* Revert "tests: Use separate virtual environment for avocado"Paolo Bonzini2023-08-281-11/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit e8e4298feadae7924cf7600bb3bcc5b0a8d7cbe9. ensuregroup allows to specify both the acceptable versions of avocado, and a locked version to be used when avocado is not installed as a system pacakge. This lets us install avocado in pyvenv/ using "mkvenv.py" and reuse the distro package on Fedora and CentOS Stream (the only distros where it's available). ensuregroup's usage of "(>=..., <=...)" constraints when evaluating the distro package, and "==" constraints when installing it from PyPI, makes it possible to avoid conflicts between the known-good version and a package plugins included in the distro. This is because package plugins have "==" constraints on the version that is included in the distro, and, using "pip install avocado==88.1" on a venv that includes system packages will result in an error: avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. But at the same time, if the venv does not include a system distribution of avocado then we can install a known-good version and stick to LTS releases. Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1663 Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests: Use separate virtual environment for avocadoPaolo Bonzini2023-06-061-7/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commits eea2d141179 ("Makefile: remove $(TESTS_PYTHON)", 2023-05-26) and 9c6692db550 ("tests: Use configure-provided pyvenv for tests", 2023-05-18). Right now, there is a conflict between wanting a ">=" constraint when using a distro-provided package and wanting a "==" constraint when installing Avocado from PyPI; this would provide the best of both worlds in terms of resiliency for both distros that have required packages and distros that don't. The conflict is visible also for meson, where we would like to install the latest 0.63.x version but also accept a distro 1.1.x version. But it is worse for avocado, for two reasons: 1) we cannot use an "==" constraint to install avocado if the venv includes a system avocado. The distro will package plugins that have "==" constraints on the version that is included in the distro, and, using "pip install avocado==88.1" on a venv that includes system packages will result in this error: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. make[1]: Leaving directory '/home/berrange/src/virt/qemu/build' 2) we cannot use ">=" either if the venv does _not_ include a system avocado, because that would result in the installation of v101.0 which is the one we've just reverted. So the idea is to encode the dependencies as an (acceptable, locked) tuple, like this hypothetical TOML that would be committed inside python/ and used by mkvenv.py: [meson] meson = { minimum = "0.63.0", install = "0.63.3", canary = "meson" } [docs] # 6.0 drops support for Python 3.7 sphinx = { minimum = "1.6", install = "<6.0", canary = "sphinx-build" } sphinx_rtd_theme = { minimum = "0.5" } [avocado] avocado-framework = { minimum = "88.1", install = "88.1", canary = "avocado" } Once this is implemented, it would also be possible to install avocado in pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora and CentOS Stream (the only distros where it's available). But until this is implemented, keep avocado in a separate venv. There is still the benefit of using a single python for meson custom_targets and for sphinx. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* Revert "tests/requirements.txt: bump up avocado-framework version to 101.0"Paolo Bonzini2023-06-061-11/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit ec5ffa0056389c3c10ea2de1e78366f66f4e5abc. Bumping avocado to version 101 has two issues. First, there are problems where Avocado is not logging of command lines or terminal output, and not collecting Python logs outside the avocado namespace. Second, the recent changes to Python handling mean that there is a single virtual environment for all the build, instead of a separate one for testing. Requiring a too-new version of avocado causes conflicts with any avocado plugins installed on the host: $ make check-venv make[1]: Entering directory '/home/berrange/src/virt/qemu/build' GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc VENVPIP install -e /home/berrange/src/virt/qemu/python/ VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible. make[1]: Leaving directory '/home/berrange/src/virt/qemu/build' To avoid this issue, tests/requirements.txt should use a ">=" constraint and the version of Avocado should be limited to what distros provide in the system packages. Only Fedora has Avocado, and more specifically version 92.0 (though 98.0 is also available as a module). As a first step, this patch reverts the introduction of a too-new Avocado. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* Makefile: remove $(TESTS_PYTHON)Paolo Bonzini2023-05-261-5/+3
| | | | | | | It is now the same as $(PYTHON), since the latter always points at pyvenv/bin/python3. Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests: Use configure-provided pyvenv for testsJohn Snow2023-05-181-5/+5
| | | | | | | | | | | This patch changes how the avocado tests are provided, ever so slightly. Instead of creating a new testing venv, use the configure-provided 'pyvenv' instead, and install optional packages into that. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-20-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests/requirements.txt: bump up avocado-framework version to 101.0Kautuk Consul2023-04-271-7/+11
| | | | | | | | | | | | | | | | | | | | | Avocado version 101.0 has a fix to re-compute the checksum of an asset file if the algorithm used in the *-CHECKSUM file isn't the same as the one being passed to it by the avocado user (i.e. the avocado_qemu python module). In the earlier avocado versions this fix wasn't there due to which if the checksum wouldn't match the earlier checksum (calculated by a different algorithm), the avocado code would start downloading a fresh image from the internet URL thus making the test-cases take longer to execute. Bump up the avocado-framework version to 101.0. Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> Message-Id: <20230424092249.58552-2-alex.bennee@linaro.org>
* configure: move tests/tcg/Makefile.prereqs to root build directoryPaolo Bonzini2022-10-061-3/+0
| | | | | | | | | It will not be specific to tests/tcg anymore, since it will be possible to build firmware using container-based cross compilers too. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220929114231.583801-26-alex.bennee@linaro.org>
* tests/tcg: add distclean rulePaolo Bonzini2022-10-061-0/+11
| | | | | | Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220929114231.583801-22-alex.bennee@linaro.org>
* tests/tcg: remove -f from Makefile invocationPaolo Bonzini2022-10-061-4/+3
| | | | | | | | | | Instead of linking tests/tcg/Makefile.target into the build tree, name the symbolic link "Makefile" and create it in every target subdirectory. This makes it possible to just invoke "make" in tests/tcg subdirectories. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220929114231.583801-21-alex.bennee@linaro.org>
* tests: simplify Makefile invocation for tests/tcgPaolo Bonzini2022-10-061-6/+3
| | | | | | | | | | | | | | | Remove the DOCKER_SCRIPT and TARGET variable from the Makefile invocation for tests/tcg. For DOCKER_SCRIPT, resolve the path to docker.py in configure; for TARGET, move it to config-$(TARGET).mak and use a symbolic link to break the cycle. The symbolic link is still needed because tests/tcg includes dummy config files for targets that are not buildable. Once that is cleaned up, the symbolic link will go away too. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220929114231.583801-20-alex.bennee@linaro.org>
* tests: install "qemu" namespace package into venvJohn Snow2022-06-061-0/+1
| | | | | | | | | | | | | | | | This patch adds the "qemu" namespace package to the $build/tests/venv directory. It does so in "editable" mode, which means that changes to the source python directory will actively be reflected by the venv. This patch also then removes any sys.path hacking from the avocado test scripts directly. By doing this, the environment of where to find these packages is managed entirely by the virtual environment and not by the scripts themselves. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-7-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests: add quiet-venv-pip macroJohn Snow2022-06-061-6/+6
| | | | | | | | | | | | Factor out the "test venv pip" macro; rewrite the "check-venv" rule to be a little more compact. Replace the "PIP" pseudo-command output with "VENVPIP" to make it 1% more clear that we are talking about using pip to install something into a venv. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-6-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests: silence pip upgrade warnings during venv creationJohn Snow2022-06-061-2/+2
| | | | | | | | | | | Turn off the nag warning coaxing us to upgrade pip. It's not really that interesting to see in CI logs, and as long as nothing is broken -- nothing is broken. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-5-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests: use python3 as the python executable nameJohn Snow2022-06-061-1/+1
| | | | | | | | | | | | | Use "python3" instead of "python" as per PEP0394: https://peps.python.org/pep-0394/ This should always be defined (in a venv, at least!), matching the preferred python shebang of "#!/usr/bin/env python3". Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-4-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests: add "TESTS_PYTHON" variable to MakefileJohn Snow2022-06-061-3/+4
| | | | | | | | | | | | | | | | | | | | This is a convenience feature: $(PYTHON) points to the Python executable we were instructed to use by the configure script. We use that Python to create a virtual environment with the "check-venv" target in tests/Makefile.include. $(TESTS_PYTHON) points to the Python executable belonging to the virtual environment tied to the build. This Python executable is a symlink to the binary used to create the venv, which will be the version provided at configure time. Using $(TESTS_PYTHON) therefore uses the $(PYTHON) executable, but with paths modified to use packages installed to the venv. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-3-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests/Makefile.include: Fix 'make check-help' outputDario Faggioli2022-06-061-15/+15
| | | | | | | | | | | | | | Since commit 3d2f73ef75e ("build: use "meson test" as the test harness"), check-report.tap is no more, and we have check-report.junit.xml. Update the output of 'make check-help', which was still listing 'check-report.tap', accordingly. Fixes: 3d2f73ef75e Signed-off-by: Dario Faggioli <dfaggioli@suse.com> Message-Id: <165366545439.6869.11633009118019728798.stgit@work> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests/tcg: merge configure.sh back into main configure scriptPaolo Bonzini2022-06-011-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | tests/tcg/configure.sh has a complicated story. In the beginning its code ran as part of the creation of config-target.mak files, and that is where it placed the information on the target compiler. However, probing for the buildability of TCG tests required multiple inclusions of config-target.mak in the _main_ Makefile (not in Makefile.target, which took care of building the QEMU executables in the pre-Meson era), which polluted the namespace. Thus, it was moved to a separate directory. It created small config-*.mak files in $(BUILD_DIR)/tests/tcg. Those were also included multiple times, but at least they were small and manageable; this was also an important step in disentangling the TCG tests from Makefile.target. Since then, Meson has allowed the configure script to go on a diet. A few compilation tests survive (mostly for sanitizers) but these days it mostly takes care of command line parsing, looking for tools, and setting up the environment for Meson to do its stuff. It's time to extend configure with the capability to build for more than just one target: not just tests, but also firmware. As a first step, integrate all the logic to find cross compilers in the configure script, and move tests/tcg/configure.sh back there (though as a separate loop, not integrated in the one that generates target configurations for Meson). tests/tcg is actually very close to being buildable as a standalone project, so I actually expect the compiler tests to move back to tests/tcg, as a "configure" script of sorts which would run at Make time after the docker images are built. The GCC tree has a similar idea of doing only bare-bones tree-wide configuration and leaving the rest for Make time. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Acked-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220517092616.1272238-8-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220527153603.887929-19-alex.bennee@linaro.org>
* build: do a full build before running TCG testsPaolo Bonzini2022-06-011-1/+2
| | | | | | | | | | | TCG tests need both QEMU and firmware to be built, so do "ninja all" before trying to run them. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220517092616.1272238-5-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220527153603.887929-16-alex.bennee@linaro.org>
* tests/tcg: invoke Makefile.target directly from QEMU's makefilePaolo Bonzini2022-04-201-19/+21
| | | | | | | | | | | | | | | | Build the "docker.py cc" invocation directly in tests/tcg/configure.sh, and remove the Makefile.qemu wrapper around Makefile.target. The config-*.mak files now include the actual variables used when building the tests, rather than the CROSS_* variables that Makefile.qemu used to "translate". This is a first step towards generalizing the cross-compilation infrastructure so that it can be used for firmware as well. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220401141326.1244422-15-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220419091020.3008144-18-alex.bennee@linaro.org>
* tests/tcg: list test targets in Makefile.prereqsPaolo Bonzini2022-04-201-7/+7
| | | | | | | | | | | Omit the rules altogether for targets that do not have a compiler. Makefile.qemu now is only invoked if the tests are actually built/run. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220401141326.1244422-14-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220419091020.3008144-17-alex.bennee@linaro.org>
* tests/tcg: prepare Makefile.prereqs at configure timePaolo Bonzini2022-04-201-3/+6
| | | | | | | | | | | | List the dependencies of the build-tcg-tests-* and run-tcg-tests-* targets in a Makefile fragment, without going through Makefile.prereqs's "parsing" of config-*.mak. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220401141326.1244422-13-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220419091020.3008144-16-alex.bennee@linaro.org>
* tests/Makefile.include: Let "make clean" remove the TCG tests, tooThomas Huth2022-03-231-1/+1
| | | | | | | | | | | "make clean" should clear all binaries that have been built, but so far it left the TCG tests still in place. Let's make sure that they are now removed, too. Signed-off-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220301085900.1443232-1-thuth@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
* tests: Do not treat the iotests as separate meson test target anymoreThomas Huth2022-03-221-8/+1
| | | | | | | | | | | | | | | | | | | | If there is a failing iotest, the output is currently not logged to the console anymore. To get this working again, we need to run the meson test runner with "--print-errorlogs" (and without "--verbose" due to a current meson bug that will be fixed here: https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ). We could update the "meson test" call in tests/Makefile.include, but actually it's nicer and easier if we simply do not treat the iotests as separate test target anymore and integrate them along with the other test suites. This has the disadvantage of not getting the detailed progress indication there anymore, but since that was only working right in single-threaded "make -j1" mode anyway, it's not a huge loss right now. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-Id: <20220310075048.2303495-1-thuth@redhat.com> Tested-by: Hanna Reitz <hreitz@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
* configure, meson: move guest-agent, tools to mesonPaolo Bonzini2022-02-211-1/+1
| | | | Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests/Makefile.include: clean-up old codeAlex Bennée2022-02-091-4/+0
| | | | | | | | | This is no longer needed since a2ce7dbd91 ("meson: convert tests/qtest to meson", 2020-08-21) Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20220204204335.1689602-2-alex.bennee@linaro.org>
* build: make check-block a meson testPaolo Bonzini2022-01-281-13/+3
| | | | | | | | | | | | "meson test" can be asked to run tests verbosely; this makes it usable also for qemu-iotests's own harness, and it lets "make check-block" reuse mtest2make.py's infrastructure to find and build test dependencies. Adjust check-block.sh to use the standard exit code that reports a test as skipped. Alternatively, in the future we could make it produce TAP output, which is consistent with all other "make check" tests. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests: Fix typo in check-help outputPhilippe Mathieu-Daudé2022-01-181-1/+1
| | | | | | | | Fix typo in 'make check-help' output. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20220111175528.22294-1-f4bug@amsat.org> Signed-off-by: Laurent Vivier <laurent@vivier.eu>
* tests/acceptance: rename tests acceptance to tests avocadoWillian Rampazzo2021-11-081-2/+2
| | | | | | | | | | | | | | | | | | | In the discussion about renaming the `tests/acceptance` [1], the conclusion was that the folders inside `tests` are related to the framework running the tests and not directly related to the type of the tests. This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the CI related files and the documentation. [1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html Reviewed-by: Niek Linnenbank <nieklinnenbank@gmail.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20211105155354.154864-3-willianr@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
* tests/acceptance: introduce new check-avocado targetWillian Rampazzo2021-11-081-5/+12
| | | | | | | | | | | | | | | | | | | | | | This introduces a new `make` target, `check-avocado`, and adds a deprecation message about the `check-acceptance` target. This is a preparation for renaming the `tests/acceptance` folder to `tests/avocado`. The plan is to remove the call to the `check-avocado` target one or two months after the release and leave the warning to force people to move to the new `check-avocado` target. Later, the `check-acceptance` target can be removed. The intent is to avoid a direct impact during the current soft freeze. Suggested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Acked-by: Thomas Huth <thuth@redhat.com> Message-Id: <20211105155354.154864-2-willianr@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
* build: fix "make check" without earlier "make"Paolo Bonzini2021-10-131-5/+12
| | | | | | | | | | | "make check", if not preceded by "make", will not build the tools needed by qemu-iotests. This happens because qemu-iotests, aka "make check-block", is not yet part of meson.build. While at it, remove the reference to the now-dead QEMU_IOTESTS_HELPERS-y variable. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tests: add missing dependency for check-blockPaolo Bonzini2021-10-131-0/+1
| | | | | | qemu-iotests run qemu-storage-daemon, make sure it is up to date. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* python, iotests: remove socket_scm_helperJohn Snow2021-10-121-1/+0
| | | | | | | | | | It's not used anymore, now. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Hanna Reitz <hreitz@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-id: 20210923004938.3999963-11-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
* Acceptance Tests: improve check-acceptance descriptionCleber Rosa2021-09-271-1/+1
| | | | | | | | | | | | The "check-acceptance" make rule won't necessarily run *all* available tests, because it employs a filter based on the currently configured targets. This change in the description of the rule makes that behavior extra clear. Signed-off-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210924185506.2542588-3-crosa@redhat.com>
* tests/Makefile: add AVOCADO_TESTS option to make check-acceptanceWillian Rampazzo2021-09-271-1/+4
| | | | | | | | | | | | | Add the possibility of running all the tests from a single file, or multiple files, running a single test within a file or multiple tests within multiple files using `make check-acceptance` and the AVOCADO_TESTS environment variable. Suggested-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210923161141.232208-4-willianr@redhat.com>
* tests/Makefile: allow control over tags during check-acceptanceWillian Rampazzo2021-09-271-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | Although it is possible to run a specific test using the avocado command-line, a user may want to use a specific tag while running the ``make check-acceptance`` during the development or debugging. This allows using the AVOCADO_TAGS environment variable where the user takes total control of which tests should run based on the tags defined. This also makes the check-acceptance command flexible to restrict tests based on tags while running on CI. e.g.: AVOCADO_TAGS="foo bar baz" make check-acceptance Signed-off-by: Willian Rampazzo <willianr@redhat.com> Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210923161141.232208-2-willianr@redhat.com>
* Acceptance tests: do not try to reuse packages from the systemCleber Rosa2021-07-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | The premise behind the original behavior is that it would save people from downloading Avocado (and other dependencies) if already installed on the system. To be honest, I think it's extremely rare that the same versions described as dependencies will be available on most systems. But, the biggest motivations here are that: 1) Hacking on QEMU in the same system used to develop Avocado leads to confusion with regards to the exact bits that are being used; 2) Not reusing Python packages from system wide installations gives extra assurance that the same behavior will be seen from tests run on different machines; With regards to downloads, pip already caches the downloaded wheels and tarballs under ~/.cache/pip, so there should not be more than one download even if the venv is destroyed and recreated. Signed-off-by: Cleber Rosa <crosa@redhat.com> Message-Id: <20210415215141.1865467-3-crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Signed-off-by: Cleber Rosa <crosa@redhat.com>
* configs: rename default-configs to configs and reorganiseAlex Bennée2021-07-091-1/+1
| | | | | | | | | | | | | | | In preparation for offering variation to our build configurations lets move everything and rename it to default. Common included base configs are also renamed. During the cleanup the stale usb.mak and pci.mak references were removed from MAINTAINERS. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20210707131744.26027-5-alex.bennee@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>