summary refs log tree commit diff stats
path: root/tests/functional/qemu_test/testcase.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* tests/functional: purge scratch dir on test startupDaniel P. Berrangé2025-09-091-0/+4
| | | | | | | | | | | | | | | The test suite purges the scratch dir in the tearDown method, but if python crashes (or is non-gracefully killed) this won't get run. Also the user can set QEMU_TEST_KEEP_SCRATCH to disable cleanup. Purging the scratch dir on startup ensures that tests always run from a clean state. Reported-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-ID: <20250908135722.3375580-5-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: avoid tearDown failure when QEMU diesDaniel P. Berrangé2025-09-091-1/+4
| | | | | | | | | | | | | | | | | | | | | | | In a QEMU process under test dies unexpectedly, the 'shutdown' method may well raise an exception. This causes the tearDown method to fail, which means any later cleanup code fails to get run. Most notably the log handlers don't get removed so the base.log file from an earlier test will get polluted with messages from any subsequent tests. The tearDown failure also results in pages of exceptions printed on the console, which obscures the real failure message / trace printed by the test. Ignore any shutdown failures in the tearDown method, since any test which cares about clean shutdown should have already cleaned up any running VMs. The tearDown method is just there as a safety net to cleanup resources. The base.log file will still containing log messages from the failed 'vm.shutdown' call too. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-ID: <20250908135722.3375580-4-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: avoid duplicate messages on failuresDaniel P. Berrangé2025-09-091-2/+3
| | | | | | | | | | | | | | | | In some scenarios the same tests is mentioned in both the 'res.results.errors' and 'res.results.failures' array returned by unittest.main(). This was seen when the 'tearDown' method raised an exception. In such a case, we printed out the same information about where to find a log file twice for each test. Track which tests we have already reported on, to avoid the duplication. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-ID: <20250908135722.3375580-3-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: Mark main in QemuBaseTest class as a static methodGustavo Romero2025-08-271-0/+1
| | | | | | | | | | | | | | | The main() method in the QemuBaseTest class has no parameters but is defined as a regular method. Currently, this does not cause any issues because in the functional tests main() is always called directly from QemuBaseTest (never from instances), but the way this method is defined makes its signature wrong, implying a 'self'. Hence, it's best practice to define such a method as a static method, so decorate it with @staticmethod. Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org> Message-ID: <20250819143916.4138035-4-gustavo.romero@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: expose sys.argv to unittest.mainAlex Bennée2025-07-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With this we can call the supported we can take advantage of the argument the module supports: env PYTHONPATH=/home/alex/lsrc/qemu.git/python:/home/alex/lsrc/qemu.git/tests/functional ./pyvenv/bin/python /home/alex/lsrc/qemu.git/tests/functional/test_aarch64_kvm.py --help usage: test_aarch64_kvm.py [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b] [-k TESTNAMEPATTERNS] [tests ...] positional arguments: tests a list of any number of test modules, classes and test methods. options: -h, --help show this help message and exit -v, --verbose Verbose output -q, --quiet Quiet output --locals Show local variables in tracebacks --durations N Show the N slowest test cases (N=0 for all) -f, --failfast Stop on first fail or error -c, --catch Catch Ctrl-C and display results so far -b, --buffer Buffer stdout and stderr during tests -k TESTNAMEPATTERNS Only run tests which match the given substring Examples: test_aarch64_kvm.py test_module - run tests from test_module test_aarch64_kvm.py module.TestClass - run tests from module.TestClass test_aarch64_kvm.py module.Class.test_method - run specified test method test_aarch64_kvm.py path/to/test_file.py - run tests from test_file.py usage: test_aarch64_kvm.py discover [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b] [-k TESTNAMEPATTERNS] [-s START] [-p PATTERN] [-t TOP] options: -h, --help show this help message and exit -v, --verbose Verbose output -q, --quiet Quiet output --locals Show local variables in tracebacks --durations N Show the N slowest test cases (N=0 for all) -f, --failfast Stop on first fail or error -c, --catch Catch Ctrl-C and display results so far -b, --buffer Buffer stdout and stderr during tests -k TESTNAMEPATTERNS Only run tests which match the given substring -s, --start-directory START Directory to start discovery ('.' default) -p, --pattern PATTERN Pattern to match tests ('test*.py' default) -t, --top-level-directory TOP Top level directory of project (defaults to start directory) For test discovery all test modules must be importable from the top level directory of the project. Suggested-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-ID: <20250725154517.3523095-15-alex.bennee@linaro.org>
* functional: always enable all python warningsDaniel P. Berrangé2025-07-211-0/+4
| | | | | | | | | | | | | | | Of most importance is that this gives us a heads-up if anything we rely on has been deprecated. The default python behaviour only emits a warning if triggered from __main__ which is very limited. Setting the env variable further ensures that any python child processes will also display warnings. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Acked-by: Thomas Huth <thuth@redhat.com> Message-ID: <20250715143023.1851000-11-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* functional: ensure log handlers are closedDaniel P. Berrangé2025-07-211-0/+2
| | | | | | | | | | This avoids a resource leak warning from python when the log handler is garbage collected. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-ID: <20250715143023.1851000-9-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: Add hvf_available() helperPeter Maydell2025-07-011-2/+4
| | | | | | | | Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-id: 20250623121845.7214-26-philmd@linaro.org [PMM: tweaks to satisfy the python linter CI job] Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
* tests/functional: remove all class level fieldsDaniel P. Berrangé2025-03-191-6/+0
| | | | | | | | | | | | | | | | A number of fields are set at the class level on QemuBaseTest, even though the exact same named field is then set at the object level later in most cases. The 'self.logger' initialization in ACPI bits test needs to be removed since 'self.log' won't exist at that point in the flow. It already initialized 'self.logger' later in the setUp() method, so the __init__ method was redundant. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20250317124300.84266-1-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: Allow running TCG plugins tests on non-Linux/BSD hostsPhilippe Mathieu-Daudé2025-03-101-1/+11
| | | | | | | | | | | | | | | | | Not all platforms use the '.so' suffix for shared libraries, which is how plugins are built. Use the recently introduced dso_suffix() helper to get the proper host suffix. Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2804 Suggested-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Suggested-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20250220080215.49165-4-philmd@linaro.org> [AJB: moved plugin_file into testcase.py] Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20250304222439.2035603-11-alex.bennee@linaro.org>
* tests/functional: set 'qemu_bin' as an object level fieldDaniel P. Berrangé2025-03-061-1/+1
| | | | | | | | | | | | | | The 'qemu_bin' field is currently set on the class, despite being accessed as if it were an object instance field with 'self.qemu_bin'. This is no obvious need to have it as a class field, so move it into the object instance. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20250228102738.3064045-3-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: remove unused 'bin_prefix' variableDaniel P. Berrangé2025-03-061-3/+3
| | | | | | | | | | | This was copied over from avocado but has not been used in the new functional tests. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20250228102738.3064045-2-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: skip tests if assets are not availableDaniel P. Berrangé2024-12-171-0/+11
| | | | | | | | | | If downloading of assets has been disabled, then skip running a test if the assets it has registered are not already downloaded. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20241217155953.3950506-31-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: replace 'run_cmd' with subprocess helpersDaniel P. Berrangé2024-12-171-14/+17
| | | | | | | | | | | The 'run_cmd' helper is re-implementing a convenient helper that already exists in the form of the 'run' and 'check_call' methods provided by 'subprocess'. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20241217155953.3950506-29-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: add 'uncompress' to QemuBaseTestDaniel P. Berrangé2024-12-171-0/+25
| | | | | | | | | | | This helper wrappers utils.uncompress, forcing the use of the scratch directory, to ensure any uncompressed files are cleaned at test termination. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20241217155953.3950506-26-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: add 'archive_extract' to QemuBaseTestDaniel P. Berrangé2024-12-171-0/+32
| | | | | | | | | | | | This helper wrappers archive.archive_extract, forcing the use of the scratch directory, to ensure any extracted files are cleaned at test termination. If a specific member is requested, then the path to the extracted file is also returned. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20241217155953.3950506-23-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: switch over to using self.build_file(...)Daniel P. Berrangé2024-12-171-2/+2
| | | | | | | | | This removes direct access of the 'BUILD_DIR' variable. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20241217155953.3950506-13-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: switch over to using self.log_file(...)Daniel P. Berrangé2024-12-171-5/+4
| | | | | | | | | This removes direct access of the 'self.logdir' variable. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20241217155953.3950506-12-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: add helpers for building file pathsDaniel P. Berrangé2024-12-171-0/+95
| | | | | | | | | | | | | | | | | | | Add helper methods that construct paths for * log files - to be preserved at the end of a test * scratch files - to be purged at the end of a test * build files - anything relative to the build root * data files - anything relative to the functional test source root * socket files - a short temporary dir to avoid UNIX socket limits These are to be used instead of direct access to the self.workdir, or self.logdir variables, or any other place where paths are built manually. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20241217155953.3950506-11-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: avoid accessing log_filename on earlier failuresDaniel P. Berrangé2024-11-251-4/+6
| | | | | | | | | | | | | | | If a failure occurs early in the QemuBaseTest constructor, the 'log_filename' object atttribute may not exist yet. This happens most notably if the QEMU_TEST_QEMU_BINARY is not set. We can't initialize 'log_filename' earlier as we use the binary to identify the architecture which is then used to build the path in which the logs are stored. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20241121154218.1423005-19-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20241121165806.476008-19-alex.bennee@linaro.org>
* tests/functional: add a QMP backdoor for debugging stalled testsDaniel P. Berrangé2024-11-251-0/+7
| | | | | | | | | | | | | | | | | | | | | Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a way to get a QMP backdoor for debugging a stalled QEMU test. Most typically this would be used if running the tests directly: $ QEMU_TEST_QMP_BACKDOOR=backdoor.sock \ QEMU_TEST_QEMU_BINARY=./build/qemu-system-arm \ PYTHONPATH=./python \ ./tests/functional/test_arm_tuxrun.py And then, when the test stalls, in a second shell run: $ ./scripts/qmp/qmp-shell backdoor.sock Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20241121154218.1423005-18-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20241121165806.476008-18-alex.bennee@linaro.org>
* tests/functional: enable debug logging for QEMUMachineDaniel P. Berrangé2024-11-251-0/+6
| | | | | | | | | | | | | | | Set the 'qemu.machine' logger to 'DEBUG' level, to ensure we see log messages related to the QEMUMachine class. Most importantly this ensures we capture the full QEMU command line args for instances we spawn. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20241121154218.1423005-12-berrange@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20241121165806.476008-12-alex.bennee@linaro.org>
* tests/functional: honour requested test VM name in QEMUMachineDaniel P. Berrangé2024-11-251-0/+1
| | | | | | | | | | | | | | The functional test case class is going to the trouble of passing around a machine name, but then fails to give this QEMUMachine. As a result, QEMUMachine will create a completely random name. Since log file names match the machine name, this results in log files accumulating over time. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20241121154218.1423005-11-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20241121165806.476008-11-alex.bennee@linaro.org>
* tests/functional: put QEMUMachine logs in testcase log directoryDaniel P. Berrangé2024-11-251-2/+3
| | | | | | | | | | | | | | | | We are not passing the 'log_dir' parameter to QEMUMachine, so the QEMU stdout/err logs are being placed in a temp directory and thus deleted after execution. This makes them inaccessible as gitlab CI artifacts. Pass the testcase log directory path into QEMUMachine to make the logs persistent. Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20241121154218.1423005-10-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20241121165806.476008-10-alex.bennee@linaro.org>
* tests/functional: automatically clean up scratch files after testsDaniel P. Berrangé2024-11-251-5/+9
| | | | | | | | | | | | | | | | The build/tests/functional subdirectories are consuming huge amounts of disk space. Split the location for scratch files into a 'scratch' sub-directory, separate from log files, and delete it upon completion of each test. The new env variable QEMU_TEST_KEEP_SCRATCH can be set to preserve this scratch dir for debugging access if required. Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20241121154218.1423005-3-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20241121165806.476008-3-alex.bennee@linaro.org>
* tests/functional: Provide the user with hints where to find more log filesThomas Huth2024-11-081-5/+13
| | | | | | | | | | | | | | Since the base.log and console.log files are not referenced from the meson test logs yet, they might be hard to find for the casual users. Thus let's print some hints in case a test case failed. For this we have to run unittest.main() with exit=False to get the results of the testing. Then we can iterate through the failed test cases to print out the information accordingly. Message-ID: <20241105123849.359391-1-thuth@redhat.com> Reviewed-by: Cédric Le Goater <clg@redhat.com> Tested-by: Cédric Le Goater <clg@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: Add QemuUserTest classPhilippe Mathieu-Daudé2024-09-041-0/+17
| | | | | | | | | | | | | | | Per commit 5334df4822 ("tests/avocado: Introduce QemuUserTest base class"): Similarly to the 'System' Test base class with methods for testing system emulation, the QemuUserTest class contains methods useful to test user-mode emulation. Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20240822104238.75045-2-philmd@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-ID: <20240830133841.142644-38-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: enable pre-emptive caching of assetsDaniel P. Berrangé2024-09-041-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Many tests need to access assets stored on remote sites. We don't want to download these during test execution when run by meson, since this risks hitting test timeouts when data transfers are slow. Add support for pre-emptive caching of assets by setting the env var QEMU_TEST_PRECACHE to point to a timestamp file. When this is set, instead of running the test, the assets will be downloaded and saved to the cache, then the timestamp file created. A meson custom target is created as a dependency of each test suite to trigger the pre-emptive caching logic before the test runs. When run in caching mode, it will locate assets by looking for class level variables with a name prefix "ASSET_", and type "Asset". At the ninja level ninja test --suite functional will speculatively download any assets that are not already cached, so it is advisable to set a timeout multiplier. QEMU_TEST_NO_DOWNLOAD=1 ninja test --suite functional will fail the test if a required asset is not already cached ninja precache-functional will download and cache all assets required by the functional tests At the make level, precaching is always done by make check-functional Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Tested-by: Richard Henderson <richard.henderson@linaro.org> [thuth: Remove the duplicated "path = os.path.basename(...)" line] Message-ID: <20240830133841.142644-16-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: Set up loggingThomas Huth2024-09-041-1/+26
| | | | | | | | | Create log files for each test separately, one file that contains the basic logging and one that contains the console output. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20240830133841.142644-10-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: Add base classes for the upcoming pytest-based testsThomas Huth2024-09-041-0/+153
The files are mostly a copy of the tests/avocado/avocado_qemu/__init__.py file with some adjustments to get rid of the Avocado dependencies (i.e. we also have to drop the LinuxSSHMixIn and LinuxTest for now). The emulator binary and build directory are now passed via environment variables that will be set via meson.build later. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20240830133841.142644-9-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>