summary refs log tree commit diff stats
path: root/tests/avocado/tuxrun_baselines.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* tests/functional: Convert the Avocado aarch64 tuxrun testsThomas Huth2024-11-251-224/+0
| | | | | | | | | | | | Move the tests to a new file so that they can be run via qemu-system-aarch64 in the functional framework. Since these were the last tests in tests/avocado/tuxrun_baselines.py, we can now remove that file, too. Signed-off-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20241121165806.476008-20-alex.bennee@linaro.org>
* tests/functional: Convert the RV32-on-RV64 riscv testThomas Huth2024-11-071-16/+0
| | | | | | | | | | | | A straggler that has been added to the Avocado framework while the conversion to the functional framework was already in progress... Move it over now, too! Signed-off-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Alistair Francis <alistair.francis@wdc.com> Reviewed-by: LIU Zhiwei <zhiwei_liu@linux.alibaba.com> Message-ID: <20241105103519.341304-1-thuth@redhat.com> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
* tests/avocado: Boot Linux for RV32 cpu on RV64 QEMULIU Zhiwei2024-10-301-0/+16
| | | | | | | | | | | | make check-avocado AVOCADO_TESTS=tests/avocado/tuxrun_baselines.py: \ TuxRunBaselineTest:test_riscv64_rv32 Signed-off-by: LIU Zhiwei <zhiwei_liu@linux.alibaba.com> Suggested-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com> Acked-by: Alistair Francis <alistair.francis@wdc.com> Reviewed-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com> Message-ID: <20240919055048.562-9-zhiwei_liu@linux.alibaba.com> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
* tests/functional: Convert the Avocado sh4 tuxrun testThomas Huth2024-10-211-35/+0
| | | | | | | | Move the test into a new file so that it can be run via qemu-system-sh4 in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-18-thuth@redhat.com>
* tests/functional: Convert the Avocado ppc32 tuxrun testThomas Huth2024-10-211-16/+0
| | | | | | | | Move the test into a new file so that it can be run via qemu-system-ppc in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-16-thuth@redhat.com>
* tests/functional: Convert the Avocado mips64el tuxrun testThomas Huth2024-10-211-16/+0
| | | | | | | | Move the test into a new file so that it can be run via qemu-system-mips64el in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-15-thuth@redhat.com>
* tests/functional: Convert the Avocado mips64 tuxrun testThomas Huth2024-10-211-17/+0
| | | | | | | | Move the test into a new file so that it can be run via qemu-system-mips64 in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-14-thuth@redhat.com>
* tests/functional: Convert the Avocado mipsel tuxrun testThomas Huth2024-10-211-17/+0
| | | | | | | | Move the test into a new file so that it can be run via qemu-system-mipsel in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-13-thuth@redhat.com>
* tests/functional: Convert the Avocado mips tuxrun testThomas Huth2024-10-211-18/+0
| | | | | | | | Move the test into a new file so that it can be run via qemu-system-mips in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-12-thuth@redhat.com>
* tests/functional: Convert the Avocado x86_64 tuxrun testThomas Huth2024-10-211-18/+0
| | | | | | | | Move the tests to a new file so that they can be run via qemu-system-x86_64 in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-11-thuth@redhat.com>
* tests/functional: Convert the Avocado i386 tuxrun testThomas Huth2024-10-211-16/+0
| | | | | | | | Move the tests to a new file so that they can be run via qemu-system-i386 in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-10-thuth@redhat.com>
* tests/functional: Convert the Avocado riscv64 tuxrun testsThomas Huth2024-10-211-31/+0
| | | | | | | | Move the tests to a new file so that they can be run via qemu-system-riscv64 in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-9-thuth@redhat.com>
* tests/functional: Convert the Avocado riscv32 tuxrun testsThomas Huth2024-10-211-31/+0
| | | | | | | | Move the tests to a new file so that they can be run via qemu-system-riscv32 in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-8-thuth@redhat.com>
* tests/functional: Convert the Avocado arm tuxrun testsThomas Huth2024-10-211-56/+0
| | | | | | | | Move the tests to a new file so that they can be run via qemu-system-arm in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-7-thuth@redhat.com>
* tests/functional: Convert the Avocado s390x tuxrun testThomas Huth2024-10-211-17/+0
| | | | | | | | Move the test to a new file so that it can be run via qemu-system-s390x in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-6-thuth@redhat.com>
* tests/functional: Convert the Avocado sparc64 tuxrun testThomas Huth2024-10-211-16/+0
| | | | | | | | Move the test to a new file so that it can be run via qemu-system-sparc64 in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-5-thuth@redhat.com>
* tests/functional: Convert the Avocado ppc64 tuxrun testsThomas Huth2024-10-211-92/+0
| | | | | | | | Move the tests to a new file so that they can be run via qemu-system-ppc64 in the functional framework. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241011131937.377223-3-thuth@redhat.com>
* docs, tests: do not specify scsi=offPaolo Bonzini2024-06-051-1/+1
| | | | | | | This has been the default forever. Acked-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* gitlab: add optional job to run flaky avocado testsAlex Bennée2023-12-011-0/+1
| | | | | | | | | | | | One problem with flaky tests is they often only fail under CI conditions which makes it hard to debug. We add an optional allow_fail job so developers can trigger the only the flaky tests in the CI environment if they are debugging. Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Acked-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20231201093633.2551497-8-alex.bennee@linaro.org>
* docs/devel: rationalise unstable gitlab tests under FLAKY_TESTSAlex Bennée2023-12-011-2/+2
| | | | | | | | | | | | | | | | | It doesn't make sense to have two classes of flaky tests. While it may take the constrained environment of CI to trigger failures easily it doesn't mean they don't occasionally happen on developer machines. As CI is the gating factor to passing there is no point developers running the tests locally anyway unless they are trying to fix things. While we are at it update the language in the docs to discourage the QEMU_TEST_FLAKY_TESTS becoming a permanent solution. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Acked-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20231201093633.2551497-3-alex.bennee@linaro.org>
* avocado, risc-v: add tuxboot tests for 'max' CPUDaniel Henrique Barboza2023-10-121-0/+32
| | | | | | | | | | | | Add smoke tests to ensure that we'll not break the 'max' CPU type when adding new frozen/ratified RISC-V extensions. Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com> Reviewed-by: Alistair Francis <alistair.francis@wdc.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20230912132423.268494-12-dbarboza@ventanamicro.com> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
* tests/avocado: Re-enable MIPS Malta tests (GitLab issue #1884 fixed)Philippe Mathieu-Daudé2023-10-041-4/+0
| | | | | | | | | | | | Commit 18a536f1f8 ("accel/tcg: Always require can_do_io") fixed the GitLab issue #1884: we can now re-enable those tests. This reverts commit f959c3d87ccfa585b105de6964a6261e368cc1da. Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Acked-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20231003063808.66564-1-philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* tests/avocado: Disable MIPS Malta tests due to GitLab issue #1884Philippe Mathieu-Daudé2023-09-201-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | Commit 0d58c66068 ("softmmu: Use async_run_on_cpu in tcg_commit") introduced a regression which is only triggered by the MIPS Malta machine. Since those tests are gatting and disturb the CI workflow, disable them until https://gitlab.com/qemu-project/qemu/-/issues/1866 is fixed. $ make check-avocado \ AVOCADO_TAGS='arch:mipsel arch:mips64el' \ AVOCADO_ALLOW_UNTRUSTED_CODE=1 \ AVOCADO_TIMEOUT_EXPECTED=1 AVOCADO tests/avocado (04/24) tests/avocado/boot_linux_console.py:BootLinuxConsole.test_mips_malta32el_nanomips_4k: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n... (90.39 s) (05/24) tests/avocado/boot_linux_console.py:BootLinuxConsole.test_mips_malta32el_nanomips_16k_up: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n... (90.29 s) (06/24) tests/avocado/boot_linux_console.py:BootLinuxConsole.test_mips_malta32el_nanomips_64k_dbg: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n... (92.53 s) (11/24) tests/avocado/machine_mips_malta.py:MaltaMachineFramebuffer.test_mips_malta_i6400_framebuffer_logo_1core: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n... (25.78 s) RESULTS : PASS 8 | ERROR 0 | FAIL 0 | SKIP 7 | WARN 2 | INTERRUPT 5 | CANCEL 2 JOB TIME : 525.60 s ^^^^^^^^^^^ Reported-by: Thomas Huth <thuth@redhat.com> Suggested-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230913135339.9128-1-philmd@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230914155422.426639-10-alex.bennee@linaro.org>
* tests/avocado/tuxrun_baselines: Fix ppc64 tests for binaries without slirpThomas Huth2023-06-101-0/+1
| | | | | | | | | | | | | | The ppc64 tuxrun tests are currently failing if "slirp" has been disabled in the binary since they are using "-netdev user" now. We have to skip the test if this network backend is missing. Fixes: 6ee3624236 ("improve code coverage for ppc64") Signed-off-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com> Acked-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230606192802.666000-1-thuth@redhat.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com>
* tests/avocado/tuxrun_baselines.py: improve code coverage for ppc64Kautuk Consul2023-04-271-6/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb disabled the boot_linux.py test-case due to which the code coverage for ppc decreased by around 2%. As per the discussion on https://lore.kernel.org/qemu-devel/87sfdpqcy4.fsf@linaro.org/ it was mentioned that the baseline test for ppc64 could be modified to make up this 2% code coverage. This patch attempts to achieve this 2% code coverage by adding various device command line arguments (to ./qemu-system-ppc64) in the tuxrun_baselines.py test-case. The code coverage report with boot_linux.py, without it and finally with these tuxrun_baselines.py changes is as follows: With boot_linux.py ------------------ lines......: 13.8% (58006 of 420997 lines) functions..: 20.7% (7675 of 36993 functions) branches...: 9.2% (22146 of 240611 branches) Without boot_linux.py (without this patch changes) -------------------------------------------------- lines......: 11.9% (50174 of 420997 lines) functions..: 18.8% (6947 of 36993 functions) branches...: 7.4% (17580 of 239017 branches) Without boot_linux.py (with this patch changes) ----------------------------------------------- lines......: 13.8% (58287 of 420997 lines) functions..: 20.7% (7640 of 36993 functions) branches...: 8.4% (20223 of 240611 branches) Rebased on Alex Benee's testing/next branch: https://gitlab.com/stsquad/qemu/-/tree/testing/next Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> Reported-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Harsh Prateek Bora <harshpb@linux.ibm.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230424041830.1275636-1-kconsul@linux.vnet.ibm.com> Message-Id: <20230424092249.58552-13-alex.bennee@linaro.org>
* tests/avocado: use the new snapshots for testingAlex Bennée2023-04-271-26/+144
| | | | | | | | | | The tuxboot images now have a stable snapshot URL so we can enable the checksums and remove the avocado warnings. We will have to update as old snapshots retire but that won't be too frequent. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230424092249.58552-3-alex.bennee@linaro.org>
* tests/avocado: Enable TuxRun/mips64 (big-endian) testPhilippe Mathieu-Daudé2023-03-301-1/+0
| | | | | | | | | | | Now that the previous commit ("hw/mips/gt64xxx_pci: Don't endian-swap GT_PCI0_CFGADDR") fixed the issue accessing the GT64120 PCI config-address register on big-endian targets, we can enable this TuxRun test. Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Acked-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230223220404.63630-1-philmd@linaro.org>
* tests/avocado: don't use tags to define driveAlex Bennée2023-03-221-36/+24
| | | | | | | | | | | | | | | | We are abusing the avocado tags which are intended to provide test selection metadata to provide parameters to our test. This works OK up until the point you need to have ,'s in the field as this is the tag separator character which is the case for a number of the drive parameters. Fix this by making drive a parameter to the common helper function. Fixes: 267fe57c23 (tests: add tuxrun baseline test to avocado) Reviewed-by: David Woodhouse <dwmw@amazon.co.uk> Signed-off-by: David Woodhouse <dwmw@amazon.co.uk> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230315174331.2959-11-alex.bennee@linaro.org>
* tests: add tuxrun baseline test to avocadoAlex Bennée2023-03-011-0/+423
The TuxRun project (www.tuxrun.org) uses QEMU to run tests on a wide variety of kernel configurations on wide range of our emulated platforms. They publish a known good set of images at: https://storage.tuxboot.com/ to help with bisecting regressions in either the kernel, firmware or QEMU itself. The tests are pretty lightweight as they contain just a kernel with a minimal rootfs which boots a lot faster than most of the distros. In time they might be persuaded to version their known good baselines and we can then enable proper checksums. For a couple of tests we currently skip: - mips64, a regression against previous stable release - sh4, very unstable with intermittent oops Total run time: 340s (default) -> 890s (debug) Overall coverage rate (tested targets + disabled tests): lines......: 16.1% (126894 of 789848 lines) functions..: 20.6% (15954 of 77489 functions) branches...: 9.3% (40727 of 439365 branches) Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Cc: Anders Roxell <anders.roxell@linaro.org> Acked-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230228190653.1602033-11-alex.bennee@linaro.org>