Merge tag 'pull-request-2024-09-04' of https://gitlab.com/thuth/qemu into staging

* Bump Avocado to version 103
* Introduce new functional test framework for Python-based tests
* Convert many Avocado tests to the new functional test framework

# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEEJ7iIR+7gJQEY8+q5LtnXdP5wLbUFAmbYOEsRHHRodXRoQHJl
# ZGhhdC5jb20ACgkQLtnXdP5wLbUDAA/+Kdlak/nCrK5gXDyDasfy86IxgMD0QlDR
# U0MOpQyfXbM2EJjwCUhmgo8pui8qV23dKzfCwbDmkjB7mJ+yKi2ZdiFEp6onq/ke
# aAdaaZwENtWcFglRD80TOSQX6oyeNmE/PuvJGG0BfwWXyyhaEa6kCdytEPORipQs
# lZ+ZndHgXtcM3roXtgI3kp2V1nY5LLCJ044UrasKRq2xWfD/Ken90uWP5/nMLV7f
# 7YLRUIb0sgV7IdjZiT1UkXJZRB7MatV7+OsojYbG8BPbQEvXqpryXMIeygHVR9a0
# yxNDUpTZR6JoS1IaLKkHh1mTM+L1JpFltKadKkXa0zqJHHSur7Tp0xVO/GeqCek4
# 9N8K4zw2CoO/AKmN8JjW5i4GnMrFMdcvxxNwLdRoVgYt4YA731wnHrbosXZOXcuv
# H0z8Tm6ueKvfBtrQErdvqsGrP/8FUYRqZP4H6XaaC+wEis++7OmVR2nlQ/gAyr6/
# mMJtmxqVHCIcEVjDu1jYltrW3BN2CcxN2M9gxyOScq2/Xmzqtaeb4iyjxeCUjIBW
# Pc4LXlSafIg3hPrdH3EKN275ev8cx/5jp8oEgXD5We25Mj3W930zde6/STXoX318
# NVNlbrIQjGjQN7rN5oxTFxTlIN8ax2tuuzpQDFvS/4bLyMYXcZ4I5gUrM5tvWTGv
# +0UN45pJ7Nk=
# =l6Ki
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 04 Sep 2024 11:36:59 BST
# gpg:                using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5
# gpg:                issuer "thuth@redhat.com"
# gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full]
# gpg:                 aka "Thomas Huth <thuth@redhat.com>" [full]
# gpg:                 aka "Thomas Huth <huth@tuxfamily.org>" [full]
# gpg:                 aka "Thomas Huth <th.huth@posteo.de>" [unknown]
# Primary key fingerprint: 27B8 8847 EEE0 2501 18F3  EAB9 2ED9 D774 FE70 2DB5

* tag 'pull-request-2024-09-04' of https://gitlab.com/thuth/qemu: (42 commits)
  docs/devel/testing: Add documentation for functional tests
  docs/devel/testing: Rename avocado_qemu.Test class
  docs/devel/testing: Split the Avocado documentation into a separate file
  docs/devel: Split testing docs from the build docs and move to separate folder
  gitlab-ci: Add "check-functional" to the build tests
  tests/avocado: Remove unused QemuUserTest class
  tests/functional: Convert ARM bFLT linux-user avocado test
  tests/functional: Add QemuUserTest class
  tests/functional: Convert mips64el Fuloong2e avocado test (1/2)
  tests/functional: Convert Aarch64 Virt machine avocado tests
  tests/functional: Convert Aarch64 SBSA-Ref avocado tests
  tests/functional: Convert ARM Integrator/CP avocado tests
  tests/functional: Convert the linux_initrd avocado test into a standalone test
  tests/functional: Convert the rx_gdbsim avocado test into a standalone test
  tests/functional: Convert the acpi-bits test into a standalone test
  tests/functional: Convert the m68k nextcube test with tesseract
  tests/functional: Convert the ppc_hv avocado test into a standalone test
  tests/functional: Convert the ppc_amiga avocado test into a standalone test
  tests/functional: Convert most ppc avocado tests into standalone tests
  tests/functional: Convert the virtio_gpu avocado test into a standalone test
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

1 files changed, 155 insertions, 0 deletions

diff --git a/docs/devel/testing/acpi-bits.rst b/docs/devel/testing/acpi-bits.rst
new file mode 100644
index 0000000000..78aeb6aa3c
--- /dev/null
+++ b/docs/devel/testing/acpi-bits.rst
@@ -0,0 +1,155 @@
+==================================
+ACPI/SMBIOS testing using biosbits
+==================================
+************
+Introduction
+************
+Biosbits is a software written by Josh Triplett that can be downloaded
+from https://biosbits.org/. The github codebase can be found
+`here <https://github.com/biosbits/bits/tree/master>`__. It is a software that
+executes the bios components such as acpi and smbios tables directly through
+acpica bios interpreter (a freely available C based library written by Intel,
+downloadable from https://acpica.org/ and is included with biosbits) without an
+operating system getting involved in between. Bios-bits has python integration
+with grub so actual routines that executes bios components can be written in
+python instead of bash-ish (grub's native scripting language).
+There are several advantages to directly testing the bios in a real physical
+machine or in a VM as opposed to indirectly discovering bios issues through the
+operating system (the OS). Operating systems tend to bypass bios problems and
+hide them from the end user. We have more control of what we wanted to test and
+how by being as close to the bios on a running system as possible without a
+complicated software component such as an operating system coming in between.
+Another issue is that we cannot exercise bios components such as ACPI and
+SMBIOS without being in the highest hardware privilege level, ring 0 for
+example in case of x86. Since the OS executes from ring 0 whereas normal user
+land software resides in unprivileged ring 3, operating system must be modified
+in order to write our test routines that exercise and test the bios. This is
+not possible in all cases. Lastly, test frameworks and routines are preferably
+written using a high level scripting language such as python. OSes and
+OS modules are generally written using low level languages such as C and
+low level assembly machine language. Writing test routines in a low level
+language makes things more cumbersome. These and other reasons makes using
+bios-bits very attractive for testing bioses. More details on the inspiration
+for developing biosbits and its real life uses can be found in [#a]_ and [#b]_.
+
+For QEMU, we maintain a fork of bios bits in gitlab along with all the
+dependent submodules `here <https://gitlab.com/qemu-project/biosbits-bits>`__.
+This fork contains numerous fixes, a newer acpica and changes specific to
+running these functional QEMU tests using bits. The author of this document
+is the sole maintainer of the QEMU fork of bios bits repository. For more
+information, please see author's `FOSDEM talk on this bios-bits based test
+framework <https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/>`__.
+
+*********************************
+Description of the test framework
+*********************************
+
+Under the directory ``tests/functional/``, ``test_acpi_bits.py`` is a QEMU
+functional test that drives all this.
+
+A brief description of the various test files follows.
+
+Under ``tests/functional/`` as the root we have:
+
+::
+
+   ├── acpi-bits
+   │ ├── bits-config
+   │ │ └── bits-cfg.txt
+   │ ├── bits-tests
+   │   ├── smbios.py2
+   │   ├── testacpi.py2
+   │   └── testcpuid.py2
+   ├── test_acpi_bits.py
+
+* ``tests/functional``:
+
+   ``test_acpi_bits.py``:
+   This is the main python functional test script that generates a
+   biosbits iso. It then spawns a QEMU VM with it, collects the log and reports
+   test failures. This is the script one would be interested in if they wanted
+   to add or change some component of the log parsing, add a new command line
+   to alter how QEMU is spawned etc. Test writers typically would not need to
+   modify this script unless they wanted to enhance or change the log parsing
+   for their tests. In order to enable debugging, you can set **V=1**
+   environment variable. This enables verbose mode for the test and also dumps
+   the entire log from bios bits and more information in case failure happens.
+   You can also set **BITS_DEBUG=1** to turn on debug mode. It will enable
+   verbose logs and also retain the temporary work directory the test used for
+   you to inspect and run the specific commands manually.
+
+   In order to run this test, please perform the following steps from the QEMU
+   build directory (assuming that the sources are in ".."):
+   ::
+
+     $ export PYTHONPATH=../python:../tests/functional
+     $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
+     $ python3 ../tests/functional/test_acpi_bits.py
+
+   The above will run all acpi-bits functional tests (producing output in
+   tap format).
+
+   You can inspect the log files in tests/functional/x86_64/test_acpi_bits.*/
+   for more information about the run or in order to diagnoze issues.
+   If you pass V=1 in the environment, more diagnostic logs will be put into
+   the test log.
+
+* ``tests/functional/acpi-bits/bits-config``:
+
+   This location contains biosbits configuration files that determine how the
+   software runs the tests.
+
+   ``bits-config.txt``:
+   This is the biosbits config file that determines what tests
+   or actions are performed by bits. The description of the config options are
+   provided in the file itself.
+
+* ``tests/functional/acpi-bits/bits-tests``:
+
+   This directory contains biosbits python based tests that are run from within
+   the biosbits environment in the spawned VM. New additions of test cases can
+   be made in the appropriate test file. For example, new acpi tests can go
+   into testacpi.py2 and one would call testsuite.add_test() to register the new
+   test so that it gets executed as a part of the ACPI tests.
+   It might be occasionally necessary to disable some subtests or add a new
+   test that belongs to a test suite not already present in this directory. To
+   do this, please clone the bits source from
+   https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits.
+   Note that this is the "qemu-bits" branch and not the "bits" branch of the
+   repository. "qemu-bits" is the branch where we have made all the QEMU
+   specific enhancements and we must use the source from this branch only.
+   Copy the test suite/script that needs modification (addition of new tests
+   or disabling them) from python directory into this directory. For
+   example, in order to change cpuid related tests, copy the following
+   file into this directory and rename it with .py2 extension:
+   https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py
+   Then make your additions and changes here. Therefore, the steps are:
+
+       (a) Copy unmodified test script to this directory from bits source.
+       (b) Add a SPDX license header.
+       (c) Perform modifications to the test.
+
+   Commits (a), (b) and (c) preferably should go under separate commits so that
+   the original test script and the changes we have made are separated and
+   clear. (a) and (b) can sometimes be combined into a single step.
+
+   The test framework will then use your modified test script to run the test.
+   No further changes would be needed. Please check the logs to make sure that
+   appropriate changes have taken effect.
+
+   The tests have an extension .py2 in order to indicate that:
+
+   (a) They are python2.7 based scripts and not python 3 scripts.
+   (b) They are run from within the bios bits VM and is not subjected to QEMU
+       build/test python script maintenance and dependency resolutions.
+   (c) They need not be loaded by the test framework by accident when running
+       tests.
+
+
+Author: Ani Sinha <anisinha@redhat.com>
+
+References:
+-----------
+.. [#a] https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
+.. [#b] https://www.youtube.com/watch?v=36QIepyUuhg
+.. [#c] https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/