summary refs log tree commit diff stats
path: root/tests/functional/qemu_test/asset.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* tests/functional: treat unknown exceptions as transient faultsDaniel P. Berrangé2025-09-241-1/+2
| | | | | | | | | | | | | | | | | | | | | | | To maximise the robustness of the functional tests we want to treat most asset download failures as non-fatal to the test suite. Instead it should just skip the tests which need that particular asset. The only time aim to make it fatal is for 404 errors which are highly likely to reflect genuine problems to be fixed. We catch certain exception classes and handle them as transient errors, but unfortunately it is proving difficult to predict what exception classes urlopen() is capable of raising, with new possibilities being discovered. To provide a fail-safe, treat the generic Exception class as being a transient error too. This may well mask certain genuine bugs, but it is preferrable to prioritize running the test suite to the greatest extent practical. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-ID: <20250918125746.1165658-3-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: retry when seeing ConnectionError exceptionDaniel P. Berrangé2025-09-241-0/+7
| | | | | | | | | | | | | This base class is used for many different socket connection errors, corresponding to ECONNRESET, ECONNREFUSED, ECONNABORTED and more. Most of these are things you might expect to see every now and then as transient flaws. We should thus retry the asset download when seeing them. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-ID: <20250918125746.1165658-2-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: handle URLError when fetching assetsDaniel P. Berrangé2025-09-091-1/+9
| | | | | | | | | | | | | We treat most HTTP errors as non-fatal when fetching assets, but forgot to handle network level errors. This adds catching of URLError so that we retry on failure, and will ultimately trigger graceful skipping in the pre-cache task. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-ID: <20250829142616.2633254-4-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: fix formatting of exception argsDaniel P. Berrangé2025-09-091-1/+1
| | | | | | | | | | | The catch-all exception handler forgot the placeholder for the exception details. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20250829142616.2633254-3-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: enable force refresh of cached assetsDaniel P. Berrangé2025-09-091-0/+4
| | | | | | | | | | | | | | If the 'QEMU_TEST_REFRESH_CACHE' environment variable is set, then ignore any existing cached asset and download a fresh copy. This can be used to selectively refresh assets if set before running a single test script. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20250829142616.2633254-2-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional/asset: Add AssetError exception classNicholas Piggin2025-03-121-15/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | Assets are uniquely identified by human-readable-ish url, so make an AssetError exception class that prints url with error message. A property 'transient' is used to capture whether the client may retry or try again later, or if it is a serious and likely permanent error. This is used to retain the existing behaviour of treating HTTP errors other than 404 as 'transient' and not causing precache step to fail. Additionally, partial-downloads and stale asset caches that fail to resolve after the retry limit are now treated as transient and do not cause precache step to fail. For background: The NetBSD archive is, at the time of writing, failing with short transfer. Retrying the fetch at that position (as wget does) results in a "503 backend unavailable" error. We would like to get that error code directly, but I have not found a way to do that with urllib, so treating the short-copy as a transient failure covers that case (and seems like a reasonable way to handle it in general). Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Nicholas Piggin <npiggin@gmail.com> Message-ID: <20250312130002.945508-4-npiggin@gmail.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional/asset: Verify downloaded sizeNicholas Piggin2025-03-121-0/+14
| | | | | | | | | | | | If the server provides a Content-Length header, use that to verify the size of the downloaded file. This catches cases where the connection terminates early, and gives the opportunity to retry. Without this, the checksum will likely mismatch and fail without retry. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Nicholas Piggin <npiggin@gmail.com> Message-ID: <20250312130002.945508-3-npiggin@gmail.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional/asset: Fail assert fetch when retries are exceededNicholas Piggin2025-03-121-0/+3
| | | | | | | | | | | | | | Currently the fetch code does not fail gracefully when retry limit is exceeded, it just falls through the loop with no file, which ends up hitting other errors. Add a check for non-existing file, which indicates the retry limit was exceeded. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Nicholas Piggin <npiggin@gmail.com> Message-ID: <20250312130002.945508-2-npiggin@gmail.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: ignore errors when caching assets, except for 404Daniel P. Berrangé2024-12-171-1/+13
| | | | | | | | | | | | | | | We see periodic errors caching assets due to a combination of transient networking and server problems. With the previous patch to skip running a test when it has missing assets, we can now treat most cache download errors as non-fatal. Only HTTP 404 is retained as fatal, since it is a strong indicator of a fully broken test rather than a transient error. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20241217155953.3950506-32-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: skip tests if assets are not availableDaniel P. Berrangé2024-12-171-1/+7
| | | | | | | | | | If downloading of assets has been disabled, then skip running a test if the assets it has registered are not already downloaded. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20241217155953.3950506-31-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: resolve str(Asset) to cache file pathDaniel P. Berrangé2024-12-171-0/+3
| | | | | | | | | | | Allow an Asset object to be used in place of a filename but making its string representation resolve to the cache file path. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20241217155953.3950506-4-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: remove many unused importsDaniel P. Berrangé2024-12-171-1/+0
| | | | | | | | | Identified using 'pylint --disable=all --enable=W0611' Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20241217155953.3950506-3-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: make cached asset files read-onlyDaniel P. Berrangé2024-11-041-0/+3
| | | | | | | | | | | | | | | | | | | This ensures that if a functional test runs QEMU with a writable disk pointing to a cached asset, an error will be reported, rather than silently modifying the cache file. As an example, tweaking test_sbsaref.py to set snapshot=off, results in a clear error: Command: ./build/qemu-system-aarch64 ...snip... -drive file=/var/home/berrange/.cache/qemu/download/44cdbae275ef1bb6dab1d5fbb59473d4f741e1c8ea8a80fd9e906b531d6ad461,format=raw,snapshot=off -cpu max,pauth=off Output: qemu-system-aarch64: Could not open '/var/home/berrange/.cache/qemu/download/44cdbae275ef1bb6dab1d5fbb59473d4f741e1c8ea8a80fd9e906b531d6ad461': Permission denied Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Message-ID: <20241025092659.2312118-3-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: Fix hash validationThomas Huth2024-10-041-1/+1
| | | | | | | | | | | | | | The _check() function is supposed to check whether the hash of the downloaded file matches the expected one. Unfortunately, during the last rework of this function, the check was accidentally turned into returning the hash value itself instead of a True/False value, effectively accepting each hash as valid. Let's do a proper check again now. Fixes:05e303210d ("tests/functional/qemu_test: Use Python hashlib ...") Signed-off-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Michael Tokarev <mjt@tls.msk.ru> Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
* tests/functional/qemu_test: Use Python hashlib instead of external programsThomas Huth2024-09-111-5/+11
| | | | | | | | | | | Some systems (like OpenBSD) do not have the sha256sum or sha512sum programs installed by default, or use different names for those. Use the Python hashlib instead so we don't have to rely on the external programs. Reported-by: Peter Maydell <peter.maydell@linaro.org> Message-ID: <20240910201742.239559-1-thuth@redhat.com> Reviewed-by: Brian Cain <bcain@quicinc.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: Allow asset downloading with concurrent threadsThomas Huth2024-09-041-11/+51
| | | | | | | | | | When running "make -j$(nproc) check-functional", tests that use the same asset might be running in parallel. Improve the downloading to detect this situation and wait for the other thread to finish the download. Message-ID: <20240830133841.142644-17-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: enable pre-emptive caching of assetsDaniel P. Berrangé2024-09-041-0/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Many tests need to access assets stored on remote sites. We don't want to download these during test execution when run by meson, since this risks hitting test timeouts when data transfers are slow. Add support for pre-emptive caching of assets by setting the env var QEMU_TEST_PRECACHE to point to a timestamp file. When this is set, instead of running the test, the assets will be downloaded and saved to the cache, then the timestamp file created. A meson custom target is created as a dependency of each test suite to trigger the pre-emptive caching logic before the test runs. When run in caching mode, it will locate assets by looking for class level variables with a name prefix "ASSET_", and type "Asset". At the ninja level ninja test --suite functional will speculatively download any assets that are not already cached, so it is advisable to set a timeout multiplier. QEMU_TEST_NO_DOWNLOAD=1 ninja test --suite functional will fail the test if a required asset is not already cached ninja precache-functional will download and cache all assets required by the functional tests At the make level, precaching is always done by make check-functional Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Tested-by: Richard Henderson <richard.henderson@linaro.org> [thuth: Remove the duplicated "path = os.path.basename(...)" line] Message-ID: <20240830133841.142644-16-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
* tests/functional: add a module for handling asset download & cachingDaniel P. Berrangé2024-09-041-0/+97
The 'Asset' class is a simple module that declares a downloadable asset that can be cached locally. Downloads are stored in the user's home dir at ~/.cache/qemu/download, using a sha256 sum of the URL. [thuth: Drop sha1 support, use hash on file content for naming instead of URL, add the possibility to specify the cache dir via environment variable] Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20240830133841.142644-15-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>