summary refs log tree commit diff stats
path: root/tests/qemu-iotests/testrunner.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* iotests: drop compat for old version context managerDaniel P. Berrangé2025-09-151-7/+2
| | | | | | | Our minimum python is now 3.9, so back compat with prior python versions is no longer required. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
* iotests: Change imports for Python 3.13John Snow2024-07-121-3/+6
| | | | | | | | | | | | Python 3.13 isn't out yet, but it's in beta and Fedora is ramping up to make it the default system interpreter for Fedora 41. They moved our cheese for where ContextManager lives; add a conditional to locate it while we support both pre-3.9 and 3.13+. Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20240626232230.408004-4-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
* iotests: always use a unique sub-directory per testDaniel P. Berrangé2023-03-221-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | | The current test runner is only safe against parallel execution within a single instance of the 'check' process, and only if -j is given a value greater than 2. This prevents running multiple copies of the 'check' process for different test scenarios. This change switches the output / socket directories to always include the test name, image format and image protocol. This should allow full parallelism of all distinct test scenarios. eg running both qcow2 and raw tests at the same time, or both file and nbd tests at the same time. It would be possible to allow for parallelism of the same test scenario by including the pid, but that would potentially let many directories accumulate over time on failures, so is not done. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Acked-by: Hanna Czenczek <hreitz@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230303160727.3977246-7-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230315174331.2959-24-alex.bennee@linaro.org>
* iotests: connect stdin to /dev/null when running testsDaniel P. Berrangé2023-03-221-20/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently the tests have their stdin inherited from the test harness, meaning they are connected to a TTY. The QEMU processes spawned by certain tests, however, modify TTY settings and if the test exits abnormally the settings might not be restored. The python test harness thus has some logic which will capture the initial TTY settings and restore them once all tests are finished. This does not, however, take into account the possibility of many copies of the 'check' program running in parallel. With parallel execution, a later invokation may save the TTY state that QEMU has already modified, and thus restore bad state leaving the TTY non-functional. None of the I/O tests shnould actually be interactive requiring user input and so they should not require a TTY at all. To avoid this while TTY save/restore complexity we can connect the test stdin to /dev/null instead. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Acked-by: Hanna Czenczek <hreitz@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230303160727.3977246-6-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230315174331.2959-23-alex.bennee@linaro.org>
* iotests: print TAP protocol version when reporting testsDaniel P. Berrangé2023-03-221-0/+1
| | | | | | | | | | | | | | | | | Recently meson started complaining that TAP test reports don't include the TAP protocol version. While this warning is bogus and has since been removed from Meson, it looks like good practice to include this header going forward. The GLib library test harness has started unconditionally printing the version, so this brings the I/O tests into line. Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Acked-by: Hanna Czenczek <hreitz@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230303160727.3977246-5-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230315174331.2959-22-alex.bennee@linaro.org>
* tests/qemu-iotests: print intent to run a test in TAP modeDaniel P. Berrangé2022-05-121-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When running I/O tests using TAP output mode, we get a single TAP test with a sub-test reported for each I/O test that is run. The output looks something like this: 1..123 ok qcow2 011 ok qcow2 012 ok qcow2 013 ok qcow2 217 ... If everything runs or fails normally this is fine, but periodically we have been seeing the test harness abort early before all 123 tests have been run, just leaving a fairly useless message like TAP parsing error: Too few tests run (expected 123, got 107) we have no idea which tests were running at the time the test harness abruptly exited. This change causes us to print a message about our intent to run each test, so we have a record of what is active at the time the harness exits abnormally. 1..123 # running qcow2 011 ok qcow2 011 # running qcow2 012 ok qcow2 012 # running qcow2 013 ok qcow2 013 # running qcow2 217 ok qcow2 217 ... Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20220509124134.867431-2-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* iotests/testrunner: Flush after run_test()Hanna Reitz2022-05-121-0/+1
| | | | | | | | | | | | | | | | | | | | | | | When stdout is not a terminal, the buffer may not be flushed at each end of line, so we should flush after each test is done. This is especially apparent when run by check-block, in two ways: First, when running make check-block -jX with X > 1, progress indication was missing, even though testrunner.py does theoretically print each test's status once it has been run, even in multi-processing mode. Flushing after each test restores this progress indication. Second, sometimes make check-block failed altogether, with an error message that "too few tests [were] run". I presume that's because one worker process in the job pool did not get to flush its stdout before the main process exited, and so meson did not get to see that worker's test results. In any case, by flushing at the end of run_test(), the problem has disappeared for me. Signed-off-by: Hanna Reitz <hreitz@redhat.com> Message-Id: <20220506134215.10086-1-hreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* tests/qemu-iotests/testrunner: Supply a test plan in TAP modeThomas Huth2022-03-221-0/+1
| | | | | | | | | | | Quoting the TAP specification: "The plan tells how many tests will be run [...]. It’s a check that the test file hasn’t stopped prematurely." That's a good idea of course, so let's support that in the iotest testrunner, too. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-Id: <20220223095816.2663005-1-thuth@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
* tests/qemu-iotests: add QEMU_IOTESTS_REGEN=1 to update reference fileDaniel P. Berrangé2022-03-071-0/+6
| | | | | | | | | | | | | | | | | | | When developing an I/O test it is typical to add some logic to the test script, run it to view the output diff, and then apply the output diff to the reference file. This can be drastically simplified by letting the test runner update the reference file in place. By setting 'QEMU_IOTESTS_REGEN=1', the test runner will report the failure and show the diff, but at the same time update the reference file. So next time the I/O test is run it will succeed. Continuing to display the diff when updating the reference gives the developer a chance to review what was changed. Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20220304193610.3293146-7-berrange@redhat.com> Signed-off-by: Eric Blake <eblake@redhat.com>
* iotests: Write test output to TEST_DIRHanna Reitz2022-03-071-6/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Drop the use of OUTPUT_DIR (test/qemu-iotests under the build directory), and instead write test output files (.out.bad, .notrun, and .casenotrun) to TEST_DIR. With this, the same test can be run concurrently without the separate instances interfering, because they will need separate TEST_DIRs anyway. Running the same test separately is useful when running the iotests with various format/protocol combinations in parallel, or when you just want to aggressively exercise a single test (e.g. when it fails only sporadically). Putting this output into TEST_DIR means that it will stick around for inspection after the test run is done (though running the same test in the same TEST_DIR will overwrite it, just as it used to be); but given that TEST_DIR is a scratch directory, it should be clear that users can delete all of its content at any point. (And if TEST_DIR is on tmpfs, it will just disappear on shutdown.) Contrarily, alternative approaches that would put these output files into OUTPUT_DIR with some prefix to differentiate between separate test runs might easily lead to cluttering OUTPUT_DIR. (This change means OUTPUT_DIR is no longer written to by the iotests, so we can drop its usage altogether.) Signed-off-by: Hanna Reitz <hreitz@redhat.com> Message-Id: <20220221172909.762858-1-hreitz@redhat.com> [hreitz: Simplified `Path(os.path.join(x, y))` to `Path(x, y)`, as suggested by Vladimir; and rebased on 9086c7639822b6 ("tests/qemu-iotests: Rework the checks and spots using GNU sed")] Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
* tests/qemu-iotests/testrunner: Quote "case not run" lines in TAP modeThomas Huth2022-03-071-1/+4
| | | | | | | | | | | | | | | | | | | In TAP mode, the stdout is reserved for the TAP protocol, so we have to make sure to mark other lines with a comment '#' character at the beginning to avoid that the TAP parser at the other end gets confused. To test this condition, run "configure" for example with: --block-drv-rw-whitelist=copy-before-write,qcow2,raw,file,host_device,blkdebug,null-co,copy-on-read so that iotest 041 will report that some tests are not run due to the missing "quorum" driver. Without this change, "make check-block" fails since the meson tap parser gets confused by these messages. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-Id: <20220223124353.3273898-1-thuth@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
* tests/qemu-iotests/testrunner: Print diff to stderr in TAP modeThomas Huth2022-02-161-1/+4
| | | | | | | | | | When running in TAP mode, stdout is reserved for the TAP protocol. To see the "diff" of the failed test, we have to print it to stderr instead. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-Id: <20220209101530.3442837-8-thuth@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* check-block: replace -makecheck with TAP outputPaolo Bonzini2022-01-281-23/+26
| | | | | | | Let "meson test" take care of showing the results of the individual tests, consistently with other output from "make check V=1". Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* iotests/testrunner.py: refactor test_field_widthVladimir Sementsov-Ogievskiy2022-01-141-11/+10
| | | | | | | | | | | | | | | | A lot of Optional[] types doesn't make code beautiful. test_field_width defaults to 8, but that is never used in the code. More over, if we want some default behavior for single call of test_run(), it should just print the whole test name, not limiting or expanding its width, so 8 is bad default. So, just drop the default as unused for now. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20211210201450.101576-1-vsementsov@virtuozzo.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* iotests: check: multiprocessing supportVladimir Sementsov-Ogievskiy2021-12-221-8/+61
| | | | | | | | | | | | | | | | | | | | | | | Add -j <JOBS> parameter, to run tests in several jobs simultaneously. For realization - simply utilize multiprocessing.Pool class. Notes: 1. Of course, tests can't run simultaneously in same TEST_DIR. So, use subdirectories TEST_DIR/testname/ and SOCK_DIR/testname/ instead of simply TEST_DIR and SOCK_DIR 2. multiprocessing.Pool.starmap function doesn't support passing context managers, so we can't simply pass "self". Happily, we need self only for read-only access, and it just works if it is defined in global space. So, add a temporary link TestRunner.shared_self during run_tests(). Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20211203122223.2780098-4-vsementsov@virtuozzo.com> Reviewed-by: John Snow <jsnow@redhat.com> Tested-by: John Snow <jsnow@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
* iotests/testrunner.py: move updating last_elapsed to run_testsVladimir Sementsov-Ogievskiy2021-12-221-1/+3
| | | | | | | | | | | | | | | We are going to use do_run_test() in multiprocessing environment, where we'll not be able to change original runner object. Happily, the only thing we change is that last_elapsed and it's simple to do it in run_tests() instead. All other accesses to self in do_runt_test() and in run_test() are read-only. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20211203122223.2780098-3-vsementsov@virtuozzo.com> Reviewed-by: John Snow <jsnow@redhat.com> Tested-by: John Snow <jsnow@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
* iotests/testrunner.py: add doc string for run_test()Vladimir Sementsov-Ogievskiy2021-12-221-0/+13
| | | | | | | | | | | We are going to modify these methods and will add more documentation in further commit. As a preparation add basic documentation. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20211203122223.2780098-2-vsementsov@virtuozzo.com> Reviewed-by: John Snow <jsnow@redhat.com> Tested-by: John Snow <jsnow@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
* qemu-iotests: flush after every testPaolo Bonzini2021-10-131-0/+1
| | | | | | | | | This makes it possible to see what is happening, even if the output of "make check-block" is not sent to a tty (for example if it is sent to grep or tee). Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* iotests: Update for pylint 2.11.1John Snow2021-10-061-3/+4
| | | | | | | | | | | | | | | | | 1. Ignore the new f-strings warning, we're not interested in doing a full conversion at this time. 2. Just mute the unbalanced-tuple-unpacking warning, it's not a real error in this case and muting the dozens of callsites is just not worth it. 3. Add encodings to read_text(). Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Message-Id: <20210923180715.4168522-7-jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* qemu-iotests: fix pylint 2.8 consider-using-with errorEmanuele Giuseppe Esposito2021-05-141-11/+11
| | | | | | | | | | | | | | | | pylint 2.8 introduces consider-using-with error, suggesting to use the 'with' block statement when possible. Modify all subprocess.Popen call to use the 'with' statement, except the one in __init__ of QemuIoInteractive class, since it is assigned to a class field and used in other methods. Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> Message-Id: <20210510190449.65948-1-eesposit@redhat.com> [mreitz: Disable bad-option-value warning in the iotests' pylintrc, so that disabling consider-using-with in QemuIoInteractive will not produce a warning in pre-2.8 pylint versions] Signed-off-by: Max Reitz <mreitz@redhat.com>
* qemu-iotests: let "check" spawn an arbitrary test commandPaolo Bonzini2021-05-141-1/+0
| | | | | | | | | | | | | | | | | Right now there is no easy way for "check" to print a reproducer command. Because such a reproducer command line would be huge, we can instead teach check to start a command of our choice. This can be for example a Python unit test with arguments to only run a specific subtest. Move the trailing empty line to print_env(), since it always looks better and one caller was not adding it. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> Message-Id: <20210323181928.311862-5-pbonzini@redhat.com> Message-Id: <20210503110110.476887-5-pbonzini@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* qemu-iotests: move command line and environment handling from TestRunner to ↵Paolo Bonzini2021-05-141-13/+1
| | | | | | | | | | | | | | | | TestEnv In the next patch, "check" will learn how to execute a test script without going through TestRunner. To enable this, keep only the text output and subprocess handling in the TestRunner; move into TestEnv the logic to prepare for running a subprocess. Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> Message-Id: <20210323181928.311862-4-pbonzini@redhat.com> Message-Id: <20210503110110.476887-4-pbonzini@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
* iotests: Fix -makecheck outputKevin Wolf2021-02-021-2/+4
| | | | | | | | | | | | | | | | | | | | For -makecheck, the old 'check' implementation skipped the output when starting a test. It only had the condensed output at the end of a test. testrunner.py prints the normal output when starting a test even for -makecheck. This output contains '\r' at the end so that it can be overwritten with the result at the end of the test. However, for -makecheck this is shorter output in a different format, so effectively we end up with garbled output that mixes both output forms. Revert to the old behaviour of only printing a message after the test had completed in -makecheck mode. Fixes: d74c754c924ca34e90b7c96ce2f5609d82c0e628 Signed-off-by: Kevin Wolf <kwolf@redhat.com> Message-Id: <20210201161024.127921-1-kwolf@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* iotests: check: return 1 on failureVladimir Sementsov-Ogievskiy2021-02-021-1/+3
| | | | | | | | | | We should indicate failure by exit code, not only output. Reported-by: Peter Maydell Fixes: f203080bbd9f9e5b31041b1f2afcd6040c5aaec5 Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210201085041.3079-1-vsementsov@virtuozzo.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* iotests/testrunner: fix recognition of python testsVladimir Sementsov-Ogievskiy2021-01-291-1/+1
| | | | | | | | | | We should drop final '\n' before comparing with python3 shebang. Fixes: d74c754c924ca34e90b7c96ce2f5609d82c0e628 Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210129090616.84145-1-vsementsov@virtuozzo.com> Tested-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* iotests: add testrunner.pyVladimir Sementsov-Ogievskiy2021-01-271-0/+367
Add TestRunner class, which will run tests in a new python iotests running framework. There are some differences with current ./check behavior, most significant are: - Consider all tests self-executable, just run them, don't run python by hand. - Elapsed time is cached in json file - Elapsed time precision increased a bit - Instead of using "diff -w" which ignores all whitespace differences, manually strip whitespace at line end then use python difflib, which no longer ignores spacing mid-line Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210125185056.129513-5-vsementsov@virtuozzo.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>