tracetool: add test suite for tracetool with reference output

When reviewing tracetool patches it is often very unclear what the
expected output will be for the generated backends. Compounding
this is that a default build will only enable the 'log' trace
backend, so developers won't see generated code for other backends
without making a special effort. Some backends are also platform
specific, so can't be enabled in QEMU builds, even though tracetool
could generate the code.

To address this, introduce a test suite for tracetool which is
conceptually similar to the qapi-schema test. It is a simple
python program that runs tracetool and compares the actual output
to historical reference output kept in git. The test directly
emits TAP format logs for ease of integration with meson.

This can be run with

  make check-tracetool

to make it easier for developers changing generated output, the
sample expected content can be auto-recreated

  QEMU_TEST_REGENERATE=1 make check-tracetool

and the changes reviewed and added to the commit. This will also
assist reviewers interpreting the change.

Developers are reminded of this in the test output on failure:

  $ make check-tracetool
  1/6 qemu:tracetool / dtrace        OK              0.14s
  2/6 qemu:tracetool / ftrace        FAIL            0.06s   exit status 1
  ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
  1..2
  ok 1 - ftrace.c
  #
  not ok 1 - ftrace.h (set QEMU_TEST_REGENERATE=1 to recreate reference output if tracetool generator was intentionally changed)
  ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

  3/6 qemu:tracetool / log           OK              0.06s
  4/6 qemu:tracetool / simple        OK              0.06s
  5/6 qemu:tracetool / syslog        OK              0.06s
  6/6 qemu:tracetool / ust           OK              0.11s

  Summary of Failures:

  2/6 qemu:tracetool / ftrace FAIL            0.06s   exit status 1

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20250916081638.764020-6-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

1 files changed, 28 insertions, 0 deletions

diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index 2b5cb0c148..11f05c0006 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -178,6 +178,34 @@ parser (either fixing a bug or extending/modifying the syntax). To do this:
 
   ``qapi-schema += foo.json``
 
+.. _tracetool-tests:
+
+Tracetool tests
+~~~~~~~~~~~~~~~
+
+The tracetool tests validate the generated source files used for defining
+probes for various tracing backends and source formats. The test operates
+by running the tracetool program against a sample trace-events file, and
+comparing the generated output against known good reference output. The
+tests can be run with:
+
+.. code::
+
+  make check-tracetool
+
+The reference output is stored in files under tests/tracetool, and when
+the tracetool backend/format output is intentionally changed, the reference
+files need to be updated. This can be automated by setting the
+QEMU_TEST_REGENERATE=1 environment variable:
+
+.. code::
+
+   QEMU_TEST_REGENERATE=1 make check-tracetool
+
+The resulting changes must be reviewed by the author to ensure they match
+the intended results, before adding the updated reference output to the
+same commit that alters the generator code.
+
 check-block
 ~~~~~~~~~~~