assembly: 0.956 device: 0.953 other: 0.953 instruction: 0.951 semantic: 0.939 graphic: 0.935 socket: 0.932 boot: 0.932 vnc: 0.926 mistranslation: 0.888 KVM: 0.855 network: 0.818 [Qemu-devel] [BUG] gcov support appears to be broken Hello, according to out docs, here is the procedure that should produce coverage report for execution of the complete "make check": #./configure --enable-gcov #make #make check #make coverage-report It seems that first three commands execute as expected. (For example, there are plenty of files generated by "make check" that would've not been generated if "enable-gcov" hadn't been chosen.) However, the last command complains about some missing files related to FP support. If those files are added (for example, artificially, using "touch wrote: > > Hello, according to out docs, here is the procedure that should produce > coverage report for execution of the complete "make check": > > #./configure --enable-gcov > #make > #make check > #make coverage-report > > It seems that first three commands execute as expected. (For example, there > are plenty of files generated by "make check" that would've not been > generated if "enable-gcov" hadn't been chosen.) However, the last command > complains about some missing files related to FP support. If those files are > added (for example, artificially, using "touch starts complaining about missing some decodetree-generated files. Other kinds > of files are involved too. > > It would be nice to have coverage support working. Please somebody take a > look, or explain if I make a mistake or misunderstood our gcov support. Cc'ing Alex who's probably the closest we have to a gcov expert. (make/make check of a --enable-gcov build is in the set of things our Travis CI setup runs, so we do defend that part against regressions.) thanks -- PMM Peter Maydell writes: > On Mon, 5 Aug 2019 at 11:39, Aleksandar Markovic wrote: > > > > Hello, according to out docs, here is the procedure that should produce > > coverage report for execution of the complete "make check": > > > > #./configure --enable-gcov > > #make > > #make check > > #make coverage-report > > > > It seems that first three commands execute as expected. (For example, > > there are plenty of files generated by "make check" that would've not > > been generated if "enable-gcov" hadn't been chosen.) However, the > > last command complains about some missing files related to FP > > support. If those files are added (for example, artificially, using > > "touch > decodetree-generated files. Other kinds of files are involved too. The gcov tool is fairly noisy about missing files but that just indicates the tests haven't exercised those code paths. "make check" especially doesn't touch much of the TCG code and a chunk of floating point. > > > > It would be nice to have coverage support working. Please somebody > > take a look, or explain if I make a mistake or misunderstood our gcov > > support. So your failure mode is no report is generated at all? It's working for me here. > > Cc'ing Alex who's probably the closest we have to a gcov expert. > > (make/make check of a --enable-gcov build is in the set of things our > Travis CI setup runs, so we do defend that part against regressions.) We defend the build but I have just checked and it seems our check_coverage script is currently failing: https://travis-ci.org/stsquad/qemu/jobs/567809808#L10328 But as it's an after_success script it doesn't fail the build. > > thanks > -- PMM -- Alex Bennée > > #./configure --enable-gcov > > #make > > #make check > > #make coverage-report > > > > It seems that first three commands execute as expected. (For example, > > there are plenty of files generated by "make check" that would've not > > been generated if "enable-gcov" hadn't been chosen.) However, the > > last command complains about some missing files related to FP > So your failure mode is no report is generated at all? It's working for > me here. Alex, no report is generated for my test setups - in fact, "make coverage-report" even says that it explicitly deletes what appears to be the main coverage report html file). This is the terminal output of an unsuccessful executions of "make coverage-report" for recent ToT: ~/Build/qemu-TOT-TEST$ make coverage-report make[1]: Entering directory '/home/user/Build/qemu-TOT-TEST/slirp' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/home/user/Build/qemu-TOT-TEST/slirp' CHK version_gen.h GEN coverage-report.html Traceback (most recent call last): File "/usr/bin/gcovr", line 1970, in print_html_report(covdata, options.html_details) File "/usr/bin/gcovr", line 1473, in print_html_report INPUT = open(data['FILENAME'], 'r') IOError: [Errno 2] No such file or directory: 'wrap.inc.c' Makefile:1048: recipe for target '/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html' failed make: *** [/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html] Error 1 make: *** Deleting file '/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html' This instance is executed in QEMU 3.0 source tree: (so, it looks the problem existed for quite some time) ~/Build/qemu-3.0$ make coverage-report CHK version_gen.h GEN coverage-report.html Traceback (most recent call last): File "/usr/bin/gcovr", line 1970, in print_html_report(covdata, options.html_details) File "/usr/bin/gcovr", line 1473, in print_html_report INPUT = open(data['FILENAME'], 'r') IOError: [Errno 2] No such file or directory: '/home/user/Build/qemu-3.0/target/openrisc/decode.inc.c' Makefile:992: recipe for target '/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html' failed make: *** [/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html] Error 1 make: *** Deleting file '/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html' Fond regards, Aleksandar > Alex Bennée > > #./configure --enable-gcov > > #make > > #make check > > #make coverage-report > > > > It seems that first three commands execute as expected. (For example, > > there are plenty of files generated by "make check" that would've not > > been generated if "enable-gcov" hadn't been chosen.) However, the > > last command complains about some missing files related to FP > So your failure mode is no report is generated at all? It's working for > me here. Another piece of info: ~/Build/qemu-TOT-TEST$ gcov --version gcov (Ubuntu 5.5.0-12ubuntu1~16.04) 5.5.0 20171010 Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. :~/Build/qemu-TOT-TEST$ gcc --version gcc (Ubuntu 7.2.0-1ubuntu1~16.04) 7.2.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Alex, no report is generated for my test setups - in fact, "make coverage-report" even says that it explicitly deletes what appears to be the main coverage report html file). This is the terminal output of an unsuccessful executions of "make coverage-report" for recent ToT: ~/Build/qemu-TOT-TEST$ make coverage-report make[1]: Entering directory '/home/user/Build/qemu-TOT-TEST/slirp' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/home/user/Build/qemu-TOT-TEST/slirp' CHK version_gen.h GEN coverage-report.html Traceback (most recent call last): File "/usr/bin/gcovr", line 1970, in print_html_report(covdata, options.html_details) File "/usr/bin/gcovr", line 1473, in print_html_report INPUT = open(data['FILENAME'], 'r') IOError: [Errno 2] No such file or directory: 'wrap.inc.c' Makefile:1048: recipe for target '/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html' failed make: *** [/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html] Error 1 make: *** Deleting file '/home/user/Build/qemu-TOT-TEST/reports/coverage/coverage-report.html' This instance is executed in QEMU 3.0 source tree: (so, it looks the problem existed for quite some time) ~/Build/qemu-3.0$ make coverage-report CHK version_gen.h GEN coverage-report.html Traceback (most recent call last): File "/usr/bin/gcovr", line 1970, in print_html_report(covdata, options.html_details) File "/usr/bin/gcovr", line 1473, in print_html_report INPUT = open(data['FILENAME'], 'r') IOError: [Errno 2] No such file or directory: '/home/user/Build/qemu-3.0/target/openrisc/decode.inc.c' Makefile:992: recipe for target '/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html' failed make: *** [/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html] Error 1 make: *** Deleting file '/home/user/Build/qemu-3.0/reports/coverage/coverage-report.html' Fond regards, Aleksandar > Alex Bennée > > #./configure --enable-gcov > > #make > > #make check > > #make coverage-report > > > > It seems that first three commands execute as expected. (For example, > > there are plenty of files generated by "make check" that would've not > > been generated if "enable-gcov" hadn't been chosen.) However, the > > last command complains about some missing files related to FP > So your failure mode is no report is generated at all? It's working for > me here. Alex, here is the thing: Seeing that my gcovr is relatively old (2014) 3.2 version, I upgraded it from git repo to the most recent 4.1 (actually, to a dev version, from the very tip of the tree), and "make coverage-report" started generating coverage reports. It did emit some error messages (totally different than previous), but still it did not stop like it used to do with gcovr 3.2. Perhaps you would want to add some gcov/gcovr minimal version info in our docs. (or at least a statement "this was tested with such and such gcc, gcov and gcovr", etc.?) Coverage report looked fine at first glance, but it a kind of disappointed me when I digged deeper into its content - for example, it shows very low coverage for our FP code (softfloat), while, in fact, we know that "make check" contains detailed tests on FP functionalities. But this is most likely a separate problem of a very different nature, perhaps the issue of separate git repo for FP tests (testfloat) that our FP tests use as a mid-layer. I'll try how everything works with my test examples, and will let you know. Your help is greatly appreciated, Aleksandar Fond regards, Aleksandar > Alex Bennée Aleksandar Markovic writes: > >> #./configure --enable-gcov > >> #make > >> #make check > >> #make coverage-report > >> > >> It seems that first three commands execute as expected. (For example, > >> there are plenty of files generated by "make check" that would've not > >> been generated if "enable-gcov" hadn't been chosen.) However, the > >> last command complains about some missing files related to FP > > > So your failure mode is no report is generated at all? It's working for > > me here. > > Alex, here is the thing: > > Seeing that my gcovr is relatively old (2014) 3.2 version, I upgraded it from > git repo to the most recent 4.1 (actually, to a dev version, from the very > tip of the tree), and "make coverage-report" started generating coverage > reports. It did emit some error messages (totally different than previous), > but still it did not stop like it used to do with gcovr 3.2. > > Perhaps you would want to add some gcov/gcovr minimal version info in our > docs. (or at least a statement "this was tested with such and such gcc, gcov > and gcovr", etc.?) > > Coverage report looked fine at first glance, but it a kind of > disappointed me when I digged deeper into its content - for example, > it shows very low coverage for our FP code (softfloat), while, in > fact, we know that "make check" contains detailed tests on FP > functionalities. But this is most likely a separate problem of a very > different nature, perhaps the issue of separate git repo for FP tests > (testfloat) that our FP tests use as a mid-layer. I get: 68.6 % 2593 / 3782 62.2 % 1690 / 2718 Which is not bad considering we don't exercise the 80 and 128 bit softfloat code at all (which is not shared by the re-factored 16/32/64 bit code). > > I'll try how everything works with my test examples, and will let you know. > > Your help is greatly appreciated, > Aleksandar > > Fond regards, > Aleksandar > > > > Alex Bennée -- Alex Bennée > > it shows very low coverage for our FP code (softfloat), while, in > > fact, we know that "make check" contains detailed tests on FP > > functionalities. But this is most likely a separate problem of a very > > different nature, perhaps the issue of separate git repo for FP tests > > (testfloat) that our FP tests use as a mid-layer. > > I get: > > 68.6 % 2593 / 3782 62.2 % 1690 / 2718 > I would expect that kind of result too. However, I get: File: fpu/softfloat.c Lines: 8 3334 0.2 % Date: 2019-08-05 19:56:58 Branches: 3 2376 0.1 % :( OK, I'll try to figure that out, and most likely I could live with it if it is an isolated problem. Thank you for your assistance in this matter, Aleksandar > Which is not bad considering we don't exercise the 80 and 128 bit > softfloat code at all (which is not shared by the re-factored 16/32/64 > bit code). > > Alex Bennée > > it shows very low coverage for our FP code (softfloat), while, in > > fact, we know that "make check" contains detailed tests on FP > > functionalities. But this is most likely a separate problem of a very > > different nature, perhaps the issue of separate git repo for FP tests > > (testfloat) that our FP tests use as a mid-layer. > > I get: > > 68.6 % 2593 / 3782 62.2 % 1690 / 2718 > This problem is solved too. (and it is my fault) I worked with multiple versions of QEMU, and my previous low-coverage results were for QEMU 3.0, and for that version the directory tests/fp did not even exist. :D () For QEMU ToT, I get now: fpu/softfloat.c 68.8 % 2592 / 3770 62.3 % 1693 / 2718 which is identical for all intents and purposes to your result. Yours cordially, Aleksandar