diff options
Diffstat (limited to '')
| -rw-r--r-- | results/classifier/108/other/241 | 16 | ||||
| -rw-r--r-- | results/classifier/108/other/241119 | 108 | ||||
| -rw-r--r-- | results/classifier/108/other/2412 | 115 | ||||
| -rw-r--r-- | results/classifier/108/other/2413 | 48 | ||||
| -rw-r--r-- | results/classifier/108/other/2414 | 132 | ||||
| -rw-r--r-- | results/classifier/108/other/2415 | 68 | ||||
| -rw-r--r-- | results/classifier/108/other/2416 | 54 | ||||
| -rw-r--r-- | results/classifier/108/other/2417 | 20 | ||||
| -rw-r--r-- | results/classifier/108/other/2419 | 33 | ||||
| -rw-r--r-- | results/classifier/108/other/24190340 | 2066 |
10 files changed, 2660 insertions, 0 deletions
diff --git a/results/classifier/108/other/241 b/results/classifier/108/other/241 new file mode 100644 index 000000000..1e5f7fd96 --- /dev/null +++ b/results/classifier/108/other/241 @@ -0,0 +1,16 @@ +performance: 0.889 +semantic: 0.465 +graphic: 0.435 +device: 0.403 +files: 0.156 +other: 0.063 +debug: 0.045 +boot: 0.039 +vnc: 0.038 +PID: 0.023 +network: 0.019 +socket: 0.013 +KVM: 0.010 +permissions: 0.007 + +Please refactor linux-user/mips/cpu_loop.c diff --git a/results/classifier/108/other/241119 b/results/classifier/108/other/241119 new file mode 100644 index 000000000..8a6a78f0b --- /dev/null +++ b/results/classifier/108/other/241119 @@ -0,0 +1,108 @@ +graphic: 0.859 +semantic: 0.823 +permissions: 0.804 +device: 0.788 +other: 0.745 +debug: 0.743 +performance: 0.726 +PID: 0.701 +KVM: 0.681 +socket: 0.675 +network: 0.662 +files: 0.635 +boot: 0.628 +vnc: 0.556 + +usb_add of a Creative ZEN unrecognized in guest + +Binary package hint: kvm + +This happens when I add my Creative ZEN to a virtual machine running XP. The device is recognised well at first and drivers are installed correctly. But when trying to connect windows crashes with the classic blue screen It complains about something like usbohci.sys, I can't read well because it crashes too fast. +I have also tried with another virtual machine running Vista, same results. +Any help would be really appreciated! + +I'm using the module kvm-amd with Ubuntu 8.04 + +The USB device has the following ID: 041e:4157 Creative Technology, Ltd + +kvm: + Installed: 1:62+dfsg-0ubuntu7 + Candidate: 1:62+dfsg-0ubuntu7 + Version table: + *** 1:62+dfsg-0ubuntu7 0 + 500 http://archive.ubuntu.com hardy/main Packages + 100 /var/lib/dpkg/status + +Hi, thanks for the bug report. + +This bug was reported against Hardy's kvm-62. Unfortunately, kvm-62 is not able to do usb-passthrough properly. + +Would it be possible for you to test this against intrepid and/or jaunty? + +:-Dustin + +If you're running Hardy, could you please test this using the kvm-84 package in this PPA: + * https://launchpad.net/~ubuntu-virt/+archive/ppa + +Does this solve the issue, or is it reproducible? + +:-Dustin + + +I'm running Intrepid now, however I tried the package you suggested (kvm-84) but the problem persists. + +kvm: + Installed: 1:84+dfsg-0ubuntu8~ppa6 + Candidate: 1:84+dfsg-0ubuntu8~ppa6 + Version table: + *** 1:84+dfsg-0ubuntu8~ppa6 0 + 100 /var/lib/dpkg/status + 1:72+dfsg-1ubuntu6 0 + 500 http://archive.ubuntu.com intrepid/main Packages + +The console output when trying to usb_add the device is the following: + +husb: open device 1.5 +husb: config #1 need -1 +husb: 1 interfaces claimed for configuration 1 +husb: grabbed usb device 1.5 +usb_linux_update_endp_table: Protocol error + + +I forgot: + +Using the new kvm package the virtual machine now doesn't crash with the blue screen anymore. Now it seems like the device is not correctly recognized or initialized + +Okay, the problem persists in kvm-84. I'll mark it as 'Confirmed', and point upstream at the issue. Thanks. + +:-Dustin + +@Dustin +Did this hit upstream, I looked yesterday and couldn't find it to track it. + +To copy this upstream, click "Also affects project", and enter "qemu". + +kvm-84 is very old. Please try to reproduce this against the latest qemu git or at least 0.12.4. + +Hi. I have a similar problem, with a simple JetFlash usb drive. + +After adding it with usb_add, "info usb" shows nothing and I start getting the following message in the console: +husb: config #1 need -1 +husb: 1 interfaces claimed for configuration 1 +husb: grabbed usb device 1.3 +usb_linux_update_endp_table: Protocol error + +This is under Maverick using qemu-kvm 0.12.5+noroms-0ubuntu7 + +The same operation works fine in the same pc under my gentoo system, currently running qemu-kvm-0.13.0, but it has been working now for a year, without ever giving me problems. + +Hi, + +is this still an issue under natty or oneiric? + +Hi, unfortunately my Creative ZEN (for which I initially issued the bug) broke, so I am not able to verify if it is an issue anymore, I'm sorry. + +[Expired for qemu-kvm (Ubuntu) because there has been no activity for 60 days.] + +Closing this bug since the hardware is not available anymore (according to comment #11). + diff --git a/results/classifier/108/other/2412 b/results/classifier/108/other/2412 new file mode 100644 index 000000000..a589152ee --- /dev/null +++ b/results/classifier/108/other/2412 @@ -0,0 +1,115 @@ +permissions: 0.870 +KVM: 0.862 +other: 0.846 +device: 0.836 +semantic: 0.818 +performance: 0.810 +vnc: 0.789 +debug: 0.777 +graphic: 0.775 +PID: 0.761 +boot: 0.751 +network: 0.731 +files: 0.723 +socket: 0.716 + +Race condition in megasas device +Description of problem: +Race condition DoS in megasas device was found during **fuzzing**. I'm not sure about **worst case impact**, but for now I can make a suggestion: worst case might be leading to **DoS**, but probably it's a rabbit hole. So if we dig deeper we might find something like CWE-200 or CWE-202 (Exposure of Sensitive Information to an Unauthorized Actor and so on). Also, I think that we should analyse thread usage in this case and make all operations thread-safe, but it's not my business of course. As a consequence, I do not suggest any patch (at least for now). +Steps to reproduce: +This command: + +`cat << EOF | ./build/qemu-system-x86_64 \`\ +`-display none -machine accel=qtest, -m 512M -machine q35 -nodefaults \`\ +`-device megasas -device scsi-cd,drive=null0 -blockdev \`\ +`driver=null-co,read-zeroes=on,node-name=null0 -qtest stdio`\ +`outl 0xcf8 0x80000818`\ +`outl 0xcfc 0xc000`\ +`outl 0xcf8 0x80000804`\ +`outw 0xcfc 0x05`\ +`write 0x20 0x1 0x03`\ +`write 0x26 0x1 0x08`\ +`write 0x27 0x1 0x01`\ +`write 0x30 0x1 0x02`\ +`write 0x40 0x1 0x08`\ +`write 0x57 0x1 0x01`\ +`write 0x5a 0x1 0x08`\ +`outl 0xc03d 0x20000000`\ +`outl 0xc03d 0x00`\ +`EOF`\ +\ +Results in:\ +\ +`[R +0.081916] outl 0xcf8 0x80000818`\ +`[S +0.081986] OK`\ +`OK`\ +`[R +0.082033] outl 0xcfc 0xc000`\ +`[S +0.082083] OK`\ +`OK`\ +`[R +0.082102] outl 0xcf8 0x80000804`\ +`[S +0.082117] OK`\ +`OK`\ +`[R +0.082133] outw 0xcfc 0x05`\ +`[S +0.082926] OK`\ +`OK`\ +`[R +0.082961] write 0x20 0x1 0x03`\ +`[S +0.083688] OK`\ +`OK`\ +`[R +0.083731] write 0x26 0x1 0x08`\ +`[S +0.083754] OK`\ +`OK`\ +`[R +0.083780] write 0x27 0x1 0x01`\ +`[S +0.083799] OK`\ +`OK`\ +`[R +0.083817] write 0x30 0x1 0x02`\ +`[S +0.083850] OK`\ +`OK`\ +`[R +0.083872] write 0x40 0x1 0x08`\ +`[S +0.083903] OK`\ +`OK`\ +`[R +0.083925] write 0x57 0x1 0x01`\ +`[S +0.083947] OK`\ +`OK`\ +`[R +0.083962] write 0x5a 0x1 0x08`\ +`[S +0.083985] OK`\ +`OK`\ +`[R +0.084000] outl 0xc03d 0x20000000`\ +`[S +0.085531] OK`\ +`OK`\ +`[R +0.085570] outl 0xc03d 0x00`\ +`[S +0.085673] OK`\ +`OK`\ +`qemu/include/exec/memory.h:1152:12: runtime error: member access within null pointer of type 'AddressSpace' (aka 'struct AddressSpace')`\ +`SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior qemu/include/exec/memory.h:1152:12 in` \ +`AddressSanitizer:DEADLYSIGNAL`\ +`=================================================================`\ +`==168244==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000020 (pc 0x56259b9829ac bp 0x000000000001 sp 0x7ffe62140220 T0)`\ +`==168244==The signal is caused by a READ memory access.`\ +`==168244==Hint: address points to the zero page.`\ + `#0 0x56259b9829ac in address_space_to_flatview qemu/include/exec/memory.h:1152:12`\ + `#1 0x56259b9829ac in address_space_write qemu/build/../system/physmem.c:2929:14`\ + `#2 0x56259b98665e in address_space_unmap qemu/build/../system/physmem.c:3272:9`\ + `#3 0x56259af31dce in dma_memory_unmap qemu/include/sysemu/dma.h:236:5`\ + `#4 0x56259af31dce in dma_blk_unmap qemu/build/../system/dma-helpers.c:93:9`\ + `#5 0x56259af2f220 in dma_complete qemu/build/../system/dma-helpers.c:105:5`\ + `#6 0x56259af2f220 in dma_blk_cb qemu/build/../system/dma-helpers.c:129:9`\ + `#7 0x56259bce7041 in blk_aio_complete qemu/build/../block/block-backend.c:1555:9`\ + `#8 0x56259c224495 in aio_bh_call qemu/build/../util/async.c:171:5`\ + `#9 0x56259c224ca6 in aio_bh_poll qemu/build/../util/async.c:218:13`\ + `#10 0x56259c1b9b89 in aio_dispatch qemu/build/../util/aio-posix.c:423:5`\ + `#11 0x56259c228f40 in aio_ctx_dispatch qemu/build/../util/async.c:360:5`\ + `#12 0x7f2b8c0a07a8 in g_main_context_dispatch (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x547a8) (BuildId: 9f90bd7bbfcf84a1f1c5a6102f70e6264837b9d4)`\ + `#13 0x56259c22a1ed in glib_pollfds_poll qemu/build/../util/main-loop.c:287:9`\ + `#14 0x56259c22a1ed in os_host_main_loop_wait qemu/build/../util/main-loop.c:310:5`\ + `#15 0x56259c22a1ed in main_loop_wait qemu/build/../util/main-loop.c:589:11`\ + `#16 0x56259af5159e in qemu_main_loop qemu/build/../system/runstate.c:796:9`\ + `#17 0x56259baefdb4 in qemu_default_main qemu/build/../system/main.c:37:14`\ + `#18 0x7f2b8aff7249 (/lib/x86_64-linux-gnu/libc.so.6+0x27249) (BuildId: 82ce4e6e4ef08fa58a3535f7437bd3e592db5ac0)`\ + `#19 0x7f2b8aff7304 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x27304) (BuildId: 82ce4e6e4ef08fa58a3535f7437bd3e592db5ac0)`\ + `#20 0x562599f60b70 in _start (qemu/build/qemu-system-x86_64+0x20feb70) (BuildId: 48f1333e9a9d60383d8c9e0db5f690e7c26e1bb2)`\ +`AddressSanitizer can not provide additional info.`\ +`SUMMARY: AddressSanitizer: SEGV qemu/include/exec/memory.h:1152:12 in address_space_to_flatview`\ +`==168244==ABORTING` + +\ +But, if we manually put all of those qtest commands and wait for each command to complete, QEMU doesn't fail. It's because of possible race condition - while QEMU still mapping memory, we already starting to unmap it. It results in this crash. diff --git a/results/classifier/108/other/2413 b/results/classifier/108/other/2413 new file mode 100644 index 000000000..831ef5f14 --- /dev/null +++ b/results/classifier/108/other/2413 @@ -0,0 +1,48 @@ +performance: 0.906 +graphic: 0.903 +files: 0.865 +vnc: 0.832 +device: 0.805 +debug: 0.804 +PID: 0.724 +socket: 0.707 +KVM: 0.705 +other: 0.689 +network: 0.655 +boot: 0.549 +semantic: 0.539 +permissions: 0.485 + +Use of TSTEQ/TSTNE has regressed the cdrom test when running a 32 bit build of qemu-system-x86_64 using TCG +Description of problem: +The test freezes, eventually timing out. The bisect was confused by other SEV related things so I had to whittle down the config to --disable-kvm. +Steps to reproduce: +1. '../../configure' '--disable-docs' '--disable-user' '--cross-prefix=i686-linux-gnu-' '--target-list=x86_64-softmmu' '--enable-debug' '--disable-kvm' +2. ninja +3. meson test -t 0.05 qtest-x86_64/cdrom-test V=1 +Additional information: +Bisect run pointed at: + +``` +commit 15957eb9efe2da67c796612cead95cba28ba9bda +Author: Paolo Bonzini <pbonzini@redhat.com> +Date: Fri Oct 27 05:57:31 2023 +0200 + + target/i386: use TSTEQ/TSTNE to test low bits + + When testing the sign bit or equality to zero of a partial register, it + is useful to use a single TSTEQ or TSTNE operation. It can also be used + to test the parity flag, using bit 0 of the population count. + + Do not do this for target_ulong-sized values however; the optimizer would + produce a comparison against zero anyway, and it avoids shifts by 64 + which are undefined behavior. + + Reviewed-by: Richard Henderson <richard.henderson@linaro.org> + Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> + + target/i386/tcg/translate.c | 28 ++++++++++++++++++++-------- + target/i386/tcg/emit.c.inc | 5 ++--- + 2 files changed, 22 insertions(+), 11 deletions(-) +bisect found first bad commit⏎ +``` diff --git a/results/classifier/108/other/2414 b/results/classifier/108/other/2414 new file mode 100644 index 000000000..f278b4bad --- /dev/null +++ b/results/classifier/108/other/2414 @@ -0,0 +1,132 @@ +other: 0.919 +graphic: 0.913 +semantic: 0.873 +permissions: 0.830 +socket: 0.825 +debug: 0.804 +KVM: 0.803 +vnc: 0.800 +network: 0.758 +device: 0.754 +PID: 0.725 +performance: 0.685 +boot: 0.642 +files: 0.613 + +qemu 9.0.0 crashing with OpenBSD 7.5 +Description of problem: +After upgrading from Qemu 8.23 to 9.0 this virtual does not start anymore (others do). The bootloader runs fine and starts the OpenBSD kernel, some kernel messages are shown on VGA console. It never reaches userland. +Additional information: +``` +Jun 29 07:15:10 hypervisor kernel: qemu-system-x86[12021]: segfault at 14 ip 000056547310bee4 sp 00007fc6d68c8310 error 4 in qemu-system-x86_64[565472ee0000+6ea000] +Jun 29 07:15:10 hypervisor kernel: Code: 01 00 00 48 83 c4 58 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 89 f0 48 8b 8b 40 83 00 00 4c 8d 0c 40 49 c1 e1 03 4c 01 c9 <8b> 41 14 85 c0 0f 84 11 01 00 00 83 c0 01 89 41 14 41 80 bf d1 01 +Jun 29 07:15:10 hypervisor systemd[1]: Started Process Core Dump (PID 12122/UID 0). +Jun 29 07:15:39 hypervisor systemd-coredump[12123]: Process 12017 (qemu-system-x86) of user 954 dumped core. + + Stack trace of thread 12021: + #0 0x000056547310bee4 n/a (qemu-system-x86_64 + 0x397ee4) + #1 0x000056547330d5e2 n/a (qemu-system-x86_64 + 0x5995e2) + #2 0x000056547330dba6 n/a (qemu-system-x86_64 + 0x599ba6) + #3 0x000056547330e059 memory_region_dispatch_write (qemu-system-x86_64 + 0x59a059) + #4 0x00005654735c1e1f n/a (qemu-system-x86_64 + 0x84de1f) + #5 0x0000565473314a7d n/a (qemu-system-x86_64 + 0x5a0a7d) + #6 0x0000565473314b76 address_space_write (qemu-system-x86_64 + 0x5a0b76) + #7 0x000056547336cafe kvm_cpu_exec (qemu-system-x86_64 + 0x5f8afe) + #8 0x000056547336f56e n/a (qemu-system-x86_64 + 0x5fb56e) + #9 0x000056547352fca8 n/a (qemu-system-x86_64 + 0x7bbca8) + #10 0x00007fc6d93b6ded n/a (libc.so.6 + 0x92ded) + #11 0x00007fc6d943a0dc n/a (libc.so.6 + 0x1160dc) + + Stack trace of thread 12026: + #0 0x00007fc6d93b3740 n/a (libc.so.6 + 0x8f740) + #1 0x00007fc6d93ba551 pthread_mutex_lock (libc.so.6 + 0x96551) + #2 0x0000565473535858 qemu_mutex_lock_impl (qemu-system-x86_64 + 0x7c1858) + #3 0x000056547313f906 bql_lock_impl (qemu-system-x86_64 + 0x3cb906) + #4 0x00005654735c1c7f n/a (qemu-system-x86_64 + 0x84dc7f) + #5 0x0000565473313776 flatview_read_continue (qemu-system-x86_64 + 0x59f776) + #6 0x0000565473314df0 n/a (qemu-system-x86_64 + 0x5a0df0) + #7 0x0000565473314eb6 address_space_read_full (qemu-system-x86_64 + 0x5a0eb6) + #8 0x000056547336cdf5 kvm_cpu_exec (qemu-system-x86_64 + 0x5f8df5) + #9 0x000056547336f56e n/a (qemu-system-x86_64 + 0x5fb56e) + #10 0x000056547352fca8 n/a (qemu-system-x86_64 + 0x7bbca8) + #11 0x00007fc6d93b6ded n/a (libc.so.6 + 0x92ded) + #12 0x00007fc6d943a0dc n/a (libc.so.6 + 0x1160dc) + + Stack trace of thread 12018: + #0 0x00007fc6d9402f43 clock_nanosleep (libc.so.6 + 0xdef43) + #1 0x00007fc6d940ed77 __nanosleep (libc.so.6 + 0xead77) + #2 0x00007fc6d98ccee0 g_usleep (libglib-2.0.so.0 + 0x8dee0) + #3 0x0000565473545a75 n/a (qemu-system-x86_64 + 0x7d1a75) + #4 0x000056547352fca8 n/a (qemu-system-x86_64 + 0x7bbca8) + #5 0x00007fc6d93b6ded n/a (libc.so.6 + 0x92ded) + #6 0x00007fc6d943a0dc n/a (libc.so.6 + 0x1160dc) + + Stack trace of thread 12020: + #0 0x00007fc6d942c39d __poll (libc.so.6 + 0x10839d) + #1 0x00007fc6d98fd8fd n/a (libglib-2.0.so.0 + 0xbe8fd) + #2 0x00007fc6d989c787 g_main_loop_run (libglib-2.0.so.0 + 0x5d787) + #3 0x00005654733bf7c2 n/a (qemu-system-x86_64 + 0x64b7c2) + #4 0x000056547352fca8 n/a (qemu-system-x86_64 + 0x7bbca8) + #5 0x00007fc6d93b6ded n/a (libc.so.6 + 0x92ded) + #6 0x00007fc6d943a0dc n/a (libc.so.6 + 0x1160dc) + + Stack trace of thread 12017: + #0 0x00007fc6d942c910 ppoll (libc.so.6 + 0x108910) + #1 0x000056547354ae83 qemu_poll_ns (qemu-system-x86_64 + 0x7d6e83) + #2 0x000056547355800e main_loop_wait (qemu-system-x86_64 + 0x7e400e) + #3 0x000056547337a337 qemu_default_main (qemu-system-x86_64 + 0x606337) + #4 0x00007fc6d9349c88 n/a (libc.so.6 + 0x25c88) + #5 0x00007fc6d9349d4c __libc_start_main (libc.so.6 + 0x25d4c) + #6 0x0000565472ef08b5 _start (qemu-system-x86_64 + 0x17c8b5) + + Stack trace of thread 12025: + #0 0x00007fc6d942c39d __poll (libc.so.6 + 0x10839d) + #1 0x00007fc6d98fd8fd n/a (libglib-2.0.so.0 + 0xbe8fd) + #2 0x00007fc6d989c787 g_main_loop_run (libglib-2.0.so.0 + 0x5d787) + #3 0x00007fc6d78ff0cb n/a (libspice-server.so.1 + 0x530cb) + #4 0x00007fc6d93b6ded n/a (libc.so.6 + 0x92ded) + #5 0x00007fc6d943a0dc n/a (libc.so.6 + 0x1160dc) + + Stack trace of thread 12117: + #0 0x00007fc6d93b34e9 n/a (libc.so.6 + 0x8f4e9) + #1 0x00007fc6d93b6242 pthread_cond_timedwait (libc.so.6 + 0x92242) + #2 0x0000565473536546 n/a (qemu-system-x86_64 + 0x7c2546) + #3 0x00005654735367ad qemu_cond_timedwait_impl (qemu-system-x86_64 + 0x7c27ad) + #4 0x00005654735569d5 n/a (qemu-system-x86_64 + 0x7e29d5) + #5 0x000056547352fca8 n/a (qemu-system-x86_64 + 0x7bbca8) + #6 0x00007fc6d93b6ded n/a (libc.so.6 + 0x92ded) + #7 0x00007fc6d943a0dc n/a (libc.so.6 + 0x1160dc) + + Stack trace of thread 12028: + #0 0x00007fc6d93b3740 n/a (libc.so.6 + 0x8f740) + #1 0x00007fc6d93ba551 pthread_mutex_lock (libc.so.6 + 0x96551) + #2 0x0000565473535858 qemu_mutex_lock_impl (qemu-system-x86_64 + 0x7c1858) + #3 0x000056547313f906 bql_lock_impl (qemu-system-x86_64 + 0x3cb906) + #4 0x00005654735c1c7f n/a (qemu-system-x86_64 + 0x84dc7f) + #5 0x0000565473313776 flatview_read_continue (qemu-system-x86_64 + 0x59f776) + #6 0x0000565473314df0 n/a (qemu-system-x86_64 + 0x5a0df0) + #7 0x0000565473314eb6 address_space_read_full (qemu-system-x86_64 + 0x5a0eb6) + #8 0x000056547336cdf5 kvm_cpu_exec (qemu-system-x86_64 + 0x5f8df5) + #9 0x000056547336f56e n/a (qemu-system-x86_64 + 0x5fb56e) + #10 0x000056547352fca8 n/a (qemu-system-x86_64 + 0x7bbca8) + #11 0x00007fc6d93b6ded n/a (libc.so.6 + 0x92ded) + #12 0x00007fc6d943a0dc n/a (libc.so.6 + 0x1160dc) + + Stack trace of thread 12027: + #0 0x00007fc6d93b3740 n/a (libc.so.6 + 0x8f740) + #1 0x00007fc6d93ba551 pthread_mutex_lock (libc.so.6 + 0x96551) + #2 0x0000565473535858 qemu_mutex_lock_impl (qemu-system-x86_64 + 0x7c1858) + #3 0x000056547313f906 bql_lock_impl (qemu-system-x86_64 + 0x3cb906) + #4 0x00005654735c1c7f n/a (qemu-system-x86_64 + 0x84dc7f) + #5 0x0000565473313776 flatview_read_continue (qemu-system-x86_64 + 0x59f776) + #6 0x0000565473314df0 n/a (qemu-system-x86_64 + 0x5a0df0) + #7 0x0000565473314eb6 address_space_read_full (qemu-system-x86_64 + 0x5a0eb6) + #8 0x000056547336cdf5 kvm_cpu_exec (qemu-system-x86_64 + 0x5f8df5) + #9 0x000056547336f56e n/a (qemu-system-x86_64 + 0x5fb56e) + #10 0x000056547352fca8 n/a (qemu-system-x86_64 + 0x7bbca8) + #11 0x00007fc6d93b6ded n/a (libc.so.6 + 0x92ded) + #12 0x00007fc6d943a0dc n/a (libc.so.6 + 0x1160dc) + ELF object binary architecture: AMD x86-64 +Jun 29 07:15:40 hypervisor systemd[1]: systemd-coredump@2-12122-0.service: Deactivated successfully. +Jun 29 07:15:40 hypervisor systemd[1]: systemd-coredump@2-12122-0.service: Consumed 20.231s CPU time. +``` diff --git a/results/classifier/108/other/2415 b/results/classifier/108/other/2415 new file mode 100644 index 000000000..7877e8eb6 --- /dev/null +++ b/results/classifier/108/other/2415 @@ -0,0 +1,68 @@ +device: 0.898 +other: 0.891 +graphic: 0.871 +network: 0.867 +debug: 0.843 +KVM: 0.824 +vnc: 0.822 +semantic: 0.810 +permissions: 0.796 +performance: 0.792 +PID: 0.792 +files: 0.781 +boot: 0.772 +socket: 0.745 + +Assertion `r->req.aiocb == NULL' in am53c974 device +Description of problem: +The following log reveals it: + +``` +qemu-truman-x86_64-4467afcc: qemu/hw/scsi/scsi-disk.c:558: void scsi_write_data(SCSIRequest *): Assertion `r->req.aiocb == NULL' failed. +==2957464== ERROR: libFuzzer: deadly signal + #0 0x55e76f00e911 in __sanitizer_print_stack_trace llvm/compiler-rt/lib/asan/asan_stack.cpp:87:3 + #1 0x55e76ef88fb8 in fuzzer::PrintStackTrace() llvm/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5 + #2 0x55e76ef6d1b3 in fuzzer::Fuzzer::CrashCallback() llvm/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:233:3 + #3 0x7f83d604251f (/lib/x86_64-linux-gnu/libc.so.6+0x4251f) + #4 0x7f83d60969fb in __pthread_kill_implementation nptl/./nptl/pthread_kill.c:43:17 + #5 0x7f83d60969fb in __pthread_kill_internal nptl/./nptl/pthread_kill.c:78:10 + #6 0x7f83d60969fb in pthread_kill nptl/./nptl/pthread_kill.c:89:10 + #7 0x7f83d6042475 in gsignal signal/../sysdeps/posix/raise.c:26:13 + #8 0x7f83d60287f2 in abort stdlib/./stdlib/abort.c:79:7 + #9 0x7f83d602871a in __assert_fail_base assert/./assert/assert.c:92:3 + #10 0x7f83d6039e95 in __assert_fail assert/./assert/assert.c:101:3 + #11 0x55e76fbb55a5 in scsi_write_data qemu/hw/scsi/scsi-disk.c:558:5 + #12 0x55e76fb95a1f in scsi_req_continue qemu/hw/scsi/scsi-bus.c + #13 0x55e76fbfe0cc in esp_do_dma qemu/hw/scsi/esp.c + #14 0x55e76fc0be39 in handle_ti qemu/hw/scsi/esp.c:1104:9 + #15 0x55e76fc042f6 in esp_run_cmd qemu/hw/scsi/esp.c:1186:9 + #16 0x55e76fc042f6 in esp_reg_write qemu/hw/scsi/esp.c:1304:9 + #17 0x55e76fc1329b in esp_pci_io_write qemu/hw/scsi/esp-pci.c:248:9 +``` +Steps to reproduce: +``` +cat << EOF | qemu-system-x86_64 -display none\ +-machine accel=qtest, -m 512M -device am53c974,id=scsi -device \ +scsi-hd,drive=disk0 -drive id=disk0,if=none,file=null-co://,format=raw \ +-nodefaults -qtest stdio +outl 0xcf8 0x80001010 +outl 0xcfc 0xc000 +outl 0xcf8 0x80001004 +outw 0xcfc 0x05 +outl 0xc03e 0x030000 +outl 0xc009 0xc1000000 +outl 0xc008 0x8a +outl 0xc00d 0x0 +outl 0xc009 0x00 +outl 0xc00c 0x11 +outl 0xc00d 0x0 +outl 0xc00d 0x00 +outl 0xc00d 0x0 +outw 0xc00f 0x00 +outb 0xc00d 0x0 +outl 0xc00d 0x0 +outl 0xc009 0x41000000 +outb 0xc00c 0x90 +outl 0xc00d 0x0 +EOF +``` diff --git a/results/classifier/108/other/2416 b/results/classifier/108/other/2416 new file mode 100644 index 000000000..cf49f3f1f --- /dev/null +++ b/results/classifier/108/other/2416 @@ -0,0 +1,54 @@ +device: 0.875 +graphic: 0.586 +PID: 0.539 +debug: 0.473 +socket: 0.438 +performance: 0.408 +permissions: 0.388 +vnc: 0.383 +network: 0.358 +files: 0.321 +semantic: 0.293 +boot: 0.179 +KVM: 0.108 +other: 0.072 + +Assertion failure in virtio_snd_get_qemu_format() +Description of problem: +The following log reveals it: + +``` +ERROR:hw/audio/virtio-snd.c:356:virtio_snd_get_qemu_format: code should not be reached +Bail out! ERROR:hw/audio/virtio-snd.c:356:virtio_snd_get_qemu_format: code should not be reached +Aborted +``` +Steps to reproduce: +``` +cat << EOF | qemu-system-x86_64 -display none \ +-machine accel=qtest, -m 512M -machine q35 -device \ +virtio-sound,audiodev=my_audiodev,streams=2 -audiodev \ +alsa,id=my_audiodev -qtest stdio +outl 0xcf8 0x80001804 +outw 0xcfc 0x06 +outl 0xcf8 0x80001820 +outl 0xcfc 0xe0008000 +write 0xe0008020 0x4 0x00001000 +write 0xe0008028 0x4 0x00101000 +write 0xe000801c 0x1 0x01 +write 0x10c000 0x1 0x01 +write 0x10c001 0x1 0x01 +write 0x10c014 0x1 0x01 +write 0x10c015 0x1 0x51 +write 0x100001 0x1 0xc0 +write 0x100002 0x1 0x10 +write 0x100008 0x1 0x18 +write 0x10f000 0x1 0x02 +write 0x10f001 0x1 0x01 +write 0x100021 0x1 0xf0 +write 0x100022 0x1 0x10 +write 0x100028 0x1 0x08 +write 0x101006 0x1 0x02 +write 0x101002 0x1 0x02 +write 0xe000b001 0x1 0x00 +EOF +``` diff --git a/results/classifier/108/other/2417 b/results/classifier/108/other/2417 new file mode 100644 index 000000000..36db1d58a --- /dev/null +++ b/results/classifier/108/other/2417 @@ -0,0 +1,20 @@ +performance: 0.867 +device: 0.861 +graphic: 0.755 +semantic: 0.621 +network: 0.590 +debug: 0.590 +other: 0.476 +socket: 0.342 +boot: 0.305 +PID: 0.284 +permissions: 0.163 +files: 0.121 +KVM: 0.093 +vnc: 0.063 + +qemu-img allocates full size on exFAT when metadata preallocation is requested +Description of problem: +`qemu-img` seems to preallocate the full size of a qcow2 image on exFAT rather than just the metadata when that is requested. This was initially seen via libvirt/libvirt#649. exFAT does not support sparse files. +Steps to reproduce: +1. Run command diff --git a/results/classifier/108/other/2419 b/results/classifier/108/other/2419 new file mode 100644 index 000000000..d1afafe62 --- /dev/null +++ b/results/classifier/108/other/2419 @@ -0,0 +1,33 @@ +device: 0.662 +graphic: 0.619 +semantic: 0.503 +performance: 0.403 +permissions: 0.337 +other: 0.305 +network: 0.294 +vnc: 0.286 +socket: 0.282 +files: 0.234 +boot: 0.232 +PID: 0.223 +debug: 0.154 +KVM: 0.040 + +ldapr_stlr_i instructions doesn't consider signed offset +Description of problem: +The format ldapr_stlr_i models the load acquire / store release immediate instructions. \ +These instructions has a bug in the sign extension calculation of the imm field. \ +imm should be defined as s9 instead of 9. + +@ldapr_stlr_i .. ...... .. . imm:9 .. rn:5 rt:5 &ldapr_stlr_i + +Should be changed to: + +@ldapr_stlr_i .. ...... .. . imm:s9 .. rn:5 rt:5 &ldapr_stlr_i +Steps to reproduce: +1. Run ARM target +2. Generate any ldapr_stlr_i instructions (for example: LDAPUR) +3. When the imm value is negative, the immediate calculation is done wrong. In case the calculation leads to an undefined location, QEMU will fail. +Additional information: +In trans_LDAPR_i (translate-a64.c), when imm field is negative, the value of a->imm will be 512-x instead of x. \ +I already fixed the issue by adding the s9 to the imm field. This made a call to sextend32 for imm instead of extend32 in the generated file build/libqemu-aarch64-softmmu.fa.p/decode-a64.c.inc diff --git a/results/classifier/108/other/24190340 b/results/classifier/108/other/24190340 new file mode 100644 index 000000000..712d8fc51 --- /dev/null +++ b/results/classifier/108/other/24190340 @@ -0,0 +1,2066 @@ +debug: 0.876 +permissions: 0.851 +device: 0.832 +performance: 0.832 +other: 0.811 +vnc: 0.808 +boot: 0.803 +semantic: 0.793 +KVM: 0.776 +graphic: 0.775 +files: 0.770 +PID: 0.762 +network: 0.723 +socket: 0.715 + +[BUG, RFC] Block graph deadlock on job-dismiss + +Hi all, + +There's a bug in block layer which leads to block graph deadlock. +Notably, it takes place when blockdev IO is processed within a separate +iothread. + +This was initially caught by our tests, and I was able to reduce it to a +relatively simple reproducer. Such deadlocks are probably supposed to +be covered in iotests/graph-changes-while-io, but this deadlock isn't. + +Basically what the reproducer does is launches QEMU with a drive having +'iothread' option set, creates a chain of 2 snapshots, launches +block-commit job for a snapshot and then dismisses the job, starting +from the lower snapshot. If the guest is issuing IO at the same time, +there's a race in acquiring block graph lock and a potential deadlock. + +Here's how it can be reproduced: + +1. Run QEMU: +> +SRCDIR=/path/to/srcdir +> +> +> +> +> +$SRCDIR/build/qemu-system-x86_64 -enable-kvm \ +> +> +-machine q35 -cpu Nehalem \ +> +> +-name guest=alma8-vm,debug-threads=on \ +> +> +-m 2g -smp 2 \ +> +> +-nographic -nodefaults \ +> +> +-qmp unix:/var/run/alma8-qmp.sock,server=on,wait=off \ +> +> +-serial unix:/var/run/alma8-serial.sock,server=on,wait=off \ +> +> +-object iothread,id=iothread0 \ +> +> +-blockdev +> +node-name=disk,driver=qcow2,file.driver=file,file.filename=/path/to/img/alma8.qcow2 +> +\ +> +-device virtio-blk-pci,drive=disk,iothread=iothread0 +2. Launch IO (random reads) from within the guest: +> +nc -U /var/run/alma8-serial.sock +> +... +> +[root@alma8-vm ~]# fio --name=randread --ioengine=libaio --direct=1 --bs=4k +> +--size=1G --numjobs=1 --time_based=1 --runtime=300 --group_reporting +> +--rw=randread --iodepth=1 --filename=/testfile +3. Run snapshots creation & removal of lower snapshot operation in a +loop (script attached): +> +while /bin/true ; do ./remove_lower_snap.sh ; done +And then it occasionally hangs. + +Note: I've tried bisecting this, and looks like deadlock occurs starting +from the following commit: + +(BAD) 5bdbaebcce virtio: Re-enable notifications after drain +(GOOD) c42c3833e0 virtio-scsi: Attach event vq notifier with no_poll + +On the latest v10.0.0 it does hang as well. + + +Here's backtrace of the main thread: + +> +#0 0x00007fc547d427ce in __ppoll (fds=0x557eb79657b0, nfds=1, +> +timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:43 +> +#1 0x0000557eb47d955c in qemu_poll_ns (fds=0x557eb79657b0, nfds=1, +> +timeout=-1) at ../util/qemu-timer.c:329 +> +#2 0x0000557eb47b2204 in fdmon_poll_wait (ctx=0x557eb76c5f20, +> +ready_list=0x7ffd94b4edd8, timeout=-1) at ../util/fdmon-poll.c:79 +> +#3 0x0000557eb47b1c45 in aio_poll (ctx=0x557eb76c5f20, blocking=true) at +> +../util/aio-posix.c:730 +> +#4 0x0000557eb4621edd in bdrv_do_drained_begin (bs=0x557eb795e950, +> +parent=0x0, poll=true) at ../block/io.c:378 +> +#5 0x0000557eb4621f7b in bdrv_drained_begin (bs=0x557eb795e950) at +> +../block/io.c:391 +> +#6 0x0000557eb45ec125 in bdrv_change_aio_context (bs=0x557eb795e950, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../block.c:7682 +> +#7 0x0000557eb45ebf2b in bdrv_child_change_aio_context (c=0x557eb7964250, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../block.c:7608 +> +#8 0x0000557eb45ec0c4 in bdrv_change_aio_context (bs=0x557eb79575e0, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../block.c:7668 +> +#9 0x0000557eb45ebf2b in bdrv_child_change_aio_context (c=0x557eb7e59110, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../block.c:7608 +> +#10 0x0000557eb45ec0c4 in bdrv_change_aio_context (bs=0x557eb7e51960, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../block.c:7668 +> +#11 0x0000557eb45ebf2b in bdrv_child_change_aio_context (c=0x557eb814ed80, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../block.c:7608 +> +#12 0x0000557eb45ee8e4 in child_job_change_aio_ctx (c=0x557eb7c9d3f0, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../blockjob.c:157 +> +#13 0x0000557eb45ebe2d in bdrv_parent_change_aio_context (c=0x557eb7c9d3f0, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../block.c:7592 +> +#14 0x0000557eb45ec06b in bdrv_change_aio_context (bs=0x557eb7d74310, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../block.c:7661 +> +#15 0x0000557eb45dcd7e in bdrv_child_cb_change_aio_ctx +> +(child=0x557eb8565af0, ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = +> +{...}, tran=0x557eb7a87160, errp=0x0) at ../block.c:1234 +> +#16 0x0000557eb45ebe2d in bdrv_parent_change_aio_context (c=0x557eb8565af0, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../block.c:7592 +> +#17 0x0000557eb45ec06b in bdrv_change_aio_context (bs=0x557eb79575e0, +> +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +errp=0x0) +> +at ../block.c:7661 +> +#18 0x0000557eb45ec1f3 in bdrv_try_change_aio_context (bs=0x557eb79575e0, +> +ctx=0x557eb76c5f20, ignore_child=0x0, errp=0x0) at ../block.c:7715 +> +#19 0x0000557eb45e1b15 in bdrv_root_unref_child (child=0x557eb7966f30) at +> +../block.c:3317 +> +#20 0x0000557eb45eeaa8 in block_job_remove_all_bdrv (job=0x557eb7952800) at +> +../blockjob.c:209 +> +#21 0x0000557eb45ee641 in block_job_free (job=0x557eb7952800) at +> +../blockjob.c:82 +> +#22 0x0000557eb45f17af in job_unref_locked (job=0x557eb7952800) at +> +../job.c:474 +> +#23 0x0000557eb45f257d in job_do_dismiss_locked (job=0x557eb7952800) at +> +../job.c:771 +> +#24 0x0000557eb45f25fe in job_dismiss_locked (jobptr=0x7ffd94b4f400, +> +errp=0x7ffd94b4f488) at ../job.c:783 +> +--Type <RET> for more, q to quit, c to continue without paging-- +> +#25 0x0000557eb45d8e84 in qmp_job_dismiss (id=0x557eb7aa42b0 "commit-snap1", +> +errp=0x7ffd94b4f488) at ../job-qmp.c:138 +> +#26 0x0000557eb472f6a3 in qmp_marshal_job_dismiss (args=0x7fc52c00a3b0, +> +ret=0x7fc53c880da8, errp=0x7fc53c880da0) at qapi/qapi-commands-job.c:221 +> +#27 0x0000557eb47a35f3 in do_qmp_dispatch_bh (opaque=0x7fc53c880e40) at +> +../qapi/qmp-dispatch.c:128 +> +#28 0x0000557eb47d1cd2 in aio_bh_call (bh=0x557eb79568f0) at +> +../util/async.c:172 +> +#29 0x0000557eb47d1df5 in aio_bh_poll (ctx=0x557eb76c0200) at +> +../util/async.c:219 +> +#30 0x0000557eb47b12f3 in aio_dispatch (ctx=0x557eb76c0200) at +> +../util/aio-posix.c:436 +> +#31 0x0000557eb47d2266 in aio_ctx_dispatch (source=0x557eb76c0200, +> +callback=0x0, user_data=0x0) at ../util/async.c:361 +> +#32 0x00007fc549232f4f in g_main_dispatch (context=0x557eb76c6430) at +> +../glib/gmain.c:3364 +> +#33 g_main_context_dispatch (context=0x557eb76c6430) at ../glib/gmain.c:4079 +> +#34 0x0000557eb47d3ab1 in glib_pollfds_poll () at ../util/main-loop.c:287 +> +#35 0x0000557eb47d3b38 in os_host_main_loop_wait (timeout=0) at +> +../util/main-loop.c:310 +> +#36 0x0000557eb47d3c58 in main_loop_wait (nonblocking=0) at +> +../util/main-loop.c:589 +> +#37 0x0000557eb4218b01 in qemu_main_loop () at ../system/runstate.c:835 +> +#38 0x0000557eb46df166 in qemu_default_main (opaque=0x0) at +> +../system/main.c:50 +> +#39 0x0000557eb46df215 in main (argc=24, argv=0x7ffd94b4f8d8) at +> +../system/main.c:80 +And here's coroutine trying to acquire read lock: + +> +(gdb) qemu coroutine reader_queue->entries.sqh_first +> +#0 0x0000557eb47d7068 in qemu_coroutine_switch (from_=0x557eb7aa48b0, +> +to_=0x7fc537fff508, action=COROUTINE_YIELD) at +> +../util/coroutine-ucontext.c:321 +> +#1 0x0000557eb47d4d4a in qemu_coroutine_yield () at +> +../util/qemu-coroutine.c:339 +> +#2 0x0000557eb47d56c8 in qemu_co_queue_wait_impl (queue=0x557eb59954c0 +> +<reader_queue>, lock=0x7fc53c57de50, flags=0) at +> +../util/qemu-coroutine-lock.c:60 +> +#3 0x0000557eb461fea7 in bdrv_graph_co_rdlock () at ../block/graph-lock.c:231 +> +#4 0x0000557eb460c81a in graph_lockable_auto_lock (x=0x7fc53c57dee3) at +> +/home/root/src/qemu/master/include/block/graph-lock.h:213 +> +#5 0x0000557eb460fa41 in blk_co_do_preadv_part +> +(blk=0x557eb84c0810, offset=6890553344, bytes=4096, qiov=0x7fc530006988, +> +qiov_offset=0, flags=BDRV_REQ_REGISTERED_BUF) at ../block/block-backend.c:1339 +> +#6 0x0000557eb46104d7 in blk_aio_read_entry (opaque=0x7fc530003240) at +> +../block/block-backend.c:1619 +> +#7 0x0000557eb47d6c40 in coroutine_trampoline (i0=-1213577040, i1=21886) at +> +../util/coroutine-ucontext.c:175 +> +#8 0x00007fc547c2a360 in __start_context () at +> +../sysdeps/unix/sysv/linux/x86_64/__start_context.S:91 +> +#9 0x00007ffd94b4ea40 in () +> +#10 0x0000000000000000 in () +So it looks like main thread is processing job-dismiss request and is +holding write lock taken in block_job_remove_all_bdrv() (frame #20 +above). At the same time iothread spawns a coroutine which performs IO +request. Before the coroutine is spawned, blk_aio_prwv() increases +'in_flight' counter for Blk. Then blk_co_do_preadv_part() (frame #5) is +trying to acquire the read lock. But main thread isn't releasing the +lock as blk_root_drained_poll() returns true since blk->in_flight > 0. +Here's the deadlock. + +Any comments and suggestions on the subject are welcomed. Thanks! + +Andrey +remove_lower_snap.sh +Description: +application/shellscript + +On 4/24/25 8:32 PM, Andrey Drobyshev wrote: +> +Hi all, +> +> +There's a bug in block layer which leads to block graph deadlock. +> +Notably, it takes place when blockdev IO is processed within a separate +> +iothread. +> +> +This was initially caught by our tests, and I was able to reduce it to a +> +relatively simple reproducer. Such deadlocks are probably supposed to +> +be covered in iotests/graph-changes-while-io, but this deadlock isn't. +> +> +Basically what the reproducer does is launches QEMU with a drive having +> +'iothread' option set, creates a chain of 2 snapshots, launches +> +block-commit job for a snapshot and then dismisses the job, starting +> +from the lower snapshot. If the guest is issuing IO at the same time, +> +there's a race in acquiring block graph lock and a potential deadlock. +> +> +Here's how it can be reproduced: +> +> +[...] +> +I took a closer look at iotests/graph-changes-while-io, and have managed +to reproduce the same deadlock in a much simpler setup, without a guest. + +1. Run QSD:> ./build/storage-daemon/qemu-storage-daemon --object +iothread,id=iothread0 \ +> +--blockdev null-co,node-name=node0,read-zeroes=true \ +> +> +--nbd-server addr.type=unix,addr.path=/var/run/qsd_nbd.sock \ +> +> +--export +> +nbd,id=exp0,node-name=node0,iothread=iothread0,fixed-iothread=true,writable=true +> +\ +> +--chardev +> +socket,id=qmp-sock,path=/var/run/qsd_qmp.sock,server=on,wait=off \ +> +--monitor chardev=qmp-sock +2. Launch IO: +> +qemu-img bench -f raw -c 2000000 +> +'nbd+unix:///node0?socket=/var/run/qsd_nbd.sock' +3. Add 2 snapshots and remove lower one (script attached):> while +/bin/true ; do ./rls_qsd.sh ; done + +And then it hangs. + +I'll also send a patch with corresponding test case added directly to +iotests. + +This reproduce seems to be hanging starting from Fiona's commit +67446e605dc ("blockjob: drop AioContext lock before calling +bdrv_graph_wrlock()"). AioContext locks were dropped entirely later on +in Stefan's commit b49f4755c7 ("block: remove AioContext locking"), but +the problem remains. + +Andrey +rls_qsd.sh +Description: +application/shellscript + +From: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com> + +This case is catching potential deadlock which takes place when job-dismiss +is issued when I/O requests are processed in a separate iothread. + +See +https://mail.gnu.org/archive/html/qemu-devel/2025-04/msg04421.html +Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com> +--- + .../qemu-iotests/tests/graph-changes-while-io | 101 ++++++++++++++++-- + .../tests/graph-changes-while-io.out | 4 +- + 2 files changed, 96 insertions(+), 9 deletions(-) + +diff --git a/tests/qemu-iotests/tests/graph-changes-while-io +b/tests/qemu-iotests/tests/graph-changes-while-io +index 194fda500e..e30f823da4 100755 +--- a/tests/qemu-iotests/tests/graph-changes-while-io ++++ b/tests/qemu-iotests/tests/graph-changes-while-io +@@ -27,6 +27,8 @@ from iotests import imgfmt, qemu_img, qemu_img_create, +qemu_io, \ + + + top = os.path.join(iotests.test_dir, 'top.img') ++snap1 = os.path.join(iotests.test_dir, 'snap1.img') ++snap2 = os.path.join(iotests.test_dir, 'snap2.img') + nbd_sock = os.path.join(iotests.sock_dir, 'nbd.sock') + + +@@ -58,6 +60,15 @@ class TestGraphChangesWhileIO(QMPTestCase): + def tearDown(self) -> None: + self.qsd.stop() + ++ def _wait_for_blockjob(self, status) -> None: ++ done = False ++ while not done: ++ for event in self.qsd.get_qmp().get_events(wait=10.0): ++ if event['event'] != 'JOB_STATUS_CHANGE': ++ continue ++ if event['data']['status'] == status: ++ done = True ++ + def test_blockdev_add_while_io(self) -> None: + # Run qemu-img bench in the background + bench_thr = Thread(target=do_qemu_img_bench) +@@ -116,13 +127,89 @@ class TestGraphChangesWhileIO(QMPTestCase): + 'device': 'job0', + }) + +- cancelled = False +- while not cancelled: +- for event in self.qsd.get_qmp().get_events(wait=10.0): +- if event['event'] != 'JOB_STATUS_CHANGE': +- continue +- if event['data']['status'] == 'null': +- cancelled = True ++ self._wait_for_blockjob('null') ++ ++ bench_thr.join() ++ ++ def test_remove_lower_snapshot_while_io(self) -> None: ++ # Run qemu-img bench in the background ++ bench_thr = Thread(target=do_qemu_img_bench, args=(100000, )) ++ bench_thr.start() ++ ++ # While I/O is performed on 'node0' node, consequently add 2 snapshots ++ # on top of it, then remove (commit) them starting from lower one. ++ while bench_thr.is_alive(): ++ # Recreate snapshot images on every iteration ++ qemu_img_create('-f', imgfmt, snap1, '1G') ++ qemu_img_create('-f', imgfmt, snap2, '1G') ++ ++ self.qsd.cmd('blockdev-add', { ++ 'driver': imgfmt, ++ 'node-name': 'snap1', ++ 'file': { ++ 'driver': 'file', ++ 'filename': snap1 ++ } ++ }) ++ ++ self.qsd.cmd('blockdev-snapshot', { ++ 'node': 'node0', ++ 'overlay': 'snap1', ++ }) ++ ++ self.qsd.cmd('blockdev-add', { ++ 'driver': imgfmt, ++ 'node-name': 'snap2', ++ 'file': { ++ 'driver': 'file', ++ 'filename': snap2 ++ } ++ }) ++ ++ self.qsd.cmd('blockdev-snapshot', { ++ 'node': 'snap1', ++ 'overlay': 'snap2', ++ }) ++ ++ self.qsd.cmd('block-commit', { ++ 'job-id': 'commit-snap1', ++ 'device': 'snap2', ++ 'top-node': 'snap1', ++ 'base-node': 'node0', ++ 'auto-finalize': True, ++ 'auto-dismiss': False, ++ }) ++ ++ self._wait_for_blockjob('concluded') ++ self.qsd.cmd('job-dismiss', { ++ 'id': 'commit-snap1', ++ }) ++ ++ self.qsd.cmd('block-commit', { ++ 'job-id': 'commit-snap2', ++ 'device': 'snap2', ++ 'top-node': 'snap2', ++ 'base-node': 'node0', ++ 'auto-finalize': True, ++ 'auto-dismiss': False, ++ }) ++ ++ self._wait_for_blockjob('ready') ++ self.qsd.cmd('job-complete', { ++ 'id': 'commit-snap2', ++ }) ++ ++ self._wait_for_blockjob('concluded') ++ self.qsd.cmd('job-dismiss', { ++ 'id': 'commit-snap2', ++ }) ++ ++ self.qsd.cmd('blockdev-del', { ++ 'node-name': 'snap1' ++ }) ++ self.qsd.cmd('blockdev-del', { ++ 'node-name': 'snap2' ++ }) + + bench_thr.join() + +diff --git a/tests/qemu-iotests/tests/graph-changes-while-io.out +b/tests/qemu-iotests/tests/graph-changes-while-io.out +index fbc63e62f8..8d7e996700 100644 +--- a/tests/qemu-iotests/tests/graph-changes-while-io.out ++++ b/tests/qemu-iotests/tests/graph-changes-while-io.out +@@ -1,5 +1,5 @@ +-.. ++... + ---------------------------------------------------------------------- +-Ran 2 tests ++Ran 3 tests + + OK +-- +2.43.5 + +Am 24.04.25 um 19:32 schrieb Andrey Drobyshev: +> +So it looks like main thread is processing job-dismiss request and is +> +holding write lock taken in block_job_remove_all_bdrv() (frame #20 +> +above). At the same time iothread spawns a coroutine which performs IO +> +request. Before the coroutine is spawned, blk_aio_prwv() increases +> +'in_flight' counter for Blk. Then blk_co_do_preadv_part() (frame #5) is +> +trying to acquire the read lock. But main thread isn't releasing the +> +lock as blk_root_drained_poll() returns true since blk->in_flight > 0. +> +Here's the deadlock. +And for the IO test you provided, it's client->nb_requests that behaves +similarly to blk->in_flight here. + +The issue also reproduces easily when issuing the following QMP command +in a loop while doing IO on a device: + +> +void qmp_block_locked_drain(const char *node_name, Error **errp) +> +{ +> +BlockDriverState *bs; +> +> +bs = bdrv_find_node(node_name); +> +if (!bs) { +> +error_setg(errp, "node not found"); +> +return; +> +} +> +> +bdrv_graph_wrlock(); +> +bdrv_drained_begin(bs); +> +bdrv_drained_end(bs); +> +bdrv_graph_wrunlock(); +> +} +It seems like either it would be necessary to require: +1. not draining inside an exclusively locked section +or +2. making sure that variables used by drained_poll routines are only set +while holding the reader lock +? + +Those seem to require rather involved changes, so a third option might +be to make draining inside an exclusively locked section possible, by +embedding such locked sections in a drained section: + +> +diff --git a/blockjob.c b/blockjob.c +> +index 32007f31a9..9b2f3b3ea9 100644 +> +--- a/blockjob.c +> ++++ b/blockjob.c +> +@@ -198,6 +198,7 @@ void block_job_remove_all_bdrv(BlockJob *job) +> +* one to make sure that such a concurrent access does not attempt +> +* to process an already freed BdrvChild. +> +*/ +> ++ bdrv_drain_all_begin(); +> +bdrv_graph_wrlock(); +> +while (job->nodes) { +> +GSList *l = job->nodes; +> +@@ -211,6 +212,7 @@ void block_job_remove_all_bdrv(BlockJob *job) +> +g_slist_free_1(l); +> +} +> +bdrv_graph_wrunlock(); +> ++ bdrv_drain_all_end(); +> +} +> +> +bool block_job_has_bdrv(BlockJob *job, BlockDriverState *bs) +This seems to fix the issue at hand. I can send a patch if this is +considered an acceptable approach. + +Best Regards, +Fiona + +On 4/30/25 11:47 AM, Fiona Ebner wrote: +> +Am 24.04.25 um 19:32 schrieb Andrey Drobyshev: +> +> So it looks like main thread is processing job-dismiss request and is +> +> holding write lock taken in block_job_remove_all_bdrv() (frame #20 +> +> above). At the same time iothread spawns a coroutine which performs IO +> +> request. Before the coroutine is spawned, blk_aio_prwv() increases +> +> 'in_flight' counter for Blk. Then blk_co_do_preadv_part() (frame #5) is +> +> trying to acquire the read lock. But main thread isn't releasing the +> +> lock as blk_root_drained_poll() returns true since blk->in_flight > 0. +> +> Here's the deadlock. +> +> +And for the IO test you provided, it's client->nb_requests that behaves +> +similarly to blk->in_flight here. +> +> +The issue also reproduces easily when issuing the following QMP command +> +in a loop while doing IO on a device: +> +> +> void qmp_block_locked_drain(const char *node_name, Error **errp) +> +> { +> +> BlockDriverState *bs; +> +> +> +> bs = bdrv_find_node(node_name); +> +> if (!bs) { +> +> error_setg(errp, "node not found"); +> +> return; +> +> } +> +> +> +> bdrv_graph_wrlock(); +> +> bdrv_drained_begin(bs); +> +> bdrv_drained_end(bs); +> +> bdrv_graph_wrunlock(); +> +> } +> +> +It seems like either it would be necessary to require: +> +1. not draining inside an exclusively locked section +> +or +> +2. making sure that variables used by drained_poll routines are only set +> +while holding the reader lock +> +? +> +> +Those seem to require rather involved changes, so a third option might +> +be to make draining inside an exclusively locked section possible, by +> +embedding such locked sections in a drained section: +> +> +> diff --git a/blockjob.c b/blockjob.c +> +> index 32007f31a9..9b2f3b3ea9 100644 +> +> --- a/blockjob.c +> +> +++ b/blockjob.c +> +> @@ -198,6 +198,7 @@ void block_job_remove_all_bdrv(BlockJob *job) +> +> * one to make sure that such a concurrent access does not attempt +> +> * to process an already freed BdrvChild. +> +> */ +> +> + bdrv_drain_all_begin(); +> +> bdrv_graph_wrlock(); +> +> while (job->nodes) { +> +> GSList *l = job->nodes; +> +> @@ -211,6 +212,7 @@ void block_job_remove_all_bdrv(BlockJob *job) +> +> g_slist_free_1(l); +> +> } +> +> bdrv_graph_wrunlock(); +> +> + bdrv_drain_all_end(); +> +> } +> +> +> +> bool block_job_has_bdrv(BlockJob *job, BlockDriverState *bs) +> +> +This seems to fix the issue at hand. I can send a patch if this is +> +considered an acceptable approach. +> +> +Best Regards, +> +Fiona +> +Hello Fiona, + +Thanks for looking into it. I've tried your 3rd option above and can +confirm it does fix the deadlock, at least I can't reproduce it. Other +iotests also don't seem to be breaking. So I personally am fine with +that patch. Would be nice to hear a word from the maintainers though on +whether there're any caveats with such approach. + +Andrey + +On Wed, Apr 30, 2025 at 10:11â¯AM Andrey Drobyshev +<andrey.drobyshev@virtuozzo.com> wrote: +> +> +On 4/30/25 11:47 AM, Fiona Ebner wrote: +> +> Am 24.04.25 um 19:32 schrieb Andrey Drobyshev: +> +>> So it looks like main thread is processing job-dismiss request and is +> +>> holding write lock taken in block_job_remove_all_bdrv() (frame #20 +> +>> above). At the same time iothread spawns a coroutine which performs IO +> +>> request. Before the coroutine is spawned, blk_aio_prwv() increases +> +>> 'in_flight' counter for Blk. Then blk_co_do_preadv_part() (frame #5) is +> +>> trying to acquire the read lock. But main thread isn't releasing the +> +>> lock as blk_root_drained_poll() returns true since blk->in_flight > 0. +> +>> Here's the deadlock. +> +> +> +> And for the IO test you provided, it's client->nb_requests that behaves +> +> similarly to blk->in_flight here. +> +> +> +> The issue also reproduces easily when issuing the following QMP command +> +> in a loop while doing IO on a device: +> +> +> +>> void qmp_block_locked_drain(const char *node_name, Error **errp) +> +>> { +> +>> BlockDriverState *bs; +> +>> +> +>> bs = bdrv_find_node(node_name); +> +>> if (!bs) { +> +>> error_setg(errp, "node not found"); +> +>> return; +> +>> } +> +>> +> +>> bdrv_graph_wrlock(); +> +>> bdrv_drained_begin(bs); +> +>> bdrv_drained_end(bs); +> +>> bdrv_graph_wrunlock(); +> +>> } +> +> +> +> It seems like either it would be necessary to require: +> +> 1. not draining inside an exclusively locked section +> +> or +> +> 2. making sure that variables used by drained_poll routines are only set +> +> while holding the reader lock +> +> ? +> +> +> +> Those seem to require rather involved changes, so a third option might +> +> be to make draining inside an exclusively locked section possible, by +> +> embedding such locked sections in a drained section: +> +> +> +>> diff --git a/blockjob.c b/blockjob.c +> +>> index 32007f31a9..9b2f3b3ea9 100644 +> +>> --- a/blockjob.c +> +>> +++ b/blockjob.c +> +>> @@ -198,6 +198,7 @@ void block_job_remove_all_bdrv(BlockJob *job) +> +>> * one to make sure that such a concurrent access does not attempt +> +>> * to process an already freed BdrvChild. +> +>> */ +> +>> + bdrv_drain_all_begin(); +> +>> bdrv_graph_wrlock(); +> +>> while (job->nodes) { +> +>> GSList *l = job->nodes; +> +>> @@ -211,6 +212,7 @@ void block_job_remove_all_bdrv(BlockJob *job) +> +>> g_slist_free_1(l); +> +>> } +> +>> bdrv_graph_wrunlock(); +> +>> + bdrv_drain_all_end(); +> +>> } +> +>> +> +>> bool block_job_has_bdrv(BlockJob *job, BlockDriverState *bs) +> +> +> +> This seems to fix the issue at hand. I can send a patch if this is +> +> considered an acceptable approach. +Kevin is aware of this thread but it's a public holiday tomorrow so it +may be a little longer. + +Stefan + +Am 24.04.2025 um 19:32 hat Andrey Drobyshev geschrieben: +> +Hi all, +> +> +There's a bug in block layer which leads to block graph deadlock. +> +Notably, it takes place when blockdev IO is processed within a separate +> +iothread. +> +> +This was initially caught by our tests, and I was able to reduce it to a +> +relatively simple reproducer. Such deadlocks are probably supposed to +> +be covered in iotests/graph-changes-while-io, but this deadlock isn't. +> +> +Basically what the reproducer does is launches QEMU with a drive having +> +'iothread' option set, creates a chain of 2 snapshots, launches +> +block-commit job for a snapshot and then dismisses the job, starting +> +from the lower snapshot. If the guest is issuing IO at the same time, +> +there's a race in acquiring block graph lock and a potential deadlock. +> +> +Here's how it can be reproduced: +> +> +1. Run QEMU: +> +> SRCDIR=/path/to/srcdir +> +> +> +> +> +> +> +> +> +> $SRCDIR/build/qemu-system-x86_64 -enable-kvm \ +> +> +> +> -machine q35 -cpu Nehalem \ +> +> +> +> -name guest=alma8-vm,debug-threads=on \ +> +> +> +> -m 2g -smp 2 \ +> +> +> +> -nographic -nodefaults \ +> +> +> +> -qmp unix:/var/run/alma8-qmp.sock,server=on,wait=off \ +> +> +> +> -serial unix:/var/run/alma8-serial.sock,server=on,wait=off \ +> +> +> +> -object iothread,id=iothread0 \ +> +> +> +> -blockdev +> +> node-name=disk,driver=qcow2,file.driver=file,file.filename=/path/to/img/alma8.qcow2 +> +> \ +> +> -device virtio-blk-pci,drive=disk,iothread=iothread0 +> +> +2. Launch IO (random reads) from within the guest: +> +> nc -U /var/run/alma8-serial.sock +> +> ... +> +> [root@alma8-vm ~]# fio --name=randread --ioengine=libaio --direct=1 --bs=4k +> +> --size=1G --numjobs=1 --time_based=1 --runtime=300 --group_reporting +> +> --rw=randread --iodepth=1 --filename=/testfile +> +> +3. Run snapshots creation & removal of lower snapshot operation in a +> +loop (script attached): +> +> while /bin/true ; do ./remove_lower_snap.sh ; done +> +> +And then it occasionally hangs. +> +> +Note: I've tried bisecting this, and looks like deadlock occurs starting +> +from the following commit: +> +> +(BAD) 5bdbaebcce virtio: Re-enable notifications after drain +> +(GOOD) c42c3833e0 virtio-scsi: Attach event vq notifier with no_poll +> +> +On the latest v10.0.0 it does hang as well. +> +> +> +Here's backtrace of the main thread: +> +> +> #0 0x00007fc547d427ce in __ppoll (fds=0x557eb79657b0, nfds=1, +> +> timeout=<optimized out>, sigmask=0x0) at +> +> ../sysdeps/unix/sysv/linux/ppoll.c:43 +> +> #1 0x0000557eb47d955c in qemu_poll_ns (fds=0x557eb79657b0, nfds=1, +> +> timeout=-1) at ../util/qemu-timer.c:329 +> +> #2 0x0000557eb47b2204 in fdmon_poll_wait (ctx=0x557eb76c5f20, +> +> ready_list=0x7ffd94b4edd8, timeout=-1) at ../util/fdmon-poll.c:79 +> +> #3 0x0000557eb47b1c45 in aio_poll (ctx=0x557eb76c5f20, blocking=true) at +> +> ../util/aio-posix.c:730 +> +> #4 0x0000557eb4621edd in bdrv_do_drained_begin (bs=0x557eb795e950, +> +> parent=0x0, poll=true) at ../block/io.c:378 +> +> #5 0x0000557eb4621f7b in bdrv_drained_begin (bs=0x557eb795e950) at +> +> ../block/io.c:391 +> +> #6 0x0000557eb45ec125 in bdrv_change_aio_context (bs=0x557eb795e950, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../block.c:7682 +> +> #7 0x0000557eb45ebf2b in bdrv_child_change_aio_context (c=0x557eb7964250, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../block.c:7608 +> +> #8 0x0000557eb45ec0c4 in bdrv_change_aio_context (bs=0x557eb79575e0, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../block.c:7668 +> +> #9 0x0000557eb45ebf2b in bdrv_child_change_aio_context (c=0x557eb7e59110, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../block.c:7608 +> +> #10 0x0000557eb45ec0c4 in bdrv_change_aio_context (bs=0x557eb7e51960, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../block.c:7668 +> +> #11 0x0000557eb45ebf2b in bdrv_child_change_aio_context (c=0x557eb814ed80, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../block.c:7608 +> +> #12 0x0000557eb45ee8e4 in child_job_change_aio_ctx (c=0x557eb7c9d3f0, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../blockjob.c:157 +> +> #13 0x0000557eb45ebe2d in bdrv_parent_change_aio_context (c=0x557eb7c9d3f0, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../block.c:7592 +> +> #14 0x0000557eb45ec06b in bdrv_change_aio_context (bs=0x557eb7d74310, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../block.c:7661 +> +> #15 0x0000557eb45dcd7e in bdrv_child_cb_change_aio_ctx +> +> (child=0x557eb8565af0, ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = +> +> {...}, tran=0x557eb7a87160, errp=0x0) at ../block.c:1234 +> +> #16 0x0000557eb45ebe2d in bdrv_parent_change_aio_context (c=0x557eb8565af0, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../block.c:7592 +> +> #17 0x0000557eb45ec06b in bdrv_change_aio_context (bs=0x557eb79575e0, +> +> ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +> +> errp=0x0) +> +> at ../block.c:7661 +> +> #18 0x0000557eb45ec1f3 in bdrv_try_change_aio_context (bs=0x557eb79575e0, +> +> ctx=0x557eb76c5f20, ignore_child=0x0, errp=0x0) at ../block.c:7715 +> +> #19 0x0000557eb45e1b15 in bdrv_root_unref_child (child=0x557eb7966f30) at +> +> ../block.c:3317 +> +> #20 0x0000557eb45eeaa8 in block_job_remove_all_bdrv (job=0x557eb7952800) at +> +> ../blockjob.c:209 +> +> #21 0x0000557eb45ee641 in block_job_free (job=0x557eb7952800) at +> +> ../blockjob.c:82 +> +> #22 0x0000557eb45f17af in job_unref_locked (job=0x557eb7952800) at +> +> ../job.c:474 +> +> #23 0x0000557eb45f257d in job_do_dismiss_locked (job=0x557eb7952800) at +> +> ../job.c:771 +> +> #24 0x0000557eb45f25fe in job_dismiss_locked (jobptr=0x7ffd94b4f400, +> +> errp=0x7ffd94b4f488) at ../job.c:783 +> +> --Type <RET> for more, q to quit, c to continue without paging-- +> +> #25 0x0000557eb45d8e84 in qmp_job_dismiss (id=0x557eb7aa42b0 +> +> "commit-snap1", errp=0x7ffd94b4f488) at ../job-qmp.c:138 +> +> #26 0x0000557eb472f6a3 in qmp_marshal_job_dismiss (args=0x7fc52c00a3b0, +> +> ret=0x7fc53c880da8, errp=0x7fc53c880da0) at qapi/qapi-commands-job.c:221 +> +> #27 0x0000557eb47a35f3 in do_qmp_dispatch_bh (opaque=0x7fc53c880e40) at +> +> ../qapi/qmp-dispatch.c:128 +> +> #28 0x0000557eb47d1cd2 in aio_bh_call (bh=0x557eb79568f0) at +> +> ../util/async.c:172 +> +> #29 0x0000557eb47d1df5 in aio_bh_poll (ctx=0x557eb76c0200) at +> +> ../util/async.c:219 +> +> #30 0x0000557eb47b12f3 in aio_dispatch (ctx=0x557eb76c0200) at +> +> ../util/aio-posix.c:436 +> +> #31 0x0000557eb47d2266 in aio_ctx_dispatch (source=0x557eb76c0200, +> +> callback=0x0, user_data=0x0) at ../util/async.c:361 +> +> #32 0x00007fc549232f4f in g_main_dispatch (context=0x557eb76c6430) at +> +> ../glib/gmain.c:3364 +> +> #33 g_main_context_dispatch (context=0x557eb76c6430) at ../glib/gmain.c:4079 +> +> #34 0x0000557eb47d3ab1 in glib_pollfds_poll () at ../util/main-loop.c:287 +> +> #35 0x0000557eb47d3b38 in os_host_main_loop_wait (timeout=0) at +> +> ../util/main-loop.c:310 +> +> #36 0x0000557eb47d3c58 in main_loop_wait (nonblocking=0) at +> +> ../util/main-loop.c:589 +> +> #37 0x0000557eb4218b01 in qemu_main_loop () at ../system/runstate.c:835 +> +> #38 0x0000557eb46df166 in qemu_default_main (opaque=0x0) at +> +> ../system/main.c:50 +> +> #39 0x0000557eb46df215 in main (argc=24, argv=0x7ffd94b4f8d8) at +> +> ../system/main.c:80 +> +> +> +And here's coroutine trying to acquire read lock: +> +> +> (gdb) qemu coroutine reader_queue->entries.sqh_first +> +> #0 0x0000557eb47d7068 in qemu_coroutine_switch (from_=0x557eb7aa48b0, +> +> to_=0x7fc537fff508, action=COROUTINE_YIELD) at +> +> ../util/coroutine-ucontext.c:321 +> +> #1 0x0000557eb47d4d4a in qemu_coroutine_yield () at +> +> ../util/qemu-coroutine.c:339 +> +> #2 0x0000557eb47d56c8 in qemu_co_queue_wait_impl (queue=0x557eb59954c0 +> +> <reader_queue>, lock=0x7fc53c57de50, flags=0) at +> +> ../util/qemu-coroutine-lock.c:60 +> +> #3 0x0000557eb461fea7 in bdrv_graph_co_rdlock () at +> +> ../block/graph-lock.c:231 +> +> #4 0x0000557eb460c81a in graph_lockable_auto_lock (x=0x7fc53c57dee3) at +> +> /home/root/src/qemu/master/include/block/graph-lock.h:213 +> +> #5 0x0000557eb460fa41 in blk_co_do_preadv_part +> +> (blk=0x557eb84c0810, offset=6890553344, bytes=4096, +> +> qiov=0x7fc530006988, qiov_offset=0, flags=BDRV_REQ_REGISTERED_BUF) at +> +> ../block/block-backend.c:1339 +> +> #6 0x0000557eb46104d7 in blk_aio_read_entry (opaque=0x7fc530003240) at +> +> ../block/block-backend.c:1619 +> +> #7 0x0000557eb47d6c40 in coroutine_trampoline (i0=-1213577040, i1=21886) +> +> at ../util/coroutine-ucontext.c:175 +> +> #8 0x00007fc547c2a360 in __start_context () at +> +> ../sysdeps/unix/sysv/linux/x86_64/__start_context.S:91 +> +> #9 0x00007ffd94b4ea40 in () +> +> #10 0x0000000000000000 in () +> +> +> +So it looks like main thread is processing job-dismiss request and is +> +holding write lock taken in block_job_remove_all_bdrv() (frame #20 +> +above). At the same time iothread spawns a coroutine which performs IO +> +request. Before the coroutine is spawned, blk_aio_prwv() increases +> +'in_flight' counter for Blk. Then blk_co_do_preadv_part() (frame #5) is +> +trying to acquire the read lock. But main thread isn't releasing the +> +lock as blk_root_drained_poll() returns true since blk->in_flight > 0. +> +Here's the deadlock. +> +> +Any comments and suggestions on the subject are welcomed. Thanks! +I think this is what the blk_wait_while_drained() call was supposed to +address in blk_co_do_preadv_part(). However, with the use of multiple +I/O threads, this is racy. + +Do you think that in your case we hit the small race window between the +checks in blk_wait_while_drained() and GRAPH_RDLOCK_GUARD()? Or is there +another reason why blk_wait_while_drained() didn't do its job? + +Kevin + +On 5/2/25 19:34, Kevin Wolf wrote: +Am 24.04.2025 um 19:32 hat Andrey Drobyshev geschrieben: +Hi all, + +There's a bug in block layer which leads to block graph deadlock. +Notably, it takes place when blockdev IO is processed within a separate +iothread. + +This was initially caught by our tests, and I was able to reduce it to a +relatively simple reproducer. Such deadlocks are probably supposed to +be covered in iotests/graph-changes-while-io, but this deadlock isn't. + +Basically what the reproducer does is launches QEMU with a drive having +'iothread' option set, creates a chain of 2 snapshots, launches +block-commit job for a snapshot and then dismisses the job, starting +from the lower snapshot. If the guest is issuing IO at the same time, +there's a race in acquiring block graph lock and a potential deadlock. + +Here's how it can be reproduced: + +1. Run QEMU: +SRCDIR=/path/to/srcdir +$SRCDIR/build/qemu-system-x86_64 -enable-kvm \ +-machine q35 -cpu Nehalem \ + -name guest=alma8-vm,debug-threads=on \ + -m 2g -smp 2 \ + -nographic -nodefaults \ + -qmp unix:/var/run/alma8-qmp.sock,server=on,wait=off \ + -serial unix:/var/run/alma8-serial.sock,server=on,wait=off \ + -object iothread,id=iothread0 \ + -blockdev +node-name=disk,driver=qcow2,file.driver=file,file.filename=/path/to/img/alma8.qcow2 + \ + -device virtio-blk-pci,drive=disk,iothread=iothread0 +2. Launch IO (random reads) from within the guest: +nc -U /var/run/alma8-serial.sock +... +[root@alma8-vm ~]# fio --name=randread --ioengine=libaio --direct=1 --bs=4k +--size=1G --numjobs=1 --time_based=1 --runtime=300 --group_reporting +--rw=randread --iodepth=1 --filename=/testfile +3. Run snapshots creation & removal of lower snapshot operation in a +loop (script attached): +while /bin/true ; do ./remove_lower_snap.sh ; done +And then it occasionally hangs. + +Note: I've tried bisecting this, and looks like deadlock occurs starting +from the following commit: + +(BAD) 5bdbaebcce virtio: Re-enable notifications after drain +(GOOD) c42c3833e0 virtio-scsi: Attach event vq notifier with no_poll + +On the latest v10.0.0 it does hang as well. + + +Here's backtrace of the main thread: +#0 0x00007fc547d427ce in __ppoll (fds=0x557eb79657b0, nfds=1, timeout=<optimized +out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:43 +#1 0x0000557eb47d955c in qemu_poll_ns (fds=0x557eb79657b0, nfds=1, timeout=-1) +at ../util/qemu-timer.c:329 +#2 0x0000557eb47b2204 in fdmon_poll_wait (ctx=0x557eb76c5f20, +ready_list=0x7ffd94b4edd8, timeout=-1) at ../util/fdmon-poll.c:79 +#3 0x0000557eb47b1c45 in aio_poll (ctx=0x557eb76c5f20, blocking=true) at +../util/aio-posix.c:730 +#4 0x0000557eb4621edd in bdrv_do_drained_begin (bs=0x557eb795e950, parent=0x0, +poll=true) at ../block/io.c:378 +#5 0x0000557eb4621f7b in bdrv_drained_begin (bs=0x557eb795e950) at +../block/io.c:391 +#6 0x0000557eb45ec125 in bdrv_change_aio_context (bs=0x557eb795e950, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../block.c:7682 +#7 0x0000557eb45ebf2b in bdrv_child_change_aio_context (c=0x557eb7964250, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../block.c:7608 +#8 0x0000557eb45ec0c4 in bdrv_change_aio_context (bs=0x557eb79575e0, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../block.c:7668 +#9 0x0000557eb45ebf2b in bdrv_child_change_aio_context (c=0x557eb7e59110, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../block.c:7608 +#10 0x0000557eb45ec0c4 in bdrv_change_aio_context (bs=0x557eb7e51960, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../block.c:7668 +#11 0x0000557eb45ebf2b in bdrv_child_change_aio_context (c=0x557eb814ed80, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../block.c:7608 +#12 0x0000557eb45ee8e4 in child_job_change_aio_ctx (c=0x557eb7c9d3f0, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../blockjob.c:157 +#13 0x0000557eb45ebe2d in bdrv_parent_change_aio_context (c=0x557eb7c9d3f0, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../block.c:7592 +#14 0x0000557eb45ec06b in bdrv_change_aio_context (bs=0x557eb7d74310, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../block.c:7661 +#15 0x0000557eb45dcd7e in bdrv_child_cb_change_aio_ctx + (child=0x557eb8565af0, ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +tran=0x557eb7a87160, errp=0x0) at ../block.c:1234 +#16 0x0000557eb45ebe2d in bdrv_parent_change_aio_context (c=0x557eb8565af0, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../block.c:7592 +#17 0x0000557eb45ec06b in bdrv_change_aio_context (bs=0x557eb79575e0, +ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, tran=0x557eb7a87160, +errp=0x0) + at ../block.c:7661 +#18 0x0000557eb45ec1f3 in bdrv_try_change_aio_context (bs=0x557eb79575e0, +ctx=0x557eb76c5f20, ignore_child=0x0, errp=0x0) at ../block.c:7715 +#19 0x0000557eb45e1b15 in bdrv_root_unref_child (child=0x557eb7966f30) at +../block.c:3317 +#20 0x0000557eb45eeaa8 in block_job_remove_all_bdrv (job=0x557eb7952800) at +../blockjob.c:209 +#21 0x0000557eb45ee641 in block_job_free (job=0x557eb7952800) at +../blockjob.c:82 +#22 0x0000557eb45f17af in job_unref_locked (job=0x557eb7952800) at ../job.c:474 +#23 0x0000557eb45f257d in job_do_dismiss_locked (job=0x557eb7952800) at +../job.c:771 +#24 0x0000557eb45f25fe in job_dismiss_locked (jobptr=0x7ffd94b4f400, +errp=0x7ffd94b4f488) at ../job.c:783 +--Type <RET> for more, q to quit, c to continue without paging-- +#25 0x0000557eb45d8e84 in qmp_job_dismiss (id=0x557eb7aa42b0 "commit-snap1", +errp=0x7ffd94b4f488) at ../job-qmp.c:138 +#26 0x0000557eb472f6a3 in qmp_marshal_job_dismiss (args=0x7fc52c00a3b0, +ret=0x7fc53c880da8, errp=0x7fc53c880da0) at qapi/qapi-commands-job.c:221 +#27 0x0000557eb47a35f3 in do_qmp_dispatch_bh (opaque=0x7fc53c880e40) at +../qapi/qmp-dispatch.c:128 +#28 0x0000557eb47d1cd2 in aio_bh_call (bh=0x557eb79568f0) at ../util/async.c:172 +#29 0x0000557eb47d1df5 in aio_bh_poll (ctx=0x557eb76c0200) at +../util/async.c:219 +#30 0x0000557eb47b12f3 in aio_dispatch (ctx=0x557eb76c0200) at +../util/aio-posix.c:436 +#31 0x0000557eb47d2266 in aio_ctx_dispatch (source=0x557eb76c0200, +callback=0x0, user_data=0x0) at ../util/async.c:361 +#32 0x00007fc549232f4f in g_main_dispatch (context=0x557eb76c6430) at +../glib/gmain.c:3364 +#33 g_main_context_dispatch (context=0x557eb76c6430) at ../glib/gmain.c:4079 +#34 0x0000557eb47d3ab1 in glib_pollfds_poll () at ../util/main-loop.c:287 +#35 0x0000557eb47d3b38 in os_host_main_loop_wait (timeout=0) at +../util/main-loop.c:310 +#36 0x0000557eb47d3c58 in main_loop_wait (nonblocking=0) at +../util/main-loop.c:589 +#37 0x0000557eb4218b01 in qemu_main_loop () at ../system/runstate.c:835 +#38 0x0000557eb46df166 in qemu_default_main (opaque=0x0) at ../system/main.c:50 +#39 0x0000557eb46df215 in main (argc=24, argv=0x7ffd94b4f8d8) at +../system/main.c:80 +And here's coroutine trying to acquire read lock: +(gdb) qemu coroutine reader_queue->entries.sqh_first +#0 0x0000557eb47d7068 in qemu_coroutine_switch (from_=0x557eb7aa48b0, +to_=0x7fc537fff508, action=COROUTINE_YIELD) at ../util/coroutine-ucontext.c:321 +#1 0x0000557eb47d4d4a in qemu_coroutine_yield () at +../util/qemu-coroutine.c:339 +#2 0x0000557eb47d56c8 in qemu_co_queue_wait_impl (queue=0x557eb59954c0 +<reader_queue>, lock=0x7fc53c57de50, flags=0) at +../util/qemu-coroutine-lock.c:60 +#3 0x0000557eb461fea7 in bdrv_graph_co_rdlock () at ../block/graph-lock.c:231 +#4 0x0000557eb460c81a in graph_lockable_auto_lock (x=0x7fc53c57dee3) at +/home/root/src/qemu/master/include/block/graph-lock.h:213 +#5 0x0000557eb460fa41 in blk_co_do_preadv_part + (blk=0x557eb84c0810, offset=6890553344, bytes=4096, qiov=0x7fc530006988, +qiov_offset=0, flags=BDRV_REQ_REGISTERED_BUF) at ../block/block-backend.c:1339 +#6 0x0000557eb46104d7 in blk_aio_read_entry (opaque=0x7fc530003240) at +../block/block-backend.c:1619 +#7 0x0000557eb47d6c40 in coroutine_trampoline (i0=-1213577040, i1=21886) at +../util/coroutine-ucontext.c:175 +#8 0x00007fc547c2a360 in __start_context () at +../sysdeps/unix/sysv/linux/x86_64/__start_context.S:91 +#9 0x00007ffd94b4ea40 in () +#10 0x0000000000000000 in () +So it looks like main thread is processing job-dismiss request and is +holding write lock taken in block_job_remove_all_bdrv() (frame #20 +above). At the same time iothread spawns a coroutine which performs IO +request. Before the coroutine is spawned, blk_aio_prwv() increases +'in_flight' counter for Blk. Then blk_co_do_preadv_part() (frame #5) is +trying to acquire the read lock. But main thread isn't releasing the +lock as blk_root_drained_poll() returns true since blk->in_flight > 0. +Here's the deadlock. + +Any comments and suggestions on the subject are welcomed. Thanks! +I think this is what the blk_wait_while_drained() call was supposed to +address in blk_co_do_preadv_part(). However, with the use of multiple +I/O threads, this is racy. + +Do you think that in your case we hit the small race window between the +checks in blk_wait_while_drained() and GRAPH_RDLOCK_GUARD()? Or is there +another reason why blk_wait_while_drained() didn't do its job? + +Kevin +At my opinion there is very big race window. Main thread has +eaten graph write lock. After that another coroutine is stalled +within GRAPH_RDLOCK_GUARD() as there is no drain at the moment and only +after that main thread has started drain. That is why Fiona's idea is +looking working. Though this would mean that normally we should always +do that at the moment when we acquire write lock. May be even inside +this function. Den + +Am 02.05.2025 um 19:52 hat Denis V. Lunev geschrieben: +> +On 5/2/25 19:34, Kevin Wolf wrote: +> +> Am 24.04.2025 um 19:32 hat Andrey Drobyshev geschrieben: +> +> > Hi all, +> +> > +> +> > There's a bug in block layer which leads to block graph deadlock. +> +> > Notably, it takes place when blockdev IO is processed within a separate +> +> > iothread. +> +> > +> +> > This was initially caught by our tests, and I was able to reduce it to a +> +> > relatively simple reproducer. Such deadlocks are probably supposed to +> +> > be covered in iotests/graph-changes-while-io, but this deadlock isn't. +> +> > +> +> > Basically what the reproducer does is launches QEMU with a drive having +> +> > 'iothread' option set, creates a chain of 2 snapshots, launches +> +> > block-commit job for a snapshot and then dismisses the job, starting +> +> > from the lower snapshot. If the guest is issuing IO at the same time, +> +> > there's a race in acquiring block graph lock and a potential deadlock. +> +> > +> +> > Here's how it can be reproduced: +> +> > +> +> > 1. Run QEMU: +> +> > > SRCDIR=/path/to/srcdir +> +> > > $SRCDIR/build/qemu-system-x86_64 -enable-kvm \ +> +> > > -machine q35 -cpu Nehalem \ +> +> > > -name guest=alma8-vm,debug-threads=on \ +> +> > > -m 2g -smp 2 \ +> +> > > -nographic -nodefaults \ +> +> > > -qmp unix:/var/run/alma8-qmp.sock,server=on,wait=off \ +> +> > > -serial unix:/var/run/alma8-serial.sock,server=on,wait=off \ +> +> > > -object iothread,id=iothread0 \ +> +> > > -blockdev +> +> > > node-name=disk,driver=qcow2,file.driver=file,file.filename=/path/to/img/alma8.qcow2 +> +> > > \ +> +> > > -device virtio-blk-pci,drive=disk,iothread=iothread0 +> +> > 2. Launch IO (random reads) from within the guest: +> +> > > nc -U /var/run/alma8-serial.sock +> +> > > ... +> +> > > [root@alma8-vm ~]# fio --name=randread --ioengine=libaio --direct=1 +> +> > > --bs=4k --size=1G --numjobs=1 --time_based=1 --runtime=300 +> +> > > --group_reporting --rw=randread --iodepth=1 --filename=/testfile +> +> > 3. Run snapshots creation & removal of lower snapshot operation in a +> +> > loop (script attached): +> +> > > while /bin/true ; do ./remove_lower_snap.sh ; done +> +> > And then it occasionally hangs. +> +> > +> +> > Note: I've tried bisecting this, and looks like deadlock occurs starting +> +> > from the following commit: +> +> > +> +> > (BAD) 5bdbaebcce virtio: Re-enable notifications after drain +> +> > (GOOD) c42c3833e0 virtio-scsi: Attach event vq notifier with no_poll +> +> > +> +> > On the latest v10.0.0 it does hang as well. +> +> > +> +> > +> +> > Here's backtrace of the main thread: +> +> > +> +> > > #0 0x00007fc547d427ce in __ppoll (fds=0x557eb79657b0, nfds=1, +> +> > > timeout=<optimized out>, sigmask=0x0) at +> +> > > ../sysdeps/unix/sysv/linux/ppoll.c:43 +> +> > > #1 0x0000557eb47d955c in qemu_poll_ns (fds=0x557eb79657b0, nfds=1, +> +> > > timeout=-1) at ../util/qemu-timer.c:329 +> +> > > #2 0x0000557eb47b2204 in fdmon_poll_wait (ctx=0x557eb76c5f20, +> +> > > ready_list=0x7ffd94b4edd8, timeout=-1) at ../util/fdmon-poll.c:79 +> +> > > #3 0x0000557eb47b1c45 in aio_poll (ctx=0x557eb76c5f20, blocking=true) +> +> > > at ../util/aio-posix.c:730 +> +> > > #4 0x0000557eb4621edd in bdrv_do_drained_begin (bs=0x557eb795e950, +> +> > > parent=0x0, poll=true) at ../block/io.c:378 +> +> > > #5 0x0000557eb4621f7b in bdrv_drained_begin (bs=0x557eb795e950) at +> +> > > ../block/io.c:391 +> +> > > #6 0x0000557eb45ec125 in bdrv_change_aio_context (bs=0x557eb795e950, +> +> > > ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../block.c:7682 +> +> > > #7 0x0000557eb45ebf2b in bdrv_child_change_aio_context +> +> > > (c=0x557eb7964250, ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../block.c:7608 +> +> > > #8 0x0000557eb45ec0c4 in bdrv_change_aio_context (bs=0x557eb79575e0, +> +> > > ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../block.c:7668 +> +> > > #9 0x0000557eb45ebf2b in bdrv_child_change_aio_context +> +> > > (c=0x557eb7e59110, ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../block.c:7608 +> +> > > #10 0x0000557eb45ec0c4 in bdrv_change_aio_context (bs=0x557eb7e51960, +> +> > > ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../block.c:7668 +> +> > > #11 0x0000557eb45ebf2b in bdrv_child_change_aio_context +> +> > > (c=0x557eb814ed80, ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../block.c:7608 +> +> > > #12 0x0000557eb45ee8e4 in child_job_change_aio_ctx (c=0x557eb7c9d3f0, +> +> > > ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../blockjob.c:157 +> +> > > #13 0x0000557eb45ebe2d in bdrv_parent_change_aio_context +> +> > > (c=0x557eb7c9d3f0, ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../block.c:7592 +> +> > > #14 0x0000557eb45ec06b in bdrv_change_aio_context (bs=0x557eb7d74310, +> +> > > ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../block.c:7661 +> +> > > #15 0x0000557eb45dcd7e in bdrv_child_cb_change_aio_ctx +> +> > > (child=0x557eb8565af0, ctx=0x557eb76c5f20, visited=0x557eb7e06b60 +> +> > > = {...}, tran=0x557eb7a87160, errp=0x0) at ../block.c:1234 +> +> > > #16 0x0000557eb45ebe2d in bdrv_parent_change_aio_context +> +> > > (c=0x557eb8565af0, ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../block.c:7592 +> +> > > #17 0x0000557eb45ec06b in bdrv_change_aio_context (bs=0x557eb79575e0, +> +> > > ctx=0x557eb76c5f20, visited=0x557eb7e06b60 = {...}, +> +> > > tran=0x557eb7a87160, errp=0x0) +> +> > > at ../block.c:7661 +> +> > > #18 0x0000557eb45ec1f3 in bdrv_try_change_aio_context +> +> > > (bs=0x557eb79575e0, ctx=0x557eb76c5f20, ignore_child=0x0, errp=0x0) at +> +> > > ../block.c:7715 +> +> > > #19 0x0000557eb45e1b15 in bdrv_root_unref_child (child=0x557eb7966f30) +> +> > > at ../block.c:3317 +> +> > > #20 0x0000557eb45eeaa8 in block_job_remove_all_bdrv +> +> > > (job=0x557eb7952800) at ../blockjob.c:209 +> +> > > #21 0x0000557eb45ee641 in block_job_free (job=0x557eb7952800) at +> +> > > ../blockjob.c:82 +> +> > > #22 0x0000557eb45f17af in job_unref_locked (job=0x557eb7952800) at +> +> > > ../job.c:474 +> +> > > #23 0x0000557eb45f257d in job_do_dismiss_locked (job=0x557eb7952800) at +> +> > > ../job.c:771 +> +> > > #24 0x0000557eb45f25fe in job_dismiss_locked (jobptr=0x7ffd94b4f400, +> +> > > errp=0x7ffd94b4f488) at ../job.c:783 +> +> > > --Type <RET> for more, q to quit, c to continue without paging-- +> +> > > #25 0x0000557eb45d8e84 in qmp_job_dismiss (id=0x557eb7aa42b0 +> +> > > "commit-snap1", errp=0x7ffd94b4f488) at ../job-qmp.c:138 +> +> > > #26 0x0000557eb472f6a3 in qmp_marshal_job_dismiss (args=0x7fc52c00a3b0, +> +> > > ret=0x7fc53c880da8, errp=0x7fc53c880da0) at qapi/qapi-commands-job.c:221 +> +> > > #27 0x0000557eb47a35f3 in do_qmp_dispatch_bh (opaque=0x7fc53c880e40) at +> +> > > ../qapi/qmp-dispatch.c:128 +> +> > > #28 0x0000557eb47d1cd2 in aio_bh_call (bh=0x557eb79568f0) at +> +> > > ../util/async.c:172 +> +> > > #29 0x0000557eb47d1df5 in aio_bh_poll (ctx=0x557eb76c0200) at +> +> > > ../util/async.c:219 +> +> > > #30 0x0000557eb47b12f3 in aio_dispatch (ctx=0x557eb76c0200) at +> +> > > ../util/aio-posix.c:436 +> +> > > #31 0x0000557eb47d2266 in aio_ctx_dispatch (source=0x557eb76c0200, +> +> > > callback=0x0, user_data=0x0) at ../util/async.c:361 +> +> > > #32 0x00007fc549232f4f in g_main_dispatch (context=0x557eb76c6430) at +> +> > > ../glib/gmain.c:3364 +> +> > > #33 g_main_context_dispatch (context=0x557eb76c6430) at +> +> > > ../glib/gmain.c:4079 +> +> > > #34 0x0000557eb47d3ab1 in glib_pollfds_poll () at +> +> > > ../util/main-loop.c:287 +> +> > > #35 0x0000557eb47d3b38 in os_host_main_loop_wait (timeout=0) at +> +> > > ../util/main-loop.c:310 +> +> > > #36 0x0000557eb47d3c58 in main_loop_wait (nonblocking=0) at +> +> > > ../util/main-loop.c:589 +> +> > > #37 0x0000557eb4218b01 in qemu_main_loop () at ../system/runstate.c:835 +> +> > > #38 0x0000557eb46df166 in qemu_default_main (opaque=0x0) at +> +> > > ../system/main.c:50 +> +> > > #39 0x0000557eb46df215 in main (argc=24, argv=0x7ffd94b4f8d8) at +> +> > > ../system/main.c:80 +> +> > +> +> > And here's coroutine trying to acquire read lock: +> +> > +> +> > > (gdb) qemu coroutine reader_queue->entries.sqh_first +> +> > > #0 0x0000557eb47d7068 in qemu_coroutine_switch (from_=0x557eb7aa48b0, +> +> > > to_=0x7fc537fff508, action=COROUTINE_YIELD) at +> +> > > ../util/coroutine-ucontext.c:321 +> +> > > #1 0x0000557eb47d4d4a in qemu_coroutine_yield () at +> +> > > ../util/qemu-coroutine.c:339 +> +> > > #2 0x0000557eb47d56c8 in qemu_co_queue_wait_impl (queue=0x557eb59954c0 +> +> > > <reader_queue>, lock=0x7fc53c57de50, flags=0) at +> +> > > ../util/qemu-coroutine-lock.c:60 +> +> > > #3 0x0000557eb461fea7 in bdrv_graph_co_rdlock () at +> +> > > ../block/graph-lock.c:231 +> +> > > #4 0x0000557eb460c81a in graph_lockable_auto_lock (x=0x7fc53c57dee3) +> +> > > at /home/root/src/qemu/master/include/block/graph-lock.h:213 +> +> > > #5 0x0000557eb460fa41 in blk_co_do_preadv_part +> +> > > (blk=0x557eb84c0810, offset=6890553344, bytes=4096, +> +> > > qiov=0x7fc530006988, qiov_offset=0, flags=BDRV_REQ_REGISTERED_BUF) at +> +> > > ../block/block-backend.c:1339 +> +> > > #6 0x0000557eb46104d7 in blk_aio_read_entry (opaque=0x7fc530003240) at +> +> > > ../block/block-backend.c:1619 +> +> > > #7 0x0000557eb47d6c40 in coroutine_trampoline (i0=-1213577040, +> +> > > i1=21886) at ../util/coroutine-ucontext.c:175 +> +> > > #8 0x00007fc547c2a360 in __start_context () at +> +> > > ../sysdeps/unix/sysv/linux/x86_64/__start_context.S:91 +> +> > > #9 0x00007ffd94b4ea40 in () +> +> > > #10 0x0000000000000000 in () +> +> > +> +> > So it looks like main thread is processing job-dismiss request and is +> +> > holding write lock taken in block_job_remove_all_bdrv() (frame #20 +> +> > above). At the same time iothread spawns a coroutine which performs IO +> +> > request. Before the coroutine is spawned, blk_aio_prwv() increases +> +> > 'in_flight' counter for Blk. Then blk_co_do_preadv_part() (frame #5) is +> +> > trying to acquire the read lock. But main thread isn't releasing the +> +> > lock as blk_root_drained_poll() returns true since blk->in_flight > 0. +> +> > Here's the deadlock. +> +> > +> +> > Any comments and suggestions on the subject are welcomed. Thanks! +> +> I think this is what the blk_wait_while_drained() call was supposed to +> +> address in blk_co_do_preadv_part(). However, with the use of multiple +> +> I/O threads, this is racy. +> +> +> +> Do you think that in your case we hit the small race window between the +> +> checks in blk_wait_while_drained() and GRAPH_RDLOCK_GUARD()? Or is there +> +> another reason why blk_wait_while_drained() didn't do its job? +> +> +> +At my opinion there is very big race window. Main thread has +> +eaten graph write lock. After that another coroutine is stalled +> +within GRAPH_RDLOCK_GUARD() as there is no drain at the moment and only +> +after that main thread has started drain. +You're right, I confused taking the write lock with draining there. + +> +That is why Fiona's idea is looking working. Though this would mean +> +that normally we should always do that at the moment when we acquire +> +write lock. May be even inside this function. +I actually see now that not all of my graph locking patches were merged. +At least I did have the thought that bdrv_drained_begin() must be marked +GRAPH_UNLOCKED because it polls. That means that calling it from inside +bdrv_try_change_aio_context() is actually forbidden (and that's the part +I didn't see back then because it doesn't have TSA annotations). + +If you refactor the code to move the drain out to before the lock is +taken, I think you end up with Fiona's patch, except you'll remove the +forbidden inner drain and add more annotations for some functions and +clarify the rules around them. I don't know, but I wouldn't be surprised +if along the process we find other bugs, too. + +So Fiona's drain looks right to me, but we should probably approach it +more systematically. + +Kevin + |