other: 0.933 graphic: 0.844 semantic: 0.798 device: 0.787 boot: 0.719 KVM: 0.709 socket: 0.681 network: 0.666 vnc: 0.633 [Qemu-devel] [BUG] QEMU crashes with dpdk virtio pmd Qemu crashes, with pre-condition: vm xml config with multiqueue, and the vm's driver virtio-net support multi-queue reproduce steps: i. start dpdk testpmd in VM with the virtio nic ii. stop testpmd iii. reboot the VM This commit "f9d6dbf0 remove virtio queues if the guest doesn't support multiqueue" is introduced. Qemu version: QEMU emulator version 2.9.50 (v2.9.0-137-g32c7e0a) VM DPDK version: DPDK-1.6.1 Call Trace: #0 0x00007f60881fe5d7 in raise () from /usr/lib64/libc.so.6 #1 0x00007f60881ffcc8 in abort () from /usr/lib64/libc.so.6 #2 0x00007f608823e2f7 in __libc_message () from /usr/lib64/libc.so.6 #3 0x00007f60882456d3 in _int_free () from /usr/lib64/libc.so.6 #4 0x00007f608900158f in g_free () from /usr/lib64/libglib-2.0.so.0 #5 0x00007f6088fea32c in iter_remove_or_steal () from /usr/lib64/libglib-2.0.so.0 #6 0x00007f608edc0986 in object_property_del_all (obj=0x7f6091e74800) at qom/object.c:410 #7 object_finalize (data=0x7f6091e74800) at qom/object.c:467 #8 object_unref (address@hidden) at qom/object.c:903 #9 0x00007f608eaf1fd3 in phys_section_destroy (mr=0x7f6091e74800) at git/qemu/exec.c:1154 #10 phys_sections_free (map=0x7f6090b72bb0) at git/qemu/exec.c:1163 #11 address_space_dispatch_free (d=0x7f6090b72b90) at git/qemu/exec.c:2514 #12 0x00007f608ee91ace in call_rcu_thread (opaque=) at util/rcu.c:272 #13 0x00007f6089b0ddc5 in start_thread () from /usr/lib64/libpthread.so.0 #14 0x00007f60882bf71d in clone () from /usr/lib64/libc.so.6 Call Trace: #0 0x00007fdccaeb9790 in ?? () #1 0x00007fdcd82d09fc in object_property_del_all (obj=0x7fdcdb8acf60) at qom/object.c:405 #2 object_finalize (data=0x7fdcdb8acf60) at qom/object.c:467 #3 object_unref (address@hidden) at qom/object.c:903 #4 0x00007fdcd8001fd3 in phys_section_destroy (mr=0x7fdcdb8acf60) at git/qemu/exec.c:1154 #5 phys_sections_free (map=0x7fdcdc86aa00) at git/qemu/exec.c:1163 #6 address_space_dispatch_free (d=0x7fdcdc86a9e0) at git/qemu/exec.c:2514 #7 0x00007fdcd83a1ace in call_rcu_thread (opaque=) at util/rcu.c:272 #8 0x00007fdcd301ddc5 in start_thread () from /usr/lib64/libpthread.so.0 #9 0x00007fdcd17cf71d in clone () from /usr/lib64/libc.so.6 The q->tx_bh will free in virtio_net_del_queue() function, when remove virtio queues if the guest doesn't support multiqueue. But it might be still referenced by others (eg . virtio_net_set_status()), which need so set NULL. diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index 7d091c9..98bd683 100644 --- a/hw/net/virtio-net.c +++ b/hw/net/virtio-net.c @@ -1522,9 +1522,12 @@ static void virtio_net_del_queue(VirtIONet *n, int index) if (q->tx_timer) { timer_del(q->tx_timer); timer_free(q->tx_timer); + q->tx_timer = NULL; } else { qemu_bh_delete(q->tx_bh); + q->tx_bh = NULL; } + q->tx_waiting = 0; virtio_del_queue(vdev, index * 2 + 1); } From: wangyunjian Sent: Monday, April 24, 2017 6:10 PM To: address@hidden; Michael S. Tsirkin ; 'Jason Wang' Cc: wangyunjian ; caihe Subject: [Qemu-devel][BUG] QEMU crashes with dpdk virtio pmd Qemu crashes, with pre-condition: vm xml config with multiqueue, and the vm's driver virtio-net support multi-queue reproduce steps: i. start dpdk testpmd in VM with the virtio nic ii. stop testpmd iii. reboot the VM This commit "f9d6dbf0  remove virtio queues if the guest doesn't support multiqueue" is introduced. Qemu version: QEMU emulator version 2.9.50 (v2.9.0-137-g32c7e0a) VM DPDK version:  DPDK-1.6.1 Call Trace: #0  0x00007f60881fe5d7 in raise () from /usr/lib64/libc.so.6 #1  0x00007f60881ffcc8 in abort () from /usr/lib64/libc.so.6 #2  0x00007f608823e2f7 in __libc_message () from /usr/lib64/libc.so.6 #3  0x00007f60882456d3 in _int_free () from /usr/lib64/libc.so.6 #4  0x00007f608900158f in g_free () from /usr/lib64/libglib-2.0.so.0 #5  0x00007f6088fea32c in iter_remove_or_steal () from /usr/lib64/libglib-2.0.so.0 #6  0x00007f608edc0986 in object_property_del_all (obj=0x7f6091e74800) at qom/object.c:410 #7  object_finalize (data=0x7f6091e74800) at qom/object.c:467 #8  object_unref (address@hidden) at qom/object.c:903 #9  0x00007f608eaf1fd3 in phys_section_destroy (mr=0x7f6091e74800) at git/qemu/exec.c:1154 #10 phys_sections_free (map=0x7f6090b72bb0) at git/qemu/exec.c:1163 #11 address_space_dispatch_free (d=0x7f6090b72b90) at git/qemu/exec.c:2514 #12 0x00007f608ee91ace in call_rcu_thread (opaque=) at util/rcu.c:272 #13 0x00007f6089b0ddc5 in start_thread () from /usr/lib64/libpthread.so.0 #14 0x00007f60882bf71d in clone () from /usr/lib64/libc.so.6 Call Trace: #0  0x00007fdccaeb9790 in ?? () #1  0x00007fdcd82d09fc in object_property_del_all (obj=0x7fdcdb8acf60) at qom/object.c:405 #2  object_finalize (data=0x7fdcdb8acf60) at qom/object.c:467 #3  object_unref (address@hidden) at qom/object.c:903 #4  0x00007fdcd8001fd3 in phys_section_destroy (mr=0x7fdcdb8acf60) at git/qemu/exec.c:1154 #5  phys_sections_free (map=0x7fdcdc86aa00) at git/qemu/exec.c:1163 #6  address_space_dispatch_free (d=0x7fdcdc86a9e0) at git/qemu/exec.c:2514 #7  0x00007fdcd83a1ace in call_rcu_thread (opaque=) at util/rcu.c:272 #8  0x00007fdcd301ddc5 in start_thread () from /usr/lib64/libpthread.so.0 #9  0x00007fdcd17cf71d in clone () from /usr/lib64/libc.so.6 On 2017年04月25日 19:37, wangyunjian wrote: The q->tx_bh will free in virtio_net_del_queue() function, when remove virtio queues if the guest doesn't support multiqueue. But it might be still referenced by others (eg . virtio_net_set_status()), which need so set NULL. diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index 7d091c9..98bd683 100644 --- a/hw/net/virtio-net.c +++ b/hw/net/virtio-net.c @@ -1522,9 +1522,12 @@ static void virtio_net_del_queue(VirtIONet *n, int index) if (q->tx_timer) { timer_del(q->tx_timer); timer_free(q->tx_timer); + q->tx_timer = NULL; } else { qemu_bh_delete(q->tx_bh); + q->tx_bh = NULL; } + q->tx_waiting = 0; virtio_del_queue(vdev, index * 2 + 1); } Thanks a lot for the fix. Two questions: - If virtio_net_set_status() is the only function that may access tx_bh, it looks like setting tx_waiting to zero is sufficient? - Can you post a formal patch for this? Thanks From: wangyunjian Sent: Monday, April 24, 2017 6:10 PM To: address@hidden; Michael S. Tsirkin ; 'Jason Wang' Cc: wangyunjian ; caihe Subject: [Qemu-devel][BUG] QEMU crashes with dpdk virtio pmd Qemu crashes, with pre-condition: vm xml config with multiqueue, and the vm's driver virtio-net support multi-queue reproduce steps: i. start dpdk testpmd in VM with the virtio nic ii. stop testpmd iii. reboot the VM This commit "f9d6dbf0 remove virtio queues if the guest doesn't support multiqueue" is introduced. Qemu version: QEMU emulator version 2.9.50 (v2.9.0-137-g32c7e0a) VM DPDK version: DPDK-1.6.1 Call Trace: #0 0x00007f60881fe5d7 in raise () from /usr/lib64/libc.so.6 #1 0x00007f60881ffcc8 in abort () from /usr/lib64/libc.so.6 #2 0x00007f608823e2f7 in __libc_message () from /usr/lib64/libc.so.6 #3 0x00007f60882456d3 in _int_free () from /usr/lib64/libc.so.6 #4 0x00007f608900158f in g_free () from /usr/lib64/libglib-2.0.so.0 #5 0x00007f6088fea32c in iter_remove_or_steal () from /usr/lib64/libglib-2.0.so.0 #6 0x00007f608edc0986 in object_property_del_all (obj=0x7f6091e74800) at qom/object.c:410 #7 object_finalize (data=0x7f6091e74800) at qom/object.c:467 #8 object_unref (address@hidden) at qom/object.c:903 #9 0x00007f608eaf1fd3 in phys_section_destroy (mr=0x7f6091e74800) at git/qemu/exec.c:1154 #10 phys_sections_free (map=0x7f6090b72bb0) at git/qemu/exec.c:1163 #11 address_space_dispatch_free (d=0x7f6090b72b90) at git/qemu/exec.c:2514 #12 0x00007f608ee91ace in call_rcu_thread (opaque=) at util/rcu.c:272 #13 0x00007f6089b0ddc5 in start_thread () from /usr/lib64/libpthread.so.0 #14 0x00007f60882bf71d in clone () from /usr/lib64/libc.so.6 Call Trace: #0 0x00007fdccaeb9790 in ?? () #1 0x00007fdcd82d09fc in object_property_del_all (obj=0x7fdcdb8acf60) at qom/object.c:405 #2 object_finalize (data=0x7fdcdb8acf60) at qom/object.c:467 #3 object_unref (address@hidden) at qom/object.c:903 #4 0x00007fdcd8001fd3 in phys_section_destroy (mr=0x7fdcdb8acf60) at git/qemu/exec.c:1154 #5 phys_sections_free (map=0x7fdcdc86aa00) at git/qemu/exec.c:1163 #6 address_space_dispatch_free (d=0x7fdcdc86a9e0) at git/qemu/exec.c:2514 #7 0x00007fdcd83a1ace in call_rcu_thread (opaque=) at util/rcu.c:272 #8 0x00007fdcd301ddc5 in start_thread () from /usr/lib64/libpthread.so.0 #9 0x00007fdcd17cf71d in clone () from /usr/lib64/libc.so.6 CCing Paolo and Stefan, since it has a relationship with bh in Qemu. > -----Original Message----- > From: Jason Wang [ mailto:address@hidden > > > On 2017年04月25日 19:37, wangyunjian wrote: > > The q->tx_bh will free in virtio_net_del_queue() function, when remove > > virtio > queues > > if the guest doesn't support multiqueue. But it might be still referenced by > others (eg . virtio_net_set_status()), > > which need so set NULL. > > > > diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c > > index 7d091c9..98bd683 100644 > > --- a/hw/net/virtio-net.c > > +++ b/hw/net/virtio-net.c > > @@ -1522,9 +1522,12 @@ static void virtio_net_del_queue(VirtIONet *n, > int index) > > if (q->tx_timer) { > > timer_del(q->tx_timer); > > timer_free(q->tx_timer); > > + q->tx_timer = NULL; > > } else { > > qemu_bh_delete(q->tx_bh); > > + q->tx_bh = NULL; > > } > > + q->tx_waiting = 0; > > virtio_del_queue(vdev, index * 2 + 1); > > } > > Thanks a lot for the fix. > > Two questions: > > - If virtio_net_set_status() is the only function that may access tx_bh, > it looks like setting tx_waiting to zero is sufficient? Currently yes, but we don't assure that it works for all scenarios, so we set the tx_bh and tx_timer to NULL to avoid to possibly access wild pointer, which is the common method for usage of bh in Qemu. I have another question about the root cause of this issure. This below trace is the path of setting tx_waiting to one in virtio_net_handle_tx_bh() : Breakpoint 1, virtio_net_handle_tx_bh (vdev=0x0, vq=0x7f335ad13900) at /data/wyj/git/qemu/hw/net/virtio-net.c:1398 1398 { (gdb) bt #0 virtio_net_handle_tx_bh (vdev=0x0, vq=0x7f335ad13900) at /data/wyj/git/qemu/hw/net/virtio-net.c:1398 #1 0x00007f3357bddf9c in virtio_bus_set_host_notifier (bus=, address@hidden, address@hidden) at hw/virtio/virtio-bus.c:297 #2 0x00007f3357a0055d in vhost_dev_disable_notifiers (address@hidden, address@hidden) at /data/wyj/git/qemu/hw/virtio/vhost.c:1422 #3 0x00007f33579e3373 in vhost_net_stop_one (net=0x7f335ad84dc0, dev=0x7f335c6f5f90) at /data/wyj/git/qemu/hw/net/vhost_net.c:289 #4 0x00007f33579e385b in vhost_net_stop (address@hidden, ncs=, address@hidden) at /data/wyj/git/qemu/hw/net/vhost_net.c:367 #5 0x00007f33579e15de in virtio_net_vhost_status (status=, n=0x7f335c6f5f90) at /data/wyj/git/qemu/hw/net/virtio-net.c:176 #6 virtio_net_set_status (vdev=0x7f335c6f5f90, status=0 '\000') at /data/wyj/git/qemu/hw/net/virtio-net.c:250 #7 0x00007f33579f8dc6 in virtio_set_status (address@hidden, address@hidden '\000') at /data/wyj/git/qemu/hw/virtio/virtio.c:1146 #8 0x00007f3357bdd3cc in virtio_ioport_write (val=0, addr=18, opaque=0x7f335c6eda80) at hw/virtio/virtio-pci.c:387 #9 virtio_pci_config_write (opaque=0x7f335c6eda80, addr=18, val=0, size=) at hw/virtio/virtio-pci.c:511 #10 0x00007f33579b2155 in memory_region_write_accessor (mr=0x7f335c6ee470, addr=18, value=, size=1, shift=, mask=, attrs=...) at /data/wyj/git/qemu/memory.c:526 #11 0x00007f33579af2e9 in access_with_adjusted_size (address@hidden, address@hidden, address@hidden, access_size_min=, access_size_max=, address@hidden 0x7f33579b20f0 , address@hidden, address@hidden) at /data/wyj/git/qemu/memory.c:592 #12 0x00007f33579b2e15 in memory_region_dispatch_write (address@hidden, address@hidden, data=0, address@hidden, address@hidden) at /data/wyj/git/qemu/memory.c:1319 #13 0x00007f335796cd93 in address_space_write_continue (mr=0x7f335c6ee470, l=1, addr1=18, len=1, buf=0x7f335773d000 "", attrs=..., addr=49170, as=0x7f3358317060 ) at /data/wyj/git/qemu/exec.c:2834 #14 address_space_write (as=, addr=, attrs=..., buf=, len=) at /data/wyj/git/qemu/exec.c:2879 #15 0x00007f335796d3ad in address_space_rw (as=, address@hidden, attrs=..., address@hidden, buf=, address@hidden, address@hidden) at /data/wyj/git/qemu/exec.c:2981 #16 0x00007f33579ae226 in kvm_handle_io (count=1, size=1, direction=, data=, attrs=..., port=49170) at /data/wyj/git/qemu/kvm-all.c:1803 #17 kvm_cpu_exec (address@hidden) at /data/wyj/git/qemu/kvm-all.c:2032 #18 0x00007f335799b632 in qemu_kvm_cpu_thread_fn (arg=0x7f335ae82070) at /data/wyj/git/qemu/cpus.c:1118 #19 0x00007f3352983dc5 in start_thread () from /usr/lib64/libpthread.so.0 #20 0x00007f335113571d in clone () from /usr/lib64/libc.so.6 It calls qemu_bh_schedule(q->tx_bh) at the bottom of virtio_net_handle_tx_bh(), I don't know why virtio_net_tx_bh() doesn't be invoked, so that the q->tx_waiting is not zero. [ps: we added logs in virtio_net_tx_bh() to verify that] Some other information: It won't crash if we don't use vhost-net. Thanks, -Gonglei > - Can you post a formal patch for this? > > Thanks > > > From: wangyunjian > > Sent: Monday, April 24, 2017 6:10 PM > > To: address@hidden; Michael S. Tsirkin ; 'Jason > Wang' > > Cc: wangyunjian ; caihe > > Subject: [Qemu-devel][BUG] QEMU crashes with dpdk virtio pmd > > > > Qemu crashes, with pre-condition: > > vm xml config with multiqueue, and the vm's driver virtio-net support > multi-queue > > > > reproduce steps: > > i. start dpdk testpmd in VM with the virtio nic > > ii. stop testpmd > > iii. reboot the VM > > > > This commit "f9d6dbf0 remove virtio queues if the guest doesn't support > multiqueue" is introduced. > > > > Qemu version: QEMU emulator version 2.9.50 (v2.9.0-137-g32c7e0a) > > VM DPDK version: DPDK-1.6.1 > > > > Call Trace: > > #0 0x00007f60881fe5d7 in raise () from /usr/lib64/libc.so.6 > > #1 0x00007f60881ffcc8 in abort () from /usr/lib64/libc.so.6 > > #2 0x00007f608823e2f7 in __libc_message () from /usr/lib64/libc.so.6 > > #3 0x00007f60882456d3 in _int_free () from /usr/lib64/libc.so.6 > > #4 0x00007f608900158f in g_free () from /usr/lib64/libglib-2.0.so.0 > > #5 0x00007f6088fea32c in iter_remove_or_steal () from > /usr/lib64/libglib-2.0.so.0 > > #6 0x00007f608edc0986 in object_property_del_all (obj=0x7f6091e74800) > at qom/object.c:410 > > #7 object_finalize (data=0x7f6091e74800) at qom/object.c:467 > > #8 object_unref (address@hidden) at qom/object.c:903 > > #9 0x00007f608eaf1fd3 in phys_section_destroy (mr=0x7f6091e74800) at > git/qemu/exec.c:1154 > > #10 phys_sections_free (map=0x7f6090b72bb0) at git/qemu/exec.c:1163 > > #11 address_space_dispatch_free (d=0x7f6090b72b90) at > git/qemu/exec.c:2514 > > #12 0x00007f608ee91ace in call_rcu_thread (opaque=) at > util/rcu.c:272 > > #13 0x00007f6089b0ddc5 in start_thread () from /usr/lib64/libpthread.so.0 > > #14 0x00007f60882bf71d in clone () from /usr/lib64/libc.so.6 > > > > Call Trace: > > #0 0x00007fdccaeb9790 in ?? () > > #1 0x00007fdcd82d09fc in object_property_del_all (obj=0x7fdcdb8acf60) at > qom/object.c:405 > > #2 object_finalize (data=0x7fdcdb8acf60) at qom/object.c:467 > > #3 object_unref (address@hidden) at qom/object.c:903 > > #4 0x00007fdcd8001fd3 in phys_section_destroy (mr=0x7fdcdb8acf60) at > git/qemu/exec.c:1154 > > #5 phys_sections_free (map=0x7fdcdc86aa00) at git/qemu/exec.c:1163 > > #6 address_space_dispatch_free (d=0x7fdcdc86a9e0) at > git/qemu/exec.c:2514 > > #7 0x00007fdcd83a1ace in call_rcu_thread (opaque=) at > util/rcu.c:272 > > #8 0x00007fdcd301ddc5 in start_thread () from /usr/lib64/libpthread.so.0 > > #9 0x00007fdcd17cf71d in clone () from /usr/lib64/libc.so.6 > > > > On 25/04/2017 14:02, Jason Wang wrote: > > Thanks a lot for the fix. > > Two questions: > > - If virtio_net_set_status() is the only function that may access tx_bh, > it looks like setting tx_waiting to zero is sufficient? I think clearing tx_bh is better anyway, as leaving a dangling pointer is not very hygienic. Paolo > - Can you post a formal patch for this?