1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
x86: 0.958
user-level: 0.952
operating system: 0.952
mistranslation: 0.947
peripherals: 0.947
vnc: 0.946
risc-v: 0.944
TCG: 0.941
permissions: 0.939
VMM: 0.938
graphic: 0.937
debug: 0.935
semantic: 0.933
register: 0.930
virtual: 0.928
KVM: 0.928
performance: 0.927
i386: 0.919
ppc: 0.916
alpha: 0.916
hypervisor: 0.916
arm: 0.913
device: 0.911
assembly: 0.910
architecture: 0.906
files: 0.900
PID: 0.899
kernel: 0.896
socket: 0.892
network: 0.864
boot: 0.845
[BUG] Qemu abort with error "kvm_mem_ioeventfd_add: error adding ioeventfd: File exists (17)"
Hi list,
When I did some tests in my virtual domain with live-attached virtio deivces, I
got a coredump file of Qemu.
The error print from qemu is "kvm_mem_ioeventfd_add: error adding ioeventfd:
File exists (17)".
And the call trace in the coredump file displays as below:
#0 0x0000ffff89acecc8 in ?? () from /usr/lib64/libc.so.6
#1 0x0000ffff89a8acbc in raise () from /usr/lib64/libc.so.6
#2 0x0000ffff89a78d2c in abort () from /usr/lib64/libc.so.6
#3 0x0000aaaabd7ccf1c in kvm_mem_ioeventfd_add (listener=<optimized out>,
section=<optimized out>, match_data=<optimized out>, data=<optimized out>,
e=<optimized out>) at ../accel/kvm/kvm-all.c:1607
#4 0x0000aaaabd6e0304 in address_space_add_del_ioeventfds (fds_old_nb=164,
fds_old=0xffff5c80a1d0, fds_new_nb=160, fds_new=0xffff5c565080,
as=0xaaaabdfa8810 <address_space_memory>)
at ../softmmu/memory.c:795
#5 address_space_update_ioeventfds (as=0xaaaabdfa8810 <address_space_memory>)
at ../softmmu/memory.c:856
#6 0x0000aaaabd6e24d8 in memory_region_commit () at ../softmmu/memory.c:1113
#7 0x0000aaaabd6e25c4 in memory_region_transaction_commit () at
../softmmu/memory.c:1144
#8 0x0000aaaabd394eb4 in pci_bridge_update_mappings
(br=br@entry=0xaaaae755f7c0) at ../hw/pci/pci_bridge.c:248
#9 0x0000aaaabd394f4c in pci_bridge_write_config (d=0xaaaae755f7c0,
address=44, val=<optimized out>, len=4) at ../hw/pci/pci_bridge.c:272
#10 0x0000aaaabd39a928 in rp_write_config (d=0xaaaae755f7c0, address=44,
val=128, len=4) at ../hw/pci-bridge/pcie_root_port.c:39
#11 0x0000aaaabd6df328 in memory_region_write_accessor (mr=0xaaaae63898d0,
addr=65580, value=<optimized out>, size=4, shift=<optimized out>,
mask=<optimized out>, attrs=...) at ../softmmu/memory.c:494
#12 0x0000aaaabd6dcb6c in access_with_adjusted_size (addr=addr@entry=65580,
value=value@entry=0xffff817adc78, size=size@entry=4, access_size_min=<optimized
out>, access_size_max=<optimized out>,
access_fn=access_fn@entry=0xaaaabd6df284 <memory_region_write_accessor>,
mr=mr@entry=0xaaaae63898d0, attrs=attrs@entry=...) at ../softmmu/memory.c:556
#13 0x0000aaaabd6e0dc8 in memory_region_dispatch_write
(mr=mr@entry=0xaaaae63898d0, addr=65580, data=<optimized out>, op=MO_32,
attrs=attrs@entry=...) at ../softmmu/memory.c:1534
#14 0x0000aaaabd6d0574 in flatview_write_continue (fv=fv@entry=0xffff5c02da00,
addr=addr@entry=275146407980, attrs=attrs@entry=...,
ptr=ptr@entry=0xffff8aa8c028, len=len@entry=4,
addr1=<optimized out>, l=<optimized out>, mr=mr@entry=0xaaaae63898d0) at
/usr/src/debug/qemu-6.2.0-226.aarch64/include/qemu/host-utils.h:165
#15 0x0000aaaabd6d4584 in flatview_write (len=4, buf=0xffff8aa8c028, attrs=...,
addr=275146407980, fv=0xffff5c02da00) at ../softmmu/physmem.c:3375
#16 address_space_write (as=<optimized out>, addr=275146407980, attrs=...,
buf=buf@entry=0xffff8aa8c028, len=4) at ../softmmu/physmem.c:3467
#17 0x0000aaaabd6d462c in address_space_rw (as=<optimized out>, addr=<optimized
out>, attrs=..., attrs@entry=..., buf=buf@entry=0xffff8aa8c028, len=<optimized
out>, is_write=<optimized out>)
at ../softmmu/physmem.c:3477
#18 0x0000aaaabd7cf6e8 in kvm_cpu_exec (cpu=cpu@entry=0xaaaae625dfd0) at
../accel/kvm/kvm-all.c:2970
#19 0x0000aaaabd7d09bc in kvm_vcpu_thread_fn (arg=arg@entry=0xaaaae625dfd0) at
../accel/kvm/kvm-accel-ops.c:49
#20 0x0000aaaabd94ccd8 in qemu_thread_start (args=<optimized out>) at
../util/qemu-thread-posix.c:559
By printing more info in the coredump file, I found that the addr of
fds_old[146] and fds_new[146] are same, but fds_old[146] belonged to a
live-attached virtio-scsi device while fds_new[146] was owned by another
live-attached virtio-net.
The reason why addr conflicted was then been found from vm's console log. Just
before qemu aborted, the guest kernel crashed and kdump.service booted the
dump-capture kernel where re-alloced address for the devices.
Because those virtio devices were live-attached after vm creating, different
addr may been assigned to them in the dump-capture kernel:
the initial kernel booting log:
[ 1.663297] pci 0000:00:02.1: BAR 14: assigned [mem 0x11900000-0x11afffff]
[ 1.664560] pci 0000:00:02.1: BAR 15: assigned [mem
0x8001800000-0x80019fffff 64bit pref]
the dump-capture kernel booting log:
[ 1.845211] pci 0000:00:02.0: BAR 14: assigned [mem 0x11900000-0x11bfffff]
[ 1.846542] pci 0000:00:02.0: BAR 15: assigned [mem
0x8001800000-0x8001afffff 64bit pref]
I think directly aborting the qemu process may not be the best choice in this
case cuz it will interrupt the work of kdump.service so that failed to generate
memory dump of the crashed guest kernel.
Perhaps, IMO, the error could be simply ignored in this case and just let kdump
to reboot the system after memory-dump finishing, but I failed to find a
suitable judgment in the codes.
Any solution for this problem? Hope I can get some helps here.
Hao
|