summary refs log tree commit diff stats
path: root/results/classifier/108/other/1831
diff options
context:
space:
mode:
Diffstat (limited to '')
-rw-r--r--results/classifier/108/other/183116
-rw-r--r--results/classifier/108/other/1831225508
-rw-r--r--results/classifier/108/other/183135444
-rw-r--r--results/classifier/108/other/183136251
-rw-r--r--results/classifier/108/other/183147736
-rw-r--r--results/classifier/108/other/183148661
-rw-r--r--results/classifier/108/other/183175094
7 files changed, 810 insertions, 0 deletions
diff --git a/results/classifier/108/other/1831 b/results/classifier/108/other/1831
new file mode 100644
index 000000000..134aba161
--- /dev/null
+++ b/results/classifier/108/other/1831
@@ -0,0 +1,16 @@
+device: 0.751
+performance: 0.641
+socket: 0.617
+debug: 0.491
+graphic: 0.472
+permissions: 0.398
+vnc: 0.317
+semantic: 0.252
+other: 0.200
+PID: 0.177
+boot: 0.153
+network: 0.085
+KVM: 0.082
+files: 0.012
+
+qemu-system-m68k: ../hw/scsi/scsi-disk.c:557: scsi_write_data: Assertion `r->req.aiocb == NULL' failed.
diff --git a/results/classifier/108/other/1831225 b/results/classifier/108/other/1831225
new file mode 100644
index 000000000..472170662
--- /dev/null
+++ b/results/classifier/108/other/1831225
@@ -0,0 +1,508 @@
+permissions: 0.821
+other: 0.801
+device: 0.781
+graphic: 0.770
+vnc: 0.769
+debug: 0.767
+performance: 0.765
+files: 0.742
+socket: 0.723
+semantic: 0.720
+boot: 0.719
+KVM: 0.712
+PID: 0.704
+network: 0.607
+
+guest migration 100% cpu freeze bug
+
+# Investigate migration cpu hog(100%) bug
+
+I have some issues when migrating from kernel 4.14.63 running qemu 2.11.2 to kernel 4.19.43 running qemu 2.11.2.
+The hypervisors are running on debian jessie with libvirt v5.3.0.
+Linux, libvirt and qemu are all custom compiled.
+
+I migrated around 10.000 vms and every once in a while a vm is stuck at 100% cpu after what we can see right now is that the target hypervisor runs on linux 4.19.53. This happened with 4 vms so far. It is not that easy to debug, we found this out pretty quickly because we are running monitoring on frozen vms after migrations.
+
+Last year we were having the same "kind of" bug https://bugs.launchpad.net/qemu/+bug/1775555 when trying to upgrade qemu 2.6 to 2.11.
+This bug was fixed after applying the following patch: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg00820.html
+
+This patch is still applied as you can see because of the available pre_load var on the kvmclock_vmsd struct:
+(gdb) ptype kvmclock_vmsd
+type = const struct VMStateDescription {
+    const char *name;
+    int unmigratable;
+    int version_id;
+    int minimum_version_id;
+    int minimum_version_id_old;
+    MigrationPriority priority;
+    LoadStateHandler *load_state_old;
+    int (*pre_load)(void *);                                                
+    int (*post_load)(void *, int);
+    int (*pre_save)(void *);
+    _Bool (*needed)(void *);
+    VMStateField *fields;
+    const VMStateDescription **subsections;
+}
+
+I attached gdb to a vcpu thread of one stuck vm, and a bt showed the following info:
+Thread 4 (Thread 0x7f3a431a4700 (LWP 37799)):
+#0  0x00007f3a576f5017 in ioctl () at ../sysdeps/unix/syscall-template.S:84
+#1  0x000055d84d15de57 in kvm_vcpu_ioctl (cpu=cpu@entry=0x55d84fca78d0, type=type@entry=44672) at /home/dbosschieter/src/qemu-pkg/src/accel/kvm/kvm-all.c:2050
+#2  0x000055d84d15dfc6 in kvm_cpu_exec (cpu=cpu@entry=0x55d84fca78d0) at /home/dbosschieter/src/qemu-pkg/src/accel/kvm/kvm-all.c:1887
+#3  0x000055d84d13ab64 in qemu_kvm_cpu_thread_fn (arg=0x55d84fca78d0) at /home/dbosschieter/src/qemu-pkg/src/cpus.c:1136
+#4  0x00007f3a579ba4a4 in start_thread (arg=0x7f3a431a4700) at pthread_create.c:456
+#5  0x00007f3a576fcd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
+
+Thread 3 (Thread 0x7f3a439a5700 (LWP 37798)):
+#0  0x00007f3a576f5017 in ioctl () at ../sysdeps/unix/syscall-template.S:84
+#1  0x000055d84d15de57 in kvm_vcpu_ioctl (cpu=cpu@entry=0x55d84fc5cbb0, type=type@entry=44672) at /home/dbosschieter/src/qemu-pkg/src/accel/kvm/kvm-all.c:2050
+#2  0x000055d84d15dfc6 in kvm_cpu_exec (cpu=cpu@entry=0x55d84fc5cbb0) at /home/dbosschieter/src/qemu-pkg/src/accel/kvm/kvm-all.c:1887
+#3  0x000055d84d13ab64 in qemu_kvm_cpu_thread_fn (arg=0x55d84fc5cbb0) at /home/dbosschieter/src/qemu-pkg/src/cpus.c:1136
+#4  0x00007f3a579ba4a4 in start_thread (arg=0x7f3a439a5700) at pthread_create.c:456
+#5  0x00007f3a576fcd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
+
+The ioctl call is a ioctl(18, KVM_RUN and it looks like it is looping inside the vm itself.
+
+I saved the state of the VM (with `virsh save`) after I found it was hanging on its vcpu threads. Then I restored this vm on a test environment running the same kernel, QEMU and libvirt version). After the restore the VM still was haning at 100% cpu usage on all the vcpus.
+I tried to use the perf kvm guest option to trace the guest vm with a copy of the kernel, modules and kallsyms files from inside the guest vm and I got to the following perf stat:
+
+ Event                                         Total %Total CurAvg/s
+ kvm_entry                                   5198993   23.1   277007
+ kvm_exit                                    5198976   23.1   277006
+ kvm_apic                                    1732103    7.7    92289
+ kvm_msr                                     1732101    7.7    92289
+ kvm_inj_virq                                1731904    7.7    92278
+ kvm_eoi                                     1731900    7.7    92278
+ kvm_apic_accept_irq                         1731900    7.7    92278
+ kvm_hv_timer_state                          1731780    7.7    92274
+ kvm_pv_eoi                                  1731701    7.7    92267
+ kvm_ple_window                                   36    0.0        2
+ Total                                      22521394         1199967
+
+We tried to run the crash tool against a dump of guest vm memory and that gave us the following backtrace:
+crash> bt
+PID: 0      TASK: ffffffff81610040  CPU: 0   COMMAND: "swapper/0"
+    [exception RIP: native_read_tsc+2]
+    RIP: ffffffff810146a9  RSP: ffff88003fc03df0  RFLAGS: 00000046
+    RAX: 000000008762c0fa  RBX: ffff88003fc13680  RCX: 0000000000000001
+    RDX: 0000000000fe4871  RSI: 0000000000000000  RDI: ffff88003fc13603
+    RBP: 000000000003052c   R8: 0000000000000200   R9: ffffffff8169b180
+    R10: 0000000000000020  R11: 0000000000000005  R12: 006a33290b40455c
+    R13: 00000000df1fd292  R14: 000000002ca284ff  R15: 00fe485f3febe21a
+    CS: 0010  SS: 0018
+ #0 [ffff88003fc03df0] pvclock_clocksource_read at ffffffff8102cbb3
+ #1 [ffff88003fc03e40] kvm_clock_read at ffffffff8102c2c9
+ #2 [ffff88003fc03e50] timekeeping_get_ns at ffffffff810691b0
+ #3 [ffff88003fc03e60] ktime_get at ffffffff810695c8
+ #4 [ffff88003fc03e90] sched_rt_period_timer at ffffffff8103e4f5
+ #5 [ffff88003fc03ee0] __run_hrtimer at ffffffff810652d3
+ #6 [ffff88003fc03f20] hrtimer_interrupt at ffffffff81065abd
+ #7 [ffff88003fc03f90] smp_apic_timer_interrupt at ffffffff81024ba8
+ #8 [ffff88003fc03fb0] apic_timer_interrupt at ffffffff813587e2
+--- <IRQ stack> ---
+ #9 [ffffffff81601e98] apic_timer_interrupt at ffffffff813587e2
+    [exception RIP: native_safe_halt+2]
+    RIP: ffffffff8102c360  RSP: ffffffff81601f40  RFLAGS: 00010246
+    RAX: 0000000000000000  RBX: ffffffff81601fd8  RCX: 00000000ffffffff
+    RDX: 00000000ffffffff  RSI: 0000000000000000  RDI: 0000000000000001
+    RBP: 0000000000000000   R8: 0000000000000000   R9: 0000000000000000
+    R10: 0000000000000020  R11: 0000000000000005  R12: ffffffff816f5d80
+    R13: ffffffffffffffff  R14: 000000000008c800  R15: 0000000000000000
+    ORIG_RAX: ffffffffffffff10  CS: 0010  SS: 0018
+#10 [ffffffff81601f40] default_idle at ffffffff81014c35
+#11 [ffffffff81601f50] cpu_idle at ffffffff8100d258
+
+So it seems like the vm is reading its clock constantly trying to catch up some time after the migration.
+
+Last time it was a bug that was only triggered on newer Gold cpu hardware, but this time we also see this coming back on older Intel E5 cpus we tried to reproduce with a migrate loop of 3 days times between kernel 4.14.63 and 4.19.43 but this gave us no results. 
+
+The vms were running ubuntu 14.04, centos 7, debian 7, debian 8 these vms are running linux kernel 3.*.
+
+The thing is that we are out of ideas for reproducing this, it seems like it the same kind of bug we are hitting, just like last time the vm is basically only trying to read the clock. Perhaps we can try to read the clock data and also try to read what the guest is actually waiting for, which value of the counter does it want to reach.
+
+I am not sure how to pinpoint the cause of this issue, I would like some help and possible some extra tips on debugging.
+We are able to read the guests kernel which makes it a bit easier to debug, reproducing and finding the source of the problem is still something we are trying to figure out.
+
+A virsh dumpxml of one of the guests:
+
+<domain type='kvm' id='15'>
+  <name>vps12</name>
+  <uuid>xxxxxxxx-953c-d629-1276-00000616xxxx</uuid>
+  <memory unit='KiB'>4194304</memory>
+  <currentMemory unit='KiB'>4194304</currentMemory>
+  <vcpu placement='static'>2</vcpu>
+  <numatune>
+    <memory mode='interleave' nodeset='0-1'/>
+  </numatune>
+  <resource>
+    <partition>/machine</partition>
+  </resource>
+  <os>
+    <type arch='x86_64' machine='pc-i440fx-2.11'>hvm</type>
+    <boot dev='hd'/>
+    <boot dev='network'/>
+  </os>
+  <features>
+    <acpi/>
+    <apic/>
+    <pae/>
+  </features>
+  <cpu mode='custom' match='exact' check='full'>
+    <model fallback='forbid'>Westmere</model>
+    <topology sockets='1' cores='2' threads='1'/>
+    <feature policy='require' name='vme'/>
+    <feature policy='require' name='pclmuldq'/>
+    <feature policy='require' name='x2apic'/>
+    <feature policy='require' name='hypervisor'/>
+    <feature policy='require' name='arat'/>
+  </cpu>
+  <clock offset='utc'>
+    <timer name='hpet' present='yes'/>
+  </clock>
+  <on_poweroff>destroy</on_poweroff>
+  <on_reboot>restart</on_reboot>
+  <on_crash>restart</on_crash>
+  <devices>
+    <emulator>/usr/bin/kvm</emulator>
+    <disk type='file' device='disk'>
+      <driver name='qemu' type='raw' cache='none'/>
+      <source file='/var/local/mnt/vps/vps12/vps12.raw'/>
+      <backingStore/>
+      <target dev='hda' bus='virtio'/>
+      <iotune>
+        <read_bytes_sec>734003200</read_bytes_sec>
+        <write_bytes_sec>576716800</write_bytes_sec>
+        <read_iops_sec>3500</read_iops_sec>
+        <write_iops_sec>2500</write_iops_sec>
+      </iotune>
+      <alias name='virtio-disk0'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
+    </disk>
+    <controller type='usb' index='0' model='piix3-uhci'>
+      <alias name='usb'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
+    </controller>
+    <controller type='pci' index='0' model='pci-root'>
+      <alias name='pci.0'/>
+    </controller>
+    <interface type='bridge'>
+      <mac address='xx:xx:xx:xx:xx:xx'/>
+      <source bridge='brxx'/>
+      <target dev='vxxxx'/>
+      <model type='virtio'/>
+      <filterref filter='firewall-vps12'/>
+      <link state='up'/>
+      <alias name='net0'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
+    </interface>
+    <input type='mouse' bus='ps2'>
+      <alias name='input0'/>
+    </input>
+    <input type='tablet' bus='usb'>
+      <alias name='input1'/>
+      <address type='usb' bus='0' port='1'/>
+    </input>
+    <input type='keyboard' bus='ps2'>
+      <alias name='input2'/>
+    </input>
+    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
+      <listen type='address' address='0.0.0.0'/>
+    </graphics>
+    <sound model='es1370'>
+      <alias name='sound0'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
+    </sound>
+    <video>
+      <model type='vga' vram='16384' heads='1' primary='yes'/>
+      <alias name='video0'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
+    </video>
+    <memballoon model='virtio'>
+      <alias name='balloon0'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
+    </memballoon>
+  </devices>
+  <seclabel type='dynamic' model='dac' relabel='yes'>
+    <label>+997:+998</label>
+    <imagelabel>+997:+998</imagelabel>
+  </seclabel>
+</domain>
+
+
+Hi Dion,
+  Since you've got a crash dump, can you check the dmesg in the guest to see if there's any messages?
+
+
+Hi Alan,
+
+
+Dmesg shows nothing special:
+[29891577.708544] IPv6 addrconf: prefix with wrong length 48
+[29891580.650637] IPv6 addrconf: prefix with wrong length 48
+[29891582.013656] IPv6 addrconf: prefix with wrong length 48
+[29891583.753246] IPv6 addrconf: prefix with wrong length 48
+[29891585.397941] IPv6 addrconf: prefix with wrong length 48
+[29891587.031141] IPv6 addrconf: prefix with wrong length 48
+[29891588.991541] IPv6 addrconf: prefix with wrong length 48
+[29891590.162395] IPv6 addrconf: prefix with wrong length 48
+[29891592.681133] IPv6 addrconf: prefix with wrong length 48
+[29891593.418342] IPv6 addrconf: prefix with wrong length 48
+[29891596.491791] IPv6 addrconf: prefix with wrong length 48
+[29891597.262282] IPv6 addrconf: prefix with wrong length 48
+[29891600.116510] IPv6 addrconf: prefix with wrong length 48
+[29891600.987599] IPv6 addrconf: prefix with wrong length 48
+[29891603.954923] IPv6 addrconf: prefix with wrong length 48
+[29891604.554989] IPv6 addrconf: prefix with wrong length 48
+[29891607.641694] IPv6 addrconf: prefix with wrong length 48
+[29891607.855495] IPv6 addrconf: prefix with wrong length 48
+[29891611.128803] IPv6 addrconf: prefix with wrong length 48
+[29891611.293230] IPv6 addrconf: prefix with wrong length 48
+[29891615.011260] IPv6 addrconf: prefix with wrong length 48
+[29891615.182883] IPv6 addrconf: prefix with wrong length 48
+[29891618.577801] IPv6 addrconf: prefix with wrong length 48
+[29891619.146512] IPv6 addrconf: prefix with wrong length 48
+[29891622.250595] IPv6 addrconf: prefix with wrong length 48
+[29891623.051844] IPv6 addrconf: prefix with wrong length 48
+[29891625.642684] IPv6 addrconf: prefix with wrong length 48
+[29891626.457455] IPv6 addrconf: prefix with wrong length 48
+
+Crash shows me the uptime though which doesn't seem right:
+        DATE: Tue Nov 29 18:55:46 2603
+      UPTIME: 106751 days, 23:55:03
+Not sure if this is related to the amount of kvm clock calls.
+
+Interesting; I'd seen something similar - in rh https://bugzilla.redhat.com/show_bug.cgi?id=1538078
+and as well as the bogus date we'd had lots of log messages of the form:
+
+  CE: lapic increasing min_delta_ns to <very big number> nsec
+
+
+we were reckoning the clock jumped a bit during the migrate, and then triggered an old guest kernel bug, https://bugzilla.redhat.com/show_bug.cgi?id=1183773  - and we'd only ever triggered it on older distros (rhel6 etc), and think we backported two newer kernel patches:
+
+commit 80a05b9ffa7dc13f6693902dd8999a2b61a3a0d7
+Author: Thomas Gleixner <email address hidden>
+Date:   Fri Mar 12 17:34:14 2010 +0100
+
+    clockevents: Sanitize min_delta_ns adjustment and prevent overflows
+
+and:
+commit d1748302f70be7469809809283fe164156a34231
+Author: Martin Schwidefsky <email address hidden>
+Date:   Tue Aug 23 15:29:42 2011 +0200
+
+    clockevents: Make minimum delay adjustments configurable
+
+but if you're getting that symptom on a RHEL7 / newer than 2011 kernel then I'm worried that there's another case.
+
+This was a very difficult to reproduce last time I looked at the rh bz's above - the slightest change anywhere and it would go away.
+
+
+Is there a way that we can check that it indeed is the case that the clock jumped a bit, we can try to read some kernel variables.
+We just got another hung guest's crash dump working, this vm also shows a weird uptime
+        DATE: Fri Dec 23 09:06:16 2603
+        UPTIME: 106752 days, 00:10:35
+
+Till now we have only see this happening on guests with linux kernel 3.*.
+
+Again dmesg shows nothing.
+
+The only way this has yet happened was when the target hypervisor kernel was 4.19.
+So it seems that something changed over there, the only thing is it not quite a small diff ;) between 4.14 and 4.19. And because it only happens once every couple of thousand migrations with unknown conditions makes it a hard to bisect this.
+Source hypervisor kernel could be 4.14 and 4.19.
+
+If the clock did jump during the migration, is there something that we could do about that on source/target hypervisor, with a libvirt xml change, etc, to prevent it I mean.
+
+It seems like the patch is applied to the guests source kernel.
+
+crash> clock_event_device
+struct clock_event_device {
+    void (*event_handler)(struct clock_event_device *);
+    int (*set_next_event)(unsigned long, struct clock_event_device *);
+    int (*set_next_ktime)(ktime_t, struct clock_event_device *);
+    ktime_t next_event;
+    u64 max_delta_ns;
+    u64 min_delta_ns;
+    u32 mult;
+    u32 shift;
+    enum clock_event_mode mode;
+    unsigned int features;
+    unsigned long retries;
+    void (*broadcast)(const struct cpumask *);
+    void (*set_mode)(enum clock_event_mode, struct clock_event_device *);
+    void (*suspend)(struct clock_event_device *);
+    void (*resume)(struct clock_event_device *);
+    unsigned long min_delta_ticks;
+    unsigned long max_delta_ticks;
+    const char *name;
+    int rating;
+    int irq;
+    const struct cpumask *cpumask;
+    struct list_head list;
+    struct module *owner;
+}
+SIZE: 0xc0
+crash> clockevents_program_min_delta
+clockevents_program_min_delta = $2 = 
+ {int (struct clock_event_device *)} 0xffffffff810daa20 <clockevents_program_min_delta>
+ 
+
+You say it's only happened since 4.19 - that's possible - but since this bug is so tricky to trigger it's also possible that any slight change in 4.19.
+
+You could try disabling kvm_clock?
+
+Dave
+
+Hi,
+
+i suffer fro this bug too (or very similar) on 4.15.0-50-generic, without the patch mentionned earlier (i use this patch last year to migrate from previous qemu version).
+
+Jean-Philippe
+
+Hi Jean,
+
+Could you elaborate, is it the qemu patch that you applied and didn't apply that to the current qem u version you are running?
+
+Could you try to get a crash dump from a frozen vm working, see if you get the same kind of backtrace in there.
+
+Which specific qemu version are you running, which cpu are you migrating from/to, which kernel version are you running from/to?
+
+Could you dump the xml in this comment section from a defined guest that was frozen?
+
+Dion
+
+Hi David,
+
+I have digged further into our issue and we have seen issues only when migrating from servers that have a different tsc frequency.
+
+Example:
+Source (kernel 4.14.63)
+[    2.068227] tsc: Refined TSC clocksource calibration: 2593.906 MHz
+[    2.068373] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2563bf907c6, max_idle_ns: 440795319401 ns                                                                            
+[    4.908200] clocksource: Switched to clocksource tsc
+
+Destination (kernel 4.19.43):
+[    0.000000] tsc: Detected 2600.000 MHz processor
+[    3.647553] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x257a3c3232d, max_idle_ns: 440795236700 ns                                                                      
+[    4.421582] clocksource: Switched to clocksource tsc-early
+[    5.747791] tsc: Refined TSC clocksource calibration: 2593.904 MHz
+[    5.747952] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2563bd843df, max_idle_ns: 440795257314 ns                                                                            
+[    5.748261] clocksource: Switched to clocksource tsc
+
+Source (kernel 4.19.43):
+[    0.000000] tsc: Fast TSC calibration using PIT
+[    0.000000] tsc: Detected 2199.852 MHz processor
+[    4.041367] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x1fb5a71147a, max_idle_ns: 440795225528 ns                                                                      
+[    4.504477] clocksource: Switched to clocksource tsc-early
+[    6.231559] tsc: Refined TSC clocksource calibration: 2200.002 MHz
+[    6.231718] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1fb634b8814, max_idle_ns: 440795202126 ns                                                                            
+[    6.801999] clocksource: Switched to clocksource tsc
+
+Destionation (kernel 4.19.43):
+[    0.000000] tsc: Fast TSC calibration using PIT
+[    0.000000] tsc: Detected 2394.538 MHz processor
+[    5.486287] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22840fdedc8, max_idle_ns: 440795293830 ns                                                                      
+[    6.182944] clocksource: Switched to clocksource tsc-early
+[    7.596489] tsc: Refined TSC clocksource calibration: 2394.454 MHz
+[    7.596641] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2283c0783fd, max_idle_ns: 440795301468 ns                                                                            
+[    9.316929] clocksource: Switched to clocksource tsc
+
+Source (kernel 4.14.63):
+[    2.068227] tsc: Refined TSC clocksource calibration: 2593.906 MHz
+[    2.068373] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2563bf907c6, max_idle_ns: 440795319401 ns                                                                            
+[    4.908200] clocksource: Switched to clocksource tsc
+
+Destination (kernel 4.19.43):
+[    0.000000] tsc: Detected 2600.000 MHz processor
+[    3.661033] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x257a3c3232d, max_idle_ns: 440795236700 ns                                                                      
+[    4.435033] clocksource: Switched to clocksource tsc-early
+[    5.761251] tsc: Refined TSC clocksource calibration: 2593.905 MHz
+[    5.761412] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2563be32fd6, max_idle_ns: 440795226961 ns                                                                            
+[    5.761731] clocksource: Switched to clocksource tsc
+
+And what I have seen from reading a bit about time  keeping(https://www.kernel.org/doc/Documentation/virtual/kvm/timekeeping.txt) and counters and the virtualisation of it, is that it is pretty difficult to get right, so it is already amazing that it goes alright most of the times.
+
+When diffing I stumbled upon this: https://lore.kernel.org/patchwork/patch/866471/
+And I was thinking about reverting those changes and then try another 10k of migrations see if that makes any difference as we still are not able to reproduce the issue.
+
+I could also try to prefer migrations from similiar refined TSC clocksource calibration HZes don't, which I would rather not do but I think I understand why having the exact same frequency is prefered, what is your thought on this matter?
+
+Disabling kvm-clock is not really an option as we don't want to restart and login on all of the running vms.
+
+Dion
+
+An update on our further research. We tried bumping the hypervisor kernel form 4.19.43 to 4.19.50 which included the following patch, which we hoped to be related to our issue:
+
+https://<email address hidden>/#t
+
+Sadly after a few thousand migrations we encountered two freezes again, and halted further migrations. Again the affected VMs seem to run pre-4.x kernels and all but one freezes occurred with Gold 6126 CPUs. 
+
+While analyzing memory dumps of the VMs with the crash utility we found a peculiar similarity. They all seem to have jumped ~584 years, which led me to this bug report from 2011: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/805341 . Does this provide any insight into what the issue might be on host-level?
+
+It is very well possible that the issue lies in the guest kernel, but as the service we provide is unmanaged we have no control over it.
+
+Could you try running the guests without the TSC_DEADLINE CPUID flag set?
+
+This round we built 4.19.55, disabled the kvm_intel.preemption_timer parameter and ensured kvm.lapic_timer_advance_ns is 0, as advised by Paolo Bonzini. Sadly, yet again we encountered a freeze.
+
+Any other suggestions?
+
+Hi, this is an update after some extended tests and a fallback migration to 4.14.
+
+After doing another >10k migrations we are sure to say that we also encounter this issue on kernel 4.14.
+We migrate vpses from servers in serial (one after the other) mode. And we notice that on some servers we encounter multiple freezes within a short period. These cases often occur in succession from the same source hypervisor, which for most (if not all) we've seen has a Gold CPU. So it seems like we are facing the same issue we were facing before in https://bugs.launchpad.net/qemu/+bug/1775555
+
+The thing is though, we applied the patches that supposedly "fixed" it last time, except that right now it seems like the issue is still there and that after applying the previous patch we are still having issues running on the same kernel with the same qemu version (v2.11.2).
+
+Are there any other changes lately that could perhaps have fixed it which are worth trying?
+We noticed that some pre migration checks were added to libvirt which would let the migration go into error if there are any severe mismatches between 2 hypervisor (hardware/kernel) configurations. 
+
+We could try to run that and see if these throw more errors which could point us into some direction, is this something that you would recommend?
+
+
+We have found a vm that recovered from a freeze and it seems like it has jumped in time.
+Below I have pasted a dump of tty1, it is ocr'd though so some characters could have been misinterpreted.
+
+hild 
+[13198552.767867] le-rss:010 Killed process 10374 (crop) total,r,4376400, anon-rss,018, tl [13230065.246354] lid Out of memory: Kill process 2009 Icron) score 1 or sacrifice c [13230065.251227] le-rss:OkB Killed process 19050 (cron) total-am:437040B, anon-r=0,42, fi [13230065.378536] Out of memory: Kill process 2003 (cron) score 1 or sacrifice 1132 Killed process 19622 Icron) total-vM:4376400, anon-rss:018, fi 
+le-rss:OkB 
+[14071563.261117] Out of memory: Kill process 2083 (cron) score 1 or sacrifice child
+[14071563.263500] Killed process 20260 (cron) total-vM:4376400, anon-rss:000, fi le-rss:OkB 
+[9223372037.903009] BUG: soft lockup - CPU#0 stuck for 4281394478s! [screen:4843]
+[9223372037.983809] Stack: 
+[9223372037.983809] 
+[9223372037.903009] Call Trace:
+[9223372037.983809] <IRQ>
+[922337203].983809] <EOI> Code: 83 c4 58 5b 5d c9 90 90 to Of al 9e 19 c0 c9 9c 50 66 .6 90 66 90 c9 57 9E .6 90 c9 f0 BO 67 00 66 66 90 66 90 fd c9 40 c7 07 c9 fa 66 00 00 00 66 90 00 40 66 66 c7 47 90 c9 fb 66 66 90 00 066: 
+
+This is VM is running Debian 7, it was approximately frozen for around 2 hours and 25 minutes before spinning back into action.
+
+The QEMU project is currently considering to move its bug tracking to
+another system. For this we need to know which bugs are still valid
+and which could be closed already. Thus we are setting older bugs to
+"Incomplete" now.
+
+If you still think this bug report here is valid, then please switch
+the state back to "New" within the next 60 days, otherwise this report
+will be marked as "Expired". Or please mark it as "Fix Released" if
+the problem has been solved with a newer version of QEMU already.
+
+Thank you and sorry for the inconvenience.
+
+Hi Thomas,
+
+We are still seeing this every once in a while. I can definitely tell you that it is connected to older Linux Guest kernels and we have not been able to identify a specific version which would make searching for a fix commit a bit easier.
+
+We are going to upgrade all our host kernels to 5.4 after which we will upgrade to a newer version of QEMU, temporarily working around this issue by selecting a specific set of hardware for migrations of older guest kernels.
+
+As time goes by the issue becomes less important as there will be less 3.* linux guest kernels running on our platform, but at the time of writing this is a significant amount.
+
+The guest kernels that we came across already had this patch applied: https://bugzilla.redhat.com/show_bug.cgi?id=1183773
+
+
+This is an automated cleanup. This bug report has been moved to QEMU's
+new bug tracker on gitlab.com and thus gets marked as 'expired' now.
+Please continue with the discussion here:
+
+ https://gitlab.com/qemu-project/qemu/-/issues/223
+
+
diff --git a/results/classifier/108/other/1831354 b/results/classifier/108/other/1831354
new file mode 100644
index 000000000..c42ffd0b1
--- /dev/null
+++ b/results/classifier/108/other/1831354
@@ -0,0 +1,44 @@
+device: 0.795
+performance: 0.680
+graphic: 0.632
+network: 0.493
+PID: 0.475
+boot: 0.421
+vnc: 0.358
+debug: 0.330
+semantic: 0.328
+other: 0.305
+permissions: 0.288
+socket: 0.212
+files: 0.172
+KVM: 0.171
+
+unable to read symlinks when mounting 9p filesystem with security_model=mapped
+
+I am trying to use clang that is mounted from a 9p filesystem that has the options 
+ -fsdev local,id=virtfs3,path=/clang,security_model=mapped-file -device virtio-9p-pci,fsdev=virtfs3,mount_tag=clang
+
+clang has symlinks to clang-9. eg /clang/clang/bin/clang is a symlink that points to clang-9 in the current directory. 
+
+the clang filesystem is on a bind mount point on /clang/clang on the host and this is mapped to the same place on the guest. If I have the same virtfs mount point with the security_model=none I don't have this problem.
+
+On the host: readlink clang
+clang-9
+
+and guest:
+
+ls: cannot read symbolic link 'clang': No such file or directory
+lrwxrwxrwx 1 root root 7 May 30 02:21 clang
+
+readlink clang
+
+returns nothing.
+
+
+This is an automated cleanup. This bug report has been moved to QEMU's
+new bug tracker on gitlab.com and thus gets marked as 'expired' now.
+Please continue with the discussion here:
+
+ https://gitlab.com/qemu-project/qemu/-/issues/173
+
+
diff --git a/results/classifier/108/other/1831362 b/results/classifier/108/other/1831362
new file mode 100644
index 000000000..71f788c65
--- /dev/null
+++ b/results/classifier/108/other/1831362
@@ -0,0 +1,51 @@
+device: 0.752
+graphic: 0.672
+vnc: 0.442
+other: 0.388
+performance: 0.335
+semantic: 0.316
+socket: 0.262
+network: 0.184
+boot: 0.179
+PID: 0.167
+permissions: 0.137
+debug: 0.088
+files: 0.050
+KVM: 0.032
+
+European keyboard PC-105 deadkey
+
+With a freshly compiled version of qemu 4.0.50 on Windows 10 (host)
+
+I am using 3 different Belgian keyboards and I have the same behaviour
+- 2 USB keyboards (Logitech and HP) and
+- the keyboard of my laptop (HP)
+
+3 characters on the same key cannot be used (the key seams to be dead):
+< (less than),
+> (greater than) used with the combination of LShift or RShift
+\ (backslash) used with the combination of AltGr
+
+Using grub command mode from an archlinux installation (5.1.4)
+The keyboard seams to be a mix of azerty and qwerty keyboard
+all letters are correctly mapped but all numbers and special
+characters are not
+
+Using sendkey in monitor
+"sendkey <" results in : \
+"sendkey shift-<" results in : |
+"sendkey ctrl-alt-<" results in : nothing
+
+REM: VirtualBox can handle this key and with the showkey command
+     from the archlinux kbd package, it shows :
+     keycode 86 press
+     keycode 86 release
+
+
+This is an automated cleanup. This bug report has been moved to QEMU's
+new bug tracker on gitlab.com and thus gets marked as 'expired' now.
+Please continue with the discussion here:
+
+ https://gitlab.com/qemu-project/qemu/-/issues/174
+
+
diff --git a/results/classifier/108/other/1831477 b/results/classifier/108/other/1831477
new file mode 100644
index 000000000..c9b86995a
--- /dev/null
+++ b/results/classifier/108/other/1831477
@@ -0,0 +1,36 @@
+device: 0.883
+graphic: 0.844
+network: 0.841
+socket: 0.799
+vnc: 0.798
+PID: 0.755
+files: 0.747
+semantic: 0.656
+boot: 0.613
+permissions: 0.603
+KVM: 0.474
+performance: 0.455
+debug: 0.422
+other: 0.393
+
+update edk2 submodule & binaries to edk2-stable201905
+
+The edk2 project will soon release edk2-stable201905. Update the edk2 submodule in QEMU, and the bundled edk2 binaries, accordingly.
+
+https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Release-Planning#edk2-stable201905-tag-planning
+https://github.com/tianocore/edk2/releases/tag/edk2-stable201905 [upcoming link]
+
+Some rebase notes can be seen at <https://bugzilla.redhat.com/show_bug.cgi?id=1701710#c12>.
+
+Posted
+[PATCH 0/6] update edk2 submodule & binaries to edk2-stable201905
+http://<email address hidden>
+
+
+[PULL 0/6] update edk2 submodule & binaries to edk2-stable201905
+http://<email address hidden>
+
+
+Merged in commit 53defa05701b ("Merge remote-tracking branch 'remotes/lersek/tags/edk2-pull-2019-06-14' into staging", 2019-06-17).
+
+
diff --git a/results/classifier/108/other/1831486 b/results/classifier/108/other/1831486
new file mode 100644
index 000000000..5f601ef2a
--- /dev/null
+++ b/results/classifier/108/other/1831486
@@ -0,0 +1,61 @@
+other: 0.903
+permissions: 0.859
+performance: 0.821
+graphic: 0.821
+semantic: 0.820
+KVM: 0.808
+files: 0.800
+debug: 0.790
+device: 0.777
+socket: 0.776
+PID: 0.763
+network: 0.755
+vnc: 0.732
+boot: 0.699
+
+qmp monitor deadlock (with spice events for ex)
+
+If an event is emitted during monitor_flush_locked() it will deadlock.
+
+Thread 1 (Thread 0x7f14f1854000 (LWP 7245)):
+#0  0x00007f14fc30592d in __lll_lock_wait () at /lib64/libpthread.so.0
+#1  0x00007f14fc2fedc9 in pthread_mutex_lock () at /lib64/libpthread.so.0
+#2  0x000055de60e19327 in qemu_mutex_lock_impl (mutex=0x55de61859e58, file=0x55de60f1a640 "/home/elmarco/src/qq/monitor.c", line=438) at /home/elmarco/src/qq/util/qemu-thread-posix.c:66
+#3  0x000055de6085c5af in monitor_puts (mon=0x55de61859d30, str=0x55de62a61d30 "{\"timestamp\": {\"seconds\": 1559585795, \"microseconds\": 508720}, \"event\": \"SPICE_DISCONNECTED\", \"data\": {\"server\": {\"port\": \"/tmp/.9IW52Z/spice.sock\", \"family\": \"unix\", \"host\": \"localhost\"}, \"client\": {"...) at /home/elmarco/src/qq/monitor.c:438
+#4  0x000055de6085c85a in qmp_send_response (mon=0x55de61859d30, rsp=0x55de61ed19a0) at /home/elmarco/src/qq/monitor.c:493
+#5  0x000055de6085c8ee in monitor_qapi_event_emit (event=QAPI_EVENT_SPICE_DISCONNECTED, qdict=0x55de61ed19a0) at /home/elmarco/src/qq/monitor.c:521
+#6  0x000055de6085c9ea in monitor_qapi_event_queue_no_reenter (event=QAPI_EVENT_SPICE_DISCONNECTED, qdict=0x55de61ed19a0) at /home/elmarco/src/qq/monitor.c:546
+#7  0x000055de6085cd7a in qapi_event_emit (event=QAPI_EVENT_SPICE_DISCONNECTED, qdict=0x55de61ed19a0) at /home/elmarco/src/qq/monitor.c:621
+#8  0x000055de60e04bc3 in qapi_event_send_spice_disconnected (server=0x55de61ee7b30, client=0x55de620c9090) at qapi/qapi-events-ui.c:101
+#9  0x000055de60c84381 in channel_event (event=3, info=0x55de6222f4c0) at /home/elmarco/src/qq/ui/spice-core.c:234
+#10 0x00007f14fc70ba3b in reds_handle_channel_event (reds=<optimized out>, event=3, info=0x55de6222f4c0) at reds.c:318
+#11 0x00007f14fc6f407b in main_dispatcher_self_handle_channel_event (info=0x55de6222f4c0, event=3, self=0x55de61a5b0b0) at main-dispatcher.c:191
+#12 0x00007f14fc6f407b in main_dispatcher_channel_event (self=0x55de61a5b0b0, event=event@entry=3, info=0x55de6222f4c0) at main-dispatcher.c:191
+#13 0x00007f14fc713cf3 in red_stream_push_channel_event (s=s@entry=0x55de6222f400, event=event@entry=3) at red-stream.c:416
+#14 0x00007f14fc713d2b in red_stream_free (s=0x55de6222f400) at red-stream.c:390
+#15 0x00007f14fc6fa67c in red_channel_client_finalize (object=0x55de62511360) at red-channel-client.c:347
+#16 0x00007f14fe4cfcf0 in g_object_unref () at /lib64/libgobject-2.0.so.0
+#17 0x00007f14fc6fca12 in red_channel_client_push (rcc=0x55de62511360) at red-channel-client.c:1340
+#18 0x00007f14fc6fca12 in red_channel_client_push (rcc=0x55de62511360) at red-channel-client.c:1303
+#19 0x00007f14fc6cd479 in red_char_device_send_msg_to_client (client=<optimized out>, msg=0x55de62512c00, dev=0x55de61a5b3b0) at char-device.c:307
+#20 0x00007f14fc6cd479 in red_char_device_send_msg_to_clients (msg=0x55de62512c00, dev=0x55de61a5b3b0) at char-device.c:307
+#21 0x00007f14fc6cd479 in red_char_device_read_from_device (dev=0x55de61a5b3b0) at char-device.c:355
+#22 0x000055de60a27dba in spice_chr_write (chr=0x55de61924c00, buf=0x55de6236c070 "{\"return\": {}, \"id\": 2}\r\n", len=25) at /home/elmarco/src/qq/chardev/spice.c:201
+#23 0x000055de60d89e29 in qemu_chr_write_buffer (s=0x55de61924c00, buf=0x55de6236c070 "{\"return\": {}, \"id\": 2}\r\n", len=25, offset=0x7ffcd5e1a860, write_all=false) at /home/elmarco/src/qq/chardev/char.c:113
+#24 0x000055de60d89f96 in qemu_chr_write (s=0x55de61924c00, buf=0x55de6236c070 "{\"return\": {}, \"id\": 2}\r\n", len=25, write_all=false) at /home/elmarco/src/qq/chardev/char.c:148
+#25 0x000055de60d8cf78 in qemu_chr_fe_write (be=0x55de61859d30, buf=0x55de6236c070 "{\"return\": {}, \"id\": 2}\r\n", len=25) at /home/elmarco/src/qq/chardev/char-fe.c:42
+#26 0x000055de6085c40f in monitor_flush_locked (mon=0x55de61859d30) at /home/elmarco/src/qq/monitor.c:404
+#27 0x000055de6085c614 in monitor_puts (mon=0x55de61859d30, str=0x55de622f6a40 "{\"return\": {}, \"id\": 2}\n") at /home/elmarco/src/qq/monitor.c:446
+#28 0x000055de6085c85a in qmp_send_response (mon=0x55de61859d30, rsp=0x55de61ecf960) at /home/elmarco/src/qq/monitor.c:493
+#29 0x000055de60865902 in monitor_qmp_respond (mon=0x55de61859d30, rsp=0x55de61ecf960) at /home/elmarco/src/qq/monitor.c:4128
+#30 0x000055de60865a19 in monitor_qmp_dispatch (mon=0x55de61859d30, req=0x55de622ec000) at /home/elmarco/src/qq/monitor.c:4157
+#31 0x000055de60865ce2 in monitor_qmp_bh_dispatcher (data=0x0) at /home/elmarco/src/qq/monitor.c:4224
+
+
+This is an automated cleanup. This bug report has been moved to QEMU's
+new bug tracker on gitlab.com and thus gets marked as 'expired' now.
+Please continue with the discussion here:
+
+ https://gitlab.com/qemu-project/qemu/-/issues/175
+
+
diff --git a/results/classifier/108/other/1831750 b/results/classifier/108/other/1831750
new file mode 100644
index 000000000..1830ae788
--- /dev/null
+++ b/results/classifier/108/other/1831750
@@ -0,0 +1,94 @@
+KVM: 0.727
+vnc: 0.693
+other: 0.662
+graphic: 0.646
+device: 0.611
+network: 0.605
+permissions: 0.597
+debug: 0.594
+performance: 0.579
+files: 0.559
+semantic: 0.556
+boot: 0.532
+socket: 0.515
+PID: 0.498
+
+virtual machine cpu soft lockup when qemu attach disk
+
+Hi,  I found a problem that virtual machine cpu soft lockup when I attach a disk to the vm in the case that
+
+backend storage network has a large delay or IO pressure is too large.
+
+1) The disk xml which I attached is:
+
+    <disk type='block' device='lun' rawio='yes'>
+      <driver name='qemu' type='raw' cache='none' io='native'/>
+      <source dev='/dev/mapper/360022a11000c1e0a0787c23a000001cb'/>
+      <backingStore/>
+      <target dev='sdb' bus='scsi'/>
+      <alias name='scsi0-0-1-0'/>
+      <address type='drive' controller='0' bus='0' target='1' unit='0'/>
+    </disk>
+
+2) The bt of qemu main thread:
+
+#0 0x0000ffff9d78402c in pread64 () from /lib64/libpthread.so.0
+#1 0x0000aaaace3357d8 in pread64 (__offset=0, __nbytes=4096, __buf=0xaaaad47a5200, __fd=202) at /usr/include/bits/unistd.h:99
+#2 raw_is_io_aligned (fd=fd@entry=202, buf=buf@entry=0xaaaad47a5200, len=len@entry=4096) at block/raw_posix.c:294
+#3 0x0000aaaace33597c in raw_probe_alignment (bs=bs@entry=0xaaaad32ea920, fd=202, errp=errp@entry=0xfffffef7a330) at block/raw_posix.c:349
+#4 0x0000aaaace335a48 in raw_refresh_limits (bs=0xaaaad32ea920, errp=0xfffffef7a330) at block/raw_posix.c:811
+#5 0x0000aaaace3404b0 in bdrv_refresh_limits (bs=0xaaaad32ea920, errp=0xfffffef7a330, errp@entry=0xfffffef7a360) at block/io.c:122
+#6 0x0000aaaace340504 in bdrv_refresh_limits (bs=bs@entry=0xaaaad09ce800, errp=errp@entry=0xfffffef7a3b0) at block/io.c:97
+#7 0x0000aaaace2eb9f0 in bdrv_open_common (bs=bs@entry=0xaaaad09ce800, file=file@entry=0xaaaad0e89800, options=<optimized out>, errp=errp@entry=0xfffffef7a450)
+at block.c:1194
+#8 0x0000aaaace2eedec in bdrv_open_inherit (filename=<optimized out>, filename@entry=0xaaaad25f92d0 "/dev/mapper/36384c4f100630193359db7a80000011d",
+reference=reference@entry=0x0, options=<optimized out>, options@entry=0xaaaad3d0f4b0, flags=<optimized out>, flags@entry=128, parent=parent@entry=0x0,
+child_role=child_role@entry=0x0, errp=errp@entry=0xfffffef7a710) at block.c:1895
+#9 0x0000aaaace2ef510 in bdrv_open (filename=filename@entry=0xaaaad25f92d0 "/dev/mapper/36384c4f100630193359db7a80000011d", reference=reference@entry=0x0,
+options=options@entry=0xaaaad3d0f4b0, flags=flags@entry=128, errp=errp@entry=0xfffffef7a710) at block.c:1979
+#10 0x0000aaaace331ef0 in blk_new_open (filename=filename@entry=0xaaaad25f92d0 "/dev/mapper/36384c4f100630193359db7a80000011d", reference=reference@entry=0x0,
+options=options@entry=0xaaaad3d0f4b0, flags=128, errp=errp@entry=0xfffffef7a710) at block/block_backend.c:213
+#11 0x0000aaaace0da1f4 in blockdev_init (file=file@entry=0xaaaad25f92d0 "/dev/mapper/36384c4f100630193359db7a80000011d", bs_opts=bs_opts@entry=0xaaaad3d0f4b0,
+errp=errp@entry=0xfffffef7a710) at blockdev.c:603
+#12 0x0000aaaace0dc478 in drive_new (all_opts=all_opts@entry=0xaaaad4dc31d0, block_default_type=<optimized out>) at blockdev.c:1116
+#13 0x0000aaaace0e3ee0 in add_init_drive (
+optstr=optstr@entry=0xaaaad0872ec0 "file=/dev/mapper/36384c4f100630193359db7a80000011d,format=raw,if=none,id=drive-scsi0-0-0-3,cache=none,aio=native")
+at device_hotplug.c:46
+#14 0x0000aaaace0e3f78 in hmp_drive_add (mon=0xfffffef7a810, qdict=0xaaaad0c8f000) at device_hotplug.c:67
+#15 0x0000aaaacdf7d688 in handle_hmp_command (mon=0xfffffef7a810, cmdline=<optimized out>) at /usr/src/debug/qemu-kvm-2.8.1/monitor.c:3199
+#16 0x0000aaaacdf7d778 in qmp_human_monitor_command (
+command_line=0xaaaacfc8e3c0 "drive_add dummy file=/dev/mapper/36384c4f100630193359db7a80000011d,format=raw,if=none,id=drive-scsi0-0-0-3,cache=none,aio=native",
+has_cpu_index=false, cpu_index=0, errp=errp@entry=0xfffffef7a968) at /usr/src/debug/qemu-kvm-2.8.1/monitor.c:660
+#17 0x0000aaaace0fdb30 in qmp_marshal_human_monitor_command (args=<optimized out>, ret=0xfffffef7a9e0, errp=0xfffffef7a9d8) at qmp-marshal.c:2223
+#18 0x0000aaaace3b6ad0 in do_qmp_dispatch (request=<optimized out>, errp=0xfffffef7aa20, errp@entry=0xfffffef7aa40) at qapi/qmp_dispatch.c:115
+#19 0x0000aaaace3b6d58 in qmp_dispatch (request=<optimized out>) at qapi/qmp_dispatch.c:142
+#20 0x0000aaaacdf79398 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-kvm-2.8.1/monitor.c:4010
+#21 0x0000aaaace3bd6c0 in json_message_process_token (lexer=0xaaaacf834c80, input=<optimized out>, type=JSON_RCURLY, x=214, y=274) at qobject/json_streamer.c:105
+#22 0x0000aaaace3f3d4c in json_lexer_feed_char (lexer=lexer@entry=0xaaaacf834c80, ch=<optimized out>, flush=flush@entry=false) at qobject/json_lexer.c:319
+#23 0x0000aaaace3f3e6c in json_lexer_feed (lexer=0xaaaacf834c80, buffer=<optimized out>, size=<optimized out>) at qobject/json_lexer.c:369
+#24 0x0000aaaacdf77c64 in monitor_qmp_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-kvm-2.8.1/monitor.c:4040
+#25 0x0000aaaace0eab18 in tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0xaaaacf90b280) at qemu_char.c:3260
+#26 0x0000ffff9dadf200 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
+#27 0x0000aaaace3c4a00 in glib_pollfds_poll () at util/main_loop.c:230
+--Type <RET> for more, q to quit, c to continue without paging--
+#28 0x0000aaaace3c4a88 in os_host_main_loop_wait (timeout=<optimized out>) at util/main_loop.c:278
+#29 0x0000aaaace3c4bf0 in main_loop_wait (nonblocking=<optimized out>) at util/main_loop.c:534
+#30 0x0000aaaace0f5d08 in main_loop () at vl.c:2120
+#31 0x0000aaaacdf3a770 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:5017
+
+
+From the bt we can see,  when do qmp sush as drive_add,  qemu main thread locks the qemu_global_mutex  and do pread in raw_probe_alignmen. Pread is a synchronous operation. If backend storage network has a large delay or IO pressure is too large,  the pread operation will not return for a long time, which make vcpu thread can't acquire qemu_global_mutex for a long time and make the vcpu thread unable to be scheduled for a long time.  So virtual machine cpu soft lockup happened.
+
+
+I thank  qemu main thread should not hold qemu_global_mutex for a long time when do qmp that involving IO synchronous operation sush pread , ioctl, etc.
+
+Do you have any solutions or good ideas about it? Thanks for your reply!
+
+
+This is an automated cleanup. This bug report has been moved to QEMU's
+new bug tracker on gitlab.com and thus gets marked as 'expired' now.
+Please continue with the discussion here:
+
+ https://gitlab.com/qemu-project/qemu/-/issues/176
+
+