1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
|
mistranslation: 0.717
risc-v: 0.706
VMM: 0.701
user-level: 0.696
permissions: 0.686
device: 0.685
debug: 0.668
register: 0.667
semantic: 0.663
TCG: 0.663
files: 0.659
vnc: 0.648
architecture: 0.641
KVM: 0.640
assembly: 0.639
socket: 0.636
network: 0.635
arm: 0.632
kernel: 0.628
PID: 0.628
virtual: 0.625
x86: 0.621
i386: 0.620
performance: 0.615
hypervisor: 0.609
graphic: 0.605
ppc: 0.601
peripherals: 0.590
boot: 0.564
QEMU fails during migration and reports "qemu: VQ 0 size 0x80 Guest index 0x2d6 inconsistent with Host index 0x18: delta 0x2be"
This issue is reproducing consistently on automated testing, verified on manual testing (although it may require many tries).
Steps to reproduce:
1) Start a linux guest. The command line used by automated testing was:
10/05 06:48:27 INFO | kvm_vm:1605| MALLOC_PERTURB_=1 /usr/local/autotest/tests/kvm/qemu
10/05 06:48:27 INFO | kvm_vm:1605| -S
10/05 06:48:27 INFO | kvm_vm:1605| -name 'vm1'
10/05 06:48:27 INFO | kvm_vm:1605| -nodefaults
10/05 06:48:27 INFO | kvm_vm:1605| -chardev socket,id=hmp_id_humanmonitor1,path=/tmp/monitor-humanmonitor1-20121005-062311-r6UwQhzg,server,nowait
10/05 06:48:27 INFO | kvm_vm:1605| -mon chardev=hmp_id_humanmonitor1,mode=readline
10/05 06:48:27 INFO | kvm_vm:1605| -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20121005-062311-r6UwQhzg,server,nowait
10/05 06:48:27 INFO | kvm_vm:1605| -mon chardev=qmp_id_qmpmonitor1,mode=control
10/05 06:48:27 INFO | kvm_vm:1605| -chardev socket,id=serial_id_20121005-062311-r6UwQhzg,path=/tmp/serial-20121005-062311-r6UwQhzg,server,nowait
10/05 06:48:27 INFO | kvm_vm:1605| -device isa-serial,chardev=serial_id_20121005-062311-r6UwQhzg
10/05 06:48:27 INFO | kvm_vm:1605| -chardev socket,id=seabioslog_id_20121005-062311-r6UwQhzg,path=/tmp/seabios-20121005-062311-r6UwQhzg,server,nowait
10/05 06:48:27 INFO | kvm_vm:1605| -device isa-debugcon,chardev=seabioslog_id_20121005-062311-r6UwQhzg,iobase=0x402
10/05 06:48:27 INFO | kvm_vm:1605| -device ich9-usb-uhci1,id=usb1
10/05 06:48:27 INFO | kvm_vm:1605| -drive file='/tmp/kvm_autotest_root/images/rhel62-64.qcow2',if=none,cache=none,id=virtio0
10/05 06:48:27 INFO | kvm_vm:1605| -device virtio-blk-pci,drive=virtio0
10/05 06:48:27 INFO | kvm_vm:1605| -device virtio-net-pci,netdev=idbdMz8N,mac='9a:cf:d0:d1:d2:d3',id='ida0Kc7l'
10/05 06:48:27 INFO | kvm_vm:1605| -netdev tap,id=idbdMz8N,fd=21
10/05 06:48:27 INFO | kvm_vm:1605| -m 2048
10/05 06:48:27 INFO | kvm_vm:1605| -smp 2,cores=1,threads=1,sockets=2
10/05 06:48:27 INFO | kvm_vm:1605| -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1
10/05 06:48:27 INFO | kvm_vm:1605| -vnc :0
10/05 06:48:27 INFO | kvm_vm:1605| -vga std
10/05 06:48:27 INFO | kvm_vm:1605| -rtc base=utc,clock=host,driftfix=none
10/05 06:48:27 INFO | kvm_vm:1605| -boot order=cdn,once=c,menu=off
10/05 06:48:27 INFO | kvm_vm:1605| -enable-kvm
Start a new VM in incoming mode. The example on this bug is using TCP protocol:
10/05 07:18:56 INFO | kvm_vm:1605| MALLOC_PERTURB_=1 /usr/local/autotest/tests/kvm/qemu
10/05 07:18:56 INFO | kvm_vm:1605| -S
10/05 07:18:56 INFO | kvm_vm:1605| -name 'vm1'
10/05 07:18:56 INFO | kvm_vm:1605| -nodefaults
10/05 07:18:56 INFO | kvm_vm:1605| -chardev socket,id=hmp_id_humanmonitor1,path=/tmp/monitor-humanmonitor1-20121005-071855-5QYsCtRS,server,nowait
10/05 07:18:56 INFO | kvm_vm:1605| -mon chardev=hmp_id_humanmonitor1,mode=readline
10/05 07:18:56 INFO | kvm_vm:1605| -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20121005-071855-5QYsCtRS,server,nowait
10/05 07:18:56 INFO | kvm_vm:1605| -mon chardev=qmp_id_qmpmonitor1,mode=control
10/05 07:18:56 INFO | kvm_vm:1605| -chardev socket,id=serial_id_20121005-071855-5QYsCtRS,path=/tmp/serial-20121005-071855-5QYsCtRS,server,nowait
10/05 07:18:56 INFO | kvm_vm:1605| -device isa-serial,chardev=serial_id_20121005-071855-5QYsCtRS
10/05 07:18:56 INFO | kvm_vm:1605| -chardev socket,id=seabioslog_id_20121005-071855-5QYsCtRS,path=/tmp/seabios-20121005-071855-5QYsCtRS,server,nowait
10/05 07:18:56 INFO | kvm_vm:1605| -device isa-debugcon,chardev=seabioslog_id_20121005-071855-5QYsCtRS,iobase=0x402
10/05 07:18:56 INFO | kvm_vm:1605| -device ich9-usb-uhci1,id=usb1
10/05 07:18:56 INFO | kvm_vm:1605| -drive file='/tmp/kvm_autotest_root/images/rhel62-64.qcow2',if=none,cache=none,id=virtio0
10/05 07:18:56 INFO | kvm_vm:1605| -device virtio-blk-pci,drive=virtio0
10/05 07:18:56 INFO | kvm_vm:1605| -device virtio-net-pci,netdev=idERNnUO,mac='9a:cf:d0:d1:d2:d3',id='ideI7zfw'
10/05 07:18:56 INFO | kvm_vm:1605| -netdev tap,id=idERNnUO,fd=32
10/05 07:18:56 INFO | kvm_vm:1605| -m 2048
10/05 07:18:56 INFO | kvm_vm:1605| -smp 2,cores=1,threads=1,sockets=2
10/05 07:18:56 INFO | kvm_vm:1605| -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1
10/05 07:18:56 INFO | kvm_vm:1605| -vnc :1
10/05 07:18:56 INFO | kvm_vm:1605| -vga std
10/05 07:18:56 INFO | kvm_vm:1605| -rtc base=utc,clock=host,driftfix=none
10/05 07:18:56 INFO | kvm_vm:1605| -boot order=cdn,once=c,menu=off
10/05 07:18:56 INFO | kvm_vm:1605| -enable-kvm
10/05 07:18:56 INFO | kvm_vm:1605| -incoming tcp:0:5200
Start the migration, typing on monitor
10/05 07:18:58 DEBUG|kvm_monito:0177| (monitor humanmonitor1) Sending command 'migrate -d tcp:0:5200'
The VM will start migrating state to the new qemu instance, and at some point it will stop with:
10/05 07:19:10 INFO | aexpect:0786| [qemu output] qemu: VQ 0 size 0x80 Guest index 0x2d6 inconsistent with Host index 0x18: delta 0x2be
10/05 07:19:10 INFO | aexpect:0786| [qemu output] qemu: warning: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk'
10/05 07:19:10 INFO | aexpect:0786| [qemu output] load of migration failed
10/05 07:19:10 INFO | aexpect:0786| [qemu output] (Process terminated with status 0)
Due to the large number of migrations executed during a virt job (vm state keeps being passed back and forth many times, using many protocols), we get this problem on every single virt job.
Latest commits we found this issue:
Kernel (kvm.git, avi's tree)
10/05 05:38:12 INFO | git:0153| git commit ID is 1a95620f45155ac523cd1419d89150fbb4eb858b (tag kvm-3.6-2-136-g1a95620)
Userspace (qemu.git)
10/05 06:20:30 INFO | git:0153| git commit ID is a14c74928ba1fdaada515717f4d3c3fa3275d6f7 (tag v1.2.0-546-ga14c749)
Have you bisected it already?
No, I haven't. I'll do my best to reserve time this week to do so.
FWIW this one bites me. The bug report seems fairly abandoned ... has this been fixed elsewhere perhaps?
I meant "bites me too" ...
Yep, I did let this one slip...
In any case, I don't see this problem anymore, it was fixed a long while ago. Are you trying QEMU from the latest master?
Closing this ticket now, since it has been fixed according to comment #5.
|