1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
|
semantic: 0.963
graphic: 0.952
assembly: 0.945
other: 0.943
socket: 0.931
instruction: 0.922
boot: 0.914
device: 0.911
vnc: 0.883
mistranslation: 0.882
network: 0.876
KVM: 0.874
vhost-user: qemu stops processing packets under high load of traffic
Description of problem:
- qemu socket becomes full, causing qemu to send incomplete
SET_VRING_CALL messages to vhost-user backend (without proper fd set in
ancillary data).
- after some time, some interrupts are lost, causing the VM to stop
transmitting packets.
How reproducible:
Run a stress tests of a vhost-user interface using an UDP
traffic generator. Traffic generator (IXIA) was connected to 2 physical ports that are in turn connected to 2 virtio ports through a linux bridge, VM
(running linux) doing routing to forward packets between the 2 virtio ports.
When traffic reaches high pps rates of small packets,
Actual results:
- VM stop transmitting packets
Expected results:
- VM should never stop transmitting packets
Additional info:
We do propose a fix at:
http://lists.nongnu.org/archive/html/qemu-devel/2015-12/msg00652.html
for tracking,
http://git.qemu.org/?p=qemu.git;a=patch;h=5669655aafdb88a8797c74a989dd0c0ebb1349fa
On Fri, Mar 11, 2016 at 10:51:33PM -0000, Vincent JARDIN wrote:
> for tracking,
> http://git.qemu.org/?p=qemu.git;a=patch;h=5669655aafdb88a8797c74a989dd0c0ebb1349fa
>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1556306
>
> Title:
> vhost-user: qemu stops processing packets under high load of traffic
>
> Status in QEMU:
> New
I presume you'll also close this bu at some point?
It's fixed in upstream QEMU.
> Bug description:
> Description of problem:
> - qemu socket becomes full, causing qemu to send incomplete
> SET_VRING_CALL messages to vhost-user backend (without proper fd set in
> ancillary data).
> - after some time, some interrupts are lost, causing the VM to stop
> transmitting packets.
>
> How reproducible:
> Run a stress tests of a vhost-user interface using an UDP
> traffic generator. Traffic generator (IXIA) was connected to 2 physical ports that are in turn connected to 2 virtio ports through a linux bridge, VM
> (running linux) doing routing to forward packets between the 2 virtio ports.
> When traffic reaches high pps rates of small packets,
>
> Actual results:
> - VM stop transmitting packets
>
> Expected results:
> - VM should never stop transmitting packets
>
> Additional info:
> We do propose a fix at:
> http://lists.nongnu.org/archive/html/qemu-devel/2015-12/msg00652.html
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1556306/+subscriptions
Correct, it is fixed in Qemu upstream. Just need to get it used into my ubuntu.
Let's close it. Sorry, it should be opened into:
https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/
you can also add the project 'qemu-kvm' on the bug in order to get it into the ubuntu qemu-kvm bug list.
apologize but I was corrected that for qemu issues. The bug should be in the following:
Distribution: ubuntu
package: qemu <--instead of project.
I will correct this in the bug.
Status changed to 'Confirmed' because the bug affects multiple users.
Thanks for reporting this bug. I'll push into the xenial package today.
Side question, will you apply it to qemu-kvm from
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/mitaka-staging/+files/qemu-kvm_2.5+dfsg-5ubuntu5~cloud0_amd64.deb
too?
or should I open another bug?
This bug was fixed in the package qemu - 1:2.5+dfsg-5ubuntu6
---------------
qemu (1:2.5+dfsg-5ubuntu6) xenial; urgency=medium
* Cherrypick upstream patch vhost-user-interrupt-management-fixes.patch
(LP: #1556306)
-- Serge Hallyn <email address hidden> Wed, 16 Mar 2016 16:35:22 -0700
It should also be fixed in the qemu-kvm package. No additional bug needed as this bug covers both qemu and qemu-kvm packages.
Sergey - any chance you can also push the patch into the qemu-kvm package?
it seems that the fix was not applied on ppc build:
https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842754
https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842753
neither on arm64:
https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842750
Quoting Vincent JARDIN (vincent.jardin@6wind.com):
> it seems that the fix was not applied on ppc build:
> https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842754
> https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842753
>
> neither on arm64:
> https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842750
Hi,
that is an old version. See https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-5ubuntu6
and
https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-5ubuntu6/+build/9361116/+files/buildlog_ubuntu-xenial-arm64.qemu_1%3A2.5+dfsg-5ubuntu6_BUILDING.txt.gz
Great, thanks for your ack't of the update being available for ppc.
This is marked as affecting precise, but has anyone reproduced this with qemu-kvm 1.0+noroms-0ubuntu14.27 ?
The patch is completely inapplicable to that code base, so it would need to be rewritten from scratch if so.
(if someone says they have reproduced it on 1.0+noroms-0ubuntu14.27 I'll unmark it invalid.)
Actually even porting to trusty is complicated by a set of endianness patches.
Hello Vincent, or anyone else affected,
Accepted qemu into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.3+dfsg-5ubuntu9.3 in a few hours, and then in the -proposed repository.
Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.
Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!
The wily SRU has been waiting for validation for quite some time. I'm wondering whether that is because noone is using wily, or because it's not high priority?
The patch does not apply cleanly to trusty. In particular, the chunk in ./hw/net/vhost_net.c.rej is quite obsolete in the trusty source. So I'd like to hear from someone that they are hitting this before risking an erroneous backport.
Hi,
cleaning up old issues.
In all the time we had no confirmed report on trusty, also as serge outlined in c#19 the backport would be much harder and therefore carry more risk for the SRU.
Since wily was haniging in verification so long and now is EOD this is dead.
I'm cleaning up the bug states to match that accordingly.
|