vhost-user: qemu stops processing packets under high load of traffic

Description of problem:
- qemu socket becomes full, causing qemu to send incomplete
SET_VRING_CALL messages to vhost-user backend (without proper fd set in
ancillary data).
- after some time, some interrupts are lost, causing the VM to stop
transmitting packets.

How reproducible:
Run a stress tests of a vhost-user interface using an UDP
traffic generator. Traffic generator (IXIA) was connected to 2 physical ports that are in turn connected to 2 virtio ports through a linux bridge, VM
(running linux) doing routing to forward packets between the 2 virtio ports.
When traffic reaches high pps rates of small packets,

Actual results:
- VM stop transmitting packets

Expected results:
- VM should never stop transmitting packets

Additional info:
We do propose a fix at:
  http://lists.nongnu.org/archive/html/qemu-devel/2015-12/msg00652.html

for tracking,
  http://git.qemu.org/?p=qemu.git;a=patch;h=5669655aafdb88a8797c74a989dd0c0ebb1349fa

On Fri, Mar 11, 2016 at 10:51:33PM -0000, Vincent JARDIN wrote:
> for tracking,
>   http://git.qemu.org/?p=qemu.git;a=patch;h=5669655aafdb88a8797c74a989dd0c0ebb1349fa
> 
> -- 
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1556306
> 
> Title:
>    vhost-user: qemu stops processing packets under high load of traffic
> 
> Status in QEMU:
>   New

I presume you'll also close this bu at some point?
It's fixed in upstream QEMU.

> Bug description:
>   Description of problem:
>   - qemu socket becomes full, causing qemu to send incomplete
>   SET_VRING_CALL messages to vhost-user backend (without proper fd set in
>   ancillary data).
>   - after some time, some interrupts are lost, causing the VM to stop
>   transmitting packets.
> 
>   How reproducible:
>   Run a stress tests of a vhost-user interface using an UDP
>   traffic generator. Traffic generator (IXIA) was connected to 2 physical ports that are in turn connected to 2 virtio ports through a linux bridge, VM
>   (running linux) doing routing to forward packets between the 2 virtio ports.
>   When traffic reaches high pps rates of small packets,
> 
>   Actual results:
>   - VM stop transmitting packets
> 
>   Expected results:
>   - VM should never stop transmitting packets
> 
>   Additional info:
>   We do propose a fix at:
>     http://lists.nongnu.org/archive/html/qemu-devel/2015-12/msg00652.html
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1556306/+subscriptions


Correct, it is fixed in Qemu upstream. Just need to get it used into my ubuntu.

Let's close it. Sorry, it should  be opened into:
  https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/

you can also add the project 'qemu-kvm' on the bug in order to get it into the ubuntu qemu-kvm bug list.

apologize but I was corrected that for qemu issues. The bug should be in the following:

Distribution: ubuntu
package: qemu  <--instead of project.

I will correct this in the bug.

Status changed to 'Confirmed' because the bug affects multiple users.

Thanks for reporting this bug.  I'll push into the xenial package today.

Side question, will you apply it to qemu-kvm from
  https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/mitaka-staging/+files/qemu-kvm_2.5+dfsg-5ubuntu5~cloud0_amd64.deb
too?

or should I open another bug?

This bug was fixed in the package qemu - 1:2.5+dfsg-5ubuntu6

---------------
qemu (1:2.5+dfsg-5ubuntu6) xenial; urgency=medium

  * Cherrypick upstream patch vhost-user-interrupt-management-fixes.patch
    (LP: #1556306)

 -- Serge Hallyn <email address hidden>  Wed, 16 Mar 2016 16:35:22 -0700

It should also be fixed in the qemu-kvm package. No additional bug needed as this bug covers both qemu and qemu-kvm packages. 

Sergey - any chance you can also push the patch into the qemu-kvm package?

it seems that the fix was not applied on ppc build:
   https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842754
   https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842753

neither on arm64:
  https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842750



Quoting Vincent JARDIN (vincent.jardin@6wind.com):
> it seems that the fix was not applied on ppc build:
>    https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842754
>    https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842753
> 
> neither on arm64:
>   https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-1ubuntu3/+build/8842750

Hi,

that is an old version.  See https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-5ubuntu6
and
https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-5ubuntu6/+build/9361116/+files/buildlog_ubuntu-xenial-arm64.qemu_1%3A2.5+dfsg-5ubuntu6_BUILDING.txt.gz


Great, thanks for your ack't of the update being available for ppc.

This is marked as affecting precise, but has anyone reproduced this with qemu-kvm 1.0+noroms-0ubuntu14.27 ?

The patch is completely inapplicable to that code base, so it would need to be rewritten from scratch if so.


(if someone says they have reproduced it on 1.0+noroms-0ubuntu14.27 I'll unmark it invalid.)


Actually even porting to trusty is complicated by a set of endianness patches.


Hello Vincent, or anyone else affected,

Accepted qemu into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.3+dfsg-5ubuntu9.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed.  Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed.  In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in advance!

The wily SRU has been waiting for validation for quite some time.  I'm wondering whether that is because noone is using wily, or because it's not high priority?

The patch does not apply cleanly to trusty.  In particular, the chunk in ./hw/net/vhost_net.c.rej is quite obsolete in the trusty source.  So I'd like to hear from someone that they are hitting this before risking an erroneous backport.


Hi,
cleaning up old issues.
In all the time we had no confirmed report on trusty, also as serge outlined in c#19 the backport would be much harder and therefore carry more risk for the SRU.
Since wily was haniging in verification so long and now is EOD this is dead.

I'm cleaning up the bug states to match that accordingly.