MTU not honored in virtio vnet I am observing a potential regression/different behavior between rel. 14.04 (dev branch) and release 13.04. My hardware is: Cisco UCS blade B200-M3 and the network adapter card: Cisco UCS VIC 1240. lsb_release -rd Description: Ubuntu Trusty Tahr (development branch) Release: 14.04 uname -a Linux konan2 3.13.0-19-generic #40-Ubuntu SMP Mon Mar 24 02:36:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux The problem: After starting a kvm with virtio interfaces I am passing HTTP traffic via an external network traffic simulator. The tool is sending a TCP packet of 3481B, because the tool MTU is set to 1400B, it splits the packets into 3 TCP segments. When the 3 segments are received at the host eth1 interface, the host (ubuntu 14.04) reassembles the TCP packets into a larger packet (GRO), then passes the packet up on vnet1. At this point, because vnet1 MTU is 1500B, it is supposed to re-segment the packet and pass the 3 segments up to the VM. But it passes the big 3481B packet instead. This behavior did not happen when I tried the same scenario in release 13.04 I can disable this behavior by disabling TSO (TCP segment offloading in the vnet), but I did not have to do this in rel. 13.04 and I feel the MTU is not honored as it should be with rel. 14.04. ip link show | grep eth1 3: eth1: mtu 1500 qdisc mq master br1 state UP mode DEFAULT group default qlen 1000 ip link show | grep vnet1 16: vnet1: mtu 1500 qdisc pfifo_fast master br1 state UNKNOWN mode DEFAULT group default qlen 500 I am attaching two tcpdump/pcap traces that show a TCP transaction passing on vnet1 when TSO is on and when TSO is off. Please see: - vnet1_tso_on.pcap - vnet1_tso_off.pcap in attachment. I noticed there was a driver upgrade in rel. 14.04: in 14.04: ethtool -i eth1 driver: enic version: 2.1.1.50 firmware-version: 2.1(3a) bus-info: 0000:07:00.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no ethtool -i vnet1 driver: tun version: 1.6 firmware-version: bus-info: tap supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no ethtool -k vnet1 Features for vnet1: rx-checksumming: off [fixed] tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: on tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off tx-tcp6-segmentation: off udp-fragmentation-offload: on generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: on ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: on loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: on rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] in 13.04 : ethtool -i eth1 driver: enic version: 2.1.1.39 firmware-version: 2.1(3a) bus-info: 0000:07:00.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no ethtool -i vnet1 driver: tun version: 1.6 firmware-version: bus-info: tap supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no ethtool -k vnet1 Features for vnet1: rx-checksumming: off [fixed] tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: on tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: on tx-tcp6-segmentation: on udp-fragmentation-offload: on generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] fcoe-mtu: off [fixed] Thanks for reporting this bug. I wonder if this may have been fixed since 1.7 - Are you able to reproduce this with the qemu version in ppa:ubuntu-virt/candidate? (That is the version which we hope will end up in 14.04 before release) I have updated to 1.7 but I still see the same issue. kvm --version QEMU emulator version 1.7.0 (Debian 1.7.0+dfsg-3ubuntu7), Copyright (c) 2003-2008 Fabrice Bellard drivers appears to be the same though ethtool -i eth1 driver: enic version: 2.1.1.50 firmware-version: 2.1(3a) bus-info: 0000:07:00.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no ethtool -i vnet1 driver: tun version: 1.6 firmware-version: bus-info: tap supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no --- I can definitely see my VM receiving packets bigger than 1500B tcpdump .... .... Wednesday March 26 2014 INBOUND>>>>> 19:21:31:127 [ver:4 len:3481 proto:ether hop:0 t1:none vlan:0x010b] 802.1Q vlan#2067 P0 192.188.67.15.80 > 101.8.11.217.32103: P [bad tcp cksum 1df9!] 122145215:122148626(3411) ack 120888245 win 843 (DF) (ttl 64, id 61505, len 3463) Wednesday March 26 2014 INBOUND>>>>> 19:21:31:127 [ver:4 len:2133 proto:ether hop:0 t1:none vlan:0x010b] 802.1Q vlan#2067 P0 192.188.67.17.80 > 101.8.15.65.23222: P [bad tcp cksum e6f3!] 112090738:112092801(2063) ack 107977615 win 843 (DF) (ttl 64, id 28330, len 2115) .... ip link show dev vnet1 16: vnet1: mtu 1500 qdisc mq master br1 state UNKNOWN mode DEFAULT group default qlen 2500 link/ether fe:54:00:f9:ea:e4 brd ff:ff:ff:ff:ff:ff Thanks, but that looks like the version currently in 14.04 archive. Could you test with the 2.0~git-20140325.7b770c7-0ubuntu1 package which is in ppa:ubuntu-virt/candidate? I can see the following two installed, what am I missing? sudo apt-cache policy qemu qemu: Installed: 2.0~git-20140325.7b770c7-0ubuntu1 Candidate: 2.0~git-20140325.7b770c7-0ubuntu1 Version table: *** 2.0~git-20140325.7b770c7-0ubuntu1 0 500 http://ppa.launchpad.net/ubuntu-virt/candidate/ubuntu/ trusty/main amd64 Packages 100 /var/lib/dpkg/status 1.7.0+dfsg-3ubuntu7 0 500 http://archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages sudo apt-cache policy qemu-kvm qemu-kvm: Installed: 2.0~git-20140325.7b770c7-0ubuntu1 Candidate: 2.0~git-20140325.7b770c7-0ubuntu1 Version table: *** 2.0~git-20140325.7b770c7-0ubuntu1 0 500 http://ppa.launchpad.net/ubuntu-virt/candidate/ubuntu/ trusty/main amd64 Packages 100 /var/lib/dpkg/status 1.7.0+dfsg-3ubuntu7 0 500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages Ok, sorry, in comment #4 you had said you were on 1.7. There are some netdev commits between 1.7 and 2.0 so before marking this as affecting upstream I wanted to make sure you'd tested git head (which you effectively have). Status changed to 'Confirmed' because the bug affects multiple users. Also affected by this bug with Ubuntu 12.04LTS kvm. A tracepath in a VM with virtio network card ends with: Resume: pmtu 65535. While with pcnet or rtl8139 it properly ends with: Resume: pmtu 1500 This is pretty blocking given some servers are failing to communicate because of the wrong MTU (I've the case with a mail server). The bug description said this was a regression from 13.04 to 14.04, but comment #9 says 12.04 is affected. Pierre, could you show us your kernel and qemu versions? dpkg -l | egrep "(linux|qemu)" ii libselinux1 2.1.0-4.1ubuntu1 SELinux runtime shared libraries ii linux-firmware 1.79.12 Firmware for Linux kernel drivers ii linux-headers-3.2.0-60 3.2.0-60.91 Header files related to Linux kernel version 3.2.0 ii linux-headers-3.2.0-60-generic 3.2.0-60.91 Linux kernel headers for version 3.2.0 on 64 bit x86 SMP ii linux-headers-3.2.0-61 3.2.0-61.92 Header files related to Linux kernel version 3.2.0 ii linux-headers-3.2.0-61-generic 3.2.0-61.92 Linux kernel headers for version 3.2.0 on 64 bit x86 SMP ii linux-headers-server 3.2.0.61.72 Linux kernel headers on Server Equipment. rc linux-image-2.6.32-21-server 2.6.32-21.32 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-22-server 2.6.32-22.36 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-24-server 2.6.32-24.43 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-25-server 2.6.32-25.45 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-26-server 2.6.32-26.48 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-27-server 2.6.32-27.49 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-29-server 2.6.32-29.58 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-30-server 2.6.32-30.59 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-33-server 2.6.32-33.72 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-34-server 2.6.32-34.77 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-35-server 2.6.32-35.78 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-37-server 2.6.32-37.81 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-38-server 2.6.32-38.83 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-39-server 2.6.32-39.86 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-40-server 2.6.32-40.87 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-2.6.32-41-server 2.6.32-41.88 Linux kernel image for version 2.6.32 on x86_64 rc linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-25-generic 3.2.0-25.40 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-26-generic 3.2.0-26.41 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-27-generic 3.2.0-27.43 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-29-generic 3.2.0-29.46 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-30-generic 3.2.0-30.48 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-31-generic 3.2.0-31.50 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-32-generic 3.2.0-32.51 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-33-generic 3.2.0-33.52 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-34-generic 3.2.0-34.53 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-35-generic 3.2.0-35.55 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-36-generic 3.2.0-36.57 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-37-generic 3.2.0-37.58 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-38-generic 3.2.0-38.61 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-39-generic 3.2.0-39.62 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-40-generic 3.2.0-40.64 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-41-generic 3.2.0-41.66 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-43-generic 3.2.0-43.68 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-44-generic 3.2.0-44.69 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-45-generic 3.2.0-45.70 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-48-generic 3.2.0-48.74 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-49-generic 3.2.0-49.75 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-51-generic 3.2.0-51.77 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-52-generic 3.2.0-52.78 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-53-generic 3.2.0-53.81 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-54-generic 3.2.0-54.82 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-55-generic 3.2.0-55.85 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-56-generic 3.2.0-56.86 Linux kernel image for version 3.2.0 on 64 bit x86 SMP rc linux-image-3.2.0-57-generic 3.2.0-57.87 Linux kernel image for version 3.2.0 on 64 bit x86 SMP ii linux-image-3.2.0-58-generic 3.2.0-58.88 Linux kernel image for version 3.2.0 on 64 bit x86 SMP ii linux-image-3.2.0-59-generic 3.2.0-59.90 Linux kernel image for version 3.2.0 on 64 bit x86 SMP ii linux-image-3.2.0-60-generic 3.2.0-60.91 Linux kernel image for version 3.2.0 on 64 bit x86 SMP ii linux-image-3.2.0-61-generic 3.2.0-61.92 Linux kernel image for version 3.2.0 on 64 bit x86 SMP ii linux-image-3.8.0-33-generic 3.8.0-33.48~precise1 Linux kernel image for version 3.8.0 on 64 bit x86 SMP ii linux-image-server 3.2.0.61.72 Linux kernel image on Server Equipment. ii linux-libc-dev 3.2.0-61.92 Linux Kernel Headers for development ii linux-server 3.2.0.61.72 Complete Linux kernel on Server Equipment. ii qemu-common 1.0+noroms-0ubuntu14.14 qemu common functionality (bios, documentation, etc) ii qemu-kvm 1.0+noroms-0ubuntu14.14 Full virtualization on i386 and amd64 hardware ii qemu-utils 1.0+noroms-0ubuntu14.14 qemu utilities ii util-linux 2.20.1-1ubuntu3 Miscellaneous system utilities And atm, running : root@goulash ~ # uname -a Linux goulash 3.8.0-33-generic #48~precise1-Ubuntu SMP Thu Oct 24 16:28:06 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux I believe this bug may be breaking Salt by breaking ZeroMQ in certain virtualized environments. Additional information here: https://github.com/saltstack/salt/issues/12248 Seems to me that the issue is not that the MTU isn't being honoured, but that the MTU should be checked *before* TSO assembly and not *after*. Assembly should happen outside the VM if the VM has enabled it on the interface internally, and clearly the incoming segments (not packets, at this point) may be longer than the MTU. Did anybody ever tried to reproduce this bug with upstream QEMU? Closing for QEMU since there hasn't been any response within a year.