path: root/results/scraper/launchpad/785668

bonding inside a bridge does not update ARP correctly when bridged net accessed from within a VM

Binary package hint: qemu-kvm

Description: Ubuntu 10.4.2
Release: 10.04


When setting a KVM host with a bond0 interface made of eth0 and eth1 and using this bond0 interface for a bridge to KVM VMs, the ARP tables do not get updated correctly and

Reproducible: 100%, with any of the load balancing mode

How to reproduce the problem

- Prerequisites:
1 One KVM system with 10.04.02 server,  configured as a virtual host is needed. The following /etc/network/interfaces was used for the test :

# grep  -v ^# /etc/network/interfaces
auto lo
iface lo inet loopback


auto bond0
iface bond0 inet manual
	post-up ifconfig $IFACE up
	pre-down ifconfig $IFACE down
	bond-slaves none
	bond_mode balance-rr
	bond-downdelay 250
	bond-updelay 120
auto eth0
allow-bond0 eth0
iface eth0 inet manual
	bond-master bond0
auto eth1
allow-bond0 eth1
iface eth1 inet manual
	bond-master bond0

auto br0
iface br0 inet dhcp
	# dns-* options are implemented by the resolvconf package, if installed
	bridge-ports bond0
	bridge-stp off
	bridge-fd 9
	bridge-hello 2
	bridge-maxage 12
	bridge_max_wait 0

One VM running Maverick 10.10 server, standard installation, using the following /etc/network/interfaces file :

grep -v ^# /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 10.153.107.92
        netmask 255.255.255.0
        network 10.153.107.0
        broadcast 10.153.107.255

--------------
On a remote bridged network system :
$ arp -an
? (10.153.107.188) à 00:1c:c4:6a:c0:dc [ether] sur tap0
? (16.1.1.1) à 00:17:33:e9:ee:3c [ether] sur wlan0
? (10.153.107.52) à 00:1c:c4:6a:c0:de [ether] sur tap0

On KVMhost
$ arp -an
? (10.153.107.80) at ee:99:73:68:f0:a5 [ether] on br0

On VM
$ arp -an
? (10.153.107.61) at <incomplete> on eth0

1) Test #1 : ping from VM (10.153.107.92) to remote bridged network system (10.153.107.80) :

- On remote bridged network system :
caribou@marvin:~$ arp -an
? (10.153.107.188) à 00:1c:c4:6a:c0:dc [ether] sur tap0
? (16.1.1.1) à 00:17:33:e9:ee:3c [ether] sur wlan0
? (10.153.107.52) à 00:1c:c4:6a:c0:de [ether] sur tap0

- On KVMhost
ubuntu@VMhost:~$ arp -an
? (10.153.107.80) at ee:99:73:68:f0:a5 [ether] on br0

- On VM
ubuntu@vm1:~$ ping 10.153.107.80
PING 10.153.107.80 (10.153.107.80) 56(84) bytes of data.
From 10.153.107.92 icmp_seq=1 Destination Host Unreachable
From 10.153.107.92 icmp_seq=2 Destination Host Unreachable
From 10.153.107.92 icmp_seq=3 Destination Host Unreachable
^C
--- 10.153.107.80 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3010ms
pipe 3
ubuntu@vm1:~$ arp -an
? (10.153.107.61) at <incomplete> on eth0
? (10.153.107.80) at <incomplete> on eth0

2) Test #2 : ping from remote bridged network system (10.153.107.80) to VM (10.153.107.92) :

- On remote bridged network system :
$ ping 10.153.107.92
PING 10.153.107.92 (10.153.107.92) 56(84) bytes of data.
64 bytes from 10.153.107.92: icmp_req=1 ttl=64 time=327 ms
64 bytes from 10.153.107.92: icmp_req=2 ttl=64 time=158 ms
64 bytes from 10.153.107.92: icmp_req=3 ttl=64 time=157 ms
^C
--- 10.153.107.92 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 157.289/214.500/327.396/79.831 ms
caribou@marvin:~$ arp -an
? (10.153.107.188) à 00:1c:c4:6a:c0:dc [ether] sur tap0
? (16.1.1.1) à 00:17:33:e9:ee:3c [ether] sur wlan0
? (10.153.107.52) à 00:1c:c4:6a:c0:de [ether] sur tap0
? (10.153.107.92) à 52:54:00:8c:e0:3c [ether] sur tap0

- On KVMhost
$ arp -an
? (10.153.107.80) at ee:99:73:68:f0:a5 [ether] on br0

- On VM
arp -an
? (10.153.107.61) at <incomplete> on eth0
? (10.153.107.80) at ee:99:73:68:f0:a5 [ether] on eth0


3) Test #3 : New ping from VM (10.153.107.92) to remote bridged network system (10.153.107.80) :
- On remote bridged network system :
$ arp -an
? (10.153.107.188) à 00:1c:c4:6a:c0:dc [ether] sur tap0
? (16.1.1.1) à 00:17:33:e9:ee:3c [ether] sur wlan0
? (10.153.107.52) à 00:1c:c4:6a:c0:de [ether] sur tap0
? (10.153.107.92) à 52:54:00:8c:e0:3c [ether] sur tap0

- On KVMhost
ubuntu@VMhost:~$ arp -an
? (10.153.107.80) at ee:99:73:68:f0:a5 [ether] on br0

- On VM
ubuntu@vm1:~$ ping 10.153.107.80
PING 10.153.107.80 (10.153.107.80) 56(84) bytes of data.
64 bytes from 10.153.107.80: icmp_req=1 ttl=64 time=154 ms
64 bytes from 10.153.107.80: icmp_req=2 ttl=64 time=170 ms
64 bytes from 10.153.107.80: icmp_req=3 ttl=64 time=154 ms
^C
--- 10.153.107.80 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 154.072/159.465/170.058/7.504 ms

tcpdump traces are available for those tests. Test system is available upon request.

Workaround:

Use the bonded device in "active-backup" mode

ProblemType: Bug
DistroRelease: Ubuntu 10.04.02
Package: qemu-kvm-0.12.3+noroms-0ubuntu9.6
Uname: Linux 2.6.35-25-serverr x86_64
Architecture: amd64

Thanks for reporting this bug and the detailed reproduction instructions.  I would mark it high, but since you offer a workaround I'll mark it medium instead.

What does your /etc/modprobe.d/bonding show?

I've not used this combination myself, but from those who have, a few things do appear fragile, namely:

1. if you are using 802.3ad, you need trunking enabled on the physical switch

2. some people find that turning stp on helps (http://www.linuxquestions.org/questions/linux-networking-3/bridging-a-bond-802-3ad-only-works-when-stp-is-enabled-741640/)

But I'm actually wondering whether this patch:

http://permalink.gmane.org/gmane.linux.network/159403

may be needed.  If so, then even the natty kernel does not yet have that fix.

I am marking this as affecting the kernel, since I believe that is where the bug lies.


Actually, I may be wrong about this being a kernel issue.

Are you always able to ping the remote host from the kvm host, even when you can't do so from the VM?

In addition to kvmhost's /etc/modprove.d/bonding.conf, can you also please provide the configuration info for the KVM vm?  (If a libvirt host, then the network-related (or just all) xml info, or else the 'ps -ef | grep kvm' output).  Also the network configuration insid the KVM VM.  In particular, if the KVM VM has a bridge, that one would need to have stp turned on, but I doubt you have that.

Yup, I can reproduce this 100%.

I'm setting up networking as described above, and then starting virtual machines with:

sudo tunctl -u 1000 -g 1000 -t tap0
sudo /sbin/ifconfig $1 0.0.0.0 up
sudo brctl addif br0 tap0

kvm -drive file=disk.img,if=virtio,cache=none,boot=on -m 1024 -vnc :1 -net nic,model=virtio -net tap,script=no,ifname=tap0,downscript=no

With mode=balance-rr, I can't run dhclient from the guest.  With either
bond0 as active-backup, or without bond0 (with eth0 directly in br0),
I can.


Following the advice toward the bottom of

http://forum.proxmox.com/archive/index.php/t-2676.html?s=e8a9cfc9a128659e4a61efec0b758d3e

I was able to get this to work with balance-rr by changing a few bridge properties.  The following was my /etc/network/interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
 post-up ifconfig $IFACE up
 pre-down ifconfig $IFACE down
 bond-slaves none
 bond_mode active-rr
 bond-downdelay 250
 bond-updelay 120
auto eth0
allow-bond0 eth0
iface eth0 inet manual
 bond-master bond0
auto eth1
allow-bond0 eth1
iface eth1 inet manual
 bond-master bond0

auto br0
iface br0 inet dhcp
 # dns-* options are implemented by the resolvconf package, if installed
 bridge-ports bond0
# bridge-stp off
# bridge-fd 9
# bridge-hello 2
# bridge-maxage 12
# bridge_max_wait 0
 bridge_stp on
 bridge_maxwait 0
 bridge_maxage 0
 bridge_fd 0
 bridge_ageing 0

I don't know if this is acceptable to you since stp is on.  If not, is using balance-alb (which did also work for me) acceptable?

Following your suggestions, I modified my /etc/network/interfaces & added the STP options to my test environment.  Following that, I am now able to ping to the remote system using the following bonding modes :

* 802.3ad (4)
* tlb (5)
* alb (6)

For unknown reasons, I'm still unable to use balance-rr unlike your setup.  But that might not be much of an issue as those modes listed above might be sufficient. I must go & check that.  And now, the two VMs are able to ping each other.

One thing regarding your listed /etc/network/interfaces : I think that there is a typo as 'bond_mode active-rr' is not a support bonding mode.

Regarding your request for /etc/modprove.d/bonding.conf, there is no such file on my test system. Let me know if you still require the xml dump of the VM.

Quoting Louis Bouchard (<email address hidden>):
> Regarding your request for /etc/modprove.d/bonding.conf, there is no
> such file on my test system.

Right, sorry, that's obsolete as of hardy, sorry.

> Let me know if you still require the xml
> dump of the VM.

Thanks, no, as I'm able to reproduce that won't be necessary.


I can reproduce this just using lxc, which simply attaches an endpoint of a veth tunnel to the bridge.  With balance-rr mode, i can't dhcp in the guest.  With balance-alb, I can.

That means this is not actually qemu-kvm, but a bug in the kernel or (unlikely) ifenslave.

My next steps will be to test on maverick and natty, look through linux-2.6/drivers/net/bonding and linux-2.6/net/bridge/ and perhaps go to the https://lists.linux-foundation.org/pipermail/bridge/2011-May/thread.html list to ask for help if it is still broken in natty.

Maverick gives me the same result.  (Except I don't seem able, in maverick, to auto-setup the bond+bridge setup with /etc/network/interfaces, keep having to do it by hand.  Hoping I did something wrong myself,a nd it's not a maverick bug)

Natty is still affected.

Since qemu isn't needed to show the bug, you can now trivially test this inside a natty kvm container by giving it two NICs, setting up /etc/network interfaces as shown above, and using lxc as follows:

   apt-get install lxc debootstrap
   mkdir /cgroup
   mount -t cgroup cgroup /cgroup
   cat > /etc/lxc.conf << EOF
   lxc.network.type=veth
   lxc.network.link=br0
   lxc.network.flags=up
   EOF
   lxc-create -t natty -n lxc -f /etc/lxc.conf
   lxc-start -n lxc

When not using balance-rr, the container's network is fine.  When using balance-rr, it can't get a dhcp address.

Next step is probably to look at the hwaddr handling in the kernel source, and talk to upstream.


I sent an email to bonding-devel, and got this response:

http://sourceforge.net/mailarchive/forum.php?thread_name=21866.1306527811%40death&forum_name=bonding-devel

Assuming that your switch is in fact set up for Etherchannel, can you go ahead and gather the tcpdump data?

I read the mail and did a first round of test before I could check the setting of the switch.  Here are the transcript of the test with balance-rr.

Container : LXC container with fixed IP
VMhost : The host where the LXC container runs. configured with br0 & bond0
remote_host : another host on the same bridged subnet

Container : date;ping xxx.xxx.xxx.87
Mon May 30 15:40:49 UTC 2011
PING xxx.xxx.xxx.87 (xxx.xxx.xxx.87): 48 data bytes
60 bytes from xxx.xxx.xxx.92: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst Data
 4  5  00 4c00 0000   0 0040  40  01 cc4e xxx.xxx.xxx.92  xxx.xxx.xxx.87 
60 bytes from xxx.xxx.xxx.92: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst Data
 4  5  00 4c00 0000   0 0040  40  01 cc4e xxx.xxx.xxx.92  xxx.xxx.xxx.87 
60 bytes from xxx.xxx.xxx.92: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst Data
 4  5  00 4c00 0000   0 0040  40  01 cc4e xxx.xxx.xxx.92  xxx.xxx.xxx.87 
^C--- xxx.xxx.xxx.87 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss


VMhost : date;ping xxx.xxx.xxx.92
Mon May 30 15:41:14 EDT 2011
PING xxx.xxx.xxx.92 (xxx.xxx.xxx.92) 56(84) bytes of data.
64 bytes from xxx.xxx.xxx.92: icmp_req=9 ttl=64 time=10.1 ms
64 bytes from xxx.xxx.xxx.92: icmp_req=10 ttl=64 time=0.087 ms
64 bytes from xxx.xxx.xxx.92: icmp_req=11 ttl=64 time=0.076 ms
^C
--- xxx.xxx.xxx.92 ping statistics ---
11 packets transmitted, 3 received, 72% packet loss, time 10004ms
rtt min/avg/max/mdev = 0.076/3.423/10.108/4.727 ms


Container : date;ping xxx.xxx.xxx.87
Mon May 30 15:41:41 UTC 2011
PING xxx.xxx.xxx.87 (xxx.xxx.xxx.87): 48 data bytes
60 bytes from xxx.xxx.xxx.92: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst Data
 4  5  00 4c00 0000   0 0040  40  01 cc4e xxx.xxx.xxx.92  xxx.xxx.xxx.87 
60 bytes from xxx.xxx.xxx.92: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst Data
 4  5  00 4c00 0000   0 0040  40  01 cc4e xxx.xxx.xxx.92  xxx.xxx.xxx.87 
60 bytes from xxx.xxx.xxx.92: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst Data
 4  5  00 4c00 0000   0 0040  40  01 cc4e xxx.xxx.xxx.92  xxx.xxx.xxx.87 
^C--- xxx.xxx.xxx.87 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss


remote_host : date;ping xxx.xxx.xxx.92
lundi 30 mai 2011, 15:42:03 (UTC+0200)
PING xxx.xxx.xxx.92 (xxx.xxx.xxx.92) 56(84) bytes of data.
64 bytes from xxx.xxx.xxx.92: icmp_req=1 ttl=64 time=284 ms
64 bytes from xxx.xxx.xxx.92: icmp_req=2 ttl=64 time=125 ms
64 bytes from xxx.xxx.xxx.92: icmp_req=3 ttl=64 time=134 ms
^C
--- xxx.xxx.xxx.92 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 125.282/181.561/284.952/73.204 ms


Container : Mon May 30 15:42:24 UTC 2011
PING xxx.xxx.xxx.87 (xxx.xxx.xxx.87): 48 data bytes
56 bytes from xxx.xxx.xxx.87: icmp_seq=0 ttl=64 time=141.506 ms
56 bytes from xxx.xxx.xxx.87: icmp_seq=1 ttl=64 time=153.311 ms
56 bytes from xxx.xxx.xxx.87: icmp_seq=2 ttl=64 time=124.973 ms
^C--- xxx.xxx.xxx.87 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 124.973/139.930/153.311/11.622 ms

I will send you the dump data directly.  Now that I have full access to our switch, I will do more tests tomorrow.  As far as I know, the switch is doing automatic trunking so the switch should not be an issue.


Hello,

Now I am dazed and confused (and trying to continue) 

I have tested most of the combination of bonding modes with appropriate switch settings and here is what I get :

Bonding mode	switch configuration	        result(ping from Container)	With STP
============	====================	======           				========
balance-rr	         two port trunked	        OK
balance-rr	         No trunking		                NOK		                 		OK
balance-alb    	 No trunking		                OK
balance-tlb	         No trunking		                OK
802.3ad		         LACP dynamic trunk	        OK
balance-xor	         two port trunked	        OK
balance-xor	         No trunking		                NOK				                NOK

I could swear that I had tested -alb and -tlb with negative results. So apparently, the STP workaround is not required with proper switch configuration.

It looks like this has been fixed upstream.  I will close it.  If the problem still occurs, please reopen it.