Things I learned about IPVS on OpenStack

Currently, I’m working with IPVS and OpenStack. This block post records some quirks I have found while setting up IPVS within a VM on top of OpenStack. None of these are unknown for experienced networking people, but I think they are worth to write down. In my setup I use IPVS with the TUN mode. Both the IPVS director and the server are located within the same OpenStack network. I have assigned the virtual IP (VIP) to the tunl0 interface of the realservers. This causes some ARP problems since, by default most linux servers answer ARP requests on any interface. For example, let’s assume our realserver has an interface eth0 attached to the OpenStack network. The VIP (192.168.48.3) is attached to tunl0. By default our realserver will answer any ARP request for the IP 192.168.48.3 on the interface eth0 as well. To deactivate this, I have configured

1
2
3
4

sysctl -qw net.ipv4.conf.all.arp_filter=1
sysctl -qw net.ipv4.conf.all.arp_ignore=1
sysctl -qw net.ipv4.conf.eth0.arp_filter=1
sysctl -qw net.ipv4.conf.eth0.arp_ignore=1

arp_ignore=1 instructs the kernel only to answer ARP requests, if the requested IP address is assigned to the network interface, the ARP request was received on. arp_filter does the same but will also respect your source based routing setup and will answer only if the kernel had routed an outgoing packet via the interface the ARP request was received on. I have set it for both all and the eth0 interface, since the maximum value of the value configured for all and an interface is used to determine the effective value for the interface. The IPVS knowledgebase has more details and other solutions for this problem.

The second and harder problem I ran into was that my realservers got the the SYN packet of a new connection via the tunl0 interface, but never replied with a SYN-ACK. Thus, the TCP three-way handshake could never succeed. After a very helpful tip of @awlnx on twitter I disabled rp_filter on the tunl0 interface. rp_filter is a spoofing protection enabled by default. It discards all packets received on an interface where the source IP address is not reachable via that interface. This is a very secure and advised default, but since we do asymmetrical routing it will drop our SYN packets received via the tunl0 interface. I have configured:

1
2

sysctl -qw net.ipv4.conf.tunl0.rp_filter=0
sysctl -qw net.ipv4.conf.all.rp_filter=0

Also, for both all and the tunl0 interface, because the maximum value is chosen. The default for rp_filter is 1.

The last problem I encountered where related to OpenStacks port security feature. I have added the VIP as allowed-address-pair on the eth0 interface of my realservers, yet the OpenStack networking layer drops the outgoing SYN-ACK packets from my realservers. The only solutions is to disable port security for the interface in OpenStack. I have yet to find out if this is related to the used OpenStack network implementation or a general OpenStack problem. Disabling port security entirely will also disable all security groups attached to that interface.