Things I learned about IPVS on OpenStack
Currently, I’m working with IPVS and OpenStack. This block post records some quirks I have found while setting up
IPVS within a VM on top of OpenStack.
None of these are unknown for experienced networking people, but I think they are worth to write down. In my setup
I use IPVS with the
TUN mode. Both the IPVS director and the server are located within the same OpenStack network. I have assigned the
virtual IP (VIP) to the
tunl0 interface of the realservers. This causes some ARP problems since, by default most
linux servers answer ARP requests on any interface. For example, let’s assume our realserver has an interface
attached to the OpenStack network. The VIP (
192.168.48.3) is attached to
tunl0. By default our realserver will
any ARP request for the IP
192.168.48.3 on the interface
eth0 as well. To deactivate this, I have configured
arp_ignore=1 instructs the kernel only to answer ARP requests,
if the requested IP address is assigned to the
network interface, the ARP request was received on.
arp_filter does the same but will also respect your source
based routing setup and will answer only if the kernel had routed an outgoing packet via the interface the
ARP request was received on. I have set it for both
all and the
eth0 interface, since the maximum value of the
value configured for
all and an interface is used to determine the effective value for the interface.
The IPVS knowledgebase has more
details and other solutions for this problem.
The second and harder problem I ran into was that my realservers got the the SYN packet of a new connection via the
tunl0 interface, but never replied with a
SYN-ACK. Thus, the TCP three-way handshake could never succeed. After
a very helpful tip of @awlnx on twitter
rp_filter on the
rp_filter is a
spoofing protection enabled by default. It discards all packets received on an interface where the source IP address
is not reachable via that interface. This is a very secure and advised default, but since we do asymmetrical routing
it will drop our
SYN packets received via the
I have configured:
Also, for both
all and the
tunl0 interface, because the maximum value is chosen. The default for
rp_filter is 1.
The last problem I encountered where related to OpenStacks port security feature.
I have added the
eth0 interface of my realservers, yet the OpenStack networking layer drops the outgoing
from my realservers. The only solutions is to disable port security for the interface in OpenStack. I have yet to
find out if this is related to the used OpenStack network implementation or a general OpenStack problem. Disabling
port security entirely will also disable all security groups attached to that interface.