Things I learned about VMs with two interfaces on OpenStack

Sometimes you want to attach two interfaces to a OpenStack virtual machine (VM). As discussed in my last post, I had to disable OpenStacks port security feature to allow my IPVS realservers to implement direct server return. Disabling port-security has the downside, that my realserver can participate in other OpenStack security groups. For my use case, my realservers have to be part of a security group to reach certain backends. To implement that, I added a second network interface to my VM. It turned out, it was not easy as expected. This picture shows the setup:

The underlying OpenStack network implemented with Linux bridges is very picky about which traffic can be sent over which interface. By default, Linux is not configured for this setup. You have to change certain defaults to make this work. To sum the linked article up, Linux will respond by default to an ARP request received on any interface requesting an IP assigned to any local interface. Thus, in the example above, an ARP request received on ens3 requesting the IP 192.0.2.4 will be answered. This messes up the OpenStack network, since it contradicts the security model and assumes the response is spoofing. OpenStack compares ARP responses with the configured IP addresses in its network model. The second problem is, that for any packets originating on my realserver, only the interface ens3 is used. Even if the source IP is 192.0.2.4. This is due to the fact, that by default the first matching route is used. We use policy-based routing to fix this problem.

ARP

For ARP, we use the same settings as in my last post

1
2
3

sysctl -qw net.ipv4.conf.all.arp_ignore=1
sysctl -qw net.ipv4.conf.ens3.arp_ignore=1
sysctl -qw net.ipv4.conf.ens4.arp_ignore=1

As explained in my last post, the arp_ignore=1 sysctl settings will instruct the linux kernel only to answer ARP requests for IPs attached to the receiving interface. Thus, in our example, the kernel will only respond to ARP requests received via ens3 requesting the IP 192.0.2.3. ARP requests for the IP 192.0.2.4 are ignored on the interface ens3 after this setting is set.

Policy-Based Routing

The next problem we have to solve, is that the Linux kernel will send traffic originating from the IP 192.0.2.4 via the ens3 interface. This traffic will be dropped by OpenStack, since it assumes this is spoofed traffic. Furthermore, we want to use the security groups attached to ens4. To make sure, traffic originating from 192.0.2.4 is forwarded via the ens4 interface, we must use policy-based routing.

First, we create two new routing tables:

1
2

echo 200 table_ens3 > /etc/iproute2/rt_tables
echo 201 table_ens4 > /etc/iproute2/rt_tables

You can choose what ever name you want. I used table_ensX for clarity. You must however, choose a different ID for both tables. I have chosen the ID 200 and 201.

Next, we add the default routes to the two routing tables:

1
2
3
4
5
6
7
8
9

# Adding host route for our gateway
ip route add table table_ens3 192.0.2.1 dev ens3
# Adding default route via the gateway
ip route add table table_ens3 default via 192.0.2.1 dev ens3

# Adding host route for our gateway
ip route add table table_ens4 192.0.2.1 dev ens4
# Adding default route via the gateway
ip route add table table_ens4 default via 192.0.2.1 dev ens4

The ip route add table table_ensX instructs the ip route add command, to add the route to the routing table table_ensX. If you have other routing requirements, e.g. allowing use ens4 only for traffic within the subnet 192.0.2.0/24, you can do this:

1
2

# Adding route for subnet via the gateway
ip route add table table_ens4 192.0.2.0/24 via 192.0.2.1 dev ens4

The last step is to configure the policy-based routing. Therefore, we use the ip rule command:

1
2
3
4
5

# Use the table table_ens4 for routing lookups for the ip 192.0.2.4
ip rule add from 192.0.2.3 lookup table_ens3

# Use the table table_ens4 for routing lookups for the ip 192.0.2.4
ip rule add from 192.0.2.4 lookup table_ens4

To show the rules/policies in effect, you can use the command ip rule show. The output should look like this:

1
2
3
4
5

0:	from all lookup local
32764:	from 192.0.2.3 lookup table_ens3
32765:	from 192.0.2.4 lookup table_ens4
32766:	from all lookup main
32767:	from all lookup default

The first number is the priority. The rules are check from the lowest priority number (zero) to the highest (here 32767). The first matching rule is used.

To test your routing setup, the command ip route get is very handy. This let you check the where a packet would be routed. Doing this should give you an output like:

1
2
3
4
5
6
7

ip route get 8.8.8.8 from 192.0.2.3
8.8.8.8 via 192.0.2.1 dev ens3 src 192.0.2.3 table table_ens3 uid 1000 
    cache 
    
ip route get 8.8.8.8 from 192.0.2.4
8.8.8.8 via 192.0.2.1 dev ens4 src 192.0.2.4 table table_ens3 uid 1000
    cache 

As you can see, ip route get shows you the device dev ens3 and the used table to retrieve that information. The downside of this set up is, you have to manage the routes in the tables table_ens3 and table_ens4 by yourself. Thus, you should to this only if your routing and the IPs are pretty static.

To sum it up, we have learned how to configure Linux to run with two interfaces in the same subnet. First, we have to reconfigure the default ARP answering behaviour. Secondly, we have to use policy-based routing to steer the egress traffic to the proper interfaces.