Xen Hypervisor NAT Routing

My existing Xen networking setup consists of a simple eth0->xenbr0 Linux bridge that essentially moves eth0’s IP address over to xenbr0 with each Xen DomU guest reaching out to the router for DHCP assignments.

In order to move away from depending on a networked router and towards deployment-enforced Xen networking, I have reconfigured xenbr0 as a detached bridge (no bridge_ports) with IPtables/NAT MASQUERADE rules to pass traffic between my host’s network to my guests’ bridge.

Most of the configuration here was borrowed from https://ilearnedhowto.wordpress.com/tag/open-vswitch/, so credit for most of this goes to Carlos A.

The following steps are using Debian 9 as a test bed for this configuration. I configured my Xen bridge to have a static IP as if it’s now the router for the subnet that will house my DomUs:

Dom0 ~$ cat > /etc/network/interfaces.d/xenbr0 << EOF
auto xenbr0
iface xenbr0 inet static
      bridge_stp none
      address 10.10.1.1
      network 10.10.1.0
      broadcast 10.10.1.255
      netmask 255.255.255.0
EOF

It then needs some IPtables rules to pass traffic from eth0 over to xenbr0:

Dom0 ~$ cat > /etc/network/enable_nat.sh << EOF
#!/bin/bash
IFACE_WAN=eth0
IFACE_LAN=${1}
NETWORK_LAN=${2}

echo "1" > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o $IFACE_WAN -s $NETWORK_LAN ! -d $NETWORK_LAN -j MASQUERADE
iptables -A FORWARD -d $NETWORK_LAN -i $IFACE_WAN -o $IFACE_LAN -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -s $NETWORK_LAN -i $IFACE_LAN -j ACCEPT
EOF

Dom0 ~$ chmod u+x /etc/network/enable_nat.sh

Then it will need to know to run the above enable_nat.sh script whenever xenbr0 is brought up and I also had failures of xenbr0 coming online on host reboot, so I added the following to /etc/network/interfaces.d/xenbr0:

Dom0 ~$ cat >> /etc/network/interfaces.d/xenbr0 << EOF
      pre-up  brctl addbr xenbr0
      post-up /etc/network/enable_nat.sh xenbr0 10.10.1.0/24
EOF

So now we ensure that xenbr0 is added before its IP configuration is set; otherwise Debian has a tendency to complain that "xenbr0 doesn't exist". It then 'post-up' runs the enable_nat.sh script and passes the IFACE name 'xenbr0' and the network to the script as arguments which it takes in as ${1} and ${2}.

Now after any reboot, we should see xenbr0 up with a configuration (and not just a blank bridge), and IPtables should output our MASQUERADING entries:

Dom0 ~$ ifconfig xenbr0
xenbr0: flags=4163  mtu 1500
        inet 10.10.1.1  netmask 255.255.255.0  broadcast 10.10.1.255
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 133  bytes 16383 (15.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 204  bytes 20337 (19.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
Dom0 ~$ iptables -L -t nat | grep MASQ
MASQUERADE  all  --  10.10.1.0/24         !10.10.1.0/24
Dom0 ~$ iptables -L -n | grep 10.10.1.0
ACCEPT     all  --  0.0.0.0/0            10.10.1.0/24          state RELATED,ESTABLISHED
ACCEPT     all  --  10.10.1.0/24          0.0.0.0/0

Now within a Xen DomU configuration, set the 'vif' interface up with a MAC address and bridge setting. I also add an ip= setting, but have found this largely pointless since I configure the guest's network statically anyway:

vif = ["mac=00:22:19:??:??:??,ip=10.10.1.5,bridge=xenbr0"]

With a default configuration, Xen attempts to use the defined bridge and will create a new 'vif' interface on the host attached to xenbr0. If you're using a DHCP server to assign addresses to guests, you can probably stop at this point. I'll be adding DHCP services later, so I plan on writing a separate post for that as a follow-up.

In my case for now, I'm using static IPs, so I have a bit more to do on the guest:

DomU ~$ cat > /etc/network/interfaces << EOF
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
 address 10.10.1.2
 netmask 255.255.255.0
 gateway 10.10.1.1
EOF

And also set nameservers in /etc/resolv.conf. For now we'll just use Google's:

DomU ~$ cat > /etc/resolv.conf << EOF
nameserver 8.8.8.8
nameserver 8.8.4.4
EOF

And at this point, after restarting networking on the guest, it should be able to ping out to the world. There is still more needed as mentioned earlier with a DHCP server and also some DNS routing and/or IPtables rules, but this is enough to get started with configuring my Xen stack.

Leave a Reply

Your email address will not be published. Required fields are marked *