I was recently configuring a FreeBSD 13.0 box to run unbound (caching DNS Resolver) in two separate jails. The machine was connected to two WANs through two interfaces. The goal was to have one unbound jail route queries to WAN A and the other unbound jail route queries to WAN B.
Important Side Note: If you want to configure unbound to do anything other than be a basic local cache you will want to install the unbound FreeBSD package and use the unbound_enable=”YES” in your /etc/rc.conf. This is different from the pre-installed local_unbound and the local_unbound_enable=”YES|NO” setting.
I opted to use FreeBSD fib support for this. See the setfib(1) manpage for more info. Basically, fib’s allow you to have independent routing tables based on the particular fib ids. So each fib can have it’s own default gateway (default route), as well as its own custom routing setups if needed. You can then run any process in the context of a particular fib routing table, or even an entire jail, as we will do here.
Note: Your kernel needs to have fibs enabled with the ROUTETABLES option. This seems to be the default in newer GENERIC kernels. I did not have to change anything on arm64 aarch64 FreeBSD 13.0.
In order to enable fibs you have to set some options in the /boot/loader.conf to load at boot time.
# echo 'net.fibs=4' >> /boot/loader.conf # echo 'net.add_addr_allfibs=0' >> /boot/loader.conf
The first line sets the number of fib routing tables our host will have after rebooting. We really only need two at the moment, but you will have to reboot again to change later. The second line, ” net.add_addr_allfibs=0 ‘ prevents all your routing commands (and more importantly the default routing commands setup by rc.conf) from affecting ALL fibs by default. You want independent control over each fib; so this should be disabled.
Note: There have been reports that in later versions of FreeBSD (~12+) this option must be set in /etc/sysctl.conf So I added it there as well. You probably don’t need it in the /boot/loader.conf, but it doesn’t seem to hurt.
# echo 'net.add_addr_allfibs=0' >> /etc/sysctl.conf
Make sure the ips you wish your jails to use have been configured in /etc/rc.conf as usual (typically as aliases). Set your default router for the parent host there as well, but not for the jails which need special routing.
#/etc/rc.conf ... # WAN A ips # Be sure to change the ue0 to YOUR interface name ifconfig_ue0="inet 192.168.1.10 netmask 255.255.255.0" ifconfig_ue0_alias0="inet 192.168.1.20 netmask 255.255.255.0" #WAN A DEFAULT GATEWAY defaultrouter="192.168.0.1" # WAN B ips # Be sure to change ue1 to YOUR interface name ifconfig_ue1="inet 192.168.2.10 netmask 255.255.255.0" ifconfig_ue1_alias0="inet 192.168.2.20 netmask 255.255.255.0"
I my case WAN A is on 192.168.1.xxx and interface ue0 and WAN B is on 192.168.2.xxx and interface ue1. I believe that these could in fact be on the same physical interface if you desire them to be. Mine needed to be on separate Ethernets.
192.168.1.10 is the parent host’s IP on the WAN A network.
192.168.1.20 is the jail IP on the WAN A network
192.168.2.10 is the parent host’s IP on the WAN B network.
192.168.2.20 is the jail IP on the WAN B network.
We set up the default route for the parent host with the defaultrouter line. This will also be the default route for the jail on the WAN A network. Nothing extra need be done with this jail. It will work like every other jail and follow the parent host’s network routing.
We will setup the special routing rules for the WAN B jail in /etc/rc.local (or wherever you like to put your local startup scripts).
We might as well set up our routing for the jail on WAN B before we reboot. Edit or create an /etc/rc.local (or put this where ever you like to put such startup script).
# /etc/rc.local ... # fib Routing # fib 0: Default # fib 1: jail B on WAN B # fib 2: <reserved for future use> # fib 3: <reserved for future use> # # Set up the jail B on WAN B routing table # # Interface route(s) # Be sure to set the iface to YOUR iface setfib 1 route add -net 192.168.2.0/24 -iface ue1 # Default route setfib 1 route add default 192.168.2.1
You will have to change ue1 in the example above to your interface for the second jail. Note that we use the setfib command to first configure which fib the route command will affect. So these routes are added only to fib 1, which is the second fib, and the one we are going to set our jail on WAN B to use.
Speaking of the jail. We might as well set the jail to use the correct fib routing table before we restart as well. Edit your /etc/jail.conf (or use whatever jail manager you prefer to do so) to add the following option to your jail B which is on WAN B and should get the special routing table:
exec.fib=1;
This tells jail to run this jail ion the context the fib indicated. For instance, your jail.conf might look something like this sample:
# SAMPLE ONLY-- change all the pertinent data
jailB {
exec.fib=1; #Run this jail with fib 1 routing
host.hostname="jailB.domain.com";
ip4.addr="192.168.2.20";
path="/jail/jailB/root";
# For debugging. Remove for production
allow.raw_sockets = 1;
}
Make sure your jails are all configured and setup properly to reflect there respective ip addresses. Don’t add any routing settings in the jail /etc/rc.conf files. You would just specify the ip addresses only. In the case of my jail B this would just be something like this:
#/etc/rc.conf ... # Be sure to change the interface and ip address to your own! ifconfig_ue1="inet 192.168.2.20 netmask 255.255.255.0" ...
Note we also enabled allow.raw_sockets = 1; This will let us use tools like mtr (install the “nox” version to avoid all the X Windows stuff) or traceroute, etc. to work in the jail.
Note the ifconfig line has the interface name in there as well (ifconfig_ue1). Be sure to change this to your interface.
Now you need to reboot in order to make the loader.conf settings take effect. This will also make all your /etc/rc.conf settings take effect (which could be done without rebooting), but the /boot/loader.conf stuff has to be done at startup.
Assuming your parent host has jail_enable=”YES” set in /etc/rc.conf your jails should automatically start up as well. If not, service jail start should do the trick.
If you enabled raw sockets in the jail.conf you should be able to log in to your jail B on WAN B and do a traceroute or use mtr to see that your packets from this jail use the new routing you set up for fib 1. You can disable raw sockets once you get everything working.
Hopefully everything should be working at this point. You will now have a separate default route / default gateway for each jail you wish to configure.
Obviously if you want to run unbound jails like I have you will have to configure each unbound with the correct ip address, and access lists, etc.
I also put the FreeBSD void-zones package on mine in order to also block ads and telemetry at the dns level (similar to Pi-Hole, but with no GUI). It is working quite well.
If you have difficulty, or spot a mistake, feel free to leave a comment, and I will try to help.
I found the book “FreeBSD Mastery Jails” by Michael W. Lucas to be very helpful for learning about jails. Don’t forget the manpages!
