{"id":105,"date":"2021-09-14T17:50:35","date_gmt":"2021-09-14T22:50:35","guid":{"rendered":"https:\/\/sourceopen.com\/?p=105"},"modified":"2021-09-14T17:53:50","modified_gmt":"2021-09-14T22:53:50","slug":"custom-routing-for-freebsd-jails-or-how-to-set-a-unique-default-route-or-default-gateway-for-each-jail","status":"publish","type":"post","link":"https:\/\/sourceopen.com\/index.php\/custom-routing-for-freebsd-jails-or-how-to-set-a-unique-default-route-or-default-gateway-for-each-jail\/","title":{"rendered":"Custom Routing for FreeBSD Jails – or How To Set a Unique Default Route or Default Gateway for Each Jail"},"content":{"rendered":"\n

I was recently configuring a FreeBSD 13.0 box to run unbound <\/a><\/em>(caching DNS Resolver) in two separate jails. The machine was connected to two WANs through two interfaces. The goal was to have one unbound <\/em>jail route queries to WAN A and the other unbound <\/em>jail route queries to WAN B.<\/p>\n\n\n\n

Important Side Note<\/strong>: If you want to configure unbound to do anything other than be a basic local cache you will want to install the unbound <\/em>FreeBSD package and use the unbound_enable=”YES”<\/em> in your \/etc\/rc.conf. This is different <\/strong>from the pre-installed local_unboun<\/em>d and the local_unbound_enable=”YES|NO”<\/em> setting.<\/p>\n\n\n\n

I opted to use FreeBSD fib support for this. See the setfib(1)<\/a> manpage for more info. Basically, fib’s allow you to have independent routing tables based on the particular fib ids. So each fib can have it’s own default gateway (default route), as well as its own custom routing setups if needed. You can then run any process in the context of a particular fib routing table, or even an entire jail, as we will do here.<\/p>\n\n\n\n

Note<\/strong>: Your kernel needs to have fibs enabled with the ROUTETABLES option. This seems to be the default in newer GENERIC kernels. I did not have to change anything on arm64 aarch64 FreeBSD 13.0.<\/p>\n\n\n\n

In order to enable fibs you have to set some options in the \/boot\/loader.conf<\/em> to load at boot time.<\/p>\n\n\n\n

# echo 'net.fibs=4' >> \/boot\/loader.conf\n# echo 'net.add_addr_allfibs=0' >> \/boot\/loader.conf<\/pre>\n\n\n\n
The first line sets the number of fib routing tables our host will have after rebooting. We really only need two at the moment, but you will have to reboot again to change later. The second line, ” net.add_addr_allfibs=0 ‘ prevents all your routing commands (and more importantly the default routing commands setup by rc.conf<\/em>) from affecting ALL fibs by default. You want independent control over each fib; so this should be disabled. <\/p>\n\n\n\n
Note<\/strong>: There have been reports that in later versions of FreeBSD (~12+) this option must be set in \/etc\/sysctl.conf<\/em>  So I added it there as well. You probably don’t need it in the \/boot\/loader.conf<\/em>, but it doesn’t seem to hurt.<\/p>\n\n\n\n
 # echo 'net.add_addr_allfibs=0' >> \/etc\/sysctl.conf <\/pre>\n\n\n\n
Make sure the ips you wish your jails to use have been configured in \/etc\/rc.conf<\/em> as usual (typically as aliases). Set your default router for the parent host there as well, but not for the jails which need special routing.<\/p>\n\n\n\n
#\/etc\/rc.conf\n\n...\n\n# WAN A ips\n# Be sure to change the ue0 to YOUR interface name\nifconfig_ue0=\"inet 192.168.1.10 netmask 255.255.255.0\"\nifconfig_ue0_alias0=\"inet 192.168.1.20 netmask 255.255.255.0\"\n\n#WAN A DEFAULT GATEWAY\ndefaultrouter=\"192.168.0.1\"\n\n# WAN B ips\n# Be sure to change ue1 to YOUR interface name\nifconfig_ue1=\"inet 192.168.2.10 netmask 255.255.255.0\"\nifconfig_ue1_alias0=\"inet 192.168.2.20 netmask 255.255.255.0\"<\/pre>\n\n\n\n
I my case WAN A is on 192.168.1.xxx and interface ue0 and WAN B is on 192.168.2.xxx and interface ue1. I believe that these could in fact be on the same physical interface if you desire them to be. Mine needed to be on separate Ethernets.<\/p>\n\n\n\n
192.168.1.10 is the parent host’s IP on the WAN A network.<\/p>\n\n\n\n
192.168.1.20 is the jail IP on the WAN A network<\/p>\n\n\n\n
<\/p>\n\n\n\n
192.168.2.10 is the parent host’s IP on the WAN B network.<\/p>\n\n\n\n
192.168.2.20 is the jail IP on the WAN B network.<\/p>\n\n\n\n
<\/p>\n\n\n\n
We set up the default route for the parent host with the  defaultrouter  <\/em>line. This will also be the default route for the jail on the WAN A network. Nothing extra need be done with this jail. It will work like every other jail and follow the parent host’s network routing.<\/p>\n\n\n\n
We will setup the special routing rules for the WAN B jail in \/etc\/rc.loc<\/em>al (or wherever you like to put your local startup scripts).<\/p>\n\n\n\n
We might as well set up our routing for the jail on WAN B before we reboot. Edit or create an \/etc\/rc.local<\/em> (or put this where ever you like to put such startup script).<\/p>\n\n\n\n
# \/etc\/rc.local\n\n...\n\n# fib Routing \n\n# fib 0: Default\n# fib 1: jail B on WAN B\n# fib 2: <reserved for future use>\n# fib 3: <reserved for future use>\n\n#\n# Set up the jail B on WAN B routing table\n#\n# Interface route(s)\n# Be sure to set the iface to YOUR iface\nsetfib 1 route add -net 192.168.2.0\/24 -iface ue1\n\n# Default route\nsetfib 1 route add default 192.168.2.1<\/pre>\n\n\n\n
You will have to change ue1 in the example above to your interface for the second jail. Note that we use the setfib command to first configure which fib the route command will affect. So these routes are added only to fib 1, which is the second fib, and the one we are going to set our jail on WAN B to use.<\/p>\n\n\n\n
Speaking of the jail. We might as well set the jail to use the correct fib routing table before we restart as well. Edit your \/etc\/jail.conf (or use whatever jail manager you prefer to do so) to add the following option to your jail B which is on WAN B and should get the special routing table:<\/p>\n\n\n\n
 exec.fib=1; <\/pre>\n\n\n\n
This tells jail to run this jail ion the context the fib indicated. For instance, your jail.conf might look something like this sample:<\/p>\n\n\n\n
# SAMPLE ONLY-- change all the pertinent data\njailB {\n  exec.fib=1;  #Run this jail with fib 1 routing\n  host.hostname=\"jailB.domain.com\";\n  ip4.addr=\"192.168.2.20\";\n  path=\"\/jail\/jailB\/root\";\n\n  # For debugging. Remove for production\n  allow.raw_sockets = 1;\n}<\/pre>\n\n\n\n
Make sure your jails are all configured and setup properly to reflect there respective ip addresses. Don’t add any routing settings in the jail \/etc\/rc.conf<\/em> files. You would just specify the ip addresses only. In the case of my jail B this would just be something like this:<\/p>\n\n\n\n
#\/etc\/rc.conf\n\n...\n\n# Be sure to change the interface and ip address to your own!\nifconfig_ue1=\"inet 192.168.2.20 netmask 255.255.255.0\"\n\n...<\/pre>\n\n\n\n
Note we also enabled allow.raw_sockets = 1;<\/em> This will let us use tools like mtr <\/em>(install the “nox” version to avoid all the X Windows stuff) or traceroute<\/em>, etc. to work in the jail.<\/p>\n\n\n\n
Note the ifconfig <\/em>line has the interface name in there as well (ifconfig_ue1). Be sure to change this to your interface.<\/p>\n\n\n\n
Now you need to reboot <\/strong>in order to make the loader.conf<\/em> settings take effect. This will also make all your \/etc\/rc.conf settings take effect (which could be done without rebooting), but the \/boot\/loader.conf<\/em> stuff has to be done at startup.<\/p>\n\n\n\n
Assuming your parent host has jail_enable=”YES”<\/em> set in \/etc\/rc.conf your jails should automatically start up as well. If not, service jail start<\/em> should do the trick.<\/p>\n\n\n\n
If you enabled raw sockets in the jail.conf<\/em> you should be able to log in to your jail B on WAN B and do a traceroute <\/em>or use mtr <\/em>to see that your packets from this jail use the new routing you set up for fib 1. You can disable raw sockets once you get everything working.<\/p>\n\n\n\n
Hopefully everything should be working at this point. You will  now have a separate default route \/ default gateway for each jail you wish to configure.<\/p>\n\n\n\n
Obviously if you want to run unbound <\/em>jails like I have you will have to configure each unbound with the correct ip address, and access lists, etc. <\/p>\n\n\n\n
I also put the FreeBSD void-zones<\/em> package on mine in order to also block ads and telemetry at the dns level (similar to Pi-Hole, but with no GUI). It is working quite well.<\/p>\n\n\n\n
If you have difficulty, or spot a mistake, feel free to leave a comment, and I will try to help. <\/p>\n\n\n\n
I found the book “FreeBSD Mastery Jails” by Michael W. Lucas to be very helpful for learning about jails. Don’t forget the manpages!<\/p>\n","protected":false},"excerpt":{"rendered":"
I was recently configuring a FreeBSD 13.0 box to run unbound (caching DNS Resolver) in two separate jails. The machine was connected to two WANs through two interfaces. The goal was to have one unbound jail route queries to WAN … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[3,4,93,52,94,79,5,89,92,91,95,90,24],"_links":{"self":[{"href":"https:\/\/sourceopen.com\/index.php\/wp-json\/wp\/v2\/posts\/105"}],"collection":[{"href":"https:\/\/sourceopen.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sourceopen.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sourceopen.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sourceopen.com\/index.php\/wp-json\/wp\/v2\/comments?post=105"}],"version-history":[{"count":3,"href":"https:\/\/sourceopen.com\/index.php\/wp-json\/wp\/v2\/posts\/105\/revisions"}],"predecessor-version":[{"id":108,"href":"https:\/\/sourceopen.com\/index.php\/wp-json\/wp\/v2\/posts\/105\/revisions\/108"}],"wp:attachment":[{"href":"https:\/\/sourceopen.com\/index.php\/wp-json\/wp\/v2\/media?parent=105"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sourceopen.com\/index.php\/wp-json\/wp\/v2\/categories?post=105"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sourceopen.com\/index.php\/wp-json\/wp\/v2\/tags?post=105"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}