I was setting up WireGuard server on a FreeBSD machine (13.0) on a VPS. In this case the VPS was a Contabo VPS which uses KVM. (Edit: I just encountered the same issue with an Oracle Cloud VM Instance as well (which also used KVM).)
In particular the goal was to run a WireGuard server inside a jail on this VPS. I essentially followed this pretty nice guide on setting things up. This got everything working, but it was far from fast. An OpenBSD machine, running on a much inferior VPS was tons faster.
I determined that simple downloads with the likes of wget from inside the jail on the VPS were terribly slow as well. I was getting about 90K/sec vs. the many MB/sec I should have been getting. When I tried the same download from the VPS itself (outside the jail) I go full speed.
I tried NOT using the bridge interface as discussed in the article above. (Look for where it says,”But if you need only a single jail, it’s not necessary to use the bridge.”) I wondered if perhaps that extra layer was slowing things down. This made no measurable difference. However, since I got all the results I wanted without the bridge, I figured I was indeed better off without this extra layer.
I found that pf (and ipfw too I believe) need tso disabled in the vtnet driver in order to do NAT in kernel. So I added the following to the /boot/loader.conf on the VPS machine (outside the jail):
hw.vtnet.X.tso_disable="1"
hw.vtnet.tso_disable="1"
(Changing this file requires a full reboot of the VPS, not just the jail.) That really didn’t seem to do much either.
Then I found this posting from the mailing list archive. Here the gentleman suggested disabling lro as well. So now my /boot/loader.conf contained the following (along with anything else that had been in there before I started all this).
hw.vtnet.X.tso_disable="1"
hw.vtnet.tso_disable="1"
hw.vtnet.lro_disable="1"
hw.vtnet.X.lro_disable="1"
So this fixed my download speeds when doing a wget directly inside the jail! Everything was back to full speed inside the jail. Good progress!
However, transfers via WireGuard from the WireGuard client machine(s) were still quite slow (like around 1-2Mbps), and seemed to have some high latency like possibly some packets were dropped, though I could never see dropped packets or high latency via tools like ping or mtr.
So I decided, what the heck, let’s try disabling checksum (csum) offloading too! Now the /boot/loader.conf contained:
hw.vtnet.X.tso_disable="1"
hw.vtnet.tso_disable="1"
hw.vtnet.lro_disable="1"
hw.vtnet.X.lro_disable="1"
hw.vtnet.csum_disable="1"
hw.vtnet.X.csum_disable="1"
And with these settings on the vps (non-jail) machine and a reboot I was finally able to realize the full potential speed of the WireGuard VPN on the Contabo (or Oracle Cloud) KVM based VPS.
This is how I fixed the slow networking with a WireGuardVPN server in a FreeBSD Jail. This also fixed slow download speeds with a FreeBSD Jail on a VPS. I hope this helps someone.
