Two weeks with wireguard-kmod (OPNsense)

So I’ve been using the kernel implementation of WireGuard for nearly 2 weeks now, and so far its been rock solid. I haven’t seen any issues in my environments, which is 4 OPNsense devices in a mesh S2S design using WireGuard and then 2 of them have a WireGuard tunnel to PIA. Nothing has missed a beat. I have seen a 0.3ms-0.5ms drop in latency monitoring, to other OPNsense peers. So indeed there is both speed and latency improvements.

Only little issue I have noticed, if you look at the below screenshot, you’ll notice there’s some interface errors. Now are these errors… well yes and no. In the kernel implementation at the moment, if WireGuard tries to send packets of data to a peer it hasn’t got a handshake with it’ll drop the packets and then up the error counter on the interface. The same is true if WireGuard receives a data packet from a peer where the source IP inside the tunnel isn’t on its AllowedIPs list, since it’ll drop any packets of data from peers where it doesn’t match the AllowedIPs list for that peer.

Imagine you’ve just restarted WireGuard interface or OPNsense it self. You may have clients that are trying to send traffic to another WireGuard Peer, but since the handshake hasn’t happened yet it’ll drop those packets, until it has done a handshake. So seeing a few errors on the interface is normal. This is why you see errors on my interface below, because I have monitoring that is trying to contact the other peers, where the handshake hasn’t happened quite yet.
If you are seeing lots of errors, you may be trying to send traffic to a peer that is offline or handshake isn’t happening for some reason or another, which will cause WireGuard to drop that traffic.

Note: The Go implementation of WireGuard doesn’t up the interface counter in these two cases, since its not supported feature of the Go interface implementation.

If your trying to debug your interface errors, you can turn the wg interface debug mode on, where it’ll output errors in to the dmesg of the system. You can enable this by running the following command in the CLI ifconfig wg0 debug. Then when you check the dmesg, you’ll start to see WireGuard being verbose about the errors on the interface. To turn off the debug just do ifconfig wg0 -debug

An example error is “Packet has unallowed src IP from peer 3”, which means WireGuard has retrieved a packet of data from a known peer but the source IP of the packet isn’t on its AllowedIPs list, so its dropped that packet.
Now trying to workout which peer is number 3, unfortunately isn’t too easy at the moment, and its a matter of removing each peer until you stop getting the error, then add it back again, if the error returns you know which peer is causing the traffic and then you may go and investigate why that peer is sending traffic from a source IP it shouldn’t.

I’ve have actually had to do this myself when I first started using the kernel module, since I had a routing issue, where some traffic was being sent over the WireGuard tunnel, which shouldn’t have been. This is of no fault to WireGuard just a misconfiguration on my side. Which didn’t highlight itself until the move to using kmod from go, due to the counters.

Hopefully this blog post will aid others in knowing why the WireGuard kmod interface error count isn’t always 0 and how to help diagnose why their counter keeps going up, if all peers have a handshake.

Specifications of the OPNsense Firewalls encase people were wondering.
OPNsense 1: Qotom Q535G6 2 cores (4 threads) (Intel Core i3-7100U @ 2.40GHz)
OPNsense 2: Proxmox VM – 2 cores (Intel Xeon E-2236 @ 3.40GHz)
OPNsense 3: Proxmox VM – 2 cores (Intel Xeon E5-1650 v3 @ 3.50GHz)
OPNsense 4: Proxmox VM – 2 cores (Intel Xeon E3-1245 V2 @ 3.40GHz)

How to install wireguard-kmod on OPNsense