pfSense Slow Speeds: Find the Real Bottleneck

A pfSense firewall that is slower than the link behind it is almost never slow for the reason people assume. The usual assumption is CPU, the usual first action is buying a faster machine, and the usual outcome is the same number on newer hardware. The documented causes cluster somewhere else entirely: network buffers, driver behaviour, link protocol, inspection settings and the shape of the traffic itself.

What follows is a diagnostic order. It is arranged by cost, cheapest first, and every step is meant to be finished before the next one starts. Skipping to the tuning section is how firewalls end up with a page of loader tunables that fix nothing.

Step 1: decide what number you are trying to reproduce

Before touching a setting, write down three figures: the speed the link is provisioned at, the speed observed, and the traffic that was running when it was observed. If the third one is missing, the first two mean very little.

Netgate publishes the reason. Forwarding cost is charged per packet, not per bit, so the same hardware at the same packet rate delivers very different bandwidth depending on frame size. At an example rate of 500,000 packets per second, 64-byte frames yield 244 Mbps and 1500-byte frames yield 5.59 Gbps from that same packet rate, a factor of 23 between the two ends of the same table. The full four-row conversion is reproduced on the pfSense appliance sizer.

A firewall that hits its provisioned speed on a large file transfer and falls short on a workload full of small packets is not broken. It is at its packet-per-second ceiling, and that is a hardware sizing conclusion rather than a fault.

Step 2: measure to the firewall and through the firewall separately

These are different measurements and they diagnose different things. A test terminating on the firewall exercises the firewall’s own network stack and its CPU as an endpoint. A test passing through the firewall between two other hosts exercises forwarding, filtering and state tracking, which is the path production traffic actually takes.

Run both, and note which one is short. If the through-path is slow while the firewall as an endpoint is fast, the problem is in forwarding, filtering, shaping or inspection. If both are slow, look at the interface, the driver and the link before anything else.

Then take the firewall out of the question entirely by testing the uplink from a host connected directly to it, where the network design allows. Nothing configured on a firewall improves a saturated or lossy upstream link, and a surprising share of “pfSense is slow” ends at a failing cable, a negotiated duplex mismatch or an upstream device doing a second layer of address translation.

Step 3: read the interface counters before changing settings

The interface status page reports per-interface errors, collisions and drops. Non-zero and rising error or collision counters point at the physical layer, and no amount of tuning will move them. Confirm the negotiated speed and duplex on both ends of the link, not just on the firewall, because a mismatch is reported as a working link by each side.

This step exists to stop the most expensive mistake in the list, which is tuning a healthy driver to compensate for a bad cable.

Step 4: check for network buffer exhaustion

Netgate calls mbuf exhaustion a common problem on commodity hardware, and its symptoms are easy to misread. mbufs are memory buffers reserved for moving network data. When the firewall runs out of them, network loads that exhaust the pool can cause a kernel panic and reboot, and in some cases interfaces are not initialised or made available by the operating system at all.

The active count appears on the dashboard and is graphed under Status then Monitoring. From the shell, netstat -m shows whether any pool is near exhaustion. Netgate specifically notes that this condition is more common with network cards that use multiple queues or are otherwise optimised for performance over resource usage, and that mbuf usage increases when features such as Limiters are in use.

The documented remedy is a loader tunable raising the cluster count:

kern.ipc.nmbclusters="1000000"

Netgate describes one million clusters as a safe starting point on 64-bit systems with several gigabytes of RAM, and is clear about the cost: fully allocated, that setting consumes about 2.3 GB of physical memory, which is one million clusters at 2 KB each plus 256 bytes per memory buffer. Some cards also need a related value such as kern.ipc.nmbjumbop raised. Systems with little RAM should use a lower figure, and the value must not exceed available kernel memory.

Two operational notes. Loader tunable changes require a reboot to take effect. And a firewall with 2 GB of RAM cannot honour a one-million-cluster setting at all, which is one of the several ways an undersized memory specification turns into a throughput complaint. The memory arithmetic behind that is in pfSense hardware requirements, minimum versus realistic.

Step 5: if the WAN is PPPoE, this is probably the answer

PPPoE deserves its own step because the failure is structural rather than incidental. Network cards capable of multiple queues assign traffic to a queue by hashing. That hashing works well for IPv4 and IPv6 TCP and UDP traffic and fails for other protocols, PPPoE among them. The result is a card that under-performs badly with default settings, and the problem primarily affects systems with several CPU cores, precisely the systems that were bought to be fast.

Netgate’s first recommendation is not a tunable. It is to use the if_pppoe kernel module backend for PPPoE WANs, described as much faster and more efficient and capable of removing the need for further tuning. Try that before anything else.

Where tuning is still needed, the documented entry point is a system or loader tunable setting net.isr.dispatch=deferred, with net.isr.maxthreads and net.isr.numthreads as secondary adjustments. Those two are generally best left at their defaults, which match the core count, though some workloads do better at lower values. Netgate attaches a warning to deferred dispatch: it has historically caused crashes and panics on 32-bit platforms, particularly with ALTQ, with no recent reports on current releases.

Step 6: bisect the packages

Netgate states plainly that features, services and packages lower total potential throughput because they consume resources that would otherwise move traffic, and singles out packages that intercept or inspect traffic such as Snort and Suricata.

So bisect. Disable inspection on one interface and repeat the measurement from step 2. If the number moves, the question is no longer “why is the firewall slow” but “which rule categories and which interfaces are worth their cost”. Enabling every category on every interface is the configuration most likely to produce a firewall that was fast on the day it was built.

Traffic shaping belongs in the same step. A limiter or queue configured with a value that no longer matches the provisioned link caps throughput exactly like failing hardware, and it does so silently. Confirm the configured numbers against the current link speed rather than against the speed the link had when the rules were written.

Step 7: driver-level tuning, last

Only when the steps above are clean is card-specific tuning worth the reboot. The documented candidates include interrupt handling, where a card selects MSI-X, then MSI, then classic interrupts, and either can be disabled as a diagnostic through loader tunables:

hw.pci.enable_msix="0"
hw.pci.enable_msi="0"

Disabling only MSI-X nudges the card toward MSI; disabling both nudges it toward classic interrupts. Netgate also documents hardware offload settings, flow control behaviour and a set of card-specific issues covering Broadcom, Chelsio, Intel and VMware interfaces, each with its own recommended values.

Change one thing, reboot, measure, and record the result. Tuning applied in batches cannot be attributed and cannot be undone with any confidence.

When the answer is that the hardware is the answer

Some slow firewalls are correctly diagnosed as undersized. The tells are consistent: the through-path number tracks packet rate rather than bandwidth, the CPU is saturated during the test, and the shortfall scales with inspection or encryption load rather than appearing at idle.

Netgate’s guidance on where to spend is unambiguous. Inexpensive low-end cards consume significantly more CPU than better-quality cards; the first bottleneck is the CPU; and throughput improves significantly when a better card is paired with a slower processor, while a faster processor paired with a poor card does not deliver a proportional gain. Which component each workload actually stresses is covered in sizing pfSense hardware around what actually limits throughput, and the pfSense appliance sizer converts a target link speed, state count and inspection level into memory and core estimates.

If replacement hardware is on the table, the edition running on it is a separate and much smaller question, covered in pfSense CE versus pfSense Plus.

The checklist, in order

Establish the target number and the traffic shape. Measure to the firewall and through it. Read interface error and collision counters. Check mbuf usage and raise the cluster count if it is near exhaustion, within the memory available. Switch a PPPoE WAN to the if_pppoe backend before tuning anything. Disable inspection on one interface and re-measure. Verify shaper and limiter values against the current link speed. Only then change driver tunables, one at a time, with a reboot and a measurement between each.