pfSense Slow Speeds and Bufferbloat: Find the Bottleneck

Slow pfSense throughput is rarely fixed by a faster CPU. Check, in order: packet size against packet rate, interface errors, mbuf exhaustion, a PPPoE WAN, inspection packages and stale limiter values. If the Mbps is fine but calls stall during big transfers, that is bufferbloat, and the fix is FQ_CODEL limiters set to the measured WAN speed.

A pfSense firewall that is slower than the link behind it is almost never slow for the reason people assume. The usual assumption is CPU, the usual first action is buying a faster machine, and the usual outcome is the same number on newer hardware. The documented causes cluster somewhere else entirely: network buffers, driver behaviour, link protocol, inspection settings and the shape of the traffic itself.

What follows is a diagnostic order. It is arranged by cost, cheapest first, and every step is meant to be finished before the next one starts. Skipping to the tuning section is how firewalls end up with a page of loader tunables that fix nothing.

Step 1: decide what number you are trying to reproduce

Before touching a setting, write down three figures: the speed the link is provisioned at, the speed observed, and the traffic that was running when it was observed. If the third one is missing, the first two mean very little.

Netgate publishes the reason. Forwarding cost is charged per packet, not per bit, so the same hardware at the same packet rate delivers very different bandwidth depending on frame size. At an example rate of 500,000 packets per second, 64-byte frames yield 244 Mbps and 1500-byte frames yield 5.59 Gbps from that same packet rate, a factor of 23 between the two ends of the same table. The full four-row conversion is reproduced on the pfSense appliance sizer.

A firewall that hits its provisioned speed on a large file transfer and falls short on a workload full of small packets is not broken. It is at its packet-per-second ceiling, and that is a hardware sizing conclusion rather than a fault.

Step 2: run a pfSense bandwidth test to the firewall and through it

These are different measurements and they diagnose different things. A test terminating on the firewall exercises the firewall’s own network stack and its CPU as an endpoint. A test passing through the firewall between two other hosts exercises forwarding, filtering and state tracking, which is the path production traffic actually takes.

Run both, and note which one is short. If the through-path is slow while the firewall as an endpoint is fast, the problem is in forwarding, filtering, shaping or inspection. If both are slow, look at the interface, the driver and the link before anything else.

Then take the firewall out of the question entirely by testing the uplink from a host connected directly to it, where the network design allows. Nothing configured on a firewall improves a saturated or lossy upstream link, and a surprising share of “pfSense is slow” ends at a failing cable, a negotiated duplex mismatch or an upstream device doing a second layer of address translation.

Running the through-path test

Netgate’s iperf package documentation states that iperf running on pfSense is not a suitable way of testing firewall throughput, because traffic initiated or terminated on the firewall performs differently from traffic traversing it. Use the package for the to-firewall number, and iperf3 on hosts either side for the through-path:

1. Start iperf3 -s on one host, such as a server on another VLAN. It listens on TCP 5201 by default, which a filtered segment must temporarily pass.
2. From the other side, run iperf3 -c <server-ip> -t 30 (the default run is 10 seconds), then repeat with -R so the server sends, and with -P 4 for four parallel streams. One slow stream beside a full four-stream run means a per-flow limit.
3. For UDP, always pass -b: without it, iperf3 -u sends at 1 Mbit/s, which looks healthy on any link.

For the internet path, use a test that reports latency under load, such as Waveform or Cloudflare Speed Test from bufferbloat.net’s list. It screens for bufferbloat, covered in step 7, in the same run.

Step 3: read the interface counters before changing settings

The interface status page reports per-interface errors, collisions and drops. Non-zero and rising error or collision counters point at the physical layer, and no amount of tuning will move them. Confirm the negotiated speed and duplex on both ends of the link, not just on the firewall, because a mismatch is reported as a working link by each side.

This step exists to stop the most expensive mistake in the list, which is tuning a healthy driver to compensate for a bad cable.

Step 4: check for network buffer exhaustion

Netgate calls mbuf exhaustion a common problem on commodity hardware, and its symptoms are easy to misread. mbufs are memory buffers reserved for moving network data. When the firewall runs out of them, network loads that exhaust the pool can cause a kernel panic and reboot, and in some cases interfaces are not initialised or made available by the operating system at all.

The active count appears on the dashboard and is graphed under Status then Monitoring. From the shell, netstat -m shows whether any pool is near exhaustion. Netgate specifically notes that this condition is more common with network cards that use multiple queues or are otherwise optimised for performance over resource usage, and that mbuf usage increases when features such as Limiters are in use.

The documented remedy is a loader tunable raising the cluster count:

kern.ipc.nmbclusters="1000000"

Netgate describes one million clusters as a safe starting point on 64-bit systems with several gigabytes of RAM, and is clear about the cost: fully allocated, that setting consumes about 2.3 GB of physical memory, which is one million clusters at 2 KB each plus 256 bytes per memory buffer. Some cards also need a related value such as kern.ipc.nmbjumbop raised. Systems with little RAM should use a lower figure, and the value must not exceed available kernel memory.

Two operational notes. Loader tunable changes require a reboot to take effect. And a firewall with 2 GB of RAM cannot honour a one-million-cluster setting at all, which is one of the several ways an undersized memory specification turns into a throughput complaint. The memory arithmetic behind that is in pfSense workload sizing for VPN, IDS and state tables.

Step 5: if the WAN is PPPoE, this is probably the answer

PPPoE deserves its own step because the failure is structural rather than incidental. Network cards capable of multiple queues assign traffic to a queue by hashing. That hashing works well for IPv4 and IPv6 TCP and UDP traffic and fails for other protocols, PPPoE among them. The result is a card that under-performs badly with default settings, and the problem primarily affects systems with several CPU cores, precisely the systems that were bought to be fast.

Netgate’s first recommendation is not a tunable. It is to use the if_pppoe kernel module backend for PPPoE WANs, described as much faster and more efficient and capable of removing the need for further tuning. Try that before anything else.

Where tuning is still needed, the documented entry point is a system or loader tunable setting net.isr.dispatch=deferred, with net.isr.maxthreads and net.isr.numthreads as secondary adjustments. Those two are generally best left at their defaults, which match the core count, though some workloads do better at lower values. Netgate attaches a warning to deferred dispatch: it has historically caused crashes and panics on 32-bit platforms, particularly with ALTQ, with no recent reports on current releases.

Step 6: bisect the packages

Netgate states plainly that features, services and packages lower total potential throughput because they consume resources that would otherwise move traffic, and singles out packages that intercept or inspect traffic such as Snort and Suricata. How the two engines differ in what they cost is covered in Snort vs Suricata on pfSense.

So bisect. Disable inspection on one interface and repeat the measurement from step 2. If the number moves, the question is no longer “why is the firewall slow” but “which rule categories and which interfaces are worth their cost”. Enabling every category on every interface is the configuration most likely to produce a firewall that was fast on the day it was built.

Traffic shaping belongs in the same step. A limiter or queue configured with a value that no longer matches the provisioned link caps throughput exactly like failing hardware, and it does so silently. Confirm the configured numbers against the current link speed rather than against the speed the link had when the rules were written. The bufferbloat limiters in step 7 are exactly this kind of cap, so an ISP speed upgrade means remeasuring and re-entering both figures.

Step 7: fix bufferbloat on pfSense with FQ_CODEL limiters

If the speed test passes but latency climbs whenever the line is busy, the pfSense fix is two limiters using the FQ_CODEL scheduler, set to the measured WAN download and upload and applied by a floating WAN rule. Netgate’s recipe uses Tail Drop, ECN on, and a queue length of 1000 near 100 Mbit/s, or 3000 to 5000 at gigabit.

Bufferbloat is the undesirable latency that comes from network equipment buffering too much data, so it can hide behind a good Mbps figure; bufferbloat.net treats loaded latency below 15 to 25 ms as under control. pfSense limiters use dummynet rather than ALTQ, and FQ_CODEL pairs flow queueing with CoDel’s delay control. RFC 8290 sets its default target at 5 ms and interval at 100 ms; Netgate says to leave the FQ_CODEL options at their defaults.

The procedure, condensed from Netgate’s CoDel limiter recipe:

1. Run a bufferbloat test from a LAN client on an otherwise idle line and record download, upload and grade. A circuit that already scores well may not need limiters.
2. Under Firewall > Traffic Shaper > Limiters, create WANDown: bandwidth at the measured download, Mask None, Queue Management Algorithm Tail Drop, Scheduler FQ_CODEL, Queue Length 1000 (3000 to 5000 at 1 Gbit/s and up), ECN checked. Save, then use Add New Queue to create WANDownQ with Mask None and Tail Drop.
3. Repeat with WANUp at the measured upload and a WANUpQ queue, then Apply Changes.
4. Under Firewall > Rules > Floating, add a Pass rule: Quick checked, Interface WAN, Direction Out, Address Family IPv4, Protocol Any, Source WAN address, Gateway set to this WAN’s gateway, In / Out pipe WANUpQ / WANDownQ.
5. If the WAN carries IPv6, duplicate the rule with Address Family IPv6 and the LAN subnets as the source, or the post-NAT prefix where the firewall does NPt.
6. Apply, reset states so existing connections use the limiters, and retest. Netgate expects an A or higher in most cases.

Where the bufferbloat fix goes wrong

The pipe order looks inverted on purpose: pfSense names In and Out from the firewall’s own perspective, so on an outbound WAN floating rule In carries client uploads and Out carries downloads. Swapped pipes queue downloads at the upload rate, turning a latency fix into a slow-speed complaint.

If the grade does not improve and every setting matches, Netgate’s first suspect is a bandwidth value above what the circuit actually delivers; bufferbloat.net advises adjusting from the measured speeds until latency stays low while speed stays good. A multi-WAN firewall needs limiters and a floating rule per WAN, with sources that cannot match another WAN’s address, or traffic bound for a non-default WAN can be dropped. Limiters also add a small processing overhead and raise mbuf usage (step 4), and Netgate’s ALTQ scheduler notes say CoDel does not work well at 1 Mbit/s or less.

Step 8: driver-level tuning, last

Only when the steps above are clean is card-specific tuning worth the reboot. The documented candidates include interrupt handling, where a card selects MSI-X, then MSI, then classic interrupts, and either can be disabled as a diagnostic through loader tunables:

hw.pci.enable_msix="0"
hw.pci.enable_msi="0"

Disabling only MSI-X nudges the card toward MSI; disabling both nudges it toward classic interrupts. Netgate also documents hardware offload settings, flow control behaviour and a set of card-specific issues covering Broadcom, Chelsio, Intel and VMware interfaces, each with its own recommended values.

Change one thing, reboot, measure, and record the result. Tuning applied in batches cannot be attributed and cannot be undone with any confidence.

Where the tunables go on pfSense and OPNsense

On pfSense, runtime values go under System > Advanced, System Tunables tab. Loader tunables such as kern.ipc.nmbclusters and the MSI-X switches go in /boot/loader.conf.local, need a reboot, and are not saved in config.xml, so back that file up separately.

OPNsense keeps both kinds under System > Settings > Tunables, writing them to loader.conf and sysctl.conf; sysctl -a lists valid names. Its performance documentation also recommends receive-side scaling (RSS):

RSS is off by default to avoid regressions on NICs that implement it badly, and gives no benefit to Suricata in IPS mode, whose re-injection is limited to one thread. These are OPNsense’s recommendations; Netgate’s pfSense guidance in step 5 leaves the net.isr thread counts at their defaults, so do not copy one list to the other wholesale.

When the answer is that the hardware is the answer

Some slow firewalls are correctly diagnosed as undersized. The tells are consistent: the through-path number tracks packet rate rather than bandwidth, the CPU is saturated during the test, and the shortfall scales with inspection or encryption load rather than appearing at idle.

Netgate’s guidance on where to spend is unambiguous. Inexpensive low-end cards consume significantly more CPU than better-quality cards; the first bottleneck is the CPU; and throughput improves significantly when a better card is paired with a slower processor, while a faster processor paired with a poor card does not deliver a proportional gain. Which component each workload actually stresses is covered in pfSense workload sizing for routing, VPN and IDS, and the pfSense throughput and state-table sizer converts a target link speed, state count and inspection level into memory estimates and planning tiers.

If replacement hardware is on the table, the edition running on it is a separate and much smaller question, covered in pfSense CE versus pfSense Plus.

The checklist, in order

Establish the target number and the traffic shape. Measure to the firewall and through it. Read interface error and collision counters. Check mbuf usage and raise the cluster count if it is near exhaustion, within the memory available. Switch a PPPoE WAN to the if_pppoe backend before tuning anything. Disable inspection on one interface and re-measure. Verify shaper and limiter values against the current link speed. If latency rather than Mbps is the complaint, add FQ_CODEL limiters at the measured WAN rates. Only then change driver tunables, one at a time, with a reboot and a measurement between each.

FAQ

what bufferbloat grade should pfsense get

An A or better. Netgate’s FQ_CODEL limiter recipe says that in most cases the score should reach A or higher once the download and upload limiters and the floating WAN rule are in place. If it does not and the settings match the recipe, Netgate points first at a limiter bandwidth set above what the circuit actually delivers.

how do you run a bandwidth test on pfsense

Run iperf3 between two hosts on opposite sides of the firewall, not on the firewall itself. Netgate says iperf running on pfSense is not a suitable way of testing firewall throughput, so keep its iperf package for host-to-firewall checks and use a latency-aware speed test such as Waveform’s for the internet path.

what is the default pfsense console speed

The default pfSense serial console speed is 115200 bps, and Netgate says almost all hardware works well at that rate. It is set under System > Advanced, Admin Access, Serial Communications, where older installs may still show 9600 or 38400 to match an old BIOS. Changes usually need a reboot and do not affect network throughput.

where are tunables in opnsense

OPNsense tunables live under System > Settings > Tunables, where new entries can be created, and OPNsense writes them to loader.conf and sysctl.conf. Run sysctl -a in a shell to list valid names. Boot-time values such as net.inet.rss.enabled only take effect after a reboot, so schedule the change for a maintenance window.