A network card can take over part of the TCP/IP work that the processor would otherwise do:
computing checksums, cutting large blocks into packets, and merging received packets.
These functions are called offloads.Artica switches them on by default on every network card, because they give the best throughput, and lets you switch them off on one interface when its card or driver handles them badly.
The option is in Network > Network interfaces, in the settings of each interface.
¶ What each offload does
| Offload | ethtool name |
What the card does | Cost when it is off |
|---|---|---|---|
| Checksum | rx, tx |
Computes and checks the TCP/UDP/IP checksums of each packet. | The processor computes them for every packet. With tx off the kernel also drops scatter-gather and TSO, which depend on it. |
| Scatter-gather | sg |
Sends a packet from several memory areas without copying them into one buffer. | An extra memory copy per packet. TSO cannot work without it. |
| Segmentation (TSO, GSO) | tso, gso |
The kernel hands over blocks of up to 64 KB, and the card (TSO) or the end of the stack (GSO) cuts them into packets of 1500 bytes. | The kernel builds and processes every 1500-byte packet itself, about 40 times more packets per 64 KB block. |
| Receive coalescing (GRO) | gro |
Merges the packets of the same connection before they climb the network stack. | The same per-packet cost, on the receive side. |
A proxy carries many TCP connections, so segmentation and receive coalescing are where most of the gain is.
On a 10 Gbit/s card with every offload off, a processor core can saturate long before the link does, and users see a slow network while the processor looks almost idle on the other cores.
¶ What Artica applies on each interface
The rule is applied each time the network configuration is built: when you click Apply network configuration, and at every boot.
| Interface | Offloads | Can you change it? |
|---|---|---|
| Physical card (Intel, Broadcom, Mellanox…), bond, bridge | On | Yes, per interface. |
VMware vmxnet3, KVM/QEMU virtio_net |
On, always | No. The hypervisor does this work, and forcing it off made a 10 Gbit/s vmxnet3 crawl. |
e1000 or e1000e inside a virtual machine (VMware, KVM/QEMU, Red Hat, Nutanix, Xen, VirtualBox, Parallels) |
On, always | No. The hypervisor emulates this card. On real hardware, the same drivers follow the physical rule. |
Hyper-V hv_netvsc, Xen vif |
Left as the driver sets them | Yes: Switching them off is applied. |
| Interface in free mode (SPAN) | Off, always | No. A capture port must see packets as they were on the wire. |
On the interface where Suricata listens, GRO stays off even when the other offloads are on.
Suricata switches it off when it starts, and a network rebuild does not switch it back on.
¶ Advantages and drawbacks
¶ With the offloads on (default)
- Higher throughput for the same processor:
large transfers go through the stack in 64 KB blocks instead of 1500-byte packets. - Lower processor load and fewer interrupts, which leaves more processor time for the proxy, the web filter and the antivirus.
- On a virtual machine, the hypervisor does the segmentation it was designed to do.
¶ With the offloads off
- Packet captures (
tcpdump, Wireshark) on the appliance show packets exactly as they leave, with correct checksums. - It works around a card whose driver or firmware handles offloads badly.
- The cost is the one described in the first table:
every packet goes through the processor, and the throughput of a fast card drops sharply.
¶ What you should do
- Leave the offloads on.
This is the right setting for almost every appliance, physical or virtual. - Switch them off on one interface only when you see one of these symptoms on that interface, and it disappears when the offloads are off:
- TCP connections that stall or reset during large transfers while small requests work;
- checksum errors counted by the remote side or by a device in between;
- packet loss that appeared after a driver or firmware change.
- Older Broadcom (
bnx2x,tg3) and Intel cards with outdated firmware are the usual cases.
Updating the card firmware is the lasting fix;
switching off the offloads is the workaround until then. - Switch off one interface at a time, and keep the others on:
The option is per interface for this reason. - Do not use the option to debug a capture: a
tcpdumpon the appliance showingbad cksumon outgoing packets is normal when the offloads are on (see Troubleshooting).
¶ Switch the offloads off on one interface
Open Network > Network interfaces and click the name of the interface. If the row described below is named TCP/IP Checksum Offloading, read the section Appliances that show "TCP/IP Checksum Offloading" further down.
In the first tab of the interface, the row Network offloads (checksum, TSO, GSO, GRO) shows the current state. Active means the offloads are on.
Click the row to switch the offloads off. Artica stores the choice and applies it at once with ethtool: you do not need to apply the network configuration. The row then shows inactive. Click it again to switch the offloads back on.
Some drivers reset the link for a second or two when their offloads change. Prefer a quiet moment when you change the interface that carries the administration or the users' traffic.
On a vmxnet3, virtio_net or virtual e1000 card, the row is read-only and reads Always on: mandatory on this virtual network card, followed by the driver name. On an interface in free mode (SPAN), it is read-only and reads Always off on a SPAN capture port.
¶ Check the result
The Features tab of the interface lists what the card is doing right now, as reported by ethtool -k. With the offloads on, rx checksumming, tx checksumming, scatter gather, tcp segmentation offload, generic segmentation offload and generic receive offload are Active:
The same interface after the offloads were switched off:
From a shell on the appliance:
ethtool -k eth0 | grep -E "^(rx-checksumming|tx-checksumming|scatter-gather|tcp-segmentation-offload|generic-segmentation-offload|generic-receive-offload)"A feature marked [fixed] is locked by the driver or the hypervisor. Nobody can change it, and Artica does not try.
¶ After an upgrade
Earlier versions switched segmentation, scatter-gather and GRO off on every interface, and the former option TCP/IP Checksum Offloading only controlled rx and tx.
That option is no longer used. Its stored value was not reliable either: saving the settings of an interface reset it to off.
- Mandatory virtual cards (
vmxnet3,virtio_net, virtuale1000) are fixed when the Artica daemon starts after the upgrade.
Only the offloads that are off are switched on, and the system sends a notification Network: offloads switched on for virtual network cards. - Physical cards get their offloads at the next Apply network configuration or the next reboot.
Artica does not change them when the daemon restarts, to avoid resetting the link of a production server during an upgrade. - Every interface starts with the offloads on. If an interface needed them off with the previous version, switch them off again from its settings.
¶ Appliances that show "TCP/IP Checksum Offloading"
If the settings of an interface show a row named TCP/IP Checksum Offloading instead of Network offloads (checksum, TSO, GSO, GRO), the console is up to date but the Artica service (articarest) is an older version. The console then keeps the former option, because the older service cannot apply the new one.
With that older service, the option only controls the checksum (rx, tx). At every network build, the service switches scatter-gather, TSO, GSO and GRO off, whatever the row shows. With the row set to Active, ethtool -k therefore reports:
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp-segmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: offTo get every offload back:
- Update Artica so that the service includes the new option. The row is then named Network offloads (checksum, TSO, GSO, GRO).
- Click Apply network configuration, or reboot. Physical cards keep their offloads off until the network is rebuilt.
Until the update, you can switch them on by hand. This lasts until the next network build or the next reboot, when the older service switches them off again. Replace eth0 with the name of the interface:
ethtool -K eth0 rx on tx on sg on tso on gso on gro on¶ REST API
Read the state of an interface:
curl -s --unix-socket /usr/share/artica-postfix/bin/run/articarest.sock \
http://localhost/system/network/interface/features/eth0The answer holds features (the output of ethtool -k), driver, offloads_mandatory, offloads_disabled, span and offloads_mode (on, force-on, keep or disable).
Switch the offloads off, or back on:
curl -s -X POST --unix-socket /usr/share/artica-postfix/bin/run/articarest.sock \
http://localhost/system/network/interface/offloads/eth0/disable
curl -s -X POST --unix-socket /usr/share/artica-postfix/bin/run/articarest.sock \
http://localhost/system/network/interface/offloads/eth0/enableThe call is refused with "Status": false and an Error message when the card is a mandatory virtual card, when the interface is in free mode (SPAN), when the interface has no saved configuration, or while a network build is running (network change in progress, retry later).
¶ Troubleshooting
| Symptom | Cause and action |
|---|---|
| Slow network on a VMware or KVM appliance, processor mostly idle | Check the Features tab. If the offloads are off on a vmxnet3 or virtio_net card, the Artica daemon switches them on when it starts. If they are still off after that, click Apply network configuration. |
| The row is named TCP/IP Checksum Offloading, and SG, TSO, GSO and GRO stay off when it is active | The Artica service is older than the console and only controls the checksum. Update Artica, then apply the network configuration (see Appliances that show "TCP/IP Checksum Offloading"). |
| Offloads still off on a physical card although the row shows Active | The network has not been rebuilt since the upgrade. Click Apply network configuration or reboot. |
tcpdump on the appliance shows bad cksum on outgoing packets |
Normal with the offloads on: the capture sees the packet before the card writes the checksum. Capture on another device to check the real checksums. |
| GRO stays inactive on one interface | Suricata listens on that interface and keeps GRO off. This is intended. |
One feature stays off and shows [fixed] in ethtool -k |
The driver or the hypervisor does not allow the change. |
| The row cannot be clicked | The card is a mandatory virtual card, or the interface is in free mode (SPAN). The text of the row says which. |