228 lines
9.3 KiB
Markdown
228 lines
9.3 KiB
Markdown
# Simulate an unreliable network connection with tc and netem on Linux
|
|
|
|
**Side note**: Using a secondary network interface is recommended since the following commands could make a remote machine unreachable.
|
|
|
|
This a blog post about the basics of `netem` and `tc` on how to modify the **outgoing** traffic. You could modify the incoming traffic with an Intermediate Functional Block pseudo-device in Linux, but I am not too familiar with it and is out of scope for now.
|
|
|
|
# Reasons to simulate an unreliable network connection <a href="#reason" id="reason">#</a>
|
|
|
|
There are various reasons why you want to modify the traffic between devices. The last time we had to ensure that a streaming server in Frankfurt could handle incoming video streams with a high latency over an unreliable connection from the US. The other time we had to provide proof that some SAP modules can't handle the additional latency of a VPN and that the problem is on their side and not ours.
|
|
|
|
Some additional reasons besides troubleshooting could be:
|
|
: testing your network monitoring solution
|
|
: whether your application or server handles unreliable connections well
|
|
: simply do some research.
|
|
|
|
This post tries to provide enough information to help you troubleshoot various problems quickly and simulate certain scenarios.
|
|
|
|
#### Network shaping options
|
|
|
|
`tc` and `netem` provide are variety of options to shape the outgoing traffic.
|
|
|
|
We are going to cover the basics of the following options in this post:
|
|
: adding latency
|
|
: adding jitter
|
|
: sending duplicated packets
|
|
: adding a percentage of packet loss
|
|
: sending corrupt packets
|
|
|
|
Those options can be combined and will cover most of the cases.
|
|
|
|
# Basics of tc <a href="#basics" id="basics">#</a>
|
|
|
|
`tc` stands for 'traffic control' and, as the name implies, is used to configure the traffic control of the Linux kernel and is part of the `iproute2` package. [`Netem`](https://man7.org/linux/man-pages/man8/tc-netem.8.html) stands for 'network emulator' and is controlled by the `tc` command.
|
|
|
|
You can check quickly whether `tc` is available by typing `tc -V`.
|
|
|
|
```bash
|
|
kuser@pleasejustwork:~$ tc -V
|
|
tc utility, iproute2-5.15.0, libbpf 0.5.0
|
|
```
|
|
|
|
#### Show or delete current options
|
|
|
|
Show the currently applied options for an interface:
|
|
: `tc -p qdisc ls dev eth0`
|
|
|
|
```bash
|
|
kuser@pleasejustwork:~$ sudo tc -p qdisc ls dev eth0
|
|
qdisc netem 8001: root refcnt 2 limit 1000 delay 100ms 50ms
|
|
```
|
|
|
|
You can delete all options for a specific interface with the following command:
|
|
: `tc qdisc del dev eth0 root`
|
|
|
|
**Side note**: the following options are temporary and don't survive a reboot!
|
|
|
|
A breakdown of a common `tc` command can be found in the first example below.
|
|
|
|
|
|
#### Limiting to a specific IP or port
|
|
|
|
Unfortunately, it is not that easy to limit the applied options to a specific IP or port. It is possible, but outside the scope of this basic guide. To avoid problems, it is therefore recommended to use a secondary network interface.
|
|
|
|
I might rework this section at some point. For further reading, feel free to check the [official documentation](https://man7.org/linux/man-pages/man8/tc-ematch.8.html) for the filters.
|
|
|
|
# Units used for Parameters for the netem options <a href="#units" id="units">#</a>
|
|
|
|
Almost every 'nenum' option can have one or more parameters. I thought it would make sense to show you the available units before we start with the practical part.
|
|
|
|
#### Rates
|
|
|
|
The bandwidths can be specified with the following units:
|
|
: `bit` - Bits per second
|
|
: `kbit` - Kilobits per second
|
|
: `mbit` - Megabits per second
|
|
: `gbit` - Gigabits per second
|
|
: `tbit` - Terabits per second
|
|
: `bps` - Bytes per second
|
|
: `kbps` - Kilobytes per second
|
|
: `mbps` - Megabytes per second
|
|
: `gbps` - Gigabytes per second
|
|
: `tbps` - Terabytes per second
|
|
|
|
#### Time
|
|
|
|
The time for latency and other options can be specified as follows:
|
|
: `us` - Microseconds
|
|
: `ms` - Milliseconds
|
|
: `s` - Seconds
|
|
|
|
# Netem Options <a href="#options" id="options">#</a>
|
|
|
|
I am going to explain the syntax in the first scenario.
|
|
|
|
As a **reference**, this is the normal ICMP request/ ping over a separate interface.
|
|
|
|
```bash
|
|
kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1
|
|
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
|
|
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=0.458 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=0.520 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=0.453 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=0.420 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=0.513 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=0.412 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=0.550 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=0.548 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=0.402 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=0.376 ms
|
|
|
|
--- 10.10.22.1 ping statistics ---
|
|
10 packets transmitted, 10 received, 0% packet loss, time 9202ms
|
|
rtt min/avg/max/mdev = 0.376/0.465/0.550/0.060 ms
|
|
```
|
|
|
|
## Add Latency / Delay <a href="#latency" id="latency">#</a>
|
|
|
|
The netem latency will be added to the normal latency of the connection.
|
|
|
|
`DELAY := delay TIME [ JITTER [ CORRELATION ]]`
|
|
|
|
`sudo tc qdisc add dev eth0 root netem delay 100ms`
|
|
: `sudo` *# run command as `sudo`*
|
|
: `tc` *# command stands for 'traffic control'*
|
|
: `qdisc` *# stands for 'Queue discipline'*
|
|
: `add|change|del` *# is the action that `tc` should perform to a option*
|
|
: `dev eth0` *# choosing the network interface*
|
|
: `root` *# qdiscs ID*
|
|
: `netem delay 100ms` *# 'netem' option + parameter*
|
|
|
|
**Results**
|
|
|
|
```bash
|
|
kuser@pleasejustwork:~$ sudo tc qdisc add dev eth0 root netem delay 100ms
|
|
[sudo] password for kuser:
|
|
kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1
|
|
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
|
|
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=101 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=100 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=100 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=100 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=100 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=100 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=101 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=100 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=100 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=100 ms
|
|
|
|
--- 10.10.22.1 ping statistics ---
|
|
10 packets transmitted, 10 received, 0% packet loss, time 9013ms
|
|
rtt min/avg/max/mdev = 100.416/100.466/100.586/0.050 ms
|
|
```
|
|
|
|
To **remove** this `tc` rule, send the same command again, but replace `add` with `del`.
|
|
|
|
`sudo tc qdisc del dev eth0 root netem delay 100ms`
|
|
|
|
#### Add Jitter <a href="#jitter" id="jitter">#</a>
|
|
|
|
If you want to add more Jitter - or in other words - variance in latency, add another parameter at the end. This is a plus/minus value.
|
|
|
|
`sudo tc qdisc change dev eth0 root netem delay 100ms 50ms`
|
|
|
|
**Results**
|
|
|
|
```bash
|
|
kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1
|
|
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
|
|
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=105 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=88.6 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=108 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=109 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=130 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=54.5 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=141 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=102 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=124 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=146 ms
|
|
|
|
--- 10.10.22.1 ping statistics ---
|
|
10 packets transmitted, 10 received, 0% packet loss, time 9011ms
|
|
rtt min/avg/max/mdev = 54.495/110.797/145.590/25.366 ms
|
|
```
|
|
|
|
The added latency will be in a range from **50-150ms** from now on.
|
|
|
|
#### Send duplicate packets <a href="#duplicate" id="duplicate">#</a>
|
|
|
|
Sending random duplicate packets over a specific interface:
|
|
: `sudo tc qdisc change dev eth0 root netem duplicate 1%`
|
|
|
|
## Simulate Packet loss <a href="#packet-loss" id="packet-loss">#</a>
|
|
|
|
There are various reasons for packet loss: an unreliable network connection, network congestion, bugs, and so on.
|
|
|
|
To drop random packets of a specific interface, simply use the following command:
|
|
: `sudo tc qdisc add dev eth0 root netem loss 20%`
|
|
|
|
**Results**
|
|
|
|
```bash
|
|
kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1
|
|
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
|
|
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=0.833 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=0.414 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=0.576 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=0.443 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=0.449 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=0.510 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=0.515 ms
|
|
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=0.302 ms
|
|
|
|
--- 10.10.22.1 ping statistics ---
|
|
10 packets transmitted, 8 received, 20% packet loss, time 9221ms
|
|
rtt min/avg/max/mdev = 0.302/0.505/0.833/0.145 ms
|
|
```
|
|
|
|
#### Corrupt packets <a href="#corrupt" id="corrupt">#</a>
|
|
|
|
Introduced an error at a random position of the packet.
|
|
|
|
`sudo tc qdisc change dev eth0 root netem corrupt 10%`
|
|
|
|
# Conclusions
|
|
|
|
As mentioned before, there are more advanced options, but this blog post should cover the basics.
|
|
|
|
---
|