# Simulate an unreliable network connection with tc and netem on Linux **Side note**: Using a secondary network interface is recommended since the following commands could make a remote machine unreachable. This a blog post about the basics of `netem` and `tc` on how to modify the **outgoing** traffic. You could modify the incoming traffic with an Intermediate Functional Block pseudo-device in Linux, but I am not too familiar with it and is out of scope for now. # Reasons to simulate an unreliable network connection # There are various reasons why you want to modify the traffic between devices. The last time we had to ensure that a streaming server in Frankfurt could handle incoming video streams with a high latency over an unreliable connection from the US. The other time we had to provide proof that some SAP modules can't handle the additional latency of a VPN and that the problem is on their side and not ours. Some additional reasons besides troubleshooting could be: : testing your network monitoring solution : whether your application or server handles unreliable connections well : simply do some research. This post tries to provide enough information to help you troubleshoot various problems quickly and simulate certain scenarios. #### Network shaping options `tc` and `netem` provide are variety of options to shape the outgoing traffic. We are going to cover the basics of the following options in this post: : adding latency : adding jitter : sending duplicated packets : adding a percentage of packet loss : sending corrupt packets Those options can be combined and will cover most of the cases. # Basics of tc # `tc` stands for 'traffic control' and, as the name implies, is used to configure the traffic control of the Linux kernel and is part of the `iproute2` package. [`Netem`](https://man7.org/linux/man-pages/man8/tc-netem.8.html) stands for 'network emulator' and is controlled by the `tc` command. You can check quickly whether `tc` is available by typing `tc -V`. ```bash kuser@pleasejustwork:~$ tc -V tc utility, iproute2-5.15.0, libbpf 0.5.0 ``` #### Show or delete current options Show the currently applied options for an interface: : `tc -p qdisc ls dev eth0` ```bash kuser@pleasejustwork:~$ sudo tc -p qdisc ls dev eth0 qdisc netem 8001: root refcnt 2 limit 1000 delay 100ms 50ms ``` You can delete all options for a specific interface with the following command: : `tc qdisc del dev eth0 root` **Side note**: the following options are temporary and don't survive a reboot! A breakdown of a common `tc` command can be found in the first example below. #### Limiting to a specific IP or port Unfortunately, it is not that easy to limit the applied options to a specific IP or port. It is possible, but outside the scope of this basic guide. To avoid problems, it is therefore recommended to use a secondary network interface. I might rework this section at some point. For further reading, feel free to check the [official documentation](https://man7.org/linux/man-pages/man8/tc-ematch.8.html) for the filters. # Units used for Parameters for the netem options # Almost every 'nenum' option can have one or more parameters. I thought it would make sense to show you the available units before we start with the practical part. #### Rates The bandwidths can be specified with the following units: : `bit` - Bits per second : `kbit` - Kilobits per second : `mbit` - Megabits per second : `gbit` - Gigabits per second : `tbit` - Terabits per second : `bps` - Bytes per second : `kbps` - Kilobytes per second : `mbps` - Megabytes per second : `gbps` - Gigabytes per second : `tbps` - Terabytes per second #### Time The time for latency and other options can be specified as follows: : `us` - Microseconds : `ms` - Milliseconds : `s` - Seconds # Netem Options # I am going to explain the syntax in the first scenario. As a **reference**, this is the normal ICMP request/ ping over a separate interface. ```bash kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data. 64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=0.458 ms 64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=0.520 ms 64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=0.453 ms 64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=0.420 ms 64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=0.513 ms 64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=0.412 ms 64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=0.550 ms 64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=0.548 ms 64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=0.402 ms 64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=0.376 ms --- 10.10.22.1 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9202ms rtt min/avg/max/mdev = 0.376/0.465/0.550/0.060 ms ``` ## Add Latency / Delay # The netem latency will be added to the normal latency of the connection. `DELAY := delay TIME [ JITTER [ CORRELATION ]]` `sudo tc qdisc add dev eth0 root netem delay 100ms` : `sudo` *# run command as `sudo`* : `tc` *# command stands for 'traffic control'* : `qdisc` *# stands for 'Queue discipline'* : `add|change|del` *# is the action that `tc` should perform to a option* : `dev eth0` *# choosing the network interface* : `root` *# qdiscs ID* : `netem delay 100ms` *# 'netem' option + parameter* **Results** ```bash kuser@pleasejustwork:~$ sudo tc qdisc add dev eth0 root netem delay 100ms [sudo] password for kuser: kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data. 64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=101 ms 64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=100 ms 64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=100 ms 64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=100 ms 64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=100 ms 64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=100 ms 64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=101 ms 64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=100 ms 64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=100 ms 64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=100 ms --- 10.10.22.1 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9013ms rtt min/avg/max/mdev = 100.416/100.466/100.586/0.050 ms ``` To **remove** this `tc` rule, send the same command again, but replace `add` with `del`. `sudo tc qdisc del dev eth0 root netem delay 100ms` #### Add Jitter # If you want to add more Jitter - or in other words - variance in latency, add another parameter at the end. This is a plus/minus value. `sudo tc qdisc change dev eth0 root netem delay 100ms 50ms` **Results** ```bash kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data. 64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=105 ms 64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=88.6 ms 64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=108 ms 64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=109 ms 64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=130 ms 64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=54.5 ms 64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=141 ms 64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=102 ms 64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=124 ms 64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=146 ms --- 10.10.22.1 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9011ms rtt min/avg/max/mdev = 54.495/110.797/145.590/25.366 ms ``` The added latency will be in a range from **50-150ms** from now on. #### Send duplicate packets # Sending random duplicate packets over a specific interface: : `sudo tc qdisc change dev eth0 root netem duplicate 1%` ## Simulate Packet loss # There are various reasons for packet loss: an unreliable network connection, network congestion, bugs, and so on. To drop random packets of a specific interface, simply use the following command: : `sudo tc qdisc add dev eth0 root netem loss 20%` **Results** ```bash kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data. 64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=0.833 ms 64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=0.414 ms 64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=0.576 ms 64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=0.443 ms 64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=0.449 ms 64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=0.510 ms 64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=0.515 ms 64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=0.302 ms --- 10.10.22.1 ping statistics --- 10 packets transmitted, 8 received, 20% packet loss, time 9221ms rtt min/avg/max/mdev = 0.302/0.505/0.833/0.145 ms ``` #### Corrupt packets # Introduced an error at a random position of the packet. `sudo tc qdisc change dev eth0 root netem corrupt 10%` # Conclusions As mentioned before, there are more advanced options, but this blog post should cover the basics. ---