# Curl on Linux - Reference Guide

Curl is a powerful tool that is mainly used to transfer data. It has way more functions, but I won't be able to cover everything. This blog post is mainly a reference for later use and not a step-by-step guide. Therefore I won't cover everything in depth.

Most of it should work on other operating systems too, but I'll use **Linux** as reference. I'll keep this page up-to-date and add more topics in the future.

# General <a href="#general" id="general">#</a>

**Side note**: put the URL into single or double quotes if it contains special characters.

By default, curl writes the received data to **stdout** and does not encode or decode received data.

A quick example to get you public IP:
: `curl brrl.net/ip`
: `curl -L brrl.net/ip`    # `-L` to get through the HTTP>HTTP if necessary

#### Saving to disk <a href="#download" id="download">#</a>

You can redirect the content from stdout to another application, save it as a file or download the target file.

Download content to the file:
: `curl -L -o myip.txt brrl.net/ip`    # save your public IP to a file called `myip.txt` in the current directory

If you want to **download a file** and keep the **original name**, use the `-O` (capital 'o') or `--remote-name` option.

If you want to create a **new directory**, you can use `--create-dirs` like this:

`curl -L --create-dirs -o path/from/current/dir/myip.txt brrl.net/ip`

The **permission** used is 0750.


#### Specific interface <a href="#interface" id="interface">#</a>

You can use the `--interface` option to use one specific interface. You are free to use the interface name, the IP address, or the hostname.

#### Specific DNS server <a href="#dns-server" id="dns-server">#</a>

You can choose a specific DNS server with the following option. Multiple DNS servers can be chosen and must be separated by a comma.

`--dns-servers 9.9.9.9:53,149.112.112.112:53`

#### Redirects <a href="#redirects" id="redirects">#</a>

If you want curl to follow redirects, simply use the `-L` flag. 

#### Import curl options and targets from the file <a href="#import-options" id="import-options">#</a>

Some tasks require many options. To keep it organized, you can import those options from a file with the `-K` or `--config` and followed by the name of the file.

Example:
: `curl --config curl-options.txt https://example.com`

#### Data tranfer limits <a href="#transfer-limits" id="transfer-limits">#</a>

You can set up- and download limits with `--limit-rate`. The default are bytes/second, and you can use `K`,`M`,`G`,`T` for Kilo-,Mega-,Giga- and Terabyte, respectively. 

```markdown
--limit-rate 10K
--limit-rate 1000
--limit-rate 10M
```

#### Parallel function <a href="#parallel" id="parallel">#</a>

To let curl transfer data parallel, you can use the `-Z` or `--parallel` and choose `--parallel-immediate` to start immediately.

`-Z --parallel-immediate`

The **default is 50** parallel transfers but can be set with `--parallel-max NUMBER`. 

#### Continue downloads automatically

Unreliable connections are a pain, and you can tell curl to retry and continue downloads:
: `--retry 999 --retry-max-time 0 -C -`
: `--retry 999` # retry it 999 times
: `--retry-max-time 0` # prevent the default timeout between retries
: `-C -` # continue the transfer when you run the command again, and let curl figure out where to continue

[Source from StackExchange](https://superuser.com/a/142480)


# Wildcards / Multiple downloads <a href="#wildcards" id="wildcards">#</a>

**Side note**: make sure to put the full URL into single or double quotes if you work with wildcards and sequences.

#### Sets <a href="#sets" id="sets">#</a>

You can tell curl to transfer multiple files by putting the names into curly brac `{}`

Keep the original name:
: `curl -O 'http://{domain1,domain2,domain3}.com'`
: `curl -O 'http://domain.com/{uri1,uri2,uri3}'`

Rename files:
: `curl "http://{one,two}.example.com" -o "file_#1.txt"`

And you can use multiple sets, as shown in this example:

```bash
kuser@pleasejustwork:~/temp/curl$ curl "http://example.com/{1,2}/{3,4}" -o "file_#1_#2.txt"

[1/4]: http://example.com/1/3 --> file_1_3.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0   6404      0 --:--:-- --:--:-- --:--:--  6375

[2/4]: http://example.com/1/4 --> file_1_4.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12753      0 --:--:-- --:--:-- --:--:-- 12816

[3/4]: http://example.com/2/3 --> file_2_3.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12765      0 --:--:-- --:--:-- --:--:-- 12816

[4/4]: http://example.com/2/4 --> file_2_4.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12804      0 --:--:-- --:--:-- --:--:-- 12816l

kuser@pleasejustwork:~/temp/curl$ ls
file_1_3.txt  file_1_4.txt  file_2_3.txt  file_2_4.txt
```

#### Sequence <a href="#sequences" id="sequences">#</a>

Use `[]` for alphanumeric sequences:
: `curl -O 'http://example.com/picture-[1-51].img'`
: `curl -O 'http://example.com/attach-[a-z].img'`

It works with **leading zeros** too.

**Nested sequences** are not supported!

Adding steps:
: `curl -O 'http://example.com/picture-[1-50:2].img'` # every second picture


# Proxies <a href="#proxy" id="proxy">#</a>

I am not too familiar with the proxy functions. I normally just use it to download things from Tor.

If you are connected to Tor, you can reach the network through a SOCKS5 socket:
: `--socks5-hostname localhost:9150` # On Linux
: `--socks5-hostname localhost:9050` # Windows or Tor in Browser Bundle
: with this addition, you can reach the Tor network via curl.

For normal SOCKS5 sockets, you could simply use `--socks5` instead of `--socks5-hostname`, but with `--socks5-hostname`, the DNS resolution runs on the proxy. 

The usual syntax for proxies looks like this, according to the manual:
: `-x, --proxy [protocol://]host[:port]`
: `curl --proxy http://proxy.example https://example.com`
: `curl --proxy socks5://proxy.example:12345 https://example.com`

Another example of HTTP basic auth proxy:
: `curl --proxy-basic --proxy-user user:password -x http://proxy.example https://example.com`

# Authentication <a href="#authentication" id="authentication">#</a>

Example for basic authentication:
: `curl -u name:password --basic https://example.com`

Example with bearer token:
: `curl http://username:password@example.com/api/ -H "Authorization: Bearer reallysecuretoken"`

Example with oauth2 bearer:
: `curl --oauth2-bearer "mF_9.B5f-4.1JqM" https://example.com`

Example with ssh public key authentication:
: `curl --pass secret --key file https://example.com`

# What else

And there is so much more, but I'll leave it like that for now. Things I am going to add in the future:
HTTP post/get requests, certificates troubleshooting, up- and downloading data through FTP, sftp, etc., mail /SMTP. I am unfamiliar with those, so I'll test a bunch before I add those topics here.

--