ittavern.com/items/2023-04-30_long_curl-reference-guide.md

Curl on Linux - Reference Guide

Curl is a powerful tool that is mainly used to transfer data. It has way more functions, but I won't be able to cover everything. This blog post is mainly a reference for later use and not a step-by-step guide. Therefore I won't cover everything in depth.

Most of it should work on other operating systems too, but I'll use Linux as reference. I'll keep this page up-to-date and add more topics in the future.

General

Side note: put the URL into single or double quotes if it contains special characters.

By default, curl writes the received data to stdout and does not encode or decode received data.

A quick example to get you public IP:

curl brrl.net/ip

curl -L brrl.net/ip # -L to get through the HTTP>HTTP if necessary

Saving to disk

You can redirect the content from stdout to another application, save it as a file or download the target file.

Download content to the file:

curl -L -o myip.txt brrl.net/ip # save your public IP to a file called myip.txt in the current directory

If you want to download a file and keep the original name, use the -O (capital 'o') or --remote-name option.

If you want to create a new directory, you can use --create-dirs like this:

curl -L --create-dirs -o path/from/current/dir/myip.txt brrl.net/ip

The permission used is 0750.

Specific interface

You can use the --interface option to use one specific interface. You are free to use the interface name, the IP address, or the hostname.

Specific DNS server

You can choose a specific DNS server with the following option. Multiple DNS servers can be chosen and must be separated by a comma.

--dns-servers 9.9.9.9:53,149.112.112.112:53

Redirects

If you want curl to follow redirects, simply use the -L flag.

Import curl options and targets from the file

Some tasks require many options. To keep it organized, you can import those options from a file with the -K or --config and followed by the name of the file.

Example:

curl --config curl-options.txt https://example.com

Data tranfer limits

You can set up- and download limits with --limit-rate. The default are bytes/second, and you can use K,M,G,T for Kilo-,Mega-,Giga- and Terabyte, respectively.

--limit-rate 10K
--limit-rate 1000
--limit-rate 10M

Parallel function

To let curl transfer data parallel, you can use the -Z or --parallel and choose --parallel-immediate to start immediately.

-Z --parallel-immediate

The default is 50 parallel transfers but can be set with --parallel-max NUMBER.

Continue downloads automatically

Unreliable connections are a pain, and you can tell curl to retry and continue downloads:

--retry 999 --retry-max-time 0 -C -

--retry 999 # retry it 999 times

--retry-max-time 0 # prevent the default timeout between retries

-C - # continue the transfer when you run the command again, and let curl figure out where to continue

Source from StackExchange

Wildcards / Multiple downloads

Side note: make sure to put the full URL into single or double quotes if you work with wildcards and sequences.

Sets

You can tell curl to transfer multiple files by putting the names into curly brac {}

Keep the original name:

curl -O 'http://{domain1,domain2,domain3}.com'

curl -O 'http://domain.com/{uri1,uri2,uri3}'

Rename files:

curl "http://{one,two}.example.com" -o "file_#1.txt"

And you can use multiple sets, as shown in this example:

kuser@pleasejustwork:~/temp/curl$ curl "http://example.com/{1,2}/{3,4}" -o "file_#1_#2.txt"

[1/4]: http://example.com/1/3 --> file_1_3.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0   6404      0 --:--:-- --:--:-- --:--:--  6375

[2/4]: http://example.com/1/4 --> file_1_4.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12753      0 --:--:-- --:--:-- --:--:-- 12816

[3/4]: http://example.com/2/3 --> file_2_3.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12765      0 --:--:-- --:--:-- --:--:-- 12816

[4/4]: http://example.com/2/4 --> file_2_4.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12804      0 --:--:-- --:--:-- --:--:-- 12816l

kuser@pleasejustwork:~/temp/curl$ ls
file_1_3.txt  file_1_4.txt  file_2_3.txt  file_2_4.txt

Sequence

Use [] for alphanumeric sequences:

curl -O 'http://example.com/picture-[1-51].img'

curl -O 'http://example.com/attach-[a-z].img'

It works with leading zeros too.

Nested sequences are not supported!

Adding steps:

curl -O 'http://example.com/picture-[1-50:2].img' # every second picture

Proxies

I am not too familiar with the proxy functions. I normally just use it to download things from Tor.

If you are connected to Tor, you can reach the network through a SOCKS5 socket:

--socks5-hostname localhost:9150 # On Linux

--socks5-hostname localhost:9050 # Windows or Tor in Browser Bundle

with this addition, you can reach the Tor network via curl.

For normal SOCKS5 sockets, you could simply use --socks5 instead of --socks5-hostname, but with --socks5-hostname, the DNS resolution runs on the proxy.

The usual syntax for proxies looks like this, according to the manual:

-x, --proxy [protocol://]host[:port]

curl --proxy http://proxy.example https://example.com

curl --proxy socks5://proxy.example:12345 https://example.com

Another example of HTTP basic auth proxy:

curl --proxy-basic --proxy-user user:password -x http://proxy.example https://example.com

Authentication

Example for basic authentication:

curl -u name:password --basic https://example.com

Example with bearer token:

curl http://username:password@example.com/api/ -H "Authorization: Bearer reallysecuretoken"

Example with oauth2 bearer:

curl --oauth2-bearer "mF_9.B5f-4.1JqM" https://example.com

Example with ssh public key authentication:

curl --pass secret --key file https://example.com

What else

And there is so much more, but I'll leave it like that for now. Things I am going to add in the future: HTTP post/get requests, certificates troubleshooting, up- and downloading data through FTP, sftp, etc., mail /SMTP. I am unfamiliar with those, so I'll test a bunch before I add those topics here.

--