init - PoC
This commit is contained in:
commit
3484b45045
GPG key ID:
CA295D643074C68C
146 changed files with 10657 additions and 0 deletions
179
items/2023-04-30_long_curl-reference-guide.md
Normal file
179
items/2023-04-30_long_curl-reference-guide.md
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
# Curl on Linux - Reference Guide
|
||||
|
||||
Curl is a powerful tool that is mainly used to transfer data. It has way more functions, but I won't be able to cover everything. This blog post is mainly a reference for later use and not a step-by-step guide. Therefore I won't cover everything in depth.
|
||||
|
||||
Most of it should work on other operating systems too, but I'll use **Linux** as reference. I'll keep this page up-to-date and add more topics in the future.
|
||||
|
||||
# General <a href="#general" id="general">#</a>
|
||||
|
||||
**Side note**: put the URL into single or double quotes if it contains special characters.
|
||||
|
||||
By default, curl writes the received data to **stdout** and does not encode or decode received data.
|
||||
|
||||
A quick example to get you public IP:
|
||||
: `curl brrl.net/ip`
|
||||
: `curl -L brrl.net/ip` # `-L` to get through the HTTP>HTTP if necessary
|
||||
|
||||
#### Saving to disk <a href="#download" id="download">#</a>
|
||||
|
||||
You can redirect the content from stdout to another application, save it as a file or download the target file.
|
||||
|
||||
Download content to the file:
|
||||
: `curl -L -o myip.txt brrl.net/ip` # save your public IP to a file called `myip.txt` in the current directory
|
||||
|
||||
If you want to **download a file** and keep the **original name**, use the `-O` (capital 'o') or `--remote-name` option.
|
||||
|
||||
If you want to create a **new directory**, you can use `--create-dirs` like this:
|
||||
|
||||
`curl -L --create-dirs -o path/from/current/dir/myip.txt brrl.net/ip`
|
||||
|
||||
The **permission** used is 0750.
|
||||
|
||||
|
||||
#### Specific interface <a href="#interface" id="interface">#</a>
|
||||
|
||||
You can use the `--interface` option to use one specific interface. You are free to use the interface name, the IP address, or the hostname.
|
||||
|
||||
#### Specific DNS server <a href="#dns-server" id="dns-server">#</a>
|
||||
|
||||
You can choose a specific DNS server with the following option. Multiple DNS servers can be chosen and must be separated by a comma.
|
||||
|
||||
`--dns-servers 9.9.9.9:53,149.112.112.112:53`
|
||||
|
||||
#### Redirects <a href="#redirects" id="redirects">#</a>
|
||||
|
||||
If you want curl to follow redirects, simply use the `-L` flag.
|
||||
|
||||
#### Import curl options and targets from the file <a href="#import-options" id="import-options">#</a>
|
||||
|
||||
Some tasks require many options. To keep it organized, you can import those options from a file with the `-K` or `--config` and followed by the name of the file.
|
||||
|
||||
Example:
|
||||
: `curl --config curl-options.txt https://example.com`
|
||||
|
||||
#### Data tranfer limits <a href="#transfer-limits" id="transfer-limits">#</a>
|
||||
|
||||
You can set up- and download limits with `--limit-rate`. The default are bytes/second, and you can use `K`,`M`,`G`,`T` for Kilo-,Mega-,Giga- and Terabyte, respectively.
|
||||
|
||||
```markdown
|
||||
--limit-rate 10K
|
||||
--limit-rate 1000
|
||||
--limit-rate 10M
|
||||
```
|
||||
|
||||
#### Parallel function <a href="#parallel" id="parallel">#</a>
|
||||
|
||||
To let curl transfer data parallel, you can use the `-Z` or `--parallel` and choose `--parallel-immediate` to start immediately.
|
||||
|
||||
`-Z --parallel-immediate`
|
||||
|
||||
The **default is 50** parallel transfers but can be set with `--parallel-max NUMBER`.
|
||||
|
||||
#### Continue downloads automatically
|
||||
|
||||
Unreliable connections are a pain, and you can tell curl to retry and continue downloads:
|
||||
: `--retry 999 --retry-max-time 0 -C -`
|
||||
: `--retry 999` # retry it 999 times
|
||||
: `--retry-max-time 0` # prevent the default timeout between retries
|
||||
: `-C -` # continue the transfer when you run the command again, and let curl figure out where to continue
|
||||
|
||||
[Source from StackExchange](https://superuser.com/a/142480)
|
||||
|
||||
|
||||
# Wildcards / Multiple downloads <a href="#wildcards" id="wildcards">#</a>
|
||||
|
||||
**Side note**: make sure to put the full URL into single or double quotes if you work with wildcards and sequences.
|
||||
|
||||
#### Sets <a href="#sets" id="sets">#</a>
|
||||
|
||||
You can tell curl to transfer multiple files by putting the names into curly brac `{}`
|
||||
|
||||
Keep the original name:
|
||||
: `curl -O 'http://{domain1,domain2,domain3}.com'`
|
||||
: `curl -O 'http://domain.com/{uri1,uri2,uri3}'`
|
||||
|
||||
Rename files:
|
||||
: `curl "http://{one,two}.example.com" -o "file_#1.txt"`
|
||||
|
||||
And you can use multiple sets, as shown in this example:
|
||||
|
||||
```bash
|
||||
kuser@pleasejustwork:~/temp/curl$ curl "http://example.com/{1,2}/{3,4}" -o "file_#1_#2.txt"
|
||||
|
||||
[1/4]: http://example.com/1/3 --> file_1_3.txt
|
||||
% Total % Received % Xferd Average Speed Time Time Time Current
|
||||
Dload Upload Total Spent Left Speed
|
||||
100 1256 100 1256 0 0 6404 0 --:--:-- --:--:-- --:--:-- 6375
|
||||
|
||||
[2/4]: http://example.com/1/4 --> file_1_4.txt
|
||||
% Total % Received % Xferd Average Speed Time Time Time Current
|
||||
Dload Upload Total Spent Left Speed
|
||||
100 1256 100 1256 0 0 12753 0 --:--:-- --:--:-- --:--:-- 12816
|
||||
|
||||
[3/4]: http://example.com/2/3 --> file_2_3.txt
|
||||
% Total % Received % Xferd Average Speed Time Time Time Current
|
||||
Dload Upload Total Spent Left Speed
|
||||
100 1256 100 1256 0 0 12765 0 --:--:-- --:--:-- --:--:-- 12816
|
||||
|
||||
[4/4]: http://example.com/2/4 --> file_2_4.txt
|
||||
% Total % Received % Xferd Average Speed Time Time Time Current
|
||||
Dload Upload Total Spent Left Speed
|
||||
100 1256 100 1256 0 0 12804 0 --:--:-- --:--:-- --:--:-- 12816l
|
||||
|
||||
kuser@pleasejustwork:~/temp/curl$ ls
|
||||
file_1_3.txt file_1_4.txt file_2_3.txt file_2_4.txt
|
||||
```
|
||||
|
||||
#### Sequence <a href="#sequences" id="sequences">#</a>
|
||||
|
||||
Use `[]` for alphanumeric sequences:
|
||||
: `curl -O 'http://example.com/picture-[1-51].img'`
|
||||
: `curl -O 'http://example.com/attach-[a-z].img'`
|
||||
|
||||
It works with **leading zeros** too.
|
||||
|
||||
**Nested sequences** are not supported!
|
||||
|
||||
Adding steps:
|
||||
: `curl -O 'http://example.com/picture-[1-50:2].img'` # every second picture
|
||||
|
||||
|
||||
# Proxies <a href="#proxy" id="proxy">#</a>
|
||||
|
||||
I am not too familiar with the proxy functions. I normally just use it to download things from Tor.
|
||||
|
||||
If you are connected to Tor, you can reach the network through a SOCKS5 socket:
|
||||
: `--socks5-hostname localhost:9150` # On Linux
|
||||
: `--socks5-hostname localhost:9050` # Windows or Tor in Browser Bundle
|
||||
: with this addition, you can reach the Tor network via curl.
|
||||
|
||||
For normal SOCKS5 sockets, you could simply use `--socks5` instead of `--socks5-hostname`, but with `--socks5-hostname`, the DNS resolution runs on the proxy.
|
||||
|
||||
The usual syntax for proxies looks like this, according to the manual:
|
||||
: `-x, --proxy [protocol://]host[:port]`
|
||||
: `curl --proxy http://proxy.example https://example.com`
|
||||
: `curl --proxy socks5://proxy.example:12345 https://example.com`
|
||||
|
||||
Another example of HTTP basic auth proxy:
|
||||
: `curl --proxy-basic --proxy-user user:password -x http://proxy.example https://example.com`
|
||||
|
||||
# Authentication <a href="#authentication" id="authentication">#</a>
|
||||
|
||||
Example for basic authentication:
|
||||
: `curl -u name:password --basic https://example.com`
|
||||
|
||||
Example with bearer token:
|
||||
: `curl http://username:password@example.com/api/ -H "Authorization: Bearer reallysecuretoken"`
|
||||
|
||||
Example with oauth2 bearer:
|
||||
: `curl --oauth2-bearer "mF_9.B5f-4.1JqM" https://example.com`
|
||||
|
||||
Example with ssh public key authentication:
|
||||
: `curl --pass secret --key file https://example.com`
|
||||
|
||||
# What else
|
||||
|
||||
And there is so much more, but I'll leave it like that for now. Things I am going to add in the future:
|
||||
HTTP post/get requests, certificates troubleshooting, up- and downloading data through FTP, sftp, etc., mail /SMTP. I am unfamiliar with those, so I'll test a bunch before I add those topics here.
|
||||
|
||||
--
|
||||
Loading…
Add table
Add a link
Reference in a new issue