init - PoC
This commit is contained in:
commit
3484b45045
GPG key ID:
CA295D643074C68C
146 changed files with 10657 additions and 0 deletions
293
items/2023-11-08_long_getting-started-with-rclone.md
Normal file
293
items/2023-11-08_long_getting-started-with-rclone.md
Normal file
|
|
@ -0,0 +1,293 @@
|
|||
# Getting started with rclone - Data transmission
|
||||
|
||||
Rclone is an [open-source](https://github.com/rclone/rclone) cross-platform data synchronization application focusing on cloud services. It can act as the CLI for your cloud storage. Rclone provides a broad set of features, from simple data transfer to mounting your cloud storage. It provides so many features that I will work on more posts and concentrate on the initial setup and data transfer in this post.
|
||||
|
||||
I wish I had looked into rclone earlier.
|
||||
|
||||
I will use **Linux** as a reference system in this post.
|
||||
|
||||
# Further information
|
||||
|
||||
Instructions for the [download](https://rclone.org/downloads/) and [installation](https://rclone.org/install/) can be found in the official documentation.
|
||||
|
||||
After the installation, you can get more help via `rclone --help`, `man rclone` or the [official docs](https://rclone.org/docs/).
|
||||
|
||||
# Configuration File
|
||||
|
||||
With `rclone config file` you can check the configuration file that rclone would use. In my case, it hasn't been created yet.
|
||||
|
||||
```markdown
|
||||
$ rclone config file
|
||||
Configuration file doesn't exist, but rclone will use this path:
|
||||
/home/kuser/.config/rclone/rclone.conf
|
||||
```
|
||||
|
||||
You don't have to create this file. Rclone will create it after you've added the first 'Remote'.
|
||||
|
||||
|
||||
# Remotes
|
||||
|
||||
`Remotes` as the name implies, are remote storages and rclone supports a large [number of providers](https://github.com/rclone/rclone#storage-providers).
|
||||
|
||||
You can check the current list of remotes with `rclone listremotes` or the interactive `rclone config` command.
|
||||
|
||||
```markdown
|
||||
$ rclone listremotes
|
||||
hetzner-storagebox:
|
||||
```
|
||||
|
||||
```markdown
|
||||
$ rclone config
|
||||
Current remotes:
|
||||
|
||||
Name Type
|
||||
==== ====
|
||||
hetzner-storagebox sftp
|
||||
|
||||
e) Edit existing remote
|
||||
n) New remote
|
||||
d) Delete remote
|
||||
r) Rename remote
|
||||
c) Copy remote
|
||||
s) Set configuration password
|
||||
q) Quit config
|
||||
e/n/d/r/c/s/q>
|
||||
```
|
||||
|
||||
#### Adding a new remote
|
||||
|
||||
I've just added one of my StorageBoxes from Hetzner as a test. You can choose `n` and add a new 'remote'. Rclone will suggest a list of providers and if the provider of choice is unavailable, you have to choose the protocol like `sftp`, `ftp`, `webdav`, or something else to synchronise your data. Rclone will ask you for all necessary information.
|
||||
|
||||
```markdown
|
||||
[...]
|
||||
25 / Pcloud
|
||||
\ "pcloud"
|
||||
26 / Put.io
|
||||
\ "putio"
|
||||
27 / SSH/SFTP Connection
|
||||
\ "sftp"
|
||||
28 / Sugarsync
|
||||
\ "sugarsync"
|
||||
29 / Transparently chunk/split large files
|
||||
\ "chunker"
|
||||
[...]
|
||||
```
|
||||
|
||||
The information then can be found in the configuration file.
|
||||
|
||||
```markdown
|
||||
$ rclone config file
|
||||
Configuration file is stored at:
|
||||
/home/kuser/.config/rclone/rclone.conf
|
||||
|
||||
$ cat ~/.config/rclone/rclone.conf
|
||||
[hetzner-storagebox]
|
||||
type = sftp
|
||||
host = u123123.your-storagebox.de
|
||||
user = u123123
|
||||
pass = fdsfasggsghsgsdrgrsgsgsrgsgrg
|
||||
use_insecure_cipher = true
|
||||
```
|
||||
|
||||
#### Removing Remote
|
||||
|
||||
You can either use the interactive menu of `rclone config` or delete the specific remotes section from your configuration file.
|
||||
|
||||
```markdown
|
||||
$ rclone config
|
||||
Current remotes:
|
||||
|
||||
Name Type
|
||||
==== ====
|
||||
hetzner-storagebox sftp
|
||||
|
||||
e) Edit existing remote
|
||||
n) New remote
|
||||
d) Delete remote
|
||||
r) Rename remote
|
||||
c) Copy remote
|
||||
s) Set configuration password
|
||||
q) Quit config
|
||||
e/n/d/r/c/s/q> d
|
||||
Choose a number from below, or type in an existing value
|
||||
1 > hetzner-storagebox
|
||||
remote> 1
|
||||
```
|
||||
|
||||
# Show content of Remote
|
||||
|
||||
You can check the content of the root directory of the added 'remote' to verify a successful connection with the `ls` options.
|
||||
|
||||
```markdown
|
||||
ls List the objects in the path with size and path.
|
||||
lsd List all directories/containers/buckets in the path.
|
||||
lsf List directories and objects in remote:path formatted for parsing.
|
||||
lsjson List directories and objects in the path in JSON format.
|
||||
lsl List the objects in path with modification time, size and path.
|
||||
```
|
||||
|
||||
As an example:
|
||||
|
||||
```markdown
|
||||
$ rclone lsd hetzner-storagebox:/
|
||||
-1 2023-02-02 14:47:07 -1 023
|
||||
-1 2023-11-05 11:09:05 -1 2023
|
||||
-1 2023-02-02 15:08:55 -1 cam-2
|
||||
```
|
||||
|
||||
|
||||
# Basics usage
|
||||
|
||||
The basic functions of rclone are to **copy**, **move**, **sync** and **check for changes** of files and directories.
|
||||
|
||||
The basic syntax is:
|
||||
|
||||
`rclone [options] subcommand <parameters> <parameters...> src:source_path dst:destination_path`
|
||||
|
||||
**Side note**: Use the `-v` flag to get a more **verbose** output, which I highly recommend! - You can increase the verbosity further by using `-vv` or `-vvv`.
|
||||
|
||||
Use the `--dry-run` and `--interactive / -i` flags before larger transfers to check the connection and the actions rclone would take to **prevent data loss.**
|
||||
|
||||
```markdown
|
||||
$ rclone -v --dry-run sync ./test-dir hetzner-storagebox:/test-dir
|
||||
2023/11/07 22:52:57 NOTICE: another-test.txt: Skipped copy as --dry-run is set
|
||||
2023/11/07 22:52:57 NOTICE: test-image.png: Skipped delete as --dry-run is set
|
||||
2023/11/07 22:52:57 INFO :
|
||||
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
|
||||
Checks: 1 / 1, 100%
|
||||
Deleted: 1
|
||||
Transferred: 1 / 1, 100%
|
||||
Elapsed time: 0.3s
|
||||
```
|
||||
|
||||
#### Copying data
|
||||
|
||||
One of the most used functions is the simple data transfer from a source to a destination.
|
||||
|
||||
In the following example, we transfer a test image from our local directory to our 'remote' Hetzner StorageBox.
|
||||
|
||||
Usual data transfer:
|
||||
|
||||
```markdown
|
||||
┌──────────────────┐
|
||||
│ │
|
||||
│ local─────►cloud │
|
||||
│ │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
```markdown
|
||||
rclone copy ./test-image.png hetzner-storagebox:/
|
||||
```
|
||||
|
||||
I then deleted the file with rclone with the following command `rclone deletefile hetzner-storagebox:/test-image.png` and transferred it again with a more verbose output:
|
||||
|
||||
```markdown
|
||||
$ rclone -v copy ./test-image.png hetzner-storagebox:/
|
||||
2023/11/07 22:22:01 INFO : test-image.png: Copied (new)
|
||||
2023/11/07 22:22:01 INFO :
|
||||
Transferred: 261.574k / 261.574 kBytes, 100%, 920.919 kBytes/s, ETA 0s
|
||||
Transferred: 1 / 1, 100%
|
||||
Elapsed time: 0.5s
|
||||
```
|
||||
|
||||
If you copy a **directory**, make sure to label/name the folder on the destination, or you'd copy only the content of the source directory.
|
||||
|
||||
```markdown
|
||||
$ rclone lsd hetzner-storagebox:/
|
||||
-1 2023-02-02 14:47:07 -1 023
|
||||
-1 2023-11-05 11:09:05 -1 2023
|
||||
-1 2023-02-02 15:08:55 -1 cam-2
|
||||
|
||||
$ rclone -v copy ./test-dir hetzner-storagebox:/test-dir
|
||||
2023/11/07 22:29:25 INFO : test-image.png: Copied (new)
|
||||
2023/11/07 22:29:25 INFO :
|
||||
Transferred: 261.574k / 261.574 kBytes, 100%, 815.043 kBytes/s, ETA 0s
|
||||
Transferred: 1 / 1, 100%
|
||||
Elapsed time: 0.6s
|
||||
|
||||
$ rclone lsd hetzner-storagebox:/ -1 2023-02-02 14:47:07 -1 023
|
||||
-1 2023-11-05 11:09:05 -1 2023
|
||||
-1 2023-02-02 15:08:55 -1 cam-2
|
||||
-1 2023-11-07 22:29:25 -1 test-dir
|
||||
```
|
||||
|
||||
The source files and directories remain unchanged just to have it written down.
|
||||
|
||||
If you want to copy things from a **remote storage to your local system**, simply swap the source and destination and choose the file or directory in the source.
|
||||
|
||||
```markdown
|
||||
┌──────────────────┐
|
||||
│ │
|
||||
│ cloud─────►local │
|
||||
│ │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
`rclone copy hetzner-storagebox:/test-image.png ./`
|
||||
|
||||
Another often used use case for rclone is to **copy or sync data from one cloud storage provider to another**, which works pretty well!
|
||||
|
||||
```markdown
|
||||
┌──────────────────┐
|
||||
│ │
|
||||
│ cloud─────►cloud │
|
||||
│ │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
```markdown
|
||||
$ rclone -v copy hetzner-storagebox:/test-dir onedrive-storage:/test-dir-onedrive
|
||||
2023/11/08 20:01:15 INFO : another-test.txt: Copied (new)
|
||||
2023/11/08 20:01:15 INFO :
|
||||
Transferred: 261.574k / 261.574 kBytes, 100%, 975.168 kBytes/s, ETA 0s
|
||||
Transferred: 1 / 1, 100%
|
||||
Elapsed time: 1.9s
|
||||
```
|
||||
|
||||
#### Moving files
|
||||
|
||||
I'll keep this one short. Just replace `copy` with `move` from the previous section, and rclone will remove the source files after the data transfer.
|
||||
|
||||
```markdown
|
||||
$ rclone -v move ./test-image.png hetzner-storagebox:/
|
||||
2023/11/07 22:46:26 INFO : test-image.png: Copied (new)
|
||||
2023/11/07 22:46:26 INFO : test-image.png: Deleted
|
||||
2023/11/07 22:46:26 INFO :
|
||||
Transferred: 261.574k / 261.574 kBytes, 100%, 975.168 kBytes/s, ETA 0s
|
||||
Checks: 2 / 2, 100%
|
||||
Deleted: 1
|
||||
Renamed: 1
|
||||
Transferred: 1 / 1, 100%
|
||||
Elapsed time: 0.5s
|
||||
```
|
||||
|
||||
#### Syncing files
|
||||
|
||||
Syncing - with `sync` - makes the source and destination identical but only changes the destination. It works on your local device and remote storage.
|
||||
|
||||
**Side note**: As syncing can cause data loss as it deletes data that is not present in the source, use the `--dry-run` flag beforehand and check what actions rclone would take.
|
||||
|
||||
```markdown
|
||||
$ rclone -v sync ./test-dir hetzner-storagebox:/test-dir
|
||||
2023/11/07 23:06:47 INFO : another-test.txt: Copied (new)
|
||||
2023/11/07 23:06:47 INFO : test-image.png: Deleted
|
||||
2023/11/07 23:06:47 INFO :
|
||||
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
|
||||
Checks: 1 / 1, 100%
|
||||
Deleted: 1
|
||||
Transferred: 1 / 1, 100%
|
||||
Elapsed time: 0.4s
|
||||
```
|
||||
|
||||
|
||||
#### Deleting files
|
||||
|
||||
You can delete files and directories with `rclone delete` and `rclone deletefile`.
|
||||
|
||||
```markdown
|
||||
$ rclone -v delete onedrive-storage:/test-dir-onedrive
|
||||
2023/11/08 20:16:10 INFO : another-test.txt: Deleted
|
||||
```
|
||||
|
||||
---
|
||||
Loading…
Add table
Add a link
Reference in a new issue