293 lines
9.6 KiB
Markdown
293 lines
9.6 KiB
Markdown
# Getting started with rclone - Data transmission
|
|
|
|
Rclone is an [open-source](https://github.com/rclone/rclone) cross-platform data synchronization application focusing on cloud services. It can act as the CLI for your cloud storage. Rclone provides a broad set of features, from simple data transfer to mounting your cloud storage. It provides so many features that I will work on more posts and concentrate on the initial setup and data transfer in this post.
|
|
|
|
I wish I had looked into rclone earlier.
|
|
|
|
I will use **Linux** as a reference system in this post.
|
|
|
|
# Further information
|
|
|
|
Instructions for the [download](https://rclone.org/downloads/) and [installation](https://rclone.org/install/) can be found in the official documentation.
|
|
|
|
After the installation, you can get more help via `rclone --help`, `man rclone` or the [official docs](https://rclone.org/docs/).
|
|
|
|
# Configuration File
|
|
|
|
With `rclone config file` you can check the configuration file that rclone would use. In my case, it hasn't been created yet.
|
|
|
|
```markdown
|
|
$ rclone config file
|
|
Configuration file doesn't exist, but rclone will use this path:
|
|
/home/kuser/.config/rclone/rclone.conf
|
|
```
|
|
|
|
You don't have to create this file. Rclone will create it after you've added the first 'Remote'.
|
|
|
|
|
|
# Remotes
|
|
|
|
`Remotes` as the name implies, are remote storages and rclone supports a large [number of providers](https://github.com/rclone/rclone#storage-providers).
|
|
|
|
You can check the current list of remotes with `rclone listremotes` or the interactive `rclone config` command.
|
|
|
|
```markdown
|
|
$ rclone listremotes
|
|
hetzner-storagebox:
|
|
```
|
|
|
|
```markdown
|
|
$ rclone config
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
hetzner-storagebox sftp
|
|
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q>
|
|
```
|
|
|
|
#### Adding a new remote
|
|
|
|
I've just added one of my StorageBoxes from Hetzner as a test. You can choose `n` and add a new 'remote'. Rclone will suggest a list of providers and if the provider of choice is unavailable, you have to choose the protocol like `sftp`, `ftp`, `webdav`, or something else to synchronise your data. Rclone will ask you for all necessary information.
|
|
|
|
```markdown
|
|
[...]
|
|
25 / Pcloud
|
|
\ "pcloud"
|
|
26 / Put.io
|
|
\ "putio"
|
|
27 / SSH/SFTP Connection
|
|
\ "sftp"
|
|
28 / Sugarsync
|
|
\ "sugarsync"
|
|
29 / Transparently chunk/split large files
|
|
\ "chunker"
|
|
[...]
|
|
```
|
|
|
|
The information then can be found in the configuration file.
|
|
|
|
```markdown
|
|
$ rclone config file
|
|
Configuration file is stored at:
|
|
/home/kuser/.config/rclone/rclone.conf
|
|
|
|
$ cat ~/.config/rclone/rclone.conf
|
|
[hetzner-storagebox]
|
|
type = sftp
|
|
host = u123123.your-storagebox.de
|
|
user = u123123
|
|
pass = fdsfasggsghsgsdrgrsgsgsrgsgrg
|
|
use_insecure_cipher = true
|
|
```
|
|
|
|
#### Removing Remote
|
|
|
|
You can either use the interactive menu of `rclone config` or delete the specific remotes section from your configuration file.
|
|
|
|
```markdown
|
|
$ rclone config
|
|
Current remotes:
|
|
|
|
Name Type
|
|
==== ====
|
|
hetzner-storagebox sftp
|
|
|
|
e) Edit existing remote
|
|
n) New remote
|
|
d) Delete remote
|
|
r) Rename remote
|
|
c) Copy remote
|
|
s) Set configuration password
|
|
q) Quit config
|
|
e/n/d/r/c/s/q> d
|
|
Choose a number from below, or type in an existing value
|
|
1 > hetzner-storagebox
|
|
remote> 1
|
|
```
|
|
|
|
# Show content of Remote
|
|
|
|
You can check the content of the root directory of the added 'remote' to verify a successful connection with the `ls` options.
|
|
|
|
```markdown
|
|
ls List the objects in the path with size and path.
|
|
lsd List all directories/containers/buckets in the path.
|
|
lsf List directories and objects in remote:path formatted for parsing.
|
|
lsjson List directories and objects in the path in JSON format.
|
|
lsl List the objects in path with modification time, size and path.
|
|
```
|
|
|
|
As an example:
|
|
|
|
```markdown
|
|
$ rclone lsd hetzner-storagebox:/
|
|
-1 2023-02-02 14:47:07 -1 023
|
|
-1 2023-11-05 11:09:05 -1 2023
|
|
-1 2023-02-02 15:08:55 -1 cam-2
|
|
```
|
|
|
|
|
|
# Basics usage
|
|
|
|
The basic functions of rclone are to **copy**, **move**, **sync** and **check for changes** of files and directories.
|
|
|
|
The basic syntax is:
|
|
|
|
`rclone [options] subcommand <parameters> <parameters...> src:source_path dst:destination_path`
|
|
|
|
**Side note**: Use the `-v` flag to get a more **verbose** output, which I highly recommend! - You can increase the verbosity further by using `-vv` or `-vvv`.
|
|
|
|
Use the `--dry-run` and `--interactive / -i` flags before larger transfers to check the connection and the actions rclone would take to **prevent data loss.**
|
|
|
|
```markdown
|
|
$ rclone -v --dry-run sync ./test-dir hetzner-storagebox:/test-dir
|
|
2023/11/07 22:52:57 NOTICE: another-test.txt: Skipped copy as --dry-run is set
|
|
2023/11/07 22:52:57 NOTICE: test-image.png: Skipped delete as --dry-run is set
|
|
2023/11/07 22:52:57 INFO :
|
|
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
|
|
Checks: 1 / 1, 100%
|
|
Deleted: 1
|
|
Transferred: 1 / 1, 100%
|
|
Elapsed time: 0.3s
|
|
```
|
|
|
|
#### Copying data
|
|
|
|
One of the most used functions is the simple data transfer from a source to a destination.
|
|
|
|
In the following example, we transfer a test image from our local directory to our 'remote' Hetzner StorageBox.
|
|
|
|
Usual data transfer:
|
|
|
|
```markdown
|
|
┌──────────────────┐
|
|
│ │
|
|
│ local─────►cloud │
|
|
│ │
|
|
└──────────────────┘
|
|
```
|
|
|
|
```markdown
|
|
rclone copy ./test-image.png hetzner-storagebox:/
|
|
```
|
|
|
|
I then deleted the file with rclone with the following command `rclone deletefile hetzner-storagebox:/test-image.png` and transferred it again with a more verbose output:
|
|
|
|
```markdown
|
|
$ rclone -v copy ./test-image.png hetzner-storagebox:/
|
|
2023/11/07 22:22:01 INFO : test-image.png: Copied (new)
|
|
2023/11/07 22:22:01 INFO :
|
|
Transferred: 261.574k / 261.574 kBytes, 100%, 920.919 kBytes/s, ETA 0s
|
|
Transferred: 1 / 1, 100%
|
|
Elapsed time: 0.5s
|
|
```
|
|
|
|
If you copy a **directory**, make sure to label/name the folder on the destination, or you'd copy only the content of the source directory.
|
|
|
|
```markdown
|
|
$ rclone lsd hetzner-storagebox:/
|
|
-1 2023-02-02 14:47:07 -1 023
|
|
-1 2023-11-05 11:09:05 -1 2023
|
|
-1 2023-02-02 15:08:55 -1 cam-2
|
|
|
|
$ rclone -v copy ./test-dir hetzner-storagebox:/test-dir
|
|
2023/11/07 22:29:25 INFO : test-image.png: Copied (new)
|
|
2023/11/07 22:29:25 INFO :
|
|
Transferred: 261.574k / 261.574 kBytes, 100%, 815.043 kBytes/s, ETA 0s
|
|
Transferred: 1 / 1, 100%
|
|
Elapsed time: 0.6s
|
|
|
|
$ rclone lsd hetzner-storagebox:/ -1 2023-02-02 14:47:07 -1 023
|
|
-1 2023-11-05 11:09:05 -1 2023
|
|
-1 2023-02-02 15:08:55 -1 cam-2
|
|
-1 2023-11-07 22:29:25 -1 test-dir
|
|
```
|
|
|
|
The source files and directories remain unchanged just to have it written down.
|
|
|
|
If you want to copy things from a **remote storage to your local system**, simply swap the source and destination and choose the file or directory in the source.
|
|
|
|
```markdown
|
|
┌──────────────────┐
|
|
│ │
|
|
│ cloud─────►local │
|
|
│ │
|
|
└──────────────────┘
|
|
```
|
|
|
|
`rclone copy hetzner-storagebox:/test-image.png ./`
|
|
|
|
Another often used use case for rclone is to **copy or sync data from one cloud storage provider to another**, which works pretty well!
|
|
|
|
```markdown
|
|
┌──────────────────┐
|
|
│ │
|
|
│ cloud─────►cloud │
|
|
│ │
|
|
└──────────────────┘
|
|
```
|
|
|
|
```markdown
|
|
$ rclone -v copy hetzner-storagebox:/test-dir onedrive-storage:/test-dir-onedrive
|
|
2023/11/08 20:01:15 INFO : another-test.txt: Copied (new)
|
|
2023/11/08 20:01:15 INFO :
|
|
Transferred: 261.574k / 261.574 kBytes, 100%, 975.168 kBytes/s, ETA 0s
|
|
Transferred: 1 / 1, 100%
|
|
Elapsed time: 1.9s
|
|
```
|
|
|
|
#### Moving files
|
|
|
|
I'll keep this one short. Just replace `copy` with `move` from the previous section, and rclone will remove the source files after the data transfer.
|
|
|
|
```markdown
|
|
$ rclone -v move ./test-image.png hetzner-storagebox:/
|
|
2023/11/07 22:46:26 INFO : test-image.png: Copied (new)
|
|
2023/11/07 22:46:26 INFO : test-image.png: Deleted
|
|
2023/11/07 22:46:26 INFO :
|
|
Transferred: 261.574k / 261.574 kBytes, 100%, 975.168 kBytes/s, ETA 0s
|
|
Checks: 2 / 2, 100%
|
|
Deleted: 1
|
|
Renamed: 1
|
|
Transferred: 1 / 1, 100%
|
|
Elapsed time: 0.5s
|
|
```
|
|
|
|
#### Syncing files
|
|
|
|
Syncing - with `sync` - makes the source and destination identical but only changes the destination. It works on your local device and remote storage.
|
|
|
|
**Side note**: As syncing can cause data loss as it deletes data that is not present in the source, use the `--dry-run` flag beforehand and check what actions rclone would take.
|
|
|
|
```markdown
|
|
$ rclone -v sync ./test-dir hetzner-storagebox:/test-dir
|
|
2023/11/07 23:06:47 INFO : another-test.txt: Copied (new)
|
|
2023/11/07 23:06:47 INFO : test-image.png: Deleted
|
|
2023/11/07 23:06:47 INFO :
|
|
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
|
|
Checks: 1 / 1, 100%
|
|
Deleted: 1
|
|
Transferred: 1 / 1, 100%
|
|
Elapsed time: 0.4s
|
|
```
|
|
|
|
|
|
#### Deleting files
|
|
|
|
You can delete files and directories with `rclone delete` and `rclone deletefile`.
|
|
|
|
```markdown
|
|
$ rclone -v delete onedrive-storage:/test-dir-onedrive
|
|
2023/11/08 20:16:10 INFO : another-test.txt: Deleted
|
|
```
|
|
|
|
---
|