# Getting started with rclone - Data transmission Rclone is an [open-source](https://github.com/rclone/rclone) cross-platform data synchronization application focusing on cloud services. It can act as the CLI for your cloud storage. Rclone provides a broad set of features, from simple data transfer to mounting your cloud storage. It provides so many features that I will work on more posts and concentrate on the initial setup and data transfer in this post. I wish I had looked into rclone earlier. I will use **Linux** as a reference system in this post. # Further information Instructions for the [download](https://rclone.org/downloads/) and [installation](https://rclone.org/install/) can be found in the official documentation. After the installation, you can get more help via `rclone --help`, `man rclone` or the [official docs](https://rclone.org/docs/). # Configuration File With `rclone config file` you can check the configuration file that rclone would use. In my case, it hasn't been created yet. ```markdown $ rclone config file Configuration file doesn't exist, but rclone will use this path: /home/kuser/.config/rclone/rclone.conf ``` You don't have to create this file. Rclone will create it after you've added the first 'Remote'. # Remotes `Remotes` as the name implies, are remote storages and rclone supports a large [number of providers](https://github.com/rclone/rclone#storage-providers). You can check the current list of remotes with `rclone listremotes` or the interactive `rclone config` command. ```markdown $ rclone listremotes hetzner-storagebox: ``` ```markdown $ rclone config Current remotes: Name Type ==== ==== hetzner-storagebox sftp e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> ``` #### Adding a new remote I've just added one of my StorageBoxes from Hetzner as a test. You can choose `n` and add a new 'remote'. Rclone will suggest a list of providers and if the provider of choice is unavailable, you have to choose the protocol like `sftp`, `ftp`, `webdav`, or something else to synchronise your data. Rclone will ask you for all necessary information. ```markdown [...] 25 / Pcloud \ "pcloud" 26 / Put.io \ "putio" 27 / SSH/SFTP Connection \ "sftp" 28 / Sugarsync \ "sugarsync" 29 / Transparently chunk/split large files \ "chunker" [...] ``` The information then can be found in the configuration file. ```markdown $ rclone config file Configuration file is stored at: /home/kuser/.config/rclone/rclone.conf $ cat ~/.config/rclone/rclone.conf [hetzner-storagebox] type = sftp host = u123123.your-storagebox.de user = u123123 pass = fdsfasggsghsgsdrgrsgsgsrgsgrg use_insecure_cipher = true ``` #### Removing Remote You can either use the interactive menu of `rclone config` or delete the specific remotes section from your configuration file. ```markdown $ rclone config Current remotes: Name Type ==== ==== hetzner-storagebox sftp e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> d Choose a number from below, or type in an existing value 1 > hetzner-storagebox remote> 1 ``` # Show content of Remote You can check the content of the root directory of the added 'remote' to verify a successful connection with the `ls` options. ```markdown ls List the objects in the path with size and path. lsd List all directories/containers/buckets in the path. lsf List directories and objects in remote:path formatted for parsing. lsjson List directories and objects in the path in JSON format. lsl List the objects in path with modification time, size and path. ``` As an example: ```markdown $ rclone lsd hetzner-storagebox:/ -1 2023-02-02 14:47:07 -1 023 -1 2023-11-05 11:09:05 -1 2023 -1 2023-02-02 15:08:55 -1 cam-2 ``` # Basics usage The basic functions of rclone are to **copy**, **move**, **sync** and **check for changes** of files and directories. The basic syntax is: `rclone [options] subcommand src:source_path dst:destination_path` **Side note**: Use the `-v` flag to get a more **verbose** output, which I highly recommend! - You can increase the verbosity further by using `-vv` or `-vvv`. Use the `--dry-run` and `--interactive / -i` flags before larger transfers to check the connection and the actions rclone would take to **prevent data loss.** ```markdown $ rclone -v --dry-run sync ./test-dir hetzner-storagebox:/test-dir 2023/11/07 22:52:57 NOTICE: another-test.txt: Skipped copy as --dry-run is set 2023/11/07 22:52:57 NOTICE: test-image.png: Skipped delete as --dry-run is set 2023/11/07 22:52:57 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Checks: 1 / 1, 100% Deleted: 1 Transferred: 1 / 1, 100% Elapsed time: 0.3s ``` #### Copying data One of the most used functions is the simple data transfer from a source to a destination. In the following example, we transfer a test image from our local directory to our 'remote' Hetzner StorageBox. Usual data transfer: ```markdown ┌──────────────────┐ │ │ │ local─────►cloud │ │ │ └──────────────────┘ ``` ```markdown rclone copy ./test-image.png hetzner-storagebox:/ ``` I then deleted the file with rclone with the following command `rclone deletefile hetzner-storagebox:/test-image.png` and transferred it again with a more verbose output: ```markdown $ rclone -v copy ./test-image.png hetzner-storagebox:/ 2023/11/07 22:22:01 INFO : test-image.png: Copied (new) 2023/11/07 22:22:01 INFO : Transferred: 261.574k / 261.574 kBytes, 100%, 920.919 kBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 0.5s ``` If you copy a **directory**, make sure to label/name the folder on the destination, or you'd copy only the content of the source directory. ```markdown $ rclone lsd hetzner-storagebox:/ -1 2023-02-02 14:47:07 -1 023 -1 2023-11-05 11:09:05 -1 2023 -1 2023-02-02 15:08:55 -1 cam-2 $ rclone -v copy ./test-dir hetzner-storagebox:/test-dir 2023/11/07 22:29:25 INFO : test-image.png: Copied (new) 2023/11/07 22:29:25 INFO : Transferred: 261.574k / 261.574 kBytes, 100%, 815.043 kBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 0.6s $ rclone lsd hetzner-storagebox:/ -1 2023-02-02 14:47:07 -1 023 -1 2023-11-05 11:09:05 -1 2023 -1 2023-02-02 15:08:55 -1 cam-2 -1 2023-11-07 22:29:25 -1 test-dir ``` The source files and directories remain unchanged just to have it written down. If you want to copy things from a **remote storage to your local system**, simply swap the source and destination and choose the file or directory in the source. ```markdown ┌──────────────────┐ │ │ │ cloud─────►local │ │ │ └──────────────────┘ ``` `rclone copy hetzner-storagebox:/test-image.png ./` Another often used use case for rclone is to **copy or sync data from one cloud storage provider to another**, which works pretty well! ```markdown ┌──────────────────┐ │ │ │ cloud─────►cloud │ │ │ └──────────────────┘ ``` ```markdown $ rclone -v copy hetzner-storagebox:/test-dir onedrive-storage:/test-dir-onedrive 2023/11/08 20:01:15 INFO : another-test.txt: Copied (new) 2023/11/08 20:01:15 INFO : Transferred: 261.574k / 261.574 kBytes, 100%, 975.168 kBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 1.9s ``` #### Moving files I'll keep this one short. Just replace `copy` with `move` from the previous section, and rclone will remove the source files after the data transfer. ```markdown $ rclone -v move ./test-image.png hetzner-storagebox:/ 2023/11/07 22:46:26 INFO : test-image.png: Copied (new) 2023/11/07 22:46:26 INFO : test-image.png: Deleted 2023/11/07 22:46:26 INFO : Transferred: 261.574k / 261.574 kBytes, 100%, 975.168 kBytes/s, ETA 0s Checks: 2 / 2, 100% Deleted: 1 Renamed: 1 Transferred: 1 / 1, 100% Elapsed time: 0.5s ``` #### Syncing files Syncing - with `sync` - makes the source and destination identical but only changes the destination. It works on your local device and remote storage. **Side note**: As syncing can cause data loss as it deletes data that is not present in the source, use the `--dry-run` flag beforehand and check what actions rclone would take. ```markdown $ rclone -v sync ./test-dir hetzner-storagebox:/test-dir 2023/11/07 23:06:47 INFO : another-test.txt: Copied (new) 2023/11/07 23:06:47 INFO : test-image.png: Deleted 2023/11/07 23:06:47 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Checks: 1 / 1, 100% Deleted: 1 Transferred: 1 / 1, 100% Elapsed time: 0.4s ``` #### Deleting files You can delete files and directories with `rclone delete` and `rclone deletefile`. ```markdown $ rclone -v delete onedrive-storage:/test-dir-onedrive 2023/11/08 20:16:10 INFO : another-test.txt: Deleted ``` ---