How to use Duplicacy to backup to Storj cloud storage
Introduction
Duplicacy is a cross-platform cloud backup tool with extensive cloud support and client-side encryption.
It is available as a web-based GUI or a command line tool. It has excellent performance and uses Lock Free Deduplication.
The software does require a license, but the command-line interface (CLI) version is free for personal use.
We'll use the CLI version for this tutorial to back up to Storj cloud storage.
Here is a list of supported backends.
Installation
Packages
Pre-compiled binaries are available for Linux, macOS, and Windows directly from the Duplicacy GitHub repository.
Download the latest version for your system. The current version is 3.1.0.
$ sudo wget -O /usr/local/bin/duplicacy https://github.com/gilbertchen/duplicacy/releases/download/v3.1.0/duplicacy_linux_x64_3.1.0
Change the file mode bit.
$ sudo chmod 755 /usr/local/bin/duplicacy
Preparing a new repository
The duplicacy init command initializes the storage.
duplicacy init [command options] <snapshot id> <storage url>
The <snapshot id> refers to the name for the backup job.
The <storage url> refers to the storage location for the backup job.
The duplicacy init command has multiple options, but we'll only be using the following:
-storage-name <name> The name assigned to the storage.
-repository <path> The directory to back up.
We'll initialize the repository using the duplicacy init command and these options in the next section.
Storj cloud storage
The Storj cloud provider offers decentralized storage built for performance with an open-source architecture.
The free tier offers 150 GB of free storage.
To backup data via Storj, create a storage bucket and API key from their website.
Sign up for a free account.
Create a new storage bucket and passphrase. The passphrase is not used for Duplicacy because you have an API key. However, it is a requirement for the storage bucket.
Create an API key. Navigate to the Access page and click the Create Keys for CLI link (rightmost option). Provide name, permissions, and optionally buckets, and select Create Keys.
Copy and save the Satellite Address and API key in a safe place or download them because they will only appear once.
You will need the Satellite Address and API key to initialize your repository.
Here is the format:
$ duplicacy init -storage-name <storage name> -repository <path to directory to backup> <storage id> <storage url>
The <storage url> for Storj uses the following format:
storj://satellite_address/bucket
Here is the complete command for this example:
$ duplicacy init -storage-name storjphotos -repository /home/poochie/Photos photosbackup storj://PrseFhdxYxKicsiFmxK@us1.storj.io:7777/photosbackup
Here is the output:
$ duplicacy init -storage-name storjphotos -repository /home/poochie/Photos photosbackup storj://PrseFhdxYxKicsiFmxK@us1.storj.io:7777/photosbackup
Enter the API access key:
Enter the passphrase:
/home/poochie/Photos will be backed up to storj://PrseFhdxYxKicsiFmxK@us1.storj.io:7777/photosbackup with id photosbackup
Please note the following prompts:
Enter the API access key:
Enter the passphrase:
Copy and paste in your API key.
Copy and paste in a passphrase.
Saving configuration information
Duplicacy automatically creates a configuration folder named .duplicacy when the initialization is complete. The preferences file in this directory contains the configuration data.
The duplicacy set command allows you to store your keys and passphrase in the configuration file to avoid manually inputting them when backing up or restoring files.
Setting this will allow you to automate your backups with bash scripts or schedule them via cron jobs.
duplicacy set -storage <storage name> -key -value
Go to the official Duplicacy wiki for a complete list of environment variables.
We'll be setting the following variable keys:
<storagename>_storj_key
<storagename>_storj_passphrase
Here is the complete command for our example:
$ duplicacy set -storage storjphotos -key 'storjphotos_storj_key' -value 'MyApiKey'
$ duplicacy set -storage storjphotos -key 'storjphotos_storj_passphrase' -value 'Mypassphrase'
You can run the backup command without being prompted to provide your credentials when finished.
Backing up
Use the duplicacy backup command to back up your data.
duplicacy backup -storage <storagename>
Here is the complete command using our example:
$ duplicacy backup -storage storjphotos
Here is the output:
$ duplicacy backup -storage storjphotos
Repository set to /home/poochie/Photos
Storage set to storj://PrseFhdxYxKicsiFmxK@us1.storj.io:7777/photosbackup
No previous backup found
Indexing /home/poochie/Photos
Parsing filter file /home/poochie/.duplicacy/filters
Loaded 0 include/exclude pattern(s)
Listing all chunks
Packed profile.png (133686)
Backup for /home/poochie/Photos at revision 1 completed
Duplicacy assigns a revision number starting with one and increments it every time you run the backup.
Here is what a second backup will look like if there are no file changes.
$ duplicacy backup -storage storjphotos
Repository set to /home/poochie/Photos
Storage set to storj://PrseFhdxYxKicsiFmxK@us1.storj.io:7777/photosbackup
Last backup at revision 1 found
Indexing /home/poochie/Photos
Parsing filter file /home/poochie/.duplicacy/filters
Loaded 0 include/exclude pattern(s)
Backup for /home/poochie/Photos at revision 2 completed
Restoring files
Use the duplicacy restore command to restore files from your backups.
duplicacy restore [command options] [--] [pattern] ...
The duplicacy restore command has multiple options, but we'll only be discussing the following:
-r <revision> - revision number of the snapshot
-stats - show statistics
-overwrite - overwrite existing files
-storage <storage name> - name of storage
To restore the entire snapshot from the initial backup, run the following command:
$ duplicacy restore -storage storjphotos -stats -r 1
Here is the output:
$ duplicacy restore -storage storjphotos -stats -r 1
Repository set to /home/poochie/Photos
Storage set to storj://PrseFhdxYxKicsiFmxK@us1.storj.io:7777/photosbackup
Loaded 0 include/exclude pattern(s)
Forcing in-place mode with a non-default preference path
Parsing filter file /home/poochie/.duplicacy/filters
Loaded 0 include/exclude pattern(s)
Restoring /home/poochie/Photos to revision 1
Downloaded chunk 1 size 133686, 131KB/s 00:00:01 100.0%
Downloaded profile.png (133686)
Restored /home/poochie/Photos to revision 1
Files: 1 total, 131K bytes
Downloaded 1 file, 131K bytes, 1 chunks
Skipped 0 file, 0 bytes
Total running time: 00:00:01
The restore command only restores missing files. It won't overwrite any existing files without specifying the overwrite option.
The overwrite option only overwrites existing files that have changed. The restore command will ignore the overwrite option if there are no changes.
To restore specific files, specify a pattern.
To restore all files with the .png extension:
$ duplicacy restore -storage storjphotos -stats -r 1 -- '*.png'
To restore a file named profile.png:
$ duplicacy restore -storage storjphotos -stats -r 1 profile.png
See more details about the restore option.
Listing files in the backups
Use the duplicacy list command is used to list files in the backup snapshots.
duplicacy list [command options]
The duplicacy list command has multiple options, but we'll only be discussing the following:
-r - revision number of the snapshot
-files - print the file list in each snapshot
-storage <storage name> - name of storage
To list the files from a 4th backup:
$ duplicacy list -storage storjphotos -r 4 -files
Here is the output:
$ duplicacy list -storage storjphotos -r 4 -files
Repository set to /home/poochie/Photos
Storage set to storj://PrseFhdxYxKicsiFmxK@us1.storj.io:7777/photosbackup
Snapshot photosbackup revision 4 created at 2022-11-11 15:28
Files: 3
fbf8fb3184a7e2c55875707f profile.png
298cefc3919c3ad78c5a50 redhat.png
a23217febc1e446aa47812 test.log
Total size: 183974, file chunks: 3, metadata chunks: 3
Conclusion
Duplicacy is a powerful backup tool. This tutorial is only a brief introduction to what it has to offer.
Check out the Duplicacy forum for complete documentation and helpful community support.