BorgBackup - Securely Backup your Homelab Offsite on a Schedule

What is BorgBackup?

BorgBackup is an open-source secure backup utility for Unix-like operating systems.

The backups are:

• Secure and authenticated.
• Compressed with either LZ4, zlib, LZMA or zstd.
• Deduplicated. The same block of data will not be stored more than once.
• Mountable with FUSE.

Prerequisites

• Access to an SSH server somewhere reliable. Could be a friend's homelab or one from here. 'Offsite' is key.
• Disk space on that server equal to the amount of data you wish to backup with some headroom for future snapshots.
• Your machine's root user's SSH public key authorized on that server so Borg can connect without human interaction.
• That server's SSH host key trusted and stored in the root user's known_hosts file. To do this, run ssh-keyscan -p port hostname.tld then append the output to ~/.ssh/known_hosts.

Installing Borg and borgmatic

We'll assume you're running Ubuntu 20.04+. For other distros, refer to the official installation instructions here.

We'll also assume you're logged in as root, which is highly recommended. This is to allow borg to access system directories such as /etc, /var. Same applies when restoring. To correctly restore file permissions and set ownership, borg needs to be run as root.

## login as root
sudo su

## borg
add-apt-repository ppa:costamagnagianfranco/borgbackup
apt update apt install -y borgbackup

## borgmatic
apt install -y python3-pip
pip3 install --upgrade borgmatic

## logout and log back in as root
## then verify installation
borg -V
borgmatic --version

Borgmatic will not be upgraded to newer versions automatically. Upgrade manually with pip3 occasionally to keep it up-to-date.

pip3 install --upgrade borgmatic

Configuration

We'll generate a sample configuration file

generate-borgmatic-config

This generates a sample configuration file at /etc/borgmatic/config.yaml.

You should edit the configuration file to suit your needs, as the generated values are only representative. All options are optional except where indicated, so feel free to ignore anything you don't need.

Mine for example looks like this:

location:
    source_directories:
        - /home
        - /etc
        - /root
        - /var/lib/docker/volumes
        - /crons
        - /certs
        - /mnt/nvme/prism
        - /media/prismoriginals
        - /mnt/nvme/jellyfin

    repositories:
        - ssh://[email protected]:23/./borg ## offsite
        - /mnt/backups/borg ## attached ext drive, redundant, optional

    exclude_patterns:
        - /home/*/.cache ## exclude caches
        - /root/.launchpadlib
        - /root/.vscode-server
        - /root/.cache
        - '*/.vim*.tmp' ## exclude tmps
        - /mnt/nvme/jellyfin/logs ## exclude logs
        - /mnt/nvme/jellyfin/cache

    borgmatic_source_directory: /mnt/nvme/borgdata/.borgmatic
    exclude_caches: true
    keep_exclude_tags: false
    exclude_if_present:
        - .nobackup

storage:
    checkpoint_interval: 300 ## seconds
    borg_base_directory: /mnt/nvme/borgdata ## will contain keyfile
    encryption_passphrase: vsmYPCOqxX6VpgrBPRTi77Q1AMRsEapS ## use long pass
    archive_name_format: "homelab-{now:%Y%m%d_%H%M}" ## use preferred format

retention:
    keep_hourly: 24
    keep_daily: 7
    keep_weekly: 2
    keep_monthly: 1
    prefix: "homelab-"

consistency:
    prefix: "homelab-"

Correctly specify the SSH port number. In my setup, to exclude a new directory from being backed up, all I have to do is create a new file in that directory.

touch /included_path/subdir_to_exclude/.nobackup

In my example, 24 hourly, 7 daily, 2 weekly and 1 monthly snapshots (aka archives) will be kept. I find this to work very well for me. Tune this as you wish.

If you're lost, you can add official comments back to your config file without losing your modifications. Then refer to the comments for help.

generate-borgmatic-config -s /etc/borgmatic/config.yaml --overwrite

Validating the config file

borgmatic comes with a handy tool to validate your config file:

validate-borgmatic-config

‌Initializing the repository

We'll be using the keyfile encryption option, which requires both this file and the passphrase to access backups.

borgmatic -v2 init -e keyfile

In my example the keyfile is saved at /mnt/nvme/borgdata/.config/borg/keys

If you prefer not to deal with keyfiles, you can use the repokey option; the key will be stored on the backup server instead. This is still secure as your passphrase is still required to decrypt that key.

borgmatic -v2 init -e repokey

Dry running your backup

Let's test that borg can connect to the server and backup without issues.

borgmatic -v2 --dry-run

If summary says successful, you're good to go

Scheduling a cronjob

I like to run my backups, consistency checks and prunes every hour. You may want to tweak this to your taste.

Before we add a cronjob, we'll write a script to make future modifications easier. I like to store all of my cron scripts at /crons.

Create a file with your favorite editor at /crons/borg.sh with the following content:

#!/bin/bash
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.local/bin

/usr/local/bin/borgmatic \
    --verbosity -1 \
    --syslog-verbosity 1 \
    prune create check compact \
    --stats

Then let's make that executable:

chmod +x /crons/borg.sh

We're ready to add this to our crontab.

## edit your crontab
crontab -e

## then add this line and save
30 * * * * /crons/borg.sh

This runs every hour at minute 30.

More information about cron schedule syntax here.

Listing archives

borgmatic list

Should return something like this:

homelab-20230110_1531             Tue, 2023-01-10 15:31:47 [2d0958d602a0cc0056576186c12f5ee3f9db3dc8d8815c557bb526b344654da6]
homelab-20230110_1632             Tue, 2023-01-10 16:32:37 [673002d0c57a7bbb86fdcafa5e17b672630f9431bafd601e38afc6066e200bd3]
homelab-20230110_1733             Tue, 2023-01-10 17:33:11 [dc21a6bb7f40121865c6021b420db419bff527cba7a694b4ccf4ea15bf197066]
homelab-20230110_1831             Tue, 2023-01-10 18:31:15 [5e0ee9cef2e1fb438259ec7e16f20833bce5c5754188c14ae67e2889504b4616]
homelab-20230110_1932             Tue, 2023-01-10 19:32:09 [c35595bfcd08d0dea748a054938a4423ba75db999af0efc826face3515c22c16]
homelab-20230110_2031             Tue, 2023-01-10 20:31:08 [1af4adc261063f5958d3b83ff844adaa63d3b96a17e607b153dd63a9d1b62db1]
homelab-20230110_2132             Tue, 2023-01-10 21:32:24 [a72dcd39a4a3b228ba525dd4e71a63c95be513e50e797072bed4a02a95c3bbfa]

Manually backup single directory

Sometimes it's handy to quickly archive a single directory. Since Borgmatic don't support this yet, we have to use borg directly.

Let's set environment variables for repo, passphrase and base directory.

export BORG_REPO=ssh://[email protected]:23/./borg
export BORG_BASE_DIR=/mnt/nvme/borgdata
export BORG_PASSPHRASE="vsmYPCOqxX6VpgrBPRTi77Q1AMRsEapS"

Then to backup /certs we simply run:

borg -v create ::archive_name /certs

The variables can be added to .bashrc/.zshrc so they're loaded every time you log in.

echo "export BORG_REPO=ssh://[email protected]:23/./borg" >> .zshrc
echo "BORG_BASE_DIR=/mnt/nvme/borgdata" >> .zshrc
echo 'BORG_PASSPHRASE="vsmYPCOqxX6VpgrBPRTi77Q1AMRsEapS"' >> .zshrc

Restoring

To restore the path /certs from an archive named homelab-20230110_1531 from the ssh://[email protected]:23/./borg repository into /restore:

mkdir /restore
cd /restore
borgmatic -v2 extract \
    --repository ssh://[email protected]:23/./borg \
    --archive homelab-20230110_1531 \
    --path certs

--path should not contain leading slash (similar to tar) and can be omitted if you want to extract the entire archive.
--repository can be omitted if you only have one repository specified in the borgmatic config file.

Mounting an archive

To mount the path /certs from an archive named homelab-20230110_1531 from the ssh://[email protected]:23/./borg repository at /mnt/borg:

mkdir /mnt/borg
borgmatic -v2 mount \
    --repository ssh://[email protected]:23/./borg \
    --archive homelab-20230110_1531 \
    --path certs \
    --mount-point /mnt/borg

To unmount:

borgmatic -v2 umount \
    --mount-point /mnt/borg

--path can be omitted if you want to mount the entire archive.
--repository can be omitted if you only have one repository specified in the borgmatic config file.

Bonus: get notified when backups fail

We can use a free online (self hosted!) service healthchecks.io to monitor backups and get notified if backups fail.

Grab yourself a free account, create a check, set the cron and the machine's timezone:

Copy the Ping URL and keep it handy, we'll need this later.

Open the file we created before that's in the crontab - /crons/borg.sh in an editor.
We'll make a few modifications here so cron pings the URL at the start and end of the backup process and also sends the log to healthchecks.io.

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.local/bin

curl -m 10 --retry 5 \
    "https://your_ping_url/start" \
    2>/dev/null 1>/dev/null

foo=$(/usr/local/bin/borgmatic \
    --verbosity 1 \
    --syslog-verbosity -1 \
    prune create check compact \
    --stats 2>&1)

curl -m 10 --retry 5 \
    --data-raw "$foo" \
    "https://hc-ping.com/your_ping_url" \
    2>/dev/null 1>/dev/null

Event history on healthchecks.io looks something like this:

Don't forget to logout as the root user!

Conclusion

A backup that cannot be restored is as good as no backup. Please, test your backups! Restore, access some files, make sure they open. Spin up a VM, see if you can restore files there. A backup's no good if you cannot restore to a new machine.

Let's hope it never comes to that but until then, sleep well at night knowing your homelab is automatically backing itself offsite every day