Backup a Kubernetes PassBolt deployment to a NFS server

PassBolt is a great self-hosted password manager which has support for a highly available deployment using Kubernetes. I have been using PassBolt for almost a year now and while I wish I had the ability to store files into it in a similar fashion as with KeePass, the fact that I have access to my passwords from my computers as well as my mobile device has prompted me to exclusively use PassBolt.

I favoured a deployment based on the Helm chart using MariaDB as the database backend and even though the data is stored on a PersistentVolume hosted on a Ceph cluster which provides data replication and high availability, one has to remember that RAID is not a backup™ and thus, the PassBolt data that maintainers recommend backing up has to be just that – backed up. In this post, I will show you how I backup my Kubernetes-hosted PassBolt deployment to a NFS server.

Kubernetes CronJob

I am not interested in spending days resetting passwords everywhere so the 3-2-1 principle is especially important when it comes to keeping backup copies for my secrets and credentials. Because relying only on volume snapshots for backups would not have been sufficient, I wanted a way to store a copy on my Bacula server as well as another copy in the Cloud.

For this reason, I opted to export both the MariaDB database as well as the PassBolt GPG keys to a NFS share where Bacula is already setup to backup the local file systems. This is accomplished using the following CronJob, which is deployed in the same namespace as my PassBolt workload, namely passbolt. This is important because the CronJob leverages the secrets used by the PassBolt pods.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: passbolt-nfs-backup
spec:
  schedule: "0 1 * * *" # Every day at 01:00.
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: passbolt-nfs-backup
            securityContext:
              runAsUser: 1000
              runAsGroup: 1000
            volumeMounts:
              - name: nfs-passbolt-db-backups
                mountPath: /backups
              - name: nfs-passbolt-gpg-backups
                readOnly: true
                mountPath: /etc/passbolt-gpg
            image: docker.io/bitnami/mariadb:10.6.12-debian-11-r16
            imagePullPolicy: IfNotPresent
            env:
            - name: MARIADB_HOST
              value: "helm-passbolt-mariadb-primary.passbolt.svc.cluster.local"
            - name: MARIADB_DATABASE
              value: "passbolt"
            - name: MARIADB_USER
              value: "a7bedbe93"
            - name: MARIADB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: helm-passbolt-mariadb
                  key: mariadb-password
            - name: TZ
              value: "America/New_York"
            command:
            - /bin/sh
            - -c
            - |
              export DATE=$(date +%Y%m%d_%H%M%S) TMPDIR=$(mktemp -d)
              echo "Backing up /etc/passbolt-gpg/serverkey*.asc"
              cp /etc/passbolt-gpg/serverkey*.asc ${TMPDIR}

              echo "Dumping the ${MARIADB_DATABASE} database from ${MARIADB_USER}:[hidden]@${MARIADB_HOST} to ${TMPDIR}/${MARIADB_DATABASE}_mysqldump_${DATE}.sql"
              mysqldump -h ${MARIADB_HOST} -u${MARIADB_USER} -p${MARIADB_PASSWORD} ${MARIADB_DATABASE} >${TMPDIR}/${MARIADB_DATABASE}_mysqldump_${DATE}.sql

              echo "Archiving GPG keys and MySQL database into PassBolt_Backup_${DATE}.tar.gz"
              cd ${TMPDIR}
              tar -zcvf /backups/PassBolt_Backup_${DATE}.tar.gz ./*

              echo "Deleting the following files from /backups, which are older than 30 days:"
              find /backups -type f -name "PassBolt_Backup_*" -mtime +30 -print -delete
          restartPolicy: OnFailure
          volumes:
          - name: nfs-passbolt-db-backups
            nfs:
              server: omv.homelab.local
              path: /export/passbolt-backups
          - name: nfs-passbolt-gpg-backups
            secret:
              secretName: helm-passbolt-sec-gpg

What’s going on in this CronJob?

Here is a breakdown of the important lines:

Line 6: Defines the frequency at which the CronJob will be executed. Because I’m the only user for my PassBolt workload, taking a single daily backup is more than enough for me. If you have multiple users, you may want to adjust the schedule to run a few times per day.

Lines 13 to 15: This represents the uid/gid associated with the permissions on the NFS share, i.e.: uid=1000 and gid=1000 have read/write permissions on my NFS share. If this is not set properly, the pod may fail with permission denied errors.

Line 22: The image defined here should ideally match the image you have running for the MariaDB pods in your PassBolt workload. It is probably okay if the CronJob’s image does not match the workload’s image but ideally, both should match or at least use similar versions.

Line 26: This represents the service DNS entry for your primary MariaDB replica.

Line 28 and 30: Make sure these two lines match what you have setup in your chart’s values.yaml file.

Line 34: Change this line to match the secret name for your workload.

Line 37: Because the date and time get appended to the backup file names, you may want to adjust this value to match your local time zone.

Lines 42 to 54: These lines are the actual script that creates a temporary directory, copies the GPG keys into it, then calls mysqldump to export the MariaDB database into it and creates a tarball of that temporary directory. Finally, the tarball is copied to the NFS server and backups older than 30 days are deleted.

Line 59 and 60: Change this line to either the hostname or IP address of your NFS server as well as the path to export the backup files into.

Execution logs and output files

With the CronJob running on a daily basis, you can see what was executed in its pod logs. Since I have only been running the CronJob for a few days, no files are currently being purged:

Backing up /etc/passbolt-gpg/serverkey*.asc
Dumping the passbolt database from a7bedbe93:[hidden]@helm-passbolt-mariadb-primary.passbolt.svc.cluster.local to /tmp/tmp.HlwmKXoXds/passbolt_mysqldump_20231212_192119.sql
Archiving GPG keys and MySQL database into PassBolt_Backup_20231212_192119.tar.gz
./passbolt_mysqldump_20231212_192119.sql
./serverkey.asc
./serverkey_private.asc
Deleting the following files from /backups, which are older than 30 days:

Here is the contents of the directory where the backup files are copied onto my NFS server:

user@omv:/export/passbolt-backups# ls -l
total 21724
-rw-r--r-- 1 user users 5521591 Dec 11 21:30 PassBolt_Backup_20231211_213022.tar.gz
-rw-r--r-- 1 user users 5522714 Dec 11 21:32 PassBolt_Backup_20231211_213249.tar.gz
-rw-r--r-- 1 user users 5524165 Dec 11 21:35 PassBolt_Backup_20231211_213520.tar.gz
-rw-r--r-- 1 user users 5666968 Dec 12 01:00 PassBolt_Backup_20231212_010008.tar.gz
user@omv:/export/passbolt-backups#

As for the contents of the tarball, you can see the MariaDB SQL dump as well as the GPG private and public keys:

user@omv:/export/passbolt-backups# tar -ztvf PassBolt_Backup_20231212_010008.tar.gz
-rw-r--r-- 1000/1000  25852235 2023-12-12 01:00 ./passbolt_mysqldump_20231212_010008.sql
-rw-r--r-- 1000/1000      2639 2023-12-12 01:00 ./serverkey.asc
-rw-r--r-- 1000/1000      5704 2023-12-12 01:00 ./serverkey_private.asc
user@omv:/export/passbolt-backups#

User GPG keys

I suspect that each users GPG keys are stored within the database, but the PassBolt maintainers also recommend having each user back up their GPG keys. Sadly, I found that the process is not well documented, here is how to do it:

First, head over to the /app/settings/keys path of your PassBolt web interface, i.e.: https://your-passbolt-url.domain.tld/app/settings/keys and click on the Public and Private buttons to download your GPG keys, as shown in figure 1 below:

Conclusion

With the tarballs now accumulating on a daily basis on my NFS server, I ensure that three copies of my PassBolt data exist at all times by storing two local copies on my PassBolt deployment and another on my NFS server. I also send a third copy off-site in the cloud. Don’t forget to also have your users backup their GPG keys within the PassBolt user interface!