xkcd.com

Introduction

321 backups is a well known paradigm that advises good backups should include:

  • 3 backups in total
  • 2 backups locally (i.e at home) on different media (think devices)
  • 1 offsite e.g AWS/Backblaze b2 etc

To improve backups of my data I’ve setup:

  • Synology Nas using synology hybrid raid
  • Open Media Vault (OMV)
  • Backblaze/gDrive/Dropbox via Rcloud

Rclone is really awesome - it gives a CLI interface to cloud storage and gives rsync like control as well as a mounted drive for the command line.

I’d recommend using this with https://gist.github.com/kabili207/2cd2d637e5c7617411a666d8d7e97101) . It handles automatically mounting the drive via systemd.

Surprisingly RAID 1 where data is mirrored across disks to provide redundancy is NOT considered as a backup. Instead, its a solution to increase availability of data as in the case where a drive fails, the server can continue using the ‘redundant’ disk. However, should the server fail due to PSU outage, malicious access, motherboard failure etc, then there is the possibility that we could lose our data. Therefore, I prioritised using old hardware for a secondary Nas running OMV instead of running RAID 1.

Automation of backups

  • Power management - this is done as we don’t need availability of the OpenMediaVault 24/7
    • Autostart via BIOS settings the OpenMediaVault server.
    • Autoshutdown using OpenMediaVault settings.
  • Cron jobs to run my backup shell script. This rsyncs across my data to the OpenMediaVault samba shares.

Additional safety features

  • SMART checks were added in to run daily on the OMV drive to attempt to pick up errors in advance
  • OMV alerting configured

Issues

  • If the files being cloned across are very large and the copy cannot be completed before the shutdown starts then a proper backup will not be achieved. *Not resolved
  • Hour changes. Due to the delights of the BST/GMT time zones, it turns out that the Bios respects hour changes while cron jobs do not! *Not resolved
  • Should the copy try to move data across to the remote mounted drive and the server is offline this causes the server to copy data to the local filesystem and not the remote! *See mitigation 1

Mitigation

  • To handle scenarios where the OMV server is offline, a check to the backup script has been added to identify that the server is responding to pings. If it is not then the backup script aborts. This prevents the cron job from copying the backup from the synology to the local disk instead of the OMV disk.

Future work

  • Monitoring/email alerts. Perhaps this could be done with grafana?
  • Data encryption - Some of my personal data in the cloud is not ideal. I didn’t go with this initially as I wanted to avoid the situation where I have data failure + data backups but no access as I’ve lost the keys.
  • Upgrade the synology to support JBOD disk array. Spanning was considered but upon further research this looks grim if a drive fails and data is copied across two of the drives.
  • Try out trueNas
  • Use either btrfs or ZFS to get familiarity with snapshots/versioning of files

Afterthoughts…

Rsync is amazing but…

  • using the –delete flag synchronisation of directories A and B can be achieved e.g If I remove file_a.txt from dir A then when rysnc is run, it will delete from B. Standard rsync doesnt do this deletion. However, if the cmd is not written with care, then there can be nasty surprises! To avoid this, use –dry-run cmd to test the behaviour is as expected

CMR vs SMR hardrives

  • The HD industry has moved to using SMR for consumer drives due to lower cost
  • This is not recommended for NAS drives where heavier use is expected
  • Sequential writes are better
  • “The higher density of SMR drives, combined with its random-read nature, fills a niche between the sequential-access tape storage and the random-access conventional hard drive storage. They are suited to storing data that are unlikely to be modified, but need to be read from any point efficiently. One example of the use case is Dropbox’s Magic Storage system, which runs the on-disk extents in an append-only way.[30] Device-managed SMR disks have also been marketed as “Archive HDDs” due to this property.[31]”
  • Thus, I am using in my Synology Nas - CMR HD (Seagate Ironwolf) and a cheaper SMR drive in my OMV for archival purposes