Hi all, I’m just getting my feet wet in self hosting and have a plan to start with Nextcloud on a Pi 4 for photo backups, and then try other things for calendar, phone backups, media hosting, etc.
One thing I worry about is losing my data. I have heard “if it’s not backed up in two locations, it’s not backed up.” I’m curious what all of you do for backing up the setup. Remote backup to hard drives in the garage? Pay for cloud backup and encrypt it? Just another backup site over wifi in the house?
I’d be most afraid of losing photos and if there were a house fire or something. So my inital thought was a way of backing up to a server in my detached garage in a weather resistent container, but I want to know what you all think. Thanks for any insight.
I put encrypted backups (borg or restic) on a storage box from Hetzner. One local copy on a different drive and one remote. Keep your encryption passwords safe though, otherwise they aren’t worth much.
Oh, and I plan to report status of the cron jobs that run these backup scripts via MQTT and display backup status in Home Assistant. But haven’t started that yet. So far I dump the logs and view them occasionally.
I’m a bit lazy so it’s simple but it works: for the important stuff i have a borg endpoint on onedrive.
I’d recommend using a job monitoring service that will alerts you if it doesn’t get a check in. It’s very useful if your backups fail silently or hang mid-job for some reason.
I use https://healthchecks.io/ since it has a free tier and it works well.
Or setting proper mail support on everything so you get an email with full output whenever something fails. Ubuntu’s postfix doc is really good.
I copy my data to a nas at home. That copies once a day to an off-site nas. Once in a while I connect an external disk to the nas and copy all and disconnect it.
Perfect or not, compliant with backup procedures or not … it works for me and i’m happy having with an air gap backup.
The only concern I see here is the external drive. My experience has been that powered off drives fail more often than constantly-on drives. So my external drives are always powered on, I just run a replication script to them on a schedule.
But you do have good coverage, so that’s a small risk.
I know and I agree. I still want an air-gap backup. Because …it’s air gapped
Your garage is not remote. Remote means somewhere it won’t be affected in case of theft, fire, flood, earthquake, hurricane, that sort of thing.
I would like to see a thief try and take my server rack down a flight of stairs
I couldn’t sleep at night if I didn’t have my data backed up in 6 different places. I spent way too many years as a sysadmin to deal with 2 backups.
ZFS mirrors on my Proxmox server with multiple nodes replicating to each other. Replications of those datasets to zfs.rent. Proxmox backup server taking hourly snapshots and doing it to multiple drives. Rotating USB drives on that PBS server. Backups of the data for each VM and each docker container stack via rsync. Borg backup. Multiple Nextcloud clients with each having their file syncs held locally, then rsynced to a secondary drive.
I could probably come up with a couple more that I’ve forgotten I have running. I got burned once and it made me mad.
I see lots of solutions here, but some explanation of the basics are missing for someone starting out… this is not meant to sound preachy…
RAID is not a backup. It’s just better hardware fault tolerance. Delete does the same thing on RAID as it does one 1 drive.
Everyone syncs / copies / duplicates files somewhere, but you need a way of finding the previous backup in case something was deleted. This can be done with various ways / tech, but the point is - have some history not just 1 copy. Many pointers to 3-2-1 in here, but that also doesn’t mean 3 copies of just today’s data…
Backups are nothing without Restores. Test the backups. Various ways, but do it. Often.
And consider what you’re backing up and why… ie just your data? (Ie photos), or all the config files, databases, operating systems, etc to do a full restore on new metal. If the latter, I recommend keeping your data separate from the OS / config files, etc.
Source: decades of tech disasters 😁
Nextcloud (later Opencloud) and Immich as primary data sources, backup to:
- server itself, on mirrored drives
- NAS
- 2 external drives periodically (1-2x per week)
- off site cloud data storage
This worked and works well for me.
Basically:
-
Small proxmox node (Zimablade) that basically does only operate a Proxmox Backup Server for local clients and fast backup.
-
Offsite ZFS send to a VPS I operate for that purpose. As well as Proxmox Backup Server for VMs,etc. Basically meant as a fast recovery option. (Layer7)
-
Offsite S3 storage backup to a different provider from above. Meant for a medium term backup. (Hetzner and IONOS)
-
Portable HD: I have two different portable HDs. One is hooked up to the Backup server, the other one is in a lock box in my banks safe. The “connected one” does a weekly backup (and is switched off in between). Once in a while (around 6 to 12 weeks, with 12 weeks being the hard maximum) I take the active one to the bank and both drives switch places. That provides a full backup. (WD My Book and Seagate Expansion - the differrnt manufacturers are intentional)
-
Last line of defence: The real real important things (photos of life events-weddings,etc.- important documents,Password DBs) etc.) get burned on a M-disc Archive blue ray. They are also in the bank safe and at a secure third location. They are more meant for “shit hit the fan and I might not be there anymore,but maybe the kids want these”. Additionally they provide a defence against encryption viruses - write once reas many (WORM) has it’s advantages here.
This is another thing to consider: Have detailed descriptions for others how to retrieve your data in case something happens. I operate a private wiki (on an external server) that also gets saved into the M-Discs that has step by step instructions, as they might need to be followed by someone not that tech adept. (Like my In-Laws in case both my wife and I perish.), have notes in my password DB (Vaultwarden, which has a digital heritage/emergency access function and is also exported), in the vault, and a note in my will notifying people about this.
Edit: And: Test your recovery. Almost every data loss I have witnessed in the last years was a recovery problem. Missing encryption keys, data structure issues, etc. I have seen them all. Personally I try to recover a random file (as in: A script tells me which one) twice a year from every method and try a full recovery of each method at least once six month after introduction. Thst being said: It’s nice to have encrypted backups,but that doesn’t help if you can’t find the keys/the software does no longer exist,etc. Currently a LOT of my clients have the same problem: They use Tandberg RDX for backup, including WORM. Now, Tandberg has gone bust and it’s not that unlikely that yhey won’t be able get another RDX drive in 5 or 10 years. Or 20. Which is the legal requirement for some official files here. Well,fuck. They needed to get additional drives asap when the bankruptcy became official.
Friends have used ancient LTOs and now face the same issues - LTOs are not downwards compatible. (That’s why I use “common” technology. It’s extremely likely that I will be able to find a spare BD drive in 20 years,etc.)
Have detailed descriptions for others how to retrieve your data in case something happens.
Lots of great advice here but this stands out as a really good bit that a lot of people (including myself) need to consider.
Weekend proooojeeeeeect! 🎵
Thanks a lot. I expanded it a little bit.
-
I have a file server for copy 1, an external drive with incremental backups for copy 2, and copy 3 is a physically unplugged copy in a firesafe lockbox that I update manually. I don’t use any cloud providers to back up anything.
I mailed my buddy a Raspberry pi with a large hard drive attached and rsyc to it.
With backups two is one and one is none, so you are very much in a right track. Personally I have my stuff running on proxmox VMs with a proxmox backup server (VM as well) storing backups to Hetzner Storagebox. I’m planning to set up a another host in garage to have “local” backups too, as mine is detached as well the risk of both going up in flames in event of fire is pretty low. However, a voltage spike due to lightning on the grid or something else might blow up both hosts so that’s a threat model to be aware of. Also if your connection to garage is over copper it can cause other problems, fibre or wireless is highly recommended.
With backups it’s largely about the bandwidth available. I personally have enough so uploading to cloud is not an issue, but backing up a terabyte of data over 10Mbps connection might not work out at all.
For more info search for 3-2-1 strategy, that should give you plenty of ideas what you need to think about and what are industry best practises about making sure backups are in order.
My friend who lives a thousand miles away swaps hard drives with me that are backups of critical stuff. He keeps my data, I keep his. As others have said your garage is a start but you really want some sort of geographically separate backup.
That’s a really interesting idea. That makes me think it would even be better in the sense that the data would be protected rather than risk a cloud service going out of business or changed their storage location. Not that that is a likely scenario but still.
Just for the sake of conversation, I recently did some crude math on this. I have few friends around who are well capable of running a backup server for me (hardware maintenance and stuff is always needed anyways) and at first it seemed like a good plan. Just get a 4TB SSD/NVME and throw that on a Raspberry Pi (or something small to keep electricity consumption low and setup silent), set up encryption, connect that to my network with wireguard or some other VPN and let it do it’s thing.
But I’d need to purchase everything as setting up a remote location with old hardware is just asking for trouble. The drive alone is 300€ (give or take) and the rest is easily another 100€. Currently my storagebox costs ~10€/month for 5TB. Even if I scored a fantastic black week offer and got everything for -50% discount that hardware with multiple single point of failures would cost nearly 2 years worth of cloud backups. And I’d still owe at least few beers to the friend for the trouble.
Your mileage may obviously vary, there’s a million different scenarios, but for me with my current setup it just makes sense to pick couple cloud providers and let them store my bits instead of getting more hardware to maintain and upgrade.
For my buddy and I our critical data doesn’t change often so once or twice a year when we get together we swap drives again. Simple spinning discs for us. No need for hardware or anything to keep them running. They just sit on a shelf just in case something happens we can hand it back to pull the data back onto a running server
I actually still burn 5-8 BluRay disks about twice a year with my most important data (Photos, memories etc.) which I store at my parents house as true immutable offsite backups. Furthermore I mirror my TrueNAS Backup Server every night at my brother’s place through a VPN connection .
I remember being told cdr aren’t reliable for long term storage. Are Blu-ray better? Looks like there’s something called M Disc too…
That’s why I burn new ones every 6 months. There’s always some nee stuff to add anyways and it’s easier to just replace the whole backup then.
I have three backups. One is my laptop where all the backups initially start. Then that gets copied to a plugin USB SSD. Then another copy goes to my server which has another USB SSD. That means I don’t have an off site backup.
I don’t have a place to host an off site backup and I’m not comfortable or interested in using cloud services. Instead I just decided that if it all goes up in flames. So be it.
It’s just data and backups are just nice a convenience. I’ll be upset but there’s more important things in life to worry about.
I’ve always lived a life of minimalism and to me stuff is stuff. None of it mattered before I was born and none of it will matter after I die. That happiest and most free feeling I ever experienced was when I spent years travelling with only a 34 litre backpack and that’s kind of been my baseline for happiness ever since.
