Git Annex.
Took me a while to wrap my head around it, but nothing comes close to it once you set it up.
Edit: should have read the post more carefully, I use Git Annex both locally and on a VPS I rent from openbsd.amsterdam for off-site backups.
this post was submitted on 22 Jul 2023
111 points (95.1% liked)
Selfhosted
40729 readers
571 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 2 years ago
MODERATORS
Somehow "took me a while to wrap my head around it" doesn't make me feel comfortable. Apart from git-annex themselves saying that they aren't a backup system and just a building block to maybe create one, a backup system should imho be dead simple sind easy to understand.
Once you actually start using it it is dead simple and integrates extremely well with stuff you (might) already do.
I have a Git repo which contains my dotfiles + every “large” (annexed) file I want to back up under my home directory.
Git annex automatically tracks where all annexed files are, how many copies there are on various repos, etc.
I add and modify files using mostly standard git commands.
It supports pretty much anything as a “remote”.
It’s extremely simple to restore backups locally or remotely.
Basically Git annex is the Git of backup solutions IME, allowing you extreme flexibility to do exactly what you want, provided you take the time to learn how to do what you want.
Features that are important to me are things like an easy overview of all backup jobs (ideal via a web UI), snapshots going back every day for a week and after that every month. Backup to providers like Backblaze or AWS and the ability to browse these backups and individual snapshots.
I'd assume that you can build all of this with git annex in some way. But I really want something that works out of the box. E.g. install the backup software give it some things to backup and an B2 bucket and then go.
What I'm curious about is that the git-annex site explicitly days that they aren't a backup system, but you describe it as such.
I don’t care about stuff working OOTB - half the fun is messing around with things IMO.
I also don’t care about web UIs and similar features (I always got the impression from selfhosting communities that this is considered important but I never really understood why - I don’t spend all day staring at statistics, and when I need some info I can get it through the terminal usually).
Also, first sentence on Git Annex’s website:
git-annex allows managing large files with git, without storing the file contents in git. It can sync, backup, and archive your data, offline and online. Checksums and encryption keep your data safe and secure.
Not sure why you’re saying it’s not a backup solution.
Efit: I guess the “what git-annex is not” page says this.
To quote a comment by the creator on the same page:
It's definitely possible to use git-annex in backup-like ways, but what I want to discourage is users thinking that just putting files into git-annex means that they have a backup. Proper backups need to be designed, and tested. It helps to use software that is explicitly designed as a backup solution. git-annex is more about file distribution, and some archiving, than backups.
So basically he says this just so people won’t yell at him when they fail to use it as a backup solution correctly.
I don’t care about stuff working OOTB - half the fun is messing around with things IMO.
I generally agree. Backups for me are just something I don't want to tinker with. It's important to me that they work OOTB, are easy to grasp and I have a good overview.
The web interface is important to me because it gives me that overview from any device I'm currently using without needing to type anything into a terminal. The OOTB is important to me since I want to be able to easily set this all up again even without access to my Ansible setup or previous configuration.
To each their own. I'm not saying your way of doing this is wrong. It's just not for me. This is just my reasoning / preferences. It's also the reason something like borg wasn't my chosen solution, even though it's generally considered great.
I understand your position, though I always have access to a terminal pretty much so I still don’t see the point of a web UI.
Though I realize I’m in the minority here.
I do once a day rsync my data to another drive. I can restore a file, if I accidentaly deleted it. Important stuff goes encrypted via rclone additionaly to a hetzner storagebox.
To back up my Synology: My first level is an old Synology, the second is Amazon Glacier.
I use OneDrive. Buy the Costco subscription and get like 15 months for around 110 CAD. GIVES 6 TB. I create some fake accountsink the sharing to my main account. I have an encrypted rxlone share for some things and others I GPG encryot the tar before sending it up. Been working fine for a couple years and I have multiple TB backed up.
I use nightly borg backup to a separate box and then that box uses rclone to back up the borg repo offsite. Before running the borg backup I export all databases and docker volumes so they get picked up.
Restic using resticprofile to configure and schedule backup runs.
I have been with idrive since 2009. At the time they were the only ones that allowed backups of network attached storage on their cheaper personal plans. Everyone else saw that as an "enterprise" feature which required a business plan. Which was bullsh*t, because lots of home NAS devices were being sold.
Anyway, I haven't done a recent comparison of services, but I remain happy with idrive.
Thesedays I no longer backup on a computer with a mapped drive, but directly from my NAS which runs the idrive software.
I had a catastrophic dual drive failure a few years ago, one failed and another failed during the raid rebuild! I was able to restore about 1tb of data and didn't lose anything important.
They also offer backup and restore by shipping a drive to you if you want to avoid the huge initial backup or a total restore, but I haven't used that feature.
They do also have a mobile app, but last time I tried it, it wasn't great.
My truenas backs up to B2 Backblaze. Set it up years ago and haven't touched it since.
load more comments
(2 replies)
AWS Glacier. I use the Synology plugin that does it automatically on a schedule.
load more comments
(3 replies)