this post was submitted on 07 Jun 2023
7 points (100.0% liked)

Self Hosted - Self-hosting your services.

10912 readers
1 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules

Important

Beginning of January 1st 2024 this rule WILL be enforced. Posts that are not tagged will be warned and if not fixed within 24h then removed!

Cross-posting

If you see a rule-breaker please DM the mods!

founded 3 years ago
MODERATORS
 

I am currently looking into High Availability for my work setup. I am having some problems understanding how to achive that. I have two servers, one running libvirt and a couple VM, the other one nothing much yet.

To achieve HA with keepalived, I would have to setup the exact same VMs under the second server, right? If that's the case, how would I make sure that the "mirrors" stay equal, If for example the master goes down, the backup takes over, some changes are made in a DB and the master knows nothing about these changes.

Maybe I misunderstood keepalived so far, can somebody provide me with an example setup or hints on how to achieve what I want to do?

Kind Regards

g7s

top 7 comments
sorted by: hot top controversial new old
[–] [email protected] 4 points 1 year ago (1 children)

It's been a few years since I used keepalived so my knowledge might be outdated.

You are correct that the VMs should be in different servers. To test around you can set up on the same, but this shouldn't be done in production environments, if you lose the host, you lose the service.

Keepalived will make sure your service is available in an IP. To say, you have two (it can be configured for more than two) servers with (A) 192.168.0.2 and (B) 192.168.0.3 which provide the service you want to provide. With Keepalived you'll configure a common IP for both of them, let's say 192.168.0.4

While working, server A will be available at 192.168.0.2 and 192.168.0.4 while server B will be available at 192.168.0.3. If server A fails keepalived will "move" 192.168.0.4 to server B, so 192.168.0.2 will not be available and server B will be available at 192.168.0.3 and 192.168.0.4.

No matter which server is up / primary, your service will always be available at 192.168.0.4

For the mirroring part, you need to solve it in another step outside from keepalived. For example, MariaDB provides multimaster replication "out of the box" with galera (the recommendation is at least 3 nodes)

For files, depending on your filesystem you should have to rsync, use some shared units, distribute filesystem (Ceph), ...

[–] [email protected] 1 points 1 year ago

Thank you for the explanation. I might look into heartbeat, as suggested by @arbiter. I understand now, that keepalived is only working on an IP layer, and not helping me with mirroring my actual VM's. For that I will look into other technologies.

[–] [email protected] 2 points 1 year ago (1 children)

If you plan on using this in a production environment, I'd bring in a consultant.

However, I've heard of people in the home-lab sphere use things like heartbeat and drdb. The more nodes the merrier as if you lose connection between the two you'll have a bad time.

[–] [email protected] 1 points 1 year ago (1 children)

I'm working for our department as the only IT-admin, everything runs fine and nightly downtimes for upgrades etc. are fine. However, I want to make it more available. Thanks for the suggestions, I will look into them :)

[–] [email protected] 3 points 1 year ago* (last edited 1 year ago)

Other data replication technologies worth looking in to: GlusterFs, Ceph.

Dependent on your db's they should offer replication out of the box.

You can also implement a load balancer, such as HAProxy or Nginx, to distribute incoming network traffic across multiple VMs

[–] [email protected] 1 points 1 year ago (1 children)

Whatever technology you end up using you should be aware that you will see an order of magnitude or two increase in complexity by running things in a HA way which is very likely to cause some additional downtime instead of reducing it for a while (and possibly even in the long-term).

Network block devices on clusters like Ceph or distributed filesystems have many more failure modes in addition to the ones of the underlying storage hardware due to their distributed nature. Clustered services are similar. You might also see new performance bottlenecks emerge (e.g. your network might be significantly slower in both latency and throughput than modern local SSD or NVMe storage) and new temporarily unavailable services when the failover happens too often.

My advice would be to start running something like that only on a dev/test system that sees some use for a few months at least to learn what to do when things go wrong before you even consider using them in production.

[–] [email protected] 1 points 1 year ago

Thank you for the insight. I will think about it more and set up a test lab. We have 2.5Gbit switches, so I hope the network won't be a bottleneck

load more comments
view more: next ›