this post was submitted on 14 Jun 2023
9 points (100.0% liked)

Selfhosted

40347 readers
458 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

Hi all!

I have a couple of months to create and deploy a small cluster for running docker containers.

The cluster will consist of 3 master nodes and some workers. When it is ready, it will consist of about 15 servers.

I have little experience with Docker (managing some containers on my home server), I have spent the last 4 or 5 weeks studying and testing with Kubernetes and I think it's a little overkill for what it's going to take. You run the risk of adding unnecessary complexity.

I am seeing that instead Docker Swarm seems easier to set up and manage.

To consider that I will be on my own to manage it.

What do you think?

Thanks!

top 21 comments
sorted by: hot top controversial new old
[–] [email protected] 11 points 1 year ago* (last edited 1 year ago) (3 children)

I haven't used Docker Swarm (I have barely used Docker Compose), but I have run a couple on-prem Kubernetes clusters (at my house and for clients at my day job) and cloud Kubernetes clusters, so I can speak to how complex it is it set up and run.

My background is systems administration, engineering, IT, and now DevOps. I've been using Linux since Ubuntu 6.06.

I set up my Kubernetes cluster with kubeadm because I wanted to learn, and it took me about a weekend to get my single master, two worker cluster up and running. I think you could probably do this using k3s much faster and have less learning curve (you don't have to care as much about Container Network Interfaces, for example, because k3s makes that decision for you.)

There is a lot of documentation out there on Kubernetes. Helm as a "package manager" (really a templating engine) can be nice if the software you want to deploy has a Helm chart that is well written. Writing your own Helm charts can be a learning process, I've modified some but not written one from scratch yet.

Kubernetes releases new versions about quarterly. I've done several upgrades on my primary home cluster over the course of the past 2 years and they've been pretty smooth, about an hour of time investment ~~total~~ each. And remember, I'm on the more nerdy and complex flavor of Kubernetes. I think with k3s these would be even smoother and quicker.

I feel like Kubernetes knowledge is probably more valuable out in the industry if that's a factor for you. I haven't come across any Docker Swarm clusters in my DevOps travels, just Kubernetes and some HashiCorp Nomad.

I'm curious to see what folks say about Docker Swarm. If you have any questions about Kubernetes or running your workload on it, I'd be happy to try to help!

[–] [email protected] 6 points 1 year ago (2 children)

I agree with this. I think single node or not the industry is moving towards Kubernetes for container orchestration. Docker has showed their evil intentions and it’s time to leave them in the past. Even podman has native kubernetes manifest support (albeit limited last i checked) as @[email protected] pointed out there’s good avenues to take if you want to avoid the complexities of kubernetes like k3s.

[–] bananess 2 points 1 year ago (2 children)

Is k3s good for production?

[–] [email protected] 2 points 1 year ago

I'm more of a Kubernetes the Hard way kind of person, but I think it can be suitable for certain production workloads. I'd trust a production workload on it way more then Docker Swarm

[–] [email protected] 2 points 1 year ago (1 children)

We're using it in production at my day job in a couple of places.

[–] bananess 3 points 1 year ago (1 children)

How is it about the reverse proxy like traefik? Are you using nginx ingress with k3s?

Sorry for the many questions! Thanks!

[–] [email protected] 4 points 1 year ago

Yep, using ingress-nginx on k3s as well.

[–] [email protected] 0 points 1 year ago

I honestly hate that I can't just deploy docker compose files directly to kubernetes seamlessly, and instead have to translate them to manifests. I'd say that having to drop all of that existing configuration as well as not being able to easily copy docker-compose's for random new software I find is the biggest blocker for me being able to actually commit to using kubernetes.

[–] bananess 1 points 1 year ago (1 children)

Thanks for the reply!

I also have a background as Linux Sysadmin but barely a couple of years.

I have managed to get a Kubernets cluster up and running but I have difficult setting up containers. The Classic "whoami" works flawless by Traefik for example, but doing the same for Portainer seems like impossibile for me right now, going crazy :D

I'm i choosing a difficult path? Is it possibile to have a good K8s cluster running nice without traefik?

Thanks!

[–] [email protected] 3 points 1 year ago (1 children)

I use ingress-ngnix for all my ingress controllers, I've only messed with Traefik a bit in Kubernetes and it felt like it was fighting me the whole time.

[–] bananess 2 points 1 year ago

Nice to know that I'm not the only one having this experience

[–] [email protected] 1 points 1 year ago

K8s all the way. That's the general consensus in the industry. Swarm is not popular in comparison. My company sells a product in two flavors, SaaS and On Prem. Both run on K8s. We supply Helm charts and whatnot for On Prem installations (including in private cloud). There is good demand for K8s skill in the labor market.

[–] [email protected] 3 points 1 year ago (1 children)

I'm not really sure what the state of docker swarm is, but I'd recommend taking a look at Hashicorp Nomad too. It's between swarm and k8s in complexity, but it's been great to work with so far.

Kunernetes is what I see most companies using, but it's also usually overly complicated for what they need.

What kind of workloads do you plan on running?

[–] [email protected] 4 points 1 year ago (1 children)

overly complicated for what they need.

👀 glances at the 4 Kubernetes clusters running in my house / personal VPS

[–] [email protected] 2 points 1 year ago (1 children)

There's nothing wrong with having an over complicated homelab! I manage dozens of kubernetes clusters at work and it really is a good skill to have for career reasons. My dig at it is mostly that a lot of smaller teams or companies will jump to it without really looking at their other options first.

[–] [email protected] 2 points 1 year ago

Oh sure, I totally get it. Once I was fully bought in to k8s and GitOps for one environment, it just made sense to have my other ones use the same tooling.

[–] [email protected] 2 points 1 year ago

Docker swarm is pretty easy to set up and use (and lets you use compose files directly!) and is probably more than enough average self-hosters/homelabbers, but if you want to do something super fancy related to clustering there's a good chance you'll hit functionality walls quickly.

Kubernetes is a pain to set up but is very flexible and 'scalable' to incredible levels, while being massive overkill for most applications.

[–] [email protected] 1 points 8 months ago

Homeland: portainer Work: Openshift

My thoughts is how granular you need permissions to be.

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago)

Kubernetes all the way. The Pod definitions look a little different from compose files, and the networking is a little strange at first, but it's just better in every way.

For one, you don't get any of the dumb compromises that Docker Swarm has, and the industry has standardized on Kubernetes, so you'll have much more options with Kubernetes.

[–] tupcakes 1 points 1 year ago

docker swarm is def easier to manage, however I found it somewhat limiting. particularly around IP address assignment for services. I ended up using nomad, which is a nice inbetween. feels kinda like what swarm could have been.

[–] [email protected] 0 points 1 year ago

I thought Swarm is long dead.

load more comments
view more: next ›