this post was submitted on 22 Oct 2023
440 points (98.0% liked)

Selfhosted

40374 readers
451 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

Hi all home network administrators :) Haven't posted anything here since June, when I told you about Gatekeeper 1.1.0. Back then it was a pretty bare-bones (and maybe slightly buggy) DNS + DHCP server with a web UI with a list of LAN clients. Back at 1.1.0 Gatekeeper didn't even configure your LAN interface or set up NAT rules. It used to be something like dnsmasq - but with a web UI.

I've been improving it for the past couple of months - simplified it a lot, fixed bugs, optimized internals, improved the looks & added a bunch of quality-of-life features. Now it's a program that turns any linux machine into a home internet gateway. It's closer to OpenWRT in single executable file.

One big thing that happend was that I've attempted to replace the ubiquitous nft-based NAT (where the kernel processes the packets according to a rule-based script) with nfqueue (where the kernel sends the packet to userspace so that it can be altered arbitrarily). I've expected this to be buggy & slow. Well, it was hell to implement but it turns out that it's not slow at all. On the debug build my router can push 60GiB+ / second over TCP (over virtualized ethernet of course). And I'm not even using any io_uring magic yet. Quite honestly I don't even know how to explain it since it's slightly above the peak DDR4 transfer rate (I'm running dual channel DDR4-3200). Maybe the pages are not flushed to RAM & are only exchanged through CPU caches? Anyway I'm pretty excited because userspace access to all traffic opens a lot of new possibilities...

The first thing is NAT. By default Linux only supports symmetric NAT, which is pretty secure but is also fairly hard for the peer-to-peer protocols to pierce. There are some patches that make Linux full-cone but they're not, and are not expected to become a part of the mainline kernel (at least according to OpenWRT forums). Now, since we have access to every packet we can take care of this ourselves. We can create a couple hash tables to track connections, alter the source & destination IP, recomputing the checksums if necessary. Suddenly we can have full-cone NAT, on any linux machine, without patching the kernel! At runtime it's not as configurable as netfilter + conntrack but it's a whole lot simpler - since now we can use a general purpose programming language rather than netfilter rules.

Another cool feature that we can now have are truly realtime traffic graphs. Summary of each packet traversing the network boundary is immediately streamed to the connected web UIs over WebSocket. This is way faster than the alternatives based on reading some /proc/ or /sys/ files every couple of seconds. Gatekeeper also aggregates the traffic from the last 24 hours between each pair of hosts into a histogram with 100ms resolution and allow clients to view it, scroll through it, compute stats, download as JSON or CSV. You can retroactively check which device talked with what IP, at what time with unprecedented resolution.

My next step is going to be capturing the traffic that goes through into a 5MB circular buffer (separate buffer for each LAN client) & downloading it as Wireshark-compatible pcap files. Computationally it's almost free. IoT devices usually don't transmit a lot of data. 5MB may actually cover months of traffic for the simpler ones. If any device is did anything weird, it will finally be possible to investigate it - even after it already happened.

Long-term Gatekeeper could do even more. For example offer assistance in setting up TLS MITM, perform some online grouping / analysis on the live traffic.

I still have some ground work to do - like automatically setting up Wireless LAN, bridgidng multiple interfaces into a single one and I think there may be a bug that causes crashes when checking GitHub for updates. But I wanted to share it sooner rather than later. I hope that despite its imperfections some of you will find it useful!

(I've had some issues with cross-instance posting. This is attempt #6)

all 35 comments
sorted by: hot top controversial new old
[–] thelittleblackbird 44 points 1 year ago

Incredible amount of work, respect.

If you are lacking ideas for the super long term I could suggest you:

Any kind of ids/ips (intrusion detection system) Deep inspection packet to detect any vpn or crypto tunnel Ability to create a vpn link to another instance of the program (to link geographical disperse nodes)

And many other things that honestly I am ashamed of asking :)

[–] siftmama 17 points 1 year ago (3 children)

I don't think I fully understand how this is supposed to work. Am I correct in that this service acts as the gateway on the network? So clients would be configured to see this as the gateway, and this then subsequently pushes things through the real gateway (router)? Thus essentially everything (most things?) that the router does would be done by this instead?

[–] [email protected] 14 points 1 year ago (1 children)

It sounds like it's just an actual router, but I can't find any information if it actually can route any traffic.

Either way it seems like just setting up a real router with something like opnsense would be easier and more feature complete.

[–] PeachMan 16 points 1 year ago* (last edited 1 year ago) (1 children)

More features, yes. Easier? No. This project is aiming to be simpler than an OPNsense setup at the cost of advanced features.

According to the GitHub it can act as a router, OR you can leave your ISP router in place and have this act as a DNS and DHCP gateway. Please read the documentation before saying negative things in the comments.

[–] [email protected] 3 points 1 year ago

OPNsense is ready to route pretty much after the default install, like any other off-the-shelf router. It’s only really complicated if you have complicated needs.

[–] TCB13 6 points 1 year ago* (last edited 1 year ago) (1 children)

My questions exactly. Furthermore ISP router typically also have wifi, so wifi clients on that won't be even visible to this... please clarify is this is supposed to be yet another openwrt / router solution or what you're doing here... BUT I fucking love the UI :)

[–] [email protected] 2 points 1 year ago

Right there with you on the UI. This would overlap in functionality with a lot of other items in my network, but I'm trying to find a reason to use it just so I can play with the UI.

[–] PeachMan 4 points 1 year ago

Read the documentation.

"Generally speaking Gatekeeper needs to sit between your LAN network and the internet. It can either completely replace the router provided by ISP, or sit between the ISP router and your LAN network."

[–] [email protected] 14 points 1 year ago (2 children)

I will not blame you, since it seems you spent a lot of time on this project.

What does this project actually solve? It seems like you are trying to re-implement ISP's router, just being open source.

From my understanding, it's either user who has no idea how stuff works and will not look into this project, or user knows enough how this stuff works and will use OpenWRT router/rpi or Mikrotik. I don't understand - why would someone pick your project over RPI/OpenWRT/Mikrotik router?

[–] [email protected] 28 points 1 year ago* (last edited 1 year ago) (2 children)

In terms of types of users I agree with what you're saying but I also think that there are some shades of gray in between. There are people who love to tinker and would manually configure every service on their router, compiling everything from scratch, reading manuals, understanding how things work (they'll probably choose dnsmasq, systemd-networkd, graphana over Gatekeeper). In my experience this approach is pretty exciting for the first couple of years & then gradually becomes more and more troublesome. I think Gatekeeper's target audience are the people who would like to take ownership of their network (and have some theoretical understanding) but don't want to fully dive into the rabbit's hole and configure everything manually.

In terms of problem solved: I agree that Gatekeeper solves a similar problem. I think it's different from those projects because it tightly integrates all of the home gateway functions. While this goes against the Unix philosophy, I think it creates some advantages:

  1. Possibility of cross-cutting features.
  2. Better performance (lower disk usage, lower RAM usage, lower CPU load).
  3. Seamless integration.

Functions of home routers are conventionally spread out over many components (kernel & a bunch of independently developed userspace tools) which talk to each other. Whenever we want to create a cross-cutting feature (for example live traffic graphs) we must coordinate work between many components. We need to create kernel APIs to notify userspace apps about new traffic, create userspace apps to maintain a record of this traffic & a web interface to display it. It's difficult organizationally. In a monolith, where all code is in one place, such cross-cutting features can be developed with less friction.

From the performance point the conventional approach is also less efficient. The tools must talk to each other. Quite often through files (logs & databases). It's wearing down SSDs & causing CPU load that could be otherwise avoided. A tightly integrated monolith needs to write files only periodically (if ever) - because all data can be exchanged through RAM.

From the complexity standpoint the conventional approach is also not great because each of the tools needs to know how to talk with the others. This is usually done by administrator, configuring every service according to its manual. When everything is built together as a monolith, things can "just work" and no configuration is necessary.

Edit: Please don't be offended by my verbosity. From your question I see that you know this stuff already but I'm also answering to the fresh "selfhosted" audience :)

[–] rekliner 6 points 1 year ago

Well explained. Thanks for spelling it out.

[–] MigratingtoLemmy 4 points 1 year ago

Thanks for the explanation, that really helped! I will follow this project!

[–] SheeEttin 9 points 1 year ago (1 children)

Yeah, if I wanted a router, I'd just use opnsense. This feels like it's in that weird middle ground between doing one thing well and being a swiss army knife, where it does a lot of things, but when there's something else you need it's not there and you can't easily add it.

[–] [email protected] 2 points 1 year ago

Kind of a good thing to have something in between, but the problem for the author - there wouldn't be a lot of consistent users. What if user starts needing VLANs? Or VPN? Or VPN as a client? The user would migrate to other tools.

[–] [email protected] 12 points 1 year ago* (last edited 1 year ago)

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
DNS Domain Name Service/System
HTTP Hypertext Transfer Protocol, the Web
IP Internet Protocol
IoT Internet of Things for device controllers
NAT Network Address Translation
RPi Raspberry Pi brand of SBC
SBC Single-Board Computer
SSD Solid State Drive mass storage
SSH Secure Shell for remote terminal access
SSL Secure Sockets Layer, for transparent encryption
TCP Transmission Control Protocol, most often over IP
TLS Transport Layer Security, supersedes SSL
VPN Virtual Private Network

11 acronyms in this thread; the most compressed thread commented on today has 5 acronyms.

[Thread #230 for this sub, first seen 22nd Oct 2023, 11:05] [FAQ] [Full list] [Contact] [Source code]

[–] [email protected] 7 points 1 year ago (1 children)

The readme is not so clear on this: does it also do port forwarding?

But seems pretty cool indeed 👍

[–] maf 8 points 1 year ago* (last edited 1 year ago) (3 children)

Unfortunately explicit, stable port redirections is something that is still missing. I'll have to implemnt them (with a proper UI) eventually because under the hood they are also a necessary building block for other features. At the moment there are only "ephemeral" port redirects which may be sufficient for you. They are created automatically when a LAN machine sends out a packet from some source port. That port is then implicitly forwarded back to that machine. This is actually a part of the "Full Cone NAT" thing.

This can be triggered manually for example with something like:

nc -p 80 1.2.3.4 1234 # send a dummy TCP packet from port 80

Ephemeral port redirections don't expire but can be taken over if another LAN host also uses the same source port for outgoing traffic. This may happen randomly because source ports are usually picked at random by the OS. Generally ports below ~32k should be fairly stable because Linux doesn't use those by default (I don't know about Windows). Redirecting ports below 1024 should be even more stable because they're reserved for specific well-known services.

[–] [email protected] 4 points 1 year ago (1 children)

What makes the port redirections difficult to implement in the code? I’m imagining the kernel has some way of handling this without too many external libraries but I’m not well-versed enough on this to know for sure.

[–] maf 6 points 1 year ago (1 children)

The relevant part of NAT is actually just those 20 lines.

The hardest part is actually the UI :P The difficulty in building nice UI comes from potential ports listening on the local WAN interface (for example if the machine is also running any HTTP or SSH servers). I'd like the user to see at a glance what ports are used for what (port used by a local service - what service is that?, ephemeral port redirection using the full cone nat table - where is it redirected?, any symmetric nat connections together with their last activity / timeouts / traffic summary). Ideally the same interface should also allow the user to create new redirects.

[–] [email protected] 3 points 1 year ago

I love what you did, especially the c++.

Using a unifi right now but this is the perfect replacement, especially since it's programmable, just put a few nic ports on a vm and let it run.

Just beautiful.

[–] [email protected] 3 points 1 year ago (1 children)

So you're not remapping the source ports to be unique? There's no mechanism to avoid collisions when multiple clients use the same source port? Full Cone NAT implies that you have to remember the mapping (potentially indefinitely—if you ever reassign a given external IP:port combination to a different internal IP or port after it's been used you're not implementing Full Cone NAT), but not that the internal and external ports need to be identical. It would generally only be used when you have a large enough pool of external IP addresses available to assign a unique external IP:port for every internal IP:port. Which usually implies a unique external IP for each internal IP, as you can't restrict the number of unique ports used by each client. This is why most routers only implement Symmetric NAT.

(If you do have sufficient external IPs the Linux kernel can do Full Cone NAT by translating only the IP addresses and not the ports, via SNAT/DNAT prefix mapping. The part it lacks, for very practical reasons, is support for attempting to create permanent unique mappings from a larger number of unconstrained internal IP:port combinations to a smaller number of external ones.)

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago)

So you’re not remapping the source ports to be unique? There’s no mechanism to avoid collisions when multiple clients use the same source port?

Regarding port collisions. In Gatekeeper there are both - a Symmetric NAT & Full Cone NAT. Both are used in tandem. I didn't mention the former before. Symmetric NAT takes precedence over Full Cone NAT when a connection has already been established (we observed the remote host and have a record of which LAN IP they're talking to). You're 100% correct that without Symmetric NAT there would be port collisions and computers in LAN would fight over ports. I actually started out with just the Full Cone NAT only (where collisions can happen) and used it on my network for a couple of weeks. It seemed to work in my home environment but I was a little worried about potential flakiness so I've implemented a backup mechanism eventually.

Full Cone NAT implies that you have to remember the mapping (potentially indefinitely—if you ever reassign a given external IP:port combination to a different internal IP or port after it’s been used you’re not implementing Full Cone NAT) (...)

Ah, I also recalled something like that! What you're saying about NAT assignments being permanent & requiring multiple IPs to avoid collisions. I think there was a course at my university or some Cisco course that taught that... I haven't been able to find any online sources that would confirm those definitions today but I also remember something along the lines of what you're describing. I have no idea what happened with those terms. Maybe the "permanent assignments" don't make much sense in wireless networks WiFi devices can appear and disappear at any time?

Edit: I found it - the proper term for this was "Static NAT" (as opposed to "Dynamic NAT" where the redirections expire).

(...) but not that the internal and external ports need to be identical.

Right. Port preservation is not a strictly necessary part of Full Cone NAT. It's a nice feature though. I guess the technical classification would be "Full Cone NAT with port preservation".

(If you do have sufficient external IPs the Linux kernel can do Full Cone NAT by translating only the IP addresses and not the ports, via SNAT/DNAT prefix mapping. The part it lacks, for very practical reasons, is support for attempting to create permanent unique mappings from a larger number of unconstrained internal IP:port combinations to a smaller number of external ones.)

This is very cool indeed. I didn't knew that. Thanks!

[–] [email protected] 3 points 1 year ago

Well done mate! Can’t wait to try it!

[–] [email protected] 3 points 1 year ago (1 children)

This sounds interesting, does implementing this, as a noob, make sense?

[–] [email protected] 3 points 1 year ago

I understand that your goal is to learn something new.

In my opinion ambitious, goal-oriented projects may either backfire or turn you into a legend. There will be many issues along the way and while they are all ultimately solvable, the difficulty may kill your motivation. Alternatively, if you manage to power through, then after some period of learning (potentially years) and keeping the fixation on specific problem you might emerge as a domain expert. Either way it's a risky bet.

If I might leave some advice for newcomers it would be to learn how to perform some simple tasks & focus on creating projects that you're confident can be built from things you already know. Over time you'll increase the repertoire of tasks that you can perform, and therefore be able to build increasingly advanced projects.

[–] [email protected] 2 points 1 year ago (1 children)

Very interesting project, thanks for sharing and working on this. I am actually one of your target user, where I have enough knowledge to implement my own router, at the moment running on gentoo.

I would like to use this but it lacks port forwarding and a firewall, that is a must. I'll try it out nevertheless. I'm quite impressed by the stylish HTML graphics, and I appreciate your departure from the typical "modern" gray corporate Bootstrap UI design. It's really, really cool.

One question. how do you envision exposing this service to the internet? I quite despise rust but I wonder if the use of a memory safe language would help with the inevitable bugs, especially if you put even more features into gatekeeper.

[–] [email protected] 1 points 1 year ago (1 children)

Thank you for the feedback! I have to admit I wasn't aware of how important port forwarding is. Stepping back I guess I'll need a better way of gauging how important specific features are to people. I'll have to think about this a little bit more...

Your question about security is something that I think about a lot. I don't think of LAN & internet as significantly different in terms of security. I also worry about potentially malicious LAN devices attempting to exploit local DNS, DHCP or web UI. I've profesionally worked on anti-malware and I've seen malware preloaded on new phones by factory workers & resellers, suspiciously exploitable flaws in stock firmware (which I guess was a backdoor with plausible deniability), fake monetization SDKs that are actually botnets (so application developers have been unknowingly attaching bots to their apps). There is also the problem of somebody gaining the physical access to your LAN network (for example by connecting a prepared device to an ethernet port for a couple of seconds). While those things may seem far fetched and commercial routers ignore them, I'd like to do something better here.

In terms of preventing C++ footguns, I'm relying on compilation arguments (-fstack-protector, -D_FORTIFY_SOURCE=2), safe abstractions (for example std::unique_ptr, std::span, std::array...), readability (single-threaded, avoiding advanced primitives or external libraries) & patience (I think that time pressure is the biggest source of bugs).

In terms of protocol level security, so far I've been able to secure the update path (so that MITM attackers can't inject malicious code). The web UI is a big problem for me because to do any privileged operations I'll have to authenticate the user first. Firstly I'm not exactly sure how to even do that. Password seems like the best option but I'm still trying to think of something better. There is this new WebAuthn thing which I'll have to look into. Second issue with web UI is that I need to protect the authentication channel. This means that local web UI will need TLS. And this in turn means that I'll have to somehow obtain a TLS cert somehow. Self-signed certs produce nasty security warnings. Obtaining one from LetsEncrypt seems easy - assuming the router has public IP (which may not always be the case). But even if I obtain a LetsEncrypt cert, any LAN device can do the same thing, so the whole TLS can still be MITM-ed. It would be really great if web browsers could "just establish encrypted channel" and not show any security warnings along the way...

[–] [email protected] 1 points 1 year ago (1 children)

thank you for the reply. All the stuff you wrote makes sense.

But even if I obtain a LetsEncrypt cert, any LAN device can do the same thing, so the whole TLS can still be MITM-ed.

can you elaborate?

[–] [email protected] 1 points 1 year ago (1 children)

Yeah. LetsEncrypt usually verifies whether the client asking for a certificate owns the domain by sending a HTTP-based challenge. Gatekeeper could pass it by intercepting traffic on port 80. But any LAN device could also pass it by asking for port 80 to be temporarily forwarded. This means that LetsEncrypt TLS certificates are not worth much in LAN environment. Malicious IoT device could convince other LAN hosts that it owns the router IP be sending spoofed ARP announcements. Whenever any LAN device would try to visit Gatekeeper web UI, it would actually visit a fake web UI hosted by the malicious IoT device. The IoT device could then sniff the administrator password and perform privileged actions in the real web UI.

[–] [email protected] 1 points 1 year ago (1 children)

interesting. This could all be solved if gatekeeper doesn't allow port redirection on 80 unless explicitly configured by the administrator, right?

[–] [email protected] 1 points 1 year ago

Oh, yeah that would make sense. I think that would solve the whole security aspect :)

[–] [email protected] -3 points 1 year ago

Monolithic announcement about monolithic project to reinvent a wheel.

Super-binaries were dead before systemd started eating everything around it, but it's got Plot Armour protecting it.