this post was submitted on 18 Jan 2024
21 points (86.2% liked)

Linux

47307 readers
608 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
21
[not solved] I think I messed something up in the router settings but cannot understand what... now I have no internet connection (lemmy.kde.social)
submitted 8 months ago* (last edited 8 months ago) by [email protected] to c/[email protected]
31 comments fedilink hide all child comments
 

Hello! I (tried, at least) converted an old laptop to a Debian home server, and I was trying to set up duckdns.org and to enable port forwarding on my router. internet connection was working, I installed packages, docker, immich, etc, and then suddenly (I don't know exactly when) it refuses to connect to the internet. It does connect to local addresses (I can ssh into it) but ping google.com and any other internet-involving command fail. I had set up a rule on the router to forward port 80 to the device's port 80, but I then removed the rule and it still does not connect to the internet. I rebooted the router but nothing changed. Any idea what could be? the router is a Vodafone router.

I changed the hostname to debianserver but on the router it is still written debian. Also, it's the only device with unknown ipv6

thanks in advance!

EDIT: I rebooted again the server, and now ipv6 is not unknown anymore, and the hostname is correct. however, it still does not connect to the internet

EDIT 2:

only one device (debianserver) has this problem, other devices work as before

EDIT 3:

I don't know if it's useful or not, but if I boot a live debian USB in the server internet works

~~SOLUTION:~~ aaaaand no it does not work, after restarting docker it seems to work because all the brodges are yet to be created and it takes some time, after like 30 seconds it does not work as before :(

  • edit the file /lib/systemd/system/docker.service
  • append the flag --bip=192.168.3.1/24 to ExecStart=....
  • systemctl daemon-reload
  • systemctl restart docker

docker was the fucker that messed everything up and made me lose a few hours!

EDIT 4:

it seems that ip route flush 0/0 restores the internet connectivity until reboot... I don't know what does this means but can be a temporary workaround I guess? I really have no idea how to solve this

FINAL EDIT: I gave up. I removed debian and installed fedora, and now it all works like a charm

top 31 comments
sorted by: hot top controversial new old
[–] [email protected] 7 points 8 months ago (1 children)

Does ping 8.8.8.8 work? (To check if it's DNS)

What's ip route show say? (Just to try to narrow down whether it's an issue with the server's config or the router's)

What's traceroute 8.8.8.8 display?

[–] [email protected] 3 points 8 months ago (2 children)

thanks for the answer!

ping 8.8.8.8 fails, and I don't have traceroute installed (and no internet to install it)

tubbadu@debianserver:~$ ip route show
0.0.0.0 dev veth3492bf7 scope link
0.0.0.0 dev vethc1bf668 scope link
0.0.0.0 dev vethb41fd7e scope link
0.0.0.0 dev veth2e39932 scope link
0.0.0.0 dev veth68451d9 scope link
default dev veth3492bf7 scope link
default dev vethc1bf668 scope link
default dev vethb41fd7e scope link
default via 192.168.1.1 dev enp1s0
169.254.0.0/16 dev veth68451d9 proto kernel scope link src 169.254.210.75
169.254.0.0/16 dev veth2e39932 proto kernel scope link src 169.254.242.12
169.254.0.0/16 dev vethb41fd7e proto kernel scope link src 169.254.185.90
169.254.0.0/16 dev vethc1bf668 proto kernel scope link src 169.254.225.22
169.254.0.0/16 dev veth3492bf7 proto kernel scope link src 169.254.123.220
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-56cf32fc7cde proto kernel scope link src 172.18.0.1
192.168.1.0/24 dev enp1s0 proto kernel scope link src 192.168.1.9
192.168.1.1 dev enp1s0 scope link
[–] [email protected] 4 points 8 months ago* (last edited 8 months ago) (1 children)

This is kind of a nutty network config. It looks like docker is setting up extra default routes, which I could easily see fouling everything up. As a first experiment (warning, this may ruin your networking until the next reboot):

ip route flush 0/0
ip route add default via 192.168.1.1 dev enp1s0

... and see if that makes things work (start with ping 192.168.1.1 and ping 8.8.8.8). If that solves the problem, then I think something about your docker config is adding stuff to your networking that's causing the problem; maybe remove/disable docker completely and then re-add docker things one at a time to see where the problem comes in.

[–] [email protected] 2 points 8 months ago (1 children)

okay, I thought to have solved the problem but I was wrong, here I go again. When I docker compose up -d the immich server (the only one I have installed) all those routes are created, and apparently some of them conflicts with something else and now my host has no internet connection. however it seems that ip route flush 0/0 solves the problem until the reboot, which is strange. the other command returns RTNETLINK answers: File exists

[–] [email protected] 5 points 8 months ago* (last edited 8 months ago)

Hm. Yeah, that's weird. The default routes you're seeing should basically never exist, so it sounds like there's some kind of manual network config happening inside the Docker container that's creating a broken network.

What does docker network inspect [network] say for each of your Docker networks (substituting each Docker network for "[network]")? What's the network section of docker-compose.yml look like?

[–] [email protected] 3 points 8 months ago

Try removing all the superfluous default routes.

[–] [email protected] 4 points 8 months ago (1 children)

My guess is that you have Docker configured incorrectly. Its internal IP range probably overlaps with your real network, so all requests are routed to Docker. Uninstall docker and reboot the server. If that works, reinstall docker and properly configure its internal networking.

[–] [email protected] 5 points 8 months ago

I just figured this out! adding a --bip=192.168.3.1/24 to the systemd service solves this! [reference]

[–] Aurix 3 points 8 months ago (1 children)

Go to the outer status page. The router should display whether it has an internet connection to your provider. If no, then your router/modem has no credentials or another issue preventing access.

If it shows as working, then you can narrow it down to incorrect DNS and IP routing. Perhaps dynamic IP allocation is set to off or another configuration error or bug, in which case you might need to reset all the router settings. Then, is it only broken for a single end device?

[–] [email protected] 1 points 8 months ago

thanks for the reply! Sorry it isn't very clear from the post, but yes only one device (debianserver) has this problem (no internet connection, but yes local network connection), all other devices works as before. I'll update the original post to clarify this

[–] [email protected] 2 points 8 months ago (1 children)

I understand you might be a bit stressed out, but it's very hard to make head or tails of what you're describing.

E.g. What do you mean by "no internet connection"? Does it mean just the old laptop, all devices on the network or the router itself?

[–] [email protected] 1 points 8 months ago (1 children)

thanks for the answer! Sorry I didn't specified very well: only the server has problems, all other devices work as before

[–] [email protected] 3 points 8 months ago* (last edited 8 months ago) (1 children)

I bet it has something to do with the LAN DHCP setup, since you have to set a static IP for the server...

Update - That won't work at all if it is indeed the issue. Let me think for a sec on how to pull external ip by doing a IP wget...

[–] [email protected] 0 points 8 months ago (1 children)

wget -qO - icanhazip.com

it waits like 10 seconds and then exits without any output

[–] [email protected] 3 points 8 months ago (1 children)

Yeah, I realized after the fact. This should work: wget 104.18.114.97, and you should get a ERROR 403: Forbidden if your server has any internet connection.

[–] [email protected] 2 points 8 months ago (2 children) 
tubbadu@debianserver:~$ wget 104.18.114.97
--2024-01-18 21:35:38--  http://104.18.114.97/
Connecting to 104.18.114.97:80... failed: No route to host.

really really thanks for the help!

[–] gornius 2 points 8 months ago* (last edited 8 months ago) (1 children)

You mentioned you changed firewall rules for that device. Any chance you have set outbound rule instead of inbound rule?

Anyway, what's the output of ip route?

[–] [email protected] 1 points 8 months ago

Anyway, what’s the output of ip route?

tubbadu@debianserver:~$ ip route
0.0.0.0 dev veth60f206e scope link
0.0.0.0 dev veth48a6716 scope link
0.0.0.0 dev vethdf7381d scope link
0.0.0.0 dev veth59e46aa scope link
0.0.0.0 dev vethac4830e scope link
default dev veth60f206e scope link
default dev veth48a6716 scope link
default dev vethdf7381d scope link
default dev veth59e46aa scope link
default via 192.168.1.1 dev enp1s0
169.254.0.0/16 dev vethac4830e proto kernel scope link src 169.254.241.201
169.254.0.0/16 dev veth59e46aa proto kernel scope link src 169.254.97.105
169.254.0.0/16 dev vethdf7381d proto kernel scope link src 169.254.212.49
169.254.0.0/16 dev veth48a6716 proto kernel scope link src 169.254.26.74
169.254.0.0/16 dev veth60f206e proto kernel scope link src 169.254.212.242
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-56cf32fc7cde proto kernel scope link src 172.18.0.1
192.168.1.0/24 dev enp1s0 proto kernel scope link src 192.168.1.9
192.168.1.1 dev enp1s0 scope link

You mentioned you changed firewall rules for that device. Any chance you have set outbound rule instead of inbound rule?

All I did was doing this and press Save:

(now I rewrote it just to take the screenshot)

after internet stopped working I edited it to set the port to 443, and then I removed the rule. there are no rules now

[–] [email protected] 2 points 8 months ago* (last edited 8 months ago) (2 children)

Please try wget https://104.18.114.97, if this one goes through I'd think the problem could be related to a faulty forwarding of regular internet traffic (port 80). If that fails as well I'd guess it's to due with the DHCP/Static IP's and involves your router. I'm absolutely clueless about Vodafone routers though.

Edit: Any connection would show "The certificate's owner does not match hostname ‘104.18.114.97’"

[–] [email protected] 1 points 8 months ago (1 children)

Edit: Any connection would show “The certificate’s owner does not match hostname ‘104.18.114.97’”

sorry I don't think I understand, where should this be written?

[–] [email protected] 1 points 8 months ago (1 children)

Here I've tried wget https://104.18.114.97. It's able to connect to the IP on port 443, but shows an error message since I'm not using a domain name. But at least I know I've reached the server.

[–] [email protected] 1 points 8 months ago

so this means that the problem is actually the server? or do I misunderstand?

[–] [email protected] 1 points 8 months ago (1 children)

wget https://104.18.114.97

same as before:

tubbadu@debianserver:~$ wget https://104.18.114.97
--2024-01-18 21:52:29--  https://104.18.114.97/
Connecting to 104.18.114.97:443... failed: No route to host.
[–] [email protected] 2 points 8 months ago (1 children)

Something is completely shutting your server off from the internet, despite it having full LAN access. The only time I've run into this exact issue was when I misconfigured the firewall on a server, effectively only allowing for local connections. I simply started over by reinstalling Debian, wiping all my mistakes. But it could also be a setting on your router, and without you knowing what changes you made it's hard to give any reasonable advice.

These are just shots in the dark, and other might offer better solutions but I'd try;

  1. Boot the laptop into a live session directly from USB. All settings are default. Test again, either wget or maybe ping a website. If it works, it's the server setup and I'd start over. If not...

  2. Try reversing all changes on the router, give the server a different static IP.

  3. Back up the router configuration to a file, consider making notes of important changes, reset the router and try again. If it STILL doesn't work you can restore the important settings. If it works, you can reimplement the settings from your notes. (unless we're talking manually imported VPN certificates and similar lol)

Sorry I wasn't able to help you out, I'm hosting from home and it's a fantastic thing when things

I don’t know if it’s useful or not, but if I boot a live debian USB in the server internet works

Haha yes, as mentioned. The issue is a setting on your laptop server installation. The simplest thing is just reinstalling and starting over.

[–] [email protected] 2 points 8 months ago (1 children)

The issue is a setting on your laptop server installation. The simplest thing is just reinstalling and starting over.

this is probably a good news I think, because it's only tedious but easy to solve If there's a way to do it without having to reinstall immich on docker would be awesome, but perhaps it's faster to reinstall everything than to look for a way to do this

thank you very very very VERY much for your help!!!!!!!!!

[–] [email protected] 1 points 8 months ago* (last edited 8 months ago) (1 children)

Happy to help, sorry I can't offer a simpler solution that avoids reformatting. It's probably doable if you want to do more fine grain troubleshooting but I've never found it to be worth the time.

For my home self-hosting I also prefer pre-made scripts exactly to avoid having to manually set up everything. It's such a slog and particularly if I have to restart from scratch. What's the goal of your project, perhaps something like Yunohost or even DietPi could save you a lot of work and get straight to the fun of self-hosting?

[–] [email protected] 2 points 8 months ago

What’s the goal of your project?

I have my brother's old laptop with the screen broken, so I thought to convert it to a server for my huge pictures library I have on my external ssd installing immich, it's been a while since I'm reading things about this and now I finally have the chance to do it! I finished reinstalling now and internet works fine. I'll run a ping linux.org after every single step so if this happens again I'll at least know who's the guilty!

thanks again for your time and help!

[–] [email protected] 2 points 8 months ago (1 children)

Is there a reset button somewhere on the router? Most of them have something like this in order to reset them to factory settings. If not, google for your device name and factory reset, maybe it’s something like „press button while turning on“ etc. I’d try something like that

[–] [email protected] 3 points 8 months ago (1 children)

Yes there is in the web UI a factory reset button, but I'd rather not do this because it has some settings by other people

[–] [email protected] 4 points 8 months ago (1 children)

So, several people share the router and can modify settings...
That's a recipe for trouble.

I do external support for small businesses. When I work, configure etc. sometning, I change the password. I write the password on a piece of paper and put it in an envelope. Then I sign across the sealed flap. That envelope is handed to the customer. If I cease to work for them or something happens to me, they can open it and retrive the password. If something goes belly up, I ask for the envelope. If the envelope has been tampered, I wash my hands, and chrge handsomly to solve the issue.

I'd reset it, and then make someone responsible for doing things to the router, that way everyone knows what's going on.

[–] [email protected] 2 points 8 months ago

whel this is clever XD