I was recently gifted some Mellanox 40gig network cards
Where can I find such friends/relatives?
this post was submitted on 03 Sep 2023
20 points (95.5% liked)
homelab
6649 readers
3 users here now
founded 4 years ago
MODERATORS
Marry way out of your league π
That's what I did at least
I'm already at the lowest league unless you count people who marry their animu body pillow.
Are you sure you're not using local hostnames / DNS resolution?
You may also need to updates your nfs exports file for the new subnet. And also update systems on the fstab changes (daemon-reload).
I did run a daemon-reload. I'm directly referencing the IP address in my fstab. On my desktop I have DNS turned off for the Mellanox interface.
I'll have to look into the nfs exports file. I'm using OMV, and I just have my exports wide open. But, I guess I could solve this by limiting the connection to the mounts by client IP.
There might also be some magic/weirdness with IP routing in the kernel. Have a look at net.ipv4.ip_forward system variable.
Sounds like route tables are finding a priority match on the 1gb interface. Are you 100% sure the NAS connection is truly not an overlapping subnet with the 1gb nic?
100% sure.
LAN: 192.168.69.0/24
Fiber: 10.42.69.0/24
Also: yeah, yeah. I know...
you need to look at the routing tables on your computer. these tables store the prioritized rules for how packets leave your host machine.
it might be that something is adding rules, or, there is some overly broad rule taking priority (like a rule that says all 10.0.0.0/8 traffic go to your home router over 192.168.69.0/24, etc)
itβs also suspect that you can reach the NAS over the 1gb card. That to me means one of two things:
- something is not actually using the IP youβve configured in your fstab and is using some IP that is on the 1gb interface
- you have some weird network routes configured that is leading to this issue. if 10.42.69.0/24 is accessible over the 192.168.69.0/24 network, then you might need to create a static route explicitly telling your OS to send packets out the 40gb card
ultimately, i suggest you run something like tcpdump or wireshark on your computer (ideally on the NAS too) so you can start to visualize how the packets are being addressed and transferred over your networks.
sincerely, a fellow 10.0.69.0/24 enjoyer
A bad route would be the first thing I'd check, too.
It sounds like u/[email protected] is already pretty familiar with the Linux command line, but just in case, you can check the routing table (on both the NAS and the client machine) with ip route (that should show the whole routing table) and get the specific route to the remote device with ip route get 10.42.69.XXX (change the XXX to make it the address of the remote system). If either side shows that the route is going over your default gateway, that's your problem.
It definitely sounds like he has redundant routes (this literally shouldn't be possible otherwise) so yeah, he needs to fix priority
Could add a default route via interface
Maybe a /30 going to that gateway
Autoneg / link speed problems?