this post was submitted on 23 Jun 2023

23 points (96.0% liked)

Selfhosted

40730 readers

664 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago

MODERATORS

What is the most affordable way to get into 10G networking? (self.selfhosted)

submitted 2 years ago by surfrock66 to c/selfhosted

29 comments fedilink hide all child comments

I think I'm ready to elevate from bonded 1G networking to 10GB. I tend to have used/old netgear equipment at home, purchased on the cheap. I think I'm finally seeing enough latency when trying to run OS's on my proxmox server with storage on my TrueNAS that I want to move to 10G networking.

Looking to replace my existing 12-port L3 switch (6 ports are occupied) with either RJ-45 or SFP based 10G, I think I will need realistically 16 ports. Storage will specifically be moved from my 48 port L2 witch with 4-1G bonds, carrying storage and networking, to probably 2 10G bonds dedicated to storage, which currently means I need an additional 6 ports right now, so 12 total used, then planning for future expansion, 16 port minimum.

I've been crawling ebay, but the barrier to entry is way higher than my foray from unmanaged to managed networking. I keep landing on Netgear M4300, where the price ranges from about $1000-$2400. I can save up for these, but is there a better way to get into 10G? I've seen some HP Procurve switches that seem way more cost reasonable, but my networking peers have told me to completely stay away from HP networking.

Any thoughts are appreciated!

top 23 comments

sorted by: hot top controversial new old

[–] [email protected] 6 points 2 years ago* (last edited 2 years ago) (2 children)

Mikrotik stuff is pretty good and quite inexpensive. They all run RouterOS, which is their linux distro for switches/routers. Fully managed and can do basically anything L2 or L3 you could want. Worth noting that availability and pricing has been pretty variable from their resllers for a while, and you will pay a premium via Amazon. Here's a list of relevant product product pages with the port counts and MSRPs:

A router with 12x SFP+ and 2x SFP-28/25g for $600, I run one of these as my router.
A switch with 16x SFP+ for $500 Probably the right switch for you
A switch with 24x SFP+ and 2xQSFP/40g for $600 I run one of these as my core switch

Also these may help for some corner cases or change your ideas on what you want to build:

A tiny switch with 4x SFP+ for $150, I have a few of these where I want multiple things plugged in to 10g, but don't want to run a cable for each of them as the devices aren't usually using a full 10g at the same time.
A switch with 8x SFP-28/25g and 2x QSFP-28/100g for $1000, interesting because each QSFP-28 can be broken out into 4x SFP-28, so a total of 16 25g ports. Just imagine making the jump to 25g instead of 10g, wouldn't it be glorious?
A switch with 48x 1GBASE-T, 4x SFP+, and 2x QSFP+ for $600 Who needs anything else? Break-out the QSFPs into 8 SFP+ ports, you've got 4 SFP+ ports natively, and then another 48 ports of 1g copper. Could be pretty interesting. There is also a variant with PoE output for $400 more in case you need to run cameras/APs/whatever and don't want to run injectors.
A super weird 2x SFP-28/25g network card that is actually a full router for $200 I haven't figured out a use for these yet, but I kinda want one. Otherwise my SFP-28 ports are being wasted at just 10g, right?
A switch with 16x SFP-28/25g and 2x QSFP-28/100g for $1600 Total overkill, but I just had to list it since it falls in the middle of the price range you were considering but is so much beefier than the Netgear offering.

[–] surfrock66 2 points 2 years ago

Thank you for the super in depth recommendations, this is great.

[–] [email protected] 1 points 2 years ago

I am also running CRS326-24S+2Q+RM as my core switch. It is a steal. I am thinking of getting their CRS504-4XQ-IN for... well, science🤣!

[–] [email protected] 6 points 2 years ago* (last edited 2 years ago)

I went the middle road and settled for 2.5G - still a nice boost, and a lot cheaper. Might be worth a consideration if you can't find cheap enough 10G gear.

Edit: Cost me around 300 dollars to upgrade my network. 2x 8 port switches, and 4x network cards.

Edit2: As a bonus, I could continue using the 1gbit devices directly plugged into the switch, since 2.5G use ethernet and the switches support 1g too.

[–] TurnItOff_OnAgain 6 points 2 years ago (5 children)

Check out the FS stuff. It's pretty cheap brand new

https://www.fs.com/c/10g-enterprise-switches-3256

They SFPs are cheap and work very well too.

[–] iuser 2 points 2 years ago

Agree, I’m pretty happy with their gear.

[–] [email protected] 1 points 2 years ago (1 children)

How does it compare to e.g., Netgear in terms of functionality and reliability?

[–] TurnItOff_OnAgain 1 points 2 years ago

I've only used their SFPs and DACs, which have been perfectly fine. Can't speak to their networking equipment.

[–] surfrock66 1 points 2 years ago

Thanks for the rec, still I would have to save up, but it's a good option to go new. I think it's time to shoot for saving up 2k.

[–] [email protected] 1 points 2 years ago

I've bought several products from FS (no switches but cables) and they are amazing quality

[–] [email protected] 1 points 2 years ago

This. FS or there's some shockingly good (looking) stuff on Aliexpress I've been considering and have heard good (perhaps sponsored) things about.

[–] [email protected] 4 points 2 years ago

If you can live with only 4 10G ports, those types of switches are far cheaper than 16 port ones, I've gotten them for ~$100AU off eBay auctions before.

And if you're fine with spending a bit more you could get a 8 port ubiquiti aggregation switch, and then upgrade to a bigger switch once the price comes down.

[–] nezbyte 3 points 2 years ago (2 children)

The Brocade ICX switches are very affordable for the performance. Great way to get a mix of 1G PoE, 10G, and 40G for under $200. Sadly they do consume a fair bit of power and most are loud out of the box. Although, there is a fan mod you can do to make them run quiet and cool.

Here is a link to more info: https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/

[–] surfrock66 2 points 2 years ago

This is a huge angle I didn't see; I run everything by my IRL networking friends and I'm seeking feedback on a 6610 now, but it looks like exactly what I need.

[–] surfrock66 2 points 2 years ago (1 children)

I have an actual bedroom closet with a 45U rack and dedicated power and cooling, so noise/power are not a concern, I'll look into it. How is the mgmt of them?

[–] nezbyte 2 points 2 years ago

Management is mostly done through CLI. You need a generic console cable to get it on the network and then you can do everything remotely. There is a web ui as well, I mostly use it for quick status checks.

I have only used an ICX6610-48P-E. As this was my first non-Unifi managed switch, I spent a few hours setting it all up. Now, I just go back to the guides on ServeTheHome whenever I want to change VLANs or something complicated.

My home server is connected to the switch using a Mellanox ConnectX-3 and a QSFP DAC cable. It was about the same price as a 10G card and link so I figured why not go 40G to future proof it.

[–] [email protected] 2 points 2 years ago (1 children)

Do you want managed or unmanaged switches? I run some Microtik managed 10g switches and I can provide model numbers when I get home tonight.

[–] [email protected] 1 points 2 years ago

Microtik has a 16 port SFP+ managed switch (CRS317-1G-16S+RM) for around $500 USD.

My personal setup is a CRS312-4C+8XG-RM which has 8 10G RJ45 and 4 SFP+/RJ45 Ports. You can usually get this switch for around $500-600 USD. For 10G into 1G devices I got a CRS326-24G-2S+RM which has 2 SFP+ 10G port into 24 Gig RJ45 ports.

Just as a warning, most managed switches assume you are using these in a data center or office; they can get quite loud. In the case of Microtik switches there is an ebay seller that offers quieter fans that you can replace inside the box (that void your warranty).

[–] mrclark 2 points 2 years ago

You could also look at Mikrotik. They have some amazing hardware with every bell and whistle you could ever want

[–] [email protected] 2 points 2 years ago* (last edited 2 years ago)

If you wanna dip your toe in the 10Gbps water, I recommend you giving Mikrotik a look, specially CRS305-1G-4S+IN or CRS309-1G-8S+IN. The former has one 1Gbps RJ45 port plus four 10Gbps SFP+ cages while the latter has 8 10Gbps SFP+ cages.

[–] gccalvin 2 points 2 years ago

DAC cables. Essentially copper with the transceiver's on the end. Ideally, your ports are sfp, so you can decide to do fiber or copper. Gives you the most flexibility.

[–] [email protected] 1 points 2 years ago (1 children)

https://mikrotik.com/product/crs305_1g_4s_in

For $150 you get a 4x 10g ports and can play with both RouterOS and SwitchOS.

https://www.ebay.com/itm/354640232219?

https://www.ebay.com/itm/115559006326?

none of these require branded SFP's so you can use commodity DAC's or SFPs.

[–] TurnItOff_OnAgain 1 points 2 years ago

If you require branded SFPs FS.com sells them at an incredibly cheap price

load more comments