McGarrah Technical Blog

Buying a 10Gbps or higher network on a homelab budget

This is a project I’ve been thinking about for a long time… how to get 10gbpe+ networking in a homelab without breaking the bank.

First option is just getting some DAC Cables and dual port 10Gbpe NICs then build a point-to-point ring network. That is relatively cheap and would set me up for future switched networks. DACs could be swapped out for GBics that use copper (thermal issues) or fiber (delicate).

Next is a relatively cheap at a couple hundred dollars switch with likely a low number of SFP+ ports. This is ~$200-$500 with anywhere from 2 to 16 ports at 10Gbps. Often switches with 10Gbps advertised only have one or two ports at that speed so shop carefully. You still have the cost of the NICs and cabling but only need one port on the NIC.

Lastly, you could go all in with an enterprise switch like the HP ProCurve 5406zl which is a module hosting monster of a switch. These are massively upgradable but come with a lot of complexity to set up and manage. They are also incredibly loud (intended for server rooms) and suck a ton of power which generates lots of heat (thermal load). These are getting cheaper but are heavy to ship and still usually several hundred dollars with modules that can cost thousands. Don’t expect a warranty on these as they are being pushed out of enterprise usage as end of life.

1. Point to Point network

If you buy Dual port NICs then you can run a ring network between each node as direct connections. It significantly complicates your network config but may be worth it for the cost savings depending on your budget.

2. Switched network

Here are a pile of network switches that are both new and ancient that could support 10Gbps.

Price per SFP+ port is one metric.

Power consumption and acoustic (noise) are others to consider.

Mikrotik switches

Mikrotik CRS310-1G-5S-4S+IN 10Gigabit Switch 1G port 5 x SFP ports 4 x SFP+ port

MikroTik CRS309-1G-8S+IN Switch Gigabit Ethernet Port and 8x SFP 10 Gbps Ports

MikroTik CRS305-1G-4S+IN 5-port Network Switch 10Gbps Gigabit Dual System

MikroTik CRS317-1G-16S+RM Cloud Router Switch Rack-mountable Manageable Switch with Layer 3 Features

ZL Chassis

HP Procurve 5406zl Switch

ZL modules

J9309A - HP ProCurve 4-port 10GbE SFP+ ZL Module 2C3 (module)

HP Procurve J9535A 20-Port 4-SFP Gig-T/SFP PoE V2 ZL Module for 5400zl Switch

HP ProCurve 24 Port Gig-T PoE+ V2 zl Module - J9534A

This only gets you part of the way to the price.

Dual NIC 10Gbps

DAC

Cable for direct connection without gbics in sfp ports

Summary

I have no decision on any of this and have been just thinking about it a lot as I start my journey into using Proxmox and Ceph on a local network using small 4 to 8 port 1Gbps Netgear switches that I happened to have on a shelf. My HP ProCurve 2510g-24 j9279a switches are completely out of date with their 100Mbps ports and their two SPF (not SPF+) DAC capable 1Gbps cross-connector.

Another options is 2.5Gbps switches with existing CAT5/6 cabling I have all over the place but it seems like a waste not to jump up to 10Gbps if I’m rebuilding stuff.

So final word is no decision on this and I will limp along with multiple 1Gbps networks.