Unfortunately the blog didn't link to the SFP+ module they're using, but everyone should know there's effectively 2 different generations of 10gbit sfp+ to ethernet^H10BASE-T modules. The old gen, labeled as 30 meters, draws ~3 W, and gets extremely hot (to the point it'll usually cause link flaps), and the newer gen, usually labeled as 100m or 80m, draws ~1.5 W, and runs much, much cooler.
This is the most important thing to know for anyone trying to do SFP+ to 10GBASE-T. It's too bad it wasn't covered in the article.
The thermal performance of the new modules is so much better that they'll make you want to throw away all of your old modules as soon as you try one. The old ones consumed a lot of power and put out a lot of heat. You can find examples of people adding active cooling to blow on them because they're so bad.
The new ones are great. They get warm but it's completely manageable. And the power draw is also closer to what the SFP+ ports on your device were probably designed to handle, so compatibility is better too.
I used to support a company that refused to run fibre, and had just run ethernet everywhere with the old 10GBE modules. Half their links were constantly flapping as the hardware powered off to cool down.
Even with the new modules, their ethernet cable runs were always in excess of what 10GBE could handle. I remember one at ~140 meters that was constantly renegotiating.
The worst part is, they had not a single element in their network that didnt support fibre. The fibre sfp modules for their radios was the same cost as the copper.
I love that these 10GBE modules exist but please (please please) just run fibre if you can.
I had this issue with old gen Unifi SFP+ to RJ45 10Gbe, 3 failed. Needed gloves to remove them. Bought newer gen and they are warm but i dont need gloves.
The difference between fiber module and RJ45 is massive. I found it funny that Ubiquity own switches (at least my Enterprise 8 PoE - the one they call "vintage" - I kid you not) will set the fan default to 35 (whatever that value means) if you use SFP+ RJ45 module (regardless of temps), and have fans at 0 without the module, when the switch is below 75 celsius.
Ironically, one of my switches with the RJ45 module runs cooler than the one without because Ubiquity does not let you do any fan control on Unifi Enterprise 8 PoE.
> Build in Marvell chip, Low Power Consumption (2.0W MAX @ 30m)
I have had 0 problems. One I use for my Sonic.net Ont and the other for my a 10G switch. Maybe I don't have issues because the cable lengths are so short (1-2')?
I just checked and both are VERY hot to the touch. Now I'm worried.
If you want to buy the cheaper old ones and are concerned about heat just add a usb fan. I have the same mikrotik switch in the post and 2 sfp to rj45 + this fan https://www.amazon.com/dp/B00G059G86?th=1 sitting on top and it makes a dramatic difference in temp.
I appreciate the craftiness of this but with the years I stopped being a fan of Frankenstein setups. The vendor should be honest; if the piece of tech is expected to heat up, just add a fan to it and manual toggle and a knob to control it or have it work auto.
I forget how I went about it, but I was able to get which chipsets their modules used. But honestly, unless they've sold out- 10GTek has been good for home use for me- at least for AOC and DAC. all my bidi stuff is second hand ciso off ebay
the AQC chips at least work with 1/10 only ports on older switches. And, it's not that hot, a single 1x1x0.5cm heatsink takes care of it. One large issue is that mikrotik's thermal design for SFP+ is a joke.
I don't agree that it is insane. It is less efficient than ideal, but 10gbit over copper is not necessarily dangerously hot or difficult to power.
I have a MikroTik CRS304-4XG-IN on my office desk with three out of four ports at 10gbit and it is perhaps 20 degrees above ambient on the outside. Warm but not hot. Passively cooled design.
A normal Windows laptop runs hotter than that when idle.
I have several 48 port 10G copper switches in my computer room with 40G uplinks. Dell OS9 jobbies. They are a bit noisy, but they don't actually run that hot. I also have some HPE 1G 48 porters and they run roughly the same temperature.
I used to daily drive Gentoo on my laptop. Lovely in the winter!
The cost, power, and length issues meant that it wasn't exactly well-received by the datacenter market back in 2006(!) when it was first released: DAC was the far more attractive option for a link from server to top-of-rack, and fiber was obviously superior (if not plain required) for anything beyond a hop to the next aisle over.
This left an incredibly tiny market, so obviously beyond the initial investment very little effort was put into developing new products for it. So now the prosumer market is hitting the limit of 1Gbps, 2.5GBASE-T and 5GBASE-T (both based on the techniques of 10GBASE-T, by the way) are becoming the norm, and suddenly network vendors remember that box of ancient 10GBASE-T transceiver chips that has been collecting dust in their warehouse.
Aaand suddenly you've got people buying what they think is a brand-new technology, but which is actually designed and manufactured using technology from a decade and a half earlier, and 10GBASE-T gets a bad name for being "hot" and "power-hungry". Turns out it is actually reasonably well-behaved if you actually make use of modern technology!
I expect we'll be using it for quite a while. 25GBASE-T and 40GBASE-T are even deader: A standard from 2016, which a decade later doesn't have a single available product? Mandatory switching to Cat8 cabling - and only a 30 meter range??? And no forwards-looking compatibility? Yeeaah, no thanks.
Right, but for example my fiber ONT only has an rj45 port so I'm stuck using one of these for the link to my UDM Pro. The router and my core switch use cheap DACs.
They're out there. Search engines are just ubiquitously poisoned to the point of uselessness for common terms[1].
But searching for specific gen4 x1 10-gig chips (like RTL8127) does produce some reasonably-inexpensive options for cards with sfp+. Give it a whirl. :)
[1]: I wrote this like it's a new problem, but looking back it may be an issue as old as the hills. I distinctly remember searching for any-brand 10-meg cards with DEC Tulip chips ~30 years ago, because that was the inexpensive PCI chip that actually worked. :)
I'm extremely happy after upgrading my network to 10gbit copper ethernet. It was much more expensive than I thought it should be, but worth it even if I only max it occasionally. Now I can easily fully saturate my 10gbit ethernet doing a first Time Machine backup or transferring files to my M.2 SSD NAS which saves me waiting rime and is satisfying to watch.
It's wild to me that 10gbit isn't the norm by now and tech people who should know better seem to think WiFi matches or even exceeds even 1gbit ethernet. My MBP connects to my WiFi7 setup(Ubiquiti E7) at a nominal 1.5-1.9gbit but Time Machine backups and file transfers are slower than plugging into 1gbit ethernet, probably in large part due to latency and retransmissions. Not to mention that ethernet works with near 100% reliability with dramatically less variation in speed and error rate.
It's wild to me Time Machine works on your network. Are you just doing "first backups" over and over again, or have you somehow achieved the very rare state where Time Machine can run for, say, a week at a time without falling over?
Sorry, this is snarky and off topic, but I'm nostalgic for the days when Time Machine "just worked".
For a very long time I thought Time Machine had become flaky, and I'm sure it's partially to blame, but with my current setup I've literally never observed it corrupt a backup and have to start over.
Before I was using one of the common Synology consumer NAS boxes that are often recommended. The NAS didn't report any errors with the drives or its own hardware, but at least once a month TM would glitch on at least one of my home laptops.
My new setup is an Asus FLASHSTOR 12 Pro Gen2 FS6812X. For a year now it's been running without a single apparent TM glitch while backing up multiple personal laptops and my work laptop. Sometimes I'm plugged in and sometimes I'm backing up over WiFi, but it's always worked.
I tried various recommended settings for the Synology and nothing helped so I strongly suspect that the Synology network protocol(SMB, AFP, etc.) implementations were either buggy themselves or at least not compatible with quirks in Apple's implementations. Synology->Asus fixed all my TM problems instantly and seemingly permanantly!
I can't remember the exact phrasing, but are you talking about the error message that essentially says:
"The Tardis is broken. Your backup has diverged into an entirely separate timeline, and I have no way of reconciling it. You may now sacrifice an entire weekend to do an initial backup again."?
I've been on a lucky streak for several years now, where I haven't gotten that one on any of my devices.
"Preparing backup..." taking an unreasonable amount of time is a regular occurrence, and some edge cases around adjusting TM backup size quotas aren't handled well. But other than that, TM has been working reasonably well for me to back up 10 TB over SMB to a Synology NAS.
My gripe is much more with Apple's abysmal support for SMB and NFS, especially after deprecating AFP. I've been back and forth between them over the years and over several OS versions, and their implementations for both are just terrible.
But over time SMB, for me, proved slightly more stable and performant, with the right tweaks in smb.conf, and authentication and permissions/ownership are easier to deal with than NFS, so I stuck with that.
I also yearn for the days where TM just worked, because somehow, the alternatives are even worse:
- Arq Backup does some things quite well, which is why I use it as part of my 3-2-1. But some of its bugs and implementation decisions just scream "hobby grade" to me.
- Kopia looks interesting, but it's not mature enough yet. Failed for me with absolutely cryptic error messages during repo init both times I tried it, with versions several months apart.
- Restic, Borg / Vorta: Not turnkey enough for me.
> "Preparing backup..." taking an unreasonable amount of time is a regular occurrence,
TM heavily throttles disk I/O used for backing up in order to ensure that normal user activity isn't affected. That makes it appear that TM is dramatically slower than you would expect which greatly annoys me. This becomes obvious after you run this command which will make both the preparing and transferring phases go closer to the theoretical speed you'd expect:
> TM heavily throttles disk I/O used for backing up
That makes sense, and I usually quite like that behavior. I barely ever notice an impact when backups are running.
However, this is happening every time on one machine (Intel iMac), and semi-regularly on another one (M3 MBP), after a fresh restart, giving mds_stores some time to settle down, and the most recent backup just hours ago, with no significant changes on disk since.
In a situation like that, I would expect the "Preparing backup..." stage to just take a second to create an APFS snapshot, and maybe a minute to diff that snapshot against the remote state. But not 10+ minutes.
But thank you for the hint about that sysctl parameter! I will certainly give this a try.
I've been using Time Machine for six months pointing to a network share on my TrueNAS box and it has worked fine. Sometimes a backup will fail when the Mac is taken off my home network (it doesn't play nice with Tailscale for whatever reason) but it will always work again if I tell it to retry the failed backup once I'm back on the local network.
> It's wild to me that 10gbit isn't the norm by now
I honestly just don't need it. Part of that is my ISP options top out around 1200Mbps, certainly. But I also just don't need that kind of speed inside my home. Streaming video needs at most 20Mbps or so, and I don't do much in the way of large file transfers. And when I do large file transfers, it's usually from or to the internet, and my home network is not the bottleneck there.
> It's wild to me that 10gbit isn't the norm by now
10G was too big of a step up from 1G. The expense and power required made it unattractive. Only recently have the interfaces for 10G over twisted pair become reasonably low power.
2.5G and even 5G are in a much better spot. It's where I recommend most people start as a default.
Yeah, 10GbE-over-copper switches are SO expensive. I bought a Ubiquiti enterprise switch for my home; 10Gb uplink and the copper ports are split between 2.5G and 1G. It's fine because only 1 or 2 clients can even talk 10G, and those are all across the house on Cat5 links anyway so they only negotiate to 2.5 even on a 10G port.
As much as I wanted to "future proof" by having a 24 port 10GbE switch... why? I'll just wait and buy one when I have a use for it.
> As much as I wanted to "future proof" by having a 24 port 10GbE switch... why? I'll just wait and buy one when I have a use for it.
As much as I enjoy looking at those wiring cabinets where every cable is cut to exactly the right length to reach a single port on the switch, this is why I prefer to leave an amount of slack in the wiring: It's good to be able to pull different wires to different switches depending on your needs.
One small high speed switch with enough ports for the couple of devices that can use it. One gigabit switch with a lot of ports to provide connectivity everywhere else.
This is what I ended up doing. A 2.5G switch for the few devices that can use it and a 1Gbit+PoE switch for all the other PoE and 1Gbit and less devices.
It does, actually. I dismissed it originally because I don't need most of the capacity, but considering I have no 10GbE copper ports available, and my home is being remodeled and I'm running some new ethernet to the other side of the house, I could take advantage of these ports now.
Also, I didn't realize my UDM Pro was kneecapped at 3-4Gbps IDS/IPS throughput. I think when I bought it I only had a 1 gig internet connection so it was overkill, but now I've got 10gig. It didn't even cross my mind that I had this bottleneck.
5G barely exists and mostly is the same price as 10Gb. 2.6Gb is taking off and mostly replacing 1Gb, but try to find a 5Gb switch, there really arent any, and most appear to be 10Gb switches with 5Gb PHY in them.
>It's wild to me that 10gbit isn't the norm by now and tech people who should know better seem to think WiFi matches or even exceeds even 1gbit ethernet.
2 things here. Upthread is the discussion about the old 10GBE modules that would constantly turn off due to overheating. Thats left a sour taste in a lot of peoples mouths.
I dont have anything in my home network that matters enough to have 10GBE anywhere. If I did, I would just get fibre. My wireless is fine for most purposes except some HD streaming, and plain old 1 gig works fine there.
Getting Time Machine to work reliably over a network is painful, even the old Apple-made Airport with built in TM stopped working twice a year.
However, I have multiple Macs where I simply have a USB-C laptop SSD attached for Time Machine and they have worked without issue for years. These laptop SSDs come in huge sizes nowadays, and you don't need an especially performant one, so they can be pretty cheap.
Heh it's honestly wild to me anyone needs over a gig. My work has a one gig fiber line supporting hundreds of employees and usage generally remains below 10%.
The high expense of 10gig is, in part, because it isn't widely necessary and the people buying it are willing to pay extra.
> The high expense of 10gig is, in part, because it isn't widely necessary and the people buying it are willing to pay extra.
I think the price has more to do with where you live and how the market is structured than how necessary it is. In Japan where there is competition between ISPs, I pay about $40/mo for 10Gbps.
Routers have also come down in price to where they are pretty affordable for consumers. I use a Ubiquiti Cloud Gateway Fiber [1] which has three 10Gbs ports (two SPF+, one 10GbE) for $279. A TP-Link router [2] with an upstream 10GbE port and 2.5GbE LAN ports and Wi-Fi 7 is about $140. 2.5 GbE NICs have become cheap and ubiquitous and could commonly be found on $150 mini pcs (before memory and SSD prices went crazy).
Yeah it's more than more than most people need, but I definitely appreciate having the increased speed when downloading 50GB games, uploading 200GB files to YouTube, or backing up files to the cloud. I've probably never maxed out the full 10Gbps, but exceeding 1Gbps is pretty easy in relatively common use cases.
And if you don't need a router, you can get 10gbit ports for much cheaper than that. The Mikrotik CRS305-1G-4S gives you one copper 1gbit port, four 10gbit-capable SFP+ cages, and is ~150USD [0]. Their whole lineup of switches with SFP+ ports can seen here [1].
I put 5Gbit internet into my home (fiber) to build my startup. I'm processing terabytes of data. I have over 100TB of storage in my basement. I can regularly saturate my internet connection. That said, I remember well when a 1Gbit connection provided enough bandwidth for a 500-person call center for daily workloads (back about 10 years ago).
That puts you in an extreme minority, even amongst enterprise businesses. Many medium sized enterprises have storage that looks like "a couple dozen TB total" for hundreds of staff.
Having 100 TB of storage in your home basement is an even more extreme minority than that. ;)
A gigabit connection is more than enough for a 500-person call center today.
Agreed. We had IP phones with 100 Mbps switches in between most of our computers and the rest of the network for a long time and very few people noticed. It'd only really be when I was installing a system upgrade or something, and I'd be like "man, it'd be nice if this didn't take an extra two minutes". For normal web access, 100 Mbps and 1000 Gbps aren't really discernable, until you're downloading large files. A lot of 4K streaming videos though, you'll start to feel it quite a bit faster.
And then hilariously, once you go above a gig, the reality is most sites won't serve them to you any faster than that anyways.
I found the nicest thing about fiber is I can hit over a gb/s uploading, which is often much more critical-path for whatever I’m doing than a download.
If that 500-person call center is a BPO, they'd have trouble with today's data-heavy workloads. I do agree, though. I'm in an extreme minority when it comes to this much bandwidth in my home.
Depends a lot on your work type and your prior exposure. If you only work "locally" and upload/download rarely, you may be way less demanding of your network than if you actually do distributed work with remote storage, high-bandwidth communicating tasks, etc.
Over 20 years ago, I was used to having 1g LAN for basic workstations and laptops in an office setting and probably 10-20g uplink from the building (shared by hundreds of staff). I also used 1g at home for my very small LAN between laptop, desktop, and SAN functions. But, my home ISP links were often terrible, such as 128k ADSL or even just a tethered GPRS phone at some points.
You end up with entirely different work styles when you have these different resource constraints.
1Gb Internet service seems low these days, much less 1Gb LAN. I have 3Gb Google Fiber service and actually get 2+ for individual downloads from some internet services like Steam. Even at 2Gb it's annoying to wait tens of minutes for 100+ GiB games to download. If I go on vacation I come home with 10s of GiBs of photos and videos on multiple devices that start syncing with cloud storage.
During the day I need to pull large data files from the work VPN so it's nice that that can happen at full speed even when Steam and movie streaming are also at full throttle. Combine that with backups and moving various files back and forth to my NAS and I'm very happy to have 10Gb local wiring.
This is one of those things where I just have to express that a lot of the HN crowd is entirely divorced from the reality the rest of the world experiences. ;)
Nearly nobody has multigig anything in the home, a probably surprisingly large percentage of business networking is 1gig LAN or less. And most people would not notice the difference if they did.
I am glad it works for you, but everyone else most certainly doesn't need it. (Yet.)
Personally, I do try for mostly gigabit in my home, because I do selfhost, but I have a ~800 Mbps download service (200 Mbps upload, it's asymmetric) that was only 500 Mbps when I signed up. And to be honest most of my patch cables are CAT5e because I'm cheap. I do make sure to run CAT6 through walls though because I don't want to ever have to do it again.
Also, I used to have Astound, and I feel so much sympathy for Google Fiber customers, you have no idea what's coming. If you thought Google had a reputation for bad customer service... just wait!
I disagree. I pay $120 a month for 5Gbps symmetric connection. I could upgrade that to 10Gbps for 2x, but there is no reason at this point. Even the local max from the cable company is more than 1Gbps, 1.2Gbps down / ~300Mbps up for around $80. Everything is streaming now. I work from home, on video calls. My better half will be watching something the on AppleTV streaming, the kid will be doing the same. I have Backblaze running to do backups to the cloud. 3 different laptops that will run TimeMachine backups to the NAS. The AppleTVs also have the Infuse app on them to stream local video files from the NAS. The security cameras are a constant 60Mpbs 24/7/365 to the NVR. The laptops can push a gig wireless and 2.5Gbps when plugged into the Thunderbolt docks. It is not clear that I need 10Gbps everywhere, but it has its uses. The NAS is at 10G. The link from the main switch to the router is 10G. The 3 APs in the house are at 2.5G and the 2 outside are at 1G. There was a noticeable difference when after I right sized the shared links paths up from 1G. When I say noticeable it both perceived and measured. I used to work doing switch bring up and competitive testing so I have a pretty good idea how this all comes together. Given that a reasonably cheap set of APs can now handle clients at above 1G and internet speeds in some areas being above 1G, moving to at least 2.5G in places is useful and not divorced from reality. I am in tech, but I have help my not tech friends upgrade APs, et.al. for their normal everyday home use cases and they have all been quite happy with the change.
Not being divorced from reality is the only reason I have not dropped $5K on the new Dream Machine Beast that was just released and have not swapped out my Enterprise 48 PoE (1st gen.) for the newest version that has 12 10G-BaseT ports.
I have it more for the fast nas access and being able to treat nas disks as more or less the same performance as if they were directly sata in my machine. Significantly less so about the external network aspect.
> It was much more expensive than I thought it should be
For 10G with copper, short runs will work fine on most anything (10m), with longer working ok on cat-5e cable (20 - 30m or so).
So, if you have existing wiring, especially in wall, just give it a try first, before gutting things. Most 10g ethernet adapters give you BER stats to see the error rate.
And, if there are issues, multi-gig may be an option, where it will drop down to 5 or 2.5 gbps, which is same as 10G, but with a reduced symbol table to handle the lower SNR.
You can definitely get stable 1gig throughput through WiFi. Doesn’t even need WiFi 7/6E - possible on 6. Ran a WiFi bridge like that for years and - no packet loss, consistent gig through and maybe +1ms latency
The gotcha is both ends needs to be good radios. So a router to router bridge tends to work better than router to end user device. Also had near LoS which presumably helped a ton
I went a slightly different route. My switches are linked with 10Gbps SFP+ across the apartment, but it was way too late (and too much hassle) to pull proper in-wall fiber. Instead I used one of those ultra-thin (around 0.1mm), unshielded fiber cables, and just snaked it through door frames and taped it along the walls. I'm genuinely impressed this stuff exists, it makes retrofitting fiber into a finished space so much less painful.
Most of my edge devices are still on 2.5GbE though, and I'm increasingly aware that for anything with plain SATA disks, the drives are the real bottleneck. Once I LAG'd 2×2.5GbE to get a 5Gbps pipe, it became obvious the network wasn't the slow part anymore in a lot of cases.
And yeah, the 10GbE SFP+ modules run hot, so hot that I would not lay my fingers on them for more than 2 seconds. I stuck 2 copper heatsinks on my module, not sure they do much but the module runs smoothly. Even so, I'm pretty happy with the overall setup: from my 10GbE-equipped Mac I can saturate multiple machines at once and I no longer think about the network most of the time, which was the goal.
And after getting 10Gbps working at home I was getting greedy, and looked at InfiniBand as well, 40Gbps and proper RDMA is very tempting compared to Ethernet. The catch for me was the practical side: IB needs PCIe slots and those chunky, inflexible cables. With most of my stuff being laptops, mini PCs and Macs, I just couldn’t see a clean way to route those or even plug cards in everywhere, so in the end the "door-frame‑friendly" skinny SFP+ fiber still won out for this apartment.
Good catch — that was actually my mistake. I mixed up cm and mm from the marketing material, and when I actually measured just now it came out to around 0.9mm. So not quite as thin as I claimed, and I probably wouldn't have noticed if you hadn't asked.
I did find some Kevlar-reinforced options that are supposedly ~0.3mm, but they seem to be raw fiber without connectors, purposed for drones, and I'm not sure about global availability.
Ah, I see. Thanks for checking. I'm considering doing a similar run at home, so was curious if there were any options that thin. 0.9mm should be plenty to work with. :)
> The most important question was the structured cabling in the walls; was it CAT-5E or CAT-6, or even CAT-6A? Remember from the last post, 10GBASE-T might work over short runs of -5E (even though officially it's not meant to be able to).
This is not quite correct.
The primary problem is cross-talk. Copper wire itself will carry the relevant frequencies up to 100m without issue but even with balanced pairs the balancing is not perfect and the "dirty paper precoding" is not perfect so some cross-talk will occur. How long you can go with Cat-5e depends on how well the wire is twisted, how many wires are bundled together, are there any loops or tight bends, and other factors. Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out.
Bottom line is: for almost any normal home or apartment any quality Cat-5e cable properly terminated will carry 10GBase-T without issue. In fact if you have problems I would first re-terminate the cable before assuming you need to run new cable. Cat-6 or 6A just isn't necessary.
As a PSA: beware of "CCA". I've noticed Amazon and eBay are absolutely flooded with cheap chinese electrical and networking cable that shows nice shiny copper in the pictures but is actually "copper clad aluminum". If they mention anything at all they code it as "CCA" cable without explaining what that means.
CCA cable cannot, by definition, be ethernet cable. I won't get into the full technical details but the standard was amended to clarify that only pure copper wires are acceptable for ethernet. Personally I would not dare use CCA for anything. It has lower performance, lower current-carrying capability for the same wire diameter (inherent in aluminum), and introduces the risk of oxidation and loosening of connections as people will treat them as copper connections when aluminum needs special installation procedures and connections to avoid them coming loose over time. For electrical connections especially this not only can but absolutely will lead to a fire over time if not treated with the appropriate care. All it takes is a little bit of mechanical action scraping off the thin copper layer and you now have an effectively aluminum wire - a time bomb ticking away.
> Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out.
This isn't quite right. Cat6A guarantees the frequency and therefore effective bandwidth that is available. But it doesn't talk about physical cable type. A common structured cable for Cat6A is U/FTP which has no plastic separator, no outer shield, but rather each pair is wrapped in foil and is actually less tightly twisted than even Cat5e. Anyone can buy U/FTP cable, but to actually be Cat6A it has to be installed correctly and the installation certified to support Cat6A speeds.
The only thing I'd caution anyone else looking to do the same is doing a software router/fw like the Portectli is it's usually not hard to get the raw bandwidth to look nice with big flows but the new connection latency, connections per second, jitter, and QoS handling tend to suffer vs something with hw offloads (which is what most are used to even with cheapo gigabit home AP+router+switch combos). It's also not usually the cheapest way to get the 10G class NAT/L4 FW bandwidth, but it is usually the cheapest way to get "full" FW functionality if you don't care as much about the performance.
If you want a full FW solution that can actually FW+NAT at 10G bidirectional without breaking a sweat then something like the FortiGate 90G is the cheapest thing I've found that performs really well across the board. Great QoS, great latency, amazing throughput performance (does well with even small packet sizes in a single stream), easy enough to use UI (once you get oriented), low power. If you want to enable all of the NGFW stuff (e.g. AV and IPS) then it'll dip below line rate though.
If you just want something that NATs/connection direction oriented filtering like a "normal" home router then something like the MikroTik CCR2004 can get you better than the performance they got on the VP2440 + give you 12 ports of 10G SFP+ to work with. If you were planning to do "fancy" FWing/functionality beyond a normal home NAT FW (with decent managed switching built in) then the feature set will be a bit limiting, of course.
I don't understand the decisions to use RJ45 copper cable for anything higher than 1GbE, unless
a) You like small computers and these only comes with RJ45 ports and 0 PCIe slots
b) You enjoy fighting with super rigid cables, port flaps and hot modules
I upgraded my home to full 10Gbps network at 2018. At that time:
* 10Gbps SFP+ PCIe NICs (Mellanox CX3 and similar cards) are already dirt cheap ($15 for OCP adapted ones, $50 for native PCIe cards) because datacenters are upgrading to CX5
* 10KM SM SFP+ modules are already dirt cheap (~$6) because 4G BBU-RRU communication uses these
* Silent 10Gbps SFP+ switches start appearing (still a little pricey); 10Gbps RJ45 switches are not available anywhere
Those 10KM SM modules will happily run off <20m MM fibers so I just used whatever fiber I have at hand. The price is so cheap that I think the RJ45 upgrade path (2.5G has not been universal, 5G still very far away, 10G is so pricey) looks like a scam.
After 2024:
* 25G/100G SFP+ PCIe NICs are dirt cheap (~$80 for single port 100G NIC)
* 100G QSFP28 modules are in acceptable price; 25G SM/CWDM SFP+ modules area already dirt cheap (~$50 for 6-slot CWDM ones) because 5G BBU-RRU communication uses these
* Silent 100Gbps switches start appearing (you can get 24*25G + 4*100G under 100W; still a little pricey); 100Gbps RJ45 switches are not available anywhere
I already have an experimental 6*25G CWDM fiber connection between my 2 rooms. Given that most of my devices wouldn't need anything higher than 10Gbps, I'm thinking of giving each room a single 100G port, split them into 4 lanes in the room with passive wavelength splitters for different devices, which further reduces cost on switches.
Power usage is a good reason to stick with non-SFP ports.
And you can use CAT5e with 10Gbps networks given the run is short enough. I run 2.5Gbps over CAT5e with no issues throughout my house (I don't need that much bandwidth except when downloading Steam games).
I might have been lucky, but in the one home and one office were I've connected 10gbit switches and PCIe cards, it has just worked. Especially the office was a nice surprise, because it is at least 20 meters (probably more) of unknown cabling and at least one unknown patch panel between the utility closet where the NAS lives and the desk area. The cables were run 15 years ago, so I expected it to be cat 5, but clearly not.
It is nice moving/streaming large files across the network at 10 gbit. It really is ten times less waiting than with plain old gigabit.
Of course, most of the time I'm working with lots of small files and then the spinning disk array in the NAS has no chance to saturated the this giant pipe, or even a normal gigabit connection...
Do you mean Cat 6 or Cat 5e? Cat 6e isn't a thing and 6a didn't exist in 2006. 6 was certainly more future-proof at the time, although arguably 5e is still fine even today. (Super-gigabit consumer equipment didn't really exist until the last five years and it's till notably more expensive and less common than gigabit, which runs on 5e just fine.)
Cat 6 for sure. I thought there was some extra tweak beyond that, but I probably misremembered after all this time. Perhaps it was just that it was plenum-rated..
> The cables were run 15 years ago, so I expected it to be cat 5, but clearly not.
Did you check the jackets? I've got Cat5 (no e) marked cable running 10gbaseT. A lot of cable exceeds the specs on its jacket and specified wire provides enough signal to noise for the provided length in dense conduit. When you have shorter runs, without dense wiring, lesser cable can work.
I can't see the cable jackets. There's just outlets in the walls. I ask building management to make a link between outlet X and Y and then they make it happen via one or more patch panels that I don't know where they are. :-)
People downthread has written, cat6 was already the goto thing 15+ years ago so perhaps that is what they put in when the office space was built. It is a well built office!
Same. I ran CAT6 from one end of our house to the other, because my home office is in the opposite corner as the fiber coming in to the router. After running that, and manually crimping everything, it Just Worked from the first time I turned everything on. That felt pretty good.
I have 1.5/900 fibre to my house, and I bring a 2.5 line from the modem to my home office where a 2.5 switch delivers it to my workstation, laptop, and unraid NAS. But those devices are all themselves just gigE I think, and I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available.
More and more regular people are getting network storage appliances. More and more people have laptops with SSDs that can write at 4 or 5 GB/s. Why shouldn't they get to use all of it?
> I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available.
There are many things along the way that would get in the way of a home user downloading something from the internet that would hit that 5GB/s speed. It's not that people should be "banned" from it or something, more that the investment cost isn't worth it.
I regularly saturate my 1G home and 1G office connection syncing ~6GB files between the two. It's also nice to be able to download a 100G or so game quickly. Remote backups to cloud storage also benefit from fast upload speeds (and more importantly, restores).
We have a 5gbps pipe; routinely download games from Steam at >3gbps; when I had to reinitialize my cloud backup it was >4gbps. All of this without impacting anyone else on the pipe.
Yah, our P95 bandwidth is just a few megabits per second. But it's not that expensive and routinely saves me a few minutes here and there.
10gbps on the LAN is more broadly useful. Pegging it for a file share is a daily occurrence.
Also storage has gotten super expensive lately, and rather than upgrading my machines/consoles I've been offloading games and downloading them as needed and now am routinely downloading dozens of GB just to play a game.
My gaming time is limited so the faster the better.
I should have said most home users. My point is that more bandwidth at this point probably won’t affect 99.999% of home users.
What’s described in the post is the tech equivalent of supe-ing up a sports car and then driving it in rush hour traffic. It’s fun to geek out doing it, but practically in everyday use the difference will be negligible. Even with large file uploads and downloads, there’s a good chance that services won’t reach those throughputs end to end.
What’s telling is that the post shows screenshots and charts from artificial speed tests. No videos of the Dropbox client chugging away with throttled uploads.
In my experience unless you actually pay attention and get something with a dram cache it will sustain that speed for all of 5 seconds and then drop to near useless, and with the current dram shortages that is getting harder and harder to justify.
I just did a build out where ram cost as much as the GPU and both are individually 3x more expensive than the CPU and MB combined.
A decent 2TB nvme drive was also more expensive than the CPU and MB combined.
A 10GB network connection can happily transfer at that speed all day, your SSD is unlikely to maintain Sata speeds unless you actually shelled out for something decent, what came from the manufacturer in the laptop is not that.
I regularly hit 4.7 gbps on a 5 Gbps line pulling files (usenet is usually faster than torrent, but the latter can be equally as fast depending on the torrent & how good the client software is). It's great to just grab an entire movie series in 4k Blu Ray remux quality in 5 minutes and go. No real need to plan ahead for anything.
Yeah, Usenet is crazy quick. I've saturated 10Gbps on Usenet before, pretty easily.
Torrents (as you mention) can be equally as fast. Well seeded, private tracker torrents can easily download at above 1Gbps (depending on piece size, etc).
And here I am sitting in Brooklyn and haven't had one apartment that has had fiber as an option. I get to pay Spectrum $90/month for "400/20" and in reality get 100/10.
Yeah I foolishly paid for 500 Mb/s but the only things I ever get over even 200 Mb/s for in the UK are Steam downloads (and speed tests). Everything else seems to be roughly throttled at that rate.
Yeah the default is they supply a "modem" with a regular ethernet port, so a familiar config to someone coming from cable or DSL, but inside it's a QSFP that you can move to your own equipment if you want.
I've been running it like this in a closed comm box for the last 3 years without any issues. SFP+ modules actually do not use that much power, it's just that it's concentrated into a small package, resulting in high temps.
Cripes. When possible just do fiber and DACs. Faster and much cooler than 10Gbit. 10Gbit uses an absurd amount of power per port thus the need for all that cooling.
Folks have been saying that conduit is the way, and the fiber is the future, and all kinds of things like that for decades so far.
But the simple truth for all those decades is this: When there's already cat-whatever cable in the wall, it generally still works.
Decently-installed conduit (ie, actually-usable conduit) adds a ton of time and expense, which is why it is very seldom used for data circuits in residential structures.
The cable that exists is a lot better than the conduit that doesn't. And copper ethernet is bog-standard like MP3 is: It isn't the best in any technical sense at all, but everything supports it. Universal compatibility is pretty nice.
---
So the ongoing cost of copper 10gbe is electricity. Someone else here in the comments says that a copper 10GBe SFP+ module can use ~3 Watts, or that a newer one can use about 1.5 Watts.
We can be generous by using the larger figure of 3 Watts, and 8 devices..
With 4 ports, eight 10-gig endpoints @ 3 Watts each, and $0.19 per kWh [delivered]: That's $3.28 per month, or about $400 per decade.
If we assume 1.5 Watt endpoints, then that number halves.
If we subtract the power consumption of fiber SFP+ modules (or media converters or whatever) to make the number a relative comparison instead of an absolute, then that figure goes down further.
Most folks don't care about network cabling these days. For many/most common consumer uses, wifi (with mesh nodes or whatevr) is quite good for streaming video and playing online games.
These people aren't seeing the whole picture, but that's not new: They simply don't care, just as most folks have never cared.
And they care less and less as our (rather amazing) portable pocket supercomputers continue to supplant desktop computing in normal Internet-connected life.
Some folks are even actively against wired home networks as a premise: When I'd hang out on Reddit, sometimes I'd see photos of well-orchestrated and fairly intense home network infrastructure, but the new buyer of the home just wants it all gone from their sight and is seeking advise about getting that accomplished.
---
The house I live in right now has zero phone jacks (though the overhead wires still connect to the house before disappearing completely), and it only has 1 run of functional coax (which connects to the docsis modem).
I've added some decently-installed Cat 5e, which is completely adequate for what I'm doing. (That's been a trend for me for decades: When I live in a place, I tend to hardwire my stuff.)
But the next person probably won't care about that at all. If I move out and come back to visit in 5 years, the cabling I've added will probably have been erased.
And in the next place I live, I'll probably have to add more wire again. That's normal to me; again, I've been doing that for decades now.
---
And that's OK. I'm a bit of a geek when it comes to computers and networking.
Most people are not that way at all, and most people never have been that way. Most people don't give a fuck about this stuff at all.
And this is all completely acceptable.
(I guess I could choose to hate it instead, but life is too short to hate this kind of thing. Acceptance is both simpler and better-productive.)
I put cat6a in my last house and plan to in my current house "just in case". The cable isn't that much more expensive and nobody wants to do cabling again so why not.
But I'll keep using a gigabit switch because I have absolutely no idea what I'd use 10G for. It's crazy that gigabit was affordable for me as a student in the early 00s and between then and now we've gone from DVDs to 4K and it's still plenty fast enough. In fact, most people are happy with WiFi (not me, though).
Yep... Did this 3 years ago in my home and came to the same conclusion. If I were to do it again, I'd run fiber through the walls instead of Cat-6a. It took forever to find SFP+ modules that would work with my Unifi setup... (not wanting to pay for more than one router or switch with native 10g RJ-45 ports, which are still very expensive.) I loose POE but, on the whole, it would've been much easier and much less costly--I think--to have just run fiber.
Using SFP+ modules for 10GbE is interesting, I deployed a full 10GbE network in my home connected by 5Gbit/s symmetric fiber from AT&T and a 1Gbit/s x 100mbit/s DOCSIS 3.1 connection from Spectrum as backup, I did this around August of last year and used UniFi/Ubiquiti for everything. Their new XG and XGS products all support 10GbE natively, without needing SFP+ modules to connect. I'm using the UDM Pro Max with an XGSPON ONU flashed with the community firmware from PON Wiki (see https://pon.wiki/category/att/ ) and the 2.5GbE port as my second WAN link to the Spectrum modem. I'm not doing any sort of bonding/load balancing, just failover so I'm almost always just using AT&T. There's a short DAC cable connecting the UDM Pro Max to a Switch Pro XG 24-POE which serves 10GbE to the rest of the house. The only SFP+ module involved is the XGSPON ONU. Most of the devices in the house connect to WiFi served by the UniFi U7 Pro XGS APs, which take 10GbE w/ POE and give off WiFi 7.
I pretty much always get at least 4600Mbit/s both directions over AT&T, and generally cap it out. Spectrum typically gives me at least 800Mbit/s down. The ISPs are definitely the bottleneck because neither provides dedicated infrastructure from each house to the CO, instead you have some sort of aggregation point which has a shared backhaul and that means you compete for resources, but having the largest plan on each at least gives you traffic priority.
Realistically, I don't need any of this, I was doing just fine with normal gigabit fiber with a 2.5GbE network before I moved last year, but it's nice knowing everything is as fast as possible in my path and that eliminates congestion and local network resource contentions as causes when a problem arises, I know it's pretty much always upstream of me somewhere.
Regarding the proxmox cluster, I would advise not to set up a cluster unless you really need to, it adds complexity in order of shutting down the server, HA, Quorum votes, ceph monitor, among many others, and if one server goes offline, or even the wrong order, it will impact the others and sometimes some tough time in recovery, or data loss.
If you’re going to go to all this trouble, please do seriously consider pulling singlemode fibre instead. It is more planning up front, but when you’re ready for your next upgrade cycle, you can reuse the same fibre and change out the hardware on each end.
Why recommend single mode out of interest? I've just done some fibre in my house with OM3 because that's what I had, I couldn't really work out why you'd prefer one over the other for residential lans.
Multimode is probably fine, provided it's OM3 or higher. You might need to think about it down the road, 5+y out. Having a spool just lying around would make it a perfectly reasonable choice. The fibre itself is a little more expensive; the optics are cheaper. But multimode does run into physical limitations a little faster, so when you decide you want to go to 100G, or 400G, or beyond, you may need to pull fresh media.
Singlemode you just never have to think about again. It's likely to be more future-proof than your actual physical walls.
I was surprised that the old Cat5e in my home supports 10Gbps without any issue, so went ahead and upgraded the rest of the network with 10Gbps switches (expensive Ubiquiti gear, but worth it to talk at 10Gbps between all my machines, even though the internet is only 5Gbps Fiber).
Yeah, my home is cat5e (I think!) and I was only able to get 2.5 or 5Gbps to my _single_ 10GbE capable machine I had at the time.
So I ended up just buying a switch that supports 2.5Gbps over copper since it's such a huge expense to step up. Anyway, my devices are pretty much 2.5Gbps (APs, flex mini switches, etc).
One thing I'll add that I learnt in the process of doing my own house: It's not just the cable type but also the SFP module that can limit the distance. I used MicroTik hardware and their S+R10J modules are limited to ~30m for 10Gbps speeds.
10GBase-T can still be nice if you're going to do native PoE (which the author did not) or you expect a mix of devices with 1/2.5/5 which you don't want to or can't upgrade/replace immediately (which is where it sounds like the author was situated).
The new 10GBASE-T SFPs are actually not too bad - you can get the full 100 meters at half the wattage it takes for the old space heater generation ones to reach 30 meters. Based on the article, the author did not know there were newer cooler options for the ~the same price.
Has anyone tried the InvisiLight kits? They give you a 600 micron fiber optical link in the kit which you can hide down the side of carpet etc. I haven't tried it but a few YT vids made it look interesting.
This is the approach I used and tried to make a product from. I’m not pursuing that anymore but sharing for folks who want to run wire without dealing with drywall.
Looks like a complicated and expensive way to avoid running copper. "Up to 1Gbps" is not confidence inspiring, if you really can't put holes in your walls I'd try a powerline kit first for 1/5th the price
It depends on the topology of your circuits. If you can get both adapters on the same circuit that is ideal, same phase is often okay at reduced speed. Different phases or having other complicated stuff in the way like subpanels is probably what you were dealing with.
MoCa is fine, but if the coax was retrofit replacing it with proper ethernet cable isn't too hard.
I'm surprised they got a reliable 10Gbps USB adapter. I've tried 3 of them and they're all trash. They cannot sustain multi-gigabit speeds and end up doing like 400Mbps when you hit them hard. I think I have sabrent and ugreen and one other.
They all just sit in a drawer, if I need a USB-C ethernet adapter i just grab my trusty 1 gig one.
Even CAT5e can do 10 Gbps if you do a decent job making the cable, even though it's out of spec. 6A can do it easily, in spec. I used it in my previous home.
I much prefer to use fiber, but copper with good old RJ45 works fine for 10 gig.
For me the threshold has always been "can I stream a 4K movie from the NAS downstairs or from my seed box". No real need for anything above. Still I ran 10Gb single mode fiber in all the ducts.
I wonder what the idle vs at load difference is for power draw/heat. Would really love to see this feature in reviews some!
I wonder if you could negotiate down to 1gbit until you see some level of activity, if that would help at all?
I'm still eyeing 10Gb, but if my home needs +30w for three computers, I don't feel like it's really worth it. Would love to see more details on the power consumption from folks, especially tuning for idle.
It's 4x faster? If you have a storage server, its easy to exceed 2.5 or even 5 gigabit with SATA spinning disks if they're in an array. 10 gig is fast enough to work with network storage like it's local: do your work, store your steam library, edit videos, etc, but store it all somewhere else and share it with multiple computers.
Unfortunately the blog didn't link to the SFP+ module they're using, but everyone should know there's effectively 2 different generations of 10gbit sfp+ to ethernet^H10BASE-T modules. The old gen, labeled as 30 meters, draws ~3 W, and gets extremely hot (to the point it'll usually cause link flaps), and the newer gen, usually labeled as 100m or 80m, draws ~1.5 W, and runs much, much cooler.
Example of the new gen: https://www.amazon.com/Wiitek-Transceiver-Compatible-UF-RJ45...
Old gen: https://www.amazon.com/10Gtek-SFP-10G-T-S-Compatible-10GBase...
Typically the old gen uses a Marvell AQR113C, and the new gen uses a Broadcom chip that I forget the number of off hand.
This is the most important thing to know for anyone trying to do SFP+ to 10GBASE-T. It's too bad it wasn't covered in the article.
The thermal performance of the new modules is so much better that they'll make you want to throw away all of your old modules as soon as you try one. The old ones consumed a lot of power and put out a lot of heat. You can find examples of people adding active cooling to blow on them because they're so bad.
The new ones are great. They get warm but it's completely manageable. And the power draw is also closer to what the SFP+ ports on your device were probably designed to handle, so compatibility is better too.
I used to support a company that refused to run fibre, and had just run ethernet everywhere with the old 10GBE modules. Half their links were constantly flapping as the hardware powered off to cool down.
Even with the new modules, their ethernet cable runs were always in excess of what 10GBE could handle. I remember one at ~140 meters that was constantly renegotiating.
The worst part is, they had not a single element in their network that didnt support fibre. The fibre sfp modules for their radios was the same cost as the copper.
I love that these 10GBE modules exist but please (please please) just run fibre if you can.
10Gbase-T is here to stay because you can deliver POE over it to wifi APs.
Oh totally, heck I pitched the concept of a 10GBE Passive POE switch the other day.
That said, IIRC a lot of people running fibre + power.
BCM84891L. I like these modules (select 80 or 100 m in the drop down): https://www.luleey.com/product/10gbase-t-sfp-to-rj45-copper-...
Using this module, I was able to get a stable 10 gig over a 75 feet long, 20 year old run of Cat 5e.
I had this issue with old gen Unifi SFP+ to RJ45 10Gbe, 3 failed. Needed gloves to remove them. Bought newer gen and they are warm but i dont need gloves.
Same, just had two fail, in a qnap 100gb switch with a reasonable level of cooling… Meanwhile the 100g optics are actually doing just fine!
The difference between fiber module and RJ45 is massive. I found it funny that Ubiquity own switches (at least my Enterprise 8 PoE - the one they call "vintage" - I kid you not) will set the fan default to 35 (whatever that value means) if you use SFP+ RJ45 module (regardless of temps), and have fans at 0 without the module, when the switch is below 75 celsius.
Ironically, one of my switches with the RJ45 module runs cooler than the one without because Ubiquity does not let you do any fan control on Unifi Enterprise 8 PoE.
They can be a pig to remove from the switch too. My god.
The "new gen" link you says
> Built-in Newest Marvell chip, Max. Power Consumption 1.65W
Is that wrong?
Also, I'm using this:
https://www.amazon.com/dp/B0B3F5DSXJ?ref_=ppx_hzsearch_conn_...
which says
> Build in Marvell chip, Low Power Consumption (2.0W MAX @ 30m)
I have had 0 problems. One I use for my Sonic.net Ont and the other for my a 10G switch. Maybe I don't have issues because the cable lengths are so short (1-2')?
I just checked and both are VERY hot to the touch. Now I'm worried.
If you want to buy the cheaper old ones and are concerned about heat just add a usb fan. I have the same mikrotik switch in the post and 2 sfp to rj45 + this fan https://www.amazon.com/dp/B00G059G86?th=1 sitting on top and it makes a dramatic difference in temp.
I appreciate the craftiness of this but with the years I stopped being a fan of Frankenstein setups. The vendor should be honest; if the piece of tech is expected to heat up, just add a fan to it and manual toggle and a knob to control it or have it work auto.
Wow, and at essentially the same price!
Yeah, the new ones have gotten much cheaper it seems. About a year ago they were ~2x.
I think a lot of patents have recently expired for 10GbaseT. I’ve noticed it getting a lot more accessible.
Any fs.com recommendations? (or, just look at the power numbers?)
https://www.fs.com/c/10g-sfp-63?203=21772
I forget how I went about it, but I was able to get which chipsets their modules used. But honestly, unless they've sold out- 10GTek has been good for home use for me- at least for AOC and DAC. all my bidi stuff is second hand ciso off ebay
can't edit: I think I just asked FS.
the AQC chips at least work with 1/10 only ports on older switches. And, it's not that hot, a single 1x1x0.5cm heatsink takes care of it. One large issue is that mikrotik's thermal design for SFP+ is a joke.
Thanks! 10Gb Eth is insane for exactly this reason (optical SFP+ modules are way cheaper and more reliable)
I don't agree that it is insane. It is less efficient than ideal, but 10gbit over copper is not necessarily dangerously hot or difficult to power.
I have a MikroTik CRS304-4XG-IN on my office desk with three out of four ports at 10gbit and it is perhaps 20 degrees above ambient on the outside. Warm but not hot. Passively cooled design.
A normal Windows laptop runs hotter than that when idle.
I have several 48 port 10G copper switches in my computer room with 40G uplinks. Dell OS9 jobbies. They are a bit noisy, but they don't actually run that hot. I also have some HPE 1G 48 porters and they run roughly the same temperature.
I used to daily drive Gentoo on my laptop. Lovely in the winter!
Yes the noise is my main complaint for my 10G switch. I didnt expect that high frequency part.
Minor nitpick but they are both considered Ethernet. The 4 pair copper one is 10GBASE-T.
The main issue is that it is ancient.
The cost, power, and length issues meant that it wasn't exactly well-received by the datacenter market back in 2006(!) when it was first released: DAC was the far more attractive option for a link from server to top-of-rack, and fiber was obviously superior (if not plain required) for anything beyond a hop to the next aisle over.
This left an incredibly tiny market, so obviously beyond the initial investment very little effort was put into developing new products for it. So now the prosumer market is hitting the limit of 1Gbps, 2.5GBASE-T and 5GBASE-T (both based on the techniques of 10GBASE-T, by the way) are becoming the norm, and suddenly network vendors remember that box of ancient 10GBASE-T transceiver chips that has been collecting dust in their warehouse.
Aaand suddenly you've got people buying what they think is a brand-new technology, but which is actually designed and manufactured using technology from a decade and a half earlier, and 10GBASE-T gets a bad name for being "hot" and "power-hungry". Turns out it is actually reasonably well-behaved if you actually make use of modern technology!
I expect we'll be using it for quite a while. 25GBASE-T and 40GBASE-T are even deader: A standard from 2016, which a decade later doesn't have a single available product? Mandatory switching to Cat8 cabling - and only a 30 meter range??? And no forwards-looking compatibility? Yeeaah, no thanks.
Right, but for example my fiber ONT only has an rj45 port so I'm stuck using one of these for the link to my UDM Pro. The router and my core switch use cheap DACs.
Have you looked into using an unofficial ONT instead of the one supplied by your ISP? See: https://pon.wiki/guides/install-the-8311-community-firmware-...
I have not. I'm not sure what this does for me. My ONT is not an optional expense and I'm not bothered by using cat6 here.
Yeah but home users already often have 75-100 foot runs of cat 5e in the walls, and those work fine for 10G-baseT.
Problem is you get X1 pcie gen4 cards for 10g eth but not sfp
…and a lot of consumer boards have spare x1s
They're out there. Search engines are just ubiquitously poisoned to the point of uselessness for common terms[1].
But searching for specific gen4 x1 10-gig chips (like RTL8127) does produce some reasonably-inexpensive options for cards with sfp+. Give it a whirl. :)
[1]: I wrote this like it's a new problem, but looking back it may be an issue as old as the hills. I distinctly remember searching for any-brand 10-meg cards with DEC Tulip chips ~30 years ago, because that was the inexpensive PCI chip that actually worked. :)
What's really insane is 40G - nobody wants it so you can buy kit for next to nothing. I got a 72×QSFP Cisco Nexus (with EVPN & VXLAN) for £50 inc P&P.
Old, hot and loud.
Industry went for 25G and 40G gear is long EoS and EoL.
I'm extremely happy after upgrading my network to 10gbit copper ethernet. It was much more expensive than I thought it should be, but worth it even if I only max it occasionally. Now I can easily fully saturate my 10gbit ethernet doing a first Time Machine backup or transferring files to my M.2 SSD NAS which saves me waiting rime and is satisfying to watch.
It's wild to me that 10gbit isn't the norm by now and tech people who should know better seem to think WiFi matches or even exceeds even 1gbit ethernet. My MBP connects to my WiFi7 setup(Ubiquiti E7) at a nominal 1.5-1.9gbit but Time Machine backups and file transfers are slower than plugging into 1gbit ethernet, probably in large part due to latency and retransmissions. Not to mention that ethernet works with near 100% reliability with dramatically less variation in speed and error rate.
It's wild to me Time Machine works on your network. Are you just doing "first backups" over and over again, or have you somehow achieved the very rare state where Time Machine can run for, say, a week at a time without falling over?
Sorry, this is snarky and off topic, but I'm nostalgic for the days when Time Machine "just worked".
For a very long time I thought Time Machine had become flaky, and I'm sure it's partially to blame, but with my current setup I've literally never observed it corrupt a backup and have to start over.
Before I was using one of the common Synology consumer NAS boxes that are often recommended. The NAS didn't report any errors with the drives or its own hardware, but at least once a month TM would glitch on at least one of my home laptops.
My new setup is an Asus FLASHSTOR 12 Pro Gen2 FS6812X. For a year now it's been running without a single apparent TM glitch while backing up multiple personal laptops and my work laptop. Sometimes I'm plugged in and sometimes I'm backing up over WiFi, but it's always worked.
I tried various recommended settings for the Synology and nothing helped so I strongly suspect that the Synology network protocol(SMB, AFP, etc.) implementations were either buggy themselves or at least not compatible with quirks in Apple's implementations. Synology->Asus fixed all my TM problems instantly and seemingly permanantly!
Backup failed.
Again. I’m on a Synology, and your comment is interesting.
I can't remember the exact phrasing, but are you talking about the error message that essentially says: "The Tardis is broken. Your backup has diverged into an entirely separate timeline, and I have no way of reconciling it. You may now sacrifice an entire weekend to do an initial backup again."?
I've been on a lucky streak for several years now, where I haven't gotten that one on any of my devices.
"Preparing backup..." taking an unreasonable amount of time is a regular occurrence, and some edge cases around adjusting TM backup size quotas aren't handled well. But other than that, TM has been working reasonably well for me to back up 10 TB over SMB to a Synology NAS.
My gripe is much more with Apple's abysmal support for SMB and NFS, especially after deprecating AFP. I've been back and forth between them over the years and over several OS versions, and their implementations for both are just terrible.
But over time SMB, for me, proved slightly more stable and performant, with the right tweaks in smb.conf, and authentication and permissions/ownership are easier to deal with than NFS, so I stuck with that.
I also yearn for the days where TM just worked, because somehow, the alternatives are even worse:
- Arq Backup does some things quite well, which is why I use it as part of my 3-2-1. But some of its bugs and implementation decisions just scream "hobby grade" to me.
- Kopia looks interesting, but it's not mature enough yet. Failed for me with absolutely cryptic error messages during repo init both times I tried it, with versions several months apart.
- Restic, Borg / Vorta: Not turnkey enough for me.
> "Preparing backup..." taking an unreasonable amount of time is a regular occurrence,
TM heavily throttles disk I/O used for backing up in order to ensure that normal user activity isn't affected. That makes it appear that TM is dramatically slower than you would expect which greatly annoys me. This becomes obvious after you run this command which will make both the preparing and transferring phases go closer to the theoretical speed you'd expect:
sudo sysctl debug.lowpri_throttle_enabled=0
> TM heavily throttles disk I/O used for backing up
That makes sense, and I usually quite like that behavior. I barely ever notice an impact when backups are running.
However, this is happening every time on one machine (Intel iMac), and semi-regularly on another one (M3 MBP), after a fresh restart, giving mds_stores some time to settle down, and the most recent backup just hours ago, with no significant changes on disk since.
In a situation like that, I would expect the "Preparing backup..." stage to just take a second to create an APFS snapshot, and maybe a minute to diff that snapshot against the remote state. But not 10+ minutes.
But thank you for the hint about that sysctl parameter! I will certainly give this a try.
TM makes my M2 Air chug when it runs against a local Samsung SSD. It does no such I/O throttling.
Time Machine to a network share via Samba has been pretty reliable for me - only once has it corrupted itself in the five+ years I’ve been using it.
Amusingly enough Time Machine to a local drive failed completely.
I've been using Time Machine for six months pointing to a network share on my TrueNAS box and it has worked fine. Sometimes a backup will fail when the Mac is taken off my home network (it doesn't play nice with Tailscale for whatever reason) but it will always work again if I tell it to retry the failed backup once I'm back on the local network.
> I'm nostalgic for the days when Time Machine "just worked".
I’ve used it for a long long time and never knew these glory days.
> It's wild to me that 10gbit isn't the norm by now
I honestly just don't need it. Part of that is my ISP options top out around 1200Mbps, certainly. But I also just don't need that kind of speed inside my home. Streaming video needs at most 20Mbps or so, and I don't do much in the way of large file transfers. And when I do large file transfers, it's usually from or to the internet, and my home network is not the bottleneck there.
> It's wild to me that 10gbit isn't the norm by now
10G was too big of a step up from 1G. The expense and power required made it unattractive. Only recently have the interfaces for 10G over twisted pair become reasonably low power.
2.5G and even 5G are in a much better spot. It's where I recommend most people start as a default.
Yeah, 10GbE-over-copper switches are SO expensive. I bought a Ubiquiti enterprise switch for my home; 10Gb uplink and the copper ports are split between 2.5G and 1G. It's fine because only 1 or 2 clients can even talk 10G, and those are all across the house on Cat5 links anyway so they only negotiate to 2.5 even on a 10G port.
As much as I wanted to "future proof" by having a 24 port 10GbE switch... why? I'll just wait and buy one when I have a use for it.
> As much as I wanted to "future proof" by having a 24 port 10GbE switch... why? I'll just wait and buy one when I have a use for it.
As much as I enjoy looking at those wiring cabinets where every cable is cut to exactly the right length to reach a single port on the switch, this is why I prefer to leave an amount of slack in the wiring: It's good to be able to pull different wires to different switches depending on your needs.
One small high speed switch with enough ports for the couple of devices that can use it. One gigabit switch with a lot of ports to provide connectivity everywhere else.
This is what I ended up doing. A 2.5G switch for the few devices that can use it and a 1Gbit+PoE switch for all the other PoE and 1Gbit and less devices.
The new USM Beast looks interesting.
https://techspecs.ui.com/unifi/cloud-gateways/udm-beast?subc...
It does, actually. I dismissed it originally because I don't need most of the capacity, but considering I have no 10GbE copper ports available, and my home is being remodeled and I'm running some new ethernet to the other side of the house, I could take advantage of these ports now.
Also, I didn't realize my UDM Pro was kneecapped at 3-4Gbps IDS/IPS throughput. I think when I bought it I only had a 1 gig internet connection so it was overkill, but now I've got 10gig. It didn't even cross my mind that I had this bottleneck.
EDIT: Nevermind, 1500 american dollars??
expensive but not prohibitively so? https://www.aliexpress.us/item/3256807859212052.html https://www.servethehome.com/finally-a-cheap-8-port-10gbase-...
You can also get enterprise C3850 with 10G ports for ~100$, but they will be loud and power hungry.
5G barely exists and mostly is the same price as 10Gb. 2.6Gb is taking off and mostly replacing 1Gb, but try to find a 5Gb switch, there really arent any, and most appear to be 10Gb switches with 5Gb PHY in them.
>It's wild to me that 10gbit isn't the norm by now and tech people who should know better seem to think WiFi matches or even exceeds even 1gbit ethernet.
2 things here. Upthread is the discussion about the old 10GBE modules that would constantly turn off due to overheating. Thats left a sour taste in a lot of peoples mouths.
I dont have anything in my home network that matters enough to have 10GBE anywhere. If I did, I would just get fibre. My wireless is fine for most purposes except some HD streaming, and plain old 1 gig works fine there.
Getting Time Machine to work reliably over a network is painful, even the old Apple-made Airport with built in TM stopped working twice a year.
However, I have multiple Macs where I simply have a USB-C laptop SSD attached for Time Machine and they have worked without issue for years. These laptop SSDs come in huge sizes nowadays, and you don't need an especially performant one, so they can be pretty cheap.
Heh it's honestly wild to me anyone needs over a gig. My work has a one gig fiber line supporting hundreds of employees and usage generally remains below 10%.
The high expense of 10gig is, in part, because it isn't widely necessary and the people buying it are willing to pay extra.
> The high expense of 10gig is, in part, because it isn't widely necessary and the people buying it are willing to pay extra.
I think the price has more to do with where you live and how the market is structured than how necessary it is. In Japan where there is competition between ISPs, I pay about $40/mo for 10Gbps.
Routers have also come down in price to where they are pretty affordable for consumers. I use a Ubiquiti Cloud Gateway Fiber [1] which has three 10Gbs ports (two SPF+, one 10GbE) for $279. A TP-Link router [2] with an upstream 10GbE port and 2.5GbE LAN ports and Wi-Fi 7 is about $140. 2.5 GbE NICs have become cheap and ubiquitous and could commonly be found on $150 mini pcs (before memory and SSD prices went crazy).
Yeah it's more than more than most people need, but I definitely appreciate having the increased speed when downloading 50GB games, uploading 200GB files to YouTube, or backing up files to the cloud. I've probably never maxed out the full 10Gbps, but exceeding 1Gbps is pretty easy in relatively common use cases.
[1] https://store.ui.com/us/en/category/cloud-gateways-compact/c...
[2] https://amzn.asia/d/03EKpC8E
And if you don't need a router, you can get 10gbit ports for much cheaper than that. The Mikrotik CRS305-1G-4S gives you one copper 1gbit port, four 10gbit-capable SFP+ cages, and is ~150USD [0]. Their whole lineup of switches with SFP+ ports can seen here [1].
[0] <https://mikrotik.com/product/crs305_1g_4s_in>
[1] <https://mikrotik.com/products/group/switches?f[0]=s%3Ac&f[1]...>
I put 5Gbit internet into my home (fiber) to build my startup. I'm processing terabytes of data. I have over 100TB of storage in my basement. I can regularly saturate my internet connection. That said, I remember well when a 1Gbit connection provided enough bandwidth for a 500-person call center for daily workloads (back about 10 years ago).
That puts you in an extreme minority, even amongst enterprise businesses. Many medium sized enterprises have storage that looks like "a couple dozen TB total" for hundreds of staff.
Having 100 TB of storage in your home basement is an even more extreme minority than that. ;)
A gigabit connection is more than enough for a 500-person call center today.
Much works on 10/100 if you wanna know the truth about it - but it is really nice to hit full speeds when copying terabytes around.
Agreed. We had IP phones with 100 Mbps switches in between most of our computers and the rest of the network for a long time and very few people noticed. It'd only really be when I was installing a system upgrade or something, and I'd be like "man, it'd be nice if this didn't take an extra two minutes". For normal web access, 100 Mbps and 1000 Gbps aren't really discernable, until you're downloading large files. A lot of 4K streaming videos though, you'll start to feel it quite a bit faster.
And then hilariously, once you go above a gig, the reality is most sites won't serve them to you any faster than that anyways.
I found the nicest thing about fiber is I can hit over a gb/s uploading, which is often much more critical-path for whatever I’m doing than a download.
If that 500-person call center is a BPO, they'd have trouble with today's data-heavy workloads. I do agree, though. I'm in an extreme minority when it comes to this much bandwidth in my home.
Off topic (sorry). Interested to know what your startup is?
I appreciate the interest, but it's not public yet.
Depends a lot on your work type and your prior exposure. If you only work "locally" and upload/download rarely, you may be way less demanding of your network than if you actually do distributed work with remote storage, high-bandwidth communicating tasks, etc.
Over 20 years ago, I was used to having 1g LAN for basic workstations and laptops in an office setting and probably 10-20g uplink from the building (shared by hundreds of staff). I also used 1g at home for my very small LAN between laptop, desktop, and SAN functions. But, my home ISP links were often terrible, such as 128k ADSL or even just a tethered GPRS phone at some points.
You end up with entirely different work styles when you have these different resource constraints.
1Gb Internet service seems low these days, much less 1Gb LAN. I have 3Gb Google Fiber service and actually get 2+ for individual downloads from some internet services like Steam. Even at 2Gb it's annoying to wait tens of minutes for 100+ GiB games to download. If I go on vacation I come home with 10s of GiBs of photos and videos on multiple devices that start syncing with cloud storage.
During the day I need to pull large data files from the work VPN so it's nice that that can happen at full speed even when Steam and movie streaming are also at full throttle. Combine that with backups and moving various files back and forth to my NAS and I'm very happy to have 10Gb local wiring.
This is one of those things where I just have to express that a lot of the HN crowd is entirely divorced from the reality the rest of the world experiences. ;)
Nearly nobody has multigig anything in the home, a probably surprisingly large percentage of business networking is 1gig LAN or less. And most people would not notice the difference if they did.
I am glad it works for you, but everyone else most certainly doesn't need it. (Yet.)
Personally, I do try for mostly gigabit in my home, because I do selfhost, but I have a ~800 Mbps download service (200 Mbps upload, it's asymmetric) that was only 500 Mbps when I signed up. And to be honest most of my patch cables are CAT5e because I'm cheap. I do make sure to run CAT6 through walls though because I don't want to ever have to do it again.
Also, I used to have Astound, and I feel so much sympathy for Google Fiber customers, you have no idea what's coming. If you thought Google had a reputation for bad customer service... just wait!
I disagree. I pay $120 a month for 5Gbps symmetric connection. I could upgrade that to 10Gbps for 2x, but there is no reason at this point. Even the local max from the cable company is more than 1Gbps, 1.2Gbps down / ~300Mbps up for around $80. Everything is streaming now. I work from home, on video calls. My better half will be watching something the on AppleTV streaming, the kid will be doing the same. I have Backblaze running to do backups to the cloud. 3 different laptops that will run TimeMachine backups to the NAS. The AppleTVs also have the Infuse app on them to stream local video files from the NAS. The security cameras are a constant 60Mpbs 24/7/365 to the NVR. The laptops can push a gig wireless and 2.5Gbps when plugged into the Thunderbolt docks. It is not clear that I need 10Gbps everywhere, but it has its uses. The NAS is at 10G. The link from the main switch to the router is 10G. The 3 APs in the house are at 2.5G and the 2 outside are at 1G. There was a noticeable difference when after I right sized the shared links paths up from 1G. When I say noticeable it both perceived and measured. I used to work doing switch bring up and competitive testing so I have a pretty good idea how this all comes together. Given that a reasonably cheap set of APs can now handle clients at above 1G and internet speeds in some areas being above 1G, moving to at least 2.5G in places is useful and not divorced from reality. I am in tech, but I have help my not tech friends upgrade APs, et.al. for their normal everyday home use cases and they have all been quite happy with the change.
Not being divorced from reality is the only reason I have not dropped $5K on the new Dream Machine Beast that was just released and have not swapped out my Enterprise 48 PoE (1st gen.) for the newest version that has 12 10G-BaseT ports.
I have it more for the fast nas access and being able to treat nas disks as more or less the same performance as if they were directly sata in my machine. Significantly less so about the external network aspect.
It doesn’t have to be expensive.
You can hook up a few key components cheaply and 10gb pci cards are something like $20.
> It was much more expensive than I thought it should be
For 10G with copper, short runs will work fine on most anything (10m), with longer working ok on cat-5e cable (20 - 30m or so).
So, if you have existing wiring, especially in wall, just give it a try first, before gutting things. Most 10g ethernet adapters give you BER stats to see the error rate.
And, if there are issues, multi-gig may be an option, where it will drop down to 5 or 2.5 gbps, which is same as 10G, but with a reduced symbol table to handle the lower SNR.
You can definitely get stable 1gig throughput through WiFi. Doesn’t even need WiFi 7/6E - possible on 6. Ran a WiFi bridge like that for years and - no packet loss, consistent gig through and maybe +1ms latency
The gotcha is both ends needs to be good radios. So a router to router bridge tends to work better than router to end user device. Also had near LoS which presumably helped a ton
[dead]
I went a slightly different route. My switches are linked with 10Gbps SFP+ across the apartment, but it was way too late (and too much hassle) to pull proper in-wall fiber. Instead I used one of those ultra-thin (around 0.1mm), unshielded fiber cables, and just snaked it through door frames and taped it along the walls. I'm genuinely impressed this stuff exists, it makes retrofitting fiber into a finished space so much less painful.
Most of my edge devices are still on 2.5GbE though, and I'm increasingly aware that for anything with plain SATA disks, the drives are the real bottleneck. Once I LAG'd 2×2.5GbE to get a 5Gbps pipe, it became obvious the network wasn't the slow part anymore in a lot of cases.
And yeah, the 10GbE SFP+ modules run hot, so hot that I would not lay my fingers on them for more than 2 seconds. I stuck 2 copper heatsinks on my module, not sure they do much but the module runs smoothly. Even so, I'm pretty happy with the overall setup: from my 10GbE-equipped Mac I can saturate multiple machines at once and I no longer think about the network most of the time, which was the goal.
And after getting 10Gbps working at home I was getting greedy, and looked at InfiniBand as well, 40Gbps and proper RDMA is very tempting compared to Ethernet. The catch for me was the practical side: IB needs PCIe slots and those chunky, inflexible cables. With most of my stuff being laptops, mini PCs and Macs, I just couldn’t see a clean way to route those or even plug cards in everywhere, so in the end the "door-frame‑friendly" skinny SFP+ fiber still won out for this apartment.
What kind of fiber cables? If you have a link, I'd appreciate it. I can only find ones at 0.5mm.
Good catch — that was actually my mistake. I mixed up cm and mm from the marketing material, and when I actually measured just now it came out to around 0.9mm. So not quite as thin as I claimed, and I probably wouldn't have noticed if you hadn't asked.
I did find some Kevlar-reinforced options that are supposedly ~0.3mm, but they seem to be raw fiber without connectors, purposed for drones, and I'm not sure about global availability.
Ah, I see. Thanks for checking. I'm considering doing a similar run at home, so was curious if there were any options that thin. 0.9mm should be plenty to work with. :)
> The most important question was the structured cabling in the walls; was it CAT-5E or CAT-6, or even CAT-6A? Remember from the last post, 10GBASE-T might work over short runs of -5E (even though officially it's not meant to be able to).
This is not quite correct.
The primary problem is cross-talk. Copper wire itself will carry the relevant frequencies up to 100m without issue but even with balanced pairs the balancing is not perfect and the "dirty paper precoding" is not perfect so some cross-talk will occur. How long you can go with Cat-5e depends on how well the wire is twisted, how many wires are bundled together, are there any loops or tight bends, and other factors. Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out.
Bottom line is: for almost any normal home or apartment any quality Cat-5e cable properly terminated will carry 10GBase-T without issue. In fact if you have problems I would first re-terminate the cable before assuming you need to run new cable. Cat-6 or 6A just isn't necessary.
As a PSA: beware of "CCA". I've noticed Amazon and eBay are absolutely flooded with cheap chinese electrical and networking cable that shows nice shiny copper in the pictures but is actually "copper clad aluminum". If they mention anything at all they code it as "CCA" cable without explaining what that means.
CCA cable cannot, by definition, be ethernet cable. I won't get into the full technical details but the standard was amended to clarify that only pure copper wires are acceptable for ethernet. Personally I would not dare use CCA for anything. It has lower performance, lower current-carrying capability for the same wire diameter (inherent in aluminum), and introduces the risk of oxidation and loosening of connections as people will treat them as copper connections when aluminum needs special installation procedures and connections to avoid them coming loose over time. For electrical connections especially this not only can but absolutely will lead to a fire over time if not treated with the appropriate care. All it takes is a little bit of mechanical action scraping off the thin copper layer and you now have an effectively aluminum wire - a time bomb ticking away.
> Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out.
This isn't quite right. Cat6A guarantees the frequency and therefore effective bandwidth that is available. But it doesn't talk about physical cable type. A common structured cable for Cat6A is U/FTP which has no plastic separator, no outer shield, but rather each pair is wrapped in foil and is actually less tightly twisted than even Cat5e. Anyone can buy U/FTP cable, but to actually be Cat6A it has to be installed correctly and the installation certified to support Cat6A speeds.
The only thing I'd caution anyone else looking to do the same is doing a software router/fw like the Portectli is it's usually not hard to get the raw bandwidth to look nice with big flows but the new connection latency, connections per second, jitter, and QoS handling tend to suffer vs something with hw offloads (which is what most are used to even with cheapo gigabit home AP+router+switch combos). It's also not usually the cheapest way to get the 10G class NAT/L4 FW bandwidth, but it is usually the cheapest way to get "full" FW functionality if you don't care as much about the performance.
If you want a full FW solution that can actually FW+NAT at 10G bidirectional without breaking a sweat then something like the FortiGate 90G is the cheapest thing I've found that performs really well across the board. Great QoS, great latency, amazing throughput performance (does well with even small packet sizes in a single stream), easy enough to use UI (once you get oriented), low power. If you want to enable all of the NGFW stuff (e.g. AV and IPS) then it'll dip below line rate though.
If you just want something that NATs/connection direction oriented filtering like a "normal" home router then something like the MikroTik CCR2004 can get you better than the performance they got on the VP2440 + give you 12 ports of 10G SFP+ to work with. If you were planning to do "fancy" FWing/functionality beyond a normal home NAT FW (with decent managed switching built in) then the feature set will be a bit limiting, of course.
I don't understand the decisions to use RJ45 copper cable for anything higher than 1GbE, unless
I upgraded my home to full 10Gbps network at 2018. At that time: Those 10KM SM modules will happily run off <20m MM fibers so I just used whatever fiber I have at hand. The price is so cheap that I think the RJ45 upgrade path (2.5G has not been universal, 5G still very far away, 10G is so pricey) looks like a scam.After 2024:
I already have an experimental 6*25G CWDM fiber connection between my 2 rooms. Given that most of my devices wouldn't need anything higher than 10Gbps, I'm thinking of giving each room a single 100G port, split them into 4 lanes in the room with passive wavelength splitters for different devices, which further reduces cost on switches.Power usage is a good reason to stick with non-SFP ports.
And you can use CAT5e with 10Gbps networks given the run is short enough. I run 2.5Gbps over CAT5e with no issues throughout my house (I don't need that much bandwidth except when downloading Steam games).
10Gbps fibre uses significantly less power than 10Gbps copper ethernet.
I read that parent comment as referring to POE, but am doubting myself after reading yours.
POE is a great reason for rj45.
I might have been lucky, but in the one home and one office were I've connected 10gbit switches and PCIe cards, it has just worked. Especially the office was a nice surprise, because it is at least 20 meters (probably more) of unknown cabling and at least one unknown patch panel between the utility closet where the NAS lives and the desk area. The cables were run 15 years ago, so I expected it to be cat 5, but clearly not.
It is nice moving/streaming large files across the network at 10 gbit. It really is ten times less waiting than with plain old gigabit.
Of course, most of the time I'm working with lots of small files and then the spinning disk array in the NAS has no chance to saturated the this giant pipe, or even a normal gigabit connection...
> The cables were run 15 years ago, so I expected it to be cat 5
FWIW, Cat 5e supplanted Cat 5 25 years ago.
I know when I was doing some custom wiring in a house around 2005-6, it was clear that cat-6e was the thing to use if you wanted any future-proofing.
So I bought a reel of that even though I was only going to be using 1000-BaseT. I don't remember there being too much premium on the wire itself.
Do you mean Cat 6 or Cat 5e? Cat 6e isn't a thing and 6a didn't exist in 2006. 6 was certainly more future-proof at the time, although arguably 5e is still fine even today. (Super-gigabit consumer equipment didn't really exist until the last five years and it's till notably more expensive and less common than gigabit, which runs on 5e just fine.)
Cat 6 for sure. I thought there was some extra tweak beyond that, but I probably misremembered after all this time. Perhaps it was just that it was plenum-rated..
You might be thinking of cat6a which officially supports 10G over long lengths, unlike cat6.
Nah, it was definitely around 2006 which seems to be before cat 6a was ratified. So I couldn't have bought a reel marked that way...
> The cables were run 15 years ago, so I expected it to be cat 5, but clearly not.
Did you check the jackets? I've got Cat5 (no e) marked cable running 10gbaseT. A lot of cable exceeds the specs on its jacket and specified wire provides enough signal to noise for the provided length in dense conduit. When you have shorter runs, without dense wiring, lesser cable can work.
I can't see the cable jackets. There's just outlets in the walls. I ask building management to make a link between outlet X and Y and then they make it happen via one or more patch panels that I don't know where they are. :-)
People downthread has written, cat6 was already the goto thing 15+ years ago so perhaps that is what they put in when the office space was built. It is a well built office!
Same. I ran CAT6 from one end of our house to the other, because my home office is in the opposite corner as the fiber coming in to the router. After running that, and manually crimping everything, it Just Worked from the first time I turned everything on. That felt pretty good.
> It really is ten times less waiting than with plain old gigabit.
Man I wish I had the kind of money/hardware to get 10gbit disk io out of a nas.
€200 buys you a 1tb ssd even at todays elevated prices. That has no problem giving you 10gbit of sequential io.
Four or five 7200rpm disks in an array can also sustain it for sequential reads of big files. That costs less than a gaming laptop.
A potato can saturate a 10gbit line if the data is mostly in memory.
For lots of small files in random sequence and cold caches it gets a lot harder.
That's pretty wild.
I have 1.5/900 fibre to my house, and I bring a 2.5 line from the modem to my home office where a 2.5 switch delivers it to my workstation, laptop, and unraid NAS. But those devices are all themselves just gigE I think, and I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available.
10Gbps is enough bandwidth for 500 concurrent Netflix streams in 4K/UHD (15Mbps) AND 500 concurrent video calls (4Mbps).
Home users don’t need more bandwidth to improve their internet experiences, they need lower latency, less congestion and less loss.
https://help.netflix.com/en/node/306
https://learn.microsoft.com/en-us/microsoftteams/prepare-net...
Home users only do video calls and watch Netflix?
More and more regular people are getting network storage appliances. More and more people have laptops with SSDs that can write at 4 or 5 GB/s. Why shouldn't they get to use all of it?
To quote the previous post:
> I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available.
There are many things along the way that would get in the way of a home user downloading something from the internet that would hit that 5GB/s speed. It's not that people should be "banned" from it or something, more that the investment cost isn't worth it.
I regularly saturate my 1G home and 1G office connection syncing ~6GB files between the two. It's also nice to be able to download a 100G or so game quickly. Remote backups to cloud storage also benefit from fast upload speeds (and more importantly, restores).
We have a 5gbps pipe; routinely download games from Steam at >3gbps; when I had to reinitialize my cloud backup it was >4gbps. All of this without impacting anyone else on the pipe.
Yah, our P95 bandwidth is just a few megabits per second. But it's not that expensive and routinely saves me a few minutes here and there.
10gbps on the LAN is more broadly useful. Pegging it for a file share is a daily occurrence.
Also storage has gotten super expensive lately, and rather than upgrading my machines/consoles I've been offloading games and downloading them as needed and now am routinely downloading dozens of GB just to play a game.
My gaming time is limited so the faster the better.
I should have said most home users. My point is that more bandwidth at this point probably won’t affect 99.999% of home users.
What’s described in the post is the tech equivalent of supe-ing up a sports car and then driving it in rush hour traffic. It’s fun to geek out doing it, but practically in everyday use the difference will be negligible. Even with large file uploads and downloads, there’s a good chance that services won’t reach those throughputs end to end.
What’s telling is that the post shows screenshots and charts from artificial speed tests. No videos of the Dropbox client chugging away with throttled uploads.
> I should have said most home users. My point is that more bandwidth at this point probably won’t affect 99.999% of home users.
640k should be enough for everybody... DSL should be enough for everybody...
If you build it, they will come.
> SSDs that can write at 4 or 5 GB/s
In my experience unless you actually pay attention and get something with a dram cache it will sustain that speed for all of 5 seconds and then drop to near useless, and with the current dram shortages that is getting harder and harder to justify.
I just did a build out where ram cost as much as the GPU and both are individually 3x more expensive than the CPU and MB combined.
A decent 2TB nvme drive was also more expensive than the CPU and MB combined.
A 10GB network connection can happily transfer at that speed all day, your SSD is unlikely to maintain Sata speeds unless you actually shelled out for something decent, what came from the manufacturer in the laptop is not that.
How much $ extra are you willing to pay for the extremely occasional transfer at rates higher than gigabit? 2x? 3x?
Those ssds are very likely cached and so cannot keep that pace for more than a quick burst of a few gigs.
> Netflix streams in 4K/UHD (15Mbps)
proper 4k, like the one you watch from a blu-ray, will have peaks of 150mbps.
the 4k we see on streaming services is awfully overcompressed.
and yeah, you can see the difference, it's day and night.
4k 15mbps is arse. Try 100mbps and you’re getting close to decent quality.
I regularly hit 4.7 gbps on a 5 Gbps line pulling files (usenet is usually faster than torrent, but the latter can be equally as fast depending on the torrent & how good the client software is). It's great to just grab an entire movie series in 4k Blu Ray remux quality in 5 minutes and go. No real need to plan ahead for anything.
Yeah, Usenet is crazy quick. I've saturated 10Gbps on Usenet before, pretty easily.
Torrents (as you mention) can be equally as fast. Well seeded, private tracker torrents can easily download at above 1Gbps (depending on piece size, etc).
Can’t get much over a gigabit here I’d nz.
And here I am sitting in Brooklyn and haven't had one apartment that has had fiber as an option. I get to pay Spectrum $90/month for "400/20" and in reality get 100/10.
I was paying 21.9€ per month to TIM for a 10/2Gbit connection
Welcome to Italy :D
But Brooklyn is an old city that's hard to wire, unlike those new Italian cities... /s
Yeah I foolishly paid for 500 Mb/s but the only things I ever get over even 200 Mb/s for in the UK are Steam downloads (and speed tests). Everything else seems to be roughly throttled at that rate.
Telus gave me a good price and I'd already invested in a Unifi gateway that could accept the ONT directly, so that was fun to play with.
How else would you connect? Some ISP box?
Sorry, I’m not familiar with any other way.
Yeah the default is they supply a "modem" with a regular ethernet port, so a familiar config to someone coming from cable or DSL, but inside it's a QSFP that you can move to your own equipment if you want.
The Mikrotik switch is awesome, and it's still the most compact 10G switch available.
You can fix the thermal issue either by adding a small fan (Noctua is great) or by adding more radiators: https://pics.ealex.net/share/UxeSf_AWHLIuc-qzK5zl7JIgQvQDAZh...
I've been running it like this in a closed comm box for the last 3 years without any issues. SFP+ modules actually do not use that much power, it's just that it's concentrated into a small package, resulting in high temps.
Cripes. When possible just do fiber and DACs. Faster and much cooler than 10Gbit. 10Gbit uses an absurd amount of power per port thus the need for all that cooling.
yeah, the article though says...
"The apartment has structured cabling -- each room has one or more RJ45 sockets in the wall," ...
Which is the main problem most folks face.
wish the standard was "conduit" instead of "bake-this-years-tech-into-the-wall" which doesn't always last...
Folks have been saying that conduit is the way, and the fiber is the future, and all kinds of things like that for decades so far.
But the simple truth for all those decades is this: When there's already cat-whatever cable in the wall, it generally still works.
Decently-installed conduit (ie, actually-usable conduit) adds a ton of time and expense, which is why it is very seldom used for data circuits in residential structures.
The cable that exists is a lot better than the conduit that doesn't. And copper ethernet is bog-standard like MP3 is: It isn't the best in any technical sense at all, but everything supports it. Universal compatibility is pretty nice.
---
So the ongoing cost of copper 10gbe is electricity. Someone else here in the comments says that a copper 10GBe SFP+ module can use ~3 Watts, or that a newer one can use about 1.5 Watts.
We can be generous by using the larger figure of 3 Watts, and 8 devices..
With 4 ports, eight 10-gig endpoints @ 3 Watts each, and $0.19 per kWh [delivered]: That's $3.28 per month, or about $400 per decade.
If we assume 1.5 Watt endpoints, then that number halves.
If we subtract the power consumption of fiber SFP+ modules (or media converters or whatever) to make the number a relative comparison instead of an absolute, then that figure goes down further.
Not so bad, compared to conduit.
houses I lived in never had cat-x wiring. They had electrical cable and sometimes phone lines.
but maybe statistically speaking cat-x is more prevalent now in housing built after maybe 2000.
Most folks don't care about network cabling these days. For many/most common consumer uses, wifi (with mesh nodes or whatevr) is quite good for streaming video and playing online games.
These people aren't seeing the whole picture, but that's not new: They simply don't care, just as most folks have never cared.
And they care less and less as our (rather amazing) portable pocket supercomputers continue to supplant desktop computing in normal Internet-connected life.
Some folks are even actively against wired home networks as a premise: When I'd hang out on Reddit, sometimes I'd see photos of well-orchestrated and fairly intense home network infrastructure, but the new buyer of the home just wants it all gone from their sight and is seeking advise about getting that accomplished.
---
The house I live in right now has zero phone jacks (though the overhead wires still connect to the house before disappearing completely), and it only has 1 run of functional coax (which connects to the docsis modem).
I've added some decently-installed Cat 5e, which is completely adequate for what I'm doing. (That's been a trend for me for decades: When I live in a place, I tend to hardwire my stuff.)
But the next person probably won't care about that at all. If I move out and come back to visit in 5 years, the cabling I've added will probably have been erased.
And in the next place I live, I'll probably have to add more wire again. That's normal to me; again, I've been doing that for decades now.
---
And that's OK. I'm a bit of a geek when it comes to computers and networking.
Most people are not that way at all, and most people never have been that way. Most people don't give a fuck about this stuff at all.
And this is all completely acceptable.
(I guess I could choose to hate it instead, but life is too short to hate this kind of thing. Acceptance is both simpler and better-productive.)
DACs are terrible, they are unwieldy and are always either too short or too long.
Fiber is much better, but in my case the house already had Cat5 Ethernet wiring (originally used just for phones!) everywhere.
Another use case for 10G Ethernet is PoE for the WiFi access points. Although you can't use SFP+ modules for that, of course.
Just needs a guitar shooting fire and it’s a Mad Max prop.
I put cat6a in my last house and plan to in my current house "just in case". The cable isn't that much more expensive and nobody wants to do cabling again so why not.
But I'll keep using a gigabit switch because I have absolutely no idea what I'd use 10G for. It's crazy that gigabit was affordable for me as a student in the early 00s and between then and now we've gone from DVDs to 4K and it's still plenty fast enough. In fact, most people are happy with WiFi (not me, though).
Yep... Did this 3 years ago in my home and came to the same conclusion. If I were to do it again, I'd run fiber through the walls instead of Cat-6a. It took forever to find SFP+ modules that would work with my Unifi setup... (not wanting to pay for more than one router or switch with native 10g RJ-45 ports, which are still very expensive.) I loose POE but, on the whole, it would've been much easier and much less costly--I think--to have just run fiber.
Same.
I wish I had seen this before pulling all that Cat-6a.
Using SFP+ modules for 10GbE is interesting, I deployed a full 10GbE network in my home connected by 5Gbit/s symmetric fiber from AT&T and a 1Gbit/s x 100mbit/s DOCSIS 3.1 connection from Spectrum as backup, I did this around August of last year and used UniFi/Ubiquiti for everything. Their new XG and XGS products all support 10GbE natively, without needing SFP+ modules to connect. I'm using the UDM Pro Max with an XGSPON ONU flashed with the community firmware from PON Wiki (see https://pon.wiki/category/att/ ) and the 2.5GbE port as my second WAN link to the Spectrum modem. I'm not doing any sort of bonding/load balancing, just failover so I'm almost always just using AT&T. There's a short DAC cable connecting the UDM Pro Max to a Switch Pro XG 24-POE which serves 10GbE to the rest of the house. The only SFP+ module involved is the XGSPON ONU. Most of the devices in the house connect to WiFi served by the UniFi U7 Pro XGS APs, which take 10GbE w/ POE and give off WiFi 7.
I pretty much always get at least 4600Mbit/s both directions over AT&T, and generally cap it out. Spectrum typically gives me at least 800Mbit/s down. The ISPs are definitely the bottleneck because neither provides dedicated infrastructure from each house to the CO, instead you have some sort of aggregation point which has a shared backhaul and that means you compete for resources, but having the largest plan on each at least gives you traffic priority.
Realistically, I don't need any of this, I was doing just fine with normal gigabit fiber with a 2.5GbE network before I moved last year, but it's nice knowing everything is as fast as possible in my path and that eliminates congestion and local network resource contentions as causes when a problem arises, I know it's pretty much always upstream of me somewhere.
Regarding the proxmox cluster, I would advise not to set up a cluster unless you really need to, it adds complexity in order of shutting down the server, HA, Quorum votes, ceph monitor, among many others, and if one server goes offline, or even the wrong order, it will impact the others and sometimes some tough time in recovery, or data loss.
I was just reading this thread today: https://www.reddit.com/r/homelab/comments/1sz9li4/invisible_...
If you’re going to go to all this trouble, please do seriously consider pulling singlemode fibre instead. It is more planning up front, but when you’re ready for your next upgrade cycle, you can reuse the same fibre and change out the hardware on each end.
Why recommend single mode out of interest? I've just done some fibre in my house with OM3 because that's what I had, I couldn't really work out why you'd prefer one over the other for residential lans.
Multimode is probably fine, provided it's OM3 or higher. You might need to think about it down the road, 5+y out. Having a spool just lying around would make it a perfectly reasonable choice. The fibre itself is a little more expensive; the optics are cheaper. But multimode does run into physical limitations a little faster, so when you decide you want to go to 100G, or 400G, or beyond, you may need to pull fresh media.
Singlemode you just never have to think about again. It's likely to be more future-proof than your actual physical walls.
They are much smaller to post though holes etc too.
I was surprised that the old Cat5e in my home supports 10Gbps without any issue, so went ahead and upgraded the rest of the network with 10Gbps switches (expensive Ubiquiti gear, but worth it to talk at 10Gbps between all my machines, even though the internet is only 5Gbps Fiber).
Yeah, my home is cat5e (I think!) and I was only able to get 2.5 or 5Gbps to my _single_ 10GbE capable machine I had at the time.
So I ended up just buying a switch that supports 2.5Gbps over copper since it's such a huge expense to step up. Anyway, my devices are pretty much 2.5Gbps (APs, flex mini switches, etc).
That Mikrotik switch must be heating up like crazy if you fill all 4 ports, since it's using copper .
How about this one? https://mikrotik.com/product/crs305_1g_4s_in
You can use fiber optic SFP+ in it which should not run as hot.
Run fiber everywhere for the fun of it, instead of copper Ethernet. Here is an interesting post about it:
https://sschueller.github.io/posts/wiring-a-home-with-fiber/
You can simplify it and avoid splicing which is the most complex thing about it.
One thing I'll add that I learnt in the process of doing my own house: It's not just the cable type but also the SFP module that can limit the distance. I used MicroTik hardware and their S+R10J modules are limited to ~30m for 10Gbps speeds.
10g-base-T is a sick joke
high latency, high error rates, and terrifying heat output from SFPs (which the author noted for himself)
the only cat6 left in my home network is the link to verizon's ont, because in their infinite wisdom the ONLY connectivity offered was 10g-base-t
10GBase-T can still be nice if you're going to do native PoE (which the author did not) or you expect a mix of devices with 1/2.5/5 which you don't want to or can't upgrade/replace immediately (which is where it sounds like the author was situated).
The new 10GBASE-T SFPs are actually not too bad - you can get the full 100 meters at half the wattage it takes for the old space heater generation ones to reach 30 meters. Based on the article, the author did not know there were newer cooler options for the ~the same price.
Has anyone tried the InvisiLight kits? They give you a 600 micron fiber optical link in the kit which you can hide down the side of carpet etc. I haven't tried it but a few YT vids made it look interesting.
https://www.amazon.com/stores/InvisiLight/page/DFF3C042-0D2E...
That’s pretty cool - I hadn’t seen it.
This is the approach I used and tried to make a product from. I’m not pursuing that anymore but sharing for folks who want to run wire without dealing with drywall.
https://trywireshark.com
(yes I’m familiar with the other well established Wireshark)
Looks like a complicated and expensive way to avoid running copper. "Up to 1Gbps" is not confidence inspiring, if you really can't put holes in your walls I'd try a powerline kit first for 1/5th the price
I used powerline before. From my experience its really bad, unstable, slow.
What did work pretty well though is MoCa, basically reusing coax/antenna cabling. I get a pretty stable connection roughly equivalent to 2.5G.
It depends on the topology of your circuits. If you can get both adapters on the same circuit that is ideal, same phase is often okay at reduced speed. Different phases or having other complicated stuff in the way like subpanels is probably what you were dealing with.
MoCa is fine, but if the coax was retrofit replacing it with proper ethernet cable isn't too hard.
This is the YT video I saw about it: https://www.youtube.com/watch?v=Z2FbzCyiNr4
He used different transceivers and got 10Gbs
That is an ad, I'm not surprised he had good results.
I'm surprised they got a reliable 10Gbps USB adapter. I've tried 3 of them and they're all trash. They cannot sustain multi-gigabit speeds and end up doing like 400Mbps when you hit them hard. I think I have sabrent and ugreen and one other.
They all just sit in a drawer, if I need a USB-C ethernet adapter i just grab my trusty 1 gig one.
It's interesting how slow the uptake has been. 10GbE is from 2002, 24 years old. It's much worse than IPv6, at least we're up to 50% uptake there.
100 GbE is 15 years old too depending on which standard you think of. (The most common physical layer, SR4, is 11 yrs old)
Gigabit ethernet came much faster after 100 M.
I can install 1Gbps using copper cables + RJ45 and household tools. I can't install 10Gbps using a $10 crimper.
Therefore, I have 1Gbps all through my house and 10Gbps between equipment next to the access point.
You can’t crimp cat 6?
Crimping implies RJ45, so 1Gbps. You can't crimp SFP+.
Even CAT5e can do 10 Gbps if you do a decent job making the cable, even though it's out of spec. 6A can do it easily, in spec. I used it in my previous home.
I much prefer to use fiber, but copper with good old RJ45 works fine for 10 gig.
My 10G experience is documented here, mine was TP-Link-based: https://ben3d.ca/blog/home-network-lessons
The device nomenclature alone is worth the read! (otherwise an impressive feat to see)
For me the threshold has always been "can I stream a 4K movie from the NAS downstairs or from my seed box". No real need for anything above. Still I ran 10Gb single mode fiber in all the ducts.
> For me the threshold has always been "can I stream a 4K movie from the NAS downstairs or from my seed box".
So basically, and this is not a snark, you're happy with 100 Mbit/s?
Both impressive and surprising that thermals were the biggest barrier!
Meanwhile I'm sat here wishing I could justify running any ethernet in my apartment, but improving wi-fi tech means I never can...
I wonder what the idle vs at load difference is for power draw/heat. Would really love to see this feature in reviews some!
I wonder if you could negotiate down to 1gbit until you see some level of activity, if that would help at all?
I'm still eyeing 10Gb, but if my home needs +30w for three computers, I don't feel like it's really worth it. Would love to see more details on the power consumption from folks, especially tuning for idle.
What’s the use case for 10Gb/s Ethernet over 2.5?
It's 4x faster? If you have a storage server, its easy to exceed 2.5 or even 5 gigabit with SATA spinning disks if they're in an array. 10 gig is fast enough to work with network storage like it's local: do your work, store your steam library, edit videos, etc, but store it all somewhere else and share it with multiple computers.
10Gb Ethernet is incredible for this exact reason but optical SFP+ modules are far more affordable and reliable.
I wired my home with single mode fiber. It works like a charm and I can always upgrade the speed.
What's the point?
for the funs