I have attended some of the webinars and listen to Allan on a podcast regularly and this LLM-written post is painful to read. If I hadn't already been familiar with him and his work I would have not given much weight to what was written. He is a big proponent of tuning ZFS which requires knowing your workload for which he has written other guides.
1. From my own experience, most people who use a ZIL for faster sync writes are wasting space. You only need five seconds of peak throughput (for a 10Gb connection that's ~5 GB). Create a pair of 25 GB partitions for the ZIL rather than dedicating entire devices.
2. With the remaining space set up a mirrored Special VDEV with heavy use of small blocks. 32k block sizes will still give some room for compression which, other than saving space, can increase throughput and reduce IO/ps.
3. Pause scrubs (`zpool scrub -p`) when the workload is either expected to be heavy or is actually heavy based on moving averages and resume when its safe. Even my 24 TB NAS takes 24+ hours with no other activity.
> 1. From my own experience, most people who use a ZIL for faster sync writes are wasting space. You only need five seconds of peak throughput (for a 10Gb connection that's ~5 GB). Create a pair of 25 GB partitions for the ZIL rather than dedicating entire devices.
You might benefit from the same technique for a "special" vdev... If you run an all spinners array, but the metadata is on a small partition, you might get better seek times during metadata scans, since everything will be closer together? And/or if you use raidz for your regular vdev, but mirror for the metadata, you'll have more capacity for metadata reads.
Yeah, the only place I use SLOG is with Optane devices in front of SATA/SAS flash. Anything that needs the speed of SLOG shouldn't be on rust anyways.
I would be in a different world if my system was backing banking or other loss critical data. For all my workloads, losing 5 seconds of writes in a crash is NBD.
> Sizing is measurable rather than guesswork. zdb -bb tank breaks down allocated space by block type; sum the non-data categories and you have your current metadata footprint.
Of course, this is also not a fast command on a slow array; my half full 2x 24TB mirror is estimating ~ 12 hours to run this; the last scrub took just over 24 hours to run. (After I posted, the remaining time dropped to 10 minutes... not as slow as it seemed, but still slow, and using a lot of I/O ... left for lunch and 50 minutes later it's almost done; estimation is hard)
Isn't that kind of the point of the article? The random reads required for metadata traversal during scrubs will kill performance for sequential reads and writes as well if you don't have the IOPS to back them up, building a pool without taking this into account will make regularly scheduled scrubs a pain point.
I schedule scrubs during idle time (10pm Friday - 6am Monday), so the metadata hit doesn't affect usage, but YMMV if you don't have an idle time. I also don't generally let them get that full.
If I was building the array new, I'd probably use a special VDEV for metadata, but they didn't exist when I built my last one of these.
ZFS scrub performance shouldn’t come at the expense of production worklaods. If scrubs cause latency or downtime, your pool lacks the IOPs and metadata performance needed for healthy maintanance.
This could have been a decent article if it hadn't been run through the AI slop machine. If I wanted to read writing this bad I would have just asked ChstGPT myself.
It's so hard to wade through run on sentences like "The knobs are not the problem. The pool was designed without room for this workload."
Oh sure, just buy more disks. Who do you think I am, that guy on the Monopoly box?
I stocked up on a few shuckable externals before the start of the second Grump catastrophe. $280/24TB (Exos inside). I'm finally getting around to setting up a pool, thinking 4+1 is kind of poor redundancy and wondering about adding another disk. The things are up to over $1,000 each, in the husk or not. Fucking hell. It makes me want to drastically scale back the size of my new pool in case a drive does die so I don't end up needing to buy a replacement any time soon.
I had a disk in my ZFS pool (a relatively tiny one, among the kind of people who have a home ZFS RAID5 array) go bad earlier this year and bought a used disk for the first time ever. Never even considered it before.
Still cost me as much as a pair of new drives the same size that I bought to transition from mirrored pairs to a four-disk RAID5 array about 16 months earlier. Like, god damn it was expensive.
A new disk would have cost me almost as much as all four disks did, originally.
I have attended some of the webinars and listen to Allan on a podcast regularly and this LLM-written post is painful to read. If I hadn't already been familiar with him and his work I would have not given much weight to what was written. He is a big proponent of tuning ZFS which requires knowing your workload for which he has written other guides.
1. From my own experience, most people who use a ZIL for faster sync writes are wasting space. You only need five seconds of peak throughput (for a 10Gb connection that's ~5 GB). Create a pair of 25 GB partitions for the ZIL rather than dedicating entire devices.
2. With the remaining space set up a mirrored Special VDEV with heavy use of small blocks. 32k block sizes will still give some room for compression which, other than saving space, can increase throughput and reduce IO/ps.
3. Pause scrubs (`zpool scrub -p`) when the workload is either expected to be heavy or is actually heavy based on moving averages and resume when its safe. Even my 24 TB NAS takes 24+ hours with no other activity.
> 1. From my own experience, most people who use a ZIL for faster sync writes are wasting space. You only need five seconds of peak throughput (for a 10Gb connection that's ~5 GB). Create a pair of 25 GB partitions for the ZIL rather than dedicating entire devices.
You might benefit from the same technique for a "special" vdev... If you run an all spinners array, but the metadata is on a small partition, you might get better seek times during metadata scans, since everything will be closer together? And/or if you use raidz for your regular vdev, but mirror for the metadata, you'll have more capacity for metadata reads.
> Even my 24 TB NAS takes 24+ hours with no other activity
Your NAS sounds pretty slow, it shouldn't take an hour to scrub 1tb of data (on average)
Yeah, the only place I use SLOG is with Optane devices in front of SATA/SAS flash. Anything that needs the speed of SLOG shouldn't be on rust anyways.
I would be in a different world if my system was backing banking or other loss critical data. For all my workloads, losing 5 seconds of writes in a crash is NBD.
> Sizing is measurable rather than guesswork. zdb -bb tank breaks down allocated space by block type; sum the non-data categories and you have your current metadata footprint.
Of course, this is also not a fast command on a slow array; my half full 2x 24TB mirror is estimating ~ 12 hours to run this; the last scrub took just over 24 hours to run. (After I posted, the remaining time dropped to 10 minutes... not as slow as it seemed, but still slow, and using a lot of I/O ... left for lunch and 50 minutes later it's almost done; estimation is hard)
Every 100+ TB ZFS pool I have is optimized for streaming reads/writes. IOPS are rarely a binding factor at these sizes.
Isn't that kind of the point of the article? The random reads required for metadata traversal during scrubs will kill performance for sequential reads and writes as well if you don't have the IOPS to back them up, building a pool without taking this into account will make regularly scheduled scrubs a pain point.
I schedule scrubs during idle time (10pm Friday - 6am Monday), so the metadata hit doesn't affect usage, but YMMV if you don't have an idle time. I also don't generally let them get that full.
If I was building the array new, I'd probably use a special VDEV for metadata, but they didn't exist when I built my last one of these.
ZFS scrub performance shouldn’t come at the expense of production worklaods. If scrubs cause latency or downtime, your pool lacks the IOPs and metadata performance needed for healthy maintanance.
This could have been a decent article if it hadn't been run through the AI slop machine. If I wanted to read writing this bad I would have just asked ChstGPT myself.
It's so hard to wade through run on sentences like "The knobs are not the problem. The pool was designed without room for this workload."
Oh sure, just buy more disks. Who do you think I am, that guy on the Monopoly box?
I stocked up on a few shuckable externals before the start of the second Grump catastrophe. $280/24TB (Exos inside). I'm finally getting around to setting up a pool, thinking 4+1 is kind of poor redundancy and wondering about adding another disk. The things are up to over $1,000 each, in the husk or not. Fucking hell. It makes me want to drastically scale back the size of my new pool in case a drive does die so I don't end up needing to buy a replacement any time soon.
I had a disk in my ZFS pool (a relatively tiny one, among the kind of people who have a home ZFS RAID5 array) go bad earlier this year and bought a used disk for the first time ever. Never even considered it before.
Still cost me as much as a pair of new drives the same size that I bought to transition from mirrored pairs to a four-disk RAID5 array about 16 months earlier. Like, god damn it was expensive.
A new disk would have cost me almost as much as all four disks did, originally.