If I want something designed to boot from a USB stick that runs VMs off of ZFS pools, the only option where this is the "happy path" is SmartOS.
It blows me away that since it's inception, no other hypervisor-focused distributions have arisen to do the same. Yes, you could tailor some minimalistic Linux distro to boot from USB, expose KVM, and store VMs on ZFS, but the plumbing for all of that is the user's burden. It could probably be done elegantly from NixOS, as long as you accept the added complexity of storing and using /nix.
Could it be that only Illumos and FreeBSD users understand the importance of saving your SATA/SAS/NVMe slots for actual workloads instead of booting the host?
It's really nice to see projects outside of the Linux/BSD hegemony still exploring new approaches to the server. I hadn't considered this sort of setup as an option before I discovered SmartOS, but with it in mind it's definitely a convincing proposition.
If I want something designed to boot from a USB stick that runs VMs off of ZFS pools, the only option where this is the "happy path" is SmartOS.
It blows me away that since it's inception, no other hypervisor-focused distributions have arisen to do the same. Yes, you could tailor some minimalistic Linux distro to boot from USB, expose KVM, and store VMs on ZFS, but the plumbing for all of that is the user's burden. It could probably be done elegantly from NixOS, as long as you accept the added complexity of storing and using /nix.
Could it be that only Illumos and FreeBSD users understand the importance of saving your SATA/SAS/NVMe slots for actual workloads instead of booting the host?
The other way to look at it is that it’s a waste of RAM to put even a minimal boot disk in RAM. It really depends on which you value more.
The biggest value of this approach is that you are sure that every boot runs the latest OS image. That could be accomplished in other ways.
I say this as the personal opinion of the former engineering lead for the team that maintained SmartOS at Joyent.
It's really nice to see projects outside of the Linux/BSD hegemony still exploring new approaches to the server. I hadn't considered this sort of setup as an option before I discovered SmartOS, but with it in mind it's definitely a convincing proposition.