> Somewhere, a LLM is trained on this code as we speak :)
If we collectively have to change the way we do things to better fit our tools rather than changing our tools to fit what we would naturally like to do, then we've failed as an industry.
Back then handwriting recognition of what you scribbled with a stylus was all the rage. But the PDA CPUs weren't good enough to recognize real handwriting predictably.
So Palm came up with a very simplified all caps alphabet that the user learned how to write instead. And had 98% recognition rate because it was designed for what the CPU could do.
Interesting! I can sort of see the value of that as a short-term solution to the hardware not being ready for something more robust.
I guess my main objection to having to change the way we write for LLMs is that the writing isn't just an input method that we use to generate the "real" thing we want (like the handwriting system that you describe, where it sounds like the intermediate handwriting was basically thrown away once it was properly read into text), and more importantly, the goal wasn't to write text for other humans but for CPUs; no one else was reading the intermediate input coming from the stylus. My concern is that if we change the way we write for other humans because we're concerned about whether LLMs will understand it properly, we're essentially saying that the LLM audience is more important than the human audience, or that we'd prefer to make the human audience adapt than improve the technology to be more human-friendly. I don't think either of those would be good things.
I came up with this one on my own many many years ago. Exact order and bs=1M and everything. I taught it that :)
Though I usually do the individual partitions before the main disk, to make sure fdisk doesn't complain about existing signatures as I start making partitions again. These days many disks and SD cards support "blkdiscard" to just wipe the whole thing (and temporarily gain some performance from clearing internal info), and "wipefs" became a thing to wipe the exact sequences fdisk and mkfs look for.
The "dd" method makes most sense on BIOS systems where you want to make sure to wipe the MBR boot code, and know that nothing is accidentally being preserved when you install your preferred bootloader.
I went through a similar "thought process" to build this: https://www.endbasic.dev/endbox.html last year. I originally wanted to "just" launch my binary right after the kernel started... but in the end settled for a full NetBSD base system to get things like network and WiFi configuration to work with ease. (That said, I still hook very early in the boot sequence to launch my own program and take over the console so that the rest of the system is invisible and initializes in the background.)
Thanks. I haven't documented it (yet?) but I did give a talk last year on the process that got me there and the internals behind this system: https://www.youtube.com/watch?v=WZFYTInWAqc
> .. There was a brief moment in the early 2000s where the newly introduced SSDs made booting quick ..
Anyone else reading this and getting the feeling someone is reminiscing about a past they didn't live and thus are seeing it through rose-colored glasses because it didn't exist?
In the early 2000s, SSDs had an obscene price tag of ~$1000 per gigabyte, so even an 8GB drive would cost you about $8000. There was nothing for the tech industry to ruin because only 0.01% of people owned a setup with an SSD. And neither Linux, NT or Mach were optimized for them. Hell, to this day Linux' VM system is by default still tuned for swap and slow HDDs (swappiness 60 and all that) because no one has bothered to update the defaults.
I remember this was the time when Google started pushing the Chromebook idea and highlighting how it could boot in "just a few seconds". One of the earliest models I got as a dogfooding device probably needed 20 seconds to boot or so. Nice, compared to the awful boot times of machines with HDDs... but not stellar.
But then, in 2011, my wife bought a MacBook Air with an SSD and I was blown away. That thing booted to a full desktop (and not the joke that ChromeOS was) in... 5, 6 seconds? It was ridiculous.
And we have lost all of those gains. I find it painful to witness how a recent Mac chews through I/O during boot or doing any sort of software update (iStat Menus is great to watch this sort of thing), and how these feel slower to that early experience of 15 years ago :-/
I distinctly remember buying my first SSD laptop in 2012 and being amazed at the boot time. At the time, I didn't realize it was because of the SSD, I just thought, "wow, the devs have made some fantastic optimizations in boot time!"
I wrote a batch script that that opened the entire Creative Suite in parallel and couldn't believe how much faster it was than just launching just Flash would have been on the old drive.
It was also around the time of the early netbooks, when the premise was a lean laptop and you'd do tasks that'd usually be done with relatively demanding local clients by the browser, as google and others were improving their online app offerings. They'd be using things like SATA/IDE to compact flash adapters to keep power down and responsiveness up, they were an ideal candidate for early SSDs as they didn't need a lot of capacity to run a linux + lightweight DE. Then they became just cheap laptops running full windows.
Yes. That's closer to the time mortals started to be able to afford SATA SSD's.
Though there were some disk-on-chip things before, but they were sized in megabytes. I think I might still have a PATA one somewhere. (And you could boot various memory sticks and cards for an even slower experience.)
When I swapped my disk in my mac for a SSD for the first time... It was insanely faster. Booted in 5-6 seconds. Apps like photoshop opened up cold in 2 seconds. So no, I think it was a huge leap.
I think no matter what the advance in tech is, it will get enshittified to tolerated "computer is working" slowness within months of said upgrade.
Tried out doing this kind of stuff a while back to create a custom user space, and I found that suckless’ init, sinit, was quite educational for how to deal with init’s responsibilities in a super minimal way: https://core.suckless.org/sinit/
As I'm currently exploring kernel build things, the alternative to `make tinyconfig` is `make allnoconfig` which is supposedly will not disable expert options and might be a little bit safer starting point.
I do the “Linux from Scratch” challenge every few years and the last time I did it, I got as far as I needed to be able to build antirez’ LOAD81[1] directly from single mode in a few seconds.. it was so satisfying. I lost the image in some later admin task, but I’m going to do it again, and maybe see about adding threads and SQLite support to LOAD81 alongside the other goodies, just for the fun of it ..
> I remember seeing a linux firewall/gateway set up to run with just the kernel, without any userspace at all. Completely unhackable.
Do you remember any details that would let me search for it? Because that does sound cool, and even maybe useful; the thought has certainly crossed my mind that a router or VPN box doesn't really get a lot of use out of userspace... Although maybe it's worth keeping for control/configuration/debugging.
> To print some text or run a simple program, I belive DOS without a memory manager would be even faster.
Or just make your code boot directly. It's not hard to make a .efi, or use
https://github.com/jart/cosmopolitan
to make a binary that runs in many places including bare metal.
IIRC, it ran a script as init process that set up the network cards, set up iptables, etc. and then just exited. Kernel would panic (the "init was killed" panic), but the network would still be functional. Automatic reboot on panic was disabled.
To reconfigure, the admin would simply reset it and start the system with "init=/something/else" as kernel parameter that booted to a normal userspace.
Basically, even if your application _can_ run as the kernel, and it's desirable for it to run with kernel-level permissions, do you really want production to be a world without strace and iotop and the like?
That's a halted firewall setup. Normally as part of shutdown you would tear down networking in SysVinit or systemd but you don't actually have to do that. When shutting down you can choose whether to power off or just to halt. It's basically like the old Windows "It is now safe to power off your PC".
> I... fail to see the point of running just one process.
It makes sense if you got some legacy piece of hardware that has extremely limited resources, both in terms of RAM and storage. Write your code in Go and you don't even need libc any more.
When you ship containers or to ultra-embedded, that one actually matters. If done right you can ship a container that only contains your application's binary, not a single byte more.
Wouldn't this be useful for embedded types of applications where you have a very specific task you want to do and you want to do it now.... like that firewall example?
Do do something actually useful, the program would have to access some data: network, disk, some sensors, etc. Network alone means scanning PCIe for the network card and configuring it, disk access needs controller also on PCIe, then scanning the ports for the drive, reading partition table, mounting the partition, etc.
All that takes a lot more than 1s. The speedup might not even be significant compared to a kernel optimized for that system (all modules built-in, nothing redundant), but full-featured, plus busybox or sysvinit alone.
You may also want to build and run busybox for your tiny userspace.
Other things you may want to experiment with is gen_init_cpio.c from linux kernel tree. It makes creating initramfs file structure easier from scripts.
And finally if sys/isolinux is also fun to use for minimal boot images.
Cool article! I'm working on a tangentially related issue requiring microVMs inside isolated infrastructure environments. Latency isn't really my main priority, but I am always tempted by any option to minimize attack surface. I wonder what it would take to replace the host block mount in this configuration with `vsock` for all communications between the host and guest microVM? Then you could avoid any files being mounted on the host at all while still enabling, e.g., one-way egress to a pre-signed S3 URL via a private VPC endpoint. Very cool article!
Amazon Nitro Enclave does pretty much this, the guest has one method of communication, via vsock, and it's up to you to build the pipes on either side.
It's a huge PITA in practice because whatever you want to run inside some enclave usually ends up being a 'normal program' that needs to talk TCP/IP over sockets... so your vsock I/O becomes a weird mix between a TUN proxy or a SOCKS5 local listener inside the VM that tunnels through vsock.
For example, I have the Windows NT 3.50 kernel compiling from scratch with virtio-net drivers, it's fairly straightforward for me to add a bus driver that runs over vsock inside Nitro Enclave that exposes itself (o,o) as a NIC then handle the tunneling logic in a usermode process in the host - but I don't uderstand the point of why you would do that when you already have sufficient attestation methods that don't require you to do vsock isolation.
It's so annoying how he starts out talking about boot speed, does the entire project, and then doesn't even mention how long it takes to boot up! All he ever says is "boots a bit quicker".
> I found “Building a tiny Linux from scratch” which does most of what I do here but in Rust and a year ago
Linux from scratch seems to still be doing fine at: https://www.linuxfromscratch.org. It's going on 27 years now.
> yeah, I know, proper C code needs to be scattered with return value checks and sensible reports of errno. I’ve left these out for clarity.
Somewhere, a LLM is trained on this code as we speak :)
But anyway, it's great that people are still interested in learning this stuff for fun.
> Somewhere, a LLM is trained on this code as we speak :)
If we collectively have to change the way we do things to better fit our tools rather than changing our tools to fit what we would naturally like to do, then we've failed as an industry.
Hmm LLM hype aside, remember the first writing system (Grafitti) for PalmOS?
Change the way you write to obtain much better recognition.
Sadly, I don't remember it. I was born in the early 90s, so although I remember PalmPilots being a thing, I never had one
Back then handwriting recognition of what you scribbled with a stylus was all the rage. But the PDA CPUs weren't good enough to recognize real handwriting predictably.
So Palm came up with a very simplified all caps alphabet that the user learned how to write instead. And had 98% recognition rate because it was designed for what the CPU could do.
https://en.wikipedia.org/wiki/Graffiti_(Palm_OS)#/media/File...
Interesting! I can sort of see the value of that as a short-term solution to the hardware not being ready for something more robust.
I guess my main objection to having to change the way we write for LLMs is that the writing isn't just an input method that we use to generate the "real" thing we want (like the handwriting system that you describe, where it sounds like the intermediate handwriting was basically thrown away once it was properly read into text), and more importantly, the goal wasn't to write text for other humans but for CPUs; no one else was reading the intermediate input coming from the stylus. My concern is that if we change the way we write for other humans because we're concerned about whether LLMs will understand it properly, we're essentially saying that the LLM audience is more important than the human audience, or that we'd prefer to make the human audience adapt than improve the technology to be more human-friendly. I don't think either of those would be good things.
> Somewhere, a LLM is trained on this code as we speak :)
Had a similar thought when I saw
I came up with this one on my own many many years ago. Exact order and bs=1M and everything. I taught it that :)
Though I usually do the individual partitions before the main disk, to make sure fdisk doesn't complain about existing signatures as I start making partitions again. These days many disks and SD cards support "blkdiscard" to just wipe the whole thing (and temporarily gain some performance from clearing internal info), and "wipefs" became a thing to wipe the exact sequences fdisk and mkfs look for.
The "dd" method makes most sense on BIOS systems where you want to make sure to wipe the MBR boot code, and know that nothing is accidentally being preserved when you install your preferred bootloader.
I went through a similar "thought process" to build this: https://www.endbasic.dev/endbox.html last year. I originally wanted to "just" launch my binary right after the kernel started... but in the end settled for a full NetBSD base system to get things like network and WiFi configuration to work with ease. (That said, I still hook very early in the boot sequence to launch my own program and take over the console so that the rest of the system is invisible and initializes in the background.)
Awesome work. I wish you'd write the process of getting it done step by step. I too would love to DIY my own netbsd.
Thanks. I haven't documented it (yet?) but I did give a talk last year on the process that got me there and the internals behind this system: https://www.youtube.com/watch?v=WZFYTInWAqc
> .. There was a brief moment in the early 2000s where the newly introduced SSDs made booting quick ..
Anyone else reading this and getting the feeling someone is reminiscing about a past they didn't live and thus are seeing it through rose-colored glasses because it didn't exist?
In the early 2000s, SSDs had an obscene price tag of ~$1000 per gigabyte, so even an 8GB drive would cost you about $8000. There was nothing for the tech industry to ruin because only 0.01% of people owned a setup with an SSD. And neither Linux, NT or Mach were optimized for them. Hell, to this day Linux' VM system is by default still tuned for swap and slow HDDs (swappiness 60 and all that) because no one has bothered to update the defaults.
They probably meant 2010s.
I remember this was the time when Google started pushing the Chromebook idea and highlighting how it could boot in "just a few seconds". One of the earliest models I got as a dogfooding device probably needed 20 seconds to boot or so. Nice, compared to the awful boot times of machines with HDDs... but not stellar.
But then, in 2011, my wife bought a MacBook Air with an SSD and I was blown away. That thing booted to a full desktop (and not the joke that ChromeOS was) in... 5, 6 seconds? It was ridiculous.
And we have lost all of those gains. I find it painful to witness how a recent Mac chews through I/O during boot or doing any sort of software update (iStat Menus is great to watch this sort of thing), and how these feel slower to that early experience of 15 years ago :-/
I distinctly remember buying my first SSD laptop in 2012 and being amazed at the boot time. At the time, I didn't realize it was because of the SSD, I just thought, "wow, the devs have made some fantastic optimizations in boot time!"
I wrote a batch script that that opened the entire Creative Suite in parallel and couldn't believe how much faster it was than just launching just Flash would have been on the old drive.
It was also around the time of the early netbooks, when the premise was a lean laptop and you'd do tasks that'd usually be done with relatively demanding local clients by the browser, as google and others were improving their online app offerings. They'd be using things like SATA/IDE to compact flash adapters to keep power down and responsiveness up, they were an ideal candidate for early SSDs as they didn't need a lot of capacity to run a linux + lightweight DE. Then they became just cheap laptops running full windows.
Or they meant 2010s?
Yes. That's closer to the time mortals started to be able to afford SATA SSD's.
Though there were some disk-on-chip things before, but they were sized in megabytes. I think I might still have a PATA one somewhere. (And you could boot various memory sticks and cards for an even slower experience.)
When I swapped my disk in my mac for a SSD for the first time... It was insanely faster. Booted in 5-6 seconds. Apps like photoshop opened up cold in 2 seconds. So no, I think it was a huge leap.
I think no matter what the advance in tech is, it will get enshittified to tolerated "computer is working" slowness within months of said upgrade.
Tried out doing this kind of stuff a while back to create a custom user space, and I found that suckless’ init, sinit, was quite educational for how to deal with init’s responsibilities in a super minimal way: https://core.suckless.org/sinit/
As I'm currently exploring kernel build things, the alternative to `make tinyconfig` is `make allnoconfig` which is supposedly will not disable expert options and might be a little bit safer starting point.
There's an option `make localyesconfig` (or similar)/which uses the kernel modules loaded on your current system.
I do the “Linux from Scratch” challenge every few years and the last time I did it, I got as far as I needed to be able to build antirez’ LOAD81[1] directly from single mode in a few seconds.. it was so satisfying. I lost the image in some later admin task, but I’m going to do it again, and maybe see about adding threads and SQLite support to LOAD81 alongside the other goodies, just for the fun of it ..
[1]- https://github.com/antirez/load81
I... fail to see the point of running just one process.
If it's just a PoC, then:
1) I remember seeing a linux firewall/gateway set up to run with just the kernel, without any userspace at all. Completely unhackable.
2) To print some text or run a simple program, I belive DOS without a memory manager would be even faster.
3) It takes 1s to boot linux, but an ordinary PC takes 10s to get to that linux. Even U-boot on ARM takes some seconds to load a kernel.
BTW, if anyone knows any current platform that can XiP a linux kernel, please share.
> I remember seeing a linux firewall/gateway set up to run with just the kernel, without any userspace at all. Completely unhackable.
Do you remember any details that would let me search for it? Because that does sound cool, and even maybe useful; the thought has certainly crossed my mind that a router or VPN box doesn't really get a lot of use out of userspace... Although maybe it's worth keeping for control/configuration/debugging.
> To print some text or run a simple program, I belive DOS without a memory manager would be even faster.
Or just make your code boot directly. It's not hard to make a .efi, or use https://github.com/jart/cosmopolitan to make a binary that runs in many places including bare metal.
IIRC, it ran a script as init process that set up the network cards, set up iptables, etc. and then just exited. Kernel would panic (the "init was killed" panic), but the network would still be functional. Automatic reboot on panic was disabled.
To reconfigure, the admin would simply reset it and start the system with "init=/something/else" as kernel parameter that booted to a normal userspace.
Oh, clever; I didn't know you could make Linux panic but keep running.
Configure it with CONFIG_PANIC_TIMEOUT=0, or add "panic=0" to kernel parameters, or "echo 0 > /proc/sys/kernel/panic".
> control/configuration/debugging
This is one of several major arguments made against unikernels in that famous Triton rant from a decade ago:
https://tritondatacenter.com/blog/unikernels-are-unfit-for-p...
Basically, even if your application _can_ run as the kernel, and it's desirable for it to run with kernel-level permissions, do you really want production to be a world without strace and iotop and the like?
That's a halted firewall setup. Normally as part of shutdown you would tear down networking in SysVinit or systemd but you don't actually have to do that. When shutting down you can choose whether to power off or just to halt. It's basically like the old Windows "It is now safe to power off your PC".
That beautiful orange text on a black background gives me nostalgia warm and fuzzies
> I... fail to see the point of running just one process.
It makes sense if you got some legacy piece of hardware that has extremely limited resources, both in terms of RAM and storage. Write your code in Go and you don't even need libc any more.
At that point, it would seem that an RTOS would be even more efficient - and if multithreading is not necessary, then just run it on bare metal.
> extremely limited resources, both in terms of RAM and storage
> Write your code in Go
When you ship containers or to ultra-embedded, that one actually matters. If done right you can ship a container that only contains your application's binary, not a single byte more.
> Write your code in Go
Container not even needed, there's projects like Tamago [0] that let you run your golang application without even needing an OS
[0] https://github.com/usbarmory/tamago
Wouldn't this be useful for embedded types of applications where you have a very specific task you want to do and you want to do it now.... like that firewall example?
Yeah, but there's a problem.
Do do something actually useful, the program would have to access some data: network, disk, some sensors, etc. Network alone means scanning PCIe for the network card and configuring it, disk access needs controller also on PCIe, then scanning the ports for the drive, reading partition table, mounting the partition, etc.
All that takes a lot more than 1s. The speedup might not even be significant compared to a kernel optimized for that system (all modules built-in, nothing redundant), but full-featured, plus busybox or sysvinit alone.
Embedded devices or other SoC situations, certain limited scope situations where docker is undesirable/unnecessary, etc.
> Completely unhackable.
I don't think this could ever be true.
It's be cool to dual boot with a Linux that has a ~1s boot time, drops you into neovim and lets you save text files to a shared partition.
It's possible to do a similar thing with any old smartphone. :)
https://xnux.eu/p-boot/
> the linux kernel configuration menu, a wonderful text menu system with a thousand options which has been baffling new users for about 30 years now.
So true.
In addition to C in the article and Rust linked to in the article, Go fans can use the similar https://gokrazy.org/ project.
> cpio is a very weird and ancient program with a command line which makes tar look user-friendly. But let’s not worry about the details for now.
So true. Been using cpio sporadically for many (many) years. And I still have to look it up. But at least I don't have to do `info cpio` anymore.
Highly instructive. My long term goal with my lone lisp programming language is to do exactly this.
You may also want to build and run busybox for your tiny userspace.
Other things you may want to experiment with is gen_init_cpio.c from linux kernel tree. It makes creating initramfs file structure easier from scripts.
And finally if sys/isolinux is also fun to use for minimal boot images.
Cool article! I'm working on a tangentially related issue requiring microVMs inside isolated infrastructure environments. Latency isn't really my main priority, but I am always tempted by any option to minimize attack surface. I wonder what it would take to replace the host block mount in this configuration with `vsock` for all communications between the host and guest microVM? Then you could avoid any files being mounted on the host at all while still enabling, e.g., one-way egress to a pre-signed S3 URL via a private VPC endpoint. Very cool article!
Amazon Nitro Enclave does pretty much this, the guest has one method of communication, via vsock, and it's up to you to build the pipes on either side.
It's a huge PITA in practice because whatever you want to run inside some enclave usually ends up being a 'normal program' that needs to talk TCP/IP over sockets... so your vsock I/O becomes a weird mix between a TUN proxy or a SOCKS5 local listener inside the VM that tunnels through vsock.
For example, I have the Windows NT 3.50 kernel compiling from scratch with virtio-net drivers, it's fairly straightforward for me to add a bus driver that runs over vsock inside Nitro Enclave that exposes itself (o,o) as a NIC then handle the tunneling logic in a usermode process in the host - but I don't uderstand the point of why you would do that when you already have sufficient attestation methods that don't require you to do vsock isolation.
I do like this as an exploration.
It's possible to boot a VM noticeably faster still, though I'm unclear on whether any of that applies to hardware:
https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-l...
.img modules in initial filesystem (initramfs.img) from --add-drivers drops to shell after dracut output.
see /etc/conf.d/ deprecated log level
I bought a couple of $60 refurbs online. This is perfect!
This is how we did it when I started Dashboard Linux. I booted directly into the c app.
Nicely explained. Next time you could try to run it on BIOS/EFI
buildroot does a nice job of tying a lot of this stuff together
"Butt Naked Linux" is how I read it.
I know it's off topic. I accept my downvotes.
I could end up stealing that for my own tinkering distro that never ends up releasing because it's awful.
It's so annoying how he starts out talking about boot speed, does the entire project, and then doesn't even mention how long it takes to boot up! All he ever says is "boots a bit quicker".
The photo of his monitor shows the kernel exec'ing his binary after 0.92 seconds.