This looks interesting and useful (I haven't tried it yet), but it is important to realize that every single useful Rust program has unsafe. Every single one. Why? Stdlib usage is full of it, and it must be by definition of what it does. In the same way you can't have a useful program without some side effects, so also you can't really have a useful program without doing some level of I/O and FFI, and I/O/FFI is always going to use unsafe under the covers.
That said, there is value in limiting your own unsafe use, and there might be value in limiting unsafe in the crates you use. However, this is really a question of "who do I trust to use unsafe? How much? Under what circumstances?" and NOT "is okay to have any unsafe?" because any useful program will contain a lot of unsafe if traced far enough in its call paths.
> Stdlib usage is full of it, and it must be by definition of what it does. In the same way you can't have a useful program without some side effects, so also you can't really have a useful program without doing some level of I/O and FFI, and I/O/FFI is always going to use unsafe under the covers.
Yep, it's not uncommon for people to misunderstand the point of Rust to be that nothing is ever unsafe. The point is that you can build safe abstractions on top of unsafe code, which narrows down the potential for those types of bugs to specific areas so that they can ideally be very small so that you can audit them more effectively to try to eliminate them (or at least more easily track them down if they do happen). In practice, I think the amount of code that actually needs to be unsafe is a lot smaller than even people who are aware of this might expect, because even if an API is unsafe, you can often push it low enough that a wrapper on top of it can enforce at least some of the invariants needed to use it safely.
I haven't needed to write unsafe code for the work I do very often (outside of stuff like FFI which often is also abstracted in a way that reduces the need for doing it manually), but when I do, I document each unsafe API with a list of the invariants needed to uphold for safe usage (which is pretty common) and then document each usage of it with a full proof of why I conclude that it's safe (which is somewhat common but more often it seems to be a single statement that isn't super rigorous rather than a list of statements that explain the full deduction of how to conclude whether each invariant is met). At the end of the proof, if there's at least one invariant that I haven't been able to prove, it means the API around whatever code I'm writing also needs to be unsafe and document for the caller needs to use it. Most of the time there's some low-hanging fruit for making some of them correct by construction though, e.g. by wrapping up the data in a struct that can only be created from outside the module with a constructor that upholds the variants correctly.
The stdlib's use of unsafe is the process of formal verification [0]. Unsafe usage is still a decent rule of thumb for dependencies you should pay more attention to.
Penalizing the stdlib for using `unsafe` would be extremely counter-productive, because you could almost trivially remove all `unsafe` in the stdlib by moving those "unsafe" operations into codegen emitted by compiler (which is essentially how every other memory-safe language under the sun works, including Java, Python, etc.). Voila, no more unsafe in the stdlib... except now you have exactly the same code existing in a form that's both harder to inspect and doesn't benefit from the bevy of tools that exist to audit unsafe blocks in regular Rust code, meaning you have an implementation that's less safe in practice. And outside of the code contained in the stdlib, the majority of Rust crates don't use `unsafe` at all (exact proportion varying by domain; e.g. embedded use cases will probably all use `unsafe` somewhere).
I think what would matter from this kind of measure is whether a project's use of unsafe actually has undefined behavior. Like the number of unsafe blocks is not really my concern as much as what the unsafe blocks are doing. If you build a single faulty abstraction via unsafe, anything that uses it is broken.
In my projects, it usually comes down to a scenario like needing to write inline assembly or invoke a foreign function, where there are close to zero guarantees the language can give me.
I agree that unsafe isn’t evil and shouldn’t be “avoided at all costs”, especially when using unsafe could be eg eliminated by the compiler (very common usage, actually!) or give you far superior codegen or code complexity.
But test coverage of unsafe blocks is not a meaningful metric. The best automated solution is standalone Miri runners exercising all branches of the code (via tests or otherwise) because tests on their own won’t catch things like out of counts reads or heap corruption unless you get lucky.
This looks interesting and useful (I haven't tried it yet), but it is important to realize that every single useful Rust program has unsafe. Every single one. Why? Stdlib usage is full of it, and it must be by definition of what it does. In the same way you can't have a useful program without some side effects, so also you can't really have a useful program without doing some level of I/O and FFI, and I/O/FFI is always going to use unsafe under the covers.
That said, there is value in limiting your own unsafe use, and there might be value in limiting unsafe in the crates you use. However, this is really a question of "who do I trust to use unsafe? How much? Under what circumstances?" and NOT "is okay to have any unsafe?" because any useful program will contain a lot of unsafe if traced far enough in its call paths.
> Stdlib usage is full of it, and it must be by definition of what it does. In the same way you can't have a useful program without some side effects, so also you can't really have a useful program without doing some level of I/O and FFI, and I/O/FFI is always going to use unsafe under the covers.
Yep, it's not uncommon for people to misunderstand the point of Rust to be that nothing is ever unsafe. The point is that you can build safe abstractions on top of unsafe code, which narrows down the potential for those types of bugs to specific areas so that they can ideally be very small so that you can audit them more effectively to try to eliminate them (or at least more easily track them down if they do happen). In practice, I think the amount of code that actually needs to be unsafe is a lot smaller than even people who are aware of this might expect, because even if an API is unsafe, you can often push it low enough that a wrapper on top of it can enforce at least some of the invariants needed to use it safely.
I haven't needed to write unsafe code for the work I do very often (outside of stuff like FFI which often is also abstracted in a way that reduces the need for doing it manually), but when I do, I document each unsafe API with a list of the invariants needed to uphold for safe usage (which is pretty common) and then document each usage of it with a full proof of why I conclude that it's safe (which is somewhat common but more often it seems to be a single statement that isn't super rigorous rather than a list of statements that explain the full deduction of how to conclude whether each invariant is met). At the end of the proof, if there's at least one invariant that I haven't been able to prove, it means the API around whatever code I'm writing also needs to be unsafe and document for the caller needs to use it. Most of the time there's some low-hanging fruit for making some of them correct by construction though, e.g. by wrapping up the data in a struct that can only be created from outside the module with a constructor that upholds the variants correctly.
The stdlib's use of unsafe is the process of formal verification [0]. Unsafe usage is still a decent rule of thumb for dependencies you should pay more attention to.
[0] https://github.com/model-checking/verify-rust-std
Penalizing the stdlib for using `unsafe` would be extremely counter-productive, because you could almost trivially remove all `unsafe` in the stdlib by moving those "unsafe" operations into codegen emitted by compiler (which is essentially how every other memory-safe language under the sun works, including Java, Python, etc.). Voila, no more unsafe in the stdlib... except now you have exactly the same code existing in a form that's both harder to inspect and doesn't benefit from the bevy of tools that exist to audit unsafe blocks in regular Rust code, meaning you have an implementation that's less safe in practice. And outside of the code contained in the stdlib, the majority of Rust crates don't use `unsafe` at all (exact proportion varying by domain; e.g. embedded use cases will probably all use `unsafe` somewhere).
I think what would matter from this kind of measure is whether a project's use of unsafe actually has undefined behavior. Like the number of unsafe blocks is not really my concern as much as what the unsafe blocks are doing. If you build a single faulty abstraction via unsafe, anything that uses it is broken.
In my projects, it usually comes down to a scenario like needing to write inline assembly or invoke a foreign function, where there are close to zero guarantees the language can give me.
I worry a little that perfectly cromulent Rust will get a bad name when the culture tends towards “unsafe is bad.”
Is there real value in these statistics vs. an approach where the measure is test coverage of unsafe blocks?
I agree that unsafe isn’t evil and shouldn’t be “avoided at all costs”, especially when using unsafe could be eg eliminated by the compiler (very common usage, actually!) or give you far superior codegen or code complexity.
But test coverage of unsafe blocks is not a meaningful metric. The best automated solution is standalone Miri runners exercising all branches of the code (via tests or otherwise) because tests on their own won’t catch things like out of counts reads or heap corruption unless you get lucky.
I agree about test coverage. I’d say it’s less bad but still doesn’t necessarily mean anything rigorous.
Short of formal verification, which I think is often going to be unreasonable, we generally have a spectrum of “less bad” options.