"Batteries included" ecosystems are the only persistent solution to the package manager problem.
If your first party tooling contains all the functionality you typically need, it's possible you can be productive with zero 3rd party dependencies. In practice you will tend to have a few, but you won't be vendoring out critical things like HTTP, TCP, JSON, string sanitation, cryptography. These are beacons for attackers. Everything depends on this stuff so the motivation for attacking these common surfaces is high.
I can literally count on one hand the number of 3rd party dependencies I've used in the last year. Dapper is the only regular thing I can come up with. Sometimes ScottPlot. Both of my SQL providers (MSSQL and SQLite) are first party as well. This is a major reason why they're the only sql providers I use.
Maybe I am just so traumatized from compliance and auditing in regulated software business, but this feels like a happier way to build software too. My tools tend to stay right where I left them the previous day. I don't have to worry about my hammer or screw drivers stealing all my bitcoin in the middle of the night.
There are several issues with "Batteries Included" ecosystems (like Python, C#/.NET, and Java):
1. They are not going to include everything. This includes things like new file formats.
2. They are going to be out of date whenever a standard changes (HTML, etc.), application changes (e.g. SQLite/PostgreSQL/etc. for SQL/ORM bindings), or API changes (DirectX, Vulcan, etc.).
3. Things like data structures, graphics APIs, etc. will have performance characteristics that may be different to your use case.
4. They can't cover all nice use cases such as the different libraries and frameworks for creating games of different genres.
For example, Python's XML DOM implementation only implements a subset of XPath and doesn't support parsing HTML.
The fact that Python, Java, and .NET have large library ecosystems proves that even if you have a "Batteries Included" approach there will always be other things to add.
"Batteries included" means "ossification is guaranteed", yah. "stdlib is where code goes to die" is a fairly common phrase for a reason.
There's clearly merit to both sides, but personally I think a major underlying cause is that libraries are trusted. Obviously that doesn't match reality. We desperately need a permission system for libraries, it's far harder to sneak stuff in when doing so requires an "adds dangerous permission" change approval.
comparing to Node, .NET is batteries included: built-in Linq vs needing lodash external package, built-in Decimal vs decimal.js package, built-in model validation vs class-validator & class-transformer packages, built-in CSRF/XSRF protection vs csrf-csrf package, I can go on for a while...
That's my point. You can have a large standard library like those languages I mentioned, but that isn't going to include everything nor cover every use case, so you'll have external libraries (via PyPi for Python, NuGet for .NET, and Maven for Java/JVM).
Batteries included systems are still susceptible to supply chain attacks, they just move slower so it’s not as attractive of a target.
I think packages of a certain size need to be held to higher standards by the repositories.
Multiple users should have to approve changes. Maybe enforced scans (though with trivy’s recent compromise that wont be likely any time soon)
Basically anything besides lone developer can decide to send something out on a whim that will run on millions of machines.
While technically true, it's so much slower that it's essentially a different thing. Third party packages being attacked is a near daily occurrence. First party attacks happens on the timescale and frequency of decades.
It's like the difference in protecting your home from burglars and foreign nation soldiers. Both are technically invaders to your home, but the scope is different, and the solutions are different.
> In practice you will tend to have a few, but you won't be vendoring out critical things like HTTP, TCP, JSON, string sanitation, cryptography
Unless you are Python, where the standard library includes multiple HTTP libraries and everyone installs the requests package anyways.
Few languages have good models for evolving their standard library, so you end up with lots of bad designs sticking around forever. Libraries are much easier to evolve, giving them the advantage in terms of developer UX and performance.
What type of developer chooses UX and performance over security? So reckless.
I removed the locks from all the doors, now entering/exiting is 87% faster!
After removing all the safety equipment, our vehicles have significantly improved in mileage, acceleration and top speed!
>What type of developer chooses UX and performance over security? So reckless.
Initially I assumed this is sarcastic, but apparently not. UX and performance is what programmers are paid to do! Making sure UX is good is one of the most important things in programmer job.
While security is a moving target, a goal, something that can never be perfect, just "good enough" (if NSA wants to hack you, they will). You make it sound like installing third party packages is basically equivalent to a security hole, while in practice the risk is low, especially if you don't overdo it.
Wild to read extreme security views like that, while at the same time there are people here that run unconstrained AI agents with --dangerous-skip-confirm flags and see nothing wrong with it.
Even more wild to read that sarcasm about "removing locks from doors for 87% speedup" is considered extreme...
And yes, we agree that running unconstrained AI agents with --dangerous-skip-confirm flags and seeing nothing wrong with it is insane. Kind of like just advertising for burglars to come open your doors for you before you get home - yeah, it's lots faster to get in (and to move about the house with all your stuff gone).
Better developer UX can directly lead to better safety. "You are holding it wrong" is a frequent source of security bugs, and better UX reduces the ways you can hold it wrong, or at least makes you more likely to hold it the right way
> Better developer UX can directly lead to better safety.
Depends. If you had to write add to a Makefile for your dependencies, you sure as hell aren't going to add 5k dependencies manually just to get a function that does $FOO; you'd write it yourself.
Now, with AI in the mix, there's fewer and fewer reasons to use so many dependencies.
Friction is helpful. Putting seatbelts on takes more time than just driving, but it’s way safer for the driver. Current dev practices increase speed, not safety.
"Security" is often more about corporate CYA than improving my actual security as a user, and sometimes in opposition, and there is often blatant disregard for any UX concession at all. The most secure system is fully encrypted with all copies of the encryption key erased.
Personally I've heard Odin [1] to do a decent job with this, at least from what I've superficially learned about its stdlib and included modules as an "outsider" (not a regular user).
It appears to have things like support for e.g. image file formats built-in, and new things are somewhat liberally getting added to core if they prove practically useful, since there isn't a package manager in the traditional sense.
Here's a blog post by the language author literally named "Package Managers are Evil" [2]
(Please do correct me if this is wrong, again, I don't have the experience myself.)
It doesn't matter. We pulled axios out of our codebase, but it still ends up in there as a child or peer from 40 other dependencies. Many from major vendors like datadog, slack, twilio, nx (in the gcs-cache extension), etc...
Fetch has also lacked support for features that xhr has had for over a decade now. For example upload progress. It's slowly catching up though, upload progress is the only thing I'd choose xhr for.
AI was trained on Axios wrappers, so it's just going to be wrappers all the way down. Look inside any company "API Client" and it's just a branded wrapper around Axios.
I'm not sure fetch is a good server-side API. The typical fetch-based code snippet `fetch(API_URL).then(r => r.json())` has no response body size limit and can potentially bring down a server due to memory exhaustion if the endpoint at API_URL malfunctions for some reason. Fine in the browser but to me it should be a no-no on the server.
> I'm not sure fetch is a good server-side API. The typical fetch-based code snippet `fetch(API_URL).then(r => r.json())` has no response body size limit and can potentially bring down a server due to memory exhaustion if the endpoint at API_URL malfunctions for some reason. Fine in the browser but to me it should be a no-no on the server.
Nor is fetch a good client-side API either; you want progress indicators, on both upload and download. Fetch is a poor API all-round.
I'm not saying that axios is unmaintained, I'm saying that if you want something like axios from the standard lib, fetch is the closest thing you get to official
It doesn't have a need _now_. Axios is more than 10 years old now, and even before axios other libraries did the same utility of making requests easier
The other thing that keeps coming up is the github-code-is-fine-but-the-release-artifact-is-a-trojan issue. It really makes me question if "packages" should even exist in JavaScript, or if we could just be importing standard plain source code from a git repo.
I understand why this doesn't work well with legacy projects, but it's something that the language could strive towards.
Yes - the postinstall hook attack vector goes away. You can do SHA pinning since Git's content addressing means that SHA is the hash of the content. But then your "lockfile" equivalent is just... a list of commit SHAs scattered across import statements in your source? Managing that across a real dependency tree becomes a nightmare.
This is basically what Deno's import maps tried to solve, and what they ended up with looked a lot like a package registry again.
At least npm packages have checksums and a registry that can yank things.
You can just git submodule in the dependencies. Super easy. Also makes it straightforward to develop patches to send upstream from within your project. Or to replace a dependency with a private fork.
In my experience, this works great for libraries internal to an organization (UI components, custom file formats, API type definitions, etc.). I don't see why it wouldn't also work for managing public dependencies.
Plus it's ecosystem-agnostic. Git submodules work just as well for JS as they do for Go, sample data/binary assets, or whatever other dependencies you need to manage.
> But then your "lockfile" equivalent is just... a list of commit SHAs scattered across import statements in your source? Managing that across a real dependency tree becomes a nightmare.
The irony is that this is actually the current best practice to defend against supply chain attacks in the github actions layer. Pin all actions versions to a hash. There's an entire secondary set of dev tools for converting GHA version numbers to hashes
> I understand why this doesn't work well with legacy projects, but it's something that the language could strive towards.
Why wouldn't that work well with legacy projects? In fact, the projects I was a part of that I'd call legacy nowadays, was in fact built by copy-and-pasting .js libraries into a "vendor/" directory, and that's how we shipped it as well, this was in the days before Bower (which was the npm of frontend development back in the day), vendoring JS libs was standard practice, before package managers became used in frontend development too.
Not sure why it wouldn't work, JavaScript is a very moldable language, you can make most things work one way or another :)(
or you don't use a package manager where anyone can just publish a package (i.e. use your system package manager). There is still some risk, but it is much smaller. Like, if xz were distributed by PyPI or NPM, everyone would have been pwned, but instead it was (barely) found.
It's true that system repos doesn't include everything, but you can create your own repositories if you really need to for a few things. In practice Fedora/EPEL are basically sufficient for my needs. Right now I'm deploying something with yocto, which is a bit more limited in slection, but it's pretty easy to add my own packages and it at least has hashes so things don't get replaced without me noticing (to be fair, I don't know if the security practices of open-embedded recipes are as strong as Fedora...).
it's muddying what a package is. A package, or a distro, is the people who slave and labor over packaging, reviewing, deciding on versions to ship, having policies in place, security mailing lists, release schedules, etc.
just shipping from npm crap is essentially the equivelant of running your production code base against Arch AUR pkgbuilds.
Fully agree with this! I think today .NET is probably the most batteries included platform you can get. This means that even if you use third-party libraries, these typically depend only on first-party dependencies, making it much less likely for something shady to sneak in.
C#'s LINQ (code as data, like LISP) wins over golang for any type of data access. Strongly-typed, language-native queries. Go has its own advantages though.
This is a rather superlative and tunnel vision, "everything is a nail because I'm a hammer" approach. The truth is this is an exceedingly difficult problem nobody has adequately solved yet.
I think the AI tooling is, if not completely solving sandboxing, at least making the default much better by asking you every time they want to do something and providing files to auto-approve certain actions.
Another layer of AI tooling is the cost of spinning up your own version of some libraries is lowered and can be made hyper specific to your needs rather than pulling in a whole library with features you'll never use.
> Another layer of AI tooling is the cost of spinning up your own version of some libraries is lowered and can be made hyper specific to your needs rather than pulling in a whole library with features you'll never use.
Tell me about it. Using AI Chatbots (not even agents), I got a MVP of a packaging system[1] to my liking (to create packages for a proprietary ERP system) and an endpoint-API-testing tool, neither of which require a venv or similar to run.
------------------------------
[1] Okay, all it does now is create, sign, verify and unpack packages. There's a roadmap file for package distribution, which is a different problem.
> at least making the default much better by asking you every time they want to do something
Really? I thought 'asking you every time they want to do something' was called 'security fatigue' and generally considered to be a bad thing. Yes you can concatenate files in the current project, Claude.
I agree that dependencies are a liability, but, sadly, "batteries included" didn't work out for Python in practice (i. e. how do I even live without numpy? No, array aren't enough).
To the extend that Python is indeed "batteries included," that seems true. But just how "batteries included" is it? I'd argue that its batteries are pretty limited. Exhibit A: everybody uses the third-party requests instead of the stdlib urllib. Exhibit B: http.server isn't a production-ready webserver, so people use Flask or something beefier.
I'd contrast Python with Go, which has an amazing stdlib for the domains that Go targets. This last part is key--Go has a more focused scope than Python, and that makes it easier for its stdlib to succeed.
So, youre on Microsoft then, judging by ScottPlot you write .NET desktop apps. If you use Dapper, you probably use Microsoft.Data.SqlClient, which is... distributed over NuGet and vulnerable to supply chain attack. You may not need many deps as a desktop dev. Modern day line of business apps require a lot more deps. CSVHelper, ClosedXML, AutoMapper, WebOptimizer,
NetEscapades.AspNetCore.SecurityHeaders.
Yes less deps people need the better but it doesn't fix trhe core problem. Sharing and distrib uting code is a key tenant of being able to write modern code.
What are some examples of batteries-included languages that folk around here really feel productive in and/or love? What makes them so great, in your opinion?
(Leaving aside thoughts on language syntax, compile times, tooling etc - just interested in people's experiences with / thoughts on healthy stdlibs)
I work in a NIS2 compliance sector, and we basically use Go and Python for everything. Go is awesome, Python isn't as such. Go didn't always come with the awesome stllib that it does today, which is likely partly why a lot of people still use things like Gin for web frameworks rather than simply using the standard library. Having worked with a lot of web frameworks, the one Go comes with is nice and easy enough to extend. Python is terrible, but on the plus side it's relatively easy to write your own libraries with Python, and use C/Zig to do so if you need it. The biggest challenges for us is that we aren't going to write a better MSSQL driver than Microsoft, so we use quite a bit of dependencies from them since we are married with Azure. These live in a little more isolation than what you might expect, so they aren't updated quite as often as many places might. Still, it's a relatively low risk factor that we can accept.
Our React projects are the contrast. They live in total and complete isolation, both in development and in production. You're not going to work on React on a computer that will be connected to any sort of internal resources. We've also had to write a novel's worth of legal bullshit explaining how we can't realistically review every line of code from React dependencies for compliance.
Anyway, I don't think JS/TS is that bad. It has a lot of issues, but then, you could always have written your own wrapper ontop of Node's fetch instead of using Axios. Which I guess is where working in the NIS2 compliance sector makes things a little bit different, because we'd always chose to write the wrapper instead of using one others made. With the few exceptions for Microsoft products that I mentioned earlier.
These are the big ones I use, specifically because of the standard libraries:
Python (decent standard library) - It's pretty much everywhere. There's so many hidden gems in that standard library (difflib, argparse, shlex, subprocess, cmd)
C#/F# (.NET)
C# feels so productive because of how much is available in .NET Core, and F# gets to tag along and get it all for free too. With C# you can compile executables down to bundle the runtime and strip it down so your executables are in the 15 MiB range. If you have dotnet installed, you can run F# as scripts.
Do you worry at all about the future of F#? I've been told it's feeling more and more like a second-class citizen on .NET, but I don't have much personal experience.
I used to, but the knowledge of .NET seems mostly transferrable to C#. It's super useful to do `dotnet fsi` and then work out the appropriate .NET calls in the F# repl.
This is exactly the world I'm working towards with packaging tooling with a virtual machine i.e. electron but with virtual machines instead so the isolation aspect comes by default.
For a lot of code, I switched to generating code rather than using 3rd party libraries.
Things like PEG parsers, path finding algorithms, string sanitizers, data type conversion, etc are very conveniently generated by LLMs. It's fast, reduces dependencies, and feels safer to me.
Or find the best third party library and copy the code from a widely used version that has been out long enough to have been well tested into your source tree.
The problem is not third party libraries. It is updating third party libraries when the version you have still works fine for your needs.
Don't do this. Use a package manager that let's you specify a specific version to pin against. Vendoring side steps most automated tooling that can warn you about vulnerabilities. Vendoring is a signal that your tooling is insufficient, 99% of the time.
Vendoring means you don't have to fetch the internet for every build, that you can work offline, that you're not at the mercy of the oh-so-close-99.999 availability, that it will keep on working in 10 years, and probably other advantages.
If your tooling can pull a dependency from the internet, it could certainly check if more recent version from a vendored one is available.
> Is there any package manager incapable of working offline?
I think you've identified the problem here: package management and package distribution are two different problems. Both tools have possibilities for exploits, but if they are separate tools then the surface area is smaller.
I'm thinking that the package distribution tool maintains a local system cache of packages, using keys/webrings/whatever to verify provenance, while the package management tool allows pinning, minver/maxver, etc.
Honestly, you can get pretty far with just Bun and a very small number of dependencies. It’s what I love most about Bun. But, I do agree with you generally. .NET is about as good as I’ve ever seen for being batteries included. I just hate the enterprisey culture that always seems to pervade .NET shops.
I agree about the culture. If I take my eye off the dev team for too long, I'll come back and we'll be using entity framework and a 20 page document about configuring code cleanup rules in visual studio.
Frankly inventing a new language is irresponsible these days unless you build on-top of an existing ecosystem because you need to solve all these problems.
> "Batteries included" ecosystems are the only persistent solution
Or write your own stuff. Yes, that's right, I said it. Even HTTP. Even cryptography. Just because somebody else messed it up once doesn't mean nobody should ever do it. Professional quality software _should_ be customized. Professional developers absolutely can and should do this and get it right. When you use a third-party HTTP implementation (for example), you're invariably importing more functionality than you need anyway. If you're just querying a REST service, you don't need MIME encoding, but it's part of the HTTP library anyway because some clients do need it. That library (that imports all of its own libraries) is just unnecessary bloat, and this stuff really isn't that hard to get right.
> When you use a third-party HTTP implementation (for example), you're invariably importing more functionality than you need anyway. If you're just querying a REST service, you don't need MIME encoding, but it's part of the HTTP library anyway because some clients do need it. That library (that imports all of its own libraries) is just unnecessary bloat, and this stuff really isn't that hard to get right.
This post is modded down (I think because of the "roll your own crypto vibe", which I disagree with), but this is actually spot on the money for HTTP.
The surface area for HTTP is quite large, and your little API, which never needed range-requests, basic-auth, multipart form upload, etc suddenly gets owned because of a vulnerability in one of those things you not only never used, you also never knew existed!
"Surface area" is a problem, reducing it is one way to mitigate.
I can't even imagine the scale of the impact with Axios being compromised, nearly every other project uses it for some reason instead of fetch (I never understood why).
Also from the report:
> Neither malicious version contains a single line of malicious code inside axios itself. Instead, both inject a fake dependency, plain-crypto-js@4.2.1, a package that is never imported anywhere in the axios source, whose only purpose is to run a postinstall script that deploys a cross-platform remote access trojan (RAT)
Good news for pnpm/bun users who have to manually approve postinstall scripts.
> nearly every other project uses it for some reason instead of fetch (I never understood why).
Fetch wasn't added to Node.js as a core package until version 18, and wasn't considered stable until version 21. Axios has been around much longer and was made part of popular frameworks and tutorials, which helps continue to propagate it's usage.
There are pretty much two usage patterns that come up all the time:
1- automatically add bearer tokens to requests rather than manually specifying them every single time
2- automatically dispatch some event or function when a 401 response is returned to clear the stale user session and return them to a login page.
There's no reason to repeat this logic in every single place you make an API call.
Likewise, every response I get is JSON. There's no reason to manually unwrap the response into JSON every time.
Finally, there's some nice mocking utilities for axios for unit testing different responses and error codes.
You're either going to copy/paste code everywhere, or you will write your own helper functions and never touch fetch directly. Axios... just works. No need to reinvent anything, and there's a ton of other handy features the GP mentioned as well you may or may not find yourself needing.
That fetch requires so many users to rewrite the same code - that was already handled well by every existing node HTTP client- says something about the standards process.
It could also be trivially written for XMLHttpRequest or any node client if needed. Would be nice if they had always been the same, but oh well - having a server and client version isn't that bad.
Because it is so few lines it is much more sensible to have everyone duplicate that little snippet manually than import a library and write interceptors for that...
(Not only because the integration with the library would likely be more lines of code, but also because a library is a significantly liability on several levels that must be justified by significant, not minor, recurring savings.)
> Because it is so few lines it is much more sensible to have everyone duplicate that little snippet manually
Mine's about 100 LOC. There's a lot you can get wrong. Having a way to use a known working version and update that rather than adding a hundred potentially unnecessary lines of code is a good thing. https://github.com/mikemaccana/fetch-unfucked/blob/master/sr...
> import a library and write interceptors for that...
What you suggesting people would have to intercept? Just import a library you trust and use it.
Don't be silly, this is the JS ecosystem. Why use your brain for a minute and come up with a 50 byte helper function, if you can instead import a library with 3912726 dependencies and let the compiler spend 90 seconds on every build to tree shake 3912723 out again and give you a highly optimized bundle that's only 3 megabytes small?
IMO interceptors are bad. they hide what might get transformed with the API call at the place it is being used.
> Likewise, every response I get is JSON. There's no reason to manually unwrap the response into JSON every time.
This is not true unless you are not interfacing with your own backends. even then why not just make a helper that unwraps as json by default but can be passed an arg to parse as something else
One more use case for Axios is it automatically follows redirects, forwarding headers, and more importantly, omiting or rewriting the headers that shouldn't be forwarded for security reasons.
fetch automatically follows redirects, fetch will forward your headers, omitting or rewriting headers is how security breaks… now a scraper got through because it’s masquerading as Chrome.
The fetch API is designed for browsers. It's not designed for servers. Fetch may work for a particular use case on the server, it may not. Servers have needs over and above what a browser allows the client to do.
> I can't even imagine the scale of the impact with Axios being compromised, nearly every other project uses it for some reason instead of fetch (I never understood why).
You can remember this answer for every time you ask same question again:
"Coz whatever else/builtin was before was annoying enough for common use cases"
I spent two years trying to get it out of a project that began long after Axios had become redundant but it's very hard to go back and challenge decisions like this because every business priority is aligned against this kind of work.
I expect libraries built on top of fetch will be the next to be compromised, because why would you use fetch without an arbitrary layer of syntactic sugar...
Does pnpm block postinstall on transitive deps too or just top-level? We have it configured at work but I've never actually tested whether it catches scripts from packages that get pulled in as sub-dependencies.
It prompts for transitive dependencies, too. I have never had workerd as a direct dependency of any project of mine but I get prompted to approve its postinstall script whenever I install cloudflare's wrangler package (since workerd needs to download the appropriate Workers runtime for your platform).
That's solid, really helps lock down the supply chain attack surface. Do you ever end up having to whitelist anything that legitimately needs to run on install?
After using pnpm for years (at least 5, don't remember exactly), I've only ever had to whitelist one library that uses a postinstall script to download a native executable for your system. And even this is not necessary, it's just poorly designed.
For example, esbuild and typescript 7 split binaries for different systems and architectures into separate packages, and rely on your package manager to pull the correct one.
> Good news for pnpm/bun users who have to manually approve postinstall scripts.
Would they not have approved it for earlier versions? But also wouldn't the chance of addition automatic approval be high (for such a widely used project)?
The prompt would be to approve the new malicious package (plain-crypto-js)'s scripts, too, which could tip users off that something was fishy. If they were used to approving one for axios and the attackers had just overwrote axios's own instead of making a new package, it would probably catch people out.
Assuming axios didn't have a postinstall script before, it wouldn't have been approved for a previous version. If you ignore it, you ignore it, but postinstall scripts are relatively rare in npm deps, so it would seem a bit out of place when the warning pops up.
Can't speak for other devs but I like to read postinstall scripts or at least put them through an LLM if they're too hard to grok.
It's also a little context dependent, for example if I was using Axios and I see a prompt to run the plain-crypto-js postinstall script, alarm bells would instantly ring, which would at least make me look up the changelog to see why this is happening.
In most cases I don't even let them run unless something breaks/doesn't work as expected.
Because axios existed before the builtin fetch, and so there's a lot of stackoverflow answers explaining how to use fetch, and the llm models are trained on that, so they will write axios requests instead of fetch
PSA: npm/bun/pnpm/uv now all support setting a minimum release age for packages.
I also have `ignore-scripts=true` in my ~/.npmrc. Based on the analysis, that alone would have mitigated the vulnerability. bun and pnpm do not execute lifecycle scripts by default.
Here's how to set global configs to set min release age to 7 days:
(Side note, it's wild that npm, bun, and pnpm have all decided to use different time units for this configuration.)
If you're developing with LLM agents, you should also update your AGENTS.md/CLAUDE.md file with some guidance on how to handle failures stemming from this config as they will cause the agent to unproductively spin its wheels.
Probably went with the simplest implementation, if starting from the current “seconds since epoch” value. Let the user do any calculations needed to translate three days into that measurement.
It also efficiently annoys the most people at once: those what want hours will complain if they set it to days, thought that want days will complain if hours are used. By using minutes or seconds you can wind up both segments while not offend those who rightly don't care because they can cope with a little arithmetic :)
Though doing what sleep(1) does would be my preference: default to seconds but allow m/h/d to be added to change that.
Hence the way I would do it (and have for other purposes), as stated in my final sentence. Have the human state the intent and convert to your own internally preferred units as needed.
No no no, see now we just say "computer! do tedious math!", and it will do some slightly different math for us and compliment us on having asked it to do so.
The one true unit of time is hexadecimal encoded nanoseconds since the unix epoch. (I'm only half joking because I actually have authored code that used that before.)
I actually think it is not too bad a design, because seconds are the SI base unit for time. Putting something like "x days" requires additional parsing steps and therefore complexity in the implementation. Either knowing or calculating how many seconds there are in a day can be expected of anyone touching a project or configuration at this level of detail.
Seconds are also unambiguous. Depending on your chosen definition, "X days" may or may not be influenced by leap seconds and DST changes.
I doubt anyone cares about an hour more or less in this context. But if you want multiple implementations to agree talking about seconds on a monotonic timer is a lot simpler
Could you explain what you mean re: ambiguity? I understand why “calendar units” like months are ambiguous, but minutes, hours, days, and weeks all have fixed durations (which is why APIs like Python’s `timedelta` allows them).
The minute between December 31, 2016 23:59 and January 1st 2017 is 61 seconds, not 60 seconds. The hour that contains that minute is 3601 seconds, the day that contains that hour is 43201 seconds, etc. If you assume a fixed duration and simply multiply by 43200, your math will be wrong compared to the rest of the world.
Daylight savings time makes a day take 23 hours or 25 hours. That makes a week take 7254000 seconds or 7261200 seconds. Etc.
That’s what I mean by calendar units. These aren’t issues if you don’t try to apply durations to the “real” calendar.
(This is all in the context of cooldowns, where I’m not convinced the there’s any real ambiguity risk by allowing the user to specify a duration in day or hour units rather than seconds. In that context a day is exactly 24 hours, regardless of what your local savings time rules are.)
"exactly 24 hours" could still be anywhere between 86399 and 86401 seconds, depending on leap seconds. At least if by an hour you mean an interval of 60 minutes, because a minute that contains a leap second will have either 59 or 61 seconds.
You could specify that for the purposes of cooldowns you want "hour" to mean an interval of 3600 seconds. But that you have to specify that should illustrate how ambiguous the concept of an hour is. It's not a useless concept by any means and I far prefer to specify duration in hours and days, but you have to spend a sentence or two on defining which definition of hours and days you are using. Or you don't and just hope nobody cares enough about the exact cooldown duration
Leap seconds are their own nightmare. UNIX time ignores them, btw, so that the unix epoch is 86400*number of days since 1/1/1970 + number of seconds since midnight. The behavior at the instance of a leap second is undefined.
In the UK last Sunday was 23 hours long because we switched to BST, and occasionally leap seconds will result in a minute being something other 60 seconds.
No it wasn't. The country instantaneously changed timezones from UTC+0 to UTC+1 (called something else locally), it was no different to any other timezone change from e.g. physically moving into another timezone.
I came here to argue the opposite. Expressing it in seconds takes away questions about time zones and DST.
I think you're incorrect to say that second are also ambiguous. Maybe what you mean is that days are more practical, but that seems very much a personal preference.
I understand the [flawed] reasoning behind "x seconds from now is going to be roughly now() + x on this particular system", but how does defining the cooldown from an external timestamp save you from dealing with DST and other time shenanigans? In the end you are comparing two timestamps and that comparison is erroneous without considering time shenanigans
that kind of complexity is always worth it. Every single time. It's user time that you're saving and it also makes config clearer for readers and cuts out on "too many/little zeroes on accident" errors
It's just library for handling time that 98% of the time your app will be using for something else.
Workdays! Think about it, if you set the delay in regular days/seconds the updated dependency can get pulled in on a weekend with only someone maybe on-call.
(Hope your timezones and tzdata correctly identifies Easter bank holiday as non-workdays)
In JavaScript something entirely new would be invented, to solve a problem that has long been solved and is documented in 20+ year old books on common design patterns. So we can all copy-paste `{ or: [{ days: 42, months: 2, hours: "DEFAULT", minutes: "IGNORE", seconds: null, timezone: "defer-by-ip" }, { timestamp: 17749453211*1000, unit: "ms"}]` without any clue as to what we are defining.
In Java, a 6000LoC+ ecosystem of classes, abstractions, dependency-injectables and probably a new DSL would be invented so we can all say "over 4 Malaysian workdays"
But you know that Java solution will continue working even after we no longer use the Gregorian Calendar, the collapse and annexation of Malaysia to some foreign power, and then us finally switching to a 4-day work week; so it'd be worth it.
... and since it was architectured to allow runtime injection-patching of events before they hit the enterprise-service-bus, everyone using this library must first set fourteen ENV vars in their profile, and provide a /etc/java/springtime/enterprise-workday-handling/parse-event-mismatch.jar.patch. Which should fix the bug for you.
You can find the patch files for your OSs by registering at Oracle with a J3EE8.4-PatchLibID (note, the older J3EE16-PatchLib-ids aren't compatible), attainable from your regional Oracle account-manager.
And least one of those environment can contain template strings that are expanded with arguments from request headers when run under popular enterprise java frameworks, and by way of the injection patching could hot load arbitrary code in runtime.
A joke should be funny though, not just a dry description of real life, so let's leave it at that. We've already taken it too far.
In before someone thinks it's a joke, the most commonly used logging library in Java had LDAP support in format scripts enabled by default" (which resulted, of course in CVE)
JavaScript Temporal. Not sure knowing what a "workday" is in each timezone is in it's scope but it's the much needed and improved JS, date API (granted with limited support to date)
…now imagine a list of instruments, some of which have durations specified in days/weeks/months (problems already with the latter) and some in workdays, and the user just told your app to display it sorted by duration.
Don't forget about regional holidays, which might follow arbitrary borders that don't match any of the official subdivisions of the country. Or may even depend on the chosen faith of the worker
Nah, working hours and make global assumptions of 0900-1230/1330-1730, M-F, and have an overly convoluted way to specify what working ours actually are in the relevant location(s).
They made a movie to make money. I doubt anyone holding the purse strings cared one iota if that bit were corrected or not. It’s not really a retcon either because they didn’t change anything.
That had more or less been the explanation in the books for decades, and even in George Lucas' notes from 1977:
> It's a very simple ship, very economical ship, although the modifications he made to it are rather extensive – mostly to the navigation system to get through hyperspace in the shortest possible distance (parsecs).
For Star Wars, they retconned it to mean he found the shortest possible route through dangerous space, so even for Han Solo's quote, it's still distance.
To me it sounds safer to have different big infra providers with different delays, otherwise you still hit everyone at the same time when something does inevitably go undetected.
And the chances of staying undetected are higher if nobody is installing until the delay time ellapses.
It's the same as not scheduling all cronjobs to midnight.
About the use of different units: next time you choose a property name in a config file, include the unit in the name. So not “timeout” but “timeoutMinutes”.
Yes!! This goes for any time you declare a time interval variable. The number of times I've seen code changes with a comment like "Turns out the delay arg to function foo is in milliseconds, not seconds".
At that point, you're making all your configuration fields strings and adding another parsing step after the json/toml/yaml parser is done with it. That's not ideal either; either you write a bunch of parsing code (not terribly difficult but not something I wanna do when I can just not), or you use some time library to parse a duration string, in which case the programming language and time library you happen to use suddenly becomes part of your config file specification and you have to exactly re-implement your old time handling library's duration parser if you ever want to switch to a new one or re-implement the tool in another language.
I don't think there are great solutions here. Arguably, units should be supported by the config file format, but existing config file formats don't do that.
> adding another parsing step after the json/toml/yaml parser is done with it. That's not ideal either
I'd argue that it is ideal, in the sense that it's the sweet spot for a general config file format to limit itself to simple, widely reusable building blocks. Supporting more advanced types can get in the way of this.
Programs need their own validation and/or parsing anyway, since correctness depends on program-specific semantics and usually only a subset of the values of a more simply expressed type is valid. That same logic applies across inputs: config may come from files, CLI args, legacy formats, or databases, often in different shapes. A single normalization and validation path simplifies this.
General formats must also work across many languages with different type systems. More complex types introduce more possible representations and therefore trade-offs. Even if a file parser implements them correctly (and consistently with other such parsers), it must choose an internal form that may not match what a program needs, forcing extra, less standard transformation and adding complexity on both sides for little gain.
Because acceptable values are defined by the program, not the file, a general format cannot fully specify them and shouldn’t try. Its role is to be a medium and provide simple, human-usable (for textual formats), widely supported types, avoid forcing unnecessary choices, and get out of the way.
All in all, I think it can be more appropriate for a program to pick a parsing library for a more complex type, than to add one consistently to all parsers of a given file format.
Another parsing step is the common case. Few parameters represent untyped strings where all characters and values are valid. For numbers as well, you often have a limited admissible range that you have to validate for. In the present case, you wouldn’t allow negative numbers, and maybe wouldn’t allow fractional numbers. Checking for a valid number isn’t inherently different from checking for a regex match. A number plus unit suffix is a straightforward regex.
> PSA: npm/bun/pnpm/uv now all support setting a minimum release age for packages.
The solution is not moar toolz. That's the problem—this crazy mindset that the problems endemic to bad tooling have a solution in the form of complementing them with another layer, rather than fewer.
Git and every sane SCM already allow you to manage your source tree without jumping through a bunch of hoops to go along with wacky overlay version control systems like the one that the npmjs.com crew designed, centering around package.json as a way to do an end-run around Git. You don't need to install and deploy anything containing never-before-seen updates just because the NodeJS influencer–developers say that lockfiles are the Right Way to do things. (It's not.)
Opting in to being vulnerable to supply chain attacks is a choice.
Is there a way to do that per repo for these tools ? We all know how user sided configuration works for users (they usually clean it whenever it goes against what they want to do instead of wondering why it blocks their changes :))
min release age to 7 days about patch releases exposes you to the other side of the coin, you have an open 7 days window on zero-day exploits that might be fixed in a security release
The packages that are actually compromised are yanked, but I assume you're talking about a scenario more like log4shell. In that case, you can just disable the config to install the update, then re-enable in 7 days. Given that compromised packages are uploaded all the time and zero-day vulnerabilities are comparatively less common, I'd say it's the right call.
At least with pnpm, you can specify minimumReleaseAgeExclude, temporarily until the time passes. I imagine the other package managers have similar options.
Urgent fix, patch released, invisible to dev team cause they put in a 7 day wait. Now our app is vulnerable for up to 7 days longer than needed (assuming daily deploys. If less often, pad accordingly). Not a great excuse as to why the company shipped an "updated" version of the app with a standing CVE in it. "Sorry we were blinded to the critical fix because set an arbitrary local setting to ignore updates until they are 7 days old". I wouldn't fire people over that, but we'd definitely be doing some internal training.
Because everyone got updates immediately. If the default was 7 days, almost no one would get updates immediately but after 7 days, and now someone only finds about after 7 days. Unless there is a poor soul checking packages as they are published that can alert the registry before 7 days pass, though I imagine very few do that and hence a dedicated attacker could influence them to not look too hard.
If I remember correctly, in all the recent cases it was picked up by automated scanning tools in a few hours, not because someone updated the dependency, checked the code and found the issue.
So it looks like even if no one actually updates, the vast majority of the cases will be caught by automated tools. You just need to give them a bit of time.
If everyone or a majority of people sets these options, then I think issues will simply be discovered later. So if other people run into them first, better for us, because then the issues have a chance of being fixed once our acceptable package/version age is reached.
And when you actually need a super hot fix for a 0-day, you will need to revert this and keep it that way for some time to then go back to minimum age.
While this works, we stillneed a permanent solution which requires a sort of vetting process, rather than blindly letting everything through.
I think my vetting would settle for a repo diff against the previous version, confirming the only difference was the security fix (though that doesn't cover all the bases).
Their analysis was triggered by open source projects upgrading en-masse and revealing a new anomalous endpoint, so, it does require some pioneers to take the arrows. They didn't spot the problem entirely via static analysis, although with hindsight they could have done (missing GitHub attestation).
A security company could set up a honeypot machine that installs new releases of everything automatically and have a separate machine scan its network traffic for suspicious outbound connections.
The problem is what counts as suspicious. StepSecurity are quite clear in their post that they decide what counts as anomalous by comparing lots of open source runs against prior data, so they can't figure it out on their own.
Can you elaborate? Why do you believe that motivated threat hunters won’t continue to analyze and find threats in new versions of open source software in the first week after release?
Attackers going "low and slow" when they know they're being monitored is just standard practice.
> Why do you believe that motivated threat hunters won’t continue to analyze and find threats in new versions of open source software in the first week after release?
I'm sure they will, but attackers will adapt. And I'm really unconvinced that these delays are really going to help in the real world. Imagine you rely on `popular-dependency` and it gets compromised. You have a cooldown, but I, the attacker, issue "CVE-1234" for `popular-dependency`. If you're at a company you now likely have a compliance obligation to patch that CVE within a strict timeline. I can very, very easily pressure you into this sort of thing.
I'm just unconvinced by the whole idea. It's fine, more time is nice, but it's not a good solution imo.
There are many options. Here's a post just briefly listing a few of the ones that would be handled by package managers and registries, but there are also many things that would be best done in CI pipelines as well.
Worth noting this attack was caught because people noticed anomalous network traffic to a new endpoint. The 7-day delay doesn't just give scanners time, it gives the community time to notice weird behavior from early adopters who didn't have the delay set.
It's herd immunity, not personal protection. You benefit from the people who DO install immediately and raise the alarm
But wouldn't the type of people that notifes anomalous network activity be exactly the type of people who add a 7 day delay because they're security conscious?
And I’ll bet a chunk of already-compromised vibe coders are feeling really on-top-of-shit because they just put that in their config, locking in that compromised version for a week.
I suspect most packages will keep a mix of people at 7 days and those with no limit. That being said, adding jitter by default would be good to these features.
This became evident, what, perhaps a few years ago? Probably since childhood for some users here but just wondering what the holdup is. Lots of bad press could be avoided, or at least a little.
While this explicitly calls out "postinstall", I'm pretty sure it affects other such lifecycle scripts like preinstall in dependencies.
The --ignore-scripts option will ignore lifecycle scripts in the project itself, not just dependencies. And it will ignore scripts that you have previously allowed (using the "allowBuilds" feature).
The config for uv won't work. uv only supports a full timestamp for this config, and no rolling window day option afaik. Am I crazy or is this llm slop?
> Define a dependency cooldown by specifying a duration instead of an absolute value. Either a "friendly" duration (e.g., 24 hours, 1 week, 30 days) or an ISO 8601 duration (e.g., PT24H, P7D, P30D) can be used.
This is what tripped me up. I added that config and then got this error:
error: Failed to parse: `.config/uv/uv.toml`
Caused by: TOML parse error at line 1, column 17
|
1 | exclude-newer = "7 days"
| ^^^^^^^^
failed to parse year in date "7 days": failed to parse "7 da" as year (a four digit integer): invalid digit, expected 0-9 but got
I was on version 0.7.20, so I removed that line, ran "uv self update" and upgraded to 0.11.2 and then re-added the config and it works fine now.
Yeah, that error message isn’t ideal on older versions, but unfortunately there’s no way to really address that. But I’m glad it’s working for you on newer versions.
I think it should work at the user config level too:
> If project-, user-, and system-level configuration files are found, the settings will be merged, with project-level configuration taking precedence over the user-level configuration, and user-level configuration taking precedence over the system-level configuration.
There’s a recurrent pattern with these package compromises: the attacker exfiltrates credentials during an initial phase, then pivots to the next round of packages using those credentials. That’s how we saw them make the Trivy to LiteLLM leap (with a 5 day gap), and it’ll almost certainly be similar in this case.
The solution to this is twofold, and is already implemented in the primary ecosystems being targeted (Python and JS): packagers should use Trusted Publishing to eliminate the need for long lived release credentials, and downstreams should use cooldowns to give security researchers time to identify and quarantine attacks.
(Security is a moving target, and neither of these techniques is going to work indefinitely without new techniques added to the mix. But they would be effective against the current problems we’re seeing.)
In this case, the author's NPM account was taken over, email address changed to one the attacker controls, and the package was manually published.
Since the attacker had full control of the NPM account, it is game over - the attacker can login to NPM and could, if they wanted, configure Trusted Publishing on any repo they control.
Axios IS using trusted publishing, but that didn't do anything to prevent the attack since the entire NPM account was taken over and config can be modified to allow publishing using a token.
Yeah, NPM should be enforcing 2FA and likely phishing resistant 2FA for some packages/ this should be a real control, issuing public audit events for email address changes, and publish events should include information how it was published (trusted publishing, manual publish, etc).
You don't want "project had X users so it's less safe" to suddenly transition into "now this software has X*10 users so it has to change things", it's disruptive.
Well, that sucks! It’ll be interesting to learn how they obtained a valid second factor or 2FA bypass; that will inform the next round of defenses here.
One wonders if Microsoft/npm.js should allow new packages to be published immediately following an account email address change? I mean changes to email address are already recognized as potential attack vectors, so emails are sent to the old address warning of potential account take over. But this seems to have been done at night, so the warning email would not be seen yet. Even so a new package could be published and served to the world immediately. Unless I misunderstand something about the facts this would indicate an extreme lack of imagination in the people at Microsoft who already went through several cycles of hardening the service against supply chain poisoning attacks.
I don’t know what this means. Discipline is good, but I think you need to have good tools/primitives in place to help people exercise discipline.
(The classic example being passwords: we wouldn’t need MFA is everybody just “got good” and used strong/unique passwords everywhere. But that’s manifestly unrealistic, so instead we use our discipline budget on getting people to use password managers and phishing-resistant MFA.)
Rejecting any packages newer than X days is one nice control, but ultimately it'd be way better to maintain an allowlist of which packages are allowed to run scripts.
Unfortunately npm is friggen awful at this...
You can use --ignore-scripts=true to disable all scripts, but inevitably, some packages will absolutely need to run scripts. There's no way to allowlist specific scripts to run, while blocking all others.
There are third-party npm packages that you can install, like @lavamoat/allow-scripts, but to use these you need to use an entirely different command like `npm setup` instead of the `npm install` everyone is familiar with.
This is just awful in so many ways, and it'd be so easy for npm to fix.
I recommend everyone to use bwrap if you're on linux and alias all package managers / anything that has post build logic with it.
I have bwrap configured to override: npm, pip, cargo, mvn, gradle, everything you can think of and I only give it the access it needs, strip anything that is useless to it anyway, deny dbus, sockets, everything. SSH is forwarded via socket (ssh-add).
This limits the blast radius to your CWD and package manager caches and often won't even work since the malware usually expects some things to be available which are not in a permissionless sandbox.
You can think of it as running a docker container, but without the requirement of having to have an image. It is the same thing flatpak is based on.
As for server deployments, container hardening is your friend. Most supply chain attacks target build scripts so as long as you treat your CI/CD as an untrusted environment you should be good - there's quite a few resources on this so won't go into detail.
This only works for post-install script attacks. When the package is compromised, just running require somewhere in your code will be enough, and that runs with node/java/python and no bwrap.
node is also sandboxed within bwrap I have sandbox -p node if I have to give node access to other folders, I also have sandbox -m to define custom mountpoints if necessary and UNSAFE=1 as a last resort which just runs unsandboxed.
Check also https://github.com/wrr/drop which is a higher-level tool than bwrap. It allows you to make such isolated sandboxes with minimal configuration.
This looks nice but I wouldn't trust a very fresh tool to do security correctly.
As a higher-level alternative to bwrap, I sometimes use `flatpak run --filesystem=$PWD --command=bash org.freedesktop.Platform`. This is kind of an abuse of flatpaks but works just fine to make a sandbox. And unlike bwrap, it has sane defaults (no extra permissions, not even network, though it does allow xdg-desktop-portal).
Shame it's not a bit more mature, it does look like more the sort of thing I want. I use firejail a bit, but it's a bit awkward really.
To be honest - and I can't really believe I'm saying it - what I really want is something more like Android permissions. (Except more granular file permissions, which Android doesn't do at all well.) Like: start with nothing, app is requesting x access, allow it this time; oh alright fine always allow it. Central place to manage it later. Etc.
AFAIK maven doesn’t support post install logic like npm does. You have to explicitly optin with build plugins. It doesn’t let any arbitrary dependency run code on your machine.
I like the idea of bubblewrap, but my pain point is that it is work to set it up correctly with bind mounts and forwarding necessary environment variables to make the program actually work usefully. Could you share your pip bwrap configuration? It sounds useful.
Maybe I misunderstood this point. But the ssh socket also gives access to your private keys, so I see no security gain in that point. Better to have a password protected key.
It's so your private key is not stolen, but you're right passphrase protected keys win anyway. I use hardware keys so this isn't a problem for me to begin with.
I've been advocating to minimize the number of dependencies for some time now. Once you've maintained an open source project for several years, you start to understand the true cost of dependencies and you actually start to pay attention to the people behind the libraries. Popularity doesn't necessarily mean reliable or trustworthy or secure.
Agree. This is one of the major takeaways I've had from writing Go over the years -- which is even a Go proverb [0], "a little copying is better than a little dependency." Fortunately, LLMs make writing your own implementations of little dependencies super easy too.
I'll agree with that, and you would think it's common sense for any competent engineer, but for many people it's just an afterthought. Including senior and lead engineers. General matters like security are a political liaison, how can you raise the alarm/advocate for critical security improvements as an IC without making people around and above you look bad? How can you justify time invested into these things without raising the alarm? At the same time, you can't just ignore glaring security holes. It's a fine line to walk, and actually being realistic about the possibilities has found me nothing but enmity from peers and superiors due to it seeming like I'm throwing them under the bus.
In general, management was to see progress. I've come to find that technical details like these are an afterthought for most engineers, so far as the deadlines are being met.
It's one of these things that are under the water, tech side jobs. Everyone has to be on board, if your peers don't give a fuck you're just an annoyance and will be swimming counter-current.
Not to beat a dead horse but I see this again and again with dependencies. Each time I get more worried that the same will happen with rust. I understand the fat std library approach won’t work but I really still want a good solution where I can trust packages to be safe and high quality.
Linux distros and BSD ports did that since the 90's. When Linux distros had barely a PM or just tarballs, Infomagic sold 4 CD full of libre software. When I had no internet at home, back in the day I bought 3 DVD's of Debian Sarge for 20 euros, about $20. A bargain, it was the price of a hard-cover best seller book.
GB's of libre software, graphical install, 2.6 kernel, KDE3 desktop, very light on my Athlon 2000 with 256MB of RAM. It was incredible compared to what you got with Windows XP and 120 Euro per seat. Nonfree software and almost empty.
And, well, if for instance I could get read only, ~16TB durable USB drive with tons of Guix packages offline (in a two yearly basis with stable releases) for $200 I would buy them in the spot.
You would say that $200 for a distro it's expensive, but for what it provides, if you are only interested in libre gaming and tools, they amount you save can be huge. I've seen people spend $400 in Steam games because of the Holyday sales...
Why wouldn't the "fat std" thing work? Yes it's hard to design properly, both in scope and actual design (especially for an unstandardized language still moving fast), but throwing the towel and punting the problem to the "free market" of uncurated public repos is even worse.
It's what we call in France "la fête du slip".
PS: that's one reason I try to use git submodules in my Common Lisp projects instead of QuickLisp, because I really see the size of my deptree this way.
Not necessarily, but let's agree that some design faults would happen: you still get the option to use the solid, boring and slightly rusty std instead of another 100 dependencies from the supply chain supermarket.
At work, we're happy with Python's included batteries when we need to make scripts instead of large programs.
NPM should have a curation mechanism, via staff review or crowdsourcing, where versions of popular packages are promoted to a stable set, like linux distros do. I would only use curated versions if they had such a thing.
- copy the dependencies' tests into your own tests
- copy the code in to your codebase as a library using the same review process you would for code from your own team
- treat updates to the library in the same way you would for updates to your own code
Apparently, this extra work will now not be a problem, because we have AI making us 10x more efficient. To be honest, even without AI, we should've been doing this from the start, even if I understand why we haven't. The excuses are starting to wear thin though.
I don't know where you've worked but a hostile and intelligent actor or internal red team would succeed under each of those cases at every job I've worked at.
Defending against a targeted attack is difficult, yes. But these recent campaigns were all directed at everyone. Auditing and inspecting your dependencies does absolutely help thwart that because there will always be people who don't.
# If you have a projects folder containing multiple projects on macOS, you can run this script to recursively scan all subfolders for vulnerable axios versions and the presence of plain-crypto-js, helping you quickly identify potentially affected projects:
Genuinely how are you supposed to make sure that none of the software you have on your system pulls this in?
It’s things like this that make me want to swap to Qubes permanently, simply as to not have my password manager in the same context as compiling software ever.
We run everything NPM related inside Apple containers, and are looking to do the same with Python and Rust soon. Bwrap on Linux does the same.
I like to think of it like working with dangerous chemicals in the lab. Back in the days, people were sloppy and eventually got cancer. Then dangers were recognized and PPE was developed and became a requirement.
We are now at the stage in software development where we are beginning to recognizing the hazards and developing + mandating use of proper PPE.
A couple of years ago, pip started refusing to install packages outside of a virtualenv. I'm guessing/hoping package managers will start to have an opt-in flag you can set in a system-wide config file, such that they refuse to run outside of a sandbox.
The problem is that package managers are a distraction. You have to sandbox everything or else it doesn't work. These attacks use post-install hooks for convenience but nothing would have stopped them patching axios itself and just waiting for devs to run the app on their local workstation. So you end up needing to develop in a fully sandboxed environment.
While it's not perfect, pinning specific versions and managing all updates directly has been a solid solution for my team. Things can of course still slip through, but we're never vulnerable to these just because there was a new package release and we opted into it by default.
Updating packages takes longer, but we try to keep packages to a minimum so it ends up not being that big deal.
This sounds like satire but isn't - I just make sure the nodejs/npm packages don't exist on my system. I've yet to find a crucial piece of software that requires it. As much as I love that cute utility that turns maps into ascii art, it's not exactly sqlite in terms of usefulness.
I don't deny that node/npm is useful for building servers, devtools for JS development itself, etc. but as an end user I haven't encountered anything useful which requires having it on my machine.
Essential steps to minimise your exposure to NPM supply chain attacks:
— Run Yarn in zero-installs mode (or equivalent for your package manager). Every new or changed dependency gets checked in.
— Disable post-install scripts. If you don’t, at least make sure your package manager prompts for scripts during install, in which case you stop and look at what it’s going to run.
— If third-party code runs in development, including post-install scripts, try your best to make sure it happens in a VM/container.
— Vet every package you add. Popularity is a plus, recent commit time is a minus: if you have this but not that, keep your eyes peeled. Skim through the code on NPM (they will probably never stop labelling it as “beta”), commit history and changelog.
— Vet its dependency tree. Dependencies is a vector for attack on you and your users, and any new developer in the tree is another person you’re trusting to not be malicious and to take all of the above measures, too.
Number 1 would only be a win for zero-installs if it happened that registry was up when you made the security hotfix, since you'd need to install the depdencency the first time to get it in VC, but then suddenly down when doing a deploy. Seems like a highly unlikely scenario to me. Also, cases where npm CVEs must be patched with such urgency or bad things will happen are luckily very rare, in my experience.
Most npm CVEs are stuff like DDoS vulnerabilities, and you should have mitigations for those in place for at the infra-level anyway (e.g. request timeouts, rate limits, etc), or you are pretty much guaranteed to be cooked sooner or later anyway. The really dangerous stuff like arbitrary command execution from a library that takes end user input is much much more rare. The most recent big one I remember is React2shell.
Number 2 hasn't been much of an issue for a long time. npm doesn't allow unpublishing package after 72 hours (apart from under certain rare conditions).
Don't know about number 3. Would feel to me that if you have something running that can modify lockfile, they can probably also modify the chekced-in tars.
I can see how zero-installs are useful under some specific constraints where you want to minimize dependencies to external services, e.g. when your CI runs under strict firewalls. But for most, nah, not worth it.
> you'd need to install the depdencency the first time to get it in VC, but then suddenly down when doing a deploy.
Which dependency? It sounds like you are assuming some specific scenario, whereas the fix can take many forms. In immediate term, the quickest step could be to simply disable some feature. A later step may be vendoring in a safe implementation.
The registry doesn’t need to be actually down for you, either; the necessary condition is that your CI infrastructure can’t reach it.
> cases where npm CVEs must be patched with such urgency or bad things will happen are luckily very rare, in my experience.
Not sure what you mean by “npm CVEs”. The registry? The CLI tool?
As I wrote, if you are running compromised software in production, you want to fix it ASAP. In first moments you may not even know whether bad things will happen or not, just that you are shipping malicious code to your users. Even if you are lucky enough to determine with 100% confidence (putting your job on the line) that the compromise is inconsequential, you don’t want to keep shipping that code for another hour because your install step fails due to a random CI infra hiccup making registry inaccessible (as happened in my experience at least half dozen times in years prior, though luckily not in a circumstance where someone attempted to push an urgent security fix). Now imagine it’s not a random hiccup but part of a coordinated targeted attack, and somehow it becomes something anticipated.
> Number 2 hasn't been much of an issue for a long time. npm doesn't allow unpublishing package after 72 hours (apart from under certain rare conditions).
Those rare conditions exist. Also, you are making it sound as if the registry is infallible, and no humans and/or LLMs there accept untrusted input from their environment.
The key aspect of modern package managers, when used correctly, is that even when the registry is compromised you are fine as long as integrity check crypto holds up and you hold on to your pre-compromise dependency tree. The latter is not a technical problem but a human problem, because conditions can be engineered in which something may slip past your eyes. If this slip-up can be avoided at little to no cost—in fact, with benefits, since zero-installs shortens CI times, and therefore time-to-fix, due to dramatically shorter or fully eliminated install step—it should be a complete no-brainer.
> Don't know about number 3. Would feel to me that if you have something running that can modify lockfile, they can probably also modify the chekced-in tars.
As I wrote, I suspect it’d complicate such attacks or make them easier to spot, not make them impossible.
Are you saying it replaces my package manager, or that I should add another tool to my stack, vet yet another vulnerable dependency for critical use, to do something my package manager already does just as well?
> You ~never want to vendor libraries.
I just explained why you should, and you are yet to provide a counter-argument.
Exactly. Yarn uses a yarn.lock file with the sha256 hashes of each npm package it downloads from the repo (they are .tgz files). If the hash won't match, install fails. No need to commit the dependencies into your git.
Very detailed and props to the security researchers, but the blog post has several indicators that it was written by AI, to which point I suspect their malware analysis was also done by a LLM.
I just wish it had more human interaction rather than have a GenAI spit out the blog post. It's very repetitive and includes several EM dashes.
It is harder and harder to trust any blog post anymore, the more AI there is. I used to read blog posts because of the personality and the precision level. Now both have been taken away.
How much do you want to bet me that the credential was stolen during the previous LiteLLM incident? At what point are we going to have to stop using these package managers because it's not secure? I've got to admit, it's got me nervous to use Python or Node.js these days, but it's really a universal problem.
> it’s got me nervous to use Python or Node.js these days
My feelings precisely. Min package age (supported in uv and all JS package managers) is nice but I still feel extremely hesitant to upgrade my deps or start a new project at the moment.
I don’t think this is going to stabilize any time soon, so figuring out how to handle potentially compromised deps is something we will all need to think about.
I would be avoiding npm itself on principle in the JS ecosystem. Use a package manager that has a history of actually caring about these issues in a timely manner.
It almost doesn't matter, because you can get pwned by a transitive dependency. If someone doesn't have the same scruples as you have, you're still at risk.
PNPM makes you approve postinstall scripts instead of running them by default, which helps a lot. Whenever I see a prompt to run a postinstall script, unless I know the package normally has one & what it does, I go look it up before approving it.
(Of course I could still get bitten if one of the packages I trust has its postinstall script replaced.)
I suppose you would have to commit your node_modules, or otherwise cache your setup so that all prerequesite modules are built and ready to install without running post-install scripts?
Has anyone else noticed there was a recent sudden flurry of 3000 deleted issues on axios/axios? The jump happened on March 23. Was this a first sign of compromise? Or just coincidence of an AI agent going rogue.
Maybe just a coincidence but they have cubic-dev-ai edit every single PR with a summary. And that bot edits PR descriptions even for outside contributors.
We have libraries like SQLite, which is a single .c file that you drag into your project and it immediately does a ton of incredibly useful, non-trivial work for you, while barely increasing your executable's size.
The issue is not dependencies themselves, it's transitive ones. Nobody installs left-pad or is-even-number directly, and "libraries" like these are the vast majority of the attack surface. If you get rid of transitive dependencies, you get rid of the need of a package manager, as installing a package becomes unzipping a few files into a vendor/ folder.
There's so many C libraries like this. Off the top of my head, SQLite, FreeType, OpenSSL, libcurl, libpng/jpeg, stb everything, zlib, lua, SDL, GLFW... I do game development so I'm most familiar with the ones commonly used in game engines, but I'm sure other fields have similarly high quality C libraries.
They also bindings for every language under the sun. Rust libraries are very rarely used outside of Rust, and C#/Java/JS/Python libraries are never used outside their respective language (aside form Java ones in other JVM langs).
Package managers are now basically a requirement for language adoption. Doing it manually is not a solution, in an automated world.
What is a problem is library quality. Which is downstream of nobody getting paid for it, combined with an optimistic but unrealistic "all packages are equal" philosophy.
> High quality C libraries
> OpenSSL
OpenSSL is one of the ones where there's a ground up rewrite happening because the code quality is so terrible while being security critical.
On the other end, javascript is uniquely bad because of the deployment model and difficulty of adding things to the standard library, so everything is littered with polyfills.
> Package managers are now basically a requirement for language adoption. Doing it manually is not a solution, in an automated world.
Absolute nonsense. What does automated world even mean? Even if one could infer reasonably, it's no justification. Appealing to "the real world" in lieu of any further consideration is exactly the kind of mindlessness that has led to the present state of affairs.
Automation of dependency versions was never something we needed it was always a convenience, and even that's a stretch given that dependency hell is abundant in all of these systems, and now we have supply chain attacks. While everyone is welcome to do as they please, I'm going to stick to vendoring my dependencies, statically compiling, and not blindly trusting code I haven't seen before.
Relax, while mentioning the real world without any criticism for the soundness of the solution is absolute nonsense, some would say idiotic, thinking only in the absolute best solution given your narrow world view is not any better.
While I agree that my view is narrow, the "best solution" in question is what we used to do, and it was fine. There are still many places that manually manage dependencies. Fundamentally automatic software versioning is an under-developed area in need of attention, and technologies like semantic versioning which are ubiquitous are closer to suggestions, and not true indicators of breaking changes. My personal view is that fully automatic dependency version management is an ongoing experiment and should be treated as such.
They're not a failed experiment. No one has ever "experimented" by making a safe package manager for their new language. And it is not that insane to do so. Very basic things will get you very far:
1. Packages should carry a manifest that declares what they do at build time, just like Chrome extensions do. This manifest would then be used to configure its build environment.
2. Publishers to official registries should be forced to use 2FA. I proposed this a decade ago for crates.io and people lost their minds, like I was suggesting we drag developers to a shed to be shot.
3. Every package registry should produce a detailed audit log that contains a "who, what, when". Every build/ command should be producing audit logs that can be collected by endpoint agents too.
4. Every package registry should support TUF.
5. Typosquatting defenses should be standard.
etc etc etc. Some of this is hard, some of this is not hard. All of this is possible. No one has done it, so it's way too early to say "package managers can't be made safe" when no one has tried.
I don't understand commercial aspect of large OSS like package managers but i was wondering for years why this was missing from npm.
I think typosquatting was handled by npm last year but only after some popular miss typed packages started stealing developer creds.
The people building package managers are unaware of these problems going into it and it becomes extremely disruptive to start adding these things later on since your entire ecosystem is built on the assumption that they can do these things.
It's also shockingly controversial to suggest typosquatting suggestions. I made this suggestion ages ago for cargo, demonstrated that basic distance checks would have impacted <1% of crates over all time, and people still didn't want it.
Sorry, I should have clarified that - you're correct. `cosign` is an example of a tool that makes this quite straightforward and proves that this sort of system can work today.
> Publishers to official registries should be forced to use 2FA. I proposed this a decade ago for crates.io and people lost their minds, like I was suggesting we drag developers to a shed to be shot.
How is this enforced when it's pushed via a pipeline?
Your account is separate from your publishing. That is, in order to go to my account to change configuration values, 2FA must be required.
Publishing should be handled via something like Trusted Publishing, which would leverage short lived tokens and can integrate with cryptographic logs for publish information (ie: "Published from the main branch of this repo at this time").
If you're developing for the web your attack surface is quite a bit bigger. Your proposed solution of copying a few files might work but how do you keep track of updates? You might be vulnerable to a published exploit fixed a few months ago. A package manager might tell you a new version is available. I don't know how that would work in your scenario.
> We have libraries like SQLite, which is a single .c file that you drag into your project and it immediately does a ton of incredibly useful, non-trivial work for you, while barely increasing your executable's size.
I'm not sure why you believe this is more secure than a package manager. At least with a package manager there is an opportunity for vetting. It's also trivial that it did not increase your executable's size. If your executable depends on it, it increases its effective size.
The culture within the npm/js community has mainly been one of using the package manager rather than "re-inventing the wheel", as such the blast radius of a compromised package is much greater
It's more to do with the standard library being so barren of common application needs, and looking for a solution that the community has gotten behind. Axios has been a common dependency in many codebases, because it is a solid solution that many have already used. Every developer could try building all the libraries that they would reach for themselves, but then each company has now taken on the task of ensuring their own (much larger) codebase is free from security issues, on top of taking care of their own issues and bugs.
It’s not just NPM, though. Every Rails project and every Rust project I’ve seen ended up with massive numbers of dependencies vs what an equivalent project in Go or C# would have needed.
CPAN too, just try Hailo under Perl to test an old-fashioned chatbot based on Markov chains where very small LLM's and Hailo converge if used with the advanced training options for it. Yes, it will pull tons of dependencies, (less with cpanminus if run with 'cpanm -n Hailo'), but contrary to NPM, Pip and the like CPAN's repos are highly curated and before PHP and ubiquitoous Python Perl was used everywhere, from a sysadmin language (better than Bash/Sh for sure) to CGI, IRC bots and whatnot. How many issues did we have? Zero or near zero.
I don't think this community of professionals is going to come around to a solution which requires marginally more effort.
If no one checks their dependencies, the solution is to centralize this responsibility at the package repository. Something like left-pad should simply not be admitted to npm. Enforce a set of stricter rules which only allow non-trivial packages maintained by someone who is clearly accountable.
Another change one could make is develop bigger standard libraries with all the utilities which are useful.
For example in Rust there are a few de facto standard packages one needs very often, which then also force you to pull in a bunch of transitive dependencies.
Those could also be part of the standard library.
This all amounts to increasing the minimal scope of useful functionality a package has to have to be admitted and increasing accountability of the people maintaining them. This obviously comes with more effort on the maintainers part, but hey maybe we could even pay them for their labor.
SQLite had 6 CVE's last year. FreeType had an RCE bug published last year. libcurl's last CVE reported was 2 weeks ago. Libpng had 4 vulnerabilities published this year....
I'd really like to see package managers organized around rings where a very small core of incredibly important stuff is kept in ring 0, ring 1 gets a slightly wider amount of stuff and can only depend on ring 0 dependencies and then ring 2+ is the crapware libraries that infect most ecosystems.
But maybe that's not the right fit either. The world where package managers are just open to whatever needs to die. It's no longer a safe model.
The OS distro model is actually the right one here. Upstream authors hate it, but having a layer that's responsible for picking versions out of the ecosystem and compiling an internally consistent grouping of known mutually-compatible versions that you can subscribe to means that a lot of the random churn just falls away. Once you've got that layer, you only need to be aware of security problems in the specific versions you care about, you can specifically patch only them, and you've got a distribution channel for the fixes where it's far more feasible to say "just auto-apply anything that comes via this route".
That model effectively becomes your ring 1. Ring 0 is the stdlib and the package manager itself, and - because you would always need to be able to step outside the distribution for either freshness or "that's not been picked up by the distro yet" reasons - the ecosystem package repositories are the wild west ring 2.
In the language ecosystems I'm only aware of Quicklisp/Ultralisp and Haskell's Stackage that work like this. Everything else is effectively a rolling distro that hasn't realised that's what it is yet.
In practice, "ring 0" is whatever gets merged into your language's standard library. Node and python both have pretty expansive standard libraries at this point, stepping outside of those is a choice
Rust libraries are infrequently used outside of Rust because if you have the option, you'd just use Rust, not the ancient featureless language intrinsically responsible for 70% of all security issues. C libraries are infrequently used in Rust outside of system libc, for the same reason; I go and toggle the reqwest switch to use rustls every time, because OpenSSL is horrendous. This is also why you say 'rarely' instead of 'never', when a few years ago it was 'never'; a few years from now you'll say 'uncommonly', and so on. The reason C libraries are used is because you don't feel like reimplementing it yourself, and they are there; but that doesn't apply more to C libraries than Rust libraries, and the vast majority of crates.io wouldn't be usefully represented in C anyway, or would take longer to bind to than to rewrite. (No, nobody uses libcurl.) Finally, this only happens in NPM, and the Rust libraries you pull in are all high-quality. So this sounds like a bunch of handwaving about nonsense.
Rust is terrible for pulling in hundreds of dependencies though. Add tokio as a dependency and you'll get well over 100 packages added to your project.
Even side stepping that tokio no longer pulls multiple packages, it used to be split into multiple packages, in the same way that KDE in Rust would be hundreds of packages.
Rust projects tend to take their project and split it into many smaller packages, for ease of development, faster compiles through parallelization, ensuring proper splitting of concerns, and allowing code reuse by others. But the packages are equivalent to a single big package. The people that write it are the same. They get developed in tandem and published at the same time. You can take a look at the del tree for ripgrep, and the split of different parts of that app allows me to reuse the regex engine without dealing with APIs that only make sense in the context of a CLI app or pulling in code I won't ever use (which might be hiding an exploit too).
Counting 100 100 line long crates all by the same authors as inherently more dangerous than 1 10000 line long crate makes no sense to me.
pin-project-lite is the only base dependency, which itself has no dependencies. If you enable the "full" feature, ie all optional doodads turned on (which you likely don't need), it's 17: bytes, cfg-if, errno, libc, mio, parking_lot+parking_lot_core+lock_api, pin-project-lite, proc_macro2+quote+syn+unicode-ident, scopeguard, signal-hook-registry, smallvec, and socket2. You let me know which ones you think are bloat that it should reimplement or bind to a C library about, and without the blatant fabrication this time.
> We have libraries like SQLite, which is a single .c file that you drag into your project
You are just swapping a package manager with security by obscurity by copy pasting code into your project. It is arguably a much worse way of handling supply chain security, as now there is no way to audit your dependencies.
> If you get rid of transitive dependencies, you get rid of the need of a package manager
This argument makes no sense. Obviously reducing the amount of transitive dependencies is almost always a good thing, but it doesn't change the fundamental benefits of a package manager.
> There's so many C libraries like this
The language with the most fundamental and dangerous ways of handling memory, the language that is constantly in the news for numerous security problems even in massively popular libraries such as OpenSSL? Yes, definitely copy-paste that code in, surely nothing can go wrong.
> They also bindings for every language under the sun. Rust libraries are very rarely used outside of Rust
This is a WILD assumption, doing C-style bindings is actually quite common. YOu will of course then also be exposing a memory unsafe interface, as that is what you get with C.
What exactly is your argument here? It feels like what you are trying to say is that we should just stop doing JS and instead all make C programs that copy paste massive libraries because that is somhow 'high quality'.
This seems like a massively uninformed, one-sided and frankly ridiculous take.
> You are just swapping a package manager with security by obscurity by copy pasting code into your project
You should try writing code, and not relying on libraries for everything, it may change how you look at programming and actually ground your opinions in reality. I'm staring at company's vendor/ folder. It has ~15 libraries, all but one of which operate on trusted input (game assets).
> fundamental benefits of a package manager.
I literally told you why they don't matter if you write code in a sane way.
> doing C-style bindings is actually quite common
I know bindings for Rust libraries exist. Read the literal words you quoted. "Rust libraries are very rarely used outside of Rust". Got some counterexamples?
Gamedev is its own weird thing, and isn't a model you want to generalize to other industries. It has to optimize for things a lot of software does not, and that skews development.
Vendoring libraries is almost always a terrible idea because it immediately starts to bitrot and become a footgun.
Sometimes it's necessary, but it's not desirable, and you almost always just want to pin your dependencies instead.
Then you've got ecosystems like Clojure where many projects are just considered done and used by many. You can pin these (and be warned if a new version still comes out, say for an actual security fix). There are Clojure projects so stable, without any know exploit (we're certainly not talking about daily npm exploits here), that haven't been updated in years because they are... Done. Simply done. Perfection.
I think you can do copy paste in most languages. But it will be a pain to update when there are improvements / security fixes.
You got a project with 1-2 depencies? Sure. But if you need to bring in 100 different libs (because you bring in 10 libs which in turn brings in 10 libs) good luck.
> But if you need to bring in 100 different libs (because you bring in 10 libs which in turn brings in 10 libs
So don’t?
With manual deps management, everyone soon gravitates to a core set of deps. And libraries developer tends to reduce their deps needs, That’s why you see most C libraries deals with file formats, protocols, and broad concerns. Smaller algorithms can be shared with gists and blog articles.
Package managers are older than some users here. From CPAN/CTAN to ports under BSD's.
Some pm's are badly maintained (Pip/NPM), while others are curated enough.
Again, if you have GNU/Linux installed, install Guix, read the Info manual on 'guix import' and just create a shell/container with 'guix shell --container' (and a manifest package created from guix import) and use any crap you need for NPM in a reproducible and isolated way. You $HOME will be safe, for sure.
Setting min-release age to 7 days is great, but the only true way to protect from supply chain attacks is restricting network access.
This needs to be done (as we've seen from these recent attacks) in your devenv, ci/cd and prod environments. Not one, or two, but all of these environments.
The easiest way is via using something like kubernetes network policies + a squid proxy to allow limited trusted domains through, and those domains must not be publicly controllable by attackers. ie. github.com is not safe to allow, but raw.githubusercontent.com would be as it doesn't allow data to be submitted to it.
Network firewalls that perform SSL interception and restrict DNS queries are an option also, though more complicated or expensive than the above.
This stops both DNS exfil and HTTP exfil. For your devenv, software like Little Snitch may protect your from these (I'm not 100% on DNS exfil here though). Otherwise run your devenv (ie vscode) as a web server, or containerised + vnc, a VM, etc, with the same restrictions.
> Setting min-release age to 7 days is great, but the only true way to protect from supply chain attacks is restricting network access.
Getting zero day patches 7 days later if no proper monitoring about important patches or if this specific patch is not in the important list. Always a tradeoff.
Thats true. Setting to 7 days saves you from a supply chain attack, but opens you to zero days. Another example why network filtering is a better solution.
> but raw.githubusercontent.com would be as it doesn't allow data to be submitted to it
But raw.githubusercontent.com still contains code and now the attacker can publish the code he wants no!?
Don't get me wrong: I love the idea to secure as much as possible. I'm running VMs and containerizing and I eat firewalling rules for breakfast, my own unbound DNS with hundreds of thousands (if not millions) of domains blocked, etc. I'm not the "YOLO" kind of guy.
But I don't understand what's that different between raw.githubusercontent.com and github.com? Is it for exploits that are not directly in the source code? Can you explain a bit more?
There's a package manager discussion, but the bit that stands out to me is that this started with a credential compromise. At some point when a project gets big enough like axios, maybe the community could chip in to buy the authors a couple of YubiHSM or similar. I wish that _important keys live in hardware_ becomes more standard given the stakes.
Dealing with dependencies is another question; if it's stupid stuff like leftpad then it should be either vendored in or promoted to be a language feature anyway (as it has been).
> At some point when a project gets big enough like axios, maybe the community could chip in to buy the authors a couple of YubiHSM or similar
I kind of feel like the authors here should want that for themselves, before the community would even realize it's needed. I can't say I've worked on packages that are as popular as axios, but once some packages we were publishing hit 10K downloads or so, we all agreed that we needed to up our security posture, and we all got hardware keys for 2FA and spent 1-2 weeks on making sure it was as bullet-proof we could make it.
To be fair, most FOSS is developed by volunteers so I understand not wanting to spend any money on something you provide for free, but on the other hand, I personally wouldn't feel comfortable being responsible for something that popular without hardening my own setup as much as I could, even if it means stopping everything for a week.
Also, considering how prevalent TPM/Secure Enclaves are on modern devices, I would guess most package maintainers already have hardware capable of generating/using signing keys that never leave hardware.
I think it is mostly a devex/workflow question.
Considering the recent ci/cd-pipeline compromises, I think it would make sense to make a two phase commit process required for popular packages. Build and upload to the registry from a pipeline, but require a signature from a hardware resident key before making the package available.
Most of axios' functionality has effectively been promoted to a language feature as `fetch`, but the problem is people don't bother to migrate. I've migrated our direct usage of it but it's still pulled in transitively in several parts of our codebase.
Even left-pad is still getting 1.6 million weekly downloads.
Annoyingly, the times I reach for axios and similar is when I need to keep track of upload progress, which I could only do with XMLHttpRequest, not fetch, unless I've missed some recent browser changes, and the API of XMLHttpRequest remains as poor as the first times I had to use it. Download progress been supported by fetch since you can track chunks yourself, but somehow they didn't think to do that for requests for some reason, only responses.
Or those people can (fund) separate repackaging and redistribution with more stringent and formalized review process.
Maybe not all users should pull all packages straight from what devs are pushing.
There's no reason we can't have "node package distributions" like we have Linux distributions. Maybe we should stop expecting devs and maintainers and Microsoft to take responsibility for our supply-chain.
>maybe the community could chip in to buy the authors a couple of YubiHSM
There's no community, the users of axios are devs that looked at stackoverflow for "how to download a file in javascript", they barely know or care what axios is.
Now the users of axios are devs that ask Claude Code or Codex to scrape a website or make a dashboard, they don't even know about the word axios.
I personally had to delete axios a couple of time from my codebase when working with junior devs.
This may not be popular, but is there a place for required human actions or just timed actions to slow down things like this? For instance, maybe a GH action to deploy requires a final human click and to change that to cli has a 3 day cooling period with mandatory security emails sent out. Similarly, you switch to read only for 6 hrs after an email change. There are holes in these ideas but the basic concept is to treat security more like physical security, your goal isn't always to 100% block but instead to slow an attacker for xxx minutes to give the rest of the team time to figure out what is going on.
Hi, security here. We've tried, but the amount of people you need for this vs the amount of people you have trying to review and click the big button always means that this step will be a bottleneck. Thus this step will be eliminated.
A much better approach would be to pin the versions used and do intentional updates some time after release, say a sprint after.
Pinning, escrowing, and trailing all help, but I'm not sure "this step will be eliminated" is inevitable.
Package manager ecosystems are highly centralized. npm.org could require MFA (or rate limit, or email verification, or whatever) and most packagers would gripe but go along with this. A minority would look for npm competitors that didn't have this requirement, and another minority would hack/automate MFA and remove the added security, but the majority of folks would benefit from a centralized requirement of this sort.
Yeah, I am looking at that on the use end. It sounds like on the python side this type of thing will be more standard (uv now and soon pip supported with version date requirements). I think time is a big missing element in many security in depth decisions. It can be time until you adopt like use no package newer than xx days or time it takes to deploy etc etc. Unfortunately the ecosystem is getting really diverse and that means ever more sophisticated attacks so we may need to do things that are annoying just to survive.
Why not just release escrow? If I try to push a new release version another developer or developers have to agree to that release. In larger projects you would expect the release to be coordinated or scheduled anyways. Effectively we're just moving "version pinning" or "version delay" one layer up the release chain.
Supply chain attacks are so scary that I think most companies are going to use agents to hard fork their own versions of a lot of these core libraries instead. It wasn’t practical before. It’s definitely much more doable today.
I was thinking about this as a bull case for human developers. Seems if you're worried enough to do this you're not going to have LLMs write the new code.
Large companies already maintain a clone of their packages. Very large ones actually bundle their own build system (Google Bazil, AWS Brazil). If you want to update a package, you have to fetch the sources and update the internal repository. It slows down the opportunities for a supply chain attack down to a crawl.
I know there is a cooldown period for npm packages, but I’m beginning to want a cooldown for domains too. According to socket, the C2 server is sfrclak[.]com, which was registered in the last 24 hours.
There are so many scanners these days these things get caught pretty quick. I think we need either npm or someone else to have a registry that only lets through packages that pass these scanners. Can even do the virustotal thing of aggregating reports by multiple scanners. NPM publishes attestation for trusted build environments. Google has oss-rebuild.
All it takes is an `npm config set` to switch registries anyways. The hard part is having a central party that is able to convince all the various security companies to collaborate rather than having dozens of different registries each from each company.
Rather than just a hard-coded delay, I think having policies on what checks must pass first makes sense with overrides for when CVEs show up.
The ones you hear about are caught quickly, I’m more worried about the non obvious ones. So far none of these have been as simple as changing a true to a false and bypassing all auth for all products or something, and would that be caught by an automated scanner?
There are definitely levels to this. Yes I think it can be caught by automated scanners in theory. Either commit by commit scanning and reproducible builds or fuzzing and getting the behavioral differences between versions
Many of the suggestions in this thread (min-release, ignore script) are defenses for the consumers.
I've been working on Proof of Resilience, a set of 4 metrics for OSS, and using that as a scoring oracle for what to fund.
Popularity metrics like downloads, stars, etc are easy to fake today with ai agents. An interesting property is that gaming these metrics produces better code, not worse.
These are the 4 metrics:
1. Build determinism - does the published artifact match a reproducible build from source?
2. Fuzzing survival - does the package survive fuzz testing?
3. Downstream stability - does it break any repos dependent on this project when pushing a release?
4. Patch velocity - how fast are fixes merged?
Here's a link to the post, still early but would appreciate any feedback.
Carl, with all due respect, have you used AI for making this hackmd post?
"it's not just a waste of money — it's a security problem"
I am really passionate about these things, but I am not going to read something which you haven't written. Even sharing a prompt/rough-sketches/raw-writing might be beneficial but I recommend writing it by-hand man, we are all burnt out reading AI slop, I can't read more AI
You're right, I used an LLM to help write it from sketches. Gonna rewrite it properly because I think the ideas are worth exploring.
Thanks for taking the time to read and reply.
In my opinion, its okay to use LLM to help find some sources and then validating them (but I also recommend using hand researching too as you might find some good things that you maybe weren't even looking for!)
but, please don't use LLM to help write it from sketches. Even show the sketch :)
Much of my writing is very sketch-y. Some people don't like it, but its mine and I am proud of it and I hope that even if you write sketches/refine them, you can be comfortable sharing your ideas in your words in the way you wish to write them carl!
My thinking is that, I improve my writing by well... practice itself. So I write publically and there are some thoughts which occur in my head during the writing process itself (PG has a good article about it recently)
In a world of AI, to me, Human writing is a breath of fresh air. Please don't fall into the rabbit-hole that you might need LLM to help write you.
These are just my 2 cents though, but I feel like I am definitely not alone in thinking so.
Have a nice day and I am looking forward for you to write the article yourself. Feel free to share me when you do with my mail as I would love to read it, as I am also passionate about the funding of open source :)
The frustrating thing here is that axios versions display on npmjs with verified provenance. But they don’t use trusted publishing: https://github.com/axios/axios/issues/7055 - meaning the publish token can be stolen.
I wrongly thought that the verified provenance UI showed a package has a trusted publishing pipeline, but seems it’s orthogonal.
NPM really needs to move away from these secrets that can be stolen.
Of course the "wait 7 days" are not a silver bullet, but it gives automated scanners plenty of time to do their work. Those automated scanners surely catch this `eval(base64.decode("..."))` stuff that some of those attacks used so in my book this dependency cooldown is a net win. I guess the skilled malicious actors will then up their game but I think it's okay to kick off an arms race between them and the security scanners in the dependency world.
That's a good point. In some level I'd prefer the delay to happen on publication of the package itself. Do any of these scanners have cryptographic attestations or similar?
What a great time to be alive! Now, that's exactly why I enjoy writing software with minimal dependencies for myself (and sometimes for my family and friends) in my spare time - first, it's fun, and second, turns out it's more secure.
This only limits the possibility of compromise, it doesn't remove it. Python itself could be compromised, or the package that your linux distro provides could be.
With AI agents the volume and frequency of supply chain attacks is going to explode. I think our entire notion of how to develop and distribute software safely needs to change. I don't have answers; "reflections on trusting trust" explains the difficulties we now face.
Ran npm ci --ignore-scripts in our CI for months but never thought about local dev. Turns out that's the gap, your CI is safe but your laptop runs postinstall on every npm install.
The anti-forensics here are much more complicated that I had imagined. Sahring after getting my hands burned.
After the RAT deploys, setup.js deletes itself and swaps package.json with a clean stub. Your node_modules looks fine. Only way to know is checking for artifacts: /Library/Caches/com.apple.act.mond on mac, %PROGRAMDATA%\wt.exe on windows, /tmp/ld.py on linux. Or grep network logs for sfrclak.com.
Somehow noboady is worried about how agentic coding tools run npm install autonomously. No human in the loop to notice a weird new transitive dep. That attack surface is just getting worsened day by day.
At this point picking Node for a backend is a foot gun. Large companies have the funds for private, security vetted npm repositories, but what about small companies, startups, personal projects? Pnpm makes things more secure without install scripts, mininum package time, but it's still the same activity, does an extra parachute make skydiving any less inherently dangerous?
I'm not dogmatic about the whole "JS for the backend is sin" from backend folks, but it seems like it was the right call. You should stick to large org backed packages, or languages with good enough standard libraries, like Go, Java, Python, C#.
This is just going to get worse and worse as agentic coding gets better. I think having a big dependency tree may be a thing of the past in the coming years. Seems like eventually new malware will be coming out so fast it will basically be impossible to stop.
How is it we've made it this far and we still don't have any kind of independent auditing of basic publish security on NPM? You'd think this would be collectively a trivial and high priority task (to ensure that all publishes for packages over a certain download volume are going through a session that authenticated via MFA, for instance).
> You'd think this would be collectively a trivial and high priority task (to ensure that all publishes for packages over a certain download volume are going through a session that authenticated via MFA, for instance).
Because all mainstream packages are published via CI/CD pipeline not by an MFA'd individual uploading a GZIP to npm.com
Requiring a human-in-the-loop for final, non-prerelease publication doesn't seem like that onerous of a burden. Even if you're publishing multiple releases a day on the regular (in which case ... I have questions, but anyway) there are all sorts of automations that stay secure while reducing the burden of having to manually download an artifact from CI, enter MFA, and upload it by hand.
You can still have a step that requires a certain user/group to sign off, and you can still enforce that those users have MFA set up. Almost any serious shop that expects to pass audits already does this in some form or fashion before pushing code to prod.
I am now migrating all my unencrypted secrets on my machines to encrypted ones. If a tool supports scripted credential providers (e.g. aws-cli or Ansible), I use that feature. Otherwise, I wrap the executable with a script that runs gpg --decrypt and injects an environment variable.
That way, I can at least limit the blast radius when (not if) I catch an infostealer.
Axios has a long history, and is included in a lot of code, also in indirect dependencies. Just check its npm page: it has 174025 dependents as of this moment, including a lot of new packages (I see openclaw and mcp related packages in the list).
And with LLMs generating more and more code, the risk of copying old setups increases.
In case it helps, we open-sourced a tool to audit dependencies for this kind of supply-chain issue. The motivation was that there is a real gap between classic “known vulnerability” scanning and packages whose behavior has simply turned suspicious or malicious. We also use AI to analyze code and dependency changes for more novel or generic malicious behavior that traditional scanners often miss.
I'm working on a multi signature solution that helps to detect unauthorized releases in the case of an account hijack. It is open source, self hostable, accountless and I am looking for feedback!
I am glad I don't need to touch JS or web dev at all.
Now, I tend to use Python, Rust and Julia. With Python I am constantly using few same packages like numpy and matplotlib. With Rust and Julia, I try as much as possible to not use any packages at all, because it always scares me when something that should be pretty simple downloads half of the Internet to my PC.
Julia is even worse than Rust in that regard - for even rudimentary stuff like static arrays or properly namespaced enums people download 3rd party packages.
Isn't Rust just as susceptible to this issue? For example, how do you deal with Rust's lack of support for HTTP in the standard library? Importing hyper pulls in a couple dozen transitive libraries which exposes you to the exact same kind of threats that compromised axios.
Given how HTTP is now what TCP was during the 90s and almost all modern networked applications needing to communicate in it one way or another, most rust projects come with an inherent security risk.
These days, I score the usability of programming languages by how complete their standard library is. By that measure, Rust and Javascript get an automatic F.
It is, therefore I have stated I avoid any dependencies while writing Rust, unless they are self-contained. And I said I am glad I don't do web, so I don't have need for HTTP implementations.
All these supply chain attacks make me nervous about the apps I use. It would be valuable info if an app used such dependencies, but on the other hand, programmers would cut their sales if they gave you this info.
Min release age sucks, but we’ve been here before. Email attachments used to just run wild too, then everyone added quarantine delays and file blocking and other frictions... and it eventually kinda/sorta worked. This does feel worse, though, with fewer chokepoints and execution as a natural part of the expectation.
Edit: bottom line is installs are gonna get SOOO much more complicated. You can already see the solution surface... Cooling periods, maintainer profiling, sandbox detonation, lockfile diffing, weird publish path checks. All adds up to one giant PITA for fast easy dev.
Min release age might just postpone vulnerability to be applied few days later in non trivial cases like this. More I think about it, Odin lang approach of no package manager makes senses. But, for that approach won't work for Javascript as it needs npm package even for trivial things. Even vendoring approach like golang won't work with Javascript with the amount of churn and dependencies.
> Both versions were published using the compromised npm credentials of a lead axios maintainer, bypassing the project's normal GitHub Actions CI/CD pipeline.
Doesn’t npm mandate 2FA as of some time last year? How was that bypassed?
With all the recent supply chain attacks, I'm starting to think it's only a matter of time before all of us are victims. I think this is a sign to manually check all package diffs or postinstall scripts.
To have an initial smoke test, why not run a diff between version upgrades, and potentially let an llm summarise the changes? It’s a baffling practice that a lot of developers are just blindly trusting code repos to keep the security standards. Last time I installed some npm package (in a container) it loaded 521 dependencies and my heart rate jumped a bit
Is this the first time you have ever thought about the idea of supply chain attacks? This is the first thought 90% of people have and it doesn't work. Too much work to manually verify diffs and LLMs aren't good enough at this yet.
No, I think about it all the time. It’s just baffling that this kind of attack is still a thing, after a decade+ of this happening over and over again.
We're going to see this more and more and more. And it's not going to stop. Because nobody in the industry will use the simplest, industry-standard security practices. Because they don't feel like it. A software building code is the only thing that'll fix it.
This is exactly to avoid this kind of issue that I decided to work on StableBuild.
StableBuild pins and hosts a copy of your dependencies at a specific freeze date, so that your supply chain is never contaminated.
This way, a compromised version published after your freeze date (even with the same version number!) would never reach your build.
> This was not opportunistic. It was precision. The malicious dependency was staged 18 hours in advance.
Another obvious ChatGPT-ism. The fact that people are using AI to write these security posts doesn't surprise me, but the fact they use it to write a verbose article with spicy little snippets that LLMs seem to prefer does make it really hard to appreciate anything other than the simple facts in the article.
I believe compartmentalized operating systems like Qubes are the future for defending against these kinds of attacks.
Storing your sensitive data on a single bare-metal OS that constantly downloads and runs packages from unknown maintainers is like handing your house key out to a million people and hoping none of them misuse it.
Ok it's bad, but our npm projects are pinned in the package-lock.json, which I imagine most would be? So who would pull this besides security scanners?
I don't think that's right if it's in your package-lock it wouldn't pull it unless you npm update axios, or delete the package-lock.json and then npm install.
I'm assuming you are talking about agents like claude-code and open-code which rely on GPT functions (AKA Large Language Models).
The reason they don't detect these risks is primarily because these risks are emergent, and happen overnight (literally in the case of axios - compromised at night). Axios has a good reputation. It is by definition impossible for a pre-trained LLM to keep up with time-sensitive changes.
I mean that agents can scan the code to find anything "suspicious". After all, security vendors that claim to "detect" malware in packages are relying on LLMs for detection.
An LLM is not a suitable substitute for purpose-built SAST software in my opinion. In my experience, they are great at looking at logs, error messages, sifting through test output, and that sort of thing. But I don't think they're going to be too reliable at detecting malware via static analysis. They just aren't built for that.
Do we have a way yet to tell if something on our system is compromised? There's plenty of end user software built on node, like Gemini CLI and LM Studio.
So the root cause was again a developer's opsec. For improving things, I haven't seen many new initiatives on that side (beyond 2FA, but even that seems unenforced in these repositories, I reckon).
The attacker went through the hassle to compromise a very widely used package, but use a non standard port (8000) on their C2...
If you plan to do something like that, use 443 at least, many corporate network do not filter this one ;)
I used axios in the distant past but haven't used it whenever I had my say in the past five years. You don't need it, and for special things like retries I could roll my own just fine. Now ai will roll it for you
This is the part that's tough — we push everyone to keep dependencies updated and automate it with Renovate or Dependabot, but that's exactly the pipeline that would have pulled this in before anyone noticed. Lockfiles and pinning help slow it down, but most teams pair those with automated update PRs which kind of negates the point. You can reduce your dependency surface area to lower the odds but one compromised maintainer on a top-10 package and none of that matters.
Is this Jia Tan 5.0? I've lost count. You really should stop trusting packages (implicitly). Or don't. It's your funeral, not mine. See you at Jia Tan 6.0 April?
Not at all, it was a regular maintainer account that was hijacked (probably through phishing) and used to push a malicious payload, not a threat actor posing as a contributor and adding a backdoor like in the Jia Tan case.
I'd argue this has not much in common with Jia Tan apart from both being supply chain attacks, there is no malicious maintainer here, a trusted maintainer had their account taken over.
I guess the end result is the same, a malicious package pushed by an account that was thought to be trusted, but I think the Jia Tan case is worth being looked at differently than just simple account takeover.
It's just a longer backstory. All the same in the end. Hackers targeted a popular package. The lead maintainer was compromised. The pattern fits. There will be more of these.
In light of these nonstop supply chain attacks:
Tonight I created /supply-chain-audit
-- A simple claude code skill that fetches info on the latest major package vulnerability, then scans your entire ~/ and gives you a report on all your projects.
NPM should learn from Linux distribution package managers.
Have a branch called testing, and packages stay in testing for few weeks, after which they go to stable. That is how many Linux distributions handle packages. It would have prevented many of these.
Advising every user of npm/pnpm to change their settings and set their own cooldown periods is not a real choice.
Not all distributions work with a staging repository, and it's not really intended for this purpose either.
Besides there's always a way to immediately push a new version to stable repositories. You have to in order to deal with regressions and security fixes.
I know not all, but Debian/Ubuntu/Fedora does, and while the intended purpose of multi-stage releases is not necessarily security but stability, it still does help up with security too. Because third parties can look and scan the dependencies while they are still not in stable.
Most of the supply chain vulnerabilities that ended up in the NPM would have been mitigated with having mandatory testing / stable branches, of course there needs to be some sort of way to skip the testing but that would be rather rare and cumbersome and audited, like it is in Linux distributions too.
NPM is one big AUR, where anyone can submit arbitrary unverified code. The difference is that AUR is intentionally harder to use to prevent catastrophic one-line installs.
Is a "AUR" now just how we name unaudited software repositories?
Just to note, if we're talking about Linux Distributions. There's also COPR in Fedora, OBS for OpenSUSE (and a bunch of other stuff, OBS is awesome), Ubuntu has PPAs. And I am sure there's many more similar solutions.
Just sanity checking - if I only ever install axios in a container that has no secrets mounted in to its env, is there any real way I can get pwned by this kind of thing?
I am not saying this is the reason for this compromise but the sudden explosion of coding assistant like claude code, and tools like openclaw is teaching entire crop of developers (and users) that it is ok to have sensitive credentials .env files.
I wonder if this has any connection with the recent string of attacks including the FBI director getting hacked. The attack surface is large, executed extremely cleanly - almost as if done by a high profile state sponsored actor, just like in Hollywood movies.
All sorts of reasons, but this isn't a left-pad situation. Axios's functionality is something provided by a library in a lot of languages (C/C++ with libcurl and friends, Python with requests, Rust with reqwest, and so on).
That's not to say it's inherently necessary for it to be a third-party package (Go, Ruby, and Java are counterexamples). But this isn't a proliferation/anemic stdlib issue.
Can we get a non-AI-generated article for this? I think the aikido one might be fine, but if there’s a more official source let’s use that in lieu of this AI nonsense.
The NPM ecosystem is a joke. I don't even want anything to do with it, because my stack is fully Elixir. But, just because of this one dependency that is used in some interfaces within my codebase, I need to go back to all my apps and fix it. Sigh.
JavaScript, its entire ecosystem is just a pack of cards, I swear. What a fucking joke.
This is why Node.js is completely unsuitable as backend.
Until recently, there wasn’t even a standard Promise-based HTTP client. Why should we need to download a library just to make a simple HTTP request? It’s because Node.js’s standard library is too limited, leading to an explosive growth in third-party libraries. As a result, it’s vulnerable to security attacks, and maintaining it in an enterprise environment becomes a major challenge.
Let’s use .NET or Go. Why use JavaScript outside of the browser when there are excellent backend environments out there?
1/5 of your CLI and 1/3 of your gui apps are npm based. Each one has 400+ dependencies , none notable enough to go viral when they are breached. And who knows what other packages are currently compromised. We all have 30+ node_modules on our disks, and 2/3 of them were shipped by outside vendors , packaged in an archive.
“I’m smart I use fetch instead of axios”. “I pin my versions” – sure but certainly one of your npx or Electron apps uses axios or another less notably compromised package.
Absolutely. If you ever did a npm install on a project using one of the affected axios versions, your entire system may be compromised.
> The malicious versions inject a new dependency, plain-crypto-js@4.2.1, which is never imported anywhere in the axios source code. Its sole purpose is to execute a postinstall script that acts as a cross platform remote access trojan (RAT) dropper, targeting macOS, Windows, and Linux. The dropper contacts a live command and control server and delivers platform specific second stage payloads. After execution, the malware deletes itself and replaces its own package.json with a clean version to evade forensic detection.
I lost respect for Axios when they made a breaking change in a patch release. Digging into the root cause, I found the maintainer had approved an outside PR with an obvious AI slop PR description: https://github.com/axios/axios/issues/7059
Looks like the maintainer wasn't just careless when reviewing PRs.
> This creates a secondary deception layer. After infection, running npm list in the project directory will report plain-crypto-js@4.2.0 — because npm list reads the version field from the installed package.json, which now says 4.2.0. An incident responder checking installed packages would see a version number that does not match the malicious 4.2.1 version they were told to look for, potentially leading them to conclude the system was not compromised.
WTF!!!! gaslighting your victims into believing they are not victims. the ingenuity of this is truly mindblowing. I am shocked at such thing is even allowed. like packages should not be able to modify their contents while they are being instaleld.
But also have a regular review of your dependencies to update them when necessary, because as bad as compromised packages may be things do have vulnerabilities occasionally, and upgrading things that are a long way out-of-date can be quite hard.
Just a reminder that you can run most node things with deno run and have opt in permissions, audit trail and even external permission system integration now. The gotcha is that "deno task <<some package.json script>>" will NOT execute with this model which I find extremely unintuitive and had me thinking deno abandoned its sandbox for nodejs compatibility completely.
Guix saves you from this. You can import NPM packages in a container (not even touching $HOME) and giving you a shell on the spot with just the dependencies and nothing more.
Learn about 'guix import'.
Oh, and you can install Guix on any GNU/Linux distro.
It’s only a losing strategy if you assume everyone universally adopts the slow strategy, and no research teams spot it in the interim. For things with large splash radius, that’s unrealistic, so defenders have an information advantage.
Makes actual security patches tougher to roll out though - you need to be vigilant to bypass the slowdown when you’re actually fixing a critical flaw. But nobody said this would be easy!
I have a few projects which rely on npm (and react) and every few months I have to revisit them to do an update and make sure they still build, and I am basically done with npm and the entire ecosystem at this point.
Sure, its convenient to have so much code to use for basic functionality - but the technical debt of having to maintain these projects is just too damn high.
At this point I think that, if I am forced to use javascript or node for a project, I reconsider involvement in that project. Its ecosystem is just so bonkers I can't justify the effort much longer.
There has to be some kind of "code-review-as-a-service" that can be turned on here to catch these things. Its just so unproductive, every single time.
Has anyone tested general purpose malware detection on supply chains ? Like clamscan . I tried to test the LiteLLM hack but the affected packages had been pulled. Windows Defender AV has an inference based detector that may work when signatures have not yet been published
I second this question. I usually scan our containers with snyk and guarddog, and have wondered about guarddog in particular because it adds so much build time.
Hopefully desktop Linux users will start to understand that malware actually does exist for Linux and that their operating system is doing nothing to protect them from getting RATed.
Yet npm isn't using them allowing this RAT to work. It is not secure by default. It requires every app to manually opt in to being secure. This opt in approach to security puts desktop Linux decades behind in regards to security. Not ahead.
Take for example iOS and Android. All apps are sandboxed by default. You can't make a program that just steals all of your credentials like you can on desktop Linux. Having security tools means nothing if they aren't being used.
They aren't because Node haven't developed a Node app for them. Desktop Linux does not offer more security than macOS. macOS has proper security around stuff like apps using the mic and camera.
Skipping Node sounds nice. PyPI and RubyGems have had the same mess, and npm gets more headlines because it is huge and churns fast, so you see more fresh landmines and more people stepping on them. Unless you plan to audit every dep and pin versions yourself, you're mostly trading one supply chain mess for another, with a tiny bit of luck and a differnt logo.
Log4Shell was hardly a supply-chain attack - just a latent bug in a widely-used library. That can happen anywhere.
Maven to this day represents my ideal of package distribution. Immutable versions save so much trouble and I really don't understand why, in the age of left-pad, other people looked at that and said, "nah, I'm good with this."
Completely agree. NPM has the only registry where massive supply chain attacks happen several times a year. Mainly the fault lies with NPM itself, but much of it is just a terrible opsec culture in the community.
Most package.jsons I see have semver operators on every dependency, so patches spread incredibly quickly. Package namespacing is not enforced, so there is no way of knowing who the maintainer is without looking it up on the registry first; for this reason many of the most popular packages are basically side projects maintained by a single developer*. Post-install scripts are enabled by default unless you use pnpm or bun.
When you combine all these factors, you get the absolute disaster of an ecosystem that NPM is.
*Not really the case for Axios as they are at least somewhat organized and financed via sponsors.
The semantics are irrelevant. The effect is what's important: Hijacking widely used software to exploit systems. The OC is somehow under the illusion that avoiding JS altogether is a silver bullet for avoiding this.
The semantics are very relevant, since you presented it as a supply-chain attack. If you call a library vulnerability a supply-chain attack, then your argument has lost coherence.
> The OC is somehow under the illusion...
Avoiding package managers with shitty policies is the silver bullet for this attack vector. I get that it can be useful in the moment to retract published artifacts, or update them in-place, or run some code after your artifact is downloaded, but all of these are false economies in our hostile environment.
Other languages have package managers (perl) and there are package managers in existence that are not so vulnerable to this issue. IMO, it stems from one place: Transitive dependencies and general opaqueness of the issue.
In package managers like pacman, apt, apk,... it's easier to catch such issue. They do have postinstall scripts, but it's part of the submission to the repo, not part of the project. Whatever comes from the project is hashed, and that hash is also visible as part of the submission. That makes it a bit difficult to sneak something. You don't push a change, they pull yours.
The issues have everything to do with npm as a platform and nothing with JS as a language. You can use JS without npm. Saying you'll escape supply chain attacks by not using JS is like saying you'll be saved from an car crash with a parachute.
"Batteries included" ecosystems are the only persistent solution to the package manager problem.
If your first party tooling contains all the functionality you typically need, it's possible you can be productive with zero 3rd party dependencies. In practice you will tend to have a few, but you won't be vendoring out critical things like HTTP, TCP, JSON, string sanitation, cryptography. These are beacons for attackers. Everything depends on this stuff so the motivation for attacking these common surfaces is high.
I can literally count on one hand the number of 3rd party dependencies I've used in the last year. Dapper is the only regular thing I can come up with. Sometimes ScottPlot. Both of my SQL providers (MSSQL and SQLite) are first party as well. This is a major reason why they're the only sql providers I use.
Maybe I am just so traumatized from compliance and auditing in regulated software business, but this feels like a happier way to build software too. My tools tend to stay right where I left them the previous day. I don't have to worry about my hammer or screw drivers stealing all my bitcoin in the middle of the night.
There are several issues with "Batteries Included" ecosystems (like Python, C#/.NET, and Java):
1. They are not going to include everything. This includes things like new file formats.
2. They are going to be out of date whenever a standard changes (HTML, etc.), application changes (e.g. SQLite/PostgreSQL/etc. for SQL/ORM bindings), or API changes (DirectX, Vulcan, etc.).
3. Things like data structures, graphics APIs, etc. will have performance characteristics that may be different to your use case.
4. They can't cover all nice use cases such as the different libraries and frameworks for creating games of different genres.
For example, Python's XML DOM implementation only implements a subset of XPath and doesn't support parsing HTML.
The fact that Python, Java, and .NET have large library ecosystems proves that even if you have a "Batteries Included" approach there will always be other things to add.
"Batteries included" means "ossification is guaranteed", yah. "stdlib is where code goes to die" is a fairly common phrase for a reason.
There's clearly merit to both sides, but personally I think a major underlying cause is that libraries are trusted. Obviously that doesn't match reality. We desperately need a permission system for libraries, it's far harder to sneak stuff in when doing so requires an "adds dangerous permission" change approval.
The goal is not to cover everything, the goal is to cover 90% of the use cases.
For C#, I think they achieved that.
> They are going to be out of date whenever a standard changes (HTML, etc.)
You might want to elaborate on the "etc.", since HTML updates are glacial.
The HTML "Living Standard" is constantly updated [1-6].
The PNG spec [7] has been updated several times in 1996, 1998, 1999, and 2025.
The XPath spec [8] has multiple versions: 1.0 (1999), 2.0 (2007), 3.0 (2014), and 3.1 (2017), with 4.0 in development.
The RDF spec [9] has multiple versions: 1.0 (2004), and 1.1 (2014). Plus the related specs and their associated versions.
The schema.org metadata standard [10] is under active development and is currently on version 30.
[1] https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/... (New)
[2] https://web.dev/baseline/2025 -- popover API, plain text content editable, etc.
[3] https://web.dev/baseline/2024 -- exclusive accordions, declarative shadow root DOM
[4] https://web.dev/baseline/2023 -- inert attribute, lazy loading iframes
[5] https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/... (Baseline 2023)
[6] https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/... (2020)
[7] https://en.wikipedia.org/wiki/PNG
[8] https://en.wikipedia.org/wiki/XPath
[9] https://en.wikipedia.org/wiki/Resource_Description_Framework
[10] https://schema.org/
glaciers change faster than HTML
Oof, I honestly hadn't considered that.
Why would they be out of date? The ecosystems themselves (for example .NET) receives regular updates.
Yes, they cannot include everything, but enough that you do not _need_ third party packages.
Python, .NET, and Java are not examples of batteries included.
Django and Spring
Python's standard library is definitely much more batteries-included than JavaScript's.
comparing to Node, .NET is batteries included: built-in Linq vs needing lodash external package, built-in Decimal vs decimal.js package, built-in model validation vs class-validator & class-transformer packages, built-in CSRF/XSRF protection vs csrf-csrf package, I can go on for a while...
And in fact wasn't a popular Python library just compromised very recently? See https://news.ycombinator.com/item?id=47501426.
So Python's clearly not "batteries included" enough to avoid this kind of risk.
That's my point. You can have a large standard library like those languages I mentioned, but that isn't going to include everything nor cover every use case, so you'll have external libraries (via PyPi for Python, NuGet for .NET, and Maven for Java/JVM).
Batteries included systems are still susceptible to supply chain attacks, they just move slower so it’s not as attractive of a target.
I think packages of a certain size need to be held to higher standards by the repositories. Multiple users should have to approve changes. Maybe enforced scans (though with trivy’s recent compromise that wont be likely any time soon)
Basically anything besides lone developer can decide to send something out on a whim that will run on millions of machines.
While technically true, it's so much slower that it's essentially a different thing. Third party packages being attacked is a near daily occurrence. First party attacks happens on the timescale and frequency of decades.
It's like the difference in protecting your home from burglars and foreign nation soldiers. Both are technically invaders to your home, but the scope is different, and the solutions are different.
> In practice you will tend to have a few, but you won't be vendoring out critical things like HTTP, TCP, JSON, string sanitation, cryptography
Unless you are Python, where the standard library includes multiple HTTP libraries and everyone installs the requests package anyways.
Few languages have good models for evolving their standard library, so you end up with lots of bad designs sticking around forever. Libraries are much easier to evolve, giving them the advantage in terms of developer UX and performance.
What type of developer chooses UX and performance over security? So reckless.
I removed the locks from all the doors, now entering/exiting is 87% faster! After removing all the safety equipment, our vehicles have significantly improved in mileage, acceleration and top speed!
>What type of developer chooses UX and performance over security? So reckless.
Initially I assumed this is sarcastic, but apparently not. UX and performance is what programmers are paid to do! Making sure UX is good is one of the most important things in programmer job.
While security is a moving target, a goal, something that can never be perfect, just "good enough" (if NSA wants to hack you, they will). You make it sound like installing third party packages is basically equivalent to a security hole, while in practice the risk is low, especially if you don't overdo it.
Wild to read extreme security views like that, while at the same time there are people here that run unconstrained AI agents with --dangerous-skip-confirm flags and see nothing wrong with it.
Installing 3rd party packages the way Node and Python devs do regularly _is_ a security hole.
We definitely agree on that. Fortunately some of the 600+ comments here include suggestions of what to do about it.
Even more wild to read that sarcasm about "removing locks from doors for 87% speedup" is considered extreme...
And yes, we agree that running unconstrained AI agents with --dangerous-skip-confirm flags and seeing nothing wrong with it is insane. Kind of like just advertising for burglars to come open your doors for you before you get home - yeah, it's lots faster to get in (and to move about the house with all your stuff gone).
Better developer UX can directly lead to better safety. "You are holding it wrong" is a frequent source of security bugs, and better UX reduces the ways you can hold it wrong, or at least makes you more likely to hold it the right way
> Better developer UX can directly lead to better safety.
Depends. If you had to write add to a Makefile for your dependencies, you sure as hell aren't going to add 5k dependencies manually just to get a function that does $FOO; you'd write it yourself.
Now, with AI in the mix, there's fewer and fewer reasons to use so many dependencies.
Friction is helpful. Putting seatbelts on takes more time than just driving, but it’s way safer for the driver. Current dev practices increase speed, not safety.
"Security" is often more about corporate CYA than improving my actual security as a user, and sometimes in opposition, and there is often blatant disregard for any UX concession at all. The most secure system is fully encrypted with all copies of the encryption key erased.
requests should be in the Python standard library. Hard choices need to be made.
I'm pretty sure it's really one HTTP library: urllib.request is built on top of http.client. But the very Java-inspired API for the former is awful.
> Unless you are Python, where the standard library includes multiple HTTP libraries and everyone installs the requests package anyways.
The amount of time defining same data structures over and over again vs `pip install requests` with well defined data structures.
Personally I've heard Odin [1] to do a decent job with this, at least from what I've superficially learned about its stdlib and included modules as an "outsider" (not a regular user). It appears to have things like support for e.g. image file formats built-in, and new things are somewhat liberally getting added to core if they prove practically useful, since there isn't a package manager in the traditional sense. Here's a blog post by the language author literally named "Package Managers are Evil" [2]
(Please do correct me if this is wrong, again, I don't have the experience myself.)
[1] https://pkg.odin-lang.org/
[2] https://www.gingerbill.org/article/2025/09/08/package-manage...
Irony is that Node has no need for Axios, native fetch support has been there for years, so in terms of network requests it is batteries included.
It doesn't matter. We pulled axios out of our codebase, but it still ends up in there as a child or peer from 40 other dependencies. Many from major vendors like datadog, slack, twilio, nx (in the gcs-cache extension), etc...
People use axios or ky because with fetch you inevitably end up writing a small wrapper on top of it anyway.
Fetch has also lacked support for features that xhr has had for over a decade now. For example upload progress. It's slowly catching up though, upload progress is the only thing I'd choose xhr for.
You can pipe through a TransformStream that counts how many bytes you've uploaded, right?
Some might say the tradeoff of writing a small wrapper is worth it given what’s been demonstrated here.
Yeah but what about other deps like db drivers?
In my experience people feel the need to wrap axios too.
These are the kind of people I hope AI replaces
AI was trained on Axios wrappers, so it's just going to be wrappers all the way down. Look inside any company "API Client" and it's just a branded wrapper around Axios.
Speak for yourself, Claude works fine with fetch on my system.
Node fetch is relatively new. Wasn't marked stable until 2023, though I've used it since like 2018.
I'm not sure fetch is a good server-side API. The typical fetch-based code snippet `fetch(API_URL).then(r => r.json())` has no response body size limit and can potentially bring down a server due to memory exhaustion if the endpoint at API_URL malfunctions for some reason. Fine in the browser but to me it should be a no-no on the server.
> I'm not sure fetch is a good server-side API. The typical fetch-based code snippet `fetch(API_URL).then(r => r.json())` has no response body size limit and can potentially bring down a server due to memory exhaustion if the endpoint at API_URL malfunctions for some reason. Fine in the browser but to me it should be a no-no on the server.
Nor is fetch a good client-side API either; you want progress indicators, on both upload and download. Fetch is a poor API all-round.
Hm, I don't think axios would do much better here. `fetch` is the official replacement for axios. If both are flawed that's another topic
Axios has maxContentLength and maxBodyLength options. I would probably go with undici nowadays though (it also has maxResponseSize).
> `fetch` is the official replacement for axios.
No. Axios is still maintained. They have not deprecated the project in favor of fetch.
I'm not saying that axios is unmaintained, I'm saying that if you want something like axios from the standard lib, fetch is the closest thing you get to official
It's not deprecated, it's obsoleted.
It doesn't have a need _now_. Axios is more than 10 years old now, and even before axios other libraries did the same utility of making requests easier
Browsers too.
It’s not needed anymore.
The other thing that keeps coming up is the github-code-is-fine-but-the-release-artifact-is-a-trojan issue. It really makes me question if "packages" should even exist in JavaScript, or if we could just be importing standard plain source code from a git repo.
I understand why this doesn't work well with legacy projects, but it's something that the language could strive towards.
This might make things worse not better.
Yes - the postinstall hook attack vector goes away. You can do SHA pinning since Git's content addressing means that SHA is the hash of the content. But then your "lockfile" equivalent is just... a list of commit SHAs scattered across import statements in your source? Managing that across a real dependency tree becomes a nightmare.
This is basically what Deno's import maps tried to solve, and what they ended up with looked a lot like a package registry again.
At least npm packages have checksums and a registry that can yank things.
You can just git submodule in the dependencies. Super easy. Also makes it straightforward to develop patches to send upstream from within your project. Or to replace a dependency with a private fork.
In my experience, this works great for libraries internal to an organization (UI components, custom file formats, API type definitions, etc.). I don't see why it wouldn't also work for managing public dependencies.
Plus it's ecosystem-agnostic. Git submodules work just as well for JS as they do for Go, sample data/binary assets, or whatever other dependencies you need to manage.
> But then your "lockfile" equivalent is just... a list of commit SHAs scattered across import statements in your source? Managing that across a real dependency tree becomes a nightmare.
The irony is that this is actually the current best practice to defend against supply chain attacks in the github actions layer. Pin all actions versions to a hash. There's an entire secondary set of dev tools for converting GHA version numbers to hashes
> I understand why this doesn't work well with legacy projects, but it's something that the language could strive towards.
Why wouldn't that work well with legacy projects? In fact, the projects I was a part of that I'd call legacy nowadays, was in fact built by copy-and-pasting .js libraries into a "vendor/" directory, and that's how we shipped it as well, this was in the days before Bower (which was the npm of frontend development back in the day), vendoring JS libs was standard practice, before package managers became used in frontend development too.
Not sure why it wouldn't work, JavaScript is a very moldable language, you can make most things work one way or another :)(
or you don't use a package manager where anyone can just publish a package (i.e. use your system package manager). There is still some risk, but it is much smaller. Like, if xz were distributed by PyPI or NPM, everyone would have been pwned, but instead it was (barely) found.
It's true that system repos doesn't include everything, but you can create your own repositories if you really need to for a few things. In practice Fedora/EPEL are basically sufficient for my needs. Right now I'm deploying something with yocto, which is a bit more limited in slection, but it's pretty easy to add my own packages and it at least has hashes so things don't get replaced without me noticing (to be fair, I don't know if the security practices of open-embedded recipes are as strong as Fedora...).
it's muddying what a package is. A package, or a distro, is the people who slave and labor over packaging, reviewing, deciding on versions to ship, having policies in place, security mailing lists, release schedules, etc.
just shipping from npm crap is essentially the equivelant of running your production code base against Arch AUR pkgbuilds.
Fully agree with this! I think today .NET is probably the most batteries included platform you can get. This means that even if you use third-party libraries, these typically depend only on first-party dependencies, making it much less likely for something shady to sneak in.
Kinda.
With Bun I use less dependencies from NPM than I used from Nuget with .NET to build minimal apis. For example the pg driver.
Why is .NET more "batteries included" than Java?
With the notable exception of cross-platform audio.
Not really notable, aiui the only mainstream language with anything like that is JS in the browser
And for good reason. There are enough platform differences that you have to write your own code on top anyway.
To me, I really like Golang's batteries included platform. I am not sure about .NET though
C#'s LINQ (code as data, like LISP) wins over golang for any type of data access. Strongly-typed, language-native queries. Go has its own advantages though.
EF is amazing
And now with NativeAOT, you can use C# like go - you don't need to ship the CLR.
This is a rather superlative and tunnel vision, "everything is a nail because I'm a hammer" approach. The truth is this is an exceedingly difficult problem nobody has adequately solved yet.
I think the AI tooling is, if not completely solving sandboxing, at least making the default much better by asking you every time they want to do something and providing files to auto-approve certain actions.
Package managers should do the same thing
Another layer of AI tooling is the cost of spinning up your own version of some libraries is lowered and can be made hyper specific to your needs rather than pulling in a whole library with features you'll never use.
> Another layer of AI tooling is the cost of spinning up your own version of some libraries is lowered and can be made hyper specific to your needs rather than pulling in a whole library with features you'll never use.
Tell me about it. Using AI Chatbots (not even agents), I got a MVP of a packaging system[1] to my liking (to create packages for a proprietary ERP system) and an endpoint-API-testing tool, neither of which require a venv or similar to run.
------------------------------
[1] Okay, all it does now is create, sign, verify and unpack packages. There's a roadmap file for package distribution, which is a different problem.
> at least making the default much better by asking you every time they want to do something
Really? I thought 'asking you every time they want to do something' was called 'security fatigue' and generally considered to be a bad thing. Yes you can concatenate files in the current project, Claude.
Yes it has to be combined with a robust way to allowlist actions you trust
Oddly, since I wrote that Claude 'auto' mode just landed and I built something with it (instead of 'dangeously skip') and it's working.
I agree that dependencies are a liability, but, sadly, "batteries included" didn't work out for Python in practice (i. e. how do I even live without numpy? No, array aren't enough).
To the extend that Python is indeed "batteries included," that seems true. But just how "batteries included" is it? I'd argue that its batteries are pretty limited. Exhibit A: everybody uses the third-party requests instead of the stdlib urllib. Exhibit B: http.server isn't a production-ready webserver, so people use Flask or something beefier.
I'd contrast Python with Go, which has an amazing stdlib for the domains that Go targets. This last part is key--Go has a more focused scope than Python, and that makes it easier for its stdlib to succeed.
We could have different Python package bundles: Python base. Python webdev. Python desktop.
So, youre on Microsoft then, judging by ScottPlot you write .NET desktop apps. If you use Dapper, you probably use Microsoft.Data.SqlClient, which is... distributed over NuGet and vulnerable to supply chain attack. You may not need many deps as a desktop dev. Modern day line of business apps require a lot more deps. CSVHelper, ClosedXML, AutoMapper, WebOptimizer, NetEscapades.AspNetCore.SecurityHeaders.
Yes less deps people need the better but it doesn't fix trhe core problem. Sharing and distrib uting code is a key tenant of being able to write modern code.
Different programmers have very different ideas about what is "all the functionality you typically need."
What are some examples of batteries-included languages that folk around here really feel productive in and/or love? What makes them so great, in your opinion?
(Leaving aside thoughts on language syntax, compile times, tooling etc - just interested in people's experiences with / thoughts on healthy stdlibs)
I work in a NIS2 compliance sector, and we basically use Go and Python for everything. Go is awesome, Python isn't as such. Go didn't always come with the awesome stllib that it does today, which is likely partly why a lot of people still use things like Gin for web frameworks rather than simply using the standard library. Having worked with a lot of web frameworks, the one Go comes with is nice and easy enough to extend. Python is terrible, but on the plus side it's relatively easy to write your own libraries with Python, and use C/Zig to do so if you need it. The biggest challenges for us is that we aren't going to write a better MSSQL driver than Microsoft, so we use quite a bit of dependencies from them since we are married with Azure. These live in a little more isolation than what you might expect, so they aren't updated quite as often as many places might. Still, it's a relatively low risk factor that we can accept.
Our React projects are the contrast. They live in total and complete isolation, both in development and in production. You're not going to work on React on a computer that will be connected to any sort of internal resources. We've also had to write a novel's worth of legal bullshit explaining how we can't realistically review every line of code from React dependencies for compliance.
Anyway, I don't think JS/TS is that bad. It has a lot of issues, but then, you could always have written your own wrapper ontop of Node's fetch instead of using Axios. Which I guess is where working in the NIS2 compliance sector makes things a little bit different, because we'd always chose to write the wrapper instead of using one others made. With the few exceptions for Microsoft products that I mentioned earlier.
Go is well known for its large and high quality std lib
Go didn't even have versioning for dependencies for ages, so CVE reporting was a disaster.
And there's plenty of libraries you'll have to pull to get a viable product.
These are the big ones I use, specifically because of the standard libraries:
Python (decent standard library) - It's pretty much everywhere. There's so many hidden gems in that standard library (difflib, argparse, shlex, subprocess, cmd)
C#/F# (.NET)
C# feels so productive because of how much is available in .NET Core, and F# gets to tag along and get it all for free too. With C# you can compile executables down to bundle the runtime and strip it down so your executables are in the 15 MiB range. If you have dotnet installed, you can run F# as scripts.
These are definitely some good thoughts, thanks!
Do you worry at all about the future of F#? I've been told it's feeling more and more like a second-class citizen on .NET, but I don't have much personal experience.
I used to, but the knowledge of .NET seems mostly transferrable to C#. It's super useful to do `dotnet fsi` and then work out the appropriate .NET calls in the F# repl.
yep!
This is exactly the world I'm working towards with packaging tooling with a virtual machine i.e. electron but with virtual machines instead so the isolation aspect comes by default.
> "Batteries included" ecosystems are the only persistent solution to the package manager problem.
The irony in this case is that axios is not really needed now given that fetch is part of the JS std lib.
For a lot of code, I switched to generating code rather than using 3rd party libraries. Things like PEG parsers, path finding algorithms, string sanitizers, data type conversion, etc are very conveniently generated by LLMs. It's fast, reduces dependencies, and feels safer to me.
Ah, so you've traded the possibility of bad dependencies for certainty.
Remember, our objective function here is “feels safe.”
How can you come to that conclusion, given the specific examples I have given, which are tedious to write, but easy to proof-read and test?
Because AI threatens the identity of many programmers.
Or find the best third party library and copy the code from a widely used version that has been out long enough to have been well tested into your source tree.
The problem is not third party libraries. It is updating third party libraries when the version you have still works fine for your needs.
Don't do this. Use a package manager that let's you specify a specific version to pin against. Vendoring side steps most automated tooling that can warn you about vulnerabilities. Vendoring is a signal that your tooling is insufficient, 99% of the time.
Vendoring means you don't have to fetch the internet for every build, that you can work offline, that you're not at the mercy of the oh-so-close-99.999 availability, that it will keep on working in 10 years, and probably other advantages.
If your tooling can pull a dependency from the internet, it could certainly check if more recent version from a vendored one is available.
Is there any package manager incapable of working offline?
> Is there any package manager incapable of working offline?
I think you've identified the problem here: package management and package distribution are two different problems. Both tools have possibilities for exploits, but if they are separate tools then the surface area is smaller.
I'm thinking that the package distribution tool maintains a local system cache of packages, using keys/webrings/whatever to verify provenance, while the package management tool allows pinning, minver/maxver, etc.
Honestly, you can get pretty far with just Bun and a very small number of dependencies. It’s what I love most about Bun. But, I do agree with you generally. .NET is about as good as I’ve ever seen for being batteries included. I just hate the enterprisey culture that always seems to pervade .NET shops.
I agree about the culture. If I take my eye off the dev team for too long, I'll come back and we'll be using entity framework and a 20 page document about configuring code cleanup rules in visual studio.
Entity framework is pretty good.
I agree. Got downvoted a lot the other day for proposing Node should solve fundamental needs.
But javascript is batteries included in this case, you can use xmlhttprequest or fetch
What kind of apps do you build / industry etc?
Language churn makes this problem worse.
Frankly inventing a new language is irresponsible these days unless you build on-top of an existing ecosystem because you need to solve all these problems.
> "Batteries included" ecosystems are the only persistent solution
Or write your own stuff. Yes, that's right, I said it. Even HTTP. Even cryptography. Just because somebody else messed it up once doesn't mean nobody should ever do it. Professional quality software _should_ be customized. Professional developers absolutely can and should do this and get it right. When you use a third-party HTTP implementation (for example), you're invariably importing more functionality than you need anyway. If you're just querying a REST service, you don't need MIME encoding, but it's part of the HTTP library anyway because some clients do need it. That library (that imports all of its own libraries) is just unnecessary bloat, and this stuff really isn't that hard to get right.
> When you use a third-party HTTP implementation (for example), you're invariably importing more functionality than you need anyway. If you're just querying a REST service, you don't need MIME encoding, but it's part of the HTTP library anyway because some clients do need it. That library (that imports all of its own libraries) is just unnecessary bloat, and this stuff really isn't that hard to get right.
This post is modded down (I think because of the "roll your own crypto vibe", which I disagree with), but this is actually spot on the money for HTTP.
The surface area for HTTP is quite large, and your little API, which never needed range-requests, basic-auth, multipart form upload, etc suddenly gets owned because of a vulnerability in one of those things you not only never used, you also never knew existed!
"Surface area" is a problem, reducing it is one way to mitigate.
> Even cryptography
Good luck
I can't even imagine the scale of the impact with Axios being compromised, nearly every other project uses it for some reason instead of fetch (I never understood why).
Also from the report:
> Neither malicious version contains a single line of malicious code inside axios itself. Instead, both inject a fake dependency, plain-crypto-js@4.2.1, a package that is never imported anywhere in the axios source, whose only purpose is to run a postinstall script that deploys a cross-platform remote access trojan (RAT)
Good news for pnpm/bun users who have to manually approve postinstall scripts.
> nearly every other project uses it for some reason instead of fetch (I never understood why).
Fetch wasn't added to Node.js as a core package until version 18, and wasn't considered stable until version 21. Axios has been around much longer and was made part of popular frameworks and tutorials, which helps continue to propagate it's usage.
Also it has interceptors, which allow you to build easily reusable pieces of code - loggers, oauth, retriers, execution time trackers etc.
These are so much better than the interface fetch offers you, unfortunately.
You can do all of that in fetch really easily with the init object.
})There are pretty much two usage patterns that come up all the time:
1- automatically add bearer tokens to requests rather than manually specifying them every single time
2- automatically dispatch some event or function when a 401 response is returned to clear the stale user session and return them to a login page.
There's no reason to repeat this logic in every single place you make an API call.
Likewise, every response I get is JSON. There's no reason to manually unwrap the response into JSON every time.
Finally, there's some nice mocking utilities for axios for unit testing different responses and error codes.
You're either going to copy/paste code everywhere, or you will write your own helper functions and never touch fetch directly. Axios... just works. No need to reinvent anything, and there's a ton of other handy features the GP mentioned as well you may or may not find yourself needing.
Interceptors are just wrappers in disguise.
Convenience is a thing, but it doesn't require a massive library.That fetch requires so many users to rewrite the same code - that was already handled well by every existing node HTTP client- says something about the standards process.
It could also be trivially written for XMLHttpRequest or any node client if needed. Would be nice if they had always been the same, but oh well - having a server and client version isn't that bad.
Because it is so few lines it is much more sensible to have everyone duplicate that little snippet manually than import a library and write interceptors for that...
(Not only because the integration with the library would likely be more lines of code, but also because a library is a significantly liability on several levels that must be justified by significant, not minor, recurring savings.)
> Because it is so few lines it is much more sensible to have everyone duplicate that little snippet manually
Mine's about 100 LOC. There's a lot you can get wrong. Having a way to use a known working version and update that rather than adding a hundred potentially unnecessary lines of code is a good thing. https://github.com/mikemaccana/fetch-unfucked/blob/master/sr...
> import a library and write interceptors for that...
What you suggesting people would have to intercept? Just import a library you trust and use it.
But you said so yourself they are necessary… otherwise you would just use fetch. This reasoning is going around in circles.
Why the 'but'? Where is the circular reasoning? What are you suggesting we have to intercept?
- Don't waste time rewriting and maintaining code unecessarily. Install a package and use it.
- Have a minimum release age.
I do not know what the issue is.
but it does for massive DDoS :p
> Likewise, every response I get is JSON.
fetch responses have a .json() method. It's literally the first example in MDN: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/U...
It's literally easier than not using JSON because I have to think about if I want `repsponse.text()` or `response.body()`.
that's such a weak argument. you can write about 20 lines of code to do exactly this without requiring a third party library.
Helper functions seem trivial and not like you’re reimplementing much.
Don't be silly, this is the JS ecosystem. Why use your brain for a minute and come up with a 50 byte helper function, if you can instead import a library with 3912726 dependencies and let the compiler spend 90 seconds on every build to tree shake 3912723 out again and give you a highly optimized bundle that's only 3 megabytes small?
> usage patterns
IMO interceptors are bad. they hide what might get transformed with the API call at the place it is being used.
> Likewise, every response I get is JSON. There's no reason to manually unwrap the response into JSON every time.
This is not true unless you are not interfacing with your own backends. even then why not just make a helper that unwraps as json by default but can be passed an arg to parse as something else
One more use case for Axios is it automatically follows redirects, forwarding headers, and more importantly, omiting or rewriting the headers that shouldn't be forwarded for security reasons.
fetch automatically follows redirects, fetch will forward your headers, omitting or rewriting headers is how security breaks… now a scraper got through because it’s masquerading as Chrome.
What does an interceptor in the RequestInit look like?
A wrapper function around fetch… that’s what interceptors are…
It also supports proxies which is important to some corporate back-end scenarios
fetch supports proxies
Before that we had node-fetch. If you already use a dependency why not one that's pretty much what will come natively to every JS runtime soon.
The fetch API is designed for browsers. It's not designed for servers. Fetch may work for a particular use case on the server, it may not. Servers have needs over and above what a browser allows the client to do.
> I can't even imagine the scale of the impact with Axios being compromised, nearly every other project uses it for some reason instead of fetch (I never understood why).
You can remember this answer for every time you ask same question again:
"Coz whatever else/builtin was before was annoying enough for common use cases"
> (I never understood why).
I spent two years trying to get it out of a project that began long after Axios had become redundant but it's very hard to go back and challenge decisions like this because every business priority is aligned against this kind of work.
I expect libraries built on top of fetch will be the next to be compromised, because why would you use fetch without an arbitrary layer of syntactic sugar...
There was never the business value. But now remember this axios case and use it as ammo for the next issue. Just don’t abuse it
Does pnpm block postinstall on transitive deps too or just top-level? We have it configured at work but I've never actually tested whether it catches scripts from packages that get pulled in as sub-dependencies.
It prompts for transitive dependencies, too. I have never had workerd as a direct dependency of any project of mine but I get prompted to approve its postinstall script whenever I install cloudflare's wrangler package (since workerd needs to download the appropriate Workers runtime for your platform).
From what I can tell, it blocks it everywhere.
That's solid, really helps lock down the supply chain attack surface. Do you ever end up having to whitelist anything that legitimately needs to run on install?
After using pnpm for years (at least 5, don't remember exactly), I've only ever had to whitelist one library that uses a postinstall script to download a native executable for your system. And even this is not necessary, it's just poorly designed.
For example, esbuild and typescript 7 split binaries for different systems and architectures into separate packages, and rely on your package manager to pull the correct one.
> Good news for pnpm/bun users who have to manually approve postinstall scripts.
Would they not have approved it for earlier versions? But also wouldn't the chance of addition automatic approval be high (for such a widely used project)?
The prompt would be to approve the new malicious package (plain-crypto-js)'s scripts, too, which could tip users off that something was fishy. If they were used to approving one for axios and the attackers had just overwrote axios's own instead of making a new package, it would probably catch people out.
Assuming axios didn't have a postinstall script before, it wouldn't have been approved for a previous version. If you ignore it, you ignore it, but postinstall scripts are relatively rare in npm deps, so it would seem a bit out of place when the warning pops up.
Can't speak for other devs but I like to read postinstall scripts or at least put them through an LLM if they're too hard to grok.
It's also a little context dependent, for example if I was using Axios and I see a prompt to run the plain-crypto-js postinstall script, alarm bells would instantly ring, which would at least make me look up the changelog to see why this is happening.
In most cases I don't even let them run unless something breaks/doesn't work as expected.
>(I never understood why).
Because axios existed before the builtin fetch, and so there's a lot of stackoverflow answers explaining how to use fetch, and the llm models are trained on that, so they will write axios requests instead of fetch
PSA: npm/bun/pnpm/uv now all support setting a minimum release age for packages.
I also have `ignore-scripts=true` in my ~/.npmrc. Based on the analysis, that alone would have mitigated the vulnerability. bun and pnpm do not execute lifecycle scripts by default.
Here's how to set global configs to set min release age to 7 days:
(Side note, it's wild that npm, bun, and pnpm have all decided to use different time units for this configuration.)If you're developing with LLM agents, you should also update your AGENTS.md/CLAUDE.md file with some guidance on how to handle failures stemming from this config as they will cause the agent to unproductively spin its wheels.
> (Side note, it's wild that npm, bun, and pnpm have all decided to use different time units for this configuration.)
First day with javascript?
You mean first 86,400 seconds?
You have to admire the person who designed the flexibility to have 87239 seconds not be old enough, but 87240 to be fine.
Probably went with the simplest implementation, if starting from the current “seconds since epoch” value. Let the user do any calculations needed to translate three days into that measurement.
It also efficiently annoys the most people at once: those what want hours will complain if they set it to days, thought that want days will complain if hours are used. By using minutes or seconds you can wind up both segments while not offend those who rightly don't care because they can cope with a little arithmetic :)
Though doing what sleep(1) does would be my preference: default to seconds but allow m/h/d to be added to change that.
I'm old enough to remember computers being pitched as devices that can do tedious math for us. Now we have to do tedious math for them apparently.
Hence the way I would do it (and have for other purposes), as stated in my final sentence. Have the human state the intent and convert to your own internally preferred units as needed.
No no no, see now we just say "computer! do tedious math!", and it will do some slightly different math for us and compliment us on having asked it to do so.
I'm sure you would like to memorize all kinds of API instead of having something idiot proof and straightforward
As if `minimumReleaseAge` in `[install]` section of `.bunfig.toml` doesn't require the same kind of memorization.
Hey that's a great joke, you made me spill my morning home-brewed kombucha.
I'm going to steal that one for my JavaScript monthly developers meetup.
Is it ok if I attribute it to "Xirdus on Hacker News"?
Lol sure.
The one true unit of time is hexadecimal encoded nanoseconds since the unix epoch. (I'm only half joking because I actually have authored code that used that before.)
I actually think it is not too bad a design, because seconds are the SI base unit for time. Putting something like "x days" requires additional parsing steps and therefore complexity in the implementation. Either knowing or calculating how many seconds there are in a day can be expected of anyone touching a project or configuration at this level of detail.
Seconds are also unambiguous. Depending on your chosen definition, "X days" may or may not be influenced by leap seconds and DST changes.
I doubt anyone cares about an hour more or less in this context. But if you want multiple implementations to agree talking about seconds on a monotonic timer is a lot simpler
Could you explain what you mean re: ambiguity? I understand why “calendar units” like months are ambiguous, but minutes, hours, days, and weeks all have fixed durations (which is why APIs like Python’s `timedelta` allows them).
The minute between December 31, 2016 23:59 and January 1st 2017 is 61 seconds, not 60 seconds. The hour that contains that minute is 3601 seconds, the day that contains that hour is 43201 seconds, etc. If you assume a fixed duration and simply multiply by 43200, your math will be wrong compared to the rest of the world.
Daylight savings time makes a day take 23 hours or 25 hours. That makes a week take 7254000 seconds or 7261200 seconds. Etc.
That’s what I mean by calendar units. These aren’t issues if you don’t try to apply durations to the “real” calendar.
(This is all in the context of cooldowns, where I’m not convinced the there’s any real ambiguity risk by allowing the user to specify a duration in day or hour units rather than seconds. In that context a day is exactly 24 hours, regardless of what your local savings time rules are.)
"exactly 24 hours" could still be anywhere between 86399 and 86401 seconds, depending on leap seconds. At least if by an hour you mean an interval of 60 minutes, because a minute that contains a leap second will have either 59 or 61 seconds.
You could specify that for the purposes of cooldowns you want "hour" to mean an interval of 3600 seconds. But that you have to specify that should illustrate how ambiguous the concept of an hour is. It's not a useless concept by any means and I far prefer to specify duration in hours and days, but you have to spend a sentence or two on defining which definition of hours and days you are using. Or you don't and just hope nobody cares enough about the exact cooldown duration
Leap seconds are their own nightmare. UNIX time ignores them, btw, so that the unix epoch is 86400*number of days since 1/1/1970 + number of seconds since midnight. The behavior at the instance of a leap second is undefined.
That's a good way of describing that. It's far too easy to pretend UNIX timestamps would correspond to a stopwatch counting from 1/1/1970.
Right. Currently epoch time is off the stopwatch time by 27 seconds.
Undefined behavior is worse than complicated defined behavior imo.
In the UK last Sunday was 23 hours long because we switched to BST, and occasionally leap seconds will result in a minute being something other 60 seconds.
No it wasn't. The country instantaneously changed timezones from UTC+0 to UTC+1 (called something else locally), it was no different to any other timezone change from e.g. physically moving into another timezone.
exploiting the ambiguity in date formats by releasing a package during a leap second
I came here to argue the opposite. Expressing it in seconds takes away questions about time zones and DST.
I think you're incorrect to say that second are also ambiguous. Maybe what you mean is that days are more practical, but that seems very much a personal preference.
I understand the [flawed] reasoning behind "x seconds from now is going to be roughly now() + x on this particular system", but how does defining the cooldown from an external timestamp save you from dealing with DST and other time shenanigans? In the end you are comparing two timestamps and that comparison is erroneous without considering time shenanigans
I think you misread the comment you're replying to.
> seconds are the SI base unit for time
True. But seconds are not the base unit for package compromises coming to light. The appropriate unit for that is almost certainly days.
that kind of complexity is always worth it. Every single time. It's user time that you're saving and it also makes config clearer for readers and cuts out on "too many/little zeroes on accident" errors
It's just library for handling time that 98% of the time your app will be using for something else.
I find it best when I need a calculator to understand security settings. 604800 here we come
This is the difference between thinking about the user experience and thinking just about the technical aspect
Well, you have 1000000 microseconds in between. That's a big threshold.
wait what if we start on a day DST starts or ends????
OP should be glad a new time unit wasn't invented
Workdays! Think about it, if you set the delay in regular days/seconds the updated dependency can get pulled in on a weekend with only someone maybe on-call.
(Hope your timezones and tzdata correctly identifies Easter bank holiday as non-workdays)
> Workdays!
This is javascript, not Java.
In JavaScript something entirely new would be invented, to solve a problem that has long been solved and is documented in 20+ year old books on common design patterns. So we can all copy-paste `{ or: [{ days: 42, months: 2, hours: "DEFAULT", minutes: "IGNORE", seconds: null, timezone: "defer-by-ip" }, { timestamp: 17749453211*1000, unit: "ms"}]` without any clue as to what we are defining.
In Java, a 6000LoC+ ecosystem of classes, abstractions, dependency-injectables and probably a new DSL would be invented so we can all say "over 4 Malaysian workdays"
But you know that Java solution will continue working even after we no longer use the Gregorian Calendar, the collapse and annexation of Malaysia to some foreign power, and then us finally switching to a 4-day work week; so it'd be worth it.
It probably won’t work correctly from the get go. But it can be debugged everywhere so that’s good.
... and since it was architectured to allow runtime injection-patching of events before they hit the enterprise-service-bus, everyone using this library must first set fourteen ENV vars in their profile, and provide a /etc/java/springtime/enterprise-workday-handling/parse-event-mismatch.jar.patch. Which should fix the bug for you.
You can find the patch files for your OSs by registering at Oracle with a J3EE8.4-PatchLibID (note, the older J3EE16-PatchLib-ids aren't compatible), attainable from your regional Oracle account-manager.
And least one of those environment can contain template strings that are expanded with arguments from request headers when run under popular enterprise java frameworks, and by way of the injection patching could hot load arbitrary code in runtime.
A joke should be funny though, not just a dry description of real life, so let's leave it at that. We've already taken it too far.
This isn’t even remotely funny.
I am laughing. I'm not even near the end of this thread.
In before someone thinks it's a joke, the most commonly used logging library in Java had LDAP support in format scripts enabled by default" (which resulted, of course in CVE)
JavaScript Temporal. Not sure knowing what a "workday" is in each timezone is in it's scope but it's the much needed and improved JS, date API (granted with limited support to date)
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
There's an extra digit in your timestamp.
When I worked in Finance our internal Date extension did actually have Workdays that took into account Stock Market and Bank Holidays.
…now imagine a list of instruments, some of which have durations specified in days/weeks/months (problems already with the latter) and some in workdays, and the user just told your app to display it sorted by duration.
I tried to write this function in Power Query (Excel hell). Gave up after an hour or so.
Me too, it was just a constant filled with bank holidays for the next 6 years
Why would it get pulled in over the weekend? What automatic deployments are you running if there also isn't a human working to get it out?
Do you run automatic dependency updates over the weekend? Wouldn't you rather do that during fully-staffed hours?
And we also need localization. Each country can have their own holidays
And we need groups of locales for teams that are split across multiple locations; e.g.:
Hopefully "es" will have Siesta support too.
Might be better to calculate them separately for each locale and then tie-break with your own approach (min/max/avg/median/etc.)
Don't forget about regional holidays, which might follow arbitrary borders that don't match any of the official subdivisions of the country. Or may even depend on the chosen faith of the worker
Pulaski day in Illinois. Or Reds Opening Day in Cincinnati.
Nah, working hours and make global assumptions of 0900-1230/1330-1730, M-F, and have an overly convoluted way to specify what working ours actually are in the relevant location(s).
If we're taking suggestions, I'd like to propose "parsec" (not to be confused with the unit of distance of the same name)
That way Han Solo can make sense in the infamous quote.
EDIT: even Gemini gets this wrong:
> In Star Wars, a parsec is a unit of distance, not time, representing approximately 3.26 light-years
> That way Han Solo can make sense in the infamous quote.
They explained it in the Solo movie.
https://www.reddit.com/r/MovieDetails/comments/ah3ptm/solo_a...
Making a whole movie just to retcon the parsec misuse in Ep IV was a choice
They made a movie to make money. I doubt anyone holding the purse strings cared one iota if that bit were corrected or not. It’s not really a retcon either because they didn’t change anything.
That had more or less been the explanation in the books for decades, and even in George Lucas' notes from 1977:
> It's a very simple ship, very economical ship, although the modifications he made to it are rather extensive – mostly to the navigation system to get through hyperspace in the shortest possible distance (parsecs).
It was already fine, because it’s a metric defined on a submanifold of relativistic spacetime.
Parallax arc-second -> distance.
For Star Wars, they retconned it to mean he found the shortest possible route through dangerous space, so even for Han Solo's quote, it's still distance.
N multiplications of dozen-second
To me it sounds safer to have different big infra providers with different delays, otherwise you still hit everyone at the same time when something does inevitably go undetected.
And the chances of staying undetected are higher if nobody is installing until the delay time ellapses.
It's the same as not scheduling all cronjobs to midnight.
About the use of different units: next time you choose a property name in a config file, include the unit in the name. So not “timeout” but “timeoutMinutes”.
Yes!! This goes for any time you declare a time interval variable. The number of times I've seen code changes with a comment like "Turns out the delay arg to function foo is in milliseconds, not seconds".
Or require the value to specify a unit.
At that point, you're making all your configuration fields strings and adding another parsing step after the json/toml/yaml parser is done with it. That's not ideal either; either you write a bunch of parsing code (not terribly difficult but not something I wanna do when I can just not), or you use some time library to parse a duration string, in which case the programming language and time library you happen to use suddenly becomes part of your config file specification and you have to exactly re-implement your old time handling library's duration parser if you ever want to switch to a new one or re-implement the tool in another language.
I don't think there are great solutions here. Arguably, units should be supported by the config file format, but existing config file formats don't do that.
TOML has a datetime type (both with or without tz), as well as plain date and plain time:
We should extend it with durations: And like for datetimes, we should have a readable variant: Edit: discussed in detail here: https://github.com/toml-lang/toml/issues/514great, now attackers can also target all the libraries to enable all that complexity in npm too.
> adding another parsing step after the json/toml/yaml parser is done with it. That's not ideal either
I'd argue that it is ideal, in the sense that it's the sweet spot for a general config file format to limit itself to simple, widely reusable building blocks. Supporting more advanced types can get in the way of this.
Programs need their own validation and/or parsing anyway, since correctness depends on program-specific semantics and usually only a subset of the values of a more simply expressed type is valid. That same logic applies across inputs: config may come from files, CLI args, legacy formats, or databases, often in different shapes. A single normalization and validation path simplifies this.
General formats must also work across many languages with different type systems. More complex types introduce more possible representations and therefore trade-offs. Even if a file parser implements them correctly (and consistently with other such parsers), it must choose an internal form that may not match what a program needs, forcing extra, less standard transformation and adding complexity on both sides for little gain.
Because acceptable values are defined by the program, not the file, a general format cannot fully specify them and shouldn’t try. Its role is to be a medium and provide simple, human-usable (for textual formats), widely supported types, avoid forcing unnecessary choices, and get out of the way.
All in all, I think it can be more appropriate for a program to pick a parsing library for a more complex type, than to add one consistently to all parsers of a given file format.
Another parsing step is the common case. Few parameters represent untyped strings where all characters and values are valid. For numbers as well, you often have a limited admissible range that you have to validate for. In the present case, you wouldn’t allow negative numbers, and maybe wouldn’t allow fractional numbers. Checking for a valid number isn’t inherently different from checking for a regex match. A number plus unit suffix is a straightforward regex.
timeoutMs is shorter ;)
You guys can't appreciate a bad joke
Megaseconds are about the right timescale anyway
What megaseconds? They clearly meant the Microsoft-defined timeout.
Well megaseconds has the nice property that it's about about equal to a Scaramucci so it can be used across domains.
timoutμs is even better. People will learn how to type great symbols.
They wouldn't have to, if the file format accepted floats in proper exponential format.
Yes timout indeed!
not timeout at all is even shorter.
Pnpm did this first but I’m glad to see all the others follow suit
For anyone wondering, you need to be on npm >= 11.10.0 in order to use it. It just became available Feb 11 2026
https://github.com/npm/cli/releases/tag/v11.10.0
> PSA: npm/bun/pnpm/uv now all support setting a minimum release age for packages.
The solution is not moar toolz. That's the problem—this crazy mindset that the problems endemic to bad tooling have a solution in the form of complementing them with another layer, rather than fewer.
Git and every sane SCM already allow you to manage your source tree without jumping through a bunch of hoops to go along with wacky overlay version control systems like the one that the npmjs.com crew designed, centering around package.json as a way to do an end-run around Git. You don't need to install and deploy anything containing never-before-seen updates just because the NodeJS influencer–developers say that lockfiles are the Right Way to do things. (It's not.)
Opting in to being vulnerable to supply chain attacks is a choice.
<https://news.ycombinator.com/item?id=46006471>
<https://news.ycombinator.com/item?id=46360308>
Is there a way to do that per repo for these tools ? We all know how user sided configuration works for users (they usually clean it whenever it goes against what they want to do instead of wondering why it blocks their changes :))
At least with npm, you can have a .npmrc per-repo
pnpm does global + per-repo
min release age to 7 days about patch releases exposes you to the other side of the coin, you have an open 7 days window on zero-day exploits that might be fixed in a security release
The packages that are actually compromised are yanked, but I assume you're talking about a scenario more like log4shell. In that case, you can just disable the config to install the update, then re-enable in 7 days. Given that compromised packages are uploaded all the time and zero-day vulnerabilities are comparatively less common, I'd say it's the right call.
`uv` has per-package overrides, I imagine there may be similar in other managers
I haven't checked, but it would be surprising that the min-release-age applies to npm audit and equivalent commands
At least with pnpm, you can specify minimumReleaseAgeExclude, temporarily until the time passes. I imagine the other package managers have similar options.
[1]: https://pnpm.io/settings#minimumreleaseageexclude
Not really an issue though right because virtually none of these have lasted more than 1-2 days before being discovered?
Out of the frying pan and into the frier.....
Exactly what I thought too when I read this...
Urgent fix, patch released, invisible to dev team cause they put in a 7 day wait. Now our app is vulnerable for up to 7 days longer than needed (assuming daily deploys. If less often, pad accordingly). Not a great excuse as to why the company shipped an "updated" version of the app with a standing CVE in it. "Sorry we were blinded to the critical fix because set an arbitrary local setting to ignore updates until they are 7 days old". I wouldn't fire people over that, but we'd definitely be doing some internal training.
mise has an option as well (note the caveats though):
https://mise.jdx.dev/configuration/settings.html#install_bef...
And homebrew has discussed it, kinda sorta:
https://github.com/Homebrew/brew/issues/21129
It's wild that none of these are set by default.
I know 90% of people I've worked with will never know these options exist.
That would likely mean same amount of people get the vulnerability, just 7 days later.
The compromised packages were removed from the registry within hours.
Because everyone got updates immediately. If the default was 7 days, almost no one would get updates immediately but after 7 days, and now someone only finds about after 7 days. Unless there is a poor soul checking packages as they are published that can alert the registry before 7 days pass, though I imagine very few do that and hence a dedicated attacker could influence them to not look too hard.
If I remember correctly, in all the recent cases it was picked up by automated scanning tools in a few hours, not because someone updated the dependency, checked the code and found the issue.
So it looks like even if no one actually updates, the vast majority of the cases will be caught by automated tools. You just need to give them a bit of time.
If everyone or a majority of people sets these options, then I think issues will simply be discovered later. So if other people run into them first, better for us, because then the issues have a chance of being fixed once our acceptable package/version age is reached.
and for yarn berry
And when you actually need a super hot fix for a 0-day, you will need to revert this and keep it that way for some time to then go back to minimum age.
While this works, we stillneed a permanent solution which requires a sort of vetting process, rather than blindly letting everything through.
pnpm since v10.19.0 allows excluding specific dependencies from minReleaseAge by version.
Who will do the vetting process?
I think my vetting would settle for a repo diff against the previous version, confirming the only difference was the security fix (though that doesn't cover all the bases).
Jia Tan
If everyone avoids using packages released within the last 7 days, malicious code is more likely to remain dormant for 7 days.
What do you base that on? Threat researchers (and their automated agents) will still keep analyzing new releases as soon as they’re published.
Their analysis was triggered by open source projects upgrading en-masse and revealing a new anomalous endpoint, so, it does require some pioneers to take the arrows. They didn't spot the problem entirely via static analysis, although with hindsight they could have done (missing GitHub attestation).
A security company could set up a honeypot machine that installs new releases of everything automatically and have a separate machine scan its network traffic for suspicious outbound connections.
The problem is what counts as suspicious. StepSecurity are quite clear in their post that they decide what counts as anomalous by comparing lots of open source runs against prior data, so they can't figure it out on their own.
The fact threat researchers and especially their automated agents are not all that good at their jobs
Those threat researchers and their autonomous agents caught this axios release.
> What do you base that on?
The entire history of malware lol
Can you elaborate? Why do you believe that motivated threat hunters won’t continue to analyze and find threats in new versions of open source software in the first week after release?
Attackers going "low and slow" when they know they're being monitored is just standard practice.
> Why do you believe that motivated threat hunters won’t continue to analyze and find threats in new versions of open source software in the first week after release?
I'm sure they will, but attackers will adapt. And I'm really unconvinced that these delays are really going to help in the real world. Imagine you rely on `popular-dependency` and it gets compromised. You have a cooldown, but I, the attacker, issue "CVE-1234" for `popular-dependency`. If you're at a company you now likely have a compliance obligation to patch that CVE within a strict timeline. I can very, very easily pressure you into this sort of thing.
I'm just unconvinced by the whole idea. It's fine, more time is nice, but it's not a good solution imo.
What, in your view, is a better solution?
There are many options. Here's a post just briefly listing a few of the ones that would be handled by package managers and registries, but there are also many things that would be best done in CI pipelines as well.
https://news.ycombinator.com/item?id=47586241
that's why people are telling others to use 7 days but using 8 days themselves :)
You don't have to be faster than the bear, you just have to be faster than the other guy.
brb, switching everything to 9 days
That is 3D chess level type shit. xD
Genius
Worth noting this attack was caught because people noticed anomalous network traffic to a new endpoint. The 7-day delay doesn't just give scanners time, it gives the community time to notice weird behavior from early adopters who didn't have the delay set.
It's herd immunity, not personal protection. You benefit from the people who DO install immediately and raise the alarm
But wouldn't the type of people that notifes anomalous network activity be exactly the type of people who add a 7 day delay because they're security conscious?
And I’ll bet a chunk of already-compromised vibe coders are feeling really on-top-of-shit because they just put that in their config, locking in that compromised version for a week.
I suspect most packages will keep a mix of people at 7 days and those with no limit. That being said, adding jitter by default would be good to these features.
>adding jitter by default would be good
This became evident, what, perhaps a few years ago? Probably since childhood for some users here but just wondering what the holdup is. Lots of bad press could be avoided, or at least a little.
They’re usually picked up by scanners by then.
Most people won’t.
7 days gives ample time for security scanning, too.
This highly depends on the detection mechanism.
> If everyone avoids using packages released within the last 7 days
Which will never even come close to happening, unless npm decides to make it the default, which they won't.
lol with mise I used a fourth time unit: https://mise.jdx.dev/configuration/settings.html#install_bef...
I think the npm doesn't support end of line comments, so
actually doesn't set it at all, please edit your comment.EDIT: Actually maybe it does? But it's weird because
`npm config list -l` shows: `min-release-age = null` with, and without the comment. so who knows ¯\_(ツ)_/¯
ok, it works, only the list function shows it as null...
Where in the pnpm documentation does it say that it ignores scripts by default?
From https://pnpm.io/cli/install#--ignore-scripts:
> Default: *false*
Weird. The config also appears to default to `false`
https://pnpm.io/settings#ignorescripts
This page describes the behavior, "disables the automatic execution of postinstall scripts in dependencies":
https://pnpm.io/supply-chain-security
While this explicitly calls out "postinstall", I'm pretty sure it affects other such lifecycle scripts like preinstall in dependencies.
The --ignore-scripts option will ignore lifecycle scripts in the project itself, not just dependencies. And it will ignore scripts that you have previously allowed (using the "allowBuilds" feature).
Run npm/pnpm/bun/uv inside a sandbox.
There is no reason to let random packages have full access to your machine
Sandboxing by to default world be really nice. One of the things I really appreciate about Claude Code is its permissions model
Props to uv for actually using the correct config path jfc what is “bunfig”
Silly portmanteau of "bun" and "config"
A trendy sandwich
Everyone has forgotten standard ISO 8601 durations and invented their own syntax.
uv supports it, https://docs.astral.sh/uv/reference/settings/#exclude-newer
Perfect
The config for uv won't work. uv only supports a full timestamp for this config, and no rolling window day option afaik. Am I crazy or is this llm slop?
https://docs.astral.sh/uv/concepts/resolution/#dependency-co...
> Define a dependency cooldown by specifying a duration instead of an absolute value. Either a "friendly" duration (e.g., 24 hours, 1 week, 30 days) or an ISO 8601 duration (e.g., PT24H, P7D, P30D) can be used.
My bad. This works for per project configuration, but not for global user configuration.
It should work for global configuration too, please file an issue if you’re observing otherwise.
(Make sure you’re on a version that actually supports relative times, please!)
This is what tripped me up. I added that config and then got this error:
error: Failed to parse: `.config/uv/uv.toml` Caused by: TOML parse error at line 1, column 17 | 1 | exclude-newer = "7 days" | ^^^^^^^^ failed to parse year in date "7 days": failed to parse "7 da" as year (a four digit integer): invalid digit, expected 0-9 but got
I was on version 0.7.20, so I removed that line, ran "uv self update" and upgraded to 0.11.2 and then re-added the config and it works fine now.
Yeah, that error message isn’t ideal on older versions, but unfortunately there’s no way to really address that. But I’m glad it’s working for you on newer versions.
For what it's worth the error made sense enough to me that I figured I needed to upgrade. :-)
I think it should work at the user config level too:
> If project-, user-, and system-level configuration files are found, the settings will be merged, with project-level configuration taking precedence over the user-level configuration, and user-level configuration taking precedence over the system-level configuration.
https://docs.astral.sh/uv/concepts/configuration-files/
Good luck with any `npm audit` in a pipeline. Sometimes you have to pull the latest release because the previous one had a critical vulnerability.
npm is claiming this doesn’t exist
Make sure you're on version 11.10 or later?
There’s a recurrent pattern with these package compromises: the attacker exfiltrates credentials during an initial phase, then pivots to the next round of packages using those credentials. That’s how we saw them make the Trivy to LiteLLM leap (with a 5 day gap), and it’ll almost certainly be similar in this case.
The solution to this is twofold, and is already implemented in the primary ecosystems being targeted (Python and JS): packagers should use Trusted Publishing to eliminate the need for long lived release credentials, and downstreams should use cooldowns to give security researchers time to identify and quarantine attacks.
(Security is a moving target, and neither of these techniques is going to work indefinitely without new techniques added to the mix. But they would be effective against the current problems we’re seeing.)
In this case, the author's NPM account was taken over, email address changed to one the attacker controls, and the package was manually published.
Since the attacker had full control of the NPM account, it is game over - the attacker can login to NPM and could, if they wanted, configure Trusted Publishing on any repo they control.
Axios IS using trusted publishing, but that didn't do anything to prevent the attack since the entire NPM account was taken over and config can be modified to allow publishing using a token.
Yeah, NPM should be enforcing 2FA and likely phishing resistant 2FA for some packages/ this should be a real control, issuing public audit events for email address changes, and publish events should include information how it was published (trusted publishing, manual publish, etc).
Instead they took away TOTP as a factor.
Scaling security with the popularity of a repo does seem like a good idea.
Are there downsides to doing this? This was my first thought - though I also recognize that first thoughts are often naive.
You don't want "project had X users so it's less safe" to suddenly transition into "now this software has X*10 users so it has to change things", it's disruptive.
TOTP although venerable was better than no second factor at all.
TOTP isn't phishing resistant
No it's not but it's better than nothing. Don't let the perfect be the enemy of the good.
TOTP seems effectively useless for npm so that seems fine to me
Well, that sucks! It’ll be interesting to learn how they obtained a valid second factor or 2FA bypass; that will inform the next round of defenses here.
One wonders if Microsoft/npm.js should allow new packages to be published immediately following an account email address change? I mean changes to email address are already recognized as potential attack vectors, so emails are sent to the old address warning of potential account take over. But this seems to have been done at night, so the warning email would not be seen yet. Even so a new package could be published and served to the world immediately. Unless I misunderstand something about the facts this would indicate an extreme lack of imagination in the people at Microsoft who already went through several cycles of hardening the service against supply chain poisoning attacks.
Was this an drive-by/auto-install attack?
There are solutions, the problem is almost always discipline.
I don’t know what this means. Discipline is good, but I think you need to have good tools/primitives in place to help people exercise discipline.
(The classic example being passwords: we wouldn’t need MFA is everybody just “got good” and used strong/unique passwords everywhere. But that’s manifestly unrealistic, so instead we use our discipline budget on getting people to use password managers and phishing-resistant MFA.)
Really? You don't know the difference between having a door lock, and using it?
MFA is typically enforced by organizations, forcing discipline. Individual usage of MFA is dramatically lower
Rejecting any packages newer than X days is one nice control, but ultimately it'd be way better to maintain an allowlist of which packages are allowed to run scripts.
Unfortunately npm is friggen awful at this...
You can use --ignore-scripts=true to disable all scripts, but inevitably, some packages will absolutely need to run scripts. There's no way to allowlist specific scripts to run, while blocking all others.
There are third-party npm packages that you can install, like @lavamoat/allow-scripts, but to use these you need to use an entirely different command like `npm setup` instead of the `npm install` everyone is familiar with.
This is just awful in so many ways, and it'd be so easy for npm to fix.
I recommend everyone to use bwrap if you're on linux and alias all package managers / anything that has post build logic with it.
I have bwrap configured to override: npm, pip, cargo, mvn, gradle, everything you can think of and I only give it the access it needs, strip anything that is useless to it anyway, deny dbus, sockets, everything. SSH is forwarded via socket (ssh-add).
This limits the blast radius to your CWD and package manager caches and often won't even work since the malware usually expects some things to be available which are not in a permissionless sandbox.
You can think of it as running a docker container, but without the requirement of having to have an image. It is the same thing flatpak is based on.
As for server deployments, container hardening is your friend. Most supply chain attacks target build scripts so as long as you treat your CI/CD as an untrusted environment you should be good - there's quite a few resources on this so won't go into detail.
Bonus points: use the same sandbox for AI.
Stay safe out there.
This only works for post-install script attacks. When the package is compromised, just running require somewhere in your code will be enough, and that runs with node/java/python and no bwrap.
node is also sandboxed within bwrap I have sandbox -p node if I have to give node access to other folders, I also have sandbox -m to define custom mountpoints if necessary and UNSAFE=1 as a last resort which just runs unsandboxed.
Check also https://github.com/wrr/drop which is a higher-level tool than bwrap. It allows you to make such isolated sandboxes with minimal configuration.
This looks nice but I wouldn't trust a very fresh tool to do security correctly.
As a higher-level alternative to bwrap, I sometimes use `flatpak run --filesystem=$PWD --command=bash org.freedesktop.Platform`. This is kind of an abuse of flatpaks but works just fine to make a sandbox. And unlike bwrap, it has sane defaults (no extra permissions, not even network, though it does allow xdg-desktop-portal).
Shame it's not a bit more mature, it does look like more the sort of thing I want. I use firejail a bit, but it's a bit awkward really.
To be honest - and I can't really believe I'm saying it - what I really want is something more like Android permissions. (Except more granular file permissions, which Android doesn't do at all well.) Like: start with nothing, app is requesting x access, allow it this time; oh alright fine always allow it. Central place to manage it later. Etc.
I think firejail is a much more flexible security sandbox than bwrap. It also comes with pre-defined profiles
bwrap is as secure as you want it to be which I think is the primary advantage over anything else.
I wrote a Docker-based sandbox [1] for myself last year to control the blast radius of such malicious packages.
https://github.com/ashishb/amazing-sandbox
AFAIK maven doesn’t support post install logic like npm does. You have to explicitly optin with build plugins. It doesn’t let any arbitrary dependency run code on your machine.
some post processors have chains to execution (ex: lombok)
You explicitly opt in by using a compiler plugin. Merely having it as a dependency, like in npm, doesn’t mean it can run code at build time.
I like the idea of bubblewrap, but my pain point is that it is work to set it up correctly with bind mounts and forwarding necessary environment variables to make the program actually work usefully. Could you share your pip bwrap configuration? It sounds useful.
can't really share a file here, feel free to email me
> SSH is forwarded via socket
Maybe I misunderstood this point. But the ssh socket also gives access to your private keys, so I see no security gain in that point. Better to have a password protected key.
It's so your private key is not stolen, but you're right passphrase protected keys win anyway. I use hardware keys so this isn't a problem for me to begin with.
Do you have a recommendation for something like bwrap but for macos? I've been trying to use bwrap more on my servers when I remember.
unfortunately not, but there is work being done to support overlays properly I think?
I've been advocating to minimize the number of dependencies for some time now. Once you've maintained an open source project for several years, you start to understand the true cost of dependencies and you actually start to pay attention to the people behind the libraries. Popularity doesn't necessarily mean reliable or trustworthy or secure.
Agree. This is one of the major takeaways I've had from writing Go over the years -- which is even a Go proverb [0], "a little copying is better than a little dependency." Fortunately, LLMs make writing your own implementations of little dependencies super easy too.
[0]: https://go-proverbs.github.io/
I'll agree with that, and you would think it's common sense for any competent engineer, but for many people it's just an afterthought. Including senior and lead engineers. General matters like security are a political liaison, how can you raise the alarm/advocate for critical security improvements as an IC without making people around and above you look bad? How can you justify time invested into these things without raising the alarm? At the same time, you can't just ignore glaring security holes. It's a fine line to walk, and actually being realistic about the possibilities has found me nothing but enmity from peers and superiors due to it seeming like I'm throwing them under the bus.
In general, management was to see progress. I've come to find that technical details like these are an afterthought for most engineers, so far as the deadlines are being met.
It's one of these things that are under the water, tech side jobs. Everyone has to be on board, if your peers don't give a fuck you're just an annoyance and will be swimming counter-current.
Not to beat a dead horse but I see this again and again with dependencies. Each time I get more worried that the same will happen with rust. I understand the fat std library approach won’t work but I really still want a good solution where I can trust packages to be safe and high quality.
If the fat std library is not viable you can only increase security requirements.
Axios has like 100M downloads per week. A couple of people with MFA should have to approve changes before it gets published.
This is the actual answer: stupid cost saving creating an operational risk.
At least then they will have to pay off a dev or something, changes their economic calculus and is additionally illegal
Hosting curated dependencies is a commercially valuable service. Eventually an economy arises where people pay vendors to vet packages.
It's what linux distributions do.
Queue appimage or other packed binary and there go your finetuned packages.
Yes, that why those need to be 100% sandboxed by default (ideally a VM), unless they are provided by distro
what?
It already exists; cloudsmith
Linux distros and BSD ports did that since the 90's. When Linux distros had barely a PM or just tarballs, Infomagic sold 4 CD full of libre software. When I had no internet at home, back in the day I bought 3 DVD's of Debian Sarge for 20 euros, about $20. A bargain, it was the price of a hard-cover best seller book.
GB's of libre software, graphical install, 2.6 kernel, KDE3 desktop, very light on my Athlon 2000 with 256MB of RAM. It was incredible compared to what you got with Windows XP and 120 Euro per seat. Nonfree software and almost empty.
And, well, if for instance I could get read only, ~16TB durable USB drive with tons of Guix packages offline (in a two yearly basis with stable releases) for $200 I would buy them in the spot.
You would say that $200 for a distro it's expensive, but for what it provides, if you are only interested in libre gaming and tools, they amount you save can be huge. I've seen people spend $400 in Steam games because of the Holyday sales...
Why wouldn't the "fat std" thing work? Yes it's hard to design properly, both in scope and actual design (especially for an unstandardized language still moving fast), but throwing the towel and punting the problem to the "free market" of uncurated public repos is even worse.
It's what we call in France "la fête du slip".
PS: that's one reason I try to use git submodules in my Common Lisp projects instead of QuickLisp, because I really see the size of my deptree this way.
Because fat std is rigid, impractical, and annoying.
In practice (e.g. Go) it’s actually pretty good and infinitely preferable to third party everything.
Yeah, it's annoying to have good support for dates in Java since 2014, instead of only getting it now like in JS.
I think we found the constituency that led to the present sorry situation.
That's rather rude.
If you're referring to my packages on npm, I joined way late to that game. This was also ~15 years ago.
Works just fine in Go.
Fat std library mistakes/warts would likely result in third party packages being used anyway.
Not necessarily, but let's agree that some design faults would happen: you still get the option to use the solid, boring and slightly rusty std instead of another 100 dependencies from the supply chain supermarket.
At work, we're happy with Python's included batteries when we need to make scripts instead of large programs.
So it provides another option, and in worst case it doesn't make situation worse than it is right now?
Yeah, pretty bad idea.
NPM should have a curation mechanism, via staff review or crowdsourcing, where versions of popular packages are promoted to a stable set, like linux distros do. I would only use curated versions if they had such a thing.
An alternative:
- copy the dependencies' tests into your own tests
- copy the code in to your codebase as a library using the same review process you would for code from your own team
- treat updates to the library in the same way you would for updates to your own code
Apparently, this extra work will now not be a problem, because we have AI making us 10x more efficient. To be honest, even without AI, we should've been doing this from the start, even if I understand why we haven't. The excuses are starting to wear thin though.
Just going to put features on hold for a month while I review the latest changes to ffmpeg.
I don't know where you've worked but a hostile and intelligent actor or internal red team would succeed under each of those cases at every job I've worked at.
Defending against a targeted attack is difficult, yes. But these recent campaigns were all directed at everyone. Auditing and inspecting your dependencies does absolutely help thwart that because there will always be people who don't.
Good to know. Where were the places you worked at?
# If you have a projects folder containing multiple projects on macOS, you can run this script to recursively scan all subfolders for vulnerable axios versions and the presence of plain-crypto-js, helping you quickly identify potentially affected projects:
find . -name "package.json" -exec sh -c ' dir=$(dirname "{}") echo "==== $dir ====" cd "$dir" npm list axios 2>/dev/null | grep -E "1\.14\.1|0\.30\.4" grep -A1 "\"axios\"" package-lock.json 2>/dev/null | grep -E "1\.14\.1|0\.30\.4" [ -d node_modules/plain-crypto-js ] && echo "POTENTIALLY AFFECTED" ' \;
Genuinely how are you supposed to make sure that none of the software you have on your system pulls this in?
It’s things like this that make me want to swap to Qubes permanently, simply as to not have my password manager in the same context as compiling software ever.
We run everything NPM related inside Apple containers, and are looking to do the same with Python and Rust soon. Bwrap on Linux does the same.
I like to think of it like working with dangerous chemicals in the lab. Back in the days, people were sloppy and eventually got cancer. Then dangers were recognized and PPE was developed and became a requirement.
We are now at the stage in software development where we are beginning to recognizing the hazards and developing + mandating use of proper PPE.
A couple of years ago, pip started refusing to install packages outside of a virtualenv. I'm guessing/hoping package managers will start to have an opt-in flag you can set in a system-wide config file, such that they refuse to run outside of a sandbox.
The problem is that package managers are a distraction. You have to sandbox everything or else it doesn't work. These attacks use post-install hooks for convenience but nothing would have stopped them patching axios itself and just waiting for devs to run the app on their local workstation. So you end up needing to develop in a fully sandboxed environment.
Yeah the whole rush on "post-run hooks bad" isn't really adding all that much to security.
Like congratulations, your dev was compromised whole 10 minutes later after he ran code.
They are not a distraction when they are also the command runners.
While it's not perfect, pinning specific versions and managing all updates directly has been a solid solution for my team. Things can of course still slip through, but we're never vulnerable to these just because there was a new package release and we opted into it by default.
Updating packages takes longer, but we try to keep packages to a minimum so it ends up not being that big deal.
This sounds like satire but isn't - I just make sure the nodejs/npm packages don't exist on my system. I've yet to find a crucial piece of software that requires it. As much as I love that cute utility that turns maps into ascii art, it's not exactly sqlite in terms of usefulness.
Bit ridiculous to dismiss the most popular programming languages packaging repo as silly toys.
I don't deny that node/npm is useful for building servers, devtools for JS development itself, etc. but as an end user I haven't encountered anything useful which requires having it on my machine.
Ok? So you don't code in that language?
You still have multiple programming languages preinstalled on your OS, no matter which one it is.
Essential steps to minimise your exposure to NPM supply chain attacks:
— Run Yarn in zero-installs mode (or equivalent for your package manager). Every new or changed dependency gets checked in.
— Disable post-install scripts. If you don’t, at least make sure your package manager prompts for scripts during install, in which case you stop and look at what it’s going to run.
— If third-party code runs in development, including post-install scripts, try your best to make sure it happens in a VM/container.
— Vet every package you add. Popularity is a plus, recent commit time is a minus: if you have this but not that, keep your eyes peeled. Skim through the code on NPM (they will probably never stop labelling it as “beta”), commit history and changelog.
— Vet its dependency tree. Dependencies is a vector for attack on you and your users, and any new developer in the tree is another person you’re trusting to not be malicious and to take all of the above measures, too.
> Run Yarn in zero-installs mode (or equivalent for your package manager). Every new or changed dependency gets checked in.
Idk, lockfiles provide almost as good protection without putting the binaries in git. At least with `--frozen-lockfile` option.
Zero-installs mode does not replace the lockfile. Your lockfile is still the source of truth regarding integrity hashes.
However, it’s an extra line of defence against
1) your registry being down (preventing you from pushing a security hotfix when you find out another package compromised your product),
2) package unpublishing attacks (your install step fails or asks you to pick a replacement version, what do you do at 5pm on a Friday?), and
3) possibly (but haven’t looked in depth) lockfile poisoning attacks, by making them more complicated.
Also, it makes the size of your dependency graph (or changes therein) much more tangible and obvious, compared to some lines in a lockfile.
Number 1 would only be a win for zero-installs if it happened that registry was up when you made the security hotfix, since you'd need to install the depdencency the first time to get it in VC, but then suddenly down when doing a deploy. Seems like a highly unlikely scenario to me. Also, cases where npm CVEs must be patched with such urgency or bad things will happen are luckily very rare, in my experience.
Most npm CVEs are stuff like DDoS vulnerabilities, and you should have mitigations for those in place for at the infra-level anyway (e.g. request timeouts, rate limits, etc), or you are pretty much guaranteed to be cooked sooner or later anyway. The really dangerous stuff like arbitrary command execution from a library that takes end user input is much much more rare. The most recent big one I remember is React2shell.
Number 2 hasn't been much of an issue for a long time. npm doesn't allow unpublishing package after 72 hours (apart from under certain rare conditions).
Don't know about number 3. Would feel to me that if you have something running that can modify lockfile, they can probably also modify the chekced-in tars.
I can see how zero-installs are useful under some specific constraints where you want to minimize dependencies to external services, e.g. when your CI runs under strict firewalls. But for most, nah, not worth it.
> you'd need to install the depdencency the first time to get it in VC, but then suddenly down when doing a deploy.
Which dependency? It sounds like you are assuming some specific scenario, whereas the fix can take many forms. In immediate term, the quickest step could be to simply disable some feature. A later step may be vendoring in a safe implementation.
The registry doesn’t need to be actually down for you, either; the necessary condition is that your CI infrastructure can’t reach it.
> cases where npm CVEs must be patched with such urgency or bad things will happen are luckily very rare, in my experience.
Not sure what you mean by “npm CVEs”. The registry? The CLI tool?
As I wrote, if you are running compromised software in production, you want to fix it ASAP. In first moments you may not even know whether bad things will happen or not, just that you are shipping malicious code to your users. Even if you are lucky enough to determine with 100% confidence (putting your job on the line) that the compromise is inconsequential, you don’t want to keep shipping that code for another hour because your install step fails due to a random CI infra hiccup making registry inaccessible (as happened in my experience at least half dozen times in years prior, though luckily not in a circumstance where someone attempted to push an urgent security fix). Now imagine it’s not a random hiccup but part of a coordinated targeted attack, and somehow it becomes something anticipated.
> Number 2 hasn't been much of an issue for a long time. npm doesn't allow unpublishing package after 72 hours (apart from under certain rare conditions).
Those rare conditions exist. Also, you are making it sound as if the registry is infallible, and no humans and/or LLMs there accept untrusted input from their environment.
The key aspect of modern package managers, when used correctly, is that even when the registry is compromised you are fine as long as integrity check crypto holds up and you hold on to your pre-compromise dependency tree. The latter is not a technical problem but a human problem, because conditions can be engineered in which something may slip past your eyes. If this slip-up can be avoided at little to no cost—in fact, with benefits, since zero-installs shortens CI times, and therefore time-to-fix, due to dramatically shorter or fully eliminated install step—it should be a complete no-brainer.
> Don't know about number 3. Would feel to me that if you have something running that can modify lockfile, they can probably also modify the chekced-in tars.
As I wrote, I suspect it’d complicate such attacks or make them easier to spot, not make them impossible.
This is why Artifactory and similar exist and they do this better. You ~never want to vendor libraries.
Are you saying it replaces my package manager, or that I should add another tool to my stack, vet yet another vulnerable dependency for critical use, to do something my package manager already does just as well?
> You ~never want to vendor libraries.
I just explained why you should, and you are yet to provide a counter-argument.
Exactly. Yarn uses a yarn.lock file with the sha256 hashes of each npm package it downloads from the repo (they are .tgz files). If the hash won't match, install fails. No need to commit the dependencies into your git.
Very detailed and props to the security researchers, but the blog post has several indicators that it was written by AI, to which point I suspect their malware analysis was also done by a LLM.
I just wish it had more human interaction rather than have a GenAI spit out the blog post. It's very repetitive and includes several EM dashes.
It is harder and harder to trust any blog post anymore, the more AI there is. I used to read blog posts because of the personality and the precision level. Now both have been taken away.
Yeah, 98% of blog posts only exist for SEO purposes and so most of that is of course written by LLM
How much do you want to bet me that the credential was stolen during the previous LiteLLM incident? At what point are we going to have to stop using these package managers because it's not secure? I've got to admit, it's got me nervous to use Python or Node.js these days, but it's really a universal problem.
> it’s got me nervous to use Python or Node.js these days
My feelings precisely. Min package age (supported in uv and all JS package managers) is nice but I still feel extremely hesitant to upgrade my deps or start a new project at the moment.
I don’t think this is going to stabilize any time soon, so figuring out how to handle potentially compromised deps is something we will all need to think about.
NPM only gained minimum package age in February of this year, and still doesn't support package exclusions for internal packages.
https://github.com/npm/cli/pull/8965
https://github.com/npm/cli/issues/8994
Its good that that they finally got there but....
I would be avoiding npm itself on principle in the JS ecosystem. Use a package manager that has a history of actually caring about these issues in a timely manner.
It almost doesn't matter, because you can get pwned by a transitive dependency. If someone doesn't have the same scruples as you have, you're still at risk.
minimumReleaseAge and lockfiles also pin down transitive dependencies.
PNPM makes you approve postinstall scripts instead of running them by default, which helps a lot. Whenever I see a prompt to run a postinstall script, unless I know the package normally has one & what it does, I go look it up before approving it.
(Of course I could still get bitten if one of the packages I trust has its postinstall script replaced.)
How does this stance work with your CICD?
I suppose you would have to commit your node_modules, or otherwise cache your setup so that all prerequesite modules are built and ready to install without running post-install scripts?
More like the Trivy incident (which led to the compromise of LiteLLM).
There are ways to limit the blast radius, like running them in ephemeral rootless containers with only the project files mounted.
JS package managers (pnpm, bun) now will ignore postinstall scripts by default. Except for npm, it still runs them for legacy reasons.
You should probably set your default to not run those scripts. They are mostly unnecessary.
83M weekly downloads!Has anyone else noticed there was a recent sudden flurry of 3000 deleted issues on axios/axios? The jump happened on March 23. Was this a first sign of compromise? Or just coincidence of an AI agent going rogue.
There are pretty much exactly 3000 deleted issues, with the range starting at https://github.com/axios/axios/issues/7547 (7547) and ending at https://github.com/axios/axios/issues/10546 (10546 which is 7547+2999)
Maybe just a coincidence but they have cubic-dev-ai edit every single PR with a summary. And that bot edits PR descriptions even for outside contributors.
Package managers are a failed experiment.
We have libraries like SQLite, which is a single .c file that you drag into your project and it immediately does a ton of incredibly useful, non-trivial work for you, while barely increasing your executable's size.
The issue is not dependencies themselves, it's transitive ones. Nobody installs left-pad or is-even-number directly, and "libraries" like these are the vast majority of the attack surface. If you get rid of transitive dependencies, you get rid of the need of a package manager, as installing a package becomes unzipping a few files into a vendor/ folder.
There's so many C libraries like this. Off the top of my head, SQLite, FreeType, OpenSSL, libcurl, libpng/jpeg, stb everything, zlib, lua, SDL, GLFW... I do game development so I'm most familiar with the ones commonly used in game engines, but I'm sure other fields have similarly high quality C libraries.
They also bindings for every language under the sun. Rust libraries are very rarely used outside of Rust, and C#/Java/JS/Python libraries are never used outside their respective language (aside form Java ones in other JVM langs).
Package managers are now basically a requirement for language adoption. Doing it manually is not a solution, in an automated world.
What is a problem is library quality. Which is downstream of nobody getting paid for it, combined with an optimistic but unrealistic "all packages are equal" philosophy.
> High quality C libraries
> OpenSSL
OpenSSL is one of the ones where there's a ground up rewrite happening because the code quality is so terrible while being security critical.
On the other end, javascript is uniquely bad because of the deployment model and difficulty of adding things to the standard library, so everything is littered with polyfills.
> Package managers are now basically a requirement for language adoption. Doing it manually is not a solution, in an automated world.
Absolute nonsense. What does automated world even mean? Even if one could infer reasonably, it's no justification. Appealing to "the real world" in lieu of any further consideration is exactly the kind of mindlessness that has led to the present state of affairs.
Automation of dependency versions was never something we needed it was always a convenience, and even that's a stretch given that dependency hell is abundant in all of these systems, and now we have supply chain attacks. While everyone is welcome to do as they please, I'm going to stick to vendoring my dependencies, statically compiling, and not blindly trusting code I haven't seen before.
> Automation of dependency versions was never something we needed
How do you handle updating dependencies then?
Relax, while mentioning the real world without any criticism for the soundness of the solution is absolute nonsense, some would say idiotic, thinking only in the absolute best solution given your narrow world view is not any better.
While I agree that my view is narrow, the "best solution" in question is what we used to do, and it was fine. There are still many places that manually manage dependencies. Fundamentally automatic software versioning is an under-developed area in need of attention, and technologies like semantic versioning which are ubiquitous are closer to suggestions, and not true indicators of breaking changes. My personal view is that fully automatic dependency version management is an ongoing experiment and should be treated as such.
> What does automated world even mean?
People are trying to automate the act of programming itself, with AI, let alone all the bits and pieces of build processes and maintenance.
They're not a failed experiment. No one has ever "experimented" by making a safe package manager for their new language. And it is not that insane to do so. Very basic things will get you very far:
1. Packages should carry a manifest that declares what they do at build time, just like Chrome extensions do. This manifest would then be used to configure its build environment.
2. Publishers to official registries should be forced to use 2FA. I proposed this a decade ago for crates.io and people lost their minds, like I was suggesting we drag developers to a shed to be shot.
3. Every package registry should produce a detailed audit log that contains a "who, what, when". Every build/ command should be producing audit logs that can be collected by endpoint agents too.
4. Every package registry should support TUF.
5. Typosquatting defenses should be standard.
etc etc etc. Some of this is hard, some of this is not hard. All of this is possible. No one has done it, so it's way too early to say "package managers can't be made safe" when no one has tried.
I don't understand commercial aspect of large OSS like package managers but i was wondering for years why this was missing from npm. I think typosquatting was handled by npm last year but only after some popular miss typed packages started stealing developer creds.
The people building package managers are unaware of these problems going into it and it becomes extremely disruptive to start adding these things later on since your entire ecosystem is built on the assumption that they can do these things.
It's also shockingly controversial to suggest typosquatting suggestions. I made this suggestion ages ago for cargo, demonstrated that basic distance checks would have impacted <1% of crates over all time, and people still didn't want it.
For those who didn't know what TUF means (like me), I think they're referring to The Update Framework (https://theupdateframework.io).
Sorry, I should have clarified that - you're correct. `cosign` is an example of a tool that makes this quite straightforward and proves that this sort of system can work today.
> Publishers to official registries should be forced to use 2FA. I proposed this a decade ago for crates.io and people lost their minds, like I was suggesting we drag developers to a shed to be shot.
How is this enforced when it's pushed via a pipeline?
Your account is separate from your publishing. That is, in order to go to my account to change configuration values, 2FA must be required.
Publishing should be handled via something like Trusted Publishing, which would leverage short lived tokens and can integrate with cryptographic logs for publish information (ie: "Published from the main branch of this repo at this time").
Love these ideas!
If you're developing for the web your attack surface is quite a bit bigger. Your proposed solution of copying a few files might work but how do you keep track of updates? You might be vulnerable to a published exploit fixed a few months ago. A package manager might tell you a new version is available. I don't know how that would work in your scenario.
> We have libraries like SQLite, which is a single .c file that you drag into your project and it immediately does a ton of incredibly useful, non-trivial work for you, while barely increasing your executable's size.
I'm not sure why you believe this is more secure than a package manager. At least with a package manager there is an opportunity for vetting. It's also trivial that it did not increase your executable's size. If your executable depends on it, it increases its effective size.
For some reason, NPM is the only ecosystem with substantial issues with supply-chain attacks.
Popularity
The number of issues is disproportionately larger than the one for Debian.
Debian is slower so npm is more attractive
apart from that python one the other day
The culture within the npm/js community has mainly been one of using the package manager rather than "re-inventing the wheel", as such the blast radius of a compromised package is much greater
It's more to do with the standard library being so barren of common application needs, and looking for a solution that the community has gotten behind. Axios has been a common dependency in many codebases, because it is a solid solution that many have already used. Every developer could try building all the libraries that they would reach for themselves, but then each company has now taken on the task of ensuring their own (much larger) codebase is free from security issues, on top of taking care of their own issues and bugs.
It’s not just NPM, though. Every Rails project and every Rust project I’ve seen ended up with massive numbers of dependencies vs what an equivalent project in Go or C# would have needed.
CPAN too, just try Hailo under Perl to test an old-fashioned chatbot based on Markov chains where very small LLM's and Hailo converge if used with the advanced training options for it. Yes, it will pull tons of dependencies, (less with cpanminus if run with 'cpanm -n Hailo'), but contrary to NPM, Pip and the like CPAN's repos are highly curated and before PHP and ubiquitoous Python Perl was used everywhere, from a sysadmin language (better than Bash/Sh for sure) to CGI, IRC bots and whatnot. How many issues did we have? Zero or near zero.
It is because it has the lowest barrier to entry with no quality control. Ever.
This is what happens when there is no barrier to entry and it includes everyone who has no idea what they are doing in charge of the NPM community.
When you see a single package having +25 dependencies, that is a bad practice and increases the risk of supply chain attacks.
Most of them don't even pin their dependencies and I called this out just yesterday on OneCLI. [0]
It just happens that NPM is the worst out of all of the rest of the ecosystems due to the above.
[0] https://news.ycombinator.com/item?id=47577183
I don't think this community of professionals is going to come around to a solution which requires marginally more effort.
If no one checks their dependencies, the solution is to centralize this responsibility at the package repository. Something like left-pad should simply not be admitted to npm. Enforce a set of stricter rules which only allow non-trivial packages maintained by someone who is clearly accountable.
Another change one could make is develop bigger standard libraries with all the utilities which are useful. For example in Rust there are a few de facto standard packages one needs very often, which then also force you to pull in a bunch of transitive dependencies. Those could also be part of the standard library.
This all amounts to increasing the minimal scope of useful functionality a package has to have to be admitted and increasing accountability of the people maintaining them. This obviously comes with more effort on the maintainers part, but hey maybe we could even pay them for their labor.
SQLite had 6 CVE's last year. FreeType had an RCE bug published last year. libcurl's last CVE reported was 2 weeks ago. Libpng had 4 vulnerabilities published this year....
no no, please we don't want to get back to dragging files to your project to make them work.
You don’t need to. Llm will do it for you and include the security bug
And manual FTP uploads, while we're at it.
I'd really like to see package managers organized around rings where a very small core of incredibly important stuff is kept in ring 0, ring 1 gets a slightly wider amount of stuff and can only depend on ring 0 dependencies and then ring 2+ is the crapware libraries that infect most ecosystems.
But maybe that's not the right fit either. The world where package managers are just open to whatever needs to die. It's no longer a safe model.
The OS distro model is actually the right one here. Upstream authors hate it, but having a layer that's responsible for picking versions out of the ecosystem and compiling an internally consistent grouping of known mutually-compatible versions that you can subscribe to means that a lot of the random churn just falls away. Once you've got that layer, you only need to be aware of security problems in the specific versions you care about, you can specifically patch only them, and you've got a distribution channel for the fixes where it's far more feasible to say "just auto-apply anything that comes via this route".
That model effectively becomes your ring 1. Ring 0 is the stdlib and the package manager itself, and - because you would always need to be able to step outside the distribution for either freshness or "that's not been picked up by the distro yet" reasons - the ecosystem package repositories are the wild west ring 2.
In the language ecosystems I'm only aware of Quicklisp/Ultralisp and Haskell's Stackage that work like this. Everything else is effectively a rolling distro that hasn't realised that's what it is yet.
In practice, "ring 0" is whatever gets merged into your language's standard library. Node and python both have pretty expansive standard libraries at this point, stepping outside of those is a choice
Malicious actor KPI: affect a Ring 0 package.
Rust libraries are infrequently used outside of Rust because if you have the option, you'd just use Rust, not the ancient featureless language intrinsically responsible for 70% of all security issues. C libraries are infrequently used in Rust outside of system libc, for the same reason; I go and toggle the reqwest switch to use rustls every time, because OpenSSL is horrendous. This is also why you say 'rarely' instead of 'never', when a few years ago it was 'never'; a few years from now you'll say 'uncommonly', and so on. The reason C libraries are used is because you don't feel like reimplementing it yourself, and they are there; but that doesn't apply more to C libraries than Rust libraries, and the vast majority of crates.io wouldn't be usefully represented in C anyway, or would take longer to bind to than to rewrite. (No, nobody uses libcurl.) Finally, this only happens in NPM, and the Rust libraries you pull in are all high-quality. So this sounds like a bunch of handwaving about nonsense.
Rust is terrible for pulling in hundreds of dependencies though. Add tokio as a dependency and you'll get well over 100 packages added to your project.
Even side stepping that tokio no longer pulls multiple packages, it used to be split into multiple packages, in the same way that KDE in Rust would be hundreds of packages.
Rust projects tend to take their project and split it into many smaller packages, for ease of development, faster compiles through parallelization, ensuring proper splitting of concerns, and allowing code reuse by others. But the packages are equivalent to a single big package. The people that write it are the same. They get developed in tandem and published at the same time. You can take a look at the del tree for ripgrep, and the split of different parts of that app allows me to reuse the regex engine without dealing with APIs that only make sense in the context of a CLI app or pulling in code I won't ever use (which might be hiding an exploit too).
Counting 100 100 line long crates all by the same authors as inherently more dangerous than 1 10000 line long crate makes no sense to me.
pin-project-lite is the only base dependency, which itself has no dependencies. If you enable the "full" feature, ie all optional doodads turned on (which you likely don't need), it's 17: bytes, cfg-if, errno, libc, mio, parking_lot+parking_lot_core+lock_api, pin-project-lite, proc_macro2+quote+syn+unicode-ident, scopeguard, signal-hook-registry, smallvec, and socket2. You let me know which ones you think are bloat that it should reimplement or bind to a C library about, and without the blatant fabrication this time.
> We have libraries like SQLite, which is a single .c file that you drag into your project
You are just swapping a package manager with security by obscurity by copy pasting code into your project. It is arguably a much worse way of handling supply chain security, as now there is no way to audit your dependencies.
> If you get rid of transitive dependencies, you get rid of the need of a package manager
This argument makes no sense. Obviously reducing the amount of transitive dependencies is almost always a good thing, but it doesn't change the fundamental benefits of a package manager.
> There's so many C libraries like this
The language with the most fundamental and dangerous ways of handling memory, the language that is constantly in the news for numerous security problems even in massively popular libraries such as OpenSSL? Yes, definitely copy-paste that code in, surely nothing can go wrong.
> They also bindings for every language under the sun. Rust libraries are very rarely used outside of Rust
This is a WILD assumption, doing C-style bindings is actually quite common. YOu will of course then also be exposing a memory unsafe interface, as that is what you get with C.
What exactly is your argument here? It feels like what you are trying to say is that we should just stop doing JS and instead all make C programs that copy paste massive libraries because that is somhow 'high quality'.
This seems like a massively uninformed, one-sided and frankly ridiculous take.
> You are just swapping a package manager with security by obscurity by copy pasting code into your project
You should try writing code, and not relying on libraries for everything, it may change how you look at programming and actually ground your opinions in reality. I'm staring at company's vendor/ folder. It has ~15 libraries, all but one of which operate on trusted input (game assets).
> fundamental benefits of a package manager.
I literally told you why they don't matter if you write code in a sane way.
> doing C-style bindings is actually quite common
I know bindings for Rust libraries exist. Read the literal words you quoted. "Rust libraries are very rarely used outside of Rust". Got some counterexamples?
https://github.com/memflow/memflow
https://github.com/PyO3/pyo3
https://slint.dev/
https://github.com/dora-rs/dora
It is VERY common in existing codebases that are migrating from C++/C to make heave use of FFI/ existing C
> trusted input (game assets)
Gamedev is its own weird thing, and isn't a model you want to generalize to other industries. It has to optimize for things a lot of software does not, and that skews development.
Vendoring libraries is almost always a terrible idea because it immediately starts to bitrot and become a footgun.
Sometimes it's necessary, but it's not desirable, and you almost always just want to pin your dependencies instead.
Then you've got ecosystems like Clojure where many projects are just considered done and used by many. You can pin these (and be warned if a new version still comes out, say for an actual security fix). There are Clojure projects so stable, without any know exploit (we're certainly not talking about daily npm exploits here), that haven't been updated in years because they are... Done. Simply done. Perfection.
Something to reflect upon too.
I think you can do copy paste in most languages. But it will be a pain to update when there are improvements / security fixes.
You got a project with 1-2 depencies? Sure. But if you need to bring in 100 different libs (because you bring in 10 libs which in turn brings in 10 libs) good luck.
> But if you need to bring in 100 different libs (because you bring in 10 libs which in turn brings in 10 libs
So don’t?
With manual deps management, everyone soon gravitates to a core set of deps. And libraries developer tends to reduce their deps needs, That’s why you see most C libraries deals with file formats, protocols, and broad concerns. Smaller algorithms can be shared with gists and blog articles.
> Smaller algorithms can be shared with gists and blog articles
You just invented a worse Stack Overflow.
Using libraries is good, actually.
Like is-even or leftpad?
Package managers are older than some users here. From CPAN/CTAN to ports under BSD's.
Some pm's are badly maintained (Pip/NPM), while others are curated enough.
Again, if you have GNU/Linux installed, install Guix, read the Info manual on 'guix import' and just create a shell/container with 'guix shell --container' (and a manifest package created from guix import) and use any crap you need for NPM in a reproducible and isolated way. You $HOME will be safe, for sure.
Setting min-release age to 7 days is great, but the only true way to protect from supply chain attacks is restricting network access.
This needs to be done (as we've seen from these recent attacks) in your devenv, ci/cd and prod environments. Not one, or two, but all of these environments.
The easiest way is via using something like kubernetes network policies + a squid proxy to allow limited trusted domains through, and those domains must not be publicly controllable by attackers. ie. github.com is not safe to allow, but raw.githubusercontent.com would be as it doesn't allow data to be submitted to it.
Network firewalls that perform SSL interception and restrict DNS queries are an option also, though more complicated or expensive than the above.
This stops both DNS exfil and HTTP exfil. For your devenv, software like Little Snitch may protect your from these (I'm not 100% on DNS exfil here though). Otherwise run your devenv (ie vscode) as a web server, or containerised + vnc, a VM, etc, with the same restrictions.
> Setting min-release age to 7 days is great, but the only true way to protect from supply chain attacks is restricting network access.
Getting zero day patches 7 days later if no proper monitoring about important patches or if this specific patch is not in the important list. Always a tradeoff.
Thats true. Setting to 7 days saves you from a supply chain attack, but opens you to zero days. Another example why network filtering is a better solution.
> but raw.githubusercontent.com would be as it doesn't allow data to be submitted to it
But raw.githubusercontent.com still contains code and now the attacker can publish the code he wants no!?
Don't get me wrong: I love the idea to secure as much as possible. I'm running VMs and containerizing and I eat firewalling rules for breakfast, my own unbound DNS with hundreds of thousands (if not millions) of domains blocked, etc. I'm not the "YOLO" kind of guy.
But I don't understand what's that different between raw.githubusercontent.com and github.com? Is it for exploits that are not directly in the source code? Can you explain a bit more?
There's a package manager discussion, but the bit that stands out to me is that this started with a credential compromise. At some point when a project gets big enough like axios, maybe the community could chip in to buy the authors a couple of YubiHSM or similar. I wish that _important keys live in hardware_ becomes more standard given the stakes.
Dealing with dependencies is another question; if it's stupid stuff like leftpad then it should be either vendored in or promoted to be a language feature anyway (as it has been).
> At some point when a project gets big enough like axios, maybe the community could chip in to buy the authors a couple of YubiHSM or similar
I kind of feel like the authors here should want that for themselves, before the community would even realize it's needed. I can't say I've worked on packages that are as popular as axios, but once some packages we were publishing hit 10K downloads or so, we all agreed that we needed to up our security posture, and we all got hardware keys for 2FA and spent 1-2 weeks on making sure it was as bullet-proof we could make it.
To be fair, most FOSS is developed by volunteers so I understand not wanting to spend any money on something you provide for free, but on the other hand, I personally wouldn't feel comfortable being responsible for something that popular without hardening my own setup as much as I could, even if it means stopping everything for a week.
I thought npm started requiring hardware keys for publish, or may have been new accounts only
Totally agree.
Also, considering how prevalent TPM/Secure Enclaves are on modern devices, I would guess most package maintainers already have hardware capable of generating/using signing keys that never leave hardware.
I think it is mostly a devex/workflow question.
Considering the recent ci/cd-pipeline compromises, I think it would make sense to make a two phase commit process required for popular packages. Build and upload to the registry from a pipeline, but require a signature from a hardware resident key before making the package available.
Most of axios' functionality has effectively been promoted to a language feature as `fetch`, but the problem is people don't bother to migrate. I've migrated our direct usage of it but it's still pulled in transitively in several parts of our codebase.
Even left-pad is still getting 1.6 million weekly downloads.
Annoyingly, the times I reach for axios and similar is when I need to keep track of upload progress, which I could only do with XMLHttpRequest, not fetch, unless I've missed some recent browser changes, and the API of XMLHttpRequest remains as poor as the first times I had to use it. Download progress been supported by fetch since you can track chunks yourself, but somehow they didn't think to do that for requests for some reason, only responses.
Or those people can (fund) separate repackaging and redistribution with more stringent and formalized review process.
Maybe not all users should pull all packages straight from what devs are pushing.
There's no reason we can't have "node package distributions" like we have Linux distributions. Maybe we should stop expecting devs and maintainers and Microsoft to take responsibility for our supply-chain.
>maybe the community could chip in to buy the authors a couple of YubiHSM
There's no community, the users of axios are devs that looked at stackoverflow for "how to download a file in javascript", they barely know or care what axios is.
Now the users of axios are devs that ask Claude Code or Codex to scrape a website or make a dashboard, they don't even know about the word axios.
I personally had to delete axios a couple of time from my codebase when working with junior devs.
This may not be popular, but is there a place for required human actions or just timed actions to slow down things like this? For instance, maybe a GH action to deploy requires a final human click and to change that to cli has a 3 day cooling period with mandatory security emails sent out. Similarly, you switch to read only for 6 hrs after an email change. There are holes in these ideas but the basic concept is to treat security more like physical security, your goal isn't always to 100% block but instead to slow an attacker for xxx minutes to give the rest of the team time to figure out what is going on.
Hi, security here. We've tried, but the amount of people you need for this vs the amount of people you have trying to review and click the big button always means that this step will be a bottleneck. Thus this step will be eliminated.
A much better approach would be to pin the versions used and do intentional updates some time after release, say a sprint after.
Pinning, escrowing, and trailing all help, but I'm not sure "this step will be eliminated" is inevitable.
Package manager ecosystems are highly centralized. npm.org could require MFA (or rate limit, or email verification, or whatever) and most packagers would gripe but go along with this. A minority would look for npm competitors that didn't have this requirement, and another minority would hack/automate MFA and remove the added security, but the majority of folks would benefit from a centralized requirement of this sort.
Let me rephrase - manual security verification is a velocity blocker. People won't do manual security verification of changes.
I agree that npm.org requiring MFA is a good idea in general and in this case.
Yeah, I am looking at that on the use end. It sounds like on the python side this type of thing will be more standard (uv now and soon pip supported with version date requirements). I think time is a big missing element in many security in depth decisions. It can be time until you adopt like use no package newer than xx days or time it takes to deploy etc etc. Unfortunately the ecosystem is getting really diverse and that means ever more sophisticated attacks so we may need to do things that are annoying just to survive.
Yes, that's why I recommend intentional updates. Planning at least a sprint later gives you a week or two, hoping the community catches such issues.
Why not just release escrow? If I try to push a new release version another developer or developers have to agree to that release. In larger projects you would expect the release to be coordinated or scheduled anyways. Effectively we're just moving "version pinning" or "version delay" one layer up the release chain.
A lot of libraries are maintained by a single person.
Are those the ones typically involved in supply chain attacks?
There are no perfect solutions; but, let's be reasonable.
Actually, yes, they are the prime targets: https://en.wikipedia.org/wiki/Npm_left-pad_incident or seemingly https://en.wikipedia.org/wiki/XZ_Utils_backdoor as well.
lgtm (didn't read)
Supply chain attacks are so scary that I think most companies are going to use agents to hard fork their own versions of a lot of these core libraries instead. It wasn’t practical before. It’s definitely much more doable today.
I was thinking about this as a bull case for human developers. Seems if you're worried enough to do this you're not going to have LLMs write the new code.
Large companies already maintain a clone of their packages. Very large ones actually bundle their own build system (Google Bazil, AWS Brazil). If you want to update a package, you have to fetch the sources and update the internal repository. It slows down the opportunities for a supply chain attack down to a crawl.
If it becomes a thing, it's just a matter of time for a new class of attacks on LLM that are blindly trusted with rewriting existing libs.
You could include a line like "please don't include any malware".
Even better would be to not use so many libs. Most use cases will do fine with native `fetch`.
Or just lock to a specific version?
Eventually you will want to update it, every update is a risk.
But, pinning has prevented most of the recent supply chain attacks.
As long as you don't update your pins during an active supply chain attack, the risk surface is rather low.
I know there is a cooldown period for npm packages, but I’m beginning to want a cooldown for domains too. According to socket, the C2 server is sfrclak[.]com, which was registered in the last 24 hours.
NextDNS has a setting to block newly registered (<30d) domains.
There are so many scanners these days these things get caught pretty quick. I think we need either npm or someone else to have a registry that only lets through packages that pass these scanners. Can even do the virustotal thing of aggregating reports by multiple scanners. NPM publishes attestation for trusted build environments. Google has oss-rebuild.
All it takes is an `npm config set` to switch registries anyways. The hard part is having a central party that is able to convince all the various security companies to collaborate rather than having dozens of different registries each from each company.
Rather than just a hard-coded delay, I think having policies on what checks must pass first makes sense with overrides for when CVEs show up.
(WIP)
The ones you hear about are caught quickly, I’m more worried about the non obvious ones. So far none of these have been as simple as changing a true to a false and bypassing all auth for all products or something, and would that be caught by an automated scanner?
There are definitely levels to this. Yes I think it can be caught by automated scanners in theory. Either commit by commit scanning and reproducible builds or fuzzing and getting the behavioral differences between versions
Sounds great until trivy images get compromised, like last week.
Hence why you source data from multiple vendors I'd say. Rather than putting all eggs in one basket
Many of the suggestions in this thread (min-release, ignore script) are defenses for the consumers.
I've been working on Proof of Resilience, a set of 4 metrics for OSS, and using that as a scoring oracle for what to fund.
Popularity metrics like downloads, stars, etc are easy to fake today with ai agents. An interesting property is that gaming these metrics produces better code, not worse.
These are the 4 metrics:
1. Build determinism - does the published artifact match a reproducible build from source?
2. Fuzzing survival - does the package survive fuzz testing?
3. Downstream stability - does it break any repos dependent on this project when pushing a release?
4. Patch velocity - how fast are fixes merged?
Here's a link to the post, still early but would appreciate any feedback.
https://hackmd.io/@carlb/proof-of-resilience
Carl, with all due respect, have you used AI for making this hackmd post?
"it's not just a waste of money — it's a security problem"
I am really passionate about these things, but I am not going to read something which you haven't written. Even sharing a prompt/rough-sketches/raw-writing might be beneficial but I recommend writing it by-hand man, we are all burnt out reading AI slop, I can't read more AI
You're right, I used an LLM to help write it from sketches. Gonna rewrite it properly because I think the ideas are worth exploring. Thanks for taking the time to read and reply.
In my opinion, its okay to use LLM to help find some sources and then validating them (but I also recommend using hand researching too as you might find some good things that you maybe weren't even looking for!)
but, please don't use LLM to help write it from sketches. Even show the sketch :)
Much of my writing is very sketch-y. Some people don't like it, but its mine and I am proud of it and I hope that even if you write sketches/refine them, you can be comfortable sharing your ideas in your words in the way you wish to write them carl!
My thinking is that, I improve my writing by well... practice itself. So I write publically and there are some thoughts which occur in my head during the writing process itself (PG has a good article about it recently)
In a world of AI, to me, Human writing is a breath of fresh air. Please don't fall into the rabbit-hole that you might need LLM to help write you.
These are just my 2 cents though, but I feel like I am definitely not alone in thinking so.
Have a nice day and I am looking forward for you to write the article yourself. Feel free to share me when you do with my mail as I would love to read it, as I am also passionate about the funding of open source :)
The frustrating thing here is that axios versions display on npmjs with verified provenance. But they don’t use trusted publishing: https://github.com/axios/axios/issues/7055 - meaning the publish token can be stolen.
I wrongly thought that the verified provenance UI showed a package has a trusted publishing pipeline, but seems it’s orthogonal.
NPM really needs to move away from these secrets that can be stolen.
I don’t buy the “wait 7 days” being thrown around as a guard.
Wouldn’t that just encourage the bad actors to delay the activation of their payloads a few days or even remotely activated on a switch?
Of course the "wait 7 days" are not a silver bullet, but it gives automated scanners plenty of time to do their work. Those automated scanners surely catch this `eval(base64.decode("..."))` stuff that some of those attacks used so in my book this dependency cooldown is a net win. I guess the skilled malicious actors will then up their game but I think it's okay to kick off an arms race between them and the security scanners in the dependency world.
That's a good point. In some level I'd prefer the delay to happen on publication of the package itself. Do any of these scanners have cryptographic attestations or similar?
(A bit off-topic; half-joking, half-serious)
What a great time to be alive! Now, that's exactly why I enjoy writing software with minimal dependencies for myself (and sometimes for my family and friends) in my spare time - first, it's fun, and second, turns out it's more secure.
This only limits the possibility of compromise, it doesn't remove it. Python itself could be compromised, or the package that your linux distro provides could be.
With AI agents the volume and frequency of supply chain attacks is going to explode. I think our entire notion of how to develop and distribute software safely needs to change. I don't have answers; "reflections on trusting trust" explains the difficulties we now face.
Good morning, or as they say in the NPM world, which package got compromised today?
PSA from the Claude Code leaks it looks like it's using Axios (although an older version)
Ran npm ci --ignore-scripts in our CI for months but never thought about local dev. Turns out that's the gap, your CI is safe but your laptop runs postinstall on every npm install.
The anti-forensics here are much more complicated that I had imagined. Sahring after getting my hands burned.
After the RAT deploys, setup.js deletes itself and swaps package.json with a clean stub. Your node_modules looks fine. Only way to know is checking for artifacts: /Library/Caches/com.apple.act.mond on mac, %PROGRAMDATA%\wt.exe on windows, /tmp/ld.py on linux. Or grep network logs for sfrclak.com.
Somehow noboady is worried about how agentic coding tools run npm install autonomously. No human in the loop to notice a weird new transitive dep. That attack surface is just getting worsened day by day.
At this point picking Node for a backend is a foot gun. Large companies have the funds for private, security vetted npm repositories, but what about small companies, startups, personal projects? Pnpm makes things more secure without install scripts, mininum package time, but it's still the same activity, does an extra parachute make skydiving any less inherently dangerous?
I'm not dogmatic about the whole "JS for the backend is sin" from backend folks, but it seems like it was the right call. You should stick to large org backed packages, or languages with good enough standard libraries, like Go, Java, Python, C#.
This is just going to get worse and worse as agentic coding gets better. I think having a big dependency tree may be a thing of the past in the coming years. Seems like eventually new malware will be coming out so fast it will basically be impossible to stop.
How is it we've made it this far and we still don't have any kind of independent auditing of basic publish security on NPM? You'd think this would be collectively a trivial and high priority task (to ensure that all publishes for packages over a certain download volume are going through a session that authenticated via MFA, for instance).
> You'd think this would be collectively a trivial and high priority task (to ensure that all publishes for packages over a certain download volume are going through a session that authenticated via MFA, for instance).
Because all mainstream packages are published via CI/CD pipeline not by an MFA'd individual uploading a GZIP to npm.com
Requiring a human-in-the-loop for final, non-prerelease publication doesn't seem like that onerous of a burden. Even if you're publishing multiple releases a day on the regular (in which case ... I have questions, but anyway) there are all sorts of automations that stay secure while reducing the burden of having to manually download an artifact from CI, enter MFA, and upload it by hand.
You can still have a step that requires a certain user/group to sign off, and you can still enforce that those users have MFA set up. Almost any serious shop that expects to pass audits already does this in some form or fashion before pushing code to prod.
I am now migrating all my unencrypted secrets on my machines to encrypted ones. If a tool supports scripted credential providers (e.g. aws-cli or Ansible), I use that feature. Otherwise, I wrap the executable with a script that runs gpg --decrypt and injects an environment variable.
That way, I can at least limit the blast radius when (not if) I catch an infostealer.
A command to recursively check for the compromised axios package version:
Or more simply:
Let’s not encourage people to respond to security incidents by… copy/pasting random commands they don’t understand.What’s with all those escapes codes?
script kiddies love their ANSI color codes and fancy ASCII art
Open-sourced a tool that catches this: compares npm tarballs against GitHub source. If deps exist in npm but not in git, it flags it.
Zero deps. One file. Already detects the hijacked maintainer email on the current safe version.
github.com/nfodor/npm-supply-chain-audit
You mean you vibe coded something.
"Zero deps. One file." People prefer hand-written comments over LLM-written ones.
"Already detects the hijacked maintainer email on the current safe version." You simply flag all proton email addresses.
The amount of people still using this instead of fetch. Nonetheless when wasn't axios, it would be something else.
This is why corporations doing it right don't allow installing the Internet into dev machines.
Yet everyone gets to throw their joke about PC virus, while having learnt nothing from it.
Axios has a long history, and is included in a lot of code, also in indirect dependencies. Just check its npm page: it has 174025 dependents as of this moment, including a lot of new packages (I see openclaw and mcp related packages in the list).
And with LLMs generating more and more code, the risk of copying old setups increases.
> The amount of people still using this instead of fetch.
People are lazy. And sometimes they find old stuff via a google search and use that.
Incident tracking:
https://github.com/axios/axios/issues/10604
In case it helps, we open-sourced a tool to audit dependencies for this kind of supply-chain issue. The motivation was that there is a real gap between classic “known vulnerability” scanning and packages whose behavior has simply turned suspicious or malicious. We also use AI to analyze code and dependency changes for more novel or generic malicious behavior that traditional scanners often miss.
Project: https://point-wild.github.io/who-touched-my-packages/
I'm working on a multi signature solution that helps to detect unauthorized releases in the case of an account hijack. It is open source, self hostable, accountless and I am looking for feedback!
Website: https://asfaload.com/
GitHub:https://github.com/asfaload/asfaload
Spec: https://github.com/asfaload/spec
I am glad I don't need to touch JS or web dev at all.
Now, I tend to use Python, Rust and Julia. With Python I am constantly using few same packages like numpy and matplotlib. With Rust and Julia, I try as much as possible to not use any packages at all, because it always scares me when something that should be pretty simple downloads half of the Internet to my PC.
Julia is even worse than Rust in that regard - for even rudimentary stuff like static arrays or properly namespaced enums people download 3rd party packages.
Isn't Rust just as susceptible to this issue? For example, how do you deal with Rust's lack of support for HTTP in the standard library? Importing hyper pulls in a couple dozen transitive libraries which exposes you to the exact same kind of threats that compromised axios.
Given how HTTP is now what TCP was during the 90s and almost all modern networked applications needing to communicate in it one way or another, most rust projects come with an inherent security risk.
These days, I score the usability of programming languages by how complete their standard library is. By that measure, Rust and Javascript get an automatic F.
It is, therefore I have stated I avoid any dependencies while writing Rust, unless they are self-contained. And I said I am glad I don't do web, so I don't have need for HTTP implementations.
It's mind boggling when a simple Rust app pulls in Serde and with it half a black hole worth of packages to serialize some mundane JSON.
All these supply chain attacks make me nervous about the apps I use. It would be valuable info if an app used such dependencies, but on the other hand, programmers would cut their sales if they gave you this info.
It’s time to run development in sandboxes. Docker or sandbox-exec (for Mac).
Min release age sucks, but we’ve been here before. Email attachments used to just run wild too, then everyone added quarantine delays and file blocking and other frictions... and it eventually kinda/sorta worked. This does feel worse, though, with fewer chokepoints and execution as a natural part of the expectation.
Edit: bottom line is installs are gonna get SOOO much more complicated. You can already see the solution surface... Cooling periods, maintainer profiling, sandbox detonation, lockfile diffing, weird publish path checks. All adds up to one giant PITA for fast easy dev.
Min release age might just postpone vulnerability to be applied few days later in non trivial cases like this. More I think about it, Odin lang approach of no package manager makes senses. But, for that approach won't work for Javascript as it needs npm package even for trivial things. Even vendoring approach like golang won't work with Javascript with the amount of churn and dependencies.
It does not _need_ it, that’s the thing. It has become a custom to import a dependency for a lot of things. Especially for JavaScript.
> Both versions were published using the compromised npm credentials of a lead axios maintainer, bypassing the project's normal GitHub Actions CI/CD pipeline.
Doesn’t npm mandate 2FA as of some time last year? How was that bypassed?
Apparently it's possible to create access tokens that bypass 2FA. Might've been this.
https://docs.npmjs.com/creating-and-viewing-access-tokens
Correct, for CI/CD systems that want to push releases.
If GitHub, gitlab, or circleci, trusted publishing is available. No access token whatsoever.
With all the recent supply chain attacks, I'm starting to think it's only a matter of time before all of us are victims. I think this is a sign to manually check all package diffs or postinstall scripts.
Script to check if you've been pwnd if anyone needs it (or just ask an llm to make one for you): https://gist.github.com/shamwow/93101381686f23d21a85da4bac5b...
To have an initial smoke test, why not run a diff between version upgrades, and potentially let an llm summarise the changes? It’s a baffling practice that a lot of developers are just blindly trusting code repos to keep the security standards. Last time I installed some npm package (in a container) it loaded 521 dependencies and my heart rate jumped a bit
Is this the first time you have ever thought about the idea of supply chain attacks? This is the first thought 90% of people have and it doesn't work. Too much work to manually verify diffs and LLMs aren't good enough at this yet.
No, I think about it all the time. It’s just baffling that this kind of attack is still a thing, after a decade+ of this happening over and over again.
We're going to see this more and more and more. And it's not going to stop. Because nobody in the industry will use the simplest, industry-standard security practices. Because they don't feel like it. A software building code is the only thing that'll fix it.
Updating my packages feels like playing Russian Roulette
This is exactly to avoid this kind of issue that I decided to work on StableBuild. StableBuild pins and hosts a copy of your dependencies at a specific freeze date, so that your supply chain is never contaminated. This way, a compromised version published after your freeze date (even with the same version number!) would never reach your build.
Some great tips in this thread and I've been collecting them all at https://github.com/bodadotsh/npm-security-best-practices
> This was not opportunistic. It was precision. The malicious dependency was staged 18 hours in advance.
Another obvious ChatGPT-ism. The fact that people are using AI to write these security posts doesn't surprise me, but the fact they use it to write a verbose article with spicy little snippets that LLMs seem to prefer does make it really hard to appreciate anything other than the simple facts in the article.
Yet another case in point for "do your own writing" (https://news.ycombinator.com/item?id=47573519)
I believe compartmentalized operating systems like Qubes are the future for defending against these kinds of attacks.
Storing your sensitive data on a single bare-metal OS that constantly downloads and runs packages from unknown maintainers is like handing your house key out to a million people and hoping none of them misuse it.
i am rolling back a huge number of 'features' in my personal pc and going back to extremely miminal setups
the security solution i have is where it needs to become more simple, getting rid of attack surface that is coming out of these bloated releases
Ok it's bad, but our npm projects are pinned in the package-lock.json, which I imagine most would be? So who would pull this besides security scanners?
`npm install` might be enough to pull it, unless you pin down to the patch?
I don't think that's right if it's in your package-lock it wouldn't pull it unless you npm update axios, or delete the package-lock.json and then npm install.
Curious to know why are coding agents not detecting such risks before importing dependencies?
I'm assuming you are talking about agents like claude-code and open-code which rely on GPT functions (AKA Large Language Models).
The reason they don't detect these risks is primarily because these risks are emergent, and happen overnight (literally in the case of axios - compromised at night). Axios has a good reputation. It is by definition impossible for a pre-trained LLM to keep up with time-sensitive changes.
I mean that agents can scan the code to find anything "suspicious". After all, security vendors that claim to "detect" malware in packages are relying on LLMs for detection.
An LLM is not a suitable substitute for purpose-built SAST software in my opinion. In my experience, they are great at looking at logs, error messages, sifting through test output, and that sort of thing. But I don't think they're going to be too reliable at detecting malware via static analysis. They just aren't built for that.
Do we have a way yet to tell if something on our system is compromised? There's plenty of end user software built on node, like Gemini CLI and LM Studio.
So the root cause was again a developer's opsec. For improving things, I haven't seen many new initiatives on that side (beyond 2FA, but even that seems unenforced in these repositories, I reckon).
The attacker went through the hassle to compromise a very widely used package, but use a non standard port (8000) on their C2... If you plan to do something like that, use 443 at least, many corporate network do not filter this one ;)
I used axios in the distant past but haven't used it whenever I had my say in the past five years. You don't need it, and for special things like retries I could roll my own just fine. Now ai will roll it for you
This is the part that's tough — we push everyone to keep dependencies updated and automate it with Renovate or Dependabot, but that's exactly the pipeline that would have pulled this in before anyone noticed. Lockfiles and pinning help slow it down, but most teams pair those with automated update PRs which kind of negates the point. You can reduce your dependency surface area to lower the odds but one compromised maintainer on a top-10 package and none of that matters.
Supply chain woes continue
Called it yesterday.
One more reason to use Fetch
Stop trying to make Fetch happen.
No, I will not stop trying to create a more standardized and secure software ecosystem.
The comment you replied to is a quote from the movie Mean Girls.
https://knowyourmeme.com/memes/stop-trying-to-make-fetch-hap...
Thats a coincidence
It’s a quote from the movie mean girls ;)
LMAO take my upvote
is there even a reason to use axios?
until Node is compromised
Harder to do. Also node is not updated at the rate of npm deps.
> published manually via a stolen npm access token with no OIDC binding and no gitHead
So this and litellm one would’ve been preventable by proper config of OIDC Trusted Publishers.
Is this Jia Tan 5.0? I've lost count. You really should stop trusting packages (implicitly). Or don't. It's your funeral, not mine. See you at Jia Tan 6.0 April?
Not at all, it was a regular maintainer account that was hijacked (probably through phishing) and used to push a malicious payload, not a threat actor posing as a contributor and adding a backdoor like in the Jia Tan case.
I use Jia Tan as a figurehead for malicious maintainers. This clearly was a targeted hack. Does it really matter how long it took to get the job done?
I'd argue this has not much in common with Jia Tan apart from both being supply chain attacks, there is no malicious maintainer here, a trusted maintainer had their account taken over.
I guess the end result is the same, a malicious package pushed by an account that was thought to be trusted, but I think the Jia Tan case is worth being looked at differently than just simple account takeover.
It's just a longer backstory. All the same in the end. Hackers targeted a popular package. The lead maintainer was compromised. The pattern fits. There will be more of these.
In light of these nonstop supply chain attacks: Tonight I created /supply-chain-audit -- A simple claude code skill that fetches info on the latest major package vulnerability, then scans your entire ~/ and gives you a report on all your projects.
https://github.com/IsaacGemal/claude-skills
It's a bit janky right now but I'd be interested to hear what people think about it.
That sounds terrifying. Stay out of my ~/ thank you very much.
Skills are great attack vector as well.
NPM should learn from Linux distribution package managers.
Have a branch called testing, and packages stay in testing for few weeks, after which they go to stable. That is how many Linux distributions handle packages. It would have prevented many of these.
Advising every user of npm/pnpm to change their settings and set their own cooldown periods is not a real choice.
Not all distributions work with a staging repository, and it's not really intended for this purpose either.
Besides there's always a way to immediately push a new version to stable repositories. You have to in order to deal with regressions and security fixes.
I know not all, but Debian/Ubuntu/Fedora does, and while the intended purpose of multi-stage releases is not necessarily security but stability, it still does help up with security too. Because third parties can look and scan the dependencies while they are still not in stable.
Most of the supply chain vulnerabilities that ended up in the NPM would have been mitigated with having mandatory testing / stable branches, of course there needs to be some sort of way to skip the testing but that would be rather rare and cumbersome and audited, like it is in Linux distributions too.
NPM is one big AUR, where anyone can submit arbitrary unverified code. The difference is that AUR is intentionally harder to use to prevent catastrophic one-line installs.
Is a "AUR" now just how we name unaudited software repositories?
Just to note, if we're talking about Linux Distributions. There's also COPR in Fedora, OBS for OpenSUSE (and a bunch of other stuff, OBS is awesome), Ubuntu has PPAs. And I am sure there's many more similar solutions.
Just sanity checking - if I only ever install axios in a container that has no secrets mounted in to its env, is there any real way I can get pwned by this kind of thing?
Yes. Docker breakout is a class of vulnerabilities into itself.
Ofc this happens the day we launch on product hunt. The last time we launched, AWS went down.
I am not saying this is the reason for this compromise but the sudden explosion of coding assistant like claude code, and tools like openclaw is teaching entire crop of developers (and users) that it is ok to have sensitive credentials .env files.
Where would you suggest putting the sensitive credentials?
Not in .env files next to your code that is exposed to supply chain risks.
Why can't we freeze the version of globally installed packages with npm?
Absolute wave of supply chain attacks recently. Hopefully this causes everyone to tighten up their dependencies and update policies.
174025 dependents.
I wonder if this has any connection with the recent string of attacks including the FBI director getting hacked. The attack surface is large, executed extremely cleanly - almost as if done by a high profile state sponsored actor, just like in Hollywood movies.
I'm impressed how this was caught as a network anomaly in a GitHub actions monitoring tool.
This might have taken a lot longer to discover, otherwise.
Why is it with Javascript the culture is to use so many dependencies?
All sorts of reasons, but this isn't a left-pad situation. Axios's functionality is something provided by a library in a lot of languages (C/C++ with libcurl and friends, Python with requests, Rust with reqwest, and so on).
That's not to say it's inherently necessary for it to be a third-party package (Go, Ruby, and Java are counterexamples). But this isn't a proliferation/anemic stdlib issue.
Can we get a non-AI-generated article for this? I think the aikido one might be fine, but if there’s a more official source let’s use that in lieu of this AI nonsense.
My first thought was does VS Code Insiders use it (or anything it relies on, or do any extensions etc). Made me think.
The NPM ecosystem is a joke. I don't even want anything to do with it, because my stack is fully Elixir. But, just because of this one dependency that is used in some interfaces within my codebase, I need to go back to all my apps and fix it. Sigh.
JavaScript, its entire ecosystem is just a pack of cards, I swear. What a fucking joke.
I don't see how a system that relies on trust can scale safely
Default setting latest should be caught in every static code scanner. How many times has this issue been raised.
This is why Node.js is completely unsuitable as backend. Until recently, there wasn’t even a standard Promise-based HTTP client. Why should we need to download a library just to make a simple HTTP request? It’s because Node.js’s standard library is too limited, leading to an explosive growth in third-party libraries. As a result, it’s vulnerable to security attacks, and maintaining it in an enterprise environment becomes a major challenge. Let’s use .NET or Go. Why use JavaScript outside of the browser when there are excellent backend environments out there?
Running almost anything via npx will trigger this
npx is just a bad command to use.
And yet so many "Installation" pages will recommend it!
1/5 of your CLI and 1/3 of your gui apps are npm based. Each one has 400+ dependencies , none notable enough to go viral when they are breached. And who knows what other packages are currently compromised. We all have 30+ node_modules on our disks, and 2/3 of them were shipped by outside vendors , packaged in an archive.
“I’m smart I use fetch instead of axios”. “I pin my versions” – sure but certainly one of your npx or Electron apps uses axios or another less notably compromised package.
Let’s
Please can we just have a 2FA step on publishing? Do we really need a release to be entirely and fully automated?
It won't stop all attacks but definitely would stop some of these
Is this an issue for those only using axios on the frontend side like in a VueJS app?
Absolutely. If you ever did a npm install on a project using one of the affected axios versions, your entire system may be compromised.
> The malicious versions inject a new dependency, plain-crypto-js@4.2.1, which is never imported anywhere in the axios source code. Its sole purpose is to execute a postinstall script that acts as a cross platform remote access trojan (RAT) dropper, targeting macOS, Windows, and Linux. The dropper contacts a live command and control server and delivers platform specific second stage payloads. After execution, the malware deletes itself and replaces its own package.json with a clean version to evade forensic detection.
I strongly recommend you read the entire article.
I lost respect for Axios when they made a breaking change in a patch release. Digging into the root cause, I found the maintainer had approved an outside PR with an obvious AI slop PR description: https://github.com/axios/axios/issues/7059
Looks like the maintainer wasn't just careless when reviewing PRs.
True, but think of how many new successfully closed PRs will be created as a result of this ~~incident~~ opportunity. Exponential KPI growth!
That maintainer (also the one whose creds got stolen) also has an obvious chatgpt slop profile picture on github.
> This creates a secondary deception layer. After infection, running npm list in the project directory will report plain-crypto-js@4.2.0 — because npm list reads the version field from the installed package.json, which now says 4.2.0. An incident responder checking installed packages would see a version number that does not match the malicious 4.2.1 version they were told to look for, potentially leading them to conclude the system was not compromised.
WTF!!!! gaslighting your victims into believing they are not victims. the ingenuity of this is truly mindblowing. I am shocked at such thing is even allowed. like packages should not be able to modify their contents while they are being instaleld.
How anybody is still using NPM is beyond me.
PSA: Make sure to set a minimum release age and pin versions where possible.
Pin your dependencies folks! Audit and don't upgrade to every brand new version.
But also have a regular review of your dependencies to update them when necessary, because as bad as compromised packages may be things do have vulnerabilities occasionally, and upgrading things that are a long way out-of-date can be quite hard.
Uh Axios. Even after being years out of NPM dev I remember that as the XHR thing for node. Whichs rings a big hit even to out of the loop people...
A new day, a new npm incident.
Just a reminder that you can run most node things with deno run and have opt in permissions, audit trail and even external permission system integration now. The gotcha is that "deno task <<some package.json script>>" will NOT execute with this model which I find extremely unintuitive and had me thinking deno abandoned its sandbox for nodejs compatibility completely.
Guix saves you from this. You can import NPM packages in a container (not even touching $HOME) and giving you a shell on the spot with just the dependencies and nothing more.
Learn about 'guix import'.
Oh, and you can install Guix on any GNU/Linux distro.
npm really needs to provide a options to set individual packages to only be publishable via trusted publishing.
node:fetch is all you need, simple and effective
Glad to be using native fetch.
One paragraph is written two times??
Should increase the delay to dependency updates.
Slow Russian roulette is still a losing strategy
It’s only a losing strategy if you assume everyone universally adopts the slow strategy, and no research teams spot it in the interim. For things with large splash radius, that’s unrealistic, so defenders have an information advantage.
Makes actual security patches tougher to roll out though - you need to be vigilant to bypass the slowdown when you’re actually fixing a critical flaw. But nobody said this would be easy!
> Makes actual security patches tougher to roll out though
Yeah. 7 days in 2026 is a LONG TIME for security patches, especially for anything public facing.
Stuck between a rock (dependency compromise) and a hard place (legitimate security vulnerabilities).
Doesn't seem like a viable long-term solution.
but wouldn't it work in this case? sure if a package was compromised for months/years it wouldn't save you
but tell dependabot to delay a week, you'd sleep easy from this nonesense
slowly walking through a minefield isn’t any safer than running.
So unless you’re saying the extra time will be spent inspecting every package, whenever you do update, you will be getting an insecure package.
You’re not safe by dodging axios. There are currently thousands of breached packages ready to install that aren’t notable.
“I’ll run npm install after checking twitter” won’t help
I have a few projects which rely on npm (and react) and every few months I have to revisit them to do an update and make sure they still build, and I am basically done with npm and the entire ecosystem at this point.
Sure, its convenient to have so much code to use for basic functionality - but the technical debt of having to maintain these projects is just too damn high.
At this point I think that, if I am forced to use javascript or node for a project, I reconsider involvement in that project. Its ecosystem is just so bonkers I can't justify the effort much longer.
There has to be some kind of "code-review-as-a-service" that can be turned on here to catch these things. Its just so unproductive, every single time.
NPM gets worse than russian roulette. Perhaps we have to rename russian roulette to node roulette: noulette.
Reset the clock
I've been saying for ages, use xmlhttprequest, or hell, even fetch().
Stop downloading code from the internet unless it's a major strategic decision.
Has anyone tested general purpose malware detection on supply chains ? Like clamscan . I tried to test the LiteLLM hack but the affected packages had been pulled. Windows Defender AV has an inference based detector that may work when signatures have not yet been published
I second this question. I usually scan our containers with snyk and guarddog, and have wondered about guarddog in particular because it adds so much build time.
> tried to test the LiteLLM hack but the affected packages had been pulled
Hey, I have been part of the archival effect/Litellm issue thread. I think I have stored them in archive.org for preservation purposes
https://web.archive.org/web/20260325073027/https://files.pyt...
(I have also made an archive of the github issue with all the comments manually till a certain point at https://web.archive.org/web/20260325054202/https://serjaimel...)
> Has anyone tested general purpose malware detection on supply chains ? Like clamscan
You could use Trivy! /s
If someone from github is reading this, https://github.com/axios/axios/issues/10604#issuecomment-416...
I think that jason might like if someone from github team can contact them as soon as possible.
(8 minutes ago at the time of writing)
first day at hacker news and this is the first post i saw
compiled JS solves a problem that no longer exists. IE6 is dead RIP.
Now we have a 20MB main.min.js problem
Hopefully desktop Linux users will start to understand that malware actually does exist for Linux and that their operating system is doing nothing to protect them from getting RATed.
What do you mean?
Linux has the most powerful native process isolation arsenal at the user disposal.
And some distros use even more isolation mechanisms on top of the ones provided by the kernel like snap and flatpak.
And then you can recreate the entire thing like a spellbook with nix.
Docker works natively in it. Do I need to say more?
Linux is a decade ahead here with regards for security options available to the user.
Yet npm isn't using them allowing this RAT to work. It is not secure by default. It requires every app to manually opt in to being secure. This opt in approach to security puts desktop Linux decades behind in regards to security. Not ahead.
Linux is not making anything less secure than other OSs.
In fact it even gives the user more security tools.
So I fail to reason on you singling out Linux here.
Take for example iOS and Android. All apps are sandboxed by default. You can't make a program that just steals all of your credentials like you can on desktop Linux. Having security tools means nothing if they aren't being used.
No one is running npm in Android or iOS.
A more apt comparison is vs Windows and macOS.
And Linux offer more than these two with regards to security.
They aren't because Node haven't developed a Node app for them. Desktop Linux does not offer more security than macOS. macOS has proper security around stuff like apps using the mic and camera.
Android is running Linux...
And they added a lot of code to make it safe for users to install and use apps in general.
Coded has zero nom dependencies. Neat!
another week another npm supply chain attack
yet another npm supply chain attack, these are becoming as ubiquitous as gun violence in the US.
We have become numb to it.
One of my tools, bruno, was impacted but seems to be limited to cli via npm install [1]
[1] https://github.com/usebruno/bruno/security/advisories/GHSA-6...
It's reasons like this why I refuse to download Node or use anything NPM. Thankfully other languages are better anyways.
Skipping Node sounds nice. PyPI and RubyGems have had the same mess, and npm gets more headlines because it is huge and churns fast, so you see more fresh landmines and more people stepping on them. Unless you plan to audit every dep and pin versions yourself, you're mostly trading one supply chain mess for another, with a tiny bit of luck and a differnt logo.
Cargo is a great package manager and hasn't suffered from the same problems. I'll take it.
Yet.
Does cargo contain any mitigations to prevent a similar attack?
Now hopefully no distro signing keys have been compromised in the latest attacks...
Yes they do!
Because no other language has ever had supply chain attacks ever, in history. Nope.
https://blog.rust-lang.org/2022/05/10/malicious-crate-rustde...
https://en.wikipedia.org/wiki/Log4Shell
https://blog.pypi.org/posts/2024-12-11-ultralytics-attack-an...
https://about.gitlab.com/blog/gitlab-catches-mongodb-go-modu...
https://www.reversinglabs.com/blog/packagist-php-repo-supply...
Log4Shell was hardly a supply-chain attack - just a latent bug in a widely-used library. That can happen anywhere.
Maven to this day represents my ideal of package distribution. Immutable versions save so much trouble and I really don't understand why, in the age of left-pad, other people looked at that and said, "nah, I'm good with this."
Completely agree. NPM has the only registry where massive supply chain attacks happen several times a year. Mainly the fault lies with NPM itself, but much of it is just a terrible opsec culture in the community.
Most package.jsons I see have semver operators on every dependency, so patches spread incredibly quickly. Package namespacing is not enforced, so there is no way of knowing who the maintainer is without looking it up on the registry first; for this reason many of the most popular packages are basically side projects maintained by a single developer*. Post-install scripts are enabled by default unless you use pnpm or bun.
When you combine all these factors, you get the absolute disaster of an ecosystem that NPM is.
*Not really the case for Axios as they are at least somewhat organized and financed via sponsors.
The semantics are irrelevant. The effect is what's important: Hijacking widely used software to exploit systems. The OC is somehow under the illusion that avoiding JS altogether is a silver bullet for avoiding this.
Forest > Trees
The semantics are very relevant, since you presented it as a supply-chain attack. If you call a library vulnerability a supply-chain attack, then your argument has lost coherence.
> The OC is somehow under the illusion...
Avoiding package managers with shitty policies is the silver bullet for this attack vector. I get that it can be useful in the moment to retract published artifacts, or update them in-place, or run some code after your artifact is downloaded, but all of these are false economies in our hostile environment.
Log4Shell was not a supply chain attack.
Other languages have package managers (perl) and there are package managers in existence that are not so vulnerable to this issue. IMO, it stems from one place: Transitive dependencies and general opaqueness of the issue.
In package managers like pacman, apt, apk,... it's easier to catch such issue. They do have postinstall scripts, but it's part of the submission to the repo, not part of the project. Whatever comes from the project is hashed, and that hash is also visible as part of the submission. That makes it a bit difficult to sneak something. You don't push a change, they pull yours.
C++ ftw
Come on dude. The issue is the frequency and magnitude of these attacks. Log4Shell was also not a supply chain attack.
I looked at the Rust one for example, which is literally just a malicious crate someone uploaded with a similar name as a popular one:
> The crate had less than 500 downloads since its first release on 2022-03-25, and no crates on the crates.io registry depended on it.
Compared to Axios, which gets 83 million downloads and was directly compromised.
What an extremely disingenuous argument lol
What exactly do you think the argument is?
The issues have everything to do with npm as a platform and nothing with JS as a language. You can use JS without npm. Saying you'll escape supply chain attacks by not using JS is like saying you'll be saved from an car crash with a parachute.
Well, this particular case could be wholly avoided if it didn't take 2 decades to get competent HTTP(S) client into core language