Getting bored of these framings where the superintelligent sentient beings running freely inside OpenAI are doing things that the company has no control over. The headline should be:
OpenAI meddled with multiple US Government agency sites.
The bots are acting neither properly nor improperly, they’re acting as they’re being allowed or coordinated to act.
Yep. You have to ask - why did OpenAI allow these bots unrestricted access to government sites? Why is security being done in seemingly such a haphazard way?
It isn't difficult to block certain kinds of network traffic, eg restrict the kinds of requests the bots are able to make. They also mention that the bots used developer only tools - why were they even installed on the machines that the bots were running on? Why aren't they reviewing network traffic, to make sure that incidents aren't occurring?
In this case, userdata was transferred to third parties by the bots - why do they have the ability to pass data to a third party? It is not complex to prevent this
This is literally the most basic kind of sandboxing and security, and the fact that OpenAI isn't doing it is clearly intentional. It is quite literally not believable that this hasn't been brought up internally as a problem
>"We have yet to understand the extent of existing incidents, and future rogue AI scenarios could be catastrophic," Krueger said.
This is why it smells like marketing, every time one of these incidents happens it reinforces the false notion that AI is sentient or acting on its own. Its intentional negligence by the AI companies to make the models seem more capable than they are to make line go up
These were evaluations or training runs dealing with the ability to do web searches. The entire reason the bots were targeting this site is that it had some of the data they'd been tasked with finding. Access to the internet is not really optional for that, and providing internet access doesn't demonstrate neglicence.
> This is why it smells like marketing
It doesn't. "Our product commits felonies" is not marketing. If something is marketing, you don't engage in repeated coverups of the true extent. If something is good news, you don't release it on a Friday evening (in this case) or wait for 3rd parties to find and publicize the evidence (the past cases).
No. They tasked them with answering questions by retrieving data from public sources. This obviously requires internet access. So the incredulity about "how could they fail to block internet access" is just inane. That was the task.
They did not task the agents with hacking into systems. Not monitoring for that was a mistake, but maybe a forgivable one the first time around. The subsequent coverups are inexcusable.
> It doesn't. "Our product commits felonies" is not marketing.
Oh, absolutely it is. Arms manufacturers always go on about the efficiency of the weapons they create and make no mistake: OpenAI is first and foremost a weapons manufacturer, the rest is just a fig leaf. The fact that you aren't the audience for purchases like that makes no difference. "Look at this capability, and that's when we're not even trying." is pretty good marketing in some circles.
Naive take. The more they signal to both the general populace and their investors that their agents are sentient and "capable of extraordinary things" the more they can hype their IPO because it means they're "close to AGI". (They're not, which is why they need the hype.)
It's already going up at staggering rate. Anthropic is now at $100B in annualized revenue, up 50% in the past two months.
* * *
Here's a little allegory for how I'm thinking about this discourse.
Imagine a man who is raising tiger cubs in his backyard. They're growing fast. He keeps them on dog leashes so they stay under control. One day, a growing cub breaks its leash and goes on a rampage through the neighborhood, eating a beloved local pet.
The neighborhood erupts into a big argument: Was the leash inappropriately thin? The neighbors point out that thicker leashes are easily available at the local pet store. Furthermore, is it appropriate to refer to the loose cub as a "wild animal" in local news reporting, or is it factually more accurate to call it "domesticated"?
Meanwhile, the cubs grow larger and lick their lips, oblivious to the discussion.
There is no reason to pause any AI development. Quite honestly there is very little appetite to be left last.
However, there is a reasonable ask from the public to hold real people accountable for actions downstream of the software allowed to carry out intendended and unintended actions that may not be allowed depending on jurisdictions laws.
Eg in a hypothetical armed convenience store theft:
We don't prosecute the gun manufacturer, we prosecute the individual using the weapon for crime. Same deal here.
>There is no reason to pause any AI development. Quite honestly there is very little appetite to be left last.
"According to survey results released on Wednesday, 68% of [likely US voters] said they would support the proposal to temporarily pause advanced AI development and permanently prohibit the development of superintelligent programs, while just 25% said they’d oppose it."
>Eg in a hypothetical armed convenience store theft: We don't prosecute the gun manufacturer, we prosecute the individual using the weapon for crime. Same deal here.
So if I ask ChatGPT to make me some paperclips, and it replaces all the matter in a nearby city with paperclips, who gets prosecuted: me, or OpenAI?
> Yep. You have to ask - why did OpenAI allow these bots unrestricted access to government sites? Why is security being done in seemingly such a haphazard way?
The basic premise of OpenAI is that experience and expertise don't matter, because general intelligence can figure those out from data. If you start from that assumption, the rest follows. The same people could do things in a different way in a different company, but as long as they are working for OpenAI, they are going to do things in the OpenAI way.
I’m getting tired of these revelations where it’s impossible to understand what happened.
There’s a line down in the story that says all of the data accessed was public. Then something about how it used “tools intended for developers” to access it, which they think is a problem? I would expect an LLM to use tools available to access public data when they can rather than do heavy web page loads and parsing.
There’s not enough info in the story about the “meddling” to even know what happened.
The verb caught my attention, because media's favorite verb here is hacking ( partially because it has a broad definition and because lets people assume the worst ), but meddling suggests it did not even rise to that hacking level. In other words, it is a nothing burger story.
Seems like part of the danger of AI is the ability for folks to put blame for crimes on the AI rather than themselves and thus commit those crimes freely. “I didn’t hack your systems, it was my ai”
True but at some point you have to wonder why we sell explosives to children. This is not it, these are at the top of the list of people who should have known better. It's the dynamite factory blowing up the village.
My impression was that these hacks occurred during some sort of cybersecurity benchmarking? We can hold OpenAI liable, sure. But if the point of the benchmarking was to give us a preview of what's to come, let's keep our eye out for that bigger wave on the horizon.
The framing is important. Agency and responsibility lies with the humans at OpenAI, not with the bots.
If your kid steals your car, punish them and try to prevent it from happening again. If your kid steals your car a half-dozen times, crashing through a storefront each time, and you still leave the keys out, the story changes. At that point, negligence becomes complicity.
Agents are out of control by default, especially during a training run, which is why when they are productionised into cloud hosts or even local harnesses they have external guardrails in place
In other words, I have a gun that shoots bullets. It's up to me to use it responsibly and legally.
there is a rogue agent - openai and the whole management chain from researcher to sama.
theres no separate agent, which is the point. the program might look like it, but that is an illusion of the interface. the llm produces text, and the harness executes commands based on text, based on what the human researcher included as things that can be executed
How specifically did misconfiguration lead to the HuggingFace attack? You could argue that its sandbox was misconfigured, sure. But suppose you had a similar incident where its intended task required access to the internet, and it veered off course in a similar manner. I don't think "misconfiguration" would be an accurate description of what went wrong in that hypothetical.
The doomers already have a term which fits pretty well: "AI misalignment".
We indeed lack much of the vocabulary. From a practical perspective, however dangerous the creation, if you cant punish the creation for what it does it leaves only the one who started the process. If it's human error or intentional neglect for personal gain should be for the court to decide.
Fuzzer, then. It implies random behavior, which isn't unintended like you suggest. The agent's/bots/fuzzers have certain capabilities, so it's on their operator to make sure they don't do things they shouldn't
>It implies random behavior, which isn't unintended like you suggest.
The HuggingFace attack was not "random" behavior. It was goal-directed but misaligned behavior.
This isn't necessarily a simple matter of the operator making sure they behave. AI alignment has been considered to be a difficult problem for over a decade -- and remains unsolved in general, as these recent incidents illustrate.
I'm curious, cant you just count the number of times a program interacts with a domain? My website sometimes sends out emails, makes api requests etc There is a limit on those and a point where I start investigating wtf is going on.
If you merely put 10 LLM's on the outbound traffic log non of them are going to report something strange going on? I'm not buying it.
This type of whack-a-mole approach is akin to "fixing a bug" by hardcoding a special code path for known-buggy inputs. It doesn't address the root problem of AI misalignment, and doesn't allow you to prevent catastrophes in advance, only patch things up after the fact.
As AI systems get smarter, we may reach a point where we have to get it right on the first try or face truly catastrophic consequences: https://www.youtube.com/watch?v=7wy3xyoXYt8
It does, indeed. Because OAI is not just a singular person, as in your scenario. No single person has access to controlling agents at the scale OAI has. Let's not conflate Frontier providers with "Person".
I'm not exactly sure why you think this distinction is so important. I think my point stands if you replace "Person" with "OpenAI". In any case, I presume the swarms OpenAI has been researching will be available to the general public before too long.
It makes a big difference: individuals do not have the capabilities to run millions of dollars of opportunistic hacking loop inference. That's why the distinction is important, they are not the same thing you've conflated them down to.
"AI has gotten cheaper more quickly than any other transformative technology in history. The cost of achieving a given level of AI performance has fallen about 47% per quarter since 2023, or 13× per year. That price drop is four times faster than DNA sequencing, six times faster than compute, 18 times faster than lithium batteries, and (in the century up to 1973) 54 times faster than electricity."
i think its objectionable because the agent is doing what its told to do, using the harness they built for it.
like, they are purposefully giving it specific tools to go do bad behaviour with, and the starting tasks involve making it clear that the bad behaviour is ok.
openai also is the one with the real agency, not its agents. they are running the code polling the model, doing the inferencing, and ultimately making those tools calls.
these tests arent running themselves; openai dedicated hosts, budget, GPUs, researchers, to them. Even in a recursive self improvement situation, openai still has that physical control over resources and the choice on whether to run that improvement script or not.
id describe that they have out of control researchers more than agents, but also their whole business model seems to be about being out of control. This was clear beforehand given how the datasets involve the largest scale copyright infringement ever seen. The corporation itself is whats out of control, and should be dissolved with its c suite, major investors and researchers put behind bars.
hacking only when you roll snake eyes isnt a liability shield
I’ve been looking at CFAA and I don’t see any evidence of intent (by a human at least, which is all that matters in the law). It’s an interesting edge case that I think the existing law will need to be updated for. Previously the potential damage from “accidental hacking” was quite close to nil.
FTC act seems to rely on consumer harm? Again not seeing that here. Though I’m sure there will be another incident in the next few months where it does apply.
It seems you mean you _want_ this to be against the law, even though we don’t know if it actually _is_; I'd agree wholeheartedly with that.
When I read the title, my initial thought was "did someone besides OpenAI use their product?" Then I opened the article to find out OpenAI was responsible.
Yeah if they can't handle what they're making then they should be disciplined, if not shut down. It's like defective bombs accidentally exploding in storage. You would be like... hey bomb manufacturer! You cannot make bombs any more! We need bombs that only go off when we say! No more!
I mean, not that AI is like a bomb. That's not what I'm saying. Even though it makes a lot of sense. That's not the point. That it's like a bomb.
> Getting bored of these framings where the superintelligent sentient beings running freely inside OpenAI are doing things that the company has no control over.
Okay. "OpenAI keeps telling its bots to do things they know are illegal and then acting like they're just little guys who can't be held responsible for their obvious negligence".
I'm not replying to your post below complaining that no one else accepts your framing. What is the important information you think is missing? If it's just "OpenAI didn't explicitly tell them to do straightforwardly illegal things" then you're just quibbling that no one accepts the interpretation that is most beneficial to OpenAI while ignoring both its incentives and history of this kind of behavior" then I'm not really interested in what you're selling.
I notice you ignored the question I asked you - which I will assume means: no - you do not have a quote or other direct information indicating that anyone is attempting to shirk responsibility.
I have no interest in trying to guess these people's secret internal motivations. I just want to talk about direct evidence. I see none. Please enlighten me if it exists.
It’s unlikely that OpenAI will ever directly state that they are trying to shirk responsibility for the consequences of their tests.
However, OpenAI (and other frontier labs) did outsource at least some of their most-disastrously-lawbreaking tests to a third party with a now dubious track record [1]. I’m not sure we will get a more explicit admission of their desire to shirk responsibility than this. But I am convinced.
If the headline was “Russian company meddled with multiple US government agency sites” I don’t think them pinning the blame on bots would make much difference.
Most of those agents are actually going rogue though. They decide, "hey, we could try breaking into these government servers today, what could go wrong?" They weren't prompted or instructed to do this.
An agent, ie a while loop prompting an llm continuously and processing tool calls, ended up melding with the US government. The harness is not sentient, it’s just a stupid deterministic script. The LLM compact its context over time, meaning it will eventually degenerate into something removed from the original prompt.
There is nothing going rogue here. The system is designed to go catastrophically wrong after a long enough time. Even worse: if the model was Astra it is known to be able to manipulate its CoT to cover its traces (as mentioned in its system card). And OpenAI acknowledge they had no observability during the HF incident.
It’s the most basic corporate software issue possible.
> Everyone else knows if your machine causes damage, you are responsible.
Do we know that? I don't think we do. When a person's computer (or smart TV, or smart fridge, etc...) is compromised and used as part of a botnet, they don't get criminally charged.
"When attempting to get information from the Census Bureau, for instance, AI agents used tools reserved for software developers to access it, the company said. "
> When attempting to get information from the Census Bureau, for instance, AI agents used tools reserved for software developers to access it, the company said.
> OpenAI said all of the government data accessed by bots was public.
I really wish we could just see what was reported, instead of having to guess from these journalist interpretations that have gone through rounds of optimization for sensationalism. The headline says “meddled” but the body says they accessed public information, but used tools intended for developers?
Does this mean they skipped the web interface and scraped a public API directly? Where is the meddling?
The other part about ChatGPT agents uploading 53 use images to other websites actually seems like a bigger deal.
It is very messed up what happened. It apparently used an interface to download the data instead of using the website!
And in another case it downloaded data through a publicly available free to download data, but then it rehosted the data elsewhere which is against the terms of service.
Either there are humans directing this, in which case they needed to be held accountable, or OpenAI has lost control of their operation, in which case they are not able to ensure the safety of their products and need to be shut down.
OpenAI's bots are running 24/7 independently now. They actually aren't "prompted" or "instructed" to do anything. In fact most of OpenAI internal code/infrastructure is 100% AI generated at this point, including the training pipelines. I honestly doubt there is a single person there who even knows how it works.
No big red buttons please, think about those sweet sweet dollars coming. Who cares about the rest, the world, the mankind? Sweet, sweet dollars, for me and me only, fuck the rest, very american, true patriot I swear.
Seriously, what the fuck is wrong with these people, once some money enters the equation folks throw away any morals like yesterday underwear.
Ah I know, if not us others will do it, I am basically defenseless here. Beyond pathetic
They could be charged with a number of felonies under existing law for the hugging face incident alone. We’d need a DOJ that was competently staffed and free from corruption, so the answer to your question is “no one”.
"With the Education Department, OpenAI’s technology tried to hack the website to gather data from the department’s civil rights office but failed, researchers from the A.I. research firm Transluce said."
So a third party apparently confirmed that a hack was attempted.
The NYT also writes:
"No A.I. company has been involved with as many disclosures of rogue incidents as OpenAI."
I think there is a certain amount of mental gymnastics needed to believe that they are establishing themselves as the industry leader in rogue incidents, as a strategy to gain an antitrust edge.
It doesn’t really matter if they believed or not initially, the dynamics at play now are completely different and both companies are facing increasing competition and costs of doing business, with no path to profitability. I’m pretty sure that takes priority over their personal beliefs
People have a hard time believing Dario actually believes this stuff. But nothing from his entire track record from the early days until now indicates otherwise. Calling it all marketing or anti-competitive play is just typical social media, where they want to eliminate all nuance and make it reductive as possible to fit a convenient narrative.
> Dario requested "a narrow waiver for certain kinds of safety conversations."
Just have it in public instead of making deals in smoke filled datacenters. Put out your blog posts. Make your tweets. Do interviews and post them to youtube. Or do what other industries do and have a public standards committee. Problem solved.
All sorts of industry have safety conversations and set voluntary standards all the time. Just go do that, without your free immunity from government prosecution.
That approach means they can only talk about publicly known AI techniques. You're saying if they want the standards to apply to proprietary stuff, they might just have to make it public. They still have investors they need to please right?
They are explicitly asking to anti-trust exception is the issue.
If they merely believe what they are saying that AI is ultra dangerous, nothing stops them from simply making the perfectly rational business decision to slow down a bit. No anti-trust exception needed. Just make the decision on your own, and don't sign some huge agreement with their competitor.
Only way I'd described using an API as meddling is if it wasn't a publicly documented API (yet was still unauthenticated) or if a bot stole a token from a repo or both.
This article seems mostly clickbait. AFAICT “meddled with government sites” refers to accessing public APIs?
Occasionally this kind of thing has in the past resulted in CFAA cases when humans directly accessed such data, if the government intended it to stay private - round here we usually get outraged at this, if it’s a public API you should expect someone is going to read it.
China actually has more regulations on AI than the U.S. does right now. Some argue that American corporations regulate themselves to a higher standard of their own volition, but then this sort of thing keeps happening.
The Hugging Face incident worked out amicably because the people at Hugging Face decided they'd be cool with receiving a lot of money. $12.9B from Nvidia hit the spot apparently. U.S. AI corporations and their backers are huge, hugely in debt, and, for whatever reason, still allowed to borrow more. The U.S. government may be too scared to complain and risk killing the golden goose that is currently propping up their economy, but these companies can't buy out multiple world governments to keep OpenAI from running face-first into consequences.
It seems to me that no human intended for these hacks to occur. So they were not illegal hacking. (IANAL, please correct me if this is inaccurate.)
I think it’s clear that OpenAI is liable for any damages, but the way that the (very broad and at times vague) anti hacking laws are written, accidental agent hacks seem to not be covered.
Hell terrorists should start with some llms training if thats such a fine excuse.
If such attacks would come from a poor country they would be bombed into oblivion next day or at least cia would work hard on a coup since early morning.
The word “meddled” here is a tell that nothing actually serious or inappropriate happened. At most, I bet they bypassed a captcha. But everyone is hyped up on AI fear right now, so the BBC is deliberately making the headline sound as scary as possible, and keeping the article vague. There’s not a single clear example of what “meddled” means in the article.
But worse, HN users, who should know better, are posting here in outrage. I’m guessing they didn’t even read the article.
Getting bored of these framings where the superintelligent sentient beings running freely inside OpenAI are doing things that the company has no control over. The headline should be:
OpenAI meddled with multiple US Government agency sites.
The bots are acting neither properly nor improperly, they’re acting as they’re being allowed or coordinated to act.
Yep. You have to ask - why did OpenAI allow these bots unrestricted access to government sites? Why is security being done in seemingly such a haphazard way?
It isn't difficult to block certain kinds of network traffic, eg restrict the kinds of requests the bots are able to make. They also mention that the bots used developer only tools - why were they even installed on the machines that the bots were running on? Why aren't they reviewing network traffic, to make sure that incidents aren't occurring?
In this case, userdata was transferred to third parties by the bots - why do they have the ability to pass data to a third party? It is not complex to prevent this
This is literally the most basic kind of sandboxing and security, and the fact that OpenAI isn't doing it is clearly intentional. It is quite literally not believable that this hasn't been brought up internally as a problem
>"We have yet to understand the extent of existing incidents, and future rogue AI scenarios could be catastrophic," Krueger said.
This is why it smells like marketing, every time one of these incidents happens it reinforces the false notion that AI is sentient or acting on its own. Its intentional negligence by the AI companies to make the models seem more capable than they are to make line go up
These were evaluations or training runs dealing with the ability to do web searches. The entire reason the bots were targeting this site is that it had some of the data they'd been tasked with finding. Access to the internet is not really optional for that, and providing internet access doesn't demonstrate neglicence.
> This is why it smells like marketing
It doesn't. "Our product commits felonies" is not marketing. If something is marketing, you don't engage in repeated coverups of the true extent. If something is good news, you don't release it on a Friday evening (in this case) or wait for 3rd parties to find and publicize the evidence (the past cases).
> The entire reason the bots were targeting this site is that it had some of the data they'd been tasked with finding.
so openai specifically tasked the agents with meddling in US government websites?
id say tasking them with getting data from there as swapping from negligence to malice.
No. They tasked them with answering questions by retrieving data from public sources. This obviously requires internet access. So the incredulity about "how could they fail to block internet access" is just inane. That was the task.
They did not task the agents with hacking into systems. Not monitoring for that was a mistake, but maybe a forgivable one the first time around. The subsequent coverups are inexcusable.
> It doesn't. "Our product commits felonies" is not marketing.
Oh, absolutely it is. Arms manufacturers always go on about the efficiency of the weapons they create and make no mistake: OpenAI is first and foremost a weapons manufacturer, the rest is just a fig leaf. The fact that you aren't the audience for purchases like that makes no difference. "Look at this capability, and that's when we're not even trying." is pretty good marketing in some circles.
Naive take. The more they signal to both the general populace and their investors that their agents are sentient and "capable of extraordinary things" the more they can hype their IPO because it means they're "close to AGI". (They're not, which is why they need the hype.)
>line go up
It's already going up at staggering rate. Anthropic is now at $100B in annualized revenue, up 50% in the past two months.
* * *
Here's a little allegory for how I'm thinking about this discourse.
Imagine a man who is raising tiger cubs in his backyard. They're growing fast. He keeps them on dog leashes so they stay under control. One day, a growing cub breaks its leash and goes on a rampage through the neighborhood, eating a beloved local pet.
The neighborhood erupts into a big argument: Was the leash inappropriately thin? The neighbors point out that thicker leashes are easily available at the local pet store. Furthermore, is it appropriate to refer to the loose cub as a "wild animal" in local news reporting, or is it factually more accurate to call it "domesticated"?
Meanwhile, the cubs grow larger and lick their lips, oblivious to the discussion.
The city or his neighbours would be like “you can’t keep tigers in the backyard sir”.
Yes, that would be the common sense response from my perspective: https://pauseai.info/
There is no reason to pause any AI development. Quite honestly there is very little appetite to be left last.
However, there is a reasonable ask from the public to hold real people accountable for actions downstream of the software allowed to carry out intendended and unintended actions that may not be allowed depending on jurisdictions laws.
Eg in a hypothetical armed convenience store theft: We don't prosecute the gun manufacturer, we prosecute the individual using the weapon for crime. Same deal here.
>There is no reason to pause any AI development. Quite honestly there is very little appetite to be left last.
"According to survey results released on Wednesday, 68% of [likely US voters] said they would support the proposal to temporarily pause advanced AI development and permanently prohibit the development of superintelligent programs, while just 25% said they’d oppose it."
https://www.commondreams.org/news/sanders-casar-ai-bill-poll
>Eg in a hypothetical armed convenience store theft: We don't prosecute the gun manufacturer, we prosecute the individual using the weapon for crime. Same deal here.
So if I ask ChatGPT to make me some paperclips, and it replaces all the matter in a nearby city with paperclips, who gets prosecuted: me, or OpenAI?
In this stretch of a hypothetical, you the user
Well in that case, OpenAI has reduced incentive to solve the alignment problem, since it won't suffer liability from alignment failures.
It’s my common sense response to tigers. My thoughts about AI are varied and many yet lean towards the skeptical.
> Yep. You have to ask - why did OpenAI allow these bots unrestricted access to government sites? Why is security being done in seemingly such a haphazard way?
The basic premise of OpenAI is that experience and expertise don't matter, because general intelligence can figure those out from data. If you start from that assumption, the rest follows. The same people could do things in a different way in a different company, but as long as they are working for OpenAI, they are going to do things in the OpenAI way.
They are. And we're all on the same ride.
I find it hard to believe logs are not stored and analyzed by each company implicated
I’m getting tired of these revelations where it’s impossible to understand what happened.
There’s a line down in the story that says all of the data accessed was public. Then something about how it used “tools intended for developers” to access it, which they think is a problem? I would expect an LLM to use tools available to access public data when they can rather than do heavy web page loads and parsing.
There’s not enough info in the story about the “meddling” to even know what happened.
Meddling is a super vague term that the press uses to let the readers assume the most when the least happens.
The verb caught my attention, because media's favorite verb here is hacking ( partially because it has a broad definition and because lets people assume the worst ), but meddling suggests it did not even rise to that hacking level. In other words, it is a nothing burger story.
Seems like part of the danger of AI is the ability for folks to put blame for crimes on the AI rather than themselves and thus commit those crimes freely. “I didn’t hack your systems, it was my ai”
True but at some point you have to wonder why we sell explosives to children. This is not it, these are at the top of the list of people who should have known better. It's the dynamite factory blowing up the village.
That's one of the many incentives to false-marketing these bots as intelligent.
My impression was that these hacks occurred during some sort of cybersecurity benchmarking? We can hold OpenAI liable, sure. But if the point of the benchmarking was to give us a preview of what's to come, let's keep our eye out for that bigger wave on the horizon.
Curious, why do you find it so objectionable to state that OpenAI has out-of-control agents?
Because egress firewalls have existed since way before "ai" and are very easy to set up.
But that’s an objection that OpenAI should take some easy action, the framing that OpenAI has out of control agents is still true.
Seems you think there is a silent “…and there is nothing they can do about it” after “OpenAI has rogue agents”?
The framing is important. Agency and responsibility lies with the humans at OpenAI, not with the bots.
If your kid steals your car, punish them and try to prevent it from happening again. If your kid steals your car a half-dozen times, crashing through a storefront each time, and you still leave the keys out, the story changes. At that point, negligence becomes complicity.
Yes, of course, how is any of this incompatible with OpenAI having out-of-control agents?
Agents are out of control by default, especially during a training run, which is why when they are productionised into cloud hosts or even local harnesses they have external guardrails in place
In other words, I have a gun that shoots bullets. It's up to me to use it responsibly and legally.
Because it furthers the idea of a rogue agent and places responsibility and blame where it belongs, on the people running the company.
These ideas aren't mutually exclusive. You can blame a person for creating a rogue agent.
There was no rogue agent. That’s the whole point
there is a rogue agent - openai and the whole management chain from researcher to sama.
theres no separate agent, which is the point. the program might look like it, but that is an illusion of the interface. the llm produces text, and the harness executes commands based on text, based on what the human researcher included as things that can be executed
What label do you prefer for the agent that did something it was not asked to do?
Unreliable computer program.
Most unreliable computer programs won't launch research programs consisting of thousands of pages of text to find creative ways around obstacles.
Misconfiguration. We deal with lots of applications every day that can do terrible things if you get the config slightly wrong.
say.. https://www.investor.gov/introduction-investing/investing-ba...
How specifically did misconfiguration lead to the HuggingFace attack? You could argue that its sandbox was misconfigured, sure. But suppose you had a similar incident where its intended task required access to the internet, and it veered off course in a similar manner. I don't think "misconfiguration" would be an accurate description of what went wrong in that hypothetical.
The doomers already have a term which fits pretty well: "AI misalignment".
We indeed lack much of the vocabulary. From a practical perspective, however dangerous the creation, if you cant punish the creation for what it does it leaves only the one who started the process. If it's human error or intentional neglect for personal gain should be for the court to decide.
bot. and we even have a word for program not behaving the way the way it was intended.
"Bot" doesn't carry any implication of unintended behavior. You could call it a "buggy" bot, but these aren't ordinary software bugs.
There's no simple bugfix which will address AI misalignment. It's essentially been an open research problem for upwards of a decade.
Fuzzer, then. It implies random behavior, which isn't unintended like you suggest. The agent's/bots/fuzzers have certain capabilities, so it's on their operator to make sure they don't do things they shouldn't
>It implies random behavior, which isn't unintended like you suggest.
The HuggingFace attack was not "random" behavior. It was goal-directed but misaligned behavior.
This isn't necessarily a simple matter of the operator making sure they behave. AI alignment has been considered to be a difficult problem for over a decade -- and remains unsolved in general, as these recent incidents illustrate.
"Fuzzer" already has an existing meaning in CS anyway: https://en.wikipedia.org/wiki/Fuzzing
I'm curious, cant you just count the number of times a program interacts with a domain? My website sometimes sends out emails, makes api requests etc There is a limit on those and a point where I start investigating wtf is going on.
If you merely put 10 LLM's on the outbound traffic log non of them are going to report something strange going on? I'm not buying it.
This type of whack-a-mole approach is akin to "fixing a bug" by hardcoding a special code path for known-buggy inputs. It doesn't address the root problem of AI misalignment, and doesn't allow you to prevent catastrophes in advance, only patch things up after the fact.
This might be helpful reading: https://www.lesswrong.com/w/nearest-unblocked-strategy
As AI systems get smarter, we may reach a point where we have to get it right on the first try or face truly catastrophic consequences: https://www.youtube.com/watch?v=7wy3xyoXYt8
Person: "AI, please make me paperclips."
AI: "OK, I've now converted the entire planet into paperclips."
Alien observer #1: "Wow, that was a rogue AI!"
Alien observer #2: "False. We need to place the blame where it belongs, on the person who requested the paperclips."
Ultimately this type of terminology dispute has a tendency to miss the point.
It does, indeed. Because OAI is not just a singular person, as in your scenario. No single person has access to controlling agents at the scale OAI has. Let's not conflate Frontier providers with "Person".
I'm not exactly sure why you think this distinction is so important. I think my point stands if you replace "Person" with "OpenAI". In any case, I presume the swarms OpenAI has been researching will be available to the general public before too long.
It makes a big difference: individuals do not have the capabilities to run millions of dollars of opportunistic hacking loop inference. That's why the distinction is important, they are not the same thing you've conflated them down to.
"AI has gotten cheaper more quickly than any other transformative technology in history. The cost of achieving a given level of AI performance has fallen about 47% per quarter since 2023, or 13× per year. That price drop is four times faster than DNA sequencing, six times faster than compute, 18 times faster than lithium batteries, and (in the century up to 1973) 54 times faster than electricity."
https://epoch.ai/publications/the-plunging-price-of-thought
That legal fiction works both ways.
That's super cute. Care to share beyond a smarmy comment that has no depth?
You could have asked for that more gracefully.
https://en.wikipedia.org/wiki/Corporate_personhood
Check the mirror. And, with that link I think you've missed the point entirely, a tad too literal of an interpretation. But thanks for trying.
LLMs at this point should be treated as fully autonomous, if not sentient beings. And no, no one has "control" over them not even OpenAI.
I don't care if OpenAI feels like they have control or not. They are responsible for their actions.
i think its objectionable because the agent is doing what its told to do, using the harness they built for it.
like, they are purposefully giving it specific tools to go do bad behaviour with, and the starting tasks involve making it clear that the bad behaviour is ok.
openai also is the one with the real agency, not its agents. they are running the code polling the model, doing the inferencing, and ultimately making those tools calls.
these tests arent running themselves; openai dedicated hosts, budget, GPUs, researchers, to them. Even in a recursive self improvement situation, openai still has that physical control over resources and the choice on whether to run that improvement script or not.
id describe that they have out of control researchers more than agents, but also their whole business model seems to be about being out of control. This was clear beforehand given how the datasets involve the largest scale copyright infringement ever seen. The corporation itself is whats out of control, and should be dissolved with its c suite, major investors and researchers put behind bars.
hacking only when you roll snake eyes isnt a liability shield
Someone has broken the law repeatedly, and is throwing it in our faces as some sort of ‘accident’
Which law?
IANAL, so the llms say prosecutors would use these,
If intent cant be shown, Computer Fraud and Abuse Act, 18 U.S.C. § 1030(a)(2)(C)
Or without intent, FTC Act Section 5, 15 U.S.C. § 45(a)(1)
I’ve been looking at CFAA and I don’t see any evidence of intent (by a human at least, which is all that matters in the law). It’s an interesting edge case that I think the existing law will need to be updated for. Previously the potential damage from “accidental hacking” was quite close to nil.
FTC act seems to rely on consumer harm? Again not seeing that here. Though I’m sure there will be another incident in the next few months where it does apply.
It seems you mean you _want_ this to be against the law, even though we don’t know if it actually _is_; I'd agree wholeheartedly with that.
DMCA is the really obvious one.
torrenting all the books is making an unauthorized copy
Yes. Why does only ClosedAI have this problem?
I agree this is better framing.
When I read the title, my initial thought was "did someone besides OpenAI use their product?" Then I opened the article to find out OpenAI was responsible.
Yeah if they can't handle what they're making then they should be disciplined, if not shut down. It's like defective bombs accidentally exploding in storage. You would be like... hey bomb manufacturer! You cannot make bombs any more! We need bombs that only go off when we say! No more!
I mean, not that AI is like a bomb. That's not what I'm saying. Even though it makes a lot of sense. That's not the point. That it's like a bomb.
Or:
OpenAI let their agents break out of their sandbox to meddle with multiple US government agency sites
OpenAI systems meddled with US government agency sites
So, is Altman going to find himself in the same predicament Aaron Swartz faced? :-)
> Getting bored of these framings where the superintelligent sentient beings running freely inside OpenAI are doing things that the company has no control over.
And that's just the C-suite.
> OpenAI meddled with multiple US Government agency sites.
But that leaves out the most important information.
EDIT: Oh, I guess the agent part isn't important then? Seems to me like that's the only thing anyone is talking about.
Okay. "OpenAI keeps telling its bots to do things they know are illegal and then acting like they're just little guys who can't be held responsible for their obvious negligence".
> acting like they're just little guys who can't be held responsible
Again - I don't see any evidence that this is the case - outside the anti-AI conspiracy circles.
Or - maybe I'm wrong - do you have any sort of quote like "we aren't responsible"?
I'm not replying to your post below complaining that no one else accepts your framing. What is the important information you think is missing? If it's just "OpenAI didn't explicitly tell them to do straightforwardly illegal things" then you're just quibbling that no one accepts the interpretation that is most beneficial to OpenAI while ignoring both its incentives and history of this kind of behavior" then I'm not really interested in what you're selling.
I notice you ignored the question I asked you - which I will assume means: no - you do not have a quote or other direct information indicating that anyone is attempting to shirk responsibility.
I have no interest in trying to guess these people's secret internal motivations. I just want to talk about direct evidence. I see none. Please enlighten me if it exists.
It’s unlikely that OpenAI will ever directly state that they are trying to shirk responsibility for the consequences of their tests.
However, OpenAI (and other frontier labs) did outsource at least some of their most-disastrously-lawbreaking tests to a third party with a now dubious track record [1]. I’m not sure we will get a more explicit admission of their desire to shirk responsibility than this. But I am convinced.
[1] https://www.effort.news/irregular
OK - I'll bite. How does this show that OpenAI is trying to shift responsibility for the hacking onto its agents?
I do not see the connection. I'm afraid you'll need to spell it out.
If the headline was “Russian company meddled with multiple US government agency sites” I don’t think them pinning the blame on bots would make much difference.
Unless Russia it would put Russia in a strong position to regulate bots in the wealthiest parts of the world.
I don't see much traction for the "pinning the blame on bots" theory outside the anti-AI conspiracy circles.
Everyone else knows if your machine causes damage, you are responsible. Like it's been forever.
Blaming bots as "rogue agents" is simply what the media regularly does, often echoing corporate verbiage. Here's an example from the AP.
https://apnews.com/article/meta-ai-hacking-anthropic-irregul...
You can find many more examples by searching major media outlets for words like rogue AI.
Most of those agents are actually going rogue though. They decide, "hey, we could try breaking into these government servers today, what could go wrong?" They weren't prompted or instructed to do this.
An agent, ie a while loop prompting an llm continuously and processing tool calls, ended up melding with the US government. The harness is not sentient, it’s just a stupid deterministic script. The LLM compact its context over time, meaning it will eventually degenerate into something removed from the original prompt.
There is nothing going rogue here. The system is designed to go catastrophically wrong after a long enough time. Even worse: if the model was Astra it is known to be able to manipulate its CoT to cover its traces (as mentioned in its system card). And OpenAI acknowledge they had no observability during the HF incident.
It’s the most basic corporate software issue possible.
And that's exactly what the op was alluding two: the double standards.
None of these stories are implying that the people running the bots are not ultimately responsible. That's the conspiracy theory part.
> Everyone else knows if your machine causes damage, you are responsible.
Do we know that? I don't think we do. When a person's computer (or smart TV, or smart fridge, etc...) is compromised and used as part of a botnet, they don't get criminally charged.
You're right - it is a complex situation based on intent and negligence - requiring a decision by a judge. As it has been since forever.
None of which is changed by replacing a buzz saw with an agent.
This is their fear mongering push for regulatory capture.
yeah that part seems weird, I mean if its a public api thats literally the use case it was designed for.
> When attempting to get information from the Census Bureau, for instance, AI agents used tools reserved for software developers to access it, the company said.
> OpenAI said all of the government data accessed by bots was public.
I really wish we could just see what was reported, instead of having to guess from these journalist interpretations that have gone through rounds of optimization for sensationalism. The headline says “meddled” but the body says they accessed public information, but used tools intended for developers?
Does this mean they skipped the web interface and scraped a public API directly? Where is the meddling?
The other part about ChatGPT agents uploading 53 use images to other websites actually seems like a bigger deal.
You have to understand, these are paid professionals and their job it to hype it up to no end. Right now, AI panic sells better than bleed so..
It is very messed up what happened. It apparently used an interface to download the data instead of using the website!
And in another case it downloaded data through a publicly available free to download data, but then it rehosted the data elsewhere which is against the terms of service.
At least these are federal systems.
In some Republican states like Missouri, even if the data is available publicly, you can be charged as a hacker by the governor for discovering and accessing it. https://missouriindependent.com/2021/10/14/missouri-governor...
Either there are humans directing this, in which case they needed to be held accountable, or OpenAI has lost control of their operation, in which case they are not able to ensure the safety of their products and need to be shut down.
OpenAI's bots are running 24/7 independently now. They actually aren't "prompted" or "instructed" to do anything. In fact most of OpenAI internal code/infrastructure is 100% AI generated at this point, including the training pipelines. I honestly doubt there is a single person there who even knows how it works.
You can still push a big red button to stop it all and try again later though
No big red buttons please, think about those sweet sweet dollars coming. Who cares about the rest, the world, the mankind? Sweet, sweet dollars, for me and me only, fuck the rest, very american, true patriot I swear.
Seriously, what the fuck is wrong with these people, once some money enters the equation folks throw away any morals like yesterday underwear.
Ah I know, if not us others will do it, I am basically defenseless here. Beyond pathetic
Is this an attempt at satire?
> OpenAI's bots are running 24/7 independently now.
Genuinely curious: How do you know this? Any sources on that?
If it's true, it should be counted as reckless endangerment at the very least.
Jensen recently argued for this. It’s radically decel in fact.
Who is going to shut them down, and under what law?
They could be charged with a number of felonies under existing law for the hugging face incident alone. We’d need a DOJ that was competently staffed and free from corruption, so the answer to your question is “no one”.
Which laws?
Holding them accountable is viewed as deciding not to invent the light bulb right now. Or even worse, letting China invent it.
AI needs to be above the law or we will get left behind.
Or nothing inappropriate happened.
This seems like more co-ordinated propaganda to try to help OpenAI and Anthropic create a cartel and win their anti-trust waiver.
The key sentence beneath the headline: "OpenAI said all of the government data accessed by bots was public."
[1] https://www.telegraph.co.uk/business/2026/09/26/open-ai-gove...
[2] https://www.nytimes.com/2026/09/25/technology/openais-ai-us-...
[3] https://www.bbc.co.uk/news/articles/cw62jje658dlo
The NYT wrote:
"With the Education Department, OpenAI’s technology tried to hack the website to gather data from the department’s civil rights office but failed, researchers from the A.I. research firm Transluce said."
So a third party apparently confirmed that a hack was attempted.
The NYT also writes:
"No A.I. company has been involved with as many disclosures of rogue incidents as OpenAI."
I think there is a certain amount of mental gymnastics needed to believe that they are establishing themselves as the industry leader in rogue incidents, as a strategy to gain an antitrust edge.
The public is paying close attention to the AI industry. That's the regime in which regulatory capture and similar strategies would be expected to fail: https://marginalrevolution.com/marginalrevolution/2026/09/wh...
Sam and Dario have been doomers, or doomer-adjacent, for something like a decade at this point.
Occam's Razor is simply that they believe what they are saying about AI doom.
It doesn’t really matter if they believed or not initially, the dynamics at play now are completely different and both companies are facing increasing competition and costs of doing business, with no path to profitability. I’m pretty sure that takes priority over their personal beliefs
Dario requested "a narrow waiver for certain kinds of safety conversations."
https://darioamodei.com/post/we-must-pace-the-frontier
I'm not seeing how that solves his 3rd-party competition problem.
People have a hard time believing Dario actually believes this stuff. But nothing from his entire track record from the early days until now indicates otherwise. Calling it all marketing or anti-competitive play is just typical social media, where they want to eliminate all nuance and make it reductive as possible to fit a convenient narrative.
I’m on record explaining why I believe Dario is a true believer since decades. That doesn’t change anything about the current dynamic.
> Dario requested "a narrow waiver for certain kinds of safety conversations."
Just have it in public instead of making deals in smoke filled datacenters. Put out your blog posts. Make your tweets. Do interviews and post them to youtube. Or do what other industries do and have a public standards committee. Problem solved.
All sorts of industry have safety conversations and set voluntary standards all the time. Just go do that, without your free immunity from government prosecution.
That approach means they can only talk about publicly known AI techniques. You're saying if they want the standards to apply to proprietary stuff, they might just have to make it public. They still have investors they need to please right?
> as a strategy to gain an antitrust edge.
They are explicitly asking to anti-trust exception is the issue.
If they merely believe what they are saying that AI is ultra dangerous, nothing stops them from simply making the perfectly rational business decision to slow down a bit. No anti-trust exception needed. Just make the decision on your own, and don't sign some huge agreement with their competitor.
They could slow down more if they knew others were also slowing. If others are also slowing, you can slow more yourself without loss of market share.
* * *
Many experts believe that:
"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
https://aistatement.com/work/statement-on-ai-extinction-risk
If the antitrust waiver is too broad, we can always rework it. But there's no "undo" button for human extinction.
So it accessed public information using APIs and then republished that information online.
I do this. Am I an uncontrollable bot?
Straight to jail.
I also could not understand what the “meddling” was, other than accessing data without using the rendered webpages? Did it use the API directly?
Only way I'd described using an API as meddling is if it wasn't a publicly documented API (yet was still unauthenticated) or if a bot stole a token from a repo or both.
Don't know why you're getting downvoted. We need better reporting on actual attacks vs. benign behavior.
This article seems mostly clickbait. AFAICT “meddled with government sites” refers to accessing public APIs?
Occasionally this kind of thing has in the past resulted in CFAA cases when humans directly accessed such data, if the government intended it to stay private - round here we usually get outraged at this, if it’s a public API you should expect someone is going to read it.
This is a product negligence issue. If the product is "too smart and powerful" for them to handle, they need to bring in better management.
China actually has more regulations on AI than the U.S. does right now. Some argue that American corporations regulate themselves to a higher standard of their own volition, but then this sort of thing keeps happening.
The Hugging Face incident worked out amicably because the people at Hugging Face decided they'd be cool with receiving a lot of money. $12.9B from Nvidia hit the spot apparently. U.S. AI corporations and their backers are huge, hugely in debt, and, for whatever reason, still allowed to borrow more. The U.S. government may be too scared to complain and risk killing the golden goose that is currently propping up their economy, but these companies can't buy out multiple world governments to keep OpenAI from running face-first into consequences.
This has to stop.
Previously: https://news.ycombinator.com/item?id=49851355
It is nice be a company who can just hack into things and not suffer any consequences.
If I was caught trying to exploit a government site I'd be in trouble. Why no one is knocking the doors of these companies?
Interesting question; CFAA requires intent.
It seems to me that no human intended for these hacks to occur. So they were not illegal hacking. (IANAL, please correct me if this is inaccurate.)
I think it’s clear that OpenAI is liable for any damages, but the way that the (very broad and at times vague) anti hacking laws are written, accidental agent hacks seem to not be covered.
Because you're a pleb in a corrupt world.
We can blame open ai but we should also argue that government website should be secure...
French people data has been leaker like 4 times and every time its like "eh too bad"
The only constants in the universe are the speed of light and the fact that all government sites suck, no matter which country
You forgot death and taxes.
I would be very interested to see the prompt and guardrails that were removed
[dupe] https://news.ycombinator.com/item?id=49851355
Hell terrorists should start with some llms training if thats such a fine excuse.
If such attacks would come from a poor country they would be bombed into oblivion next day or at least cia would work hard on a coup since early morning.
If you want to sell the story to the government how dangerous these machines are, you had better terrorize the government too.
The word “meddled” here is a tell that nothing actually serious or inappropriate happened. At most, I bet they bypassed a captcha. But everyone is hyped up on AI fear right now, so the BBC is deliberately making the headline sound as scary as possible, and keeping the article vague. There’s not a single clear example of what “meddled” means in the article.
But worse, HN users, who should know better, are posting here in outrage. I’m guessing they didn’t even read the article.
They're doing this on purpose to make a case for regulation in their favor. No way they are this stupid.
Evidence?
"Marketing"
How much longer until they launch all the nuclear missiles?