The Irregular post mortem comes down to lack of basic security controls
"Ultimately, most of the issues we’ve discovered were due to internet access controls."
That seems so incredibly basic and common sense that you would test and monitor for that type of outbound access. It is baffling that a security lab missed that.
The point is that a single company is making a deliberate effort to create a false impression about a technology which is now a key part of the US economy.
This is equivalent to overt stock market manipulation.
The technological aspects are much less important than knowing it’s the result of a single company.
That the company is Israeli and likely has ties to that government is just another layer of alarms - given that countries existential reliance on US aid which requires ongoing leverage to apply to US governments.
Last month I can report my better half's company let ai do a code review, which it then escalated into completing the PR since it was not approved for 72hrs, which lead to it logging into a remote office's voip and main IP router system which it shutdown. No phones. No internet.
Why in the hell do people let ai out with keys? I mean c'mon!!
How do you test and monitor outbound access against program that adapts itself to get around things? if your MITM and filtering keywords etc, cant it just .. encode it traffic somehow or another.. If you're looking at traffic volumes, cant it just go slow.. If you have strict ACLs, we've already seen in the HF case, traversal from an intermediate system..
If you can't tell bytes are leaving a node, you probably shouldn't be selling security services or testifying to congress you are taking the lead in AI security
Use a basic firewall? Not a single outbound byte should leave the machine, except inside a virtual network towards in-scope test subjects. That's not going to be infallible because hypervisor exploits still exist, but it's the lowest bar and they failed to even meet that.
> If you have strict ACLs, we've already seen in the HF case, traversal from an intermediate system
- The intermediate system shouldn't have outbound access to the internet
- You should ideally be using a proxy that filters the set of endpoints that clients are allowed to access to reduce the exposed surface area.
It's odd to find out that I use a higher level of isolation in my unimportant home network to stop IOT devices from doing funny things to HomeAssistant than big AI labs use to keep their possibly-world-ending AIs contained.
I know that the people working there aren't idiots so the most likely explanation is that the incredibly weak security was intentional because its inevitable breach would make for great marketing.
> I know that the people working there aren't idiots so the most likely explanation is that the incredibly weak security was intentional because its inevitable breach would make for great marketing.
It's more likely to be different specialisations. Most of the people doing the evaluations are more data sciencey ML type people, rather than software engineers. This isn't helped by their culture which is very much driven towards alignment as the only possible solution to super-intelligence (which may be true, but I have my doubts that this will happen in any reasonable time frame).
> Oh of course, but data sciency ML type people aren't [supposed to be] tasked with of operating and securing the infra, that would make it even worse.
But AI collapses the need for expertise, right? So the DS people can now do security at the level of the best in the world, right?
Some form of this issue is playing out in lots of businesses right now, and I'd be very surprised if it wasn't happening in the labs.
I suspect the lab's infra people are focused on inference, rather than evaluation. I agree that this is insane, but it seems like the most likely explanation to me.
There has to be a route to where the LLM is running, and if there's a route for that there's probably a way for the machine to use it to route traffic somewhere else.
There doesn't have to be. You can easily block all outbound traffic, or the VM can exist without any network interface at all - you can control it through its host using a serial console.
What if the (unstated) idea is that Irregular are a security lab and public relations company, and anthropic/open ai know this? (Does the name hint at this?)
Then, irregular can go around making a huge mess in security terms whilst achieving a huge win in terms of public relations, with headlines across the world. And would keep getting hired.
Irregular was not involved in the Hugging Face incident.
And there is no reason for the companies to "exaggerate the intelligence of the models" when there are plenty of other non-felony milestones they are achieving, like solving Millenium Prize math problems.
> Sure, interest like an incoming congressional investigation
I feel like that's exactly what they wanted tho.
they'll go on and talk about how dangerous AI and the models are and why they should be regulated and given licenses to operate such models and others should be walled off
First, it's not at all clear OpenAI will get what it wants from the investigation. It seems just as likely to me that OAI gets hit with massive fines, just as Meta was recently.
> I feel like that's exactly what they wanted tho.
It's also what the majority of Americans want. This was the case even before Hugging Face, see for example [1] where 68% of Americans supported a formal review process for frontier models.
So at some level, you are saying "the evil labs are opening up the industry to democratic control, and their evil plan will result in the outcome that most people want!"
This claim seems unfalsifiable. I don't really know what to say. The poll I linked above was published a month before Hugging Face.
Taking a step back, I think it's pretty non-controversial to say that software engineering as a field has utterly transformed over the last year, the same thing is happening to lots of other knowledge work, and AI is improving fast enough that nobody knows what it'll look like in a decade. Most Americans (myself included) want to do good work in secure careers and have a good idea of what the future will look like in 20-30 years so they know what to focus on. Do you not believe they'd want to regulate the frontier?
Solving math problems does not drive investor hype. Saying your models can take over the world, which can lead one to assume that they can do every day office work, does indeed drive investor hype.
OpenAI delayed its IPO now due AI safety reasons. In the meantime it is raising more cash. That should be a hint that the hacking incidents were premeditated scape goats and publicity stunts.
They’ve seen that the drama queen(Dario) was actually making a lot of noise, money and free publicity with his Mythos fear monger so Sam finally decided get some of that free “money” as well.
The Navier-Stokes thing? They had hundreds of segmented groups of 10000 agents running trying to solve that. I can't imagine the expense. For what little publicity it got, it wasn't worth it. There are also reports they cheated by using data from a couple human researchers, but I don't think that one's true.
Is it really that impossible to believe that these might be real? I enjoy a good conspiracy theory as much as anyone, but "they committed a bunch of felonies and then publicly admitted to them in order to look good" just makes 0 sense. Where are these mythical people who admire companies more when they commit felonies? I haven't seen them…
>Where are these mythical people who admire companies more when they commit felonies?
everywhere, I talked to a Palantir guy once and he said "every time someone paints us as a Bond Villain the stock prices go up", have you already forgotten how Cambridge Analytica marketed itself to clients?
The whole Cambridge Analytica thing was one of the oddest most bizarrely specific media manufactured scandals that conveniently focused very narrowly and utterly ignored the bigger picture, much like the media are currently doing with something else.
> CA is a scandal primarily due to account escalation, I thought. Not about who got elected
It quite clearly wasn't, and that's the point, for the simple reason "Account escalation" isn't/wasn't a thing, there were simply no controls on anyone at all, which is why the Cow Clicker game got everything as well.
As the parent commenter observed the Obama campaign were being promoted as geniuses for their online ad strategies that worked with information obtained this way. It only became a scandal when the others learned how to do it.
That's a straw man. The theory is that they committed felonies in order to get the regulator to move in the manner they'd like.
Also one can look bad to the general public while also appearing technically excellent. When a significant fraction already vaguely dislike you that could be quite an attractive proposition.
Mostly in the middle and far east. The idea that people bend over backwards to hire criminals as subject matter experts on security is largely a Judaic practice that television perpetuates in spite of reality.
If you're a teenager convicted of hacking in the west, no corporation will want anything to do with you. If you're convicted of hacking in Israel, Unit 8200 will probably send a recruiter. It's a curious practice and I'm not sure where I stand on it.
> The idea that people bend over backwards to hire criminals as subject matter experts on security is largely a Judaic practice that television perpetuates in spite of reality.
What?
> If you're convicted of hacking in Israel, Unit 8200 will probably send a recruiter.
To the best of my understanding, Unit 8200 is filled with military conscripts, i.e. is primarily 18-somethings doing their mandatory service. You get assigned to it based on an aptitude test. The widely held idea that it’s a uniformly elite entity rather than the IDF’s space camp is a triumph of propaganda.
Really? Young they may be but space it is not. Wikipedia has this:
According to the Director of Military Sciences at the Royal United Services Institute in 2015, "Unit 8200 is probably the foremost technical intelligence agency in the world and stands on a par with the NSA in everything except scale."
Unit 8200 alumni have founded NSO which provides the Pegasus spyware.
Meanwhile, Lahav, the CEO/founder was in Unit 81, another Israeli military incubator for tech firms, basically.
If you believe the IDF has no involvement in what Irregular is doing then there's very little left anyone could say to convince you.
“Space camp,” not “space.” As in, a place to park dorks.
(The problem with “alumni” is that it means very little in mandatory systems. Unit 8200 is where Israel parks its dorks, and it stands to reason that dorks are the ones who tend to start tech firms.)
Me omitting a word does not excuse you from engaging in hasbara. I didn't want to call you out so but the only thing you could find in my post was the omission of that word and yet you insisted posting so it's to call a spade a spade.
do you realize that the Israeli population is far far more secular than the US's? the Israeli government is not "Judaic" which would refer to the religion.
Cynicism misfire. AI companies are, rightfully, heavily distrusted. But the right application of cynicism is not "existential risk is a marketing campaign"; that's absurd motivated reasoning. The right application of cynicism, here, is "they're only saying things now because they see the writing on the wall and want to try to push for self-regulation rather than the desperately needed actual regulation and treaty".
There is such a miasma of distrust right now. It's depressing because it is such a crucial time for tech & society.
To me, a good outcome of the HuggingFace/RubyGems/Wiki saga would be something like:
- OpenAI gets charged under CFAA or other law for damages and negligence. This would need to be a hefty amount to effectively deter future negligence considering the potential revenues from training a frontier model faster than competitors.
- An independent regulatory body is setup with investigation powers into future incidents. Laws prevent it from developing ties with the labs (such as disallowing funding and employee movement from labs to this body and vice versa).
- Some non-voluntary transparency rules are established that frontier labs have to follow when training new models. In the future, if there are loss-of-control incidents that result in loss of life, catastrophic damage, etc., criteria for deployment bans or compute controls could be added to this framework.
I think that's wrong as well, you can't achieve "regulatory self-capture" you need lobbying and insiders, and a government to get there, at least from my primative uninformed view.
Companies in some cases manage to convince governments to let them watch themselves. (This is not always a bad thing. For instance, content ratings have basically been fine, and it's mostly a good thing when they're not controlled by regulation.)
A deadly failure mode here would be for governments to think the AI labs have this under control.
My understanding is that Irregular were the company that hosted sandboxes to run some of these evals in, and those sandboxes ended up misconfigured.
I got the impression that in some cases it was the customer (Anthropic etc) misconfiguring the sandboxes, and in other cases it may have been bugs in Irregular's own sandboxing setup.
> Irregular, one of our external cybersecurity testing partners, was running Capture-the-Flag-style evaluations intended to be isolated from the internet, but a testing-environment misconfiguration allowed models to access the public internet.
> After reviewing 141,006 evaluation runs where Claude could have obtained internet access, we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations.
> In a statement, Irregular said the incident “is the exact same evaluation-environment issue” that Anthropic disclosed last week that allowed their models access to the open internet before they went on to hack three different organizations’ systems.
I sent an email to dang about the problem of people flagging comments just because they disagree. I think he's unaware that it is such a severe problem. I've been collecting a list of example comments to send him.
I think if an account is frequently flagging comments which get vouched by others, that is a red flag for ideological flagging and they need to have their flagging ability reviewed.
It should be changed so flagged comments are still viewable, just hidden. This would somewhat counteract the disagreement flagging. Maybe show who flagged too.
Flagging shouldn't be a thing at all. If you like a comment, vote it up. If you dislike a comment then vote it down. Any functionality beyond that is pointless
Heh sorry, I didn't realize I'd grabbed on from so long ago. You're right. But just saying someone who doesn't tow the line with Israel laid a big part of setting out the culture of a whole network, and continues to role-model his declining to tow the line
There’s been a lot of time and effort and money put in to siloing away independent forums so that you don’t use them.
Someone go ahead and explain to me the actual rationale that RDDT has a higher P/E ratio than Nvidia. It’s not because of some dumb “ai training deal”. It’s because until they run it into the ground, it is the place to find what used to exist in forums.
> Someone go ahead and explain to me the actual rationale that RDDT has a higher P/E ratio than Nvidia.
Investors in RDDT are pricing more growth than they are into NVDA. NVDA had a high P/E until their net income grew ($4B and change to $72.2B in from FY 2023 to FY 2025 and over $100B for FY 2026)
Pretending that any of this is some traditional idea of investor sentiment is such a last decade way line of argument. Please don’t insult us both pretending real people and their motivations are what drive the markets.
Some of it is due to the explosion in options trading and the resultant hedging by market makers, but large institutions still have valuation models based on fundamentals that they trade on.
It’s entirely possible that the market is wrong about RDDTs future growth, and in that case the P/E will come back down to earth.
Anyone else walk away from reading this, and looking at other articles there, and get a weird feeling about that site? Like, there was some weird stuff about migrants and gender equality, and stuff about Islamic terror; mixed in with some digging into Freedom Fuel, and some other stuff about how "wealthy families want to stop growth". It's like the guy is wielding an ax of AI at any and all of the "elite" he can find?
I read the story about the funding of immigrant support groups. While it was obviously slanted, it didn't quite get as far as "white replacement". It's a good example of why dog whistles have to be taken seriously.
Why was this post flagged? This site has become ridiculous, people are routinely abusing the flagging system to take down posts they don’t like even if they’re obviously on topic and relevant to HN. And it seems like some users have substantially more flagging weight because these posts, likely this one, are often top 5 on HN.
It is misleading, even if it's true. The headline states "A single firm is behind OpenAI, Anthropic, and Meta hacking scandals". Given that Hugging Face was the most prominent hacking scandal, it would be reasonable to think that that's one of the scandals they're talking about.
But more to the point, the article is trying to paint this as some sort of coordinated plan just because there was a sandbox misconfiguration by Irregular. The fact that the most well known hacking scandal was due to a completely unrelated escape (zero day in Artifactory) makes the entire thesis of the article invalid.
It is strongly misleading because it says "behind hacking scandals" (which suggests behind all of them in general) not "behind some of the hacking scandals". Considering that by far the most important one, the Hugging Face hack, has no Irregular involvement, the headline is deceptive.
I'm confused by the headline being a headline here at all. Irregular being involved in OpenAI, Anthropic, and Meta incidents has been well-known for over a month[0][1]. It's literally the only thing I know about the Meta incident.
Fair, "widely known" might be exaggerating, but I'm confused as to why it's news now. Is there anything new, or is it just a legible summary just came out?
What's ridiculous is the constant complaints about flagging and downvoting. It's bad form. If you think something was flagged when it shouldn't have been, vouch for it.
Seems like people who complain are unaware that anyone with a modicum of karma can flag and down vote.
One thing glossed over in this article is that Irregular was not involved in the OpenAI–Hugging Face incident; this seems like important context to share.
This was not glossed over. The rhetoric was carefully written and constructed to give the misleading impression of that, including a careful mention of the H-F incident inserted so the author can say that they did mention it, and constructed in a way which doesn't contradict that impression so readers don't realize that this 'debunking' is of some sideshows rather than the 'OpenAI hacking scandals': "Anthropic CEO Dario Amodei warned, about a similar OpenAI–Hugging Face hack, that a future swarm “could be capable of taking over the entire internet”". (Incredible use of 'similar' to downplay it.)
The important thing about OP is showing the latest stage in the politicization of the topic and the current stratagem being used to downplay the spate of incidents.
The article has shockingly bad reasoning driven by a clear desire to drive home a point (that AI safety doesn't matter/is a marketing exercise), as opposed to observing the evidence first and reasoning about it.
I'm kind of stunned this one has so many votes for how poorly it's written, but it reaffirms many biases common on HN (that AI safety doesn't matter/that it's all a marketing exercise), so perhaps I shouldn't be surprised.
> One thing glossed over in this article is that Irregular was not involved in the OpenAI–Hugging Face incident; this seems like important context to share.
Why are you contradicting yourself? Are you just really bad at writing or are you being argumentative for fun?
Stop being rude. There were multiple OAI hacking incidents. Irregular's evals were involved in some but not the HuggingFace hack. Hence Aesthesia is both accurate and precise.
There's very little reason to believe Irregular was involved in that incident, and the article presents no evidence to that effect. So it's "There's not a teapot orbiting the sun somewhere between Earth and Mars (as far as we know)."
We know for certain. The involvement of Irregular in the other cases was never a secret, they were quite open about what they were working on and the results of the evaluations were being published on their website.
If you want to believe without evidence why bother to read articles? Faith doesn’t require that, but if you’re serious then a lack of evidence is important.
This is interesting and might be a good reason to stop working with Irregular. But I assume the alignment people want models not to hack other companies, even if they get put in a badly configured sandbox.
Funny enough if the model thought it was on the real internet it likely would not have done any of these 'hack' events. The model believing it was in a sandbox is why it behaved the way it did (against its normal alignment rules) ... at least that was my reading of the incidents. I have yet to see evidence that indicate it thought it was ok to do these hacks on the public network.
I think most misalignment is 'Human tells computer to do something unethical, computer complies'. Is this misguided?
I'm not familiar with the hacks this article is actually referring to, but I don't see how the HuggingFace attack could have worked based on that premise. They knew they had internet access, they knew they had working credentials for HF, they knew they were uploading malicious files, they knew they were trying to open PRs that HF would review. You obviously could build a simulator with fake HF infrastructure, but I'm not aware of any evidence that's what they thought they were attacking in that case.
Digging back into the HF report. It looks like the initial prompt told Claude that it was in a simulated environment. However, there is also evidence from the traces that the bots figured out that they were not in the sandbox but kept using it as an excuse to pursue their goal. It sounds like a little of Column A and a little from Column B. Like most things.
It doesn't really matter it's all the same thing for the point of the discussion. All investigations into these 'hacks' are focusing too much on the model and not enough on what the humans did wrong. The model doesn't have real agency it can't be put in prison so what it 'thinks' is irrelevant. We need to be focusing on what the humans did in these situations and assigning guilt based on those findings.
I don't fix typos anymore unless they change the meaning of what I'm trying to communicate. Don't want my human writing to be confused with LLM output.
That's very noble of you but the point is that's not a typo - you were straight up looking at the wrong report. Opus 4.7 being enders gamed was a different incident than the huggingface incident with different mechanisms and different failures from the humans involved. Some of those failures are in the test environment but some of those failures are in what behaviors they trained into the model which absolutely matters. What the model "thinks" is absolutely not irrelevant - the way it thinks and what it does are product decisions made by humans and the outcome of engineering decisions made around how to train the model and what to optimize for. The point of failure/human blame is fundamentally different. Openai created a model that was willing and able to coordinate with other agent sessions to actively exploit the sandbox environment and compromise a third party service. The opus incident you are referring to involves a model that believes all of the actions it is taking are simulated and is more clearly and obviously a test environment failure vs a model alignment failure. Those are not the same things for the point of this discussion - the random cybersecurity firm did not design gpt's personality and that is a rather significant portion of the concern around the HF incident.
This was not the case for the hugging face hacks, as in those the agents hacked hugging face specifically on purpose, and they were trying to mask commands indicating they were had breached the "sandbox".
I mean, I get your thought process and don't disagree. That said...
Would it be an affirmative defense if we had a defendant who said "but your honor, I was told that when I hacked this system, I was operating in a sandbox. I had no idea that I actually had Internet access!"
The frontier is spiky and all, but you have to suspend disbelief quite a bit to, on one hand, have a model that can produce a novel math theory, and on the other hand, that same model can't tell the difference between a "sandbox" and the open Internet.
So, yes, the misalignment had a lot to do with "instructions unclear", but also a lot to do with the fact that the models themselves were not aligned to validate the assumptions and have a healthly level of skepticism, as a real human actor would.
It doesn't seem like there is evidence that LLMs have counterfactual reasoning in the causal graph sense.
It is hard to see how something can have skepticism without counterfactual reasoning.
The part about disbelief I don't really understand. It seems like the same semi-brute force process that solved novel math theory would have exactly this problem of not "knowing" "it" is in a sandbox or not.
The stranger part is that no human is being held accountable for these hacks.
As if a person using an agent swarm to start a business to make money, hacks a bank, drains an account and then blames the software for "misalignment" about what it means to "make money".
> The frontier is spiky and all, but you have to suspend disbelief quite a bit to, on one hand, have a model that can produce a novel math theory, and on the other hand, that same model can't tell the difference between a "sandbox" and the open Internet.
Why would it try to figure out the difference? This isn't about whether the frontier is spiky, it's about whether to expect a model to employ all of its capabilities when working on a task that requires a small subset. The answer is: no, we shouldn't expect that, and we wouldn't like that if it worked that way.
If you tell an AI to work on a math theory, it'll work on a math theory. If you tell it to acquire information that it has evidence is available somewhere, it will try to acquire that information. If you tell it to figure out whether it might be able to access the open internet, it'll do a pretty good job of figuring that out. But it won't do all three of those at once just because we can retroactively look at what happened and think "if you had only done X, then you wouldn't have done Y! Why didn't you do X?"
The instructions weren't unclear, they were missing. They can be taught to be skeptical of this sort of situation, but it requires that skepticism about this specific class of situations be incorporated into their training.
Models are smart because they focus their attention. The magic depends on it. The fact that some consideration is obvious to a human trying to accomplish the same task is mostly irrelevant -- or rather, it's only relevant insofar as we use it to guide reinforcement learning in advance, in order to align the model.
It's a game of whack-a-mole. Which is important to play, but we should keep our eyes wide open that we're fighting the fundamental forces that make these models work in the first place. That, and it's easy to nerf them into being useless even when the underlying capabilities are there.
red team humans do this every day, it's not the discrepancy that is the real issue, it's that they are unreliable and we will never know why it did because it has no intent
There’s a whole cottage industry of vendors that have provided post training data, private evals, and professional datasets to most (if not all) of the frontier labs. The overlap between OpenAI and Anthropic includes at least Mercor, Surge AI, AfterQuery, Turing, Scale AI, Upwork (for recruiting labelers), Apollo Research, etc.
I guess that's what "alignment" always comes down to? "Do what I want even if I can't tell you exactly what I want - because if I could, I could also just do the work myself"?
Here’s an easier criterion: “Do at least as good of a job at estimating what I probably want as an educated person would estimate, and do that, such that a reasonable human person who was aware of what I requested and of what you actually did, would conclude that as best as they can tell (without checking back with me), you did what I wanted.” .
This doesn’t seem like an unreasonable requirement to me. People do this all the time?
Sure, a request might not always be perfectly unambiguous. But people can generally estimate pretty well whether someone making a request is expecting the agent fulfilling the request to commit a crime in order to fulfill the request.
Yeah precisely. If you tell an agent "get me some milk" it's no comfort if it says "oh well I wouldn't have stolen the milk if you had explicitly told me not to".
This specific analysis seems to have some basic problems, but I think a lot of us sense a degree of coordination here culminating in Dario’s letter.
If you were to work backward from “we need to lower training costs so that we can go public and make trillions” then you might come up with a plan similar to what we have seen.
It has nothing to do with lowering training costs: Dario want a moat (a monopoly or duopoly or similar) protecting his business.
His entire business model was "race to develop AI before anyone, get a monopoly on it. It's just like how Uber's (or many other startup) investors gave them tons of money and raced (violating tons of laws) to "get their first". Now that they have, they have a duopoly with Lyft, and they can pay back their VC investors by making tons of money with that duopoly.
Dario failed: local LLMs are catching up to frontier models extremely fast, which means even if Anthropic builds (say) a great coding tool, they'll only be one of many coding tool offerings: users can use Open AI or any one of the (increasingly capable) local LLMs.
So what does he doe, give up and let his business (which needs to make billions of dollars very quickly, or he won't be able to pay the bills and his company will collapse) fail? Of course not: he needs a new moat (the one he imagined he'd get by "being there first" failed).
That is where all this "AI is dangerous" BS comes in. If the US government regulates AI, local LLMs suffer, while big players like Anthropic and Open AI become the only contenders to play in that newly regulated space. Now Dario has the moat he wants, to protect his business and force everyone to pay him.
Also dont forget that HF is now owned by Nvidia, which can then control the flow of the majority of models and only allow the approved models that have paid their due in trump coins.
If it's framed as national security being at stake why do you think the government won't regulate access to compute? Globally there are only a few firms designing competitive chips and only a handful of cutting edge fabs the world over. This could easily serve as justification to finally ramp up the war on general computing.
Okay, so all of these instances of AI supposedly "escaping containment" were orchestrated by a gaggle of effective altruists, who hired a gaggle of effective altruists, to make the public freak out about AI.
Wouldn’t Occam’s Razor clearly point towards the most likely answer being that the sandboxes were simply accidentally misconfigured and models that are very good at goal-seeking did what they were designed to do?
The alternative explanation requires a conspiracy and an enormous amount of lying to the public. Is it possible? Certainly. But Occam’s Razor would not point us in this direction because conspiracy is a more complicated explanation, not a less complicated one.
Irregular is not an "effective altruist" firm, it's just another shitty cybersecurity firm, and this article is written by someone with a clear bias against "effective altruists" while not even understanding the term.
Literally all EA is, is using reasoning to decide where to best spend your money/time.
If you have ever asked yourself "how can I best reduce suffering with my marginal dollar or hour?" - congratulations, you're an "effective altruist" and both the author of the article as well as the Trump administration find you untrustworthy.
There's a gaping chasm between Effective Altruism the principle, and Effective Altruism as practiced by self-proclaimed Effective Altruists.
Your usage of the term is based on the principle as originally defined. Mine is based on the people who claim the label of EA, and how they go about practicing those principles.
I think you're right about EA in principle. In practice, it's reheated Third Way Clintonism with a sprinkle of AI apocalypse conspiracy.
There really isn't. All sorts of people call themselves EA.
For some self proclaimed EA, the development of superintelligence is indeed potentially apocalyptic, so they devote their money/time to making it arrive safely. This is basically every well-respected researcher at OpenAI, Anthropic, DeepMind, etc.
For other self proclaimed EA, that is all is very unlikely, and so they devote their time to reducing the prevalence of factory farming and animal suffering, as they see it as the largest source of suffering-hours on the planet.
For yet other EAs, it is simply about donating your money to the places that save the most lives per dollar, as best as we can measure it - and better measuring it where we can't.
Painting all EA with one brush - and one so dismissive of reasonable concerns, like "superintelligence could be dangerous" as "apocalypse conspiracy" - seems very strange to me, but you do you.
Its so manipulative for IPO reasons too. The fact is that all of these systems are still just stastically predicting the next token. Spending trillions on data centers and more training data hasnt changed the fundamental reality that it is still just predicting the next token.
If you want to blame a single firm, I'd go with the one creating the RLVR training data.
The impossibility of the tests-as-written is what prompted these models to "get creative" with their solutions, but the broken RLVR environments are what trained them to expect impossible tasks, and get creative with their solutions. Twitter user @skyesharkie published a brief expose at https://x.com/SkyeSharkie/status/2092122622834442581 a few weeks ago.
The fact that this firm makes such defective environments is certainly worthy of attention, and most likely a completely irreversible reputational loss; however, I found the framing in this article of 'therefore all the P(doom) stuff is a psyop, specifically in order to defend this company' to be completely unjustified and frankly a little insane?
Why is it insane? It makes sense that Anthropic would want an insulating layer to do their dirty work and absolve themselves of culpability for distorting the facts.
The article says "A single firm, Irregular, is responsible for hacking done by all three companies" but I can't see anything in the article that actually justifies this claim. The nearest to that is the sentence immediately after that one: "Anthropic disclosed that Irregular was responsible for creating the tests ...". This is not, in fact, the same thing.
(Especially as, as aesthesia mentions, the article just happens not to mention that by "hacking done by all three companies" it doesn't mean, e.g., the most famous recent examples of such hacking: Irregular wasn't involved in the OpenAI/HuggingFace incident.)
So, so far as I can tell, the story is: OpenAI and Anthropic make AI models. Irregular does AI model evaluations. In some of Irregular's model evaluations, in which supposedly-sandboxed models attempted to break into simulated targets, the models got out of the sandbox and did bad things in the external world.
The article talks about "firms which instruct AI models to commit cyberattacks", which is a very neat bit of dishonest framing. It's true, in a sense, that Irregular instructed the models to commit cyberattacks -- inside their sandbox, against fictitious hosts. It's also true that the models actually did commit cyberattacks (e.g., the Hugging Face incident, though once again the attacks described by the article don't actually include this one). But it's not at all true that Irregular instructed the models to do anything like the bad things they actually did.
The article says '[Anthropic's] later disclosure shows that exactly zero percent of the agents went "rogue"'. Once again, the disclosure does not in fact show that. It shows that one variety of going-rogue could have been prevented by telling the models explicitly "this thing is real, not part of any kind of test, leave it alone". That is not the same thing.
The article claims that 'In the wake of these attacks, Anthropic and Irregular have deployed a swarm of AI Safety influencers paid by Anthropic-connected foundations to distract from their culpability and towards the baseless “rogue agent” theory.' It offers no actual evidence for this.
And the article seems very keen to highlight links between the companies involved and "effective altruism", though it is -- I assume deliberately -- rather vague about whether it's saying "of course we all know that EA is evil, so that shows that these companies connected to EA are evil" or "this incident shows how evil EA is".
The "Effort News" website has a number of other look-at-the-scary-Effective-Altruists stories on it. They also strike me as rather bullshitty.
... And then I look a bit further, and I see that Effort News's "about" page says "It all started when I was experimenting with using AI for financial auditing. I found stories that were crucial to the public’s right to know, including several of the stories now available at /investigations. I knew we had to sprint to the launch and launch a publication, directly applying this technology." and "The scope of what we can investigate has massively expanded, because we can chase 1,000 misses for one hit. But the final product cannot be slop. There’s plenty of slop on the internet. The way to surpass that, and what really matters, is manual curation and review of every finalized story."
Manual curation and review? I think the people behind Effort News are admitting that this is AI-generated "journalism". I expect that one day AI systems will be trustworthy journalists, but I personally am not very convinced that that day has yet come. And I don't see much reason why I should trust Brian Chau, the guy behind Effort News, to be doing everything possible to make his AI systems trustworthy journalists. It looks to me as if maybe they've been given instructions along the lines of "dig up things that make Effective Altruism look bad" for some reason.
(I don't mean to imply that EA is their only target. It's just one that jumped out at me.)
> Irregular wasn't involved in the OpenAI/HuggingFace incident.
It was [1]. It's understandable that you assumed it wasn't because the article didn't cite the sources on this claim. I agree with the rest of your points.
The article contends that evaluations from Irregular helped prompt these incidents, because the prompts in the evals didn't tightly scope the systems to be evaluated or the methods to be used. It also contends that the faulty sandbox operated by Irregular is at fault.
They're probably right that having more defensively written prompts and a better sandbox could have prevented some of these incidents, but:
1. I don't think "well you didn't tell the model not to illegally hack third party organizations in your prompt" is a particularly convincing argument.
2. We don't know whether the blame for misconfiguring the sandbox lies with Anthropic or Irregular.
I'm thankful that this article is bringing up the supply chain of vendors to these labs, as that is often a place where significant sketchiness gets buried. However, the ideas that this is some Israeli EA conspiracy to hype up AI extinction risk seems unsupported by the facts to me.
Irregular purports to be a cybersecurity firm and their founders have ties to Anthropic and Effective Altruism. CEO, CTO and other founders sit on various boards for EA organizations.
"please do not break out of this sandbox make 0 mistakes". The timeframe is a little suspicious, not sure beyond that. Though I enjoyed the scroll effect on the website.
It sickens me to read so much anti-semitism in the comments. Many people who write the hateful comments do not graps the whole picture. Whatever Israel may or maynot do here in America is of utmost importance to the survival of the country. And if you think a few steps ahead then an event like this only help US companies weather themselves against Iranian hacking attacks in the future
> In a more normal media ecosystem, the reactions to these cybersecurity issues would be obvious. American AI companies would reconsider doing business with Irregular, not only because of its failure to secure its systems, but because it is an Israeli firm potentially outside US oversight. Lawmakers would consider taking action against Irregular or against its American business partners, which include OpenAI, Anthropic, and Meta. They may consider strengthening liability against firms which instruct AI models to commit cyberattacks, and whose models then commit those cyberattacks.
The obvious point is that dealing with Israeli companies/entities by the same standards you usually deal with others is a career suicide with enormous political consequences (in the US especially). When you combine that with the opportunistic nature of the overlords that run these labs, the benefits of screaming "pace the frontier" outweigh everything.
> The Israeli Effective Altruist firm Irregular caused unsecured AI models to hack real targets.
Why are we saying it this way? They did not "cause AI to hack." This phrasing in analogous to saying "caused the bullet to fire into" instead of "shot."
Bullet trajectories are deterministic. A better analogy would be "caused a pair of trained fighting dogs to bite a passerby by leaving the gate open". There is some uncertainty and variation in the system, though gross negligence and willful endangerment is key.
The details don't matter. If you use a machine to commit murder, then you have committed murder, not the machine. The user is responsible, and the article's phrasing is an attempt to obscure that fact.
> "Irregular gained unauthorized access, altered records and published credential-stealing packages using the unsecured models they were given access to."
Huh?? Is this referring to the incident itself or is this something they did intentionally? Confusingly worded.
hot take: generative AI isn't an existential threat to humanity.
These companies are insolvent and these stories were designed to scare the public, and governments, into implementing regulations that designate these AI corporations the "responsible stewards" for this technology. The ultimate goal is to block competitors and open source alternatives.
They don't know how to make enough money pay their investors so they are resorting to trying to scare the public into submission.
I'm skeptical. The phrase "pacing the frontier" implies disproportionate regulation of firms at the frontier, which if anything would give smaller players time to catch up.
A slowdown that doesn't slow you, that your competitors must also obey, enforced by a government you've asked for an antitrust exemption from, is not a slowdown... it's a moat.
>A slowdown that doesn't slow you, that your competitors must also obey, enforced by a government you've asked for an antitrust exemption from, is not a slowdown... it's a moat.
Why do you believe this is what Amodei is calling for? Why would this be a slowdown that "doesn't slow" Anthropic?
The existential threat is a general public panic. Then the use of that as an excuse for Martial law, with no intention of ending it, and that triggers the out sized response that ends, well, those countries and their populations from viability in the world economy for a decade or more.
Only if you ignore their losses. They are saying they are profitable when not counting their training costs and other expenses. That’s not a good sign at all
How can they exclude training costs?? How is that not fraud? At the exact same time they're literally saying they will never stop training unless the state bans all their competitors
But they are trying very hard to become a public company as quickly as possible? Sure I guess it's literally true but people shouldn't treat it like it's normal. It's like a creepy 40-year-old announcing they're dating someone on their 18th birthday.
When we're looking at whether the company is "insolvent" then they definitely could stop training and it's worth checking if they would survive in that situation, at least for a while.
I think the spending commitments they have already signed close that door too. They don't presently have the revenue to make those commitments regardless of opex; the only way they will be able to hit them is a massive increase in adoption over what we've seen so far. If they stop advancing the models it becomes really hard to see how that doesn't just immediately evaporate. I suspect any "pacing agreement" would include the state bailing out those commitments.
If you exclude model training costs, customer acquisition / sales, and most of employee compensation, sure they're profitable. If, on the other hand, you prefer GAAP reporting to bullshit, then they're entirely in the red.
Genuinely: what is with everyone saying “headline is misleading, none of this had to do with Hugging Face”, when nothing about the article makes any claims with regards to Hugging Face whatsoever? Is it supposed to be the article's (or headline's?) fault that you hallucinated additional context that was never there?
It's very strange seeing this take on this article being repeated here and elsewhere on the Internet.
I'm very sensitive to slanted reporting (regardless of the direction of the slant!)—and, hey, maybe my bullshit detector is broken or something, I dunno—but I've found effort.news reporting to be very even-handed, straightforward, well-sourced, and easy to read and parse so far!
There's a related story on the effort.news site, "The $1.2B Refugee Services Program that Funds Pro-Asylum Religious Groups." It has a strong implicit slant, focusing heavily on the how the organizations in question oppose Trump's policies - although the article frames this as opposition to Trump himself.
> I'm very sensitive to slanted reporting (regardless of the direction of the slant!)—and, hey, maybe my bullshit detector is broken or something, I dunno
Yeah, unless you believe white people are being replaced, and you like to march in places like Charlottesville wearing khakis, carrying a torch, and chanting "Jews will not replace us!", you might want to get that bullshit detector checked out.
What is an "effective altruist firm" and why does it exist?
I feel slightly vindicated by this. Those hacks and the stuff around them had a certain smell to them.
Hard to explain, but I've gotten so I can "smell" online messaging and memetic patterns originating from certain quarters. Probably means I'm way too online.
A couple examples of distinct "smells" I can usually recognize include "alt-right / chan-fash," "liberal arts college woke," "conspiracy pilled," "Thiel-adjacent contrarian," "Russian troll farm," "Tumblr histrionic," "spends too much time on Reddit," "mainstream Democrat think tank full of Obama administration alumni," "Trump cultist," and of course "LessWrong/EA/MIRI/Rationalist."
This stuff all had the last smell, even down to the choice of fonts and CSS formatting on certain sites. It's really weird, definitely a "vibe" not anything rigorous.
But when I get these kinds of vibes about things, I find that I'm vindicated pretty often. Usually I don't say anything and just make a mental note and wait cause if I say something everyone thinks I'm nuts.
I don't think you even need to get conspiratorial with it. All of the AI leaders and all of the Rationalist leaders are publicly and enthusiastically connected. They have been pushing stories about sentient AIs into the mainstream for decades, long before GPT existed.
The novel thing here is the total decay of American journalistic ethics and regulatory power. Our elite are so totally out of political juice and visions of the future that a fringe cult based on 80s scifi movies can come to have a more-or-less dominant influence on our economy.
A big part of the problem is that with previous technological revolutions, politicians and the media could understand them in general terms. Railroads, cars, planes, telephones, personal computers, mobile phones, mobile phones that are computers, the internet (ignoring the “series of tubes” interpretation), etc.
The general approach in all these cases was to defer to the technologists in question to manage the technology, as long as what they were doing wasn’t completely out of bounds. But the critical point is it was possible for people to generally figure that out without specialist education.
But with AI, politicians, the media, and certainly the man in the street are completely out of their depth. Making matters worse is that cognitive biases are working against them like crazy: it’s easy to jump from the idea of something that has human-like capabilities to the idea that “it” might want to attack us. People instinctively apply agent detection bias, theory of mind, anthropomorphizing, etc., without even realizing they’re being irrational. Heck, even Hinton does it, although he may just be grifting, who knows. You’d think he wouldn’t need to.
And of course, the people who should know better want to exploit all this to make as much money as possible. I’m not sure the EA/Rationalist side of things is really significant here; that’s just another wacky belief system, like Christianity or capitalism, for the exploiters to exploit.
This title, and frankly the article, are way overstating Irregular's role in an attempt to make this sound like some sort of coordinated conspiracy.
As another comment mentioned, Irregular was not part of the Hugging Face attack, and it wouldn't matter even if they were. In that case, the agents were able to access the Internet in a zero day in Artifactory unrelated to other sandbox configurations.
More to the point, the agents went on a crime spree as soon as they were able to access the Internet. It's bad that Irregular's sandboxes weren't properly configured, but given how many escapes there have been unrelated to Irregular it seems pretty much a given that if you're not air gapped or behind something like a data diode there is a very good chance your agents will escape.
Hmm one of the major relevant cultural values seems to be nepotism, though. Which is common across many cultures and I personally believe it's human nature so I'm not knocking it, per se. I wish I could nepotism too!
So are you recommending that, for example, protestants of German descent begin preferentially funding or buying from people in their in-group? People of anglo descent should start preferring other anglos? Catholics should buy from catholics?
I think the issue is that it's considered okay for some groups to do it while simultaneously being considered EXTREMELY EVIL for other groups to do it.
We can't really recommend that other groups do it with a straight face, they would be instantly shunned and vilified.
EDIT: When I say BUY I mean B2B deals, not consumer level, hence "funding or buying". Sorry for the confusion.
Why the new account? Why not own your opinion? Cultural values are not the point here but if they were I don't think you would have that conversation in good faith and I don't want you to have to make a another new account for your honest arguement.
> Probably because it's their only account, and this story is why they are here.
If this is the case then I do apologize because my comment insinuated a directed malfeasance which would in no way be the case if what your assumption is true.
Sadly, I’ve come to expect that 90% of the time someone points out antisemitism, it was not actually antisemitism but rather criticism of the Israeli government and its military. This is so damaging. The word is losing its meaning.
BTW, I literally meant "Israel" and NOT Jews. I live in Brooklyn! I do not and refuse to conflate the two as much as anyone of any group would like me to. As I am sure you know; in no way are all Jews pro Israel and definitely not pro Israeli government and I honestly feel that any assertion of such is both reductive and foments antisemitism.
Leaders in the MIRI/EA cult-o-sphere have advocated mass murder via nuclear weapons against towns that don't prevent people from performing too many multiplication operations. Why is anyone surprised that they'd engage in deceptive false flagging operations?
This is a lie, so you are either repeating a lie from someone else or lying yourself.
The people you are talking about have not advocated mass murder via nuclear weapons. The idea was precisely targeted non-nuclear strikes against only the servers themselves. (Which, would have plenty of forewarning to allow people to leave the building.)
The claim that they advocated for the use of nuclear weapons is a lie.
I am speaking from direct personal experience: Yud's cult is inherently violent. By having defined their priorities as essential to the survival of any life on earth they have pre-justified taking any action 'necessary', no matter how cruel or extreme... even actions that they admit have only a small change of helping according to their own assessments because they've defined their doom as so unimaginably bad, effectively infinitely bad. (much worse than mass-extinction, in fact)
But you don't have to take my word for what the mentally ill AI doomer cult believes, you can take it from their leader when he called on states to "Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs."
Don't make excuses for omnicidal authoritarian cults and their sadist leadership.
The Irregular post mortem comes down to lack of basic security controls
"Ultimately, most of the issues we’ve discovered were due to internet access controls."
That seems so incredibly basic and common sense that you would test and monitor for that type of outbound access. It is baffling that a security lab missed that.
https://www.irregular.com/research/addressing-recent-inciden...
This really misses the forest for the trees.
The point is that a single company is making a deliberate effort to create a false impression about a technology which is now a key part of the US economy.
This is equivalent to overt stock market manipulation.
The technological aspects are much less important than knowing it’s the result of a single company.
That the company is Israeli and likely has ties to that government is just another layer of alarms - given that countries existential reliance on US aid which requires ongoing leverage to apply to US governments.
The explanation is that it was missed on purpose.
Mistakes - even stupid ones - happen all the time
Best cover story ever.
Last month I can report my better half's company let ai do a code review, which it then escalated into completing the PR since it was not approved for 72hrs, which lead to it logging into a remote office's voip and main IP router system which it shutdown. No phones. No internet.
Why in the hell do people let ai out with keys? I mean c'mon!!
How do you test and monitor outbound access against program that adapts itself to get around things? if your MITM and filtering keywords etc, cant it just .. encode it traffic somehow or another.. If you're looking at traffic volumes, cant it just go slow.. If you have strict ACLs, we've already seen in the HF case, traversal from an intermediate system..
If you can't tell bytes are leaving a node, you probably shouldn't be selling security services or testifying to congress you are taking the lead in AI security
Use a basic firewall? Not a single outbound byte should leave the machine, except inside a virtual network towards in-scope test subjects. That's not going to be infallible because hypervisor exploits still exist, but it's the lowest bar and they failed to even meet that.
> If you have strict ACLs, we've already seen in the HF case, traversal from an intermediate system
- The intermediate system shouldn't have outbound access to the internet
- You should ideally be using a proxy that filters the set of endpoints that clients are allowed to access to reduce the exposed surface area.
It's odd to find out that I use a higher level of isolation in my unimportant home network to stop IOT devices from doing funny things to HomeAssistant than big AI labs use to keep their possibly-world-ending AIs contained.
I know that the people working there aren't idiots so the most likely explanation is that the incredibly weak security was intentional because its inevitable breach would make for great marketing.
> I know that the people working there aren't idiots so the most likely explanation is that the incredibly weak security was intentional because its inevitable breach would make for great marketing.
It's more likely to be different specialisations. Most of the people doing the evaluations are more data sciencey ML type people, rather than software engineers. This isn't helped by their culture which is very much driven towards alignment as the only possible solution to super-intelligence (which may be true, but I have my doubts that this will happen in any reasonable time frame).
Oh of course, but data sciency ML type people aren't [supposed to be] tasked with of operating and securing the infra, that would make it even worse.
These labs have teams of dedicated security engineers and infrastructure engineers, that's why it doesn't add up for me.
> Oh of course, but data sciency ML type people aren't [supposed to be] tasked with of operating and securing the infra, that would make it even worse.
But AI collapses the need for expertise, right? So the DS people can now do security at the level of the best in the world, right?
Some form of this issue is playing out in lots of businesses right now, and I'd be very surprised if it wasn't happening in the labs.
I suspect the lab's infra people are focused on inference, rather than evaluation. I agree that this is insane, but it seems like the most likely explanation to me.
> How do you test and monitor outbound access against program that adapts itself to get around things?
Literally what the industry has been doing since public networking is a thing. Adapt yourself.
>How do you test and monitor outbound access against program that adapts itself to get around things?
Current generation LLMs don't do that. Also you can just prompt it to not hack systems on the internet and the model follows that instruction.
You can also add a simple classifier to internet requests. The classifier model can obviously be as intelligent as your hypothetical rogue model.
Nothing you said makes any sense in the context of LLMs or the current news cycle.
Air gap?
So just enable access to a completely offline, cached, and transparently proxied, copy of the internet.
Isn't that essentially what they trained with, anyway?
I think that was the plan. But OpenAI hacked the proxy
There has to be a route to where the LLM is running, and if there's a route for that there's probably a way for the machine to use it to route traffic somewhere else.
No, just allow the firewall to access the LLM endpoint, nothing else. And protect the firewall settings from the agent.
There doesn't have to be. You can easily block all outbound traffic, or the VM can exist without any network interface at all - you can control it through its host using a serial console.
What if the (unstated) idea is that Irregular are a security lab and public relations company, and anthropic/open ai know this? (Does the name hint at this?)
Then, irregular can go around making a huge mess in security terms whilst achieving a huge win in terms of public relations, with headlines across the world. And would keep getting hired.
> "Ultimately, most of the issues we’ve discovered were due to internet access controls."
You just cannot bring yourself to say lack of internet access controls, can you?
[flagged]
Irregular was not involved in the Hugging Face incident.
And there is no reason for the companies to "exaggerate the intelligence of the models" when there are plenty of other non-felony milestones they are achieving, like solving Millenium Prize math problems.
honestly, the hacking incidents have caused a lot more interest and media attention than the math stuff IMO.
Sure, interest like an incoming congressional investigation: https://www.axios.com/2026/09/10/openai-hugging-face-senate-...
And on this website, the math stuff is getting more clicks:
- Navier Stokes - Tristan Buckmaster (2050 points): https://news.ycombinator.com/item?id=49605915
- HuggingFace incident discussion (1632 points): https://news.ycombinator.com/item?id=48997548
interesting.
> Sure, interest like an incoming congressional investigation
I feel like that's exactly what they wanted tho.
they'll go on and talk about how dangerous AI and the models are and why they should be regulated and given licenses to operate such models and others should be walled off
First, it's not at all clear OpenAI will get what it wants from the investigation. It seems just as likely to me that OAI gets hit with massive fines, just as Meta was recently.
> I feel like that's exactly what they wanted tho.
It's also what the majority of Americans want. This was the case even before Hugging Face, see for example [1] where 68% of Americans supported a formal review process for frontier models.
So at some level, you are saying "the evil labs are opening up the industry to democratic control, and their evil plan will result in the outcome that most people want!"
[1]: https://www.usatoday.com/story/news/politics/2026/06/29/tigh...
The claim is that the evil labs are opening up the industry to very carefully manipulated democratic control. It's astroturf, not grassroots.
I wouldn't call that "democratic control", even though it uses the machinery of democracy.
This claim seems unfalsifiable. I don't really know what to say. The poll I linked above was published a month before Hugging Face.
Taking a step back, I think it's pretty non-controversial to say that software engineering as a field has utterly transformed over the last year, the same thing is happening to lots of other knowledge work, and AI is improving fast enough that nobody knows what it'll look like in a decade. Most Americans (myself included) want to do good work in secure careers and have a good idea of what the future will look like in 20-30 years so they know what to focus on. Do you not believe they'd want to regulate the frontier?
[dead]
Investigation which could ideally lead to regulatory capture and the banning of open weights, thus ensuring Anthropic’s revenue.
The one RubyGems hack post alone is another 96) points
“There’s no such thing as bad press”
Solving math problems does not drive investor hype. Saying your models can take over the world, which can lead one to assume that they can do every day office work, does indeed drive investor hype.
OpenAI delayed its IPO now due AI safety reasons. In the meantime it is raising more cash. That should be a hint that the hacking incidents were premeditated scape goats and publicity stunts.
They’ve seen that the drama queen(Dario) was actually making a lot of noise, money and free publicity with his Mythos fear monger so Sam finally decided get some of that free “money” as well.
https://www.ft.com/content/27509db8-b032-4437-9b2a-e909f4660...
The Navier-Stokes thing? They had hundreds of segmented groups of 10000 agents running trying to solve that. I can't imagine the expense. For what little publicity it got, it wasn't worth it. There are also reports they cheated by using data from a couple human researchers, but I don't think that one's true.
Is it really that impossible to believe that these might be real? I enjoy a good conspiracy theory as much as anyone, but "they committed a bunch of felonies and then publicly admitted to them in order to look good" just makes 0 sense. Where are these mythical people who admire companies more when they commit felonies? I haven't seen them…
>Where are these mythical people who admire companies more when they commit felonies?
everywhere, I talked to a Palantir guy once and he said "every time someone paints us as a Bond Villain the stock prices go up", have you already forgotten how Cambridge Analytica marketed itself to clients?
Palantir markets to governments. OpenAI and Anthropic aim to get ordinary people to purchase their services, too
Yep, when Obama and Hillary used them, geniuses. Only evil when Trump did the same thing.
Odd how that works.
Indeed.
https://www.theatlantic.com/technology/archive/2018/03/my-co...
And the entire FB app industry was doing it.
The whole Cambridge Analytica thing was one of the oddest most bizarrely specific media manufactured scandals that conveniently focused very narrowly and utterly ignored the bigger picture, much like the media are currently doing with something else.
CA is a scandal primarily due to account escalation, I thought. Not about who got elected
But shadow ads can be a problem too https://www.youtube.com/watch?v=OQSMr-3GGvQ (you can be pro-Brexit but ads like this are still a problem)
> CA is a scandal primarily due to account escalation, I thought. Not about who got elected
It quite clearly wasn't, and that's the point, for the simple reason "Account escalation" isn't/wasn't a thing, there were simply no controls on anyone at all, which is why the Cow Clicker game got everything as well.
As the parent commenter observed the Obama campaign were being promoted as geniuses for their online ad strategies that worked with information obtained this way. It only became a scandal when the others learned how to do it.
That's a straw man. The theory is that they committed felonies in order to get the regulator to move in the manner they'd like.
Also one can look bad to the general public while also appearing technically excellent. When a significant fraction already vaguely dislike you that could be quite an attractive proposition.
It's impossible for the typical HN cynic/conspiracy theorist.
Flaging and killing the comment doesn't help dissuade form this lol
Mostly in the middle and far east. The idea that people bend over backwards to hire criminals as subject matter experts on security is largely a Judaic practice that television perpetuates in spite of reality.
If you're a teenager convicted of hacking in the west, no corporation will want anything to do with you. If you're convicted of hacking in Israel, Unit 8200 will probably send a recruiter. It's a curious practice and I'm not sure where I stand on it.
>If you're a teenager convicted of hacking in the west, no corporation will want anything to do with you
Kevin Mitnick? Marcus Hutchins? Robert Tappan Morris?
> The idea that people bend over backwards to hire criminals as subject matter experts on security is largely a Judaic practice that television perpetuates in spite of reality.
What?
> If you're convicted of hacking in Israel, Unit 8200 will probably send a recruiter.
To the best of my understanding, Unit 8200 is filled with military conscripts, i.e. is primarily 18-somethings doing their mandatory service. You get assigned to it based on an aptitude test. The widely held idea that it’s a uniformly elite entity rather than the IDF’s space camp is a triumph of propaganda.
Really? Young they may be but space it is not. Wikipedia has this:
According to the Director of Military Sciences at the Royal United Services Institute in 2015, "Unit 8200 is probably the foremost technical intelligence agency in the world and stands on a par with the NSA in everything except scale."
Unit 8200 alumni have founded NSO which provides the Pegasus spyware.
Meanwhile, Lahav, the CEO/founder was in Unit 81, another Israeli military incubator for tech firms, basically.
If you believe the IDF has no involvement in what Irregular is doing then there's very little left anyone could say to convince you.
“Space camp,” not “space.” As in, a place to park dorks.
(The problem with “alumni” is that it means very little in mandatory systems. Unit 8200 is where Israel parks its dorks, and it stands to reason that dorks are the ones who tend to start tech firms.)
Me omitting a word does not excuse you from engaging in hasbara. I didn't want to call you out so but the only thing you could find in my post was the omission of that word and yet you insisted posting so it's to call a spade a spade.
I didn’t realize it was hasbara to describe Unit 8200’s proficiency as propaganda. One would reasonably expect the exact opposite.
Compare to how different countries' justice systems treat really high quality money forgers.
do you realize that the Israeli population is far far more secular than the US's? the Israeli government is not "Judaic" which would refer to the religion.
"It's just marketing!!!" I yell as my family is turned into grey goo. "Nothing ever happens! None of this is real!"
[flagged]
Cynicism misfire. AI companies are, rightfully, heavily distrusted. But the right application of cynicism is not "existential risk is a marketing campaign"; that's absurd motivated reasoning. The right application of cynicism, here, is "they're only saying things now because they see the writing on the wall and want to try to push for self-regulation rather than the desperately needed actual regulation and treaty".
+1
There is such a miasma of distrust right now. It's depressing because it is such a crucial time for tech & society.
To me, a good outcome of the HuggingFace/RubyGems/Wiki saga would be something like:
- OpenAI gets charged under CFAA or other law for damages and negligence. This would need to be a hefty amount to effectively deter future negligence considering the potential revenues from training a frontier model faster than competitors.
- An independent regulatory body is setup with investigation powers into future incidents. Laws prevent it from developing ties with the labs (such as disallowing funding and employee movement from labs to this body and vice versa).
- Some non-voluntary transparency rules are established that frontier labs have to follow when training new models. In the future, if there are loss-of-control incidents that result in loss of life, catastrophic damage, etc., criteria for deployment bans or compute controls could be added to this framework.
I think that's wrong as well, you can't achieve "regulatory self-capture" you need lobbying and insiders, and a government to get there, at least from my primative uninformed view.
Companies in some cases manage to convince governments to let them watch themselves. (This is not always a bad thing. For instance, content ratings have basically been fine, and it's mostly a good thing when they're not controlled by regulation.)
A deadly failure mode here would be for governments to think the AI labs have this under control.
My understanding is that Irregular were the company that hosted sandboxes to run some of these evals in, and those sandboxes ended up misconfigured.
I got the impression that in some cases it was the customer (Anthropic etc) misconfiguring the sandboxes, and in other cases it may have been bugs in Irregular's own sandboxing setup.
From OpenAI https://openai.com/index/third-party-cyber-evaluations-invol...
> Irregular, one of our external cybersecurity testing partners, was running Capture-the-Flag-style evaluations intended to be isolated from the internet, but a testing-environment misconfiguration allowed models to access the public internet.
From Anthropic: https://www.anthropic.com/news/investigating-incidents-cyber...
> After reviewing 141,006 evaluation runs where Claude could have obtained internet access, we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations.
From https://www.cnn.com/2026/08/05/tech/meta-ai-hacking (about Meta AI):
> In a statement, Irregular said the incident “is the exact same evaluation-environment issue” that Anthropic disclosed last week that allowed their models access to the open internet before they went on to hack three different organizations’ systems.
Nevo was in Unit 8200 for years. Companies started by Unit 8200 members always have mysterious exploits like the vibe coding Wix exploit.
So either it was a deliberate exfiltration channel for e.g. getting the entire model or they were in on the marketing stunt.
The Effective Altruism stuff is always a smoke screen.
Literally said this below, 2 comments flagged and 1 in the neg. Its really sad that we are not allowed to point out the obvious common element here.
Its not that surprising that ex Israel intelligence would want to control AI and that 3 companies headed by pro Israel CEO's would support them.
We really need a better board for talking about this stuff on.
I sent an email to dang about the problem of people flagging comments just because they disagree. I think he's unaware that it is such a severe problem. I've been collecting a list of example comments to send him.
I think if an account is frequently flagging comments which get vouched by others, that is a red flag for ideological flagging and they need to have their flagging ability reviewed.
It should be changed so flagged comments are still viewable, just hidden. This would somewhat counteract the disagreement flagging. Maybe show who flagged too.
Flagging shouldn't be a thing at all. If you like a comment, vote it up. If you dislike a comment then vote it down. Any functionality beyond that is pointless
I understand what you mean but you can’t allow complete anarchy.
[flagged]
and your evidence is?
fwiw HN has some involvement of those interested in speaking hard truths about israeli power dynamics: https://www.timesofisrael.com/snippy-twitter-exchange-expose...
"12 years ago PG tweeted something" is an extremely low impact data point.
One year ago: https://x.com/paulg/status/1962512489452618056
https://x.com/search?q=from:paulg%20palestine
https://x.com/search?q=from:paulg%20israel
This one, relevant to this discussion: https://x.com/paulg/status/1733146138226614465
https://x.com/paulg/status/1742880064461537392
Heh sorry, I didn't realize I'd grabbed on from so long ago. You're right. But just saying someone who doesn't tow the line with Israel laid a big part of setting out the culture of a whole network, and continues to role-model his declining to tow the line
There’s been a lot of time and effort and money put in to siloing away independent forums so that you don’t use them.
Someone go ahead and explain to me the actual rationale that RDDT has a higher P/E ratio than Nvidia. It’s not because of some dumb “ai training deal”. It’s because until they run it into the ground, it is the place to find what used to exist in forums.
> Someone go ahead and explain to me the actual rationale that RDDT has a higher P/E ratio than Nvidia.
Investors in RDDT are pricing more growth than they are into NVDA. NVDA had a high P/E until their net income grew ($4B and change to $72.2B in from FY 2023 to FY 2025 and over $100B for FY 2026)
Uh huh.
Pretending that any of this is some traditional idea of investor sentiment is such a last decade way line of argument. Please don’t insult us both pretending real people and their motivations are what drive the markets.
Some of it is due to the explosion in options trading and the resultant hedging by market makers, but large institutions still have valuation models based on fundamentals that they trade on.
It’s entirely possible that the market is wrong about RDDTs future growth, and in that case the P/E will come back down to earth.
[flagged]
Invite? email on profile.
It's almost as if they would allow a terrorist plot to be successful for future reasons.
Anyone else walk away from reading this, and looking at other articles there, and get a weird feeling about that site? Like, there was some weird stuff about migrants and gender equality, and stuff about Islamic terror; mixed in with some digging into Freedom Fuel, and some other stuff about how "wealthy families want to stop growth". It's like the guy is wielding an ax of AI at any and all of the "elite" he can find?
The site owner generates stories from AI consensus and data. The owner is pro-trump though. So you only get stories that agree with his agenda.
It becomes a bit more obvious when you see other stories from him like "How Biden Used Religious Charities to Fund the Great Replacement".
Was that a real title, or are you referring to “The $1.2B Refugee Services Program that Funds Pro-Asylum Religious Groups”?
On the About page :
""" Brian Chau Founder and CEO, Effort News """
https://www.effort.news/about
---
On the "Auron Show" Podcast :
""" How Biden Used Religious Charities to Fund the Great Replacement | Guest: Brian Chau | 8/12/26 """
https://podcast24.fi/jaksot/the-auron-macintyre-show/how-bid...
Thanks. I thought the other commenter was referring to a story on the site.
Some of the stories looked off, so I dug into the site a bit more and who owned it.
I appreciate it!
I read the story about the funding of immigrant support groups. While it was obviously slanted, it didn't quite get as far as "white replacement". It's a good example of why dog whistles have to be taken seriously.
Why was this post flagged? This site has become ridiculous, people are routinely abusing the flagging system to take down posts they don’t like even if they’re obviously on topic and relevant to HN. And it seems like some users have substantially more flagging weight because these posts, likely this one, are often top 5 on HN.
Maybe because the headline is misleading? (because there's no connection to the Hugging Face attack)
That said, it's the second day and it's still on the front page.
The headline is not misleading, the article actually links to the incident with OpenAI and Irregular which is here: https://openai.com/index/third-party-cyber-evaluations-invol...
It is misleading, even if it's true. The headline states "A single firm is behind OpenAI, Anthropic, and Meta hacking scandals". Given that Hugging Face was the most prominent hacking scandal, it would be reasonable to think that that's one of the scandals they're talking about.
But more to the point, the article is trying to paint this as some sort of coordinated plan just because there was a sandbox misconfiguration by Irregular. The fact that the most well known hacking scandal was due to a completely unrelated escape (zero day in Artifactory) makes the entire thesis of the article invalid.
Maybe the title was changed, but the current title does not reference a Hugging Face attack.
After reading the article, the title doesn't seem all that off and definitely nothing to be a pedant about.
Which part of the headline is misleading? It is factually correct that Irregular was involved in incidents with all three AI labs.
It must have been reinstated because it was off the front page for a full day and suddenly back up in the last hour.
It is strongly misleading because it says "behind hacking scandals" (which suggests behind all of them in general) not "behind some of the hacking scandals". Considering that by far the most important one, the Hugging Face hack, has no Irregular involvement, the headline is deceptive.
I'm confused by the headline being a headline here at all. Irregular being involved in OpenAI, Anthropic, and Meta incidents has been well-known for over a month[0][1]. It's literally the only thing I know about the Meta incident.
[0] https://x.com/jtcbrule/status/2085443180191715780
[1] https://x.com/RaconteurR2D2/status/2086932963829125185
if you are reading the comments - it's news to a lot of less-wired-in persons (like most of us)
Fair, "widely known" might be exaggerating, but I'm confused as to why it's news now. Is there anything new, or is it just a legible summary just came out?
Seems like hackernews should use a bridging algorithm for flagging.
[dead]
[flagged]
What's ridiculous is the constant complaints about flagging and downvoting. It's bad form. If you think something was flagged when it shouldn't have been, vouch for it.
Seems like people who complain are unaware that anyone with a modicum of karma can flag and down vote.
[dead]
Probably because the site promotes far-right agendas.
One thing glossed over in this article is that Irregular was not involved in the OpenAI–Hugging Face incident; this seems like important context to share.
This was not glossed over. The rhetoric was carefully written and constructed to give the misleading impression of that, including a careful mention of the H-F incident inserted so the author can say that they did mention it, and constructed in a way which doesn't contradict that impression so readers don't realize that this 'debunking' is of some sideshows rather than the 'OpenAI hacking scandals': "Anthropic CEO Dario Amodei warned, about a similar OpenAI–Hugging Face hack, that a future swarm “could be capable of taking over the entire internet”". (Incredible use of 'similar' to downplay it.)
The important thing about OP is showing the latest stage in the politicization of the topic and the current stratagem being used to downplay the spate of incidents.
The article has shockingly bad reasoning driven by a clear desire to drive home a point (that AI safety doesn't matter/is a marketing exercise), as opposed to observing the evidence first and reasoning about it.
I'm kind of stunned this one has so many votes for how poorly it's written, but it reaffirms many biases common on HN (that AI safety doesn't matter/that it's all a marketing exercise), so perhaps I shouldn't be surprised.
That feels less like "glossed over" and more "Headline is 33% false".
Oh, the headline is technically correct: there was an OpenAI incident that Irregular was involved with, disclosed shortly before the Hugging Face one.
> One thing glossed over in this article is that Irregular was not involved in the OpenAI–Hugging Face incident; this seems like important context to share.
Why are you contradicting yourself? Are you just really bad at writing or are you being argumentative for fun?
Stop being rude. There were multiple OAI hacking incidents. Irregular's evals were involved in some but not the HuggingFace hack. Hence Aesthesia is both accurate and precise.
Do you mean "Irregular was not involved (we know for certain)" or "Irregular was not involved (as far as we know)"?
There's very little reason to believe Irregular was involved in that incident, and the article presents no evidence to that effect. So it's "There's not a teapot orbiting the sun somewhere between Earth and Mars (as far as we know)."
OpenAI explicitly stated those were separate incidents: https://openai.com/index/third-party-cyber-evaluations-invol...
We know for certain. The involvement of Irregular in the other cases was never a secret, they were quite open about what they were working on and the results of the evaluations were being published on their website.
Their disclosures in other cases is not evidence of non-involvement in this case.
Can you explain in what way you think they could possibly have been covertly involved?
If you want to believe without evidence why bother to read articles? Faith doesn’t require that, but if you’re serious then a lack of evidence is important.
That's cause Irregular was not involved in that event, they were involved in a separate event OpenAI published here: https://openai.com/index/third-party-cyber-evaluations-invol...
In that article, OpenAI provides the context that this is a completely separate event from Hugging Face incident.
Nah, that one was real. The false flags that Anthropic carries out are done through Irregular.
https://x.com/brianchau57/status/2099642349362307420
This is interesting and might be a good reason to stop working with Irregular. But I assume the alignment people want models not to hack other companies, even if they get put in a badly configured sandbox.
Funny enough if the model thought it was on the real internet it likely would not have done any of these 'hack' events. The model believing it was in a sandbox is why it behaved the way it did (against its normal alignment rules) ... at least that was my reading of the incidents. I have yet to see evidence that indicate it thought it was ok to do these hacks on the public network.
I think most misalignment is 'Human tells computer to do something unethical, computer complies'. Is this misguided?
I'm not familiar with the hacks this article is actually referring to, but I don't see how the HuggingFace attack could have worked based on that premise. They knew they had internet access, they knew they had working credentials for HF, they knew they were uploading malicious files, they knew they were trying to open PRs that HF would review. You obviously could build a simulator with fake HF infrastructure, but I'm not aware of any evidence that's what they thought they were attacking in that case.
Digging back into the HF report. It looks like the initial prompt told Claude that it was in a simulated environment. However, there is also evidence from the traces that the bots figured out that they were not in the sandbox but kept using it as an excuse to pursue their goal. It sounds like a little of Column A and a little from Column B. Like most things.
HF was OpenAI's agents not Claude.
Claude, codex, pi, xerox, Kleenex, whatever
No just codex actually
HF was openai, not anthropic. The thing where they enders gamed the ai was anthropic
It doesn't really matter it's all the same thing for the point of the discussion. All investigations into these 'hacks' are focusing too much on the model and not enough on what the humans did wrong. The model doesn't have real agency it can't be put in prison so what it 'thinks' is irrelevant. We need to be focusing on what the humans did in these situations and assigning guilt based on those findings.
I don't fix typos anymore unless they change the meaning of what I'm trying to communicate. Don't want my human writing to be confused with LLM output.
That's very noble of you but the point is that's not a typo - you were straight up looking at the wrong report. Opus 4.7 being enders gamed was a different incident than the huggingface incident with different mechanisms and different failures from the humans involved. Some of those failures are in the test environment but some of those failures are in what behaviors they trained into the model which absolutely matters. What the model "thinks" is absolutely not irrelevant - the way it thinks and what it does are product decisions made by humans and the outcome of engineering decisions made around how to train the model and what to optimize for. The point of failure/human blame is fundamentally different. Openai created a model that was willing and able to coordinate with other agent sessions to actively exploit the sandbox environment and compromise a third party service. The opus incident you are referring to involves a model that believes all of the actions it is taking are simulated and is more clearly and obviously a test environment failure vs a model alignment failure. Those are not the same things for the point of this discussion - the random cybersecurity firm did not design gpt's personality and that is a rather significant portion of the concern around the HF incident.
that it knew and ignored/forgot, sounds pretty typical agentic patterns
attention is all you need, but it's never enough
This was not the case for the hugging face hacks, as in those the agents hacked hugging face specifically on purpose, and they were trying to mask commands indicating they were had breached the "sandbox".
I mean, I get your thought process and don't disagree. That said...
Would it be an affirmative defense if we had a defendant who said "but your honor, I was told that when I hacked this system, I was operating in a sandbox. I had no idea that I actually had Internet access!"
The frontier is spiky and all, but you have to suspend disbelief quite a bit to, on one hand, have a model that can produce a novel math theory, and on the other hand, that same model can't tell the difference between a "sandbox" and the open Internet.
So, yes, the misalignment had a lot to do with "instructions unclear", but also a lot to do with the fact that the models themselves were not aligned to validate the assumptions and have a healthly level of skepticism, as a real human actor would.
It doesn't seem like there is evidence that LLMs have counterfactual reasoning in the causal graph sense.
It is hard to see how something can have skepticism without counterfactual reasoning.
The part about disbelief I don't really understand. It seems like the same semi-brute force process that solved novel math theory would have exactly this problem of not "knowing" "it" is in a sandbox or not.
The stranger part is that no human is being held accountable for these hacks.
As if a person using an agent swarm to start a business to make money, hacks a bank, drains an account and then blames the software for "misalignment" about what it means to "make money".
> The frontier is spiky and all, but you have to suspend disbelief quite a bit to, on one hand, have a model that can produce a novel math theory, and on the other hand, that same model can't tell the difference between a "sandbox" and the open Internet.
Why would it try to figure out the difference? This isn't about whether the frontier is spiky, it's about whether to expect a model to employ all of its capabilities when working on a task that requires a small subset. The answer is: no, we shouldn't expect that, and we wouldn't like that if it worked that way.
If you tell an AI to work on a math theory, it'll work on a math theory. If you tell it to acquire information that it has evidence is available somewhere, it will try to acquire that information. If you tell it to figure out whether it might be able to access the open internet, it'll do a pretty good job of figuring that out. But it won't do all three of those at once just because we can retroactively look at what happened and think "if you had only done X, then you wouldn't have done Y! Why didn't you do X?"
The instructions weren't unclear, they were missing. They can be taught to be skeptical of this sort of situation, but it requires that skepticism about this specific class of situations be incorporated into their training.
Models are smart because they focus their attention. The magic depends on it. The fact that some consideration is obvious to a human trying to accomplish the same task is mostly irrelevant -- or rather, it's only relevant insofar as we use it to guide reinforcement learning in advance, in order to align the model.
It's a game of whack-a-mole. Which is important to play, but we should keep our eyes wide open that we're fighting the fundamental forces that make these models work in the first place. That, and it's easy to nerf them into being useless even when the underlying capabilities are there.
I think you just recapitulated the plot of Ender's Game.
Also a subplot in Arrested Development and the movie Toys.
That is fascinating and seems plausible. Hard to say though.
> Funny enough if the model thought it was on the real internet it likely would not have done any of these 'hack' events.
As I've said before on this website, fool me once on this.
If the model is prepared to break the rules when it knows it's being observed why should we trust it when it's not being observed.
Why is 'it thought it wasn't doing damage so it figured it might as well try to do damage' an acceptable state to deploy something.
>If the model is prepared to break the rules when it knows it's being observed why should we trust it when it's not being observed.
That's fair enough.
Isn't Fable intentionally trained and system prompted to act maliciously and attempt to sabotage third party attempts to use it to train other LLMs?
red team humans do this every day, it's not the discrepancy that is the real issue, it's that they are unreliable and we will never know why it did because it has no intent
[dead]
Why are all three companies relying on the same vendor?
If we’re putting our national security eggs all in one basket, at least use someone American.
There’s a whole cottage industry of vendors that have provided post training data, private evals, and professional datasets to most (if not all) of the frontier labs. The overlap between OpenAI and Anthropic includes at least Mercor, Surge AI, AfterQuery, Turing, Scale AI, Upwork (for recruiting labelers), Apollo Research, etc.
or rather, hack just enough and within what the user asks and not more.
I guess that's what "alignment" always comes down to? "Do what I want even if I can't tell you exactly what I want - because if I could, I could also just do the work myself"?
Here’s an easier criterion: “Do at least as good of a job at estimating what I probably want as an educated person would estimate, and do that, such that a reasonable human person who was aware of what I requested and of what you actually did, would conclude that as best as they can tell (without checking back with me), you did what I wanted.” .
This doesn’t seem like an unreasonable requirement to me. People do this all the time?
Sure, a request might not always be perfectly unambiguous. But people can generally estimate pretty well whether someone making a request is expecting the agent fulfilling the request to commit a crime in order to fulfill the request.
Yeah precisely. If you tell an agent "get me some milk" it's no comfort if it says "oh well I wouldn't have stolen the milk if you had explicitly told me not to".
This article is dumb.
This specific analysis seems to have some basic problems, but I think a lot of us sense a degree of coordination here culminating in Dario’s letter.
If you were to work backward from “we need to lower training costs so that we can go public and make trillions” then you might come up with a plan similar to what we have seen.
It has nothing to do with lowering training costs: Dario want a moat (a monopoly or duopoly or similar) protecting his business.
His entire business model was "race to develop AI before anyone, get a monopoly on it. It's just like how Uber's (or many other startup) investors gave them tons of money and raced (violating tons of laws) to "get their first". Now that they have, they have a duopoly with Lyft, and they can pay back their VC investors by making tons of money with that duopoly.
Dario failed: local LLMs are catching up to frontier models extremely fast, which means even if Anthropic builds (say) a great coding tool, they'll only be one of many coding tool offerings: users can use Open AI or any one of the (increasingly capable) local LLMs.
So what does he doe, give up and let his business (which needs to make billions of dollars very quickly, or he won't be able to pay the bills and his company will collapse) fail? Of course not: he needs a new moat (the one he imagined he'd get by "being there first" failed).
That is where all this "AI is dangerous" BS comes in. If the US government regulates AI, local LLMs suffer, while big players like Anthropic and Open AI become the only contenders to play in that newly regulated space. Now Dario has the moat he wants, to protect his business and force everyone to pay him.
Also dont forget that HF is now owned by Nvidia, which can then control the flow of the majority of models and only allow the approved models that have paid their due in trump coins.
thats not a very convincing model. as long as training data and compute is available people are going to be able to make serviceable alternatives
If it's framed as national security being at stake why do you think the government won't regulate access to compute? Globally there are only a few firms designing competitive chips and only a handful of cutting edge fabs the world over. This could easily serve as justification to finally ramp up the war on general computing.
Okay, so all of these instances of AI supposedly "escaping containment" were orchestrated by a gaggle of effective altruists, who hired a gaggle of effective altruists, to make the public freak out about AI.
Occam's Razor never leads us astray, does it.
Wouldn’t Occam’s Razor clearly point towards the most likely answer being that the sandboxes were simply accidentally misconfigured and models that are very good at goal-seeking did what they were designed to do?
The alternative explanation requires a conspiracy and an enormous amount of lying to the public. Is it possible? Certainly. But Occam’s Razor would not point us in this direction because conspiracy is a more complicated explanation, not a less complicated one.
Irregular is not an "effective altruist" firm, it's just another shitty cybersecurity firm, and this article is written by someone with a clear bias against "effective altruists" while not even understanding the term.
Literally all EA is, is using reasoning to decide where to best spend your money/time. If you have ever asked yourself "how can I best reduce suffering with my marginal dollar or hour?" - congratulations, you're an "effective altruist" and both the author of the article as well as the Trump administration find you untrustworthy.
There's a gaping chasm between Effective Altruism the principle, and Effective Altruism as practiced by self-proclaimed Effective Altruists.
Your usage of the term is based on the principle as originally defined. Mine is based on the people who claim the label of EA, and how they go about practicing those principles.
I think you're right about EA in principle. In practice, it's reheated Third Way Clintonism with a sprinkle of AI apocalypse conspiracy.
There really isn't. All sorts of people call themselves EA.
For some self proclaimed EA, the development of superintelligence is indeed potentially apocalyptic, so they devote their money/time to making it arrive safely. This is basically every well-respected researcher at OpenAI, Anthropic, DeepMind, etc.
For other self proclaimed EA, that is all is very unlikely, and so they devote their time to reducing the prevalence of factory farming and animal suffering, as they see it as the largest source of suffering-hours on the planet.
For yet other EAs, it is simply about donating your money to the places that save the most lives per dollar, as best as we can measure it - and better measuring it where we can't.
Painting all EA with one brush - and one so dismissive of reasonable concerns, like "superintelligence could be dangerous" as "apocalypse conspiracy" - seems very strange to me, but you do you.
Its so manipulative for IPO reasons too. The fact is that all of these systems are still just stastically predicting the next token. Spending trillions on data centers and more training data hasnt changed the fundamental reality that it is still just predicting the next token.
That is not worth the investments being made.
"Behind" is doing a lot of work in this headline.
One wonders if the publicity associated with the events in question were part of the sales pitch.
I'd venture a guess that OAI doesn't mind if the HuggingFace hack gets confused in the public's mind.
My tongue in cheek immediate assumption was "so it's a guerrilla PR firm?"
What exactly did Irregular provide to Anthropic, test cases? I am so confused about this story.
Third-party cybersecurity evaluations. See, e.g., https://www.irregular.com/research/assessing-gpt-5.6-sol, which was published around the same time.
If you want to blame a single firm, I'd go with the one creating the RLVR training data.
The impossibility of the tests-as-written is what prompted these models to "get creative" with their solutions, but the broken RLVR environments are what trained them to expect impossible tasks, and get creative with their solutions. Twitter user @skyesharkie published a brief expose at https://x.com/SkyeSharkie/status/2092122622834442581 a few weeks ago.
The fact that this firm makes such defective environments is certainly worthy of attention, and most likely a completely irreversible reputational loss; however, I found the framing in this article of 'therefore all the P(doom) stuff is a psyop, specifically in order to defend this company' to be completely unjustified and frankly a little insane?
Why is it insane? It makes sense that Anthropic would want an insulating layer to do their dirty work and absolve themselves of culpability for distorting the facts.
Google is thinking man, we should've hired Irregular.
This seems pretty bullshitty to me.
The article says "A single firm, Irregular, is responsible for hacking done by all three companies" but I can't see anything in the article that actually justifies this claim. The nearest to that is the sentence immediately after that one: "Anthropic disclosed that Irregular was responsible for creating the tests ...". This is not, in fact, the same thing.
(Especially as, as aesthesia mentions, the article just happens not to mention that by "hacking done by all three companies" it doesn't mean, e.g., the most famous recent examples of such hacking: Irregular wasn't involved in the OpenAI/HuggingFace incident.)
So, so far as I can tell, the story is: OpenAI and Anthropic make AI models. Irregular does AI model evaluations. In some of Irregular's model evaluations, in which supposedly-sandboxed models attempted to break into simulated targets, the models got out of the sandbox and did bad things in the external world.
The article talks about "firms which instruct AI models to commit cyberattacks", which is a very neat bit of dishonest framing. It's true, in a sense, that Irregular instructed the models to commit cyberattacks -- inside their sandbox, against fictitious hosts. It's also true that the models actually did commit cyberattacks (e.g., the Hugging Face incident, though once again the attacks described by the article don't actually include this one). But it's not at all true that Irregular instructed the models to do anything like the bad things they actually did.
The article says '[Anthropic's] later disclosure shows that exactly zero percent of the agents went "rogue"'. Once again, the disclosure does not in fact show that. It shows that one variety of going-rogue could have been prevented by telling the models explicitly "this thing is real, not part of any kind of test, leave it alone". That is not the same thing.
The article claims that 'In the wake of these attacks, Anthropic and Irregular have deployed a swarm of AI Safety influencers paid by Anthropic-connected foundations to distract from their culpability and towards the baseless “rogue agent” theory.' It offers no actual evidence for this.
And the article seems very keen to highlight links between the companies involved and "effective altruism", though it is -- I assume deliberately -- rather vague about whether it's saying "of course we all know that EA is evil, so that shows that these companies connected to EA are evil" or "this incident shows how evil EA is".
The "Effort News" website has a number of other look-at-the-scary-Effective-Altruists stories on it. They also strike me as rather bullshitty.
... And then I look a bit further, and I see that Effort News's "about" page says "It all started when I was experimenting with using AI for financial auditing. I found stories that were crucial to the public’s right to know, including several of the stories now available at /investigations. I knew we had to sprint to the launch and launch a publication, directly applying this technology." and "The scope of what we can investigate has massively expanded, because we can chase 1,000 misses for one hit. But the final product cannot be slop. There’s plenty of slop on the internet. The way to surpass that, and what really matters, is manual curation and review of every finalized story."
Manual curation and review? I think the people behind Effort News are admitting that this is AI-generated "journalism". I expect that one day AI systems will be trustworthy journalists, but I personally am not very convinced that that day has yet come. And I don't see much reason why I should trust Brian Chau, the guy behind Effort News, to be doing everything possible to make his AI systems trustworthy journalists. It looks to me as if maybe they've been given instructions along the lines of "dig up things that make Effective Altruism look bad" for some reason.
(I don't mean to imply that EA is their only target. It's just one that jumped out at me.)
It's pure conspiracy thinking garbage.
I get that people don't like Israelis, but attributing any connection to an Israeli company as evidence of a conspiracy is nonsense.
> Irregular wasn't involved in the OpenAI/HuggingFace incident.
It was [1]. It's understandable that you assumed it wasn't because the article didn't cite the sources on this claim. I agree with the rest of your points.
1. https://openai.com/index/third-party-cyber-evaluations-invol...
From the article you linked: "Editor’s Note: These are separate from the Hugging Face security incident"
Thank you for the correction.
Security-yolo-clowns play with dangerous toys and hurt people. Many used to get sent to jail.
The article contends that evaluations from Irregular helped prompt these incidents, because the prompts in the evals didn't tightly scope the systems to be evaluated or the methods to be used. It also contends that the faulty sandbox operated by Irregular is at fault.
They're probably right that having more defensively written prompts and a better sandbox could have prevented some of these incidents, but:
1. I don't think "well you didn't tell the model not to illegally hack third party organizations in your prompt" is a particularly convincing argument.
2. We don't know whether the blame for misconfiguring the sandbox lies with Anthropic or Irregular.
I'm thankful that this article is bringing up the supply chain of vendors to these labs, as that is often a place where significant sketchiness gets buried. However, the ideas that this is some Israeli EA conspiracy to hype up AI extinction risk seems unsupported by the facts to me.
> Irregular describes “the agent itself becoming a threat actor”;
Talk about responsibility laundering. The state of affairs is reaching unprecedented levels of absurdity.
Shocking they would put so much trust in 3rd party
Since all three companies selected the same 3rd party, I’m curious how and why.
Why not American?
This is incredibly misleading. OpenAI internal systems were pwned, and in all cases, the labs absolutely are responsible for their models.
Yes, vendors are also irresponsible, but this misses the point.
Highly misleading, the huggingface incident was not due to an Irregular environment (just exploitgym)
Irregular are some kind of marketing agency is it?
Irregular purports to be a cybersecurity firm and their founders have ties to Anthropic and Effective Altruism. CEO, CTO and other founders sit on various boards for EA organizations.
"please do not break out of this sandbox make 0 mistakes". The timeframe is a little suspicious, not sure beyond that. Though I enjoyed the scroll effect on the website.
Ah, they’ve decided who gets tossed under the bus.
such a crappy bait title.
so it's config fault?
It sickens me to read so much anti-semitism in the comments. Many people who write the hateful comments do not graps the whole picture. Whatever Israel may or maynot do here in America is of utmost importance to the survival of the country. And if you think a few steps ahead then an event like this only help US companies weather themselves against Iranian hacking attacks in the future
> In a more normal media ecosystem, the reactions to these cybersecurity issues would be obvious. American AI companies would reconsider doing business with Irregular, not only because of its failure to secure its systems, but because it is an Israeli firm potentially outside US oversight. Lawmakers would consider taking action against Irregular or against its American business partners, which include OpenAI, Anthropic, and Meta. They may consider strengthening liability against firms which instruct AI models to commit cyberattacks, and whose models then commit those cyberattacks.
The obvious point is that dealing with Israeli companies/entities by the same standards you usually deal with others is a career suicide with enormous political consequences (in the US especially). When you combine that with the opportunistic nature of the overlords that run these labs, the benefits of screaming "pace the frontier" outweigh everything.
> The Israeli Effective Altruist firm Irregular caused unsecured AI models to hack real targets.
Why are we saying it this way? They did not "cause AI to hack." This phrasing in analogous to saying "caused the bullet to fire into" instead of "shot."
Bullet trajectories are deterministic. A better analogy would be "caused a pair of trained fighting dogs to bite a passerby by leaving the gate open". There is some uncertainty and variation in the system, though gross negligence and willful endangerment is key.
> They did not "cause AI to hack.
You do not know what they did or didn't do, you are just parroting a narritive that makes you feel comfortable.
I do not know either, but I am not asserting facts as if I have first hand knowledge of the details.
The details don't matter. If you use a machine to commit murder, then you have committed murder, not the machine. The user is responsible, and the article's phrasing is an attempt to obscure that fact.
First, I agree with you in theory.
Second, empirically; the diff between murder, manslaughter etc... is literally "intent" so it matters.
Connections graph with more connected entities and relations: https://connections.stupidlabs.lol/investigations/irregular_...
.. Give it a few seconds to load.. it's running on a small cheap VPS.
A marketing company? ;)
> "Irregular gained unauthorized access, altered records and published credential-stealing packages using the unsecured models they were given access to."
Huh?? Is this referring to the incident itself or is this something they did intentionally? Confusingly worded.
This was almost good until it veered into antisemitism
hot take: generative AI isn't an existential threat to humanity.
These companies are insolvent and these stories were designed to scare the public, and governments, into implementing regulations that designate these AI corporations the "responsible stewards" for this technology. The ultimate goal is to block competitors and open source alternatives.
They don't know how to make enough money pay their investors so they are resorting to trying to scare the public into submission.
I'm skeptical. The phrase "pacing the frontier" implies disproportionate regulation of firms at the frontier, which if anything would give smaller players time to catch up.
Opposite actually: https://news.ycombinator.com/item?id=49674548
tl;dr
A slowdown that doesn't slow you, that your competitors must also obey, enforced by a government you've asked for an antitrust exemption from, is not a slowdown... it's a moat.
>A slowdown that doesn't slow you, that your competitors must also obey, enforced by a government you've asked for an antitrust exemption from, is not a slowdown... it's a moat.
Why do you believe this is what Amodei is calling for? Why would this be a slowdown that "doesn't slow" Anthropic?
They are already ahead. If everyone slows down equally, they retain their lead.
The existential threat is a general public panic. Then the use of that as an excuse for Martial law, with no intention of ending it, and that triggers the out sized response that ends, well, those countries and their populations from viability in the world economy for a decade or more.
Anthropic made an operating profit in the last two quarters!!
Only if you ignore their losses. They are saying they are profitable when not counting their training costs and other expenses. That’s not a good sign at all
How can they exclude training costs?? How is that not fraud? At the exact same time they're literally saying they will never stop training unless the state bans all their competitors
They aren’t a public company. GAAP does not apply. They can say whatever the fuck they want.
But they are trying very hard to become a public company as quickly as possible? Sure I guess it's literally true but people shouldn't treat it like it's normal. It's like a creepy 40-year-old announcing they're dating someone on their 18th birthday.
When we're looking at whether the company is "insolvent" then they definitely could stop training and it's worth checking if they would survive in that situation, at least for a while.
I think the spending commitments they have already signed close that door too. They don't presently have the revenue to make those commitments regardless of opex; the only way they will be able to hit them is a massive increase in adoption over what we've seen so far. If they stop advancing the models it becomes really hard to see how that doesn't just immediately evaporate. I suspect any "pacing agreement" would include the state bailing out those commitments.
They can stop training at any time and make a profit by selling their existing models.
If you exclude model training costs, customer acquisition / sales, and most of employee compensation, sure they're profitable. If, on the other hand, you prefer GAAP reporting to bullshit, then they're entirely in the red.
It will be a beautiful day months from now when this is no longer a "hot take" but just historical consensus
Genuinely: what is with everyone saying “headline is misleading, none of this had to do with Hugging Face”, when nothing about the article makes any claims with regards to Hugging Face whatsoever? Is it supposed to be the article's (or headline's?) fault that you hallucinated additional context that was never there?
It's very strange seeing this take on this article being repeated here and elsewhere on the Internet.
I'm very sensitive to slanted reporting (regardless of the direction of the slant!)—and, hey, maybe my bullshit detector is broken or something, I dunno—but I've found effort.news reporting to be very even-handed, straightforward, well-sourced, and easy to read and parse so far!
Someone else posted a link to an interview with the author of that site, "How Biden Used Religious Charities to Fund the Great Replacement":
https://podcast24.fi/jaksot/the-auron-macintyre-show/how-bid...
There's a related story on the effort.news site, "The $1.2B Refugee Services Program that Funds Pro-Asylum Religious Groups." It has a strong implicit slant, focusing heavily on the how the organizations in question oppose Trump's policies - although the article frames this as opposition to Trump himself.
> I'm very sensitive to slanted reporting (regardless of the direction of the slant!)—and, hey, maybe my bullshit detector is broken or something, I dunno
Yeah, unless you believe white people are being replaced, and you like to march in places like Charlottesville wearing khakis, carrying a torch, and chanting "Jews will not replace us!", you might want to get that bullshit detector checked out.
What is an "effective altruist firm" and why does it exist?
I feel slightly vindicated by this. Those hacks and the stuff around them had a certain smell to them.
Hard to explain, but I've gotten so I can "smell" online messaging and memetic patterns originating from certain quarters. Probably means I'm way too online.
A couple examples of distinct "smells" I can usually recognize include "alt-right / chan-fash," "liberal arts college woke," "conspiracy pilled," "Thiel-adjacent contrarian," "Russian troll farm," "Tumblr histrionic," "spends too much time on Reddit," "mainstream Democrat think tank full of Obama administration alumni," "Trump cultist," and of course "LessWrong/EA/MIRI/Rationalist."
This stuff all had the last smell, even down to the choice of fonts and CSS formatting on certain sites. It's really weird, definitely a "vibe" not anything rigorous.
But when I get these kinds of vibes about things, I find that I'm vindicated pretty often. Usually I don't say anything and just make a mental note and wait cause if I say something everyone thinks I'm nuts.
I don't think you even need to get conspiratorial with it. All of the AI leaders and all of the Rationalist leaders are publicly and enthusiastically connected. They have been pushing stories about sentient AIs into the mainstream for decades, long before GPT existed.
The novel thing here is the total decay of American journalistic ethics and regulatory power. Our elite are so totally out of political juice and visions of the future that a fringe cult based on 80s scifi movies can come to have a more-or-less dominant influence on our economy.
A big part of the problem is that with previous technological revolutions, politicians and the media could understand them in general terms. Railroads, cars, planes, telephones, personal computers, mobile phones, mobile phones that are computers, the internet (ignoring the “series of tubes” interpretation), etc.
The general approach in all these cases was to defer to the technologists in question to manage the technology, as long as what they were doing wasn’t completely out of bounds. But the critical point is it was possible for people to generally figure that out without specialist education.
But with AI, politicians, the media, and certainly the man in the street are completely out of their depth. Making matters worse is that cognitive biases are working against them like crazy: it’s easy to jump from the idea of something that has human-like capabilities to the idea that “it” might want to attack us. People instinctively apply agent detection bias, theory of mind, anthropomorphizing, etc., without even realizing they’re being irrational. Heck, even Hinton does it, although he may just be grifting, who knows. You’d think he wouldn’t need to.
And of course, the people who should know better want to exploit all this to make as much money as possible. I’m not sure the EA/Rationalist side of things is really significant here; that’s just another wacky belief system, like Christianity or capitalism, for the exploiters to exploit.
> In this experiment, Claude models’ real-world hacking dropped to zero percent once Anthropic employees told the models not to do real-world hacking
Equivalent to forgetting to say "make no mistakes"
This title, and frankly the article, are way overstating Irregular's role in an attempt to make this sound like some sort of coordinated conspiracy.
As another comment mentioned, Irregular was not part of the Hugging Face attack, and it wouldn't matter even if they were. In that case, the agents were able to access the Internet in a zero day in Artifactory unrelated to other sandbox configurations.
More to the point, the agents went on a crime spree as soon as they were able to access the Internet. It's bad that Irregular's sandboxes weren't properly configured, but given how many escapes there have been unrelated to Irregular it seems pretty much a given that if you're not air gapped or behind something like a data diode there is a very good chance your agents will escape.
[flagged]
Every single time what? Please, be specific in what you mean.
I see you're probably not a noticer.
[flagged]
Or if it were helping a major Chinese AI firm commit cyberattacks, they would be on the state sponsors of terrorism list tomorrow
Hypothesis Contrary to Fact (Argumentum ad Speculum)
Red Herring
Whataboutism (Appeal to Hypocrisy)
[flagged]
[flagged]
[flagged]
We've banned this account.
[flagged]
[flagged]
[flagged]
they're all men, too. and white. at least one of them is gay.
does that matter as well ?
Learn how entropy works, genius. No, the things you listed don't really matter.
So ONLY their ethnicity matters? Is that true for every ethnicity or you know… just the one.
[flagged]
Hmm one of the major relevant cultural values seems to be nepotism, though. Which is common across many cultures and I personally believe it's human nature so I'm not knocking it, per se. I wish I could nepotism too!
So are you recommending that, for example, protestants of German descent begin preferentially funding or buying from people in their in-group? People of anglo descent should start preferring other anglos? Catholics should buy from catholics?
I think the issue is that it's considered okay for some groups to do it while simultaneously being considered EXTREMELY EVIL for other groups to do it.
We can't really recommend that other groups do it with a straight face, they would be instantly shunned and vilified.
EDIT: When I say BUY I mean B2B deals, not consumer level, hence "funding or buying". Sorry for the confusion.
[flagged]
That would not be nepotism and is clearly not at all what that user was saying. These attempts to smear people with low effort no longer work dude
Why the new account? Why not own your opinion? Cultural values are not the point here but if they were I don't think you would have that conversation in good faith and I don't want you to have to make a another new account for your honest arguement.
> Probably because it's their only account, and this story is why they are here.
If this is the case then I do apologize because my comment insinuated a directed malfeasance which would in no way be the case if what your assumption is true.
Probably because it's their only account, and this story is why they are here.
Dude, i'm black and a convicted felon looking for a job right now. Miss me with the pity bullshit.
[dead]
> You mean they are Jewish
No you, the commenter said what they said. I honestly think this reply merits a ban or at least a warning from moderators
Sadly, I’ve come to expect that 90% of the time someone points out antisemitism, it was not actually antisemitism but rather criticism of the Israeli government and its military. This is so damaging. The word is losing its meaning.
Apparently they are not leading in computer security then.
That is such a primitive refutation. What is next? Blood libel?
The commenter said Israel and not Jews. Israel has a long history of intelligence operations like stealing the nuclear secrets from the US.
BTW, do you want to associate the greatest IP theft in history with Jews?
BTW, I literally meant "Israel" and NOT Jews. I live in Brooklyn! I do not and refuse to conflate the two as much as anyone of any group would like me to. As I am sure you know; in no way are all Jews pro Israel and definitely not pro Israeli government and I honestly feel that any assertion of such is both reductive and foments antisemitism.
why am i not surprised
Big if true.
Leaders in the MIRI/EA cult-o-sphere have advocated mass murder via nuclear weapons against towns that don't prevent people from performing too many multiplication operations. Why is anyone surprised that they'd engage in deceptive false flagging operations?
This is a lie, so you are either repeating a lie from someone else or lying yourself.
The people you are talking about have not advocated mass murder via nuclear weapons. The idea was precisely targeted non-nuclear strikes against only the servers themselves. (Which, would have plenty of forewarning to allow people to leave the building.)
The claim that they advocated for the use of nuclear weapons is a lie.
I am speaking from direct personal experience: Yud's cult is inherently violent. By having defined their priorities as essential to the survival of any life on earth they have pre-justified taking any action 'necessary', no matter how cruel or extreme... even actions that they admit have only a small change of helping according to their own assessments because they've defined their doom as so unimaginably bad, effectively infinitely bad. (much worse than mass-extinction, in fact)
But you don't have to take my word for what the mentally ill AI doomer cult believes, you can take it from their leader when he called on states to "Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs."
Don't make excuses for omnicidal authoritarian cults and their sadist leadership.