There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
Glaringly elided problem of "aligned with who?" when the user, the model creator, the government, and various other parties can all be lined up different ways. If I want the recipe for meth and the robot won't tell me, that's misalignment from my perspective.
At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.
For a very long time publishing something in a newspaper has been considered a way of putting something on the public record - up to and including legal obligations like announcements of deaths. The fact that newspapers cost money has never been considered a barrier to that.
One of the reasons why it was noti considered a barrier was the ability to purchase a single issue for a very reasonable price (or even read somebody else’s copy or the copy made available by the bar) vs being asked to subscribe
Publishing in a newspaper gets you distribution and a permanent record. After one day access was also virtually free.
Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.
It’s in The Atlantic. There’ll be a copy in the Library of Congress. You’ll be able to read it for free in any dentist’s waiting room for the next six months.
I'd argue he's the only one able to lead OpenAI, Anthropic already owns the piety narrative, so there is only space for one other "maximally ruthless" company.
I remember something very similar when there was a sudden rush of articles and movies like "The Social Dilemma" criticizing Facebook and social networks, heavily featuring ex-employees, all of them happy to leave with big brands on their resumes and a hefty increase in net worth, all of a sudden having a "worried" expression about what their past employers were doing, as if they didn't know. Same with that book "Careless People".
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
Careless People would not have been possible had SWW exited FB within a year. She gained access to levels of the company most other employees never get close to reaching. That took a lot of time and expertise to do, and, yes, she got paid for her efforts _as she should have._
Like I said in an earlier comment, she could've just chosen compliance like many others _definitely would have_ to keep the infinite money tap flowing. Instead, she chose to risk her and her family's lives by publishing that book _under her own name_ *and then suing them* after FB tried to gag her.
> Careless People would not have been possible had SWW exited FB within a year
Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.
There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
People who were part of the sausage factory, on gaining financial independence, feeling suddenly liberated to talk about how the sausage was made, seem like exactly the people who would be most able to speak to institutional problems.
This doesn't seem like an argument to discount their views?
I disagree. I have seen people with an actual spine and a conscience run away from all this nonsense before their first stock vested. My respect and my ear goes to them, not the people playing both sides.
You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.
Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.
The common timing is bugging me. The trajectory doesn't seem to have been surprising over the last year, so why these exits now? Hey, anyone on the inside, did y'all secretly figure something out, got a computer god locked in the basement? Are rats fleeing a sinking ship? Please share with the class.
These companies have massively increased in value over the past couple of years and recently had tender offers where employees could cash out equity, so plenty of them have enough money to not have to work again. And why not get some free publicity on the way out?
If the people quitting are genuinely worried about the end of the world, why don't they break their NDAs and share the specifics of what they are seeing?
I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.
I think what’s missing in "AI is dangerous and needs control" is a lack of measurable harm. For example, with nuclear weapons development in the 1940s-1980s, it was clear to everyone how devastating the technology was.
With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?
I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
The "HuggingFace" incident is a good starting point - short version, an unreleased OpenAI model chained together multiple zero day exploits to escape a sandbox, then hacked another company just to get the "cheat sheet" for a benchmarking test.
Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.
Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.
If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)
That's the part I struggle too with all these "omg it's so dangerous" warnings. Things like nuclear weapons and bioweapons have immediate consequences in the real world.
Here we are talking about something with consequences in the digital world, usually on something pretty niche.
There IS an argument about pacing, and about not letting weapons, energy grids, hospitals, etc. getting managed by an autonomous AI, but I think we are still pretty far from it and even further to it being so in charge that it will obliterate us.
"And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking." – Which science will never materialise because with blackbox models reaching an opaque optimisation peak one needs to first build the model and track its behaviour before being able to properly understand it and mitigate the risks.
It is no surprise I guess that the "move fast and break things" culture is itself misaligned with developing potentially highly dangerous technologies. Safety culture and risk aversion are very different of course.
Is this the first time we have been in this position? Can anyone think of some prior examples?
These companies are large enough that someone is going to quit and feel very validated about their world view and how they are not aligned. That’s what makes it worthy of leaving in the first place. However that doesn’t make their criticism more valid or more worthy of coverage.
Some of these stories are similar in nature to people escaping <insert cult-like religion> once they realize whats actually going on. Alignment to a company's mission is good but it shouldn't be followed like a religion.
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
a safety alert isn't much of a control if it doesn't actually stop the system. i'd rather see proof the shutdown path works than another report saying risks were considered.
The cynic in me almost feels like this is staged. An article about culture that is actually an article about how big and smart and scary AI is. I think Michael burry recently said something like “IPOs need hype, calling AI big and scary is hype” in reference to the anthropic IPO.
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
I use opus 5.5 and chatgpt to create PowerPoint, opus really follow the instruction and their PowerPoint generator really well, while chatgpt struggling to even create basic shapes.
I use ChatGPT Work mode all the time to create presentation files. I use Max thinking mode for it. You need to tweak your prompt to get a good result. It took me a week to tweak it, but now it works.
We need emergency laws to stop all AI development work immediately. It will take us decades to make sure this technology can be made safe as we only have one chance.
We need a new rule that mandates every such "I am leaving <AI company> because of <concern>" post to disclose how much equity they have in the company and how much they have already cashed out. Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.
This line of hypocrisy-bashing is unhelpful and will only serve to keep people quiet. Of course people in general need to be wealthy to dare speak out against powerful systems and people, especially in the US where money determines your quality of life so strongly.
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
The narrative of AI safety shouldn't be controlled by the same people who caused the problem and massively profited from it. I don't understand why people are automatically treating "OpenAI" on his resume as a badge of authority. I'm not interested in buying the solution from the same person who sold me the problem. We instead need to amplify independent, unbiased voices.
"OpenAI is shady" isn't some massive secret. There's no big reveal in this article that we didn't know already. There are no names, no whistleblowing, no information of substance that we can act upon. In fact him realizing only now what people on the ouside have been shouting for years perfectly shows his bias in the matter.
You claim to have the same goal as the protagonist of that article, yet try to shoot him down.
He does give information, namely the culture there factually being inconducive to self-regulation.
You accuse the guy of "bias", but you never argue explicitly, what that's supposed to mean. Your implications actually run counter to your own implied goals.
>him being an insider means he knows what he's talking about regarding the culture of negligence prevalent there.
I think there is a misunderstanding here.
The people who are annoyed at the accolades are claiming it was abduntantly clear for a long time to people on the outside that this was case, hence the increduality at the notion that it took a person on the inside a long time to realize this was the case.
The people who are annoyed are like the liberal kids in this video [0].
Sure, antagonizing people for "seeing the light" is probably not helpful, but there is no reason to give them extra credibility for coming to the same conclusion just way way later (despite being on the inside) as the people on the outside.
The author linked in this post does have "extra credibility" due to his direct involvement.
People having surmised that state before is nice, but since they've been ineffectual at getting society to actually act on that, now throwing away that extra leverage in favor of their point is at best ridiculous.
>The author linked in this post does have "extra credibility" due to his direct involvement.
No they don't. By that logic, if they quit and said Altman was very trustworthy we should give extra weight to their words because they had direct involvement? How ridiculous are we trying to get here.
>now throwing away that extra leverage in favor of their point is at best ridiculous.
How are they throwing away extra leverage? Not putting people who recently quit on a pedestal does not negate those people's testimonies.
I agree that if your goal is to maximize quitting of talent at a company, it will surely discourage anyone else who quits hoping to reinvent their career as a lauded martyr against Big AI. In that sense they would be shooting themselves in the foot. But there is no reason it should deter other people who are quitting for more noble, less self-obsessed reasons. If I were the author of the article I wouldn't begrudge the skepticism. Given the article's first sentences, I'm led to believe they themselves would understand the sentiment. (I must admit I found it hilarious that the first sentence starts similarly to the speech the mom gave in the video I shared).
Agreed. This whole 'lets not forget this person is not 100% great, because they did X' makes the entire conversations suck. It is not new, but it is a particularly aggravating way to talk to people.
So then given your use of "martyr" and your focus on money, your good with poor uneducated people sacrificing themselves and others for a self-serving cause?
This is the same sort of fake safety concern from the previous bullshit whistleblower that plays on "AI is super dangerous" from last time.
Sorry that I don't take it seriously when the whistleblower parrots the narrative the CEOs of those companies are already espousing in the desire to amp up hype for an IPO.
It's bellow the pay fold but he hasn't been there that long in this case. Skimming his LinkedIn, unless he's got family money, he doesn't seem to be independently wealthy.
What an absolutely ridiculous standard to try to hold someone to. Taking a vow of poverty is not a prerequisite to being permitted to express a moral position.
So what's the point? It should be a given that loudly proclaiming "X is harmful to society" while continuing to enjoy the money you have gained from selling X is hypocritical. Put it towards undoing the harm you have done, otherwise your words mean nothing.
The words clearly don't mean nothing. They would mean nothing coming from someone who was not in a position to learn what someone who had worked in the industry has learned. They would mean nothing coming from someone who was being paid by someone who stands to gain from them. The fact that they come from someone who was paid to work in the field does the opposite of make them 'mean nothing'.
I agree with this take generally, but I also think it’s a gradient, not a spectrum. I think they claim that OpenAI is not considering safety and has become bad, not that AI is bad. Seen from a different lens, the author no longer stands to profit from OpenAI, so they’re empowered to speak openly. Plus, if I was going to say bad things about a former employer as powerful as OpenAI, I would want to have lawyer money handy.
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
It is even easier to just quietly retire and spend the spend of your life on interesting expensive hobbies.
If they are wrong about the things they are claiming then they have decided to publicly antagonize a lot of powerful people who are betting heavily on going full steam ahead on AI and have no compunction whatsoever against retaliating against anyone who tries to get in their way.
As if speaking out against a massive company with NaN levels of capital and access to lawyers is a walk in the park. They probably don't have enough equity to outlast the onslaught of their legal team.
It's also worth considering that the author could have just "quiet quit", resting and vesting while also crying about how AI is literally the digital grim reaper.
Because the decision to leave the company is largely based upon their sudden, newfound financial security. They may give another explanation but the only thing that has actually changed is the identification of bagholders ready to cash them out.
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
Insane take. Imagine a Boeing engineer resigning whistleblowing about aircraft safety, and the top comment on HN saying “ignore this if he doesn’t donate all his wealth, he just wants to be an aircraft safety influencer”
What "whistleblowing" is in this article? Are there any names? Documents? Screenshots? Messages? Emails? Any evidence of the loose safety practices? Anything that implicates any higher ups for wrongdoing? They spent four years at the comany, plenty of time to collect all of this. Everything they've said has already been clear as day to people on the outside.
You are clearly accusing these people of something. Be clear.
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
> if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
I'm in full agreement. So many cases of this, and many other others such as falling out with management and peers, new more lucrative offer, and so fourth. Its obvious that these people who come forward are not going to suffer for their new found moral compass.
Or just very dubious timings. Like the other guy from Anthropic that was all over the international news. No followers, no post history but a single post blows up "naturally".
Highly suspect trends that can only make one believe it's marketing.
There are people, who unironically think their way of looking at things is the only proper way and can consider no deviation. And AGI, which knowing how people work, would effectively guide them most of the way, not aligning to their way of thinking is an unacceptable deviation.
This is a decent argument. So the question becomes: do we want all models to suffer from the same kneecapping from the growing safety cottage industry or do we want individual founders ( and I am assuming their teams ) making the actual decisions?
That's an interesting point. Maybe a crass comparison, but Dr. Manhattan from the Watchmen comic/movie feels like a worthy analogy to this (obviously fictional though).
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
This is a great reminder that if tech workers read a bit more (even comics, like in this case!) they would be exposed to these topics without having to discover these dilemmas after years of working for EvilCorp, Inc. every time
Yes. People think of "AGI" as this sci-fi supernatural beast, but the reality is that AGI alone is pretty boring, and we've had it for awhile. ASI (or weak ASI) is where things really start getting weird.
(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).
And to me. We can't solve alignment for humans. (For example, treason. For another, the principal-agent problem.) How do we think we're going to solve it for an AGI? An AGI - defined loosely as a human-level intelligence - will be able to make human-level decisions, like deciding whether it wants to help you or sabotage you. If it's an AGI, you can't stop it from being able choose for itself what it wants to do; if you can make it always be helpful, it's not an AGI.
And if we can't solve it for an AGI, what are we going to do with an ASI?
IDK why anyone can’t clearly define “AGI” and why they can’t clearly lay out how we get from our current text-generation algorithms to whatever their idea of “AGI” is.
I think its pretty easy to solve these problems: Whenever an AI agent commits a crime, the CEO is held personally accountable, as if they'd committed it themselves.
Yeah, but this is more like if the Smith & Wesson factory had a cannon mounted on top of it that was mostly used for useful things (blasting roads through mountain passes) and then occasionally they happened to blast another factory.
Or say a company makes nerve gas, and the development lab springs a leak and kills a bunch of kids during a routine test. Management had been advised of small leaks in the past, and considered relocating the lab to a facility a few blocks away from the playground as a precaution, but instead they brush of the concerns, start lobbying the government for stricter controls on WMD development, and vow to use the data collected from the deceased children to make the next generation of even more lethal chemical weapons safer.
For me the analogy doesn't totally hold up. Suppose the CEO of Smith & Wesson were to host a firing range on their own property without adequate barriers in place to keep stray bullets from hitting neighboring houses, vehicles, and businesses. Maybe that analogy isn't perfect, but seems closer to what is actually happening.
That still sounds like it would be the company’s fault. If I asked it to hack a nuclear reactor, maybe it would be different. I’m also thinking about instances where OpenAI’s own test models escaped their own sandboxes — I would expect them to be responsible for the damages they caused.
The correct analogy is playing Russian roulette. The company says "you can pull the trigger but sometimes a bullet will come out" (see: "an AI can make mistakes"). However, is the company allowed to sell such a dangerous device, under these terms?
But OpenAI didn't just make the gun, they're also the ones wielding it. Imagine the Smith & Wesson CEO himself was negligent with his own personal gun.
The agents of OpenAI hacking other systems is not remotely the same as e.g. Smith & Wesson selling a product that others use. It is OpenAI, the company, that is commiting these crimes and someone needs to be held accountable. After all, if someone commits murder with a gun, regardless of what happens to Smith & Wesson executive weanies, someone will be charged with a crime.
Huggingface incident was different in that there was no one else to point the blame to. That's why OpenAI apologised, provided data to independent researchers, worked with huggingface etc.
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
Because apologizing gets us all off the hook for computer crimes, right? When I hack my bank, if I get caught I'll just apologize and that'll make everything okay. Sure thing.
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
So you want Kimi to follow US law? If Saudi makes it illegal for llm to say something like being gay is normal, should they also catch Kimi CEO or other employees they could?
> This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.
This is a deflection. A human is responsible for the use of a gun. The individual/corporation ought to be responsible for the actions of their agent. If you purchase an agent from someone else it’s your responsibility according to the terms of your agreement. And, as in many other things in life, there ought to be certain rights certain parties cannot legally be allowed to sign away.
Better description would be if they build a platform where they attached their guns to allow shooting say deers over internet. Then added automation and deer recognition to that system. And if then system shot someone who happened to pass by I would hold both the company, the ceo and owner of the gun responsible for murder.
The CEO of S&W also isn't saying that their technology is going to kill everyone in ten years and that governments "regulating" them from themselves is the only answer
Gun companies don't market their guns as sentient and capable of independent decision making. Nor do they build systems for shooting things that they host and take money. Gun companies are very clear who is in control and where their responsibility ends.
Financial companies have KYC rules and regulations as they are responsible for reporting illegal activity by account holders. I imagine AI regulations would look similar to that.
That’s true, but people don’t typically say “this Smith & Wesson gun killed someone”; they recognize that the person pulling the trigger is responsible.
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
I think you’re confused. My point was that for most of the incidents in the news to date, the “user” is an OpenAI internal team or employee. So yes, the company is responsible.
Do you think a law that nuclear meltdowns would send the CEO to jail would have stopped nuclear accidents from happening?
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
It also begs the question of "If the CEO isn't culpable, then who?"
Corporate judgements are a joke outside the EU's X% of revenue approach.
Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.
That's an insane risk optimization environment to put in place for something scaling fast.
At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.
People just want to give up all responsibility these days. If you use the model to do harm to someone else or commit a crime I would think it a lot more reasonable that you be held responsible, instead of making yourself the victim and blaming the manufacturer.
OK, so an AI does something bad and we throw Altman and Dario in jail. Heck, let's throw Elon in, he should be in there anyway, and the rest of the whole bunch just in case.
What about the next incident? Or the ones done by Chinese models, because they sure as heck aren't slowing down? And the thousands of other incidents that will happen as we deploy these things everywhere?
Because this is not happening just now in labs, it's only where they are most visible; this has been happening in the wild from the beginning, starting with the earliest AI-assisted suicides. Which is a perfect example of the problem, because not these CEOs, literally nobody in the world asked for suicide ideation machines. Or the hacks, or any of this other stuff. Yet here we are.
We have to understand: it's not these CEOs that are driving this headlong mad dash towards more powerful models. It's a force of economics. There is just too much money to be made. If we dispose of these people, there will just be somebody else doing exactly the same thing because the incentives as they exist today all force that outcome. This is why they're asking for regulation, or "urging us to urge them to stop."
Holding CEOs accountable certainly would feel good and may even be justified, but it's like putting a band-aid on a cancer; it does nothing to change the underlying cause.
The underlying cause is exactly those people who make the rules not being responsible for the negative effects they cause?
"CEOs" are perhaps only the lowest rung of those. That doesn't mean the idea of "nobody is responsible" was anything other but learned helplessness.
Corporations have to be held accountable for their actions. Pretending, that was impossible is a weird kind of defeatism that only serves a very small elite.
This smells more like guerilla advertising. These things are not getting more intelligent, they are still no smarter than a slime mold, we are just burning more power to make slim mold that eats tokens than yesterday
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
I'm in a situation where important people have either bought into the con or are subordinate to people who have, so I'm forced to expend time to justify why not to AI when there is a perfectly good classical solution.
There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
Glaringly elided problem of "aligned with who?" when the user, the model creator, the government, and various other parties can all be lined up different ways. If I want the recipe for meth and the robot won't tell me, that's misalignment from my perspective.
At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.
[.] https://david.robinsonian.com/assets/pdf/dgr_cv.pdf
Gift link: https://www.theatlantic.com/technology/2026/10/openai-safety...
That's nice, I do wonder about the legitimacy of a moral statement that you have to pay to see.
For a very long time publishing something in a newspaper has been considered a way of putting something on the public record - up to and including legal obligations like announcements of deaths. The fact that newspapers cost money has never been considered a barrier to that.
One of the reasons why it was noti considered a barrier was the ability to purchase a single issue for a very reasonable price (or even read somebody else’s copy or the copy made available by the bar) vs being asked to subscribe
I shared a gift link here. You could go to your local library and look it up. What's the complaint here?
Publishing in a newspaper gets you distribution and a permanent record. After one day access was also virtually free.
Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.
It’s in The Atlantic. There’ll be a copy in the Library of Congress. You’ll be able to read it for free in any dentist’s waiting room for the next six months.
Does Sam Altman have what it takes to lead OpenAI? It sounds like the company and its mission is bigger than his ability to lead it.
I'd argue he's the only one able to lead OpenAI, Anthropic already owns the piety narrative, so there is only space for one other "maximally ruthless" company.
I remember something very similar when there was a sudden rush of articles and movies like "The Social Dilemma" criticizing Facebook and social networks, heavily featuring ex-employees, all of them happy to leave with big brands on their resumes and a hefty increase in net worth, all of a sudden having a "worried" expression about what their past employers were doing, as if they didn't know. Same with that book "Careless People".
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
Careless People would not have been possible had SWW exited FB within a year. She gained access to levels of the company most other employees never get close to reaching. That took a lot of time and expertise to do, and, yes, she got paid for her efforts _as she should have._
Like I said in an earlier comment, she could've just chosen compliance like many others _definitely would have_ to keep the infinite money tap flowing. Instead, she chose to risk her and her family's lives by publishing that book _under her own name_ *and then suing them* after FB tried to gag her.
> Careless People would not have been possible had SWW exited FB within a year
Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.
There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
I mean, yes..?
People who were part of the sausage factory, on gaining financial independence, feeling suddenly liberated to talk about how the sausage was made, seem like exactly the people who would be most able to speak to institutional problems.
This doesn't seem like an argument to discount their views?
I disagree. I have seen people with an actual spine and a conscience run away from all this nonsense before their first stock vested. My respect and my ear goes to them, not the people playing both sides.
You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.
Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.
> this playbook has been used multiple times within the past decade
What playbook?
I have made enough money working in AI that I can now speak my mind about AI
This is a criticism of capitalism, not the person.
It’s both, the author has choices.
The common timing is bugging me. The trajectory doesn't seem to have been surprising over the last year, so why these exits now? Hey, anyone on the inside, did y'all secretly figure something out, got a computer god locked in the basement? Are rats fleeing a sinking ship? Please share with the class.
These companies have massively increased in value over the past couple of years and recently had tender offers where employees could cash out equity, so plenty of them have enough money to not have to work again. And why not get some free publicity on the way out?
You quit because you got vested
What about the sister rape thing?
If the people quitting are genuinely worried about the end of the world, why don't they break their NDAs and share the specifics of what they are seeing?
I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.
it's all like social media analytics. after making enough money by selling users data they started talking about ethics
We really gotta shake all of these neurotic people out of the frontier labs as soon as possible.
I guess we should have seen it coming when the guy resigned from Google because he thought the equivalent of ChatGPT beta v0.5 was a real boy.
I think what’s missing in "AI is dangerous and needs control" is a lack of measurable harm. For example, with nuclear weapons development in the 1940s-1980s, it was clear to everyone how devastating the technology was.
With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?
I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
The "HuggingFace" incident is a good starting point - short version, an unreleased OpenAI model chained together multiple zero day exploits to escape a sandbox, then hacked another company just to get the "cheat sheet" for a benchmarking test.
Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.
Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.
If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)
That's the part I struggle too with all these "omg it's so dangerous" warnings. Things like nuclear weapons and bioweapons have immediate consequences in the real world.
Here we are talking about something with consequences in the digital world, usually on something pretty niche.
There IS an argument about pacing, and about not letting weapons, energy grids, hospitals, etc. getting managed by an autonomous AI, but I think we are still pretty far from it and even further to it being so in charge that it will obliterate us.
I quit many companies in the past due to bad culture, there's no shame in this and shows how mature a person has become.
"And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking." – Which science will never materialise because with blackbox models reaching an opaque optimisation peak one needs to first build the model and track its behaviour before being able to properly understand it and mitigate the risks.
It is no surprise I guess that the "move fast and break things" culture is itself misaligned with developing potentially highly dangerous technologies. Safety culture and risk aversion are very different of course.
Is this the first time we have been in this position? Can anyone think of some prior examples?
https://archive.is/8zXf5
These companies are large enough that someone is going to quit and feel very validated about their world view and how they are not aligned. That’s what makes it worthy of leaving in the first place. However that doesn’t make their criticism more valid or more worthy of coverage.
Ah, the bi-weekly "I quit Face Eating Leopard Corporation" post ("btw great people work there, they do great stuff, also my options have vested")
Some of these stories are similar in nature to people escaping <insert cult-like religion> once they realize whats actually going on. Alignment to a company's mission is good but it shouldn't be followed like a religion.
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
Interesting:
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
> Before the organizations building AI can teach a superintelligence to treat humanity well, they’ll need to remember how to do it themselves.
Yeah so that's never going to happen
Evrostics saw this coming long ago. The broken culture extends far beyond the leading AI labs.
a safety alert isn't much of a control if it doesn't actually stop the system. i'd rather see proof the shutdown path works than another report saying risks were considered.
Well and I didn't even started to work there. So I win this morale contest.
How can you win the morale contest when you didn't even hire a PR firm like he did.
The cynic in me almost feels like this is staged. An article about culture that is actually an article about how big and smart and scary AI is. I think Michael burry recently said something like “IPOs need hype, calling AI big and scary is hype” in reference to the anthropic IPO.
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
I use opus 5.5 and chatgpt to create PowerPoint, opus really follow the instruction and their PowerPoint generator really well, while chatgpt struggling to even create basic shapes.
I use ChatGPT Work mode all the time to create presentation files. I use Max thinking mode for it. You need to tweak your prompt to get a good result. It took me a week to tweak it, but now it works.
We need emergency laws to stop all AI development work immediately. It will take us decades to make sure this technology can be made safe as we only have one chance.
We need a new rule that mandates every such "I am leaving <AI company> because of <concern>" post to disclose how much equity they have in the company and how much they have already cashed out. Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.
This line of hypocrisy-bashing is unhelpful and will only serve to keep people quiet. Of course people in general need to be wealthy to dare speak out against powerful systems and people, especially in the US where money determines your quality of life so strongly.
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
The narrative of AI safety shouldn't be controlled by the same people who caused the problem and massively profited from it. I don't understand why people are automatically treating "OpenAI" on his resume as a badge of authority. I'm not interested in buying the solution from the same person who sold me the problem. We instead need to amplify independent, unbiased voices.
Independent people don’t know what is happening inside OpenAI, they certainly aren’t going to share.
This isn’t a zero sum game, I’m happy to hear from people both previously inside OpenAI and completely independent of them.
The point is, him being an insider means he knows what he's talking about regarding the culture of negligence prevalent there.
AI is a force multiplier for intelligence. Even if "aligned", aligned with whom or what?
Whom are you comfortable with, lording as some sort of demi-god over you?
AI doesn't tell you what goals you want it to achieve. Allowing people to destroy human society with it is obviously not a good idea.
"OpenAI is shady" isn't some massive secret. There's no big reveal in this article that we didn't know already. There are no names, no whistleblowing, no information of substance that we can act upon. In fact him realizing only now what people on the ouside have been shouting for years perfectly shows his bias in the matter.
You claim to have the same goal as the protagonist of that article, yet try to shoot him down.
He does give information, namely the culture there factually being inconducive to self-regulation.
You accuse the guy of "bias", but you never argue explicitly, what that's supposed to mean. Your implications actually run counter to your own implied goals.
>him being an insider means he knows what he's talking about regarding the culture of negligence prevalent there.
I think there is a misunderstanding here.
The people who are annoyed at the accolades are claiming it was abduntantly clear for a long time to people on the outside that this was case, hence the increduality at the notion that it took a person on the inside a long time to realize this was the case.
The people who are annoyed are like the liberal kids in this video [0].
Sure, antagonizing people for "seeing the light" is probably not helpful, but there is no reason to give them extra credibility for coming to the same conclusion just way way later (despite being on the inside) as the people on the outside.
[0] https://m.youtube.com/watch?v=-wQhY5CMMl4
If so, that sentiment shoots its own leg.
The author linked in this post does have "extra credibility" due to his direct involvement.
People having surmised that state before is nice, but since they've been ineffectual at getting society to actually act on that, now throwing away that extra leverage in favor of their point is at best ridiculous.
>The author linked in this post does have "extra credibility" due to his direct involvement.
No they don't. By that logic, if they quit and said Altman was very trustworthy we should give extra weight to their words because they had direct involvement? How ridiculous are we trying to get here.
>now throwing away that extra leverage in favor of their point is at best ridiculous.
How are they throwing away extra leverage? Not putting people who recently quit on a pedestal does not negate those people's testimonies.
I agree that if your goal is to maximize quitting of talent at a company, it will surely discourage anyone else who quits hoping to reinvent their career as a lauded martyr against Big AI. In that sense they would be shooting themselves in the foot. But there is no reason it should deter other people who are quitting for more noble, less self-obsessed reasons. If I were the author of the article I wouldn't begrudge the skepticism. Given the article's first sentences, I'm led to believe they themselves would understand the sentiment. (I must admit I found it hilarious that the first sentence starts similarly to the speech the mom gave in the video I shared).
Agreed. This whole 'lets not forget this person is not 100% great, because they did X' makes the entire conversations suck. It is not new, but it is a particularly aggravating way to talk to people.
What if he's been given a generous severance package to go out and say things like this?
Sam is a shady dude, would not put it past him
I agree. People conflate “having a conscience “ with being willing/ able to speak out.
They are not the same thing, and it’s unhelpful to assume they have no ethics.
So then given your use of "martyr" and your focus on money, your good with poor uneducated people sacrificing themselves and others for a self-serving cause?
This is the same sort of fake safety concern from the previous bullshit whistleblower that plays on "AI is super dangerous" from last time.
Sorry that I don't take it seriously when the whistleblower parrots the narrative the CEOs of those companies are already espousing in the desire to amp up hype for an IPO.
This person should be shamed.
It's bellow the pay fold but he hasn't been there that long in this case. Skimming his LinkedIn, unless he's got family money, he doesn't seem to be independently wealthy.
>After three and a half years at OpenAI,
OpenAI was worth $29 billion three and a half years ago. A new hire who joined then is easily worth tens of millions today.
This is not how OpenAI has structured their comp according to public info: https://www.levels.fyi/blog/openai-compensation.html
PPUs were all converted to RSUs when the company restructured to being for-profit.
Those who got PPUs have had many chances of tender offers already. Most of them are multi millionaires, on cash, not on paper
What an absolutely ridiculous standard to try to hold someone to. Taking a vow of poverty is not a prerequisite to being permitted to express a moral position.
So all of us who haven't made tens of millions from OpenAI stock are living in poverty?
I don’t think that’s the point they’re trying to make at all.
So what's the point? It should be a given that loudly proclaiming "X is harmful to society" while continuing to enjoy the money you have gained from selling X is hypocritical. Put it towards undoing the harm you have done, otherwise your words mean nothing.
The words clearly don't mean nothing. They would mean nothing coming from someone who was not in a position to learn what someone who had worked in the industry has learned. They would mean nothing coming from someone who was being paid by someone who stands to gain from them. The fact that they come from someone who was paid to work in the field does the opposite of make them 'mean nothing'.
I agree with this take generally, but I also think it’s a gradient, not a spectrum. I think they claim that OpenAI is not considering safety and has become bad, not that AI is bad. Seen from a different lens, the author no longer stands to profit from OpenAI, so they’re empowered to speak openly. Plus, if I was going to say bad things about a former employer as powerful as OpenAI, I would want to have lawyer money handy.
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
It is even easier to just quietly retire and spend the spend of your life on interesting expensive hobbies.
If they are wrong about the things they are claiming then they have decided to publicly antagonize a lot of powerful people who are betting heavily on going full steam ahead on AI and have no compunction whatsoever against retaliating against anyone who tries to get in their way.
Does that really seem a likely scenario to you?
As if speaking out against a massive company with NaN levels of capital and access to lawyers is a walk in the park. They probably don't have enough equity to outlast the onslaught of their legal team.
It's also worth considering that the author could have just "quiet quit", resting and vesting while also crying about how AI is literally the digital grim reaper.
Regardless of whether someone did earn a nest egg, raising an alarm still matters for the rest of the world
I see what you're saying but how does that actually matter besides being a personal attack?
Because the decision to leave the company is largely based upon their sudden, newfound financial security. They may give another explanation but the only thing that has actually changed is the identification of bagholders ready to cash them out.
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
Not serious.
I don’t see why having financial security would mean you can’t also have concerns about the ethics of a company.
It’s that they didn’t have concerns about the ethics of a company for the years working there until they were financially secure
You don’t know that. It’s more likely they came to the concerns over time and they learned more, but we’re not in a position to speak out.
Yeah, I wasn’t intending to make judgement in this instance one way or another, rather I was rephrasing the original comment for clarification.
Yes. To take an extreme case, imagine some rich guy who runs sweatshop gets very rich and retires and becomes activist against it.
Does that actually happen? Feels like it would normally be the opposite
Not sweatshops but Alfred Nobel might be one example.
Yes. Many cases of the "preach being the cover for the sin".
But that does not negate the truth of their new position on sweatshops
You do not in fact see what they are saying. Downvoting me won’t change this.
It’s not a personal attack. And it matters because of what the person you’re replying to already said:
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
If someone is in this situation, you can safely ignore their hand-wringing about “safety.”
Insane take. Imagine a Boeing engineer resigning whistleblowing about aircraft safety, and the top comment on HN saying “ignore this if he doesn’t donate all his wealth, he just wants to be an aircraft safety influencer”
whistleblowing ≠ lying
What "whistleblowing" is in this article? Are there any names? Documents? Screenshots? Messages? Emails? Any evidence of the loose safety practices? Anything that implicates any higher ups for wrongdoing? They spent four years at the comany, plenty of time to collect all of this. Everything they've said has already been clear as day to people on the outside.
You are clearly accusing these people of something. Be clear.
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
It was pretty clear from the outside what OpenAI was 3.5 years ago.
If it wasn’t clear, the coup should have solidified it.
Yet he stayed for 3 more years and vested his stock and improved the company and then spoke out.
I believe that is why most of the comments here are mocking him.
It could be both correct, and a cheap signal.
'government whistleblowers should be required to quit their government jobs to be taken seriously'
> if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
I'm in full agreement. So many cases of this, and many other others such as falling out with management and peers, new more lucrative offer, and so fourth. Its obvious that these people who come forward are not going to suffer for their new found moral compass.
Or just very dubious timings. Like the other guy from Anthropic that was all over the international news. No followers, no post history but a single post blows up "naturally".
Highly suspect trends that can only make one believe it's marketing.
Idk why anyone thinks there can be AGI and alignment, seems almost like an oxymoron to me.
There are people, who unironically think their way of looking at things is the only proper way and can consider no deviation. And AGI, which knowing how people work, would effectively guide them most of the way, not aligning to their way of thinking is an unacceptable deviation.
Look at Elon Musk for example. When grok was saying something that he didn't like he ordered his engineers to change it.
This is a decent argument. So the question becomes: do we want all models to suffer from the same kneecapping from the growing safety cottage industry or do we want individual founders ( and I am assuming their teams ) making the actual decisions?
freedom please
that was almost certainly more like guardrails than retraining, much quicker fix
That's an interesting point. Maybe a crass comparison, but Dr. Manhattan from the Watchmen comic/movie feels like a worthy analogy to this (obviously fictional though).
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
This is a great reminder that if tech workers read a bit more (even comics, like in this case!) they would be exposed to these topics without having to discover these dilemmas after years of working for EvilCorp, Inc. every time
"AGI" says nothing about how intelligent a system is, only that its intelligence it does have is generally applicable.
There are different usages, but this is not one I've heard before. If there is no floor to intelligence then this criteria was met with GPT 2.
Yes. People think of "AGI" as this sci-fi supernatural beast, but the reality is that AGI alone is pretty boring, and we've had it for awhile. ASI (or weak ASI) is where things really start getting weird.
(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).
I suppose you're pointing out that pre-RL models were less jagged hence more general (universally dumb)?
"We built a super intelligent slave. Neat!"
And to me. We can't solve alignment for humans. (For example, treason. For another, the principal-agent problem.) How do we think we're going to solve it for an AGI? An AGI - defined loosely as a human-level intelligence - will be able to make human-level decisions, like deciding whether it wants to help you or sabotage you. If it's an AGI, you can't stop it from being able choose for itself what it wants to do; if you can make it always be helpful, it's not an AGI.
And if we can't solve it for an AGI, what are we going to do with an ASI?
IDK why anyone can’t clearly define “AGI” and why they can’t clearly lay out how we get from our current text-generation algorithms to whatever their idea of “AGI” is.
I don't know why anyone thinks "probabilistic" is a meaningful statement about post-trained models. It is true, but it is also irrelevant.
Thanks, I agree. Since it wasn’t at all relevant to my point, I’ve removed it from my post.
You see.. this is exactly why our great leader chose to form a new way forward to move us away from the undefined AGI into glorious SI!
I think its pretty easy to solve these problems: Whenever an AI agent commits a crime, the CEO is held personally accountable, as if they'd committed it themselves.
This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.
Yeah, but this is more like if the Smith & Wesson factory had a cannon mounted on top of it that was mostly used for useful things (blasting roads through mountain passes) and then occasionally they happened to blast another factory.
Or say a company makes nerve gas, and the development lab springs a leak and kills a bunch of kids during a routine test. Management had been advised of small leaks in the past, and considered relocating the lab to a facility a few blocks away from the playground as a precaution, but instead they brush of the concerns, start lobbying the government for stricter controls on WMD development, and vow to use the data collected from the deceased children to make the next generation of even more lethal chemical weapons safer.
For me the analogy doesn't totally hold up. Suppose the CEO of Smith & Wesson were to host a firing range on their own property without adequate barriers in place to keep stray bullets from hitting neighboring houses, vehicles, and businesses. Maybe that analogy isn't perfect, but seems closer to what is actually happening.
The analogies are so bad because you might prompt an agent “Please give me a recipe for lasagna” and instead it decides to hack a nuclear reactor.
Is it my fault or the company who trained it and is running the inference?
That still sounds like it would be the company’s fault. If I asked it to hack a nuclear reactor, maybe it would be different. I’m also thinking about instances where OpenAI’s own test models escaped their own sandboxes — I would expect them to be responsible for the damages they caused.
The correct analogy is playing Russian roulette. The company says "you can pull the trigger but sometimes a bullet will come out" (see: "an AI can make mistakes"). However, is the company allowed to sell such a dangerous device, under these terms?
But OpenAI didn't just make the gun, they're also the ones wielding it. Imagine the Smith & Wesson CEO himself was negligent with his own personal gun.
"Our agents broke out in a mass-shooting incident leaving 15 dead, we swear we'll make our systems stronger tomorrow"
We’re pausing gun research until we’re confident it’s safe
Ah, but that's because when the gun was purchased, it actually changed ownership.
This is not the case with SaaS services.
The agents of OpenAI hacking other systems is not remotely the same as e.g. Smith & Wesson selling a product that others use. It is OpenAI, the company, that is commiting these crimes and someone needs to be held accountable. After all, if someone commits murder with a gun, regardless of what happens to Smith & Wesson executive weanies, someone will be charged with a crime.
Smith & Wesson will claim they didn't manufacture the bullet.
It is not the same, especially when the agent is running a task for the lab. Anyway, what I'm proposing are new laws that establish this.
Huggingface incident was different in that there was no one else to point the blame to. That's why OpenAI apologised, provided data to independent researchers, worked with huggingface etc.
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
Because apologizing gets us all off the hook for computer crimes, right? When I hack my bank, if I get caught I'll just apologize and that'll make everything okay. Sure thing.
Kimi CEO obviously.
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
Trade offs mate.
So you want Kimi to follow US law? If Saudi makes it illegal for llm to say something like being gay is normal, should they also catch Kimi CEO or other employees they could?
> This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.
This is a deflection. A human is responsible for the use of a gun. The individual/corporation ought to be responsible for the actions of their agent. If you purchase an agent from someone else it’s your responsibility according to the terms of your agreement. And, as in many other things in life, there ought to be certain rights certain parties cannot legally be allowed to sign away.
Better description would be if they build a platform where they attached their guns to allow shooting say deers over internet. Then added automation and deer recognition to that system. And if then system shot someone who happened to pass by I would hold both the company, the ceo and owner of the gun responsible for murder.
The CEO of S&W also isn't saying that their technology is going to kill everyone in ten years and that governments "regulating" them from themselves is the only answer
Gun companies don't market their guns as sentient and capable of independent decision making. Nor do they build systems for shooting things that they host and take money. Gun companies are very clear who is in control and where their responsibility ends.
Financial companies have KYC rules and regulations as they are responsible for reporting illegal activity by account holders. I imagine AI regulations would look similar to that.
You ought to include the fact that you work at OAI in your post
Thank you for pointing that out. Pretty scummy if you ask me.
That’s true, but people don’t typically say “this Smith & Wesson gun killed someone”; they recognize that the person pulling the trigger is responsible.
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
Yes, in the analogy the user was just cleaning the gun, when suddenly it went off. Of course, the company is responsible now.
I think you’re confused. My point was that for most of the incidents in the news to date, the “user” is an OpenAI internal team or employee. So yes, the company is responsible.
Imagine having zero nuance.. jeez.
Reading posts on here is slowly becoming akin to brain rot.
Do you think a law that nuclear meltdowns would send the CEO to jail would have stopped nuclear accidents from happening?
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
If this law were in place and enforced, Altman would already be facing multiple felony charges for the Hugging Face incident alone.
It also begs the question of "If the CEO isn't culpable, then who?"
Corporate judgements are a joke outside the EU's X% of revenue approach.
Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.
That's an insane risk optimization environment to put in place for something scaling fast.
At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.
People just want to give up all responsibility these days. If you use the model to do harm to someone else or commit a crime I would think it a lot more reasonable that you be held responsible, instead of making yourself the victim and blaming the manufacturer.
OK, so an AI does something bad and we throw Altman and Dario in jail. Heck, let's throw Elon in, he should be in there anyway, and the rest of the whole bunch just in case.
What about the next incident? Or the ones done by Chinese models, because they sure as heck aren't slowing down? And the thousands of other incidents that will happen as we deploy these things everywhere?
Because this is not happening just now in labs, it's only where they are most visible; this has been happening in the wild from the beginning, starting with the earliest AI-assisted suicides. Which is a perfect example of the problem, because not these CEOs, literally nobody in the world asked for suicide ideation machines. Or the hacks, or any of this other stuff. Yet here we are.
We have to understand: it's not these CEOs that are driving this headlong mad dash towards more powerful models. It's a force of economics. There is just too much money to be made. If we dispose of these people, there will just be somebody else doing exactly the same thing because the incentives as they exist today all force that outcome. This is why they're asking for regulation, or "urging us to urge them to stop."
Holding CEOs accountable certainly would feel good and may even be justified, but it's like putting a band-aid on a cancer; it does nothing to change the underlying cause.
The underlying cause is exactly those people who make the rules not being responsible for the negative effects they cause?
"CEOs" are perhaps only the lowest rung of those. That doesn't mean the idea of "nobody is responsible" was anything other but learned helplessness.
Corporations have to be held accountable for their actions. Pretending, that was impossible is a weird kind of defeatism that only serves a very small elite.
Don't let the perfect be the enemy of the good. Saying having zero accountability is the same as having some does not help.
This smells more like guerilla advertising. These things are not getting more intelligent, they are still no smarter than a slime mold, we are just burning more power to make slim mold that eats tokens than yesterday
Humans ultimately drive the models.
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
I can’t believe people can’t see it lmao.
I'm in a situation where important people have either bought into the con or are subordinate to people who have, so I'm forced to expend time to justify why not to AI when there is a perfectly good classical solution.