If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money. These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong.
The carelessness is also clear in how AI is rolled out. It's being presented as a disruptive technology, as is everything else presented by modern tech companies. The goal is a very large payout in a very short period of time, it's not to make anything better, or worse, either is just a side effect.
Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset. Same for writing, it's just a better version of Grammarly, it's fine, or as a tool in drawing software. The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
> The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
I feel like you’re not acknowledging the inevitability of the technology being developed further. You’re also ignoring competition and the basic fact that people do research and push the boundaries of their field naturally.
If the major frontier AI labs had decided, as you said, to make a fancy autocomplete and a better Grammarly and then simply stop and coast, do you really believe that every AI researcher and entrepreneur in the entire world would look at the situation and decide that the status quo is where it should end?
Humans don’t work that way. People would get ideas and want to push the limit. If the frontier labs stopped developing their technology, other companies would be created to carry on. Academic researchers would push the limits further. It would be discovered that the AI could go further and we’d get back to this same eventuality with a different set of people leading it.
I also disagree with your characterization of this as a get rich quick scheme for AI leaders who want to make a lot of money and then be done. I think this characterization feels truthy if your only conception of startups is as a gamble to retire early, but the people who build and run these companies actually do like building and running companies. If the AI leaders and executive management you’re talking about were separated from their jobs today, even with billions of equity, most of them would be out starting new companies next month because that’s what people like that do. I’m not saying this to “glaze” them or whatever I’ll surely be accused of in the comments, I’m stating some facts about this space. You don’t need to look far to find examples of this already happening.
> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset.
I'm not rooting for AI companies or hustlers that are out to ride the wave and get rich.
However your claim is just hyperbole. I've used it in cross-cutting projects to extend protocol on desktop, iot device, which involves holding premise long enough to co-develop and test. It is also fantastic in debugging multi-threaded systems.
This is an interesting theme that I see a lot: Some people are firmly anchored to an idea of AI capabilities that is a year or more behind the frontier.
Some times it comes from people who try out ChatGPT free tier or the Google AI summary and think it represents the state of the art.
More recently I suspect it’s from people who build their opinion based on being contrarian and cynical: They dislike the AI people and distrust everything they say. If the AI people say it’s changing the world, then the cynical and contrarian belief is to assume that the technology is actually very bad and they’re just lying to make a lot of money and retire.
These takes were more understandable in the earlier phases of AI when it really was very limited, but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
There are a huge number of people involved in AI outside of the major AI companies both in smaller AI companies and lots of independent developers. They aren't having much success bringing AI to the masses and this is because they don't care enough about money and they aren't selling their ideas aggressively enough. There is this odd tendency to undersell achievements because they weren't motivated by the Right Stuff.
The AI companies are as much being pushed along by investors as they are in the driving seat, but the push is in the direction of getting real products in front of people and experimenting with this miraculous tech everywhere it could possible work.
Ignoring the implementation details, if we treat the whole global technology world as one single AI system made up of AI companies, independent developers/researchers, and end users, what would be the training goal and evaluation?
An excellent comment. It's very true - there's a deep sickness here which causes people to flee from the inherent value (relatively modest) and the longer, more tedious work of building a sustainable company around it.
> there's a deep sickness here which causes people to flee from the inherent value (relatively modest)
Value is simply the mutual understanding that something should cost a certain amount, and the system we've built emphasizes the need for profit above all else. When a set of companies comes along with a new, impressive technology claiming that they can automate away the white collar workforce, they're trying to capture as much value as they can.
It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
> It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end
I don't agree that "I am just a market actor behaving rationally" somehow absolves you of any moral duty
I'm not an ethicist or anything but I have studied quite a lot of ethical frameworks and I've never encountered one that I think would agree with this
> Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
I didn’t expect a full throated defense of mugging and burglary on HN this morning, but here we are.
I enthusiastically agree. I've been on this ride since GPT 2 and 3, I did start out looking at the technology in a neutral or positive light. I did not think the stewards of this technology would be so irresponsible, its consequences would rival the Stalinist purges.
No matter what practical benefits this technology might bring, if this runaway marketing campaign isn't stopped, in just a couple decades we'll see what the complete collapse of civil society looks like, a world where the institutions no longer see value in maintaining an educated democratic populous, and can relax back into the comfort of feudalism.
The AI formula is out of the bag. The Manhattan Project is over, trinity has happened.
If it weren’t OAI and Anthropic, it would be others.
If it weren’t US companies, it would be others.
It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).
Look at what happened with fossil fuels. Even if the entire global west totally ceased use, there are many countries that simply will not stop because of local utility. AI is the exact same way.
There is something deeply rotten about the US mindset though, them being American is definitely a problem.
> Look at what happened with fossil fuels.
The west had a 100 years head start, on top of that these countries are producing everything you consume, don't start pointing fingers. Chinese current co2 per capita is the same as the US in 1900
I remember reading about the development of neural networks in a magazine from the 1980s.
Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did, and the Chinese followed in.
Had AI been invented in the 20th century, it would have been kept secret and used only by state actors to facilitate genocides at mass scales. But we live in different times.
> Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did
Uh? In what universe are openai and anthropic not shadow groups?
Are you misunderstanding on purpose? OpenAI and Anthropic let the public use their AI models. Shadow groups would mean that people wouldn't even know about the existence of AI.
> If it weren’t OAI and Anthropic, it would be others.
Doing what? Rushing to make trillion dollar companies, rather then developing the tech and their companies in a sustainable way? I don't even know what in the original comment this would be a response to.
Rushing towards creating machines that can give you vast amounts of military, political or economic power without regard to it's effects on the old order.
The particular form of the pursuit of power practiced by OAI and Antrophic - the creation of trillion dollar companies, is one of the most benign in recorded history. It's so unnaturally benign, that I doubt it's going to survive for much longer.
Yes, the cat is out of the bag as to how to create this type of AI.
But that doesn't mean that everyone has to use it.
We're only starting to see the beginning of the negative impact of AI to the job market and society. If if turns out that allowing most jobs to be replaced by AI is bad for society and the economy (super high unemployment - nobody buying more then necessities), then wouldn't it be smarter to not do that?
If that's the case, then each country can decide for themselves.
Dario's "Machines of loving grace" UBI/foodstamp "utopia" sure doesn't sound like a recipe for success.
Which job are worth doing and which ones you'd rather the AI do?
AI should enable things that were previously impossible because of time constraints.
If time is money and we were previously trading with money, we will get back to trading time. Manufacturing time for instance. Compute times.
For commodities, that might be automated to some extent, I am not sure. The friction in exchanges is where the money is made and the friction here may be at the interface with the real world still.
I’m sorry, but the AI workers are as much to blame.
Or have you forgotten that Sam Altman was ousted by individuals that shared your concerns, and it was the OpenAI employees that rebelled against the board and wanted Altman back?
That HN post was infamous and nearly took down this site.
"he warned it could trigger a large-scale incident this year or next, such as a temporary blackout of financial or power systems."
I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.
Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.
So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.
We are starting to get a taste. Credit Mutuel Arkea just admitted having been hacked. Over 100 millions gone.
Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.
However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.
Don't ignore the very real human cost this would bring. Every day the power is out, more vulnerable people would be harmed(including death) directly from not having electricity for essentials or from systemic failures like in food distribution and storage. Financial systems being down could also have extreme impacts on some people, but maybe easier to mitigate via human compassion.
Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.
I agree that people only really respond when something goes wrong badly enough that its impossible to ignore.
I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.
This sort of thing was going to happen, with or without AI (state sponsored hackers, the US or China taking down systems they control, or whatever).
The later it happens, the more we will be down the path that lets it happen, and the worse the consequences. We are getting gradually worse consequences: AWS outages taking down websites, GCP outages taking down banks, attacks on fridges disrupting good supplies...
Its better for us to change path when something happens that causes dozens of people die rather than wait until its millions.
I don't think that's a good thing. But if we DON'T do anything, then even more poeple are likely to suffer. So while people dying is not a good thing, a thousand people dying is better than a million people dying because we don't do anything.
Same thing about burning your hand. It's not a good thing to burn your hand, but burning your hand once when you're young probably saves you from a more serious accident later on.
You’re describing a weapon, and if AI managed to do that (I believe we are not far away), then this would probably increase the investment not decrease it.
Would it? I don't think so. Not if the weapon is an indiscriminate one and makes everyone worse off. Yes, some people (maybe military generals) might support it. But there would be near universal hatred for it, if people wake up one day and find their bank accounts not working or their eletricity not working.
This is why I think AI is so stupid. It's just too powerful for anyone to control, especially with the inertial mass of capitalistic society powering it through the incremental, short-term advantages it gives.
> Nuclear weapons are indiscriminate and make everyone worse off if used.
The difference is most people have not experienced any immediate effects of nuclear weapons. But people will experience the immediate effects of AI taking down the electricity grid, for instance.
"most people" are in the set of things the trillionaires don't give a shit about. Their market distortion has already made PCs unaffordable and electricity prices rise, among other adverse effects. What else has to happen so you take notice?
Of course, we are.
Back in the late 80’s early 90’s, no one working on ML was aspiring to develop systems which would be used to generate the kind of AI crap which plagues us today.The good is being smothered by the absolutely soul-crushing pointless.
I hear you and i find it devastating, too. On top of that, the negative effects on the planet for having this technology vomit this soul-crushingly pointless crap is just depressing.
AI exposes the Sturgeon's law (90% of everything is crap), and amplifies it much more, and anything good is drowned out. Most of human-generated popular music is slop. Most of human generated code is buggy, unmaintainable and full of debt. Social media and youtube have always been dominated by click-bait and mediocre time-wasting crap.
Before AI I was already being smothered by soul crushing pointless junk made by humans.
You’re placing blame on the wrong spot. It’s algorithms and social media incentive systems not AI. Our digital media incentives addictive minimum effort chum and slop. AI just makes it easier to make.
My main fear is that the integrity of everything will be compromised, but everyone will keep acting as if it's not. I feel like it's already somewhat the case but I'm worried about it getting worse.
My prediction is that some people will be able to use ASI to create essentially infinite money inside their bank accounts (through hacking); the prices of assets will grow to ridiculous sums as the people with infinite money buy up assets, but yields on all assets would soon approach 0%; still, one asset is going to grow at a faster rate than all other assets (E.g. an AI company backed by blockchain/tokenized shares that are impossible to hack) and everyone is going to start piling their infinite money into that asset; every other asset will crash to $0 because of the rapid, exponential price acceleration of the top asset making every other asset unattractive by comparison. Then fiat money would cease to exist and shares of the top AI company will be used as money instead. Then prices of assets would stabilize again; denominated in the new currency... Then after that, we will finally be able to start innovating with AI.
You might be right. The weakness of current popular cryptocurrencies is the complexity and size of their code. Many above 100k lines of code. That's risky. Though the benefit is that they are somewhat heterogeneous so it may be hard to find a vulnerability which affects more than 50% of nodes; and this is a hard requirement for compromising the integrity of that system.
Traditional finance has the benefit that they can more easily reverse malicious transactions, but if hacks become too common, it will be impossible to correct fast enough and in a reliable way. With traditional banking, a hacker only needs to hack a few nodes to compromise the integrity of the entire system.
So Hinton doesn't trust the tech leaders but then the consequence of having governments censor LLMs is that only tech leaders and governments have unlimited access to AI.
If someone manages with the help of AI to hack into a vital system, then the system was already in a critically bad state beforehand. If developers can't use AI to check the code for security flaws, then security gets worse and not better.
I would posit that systems being hacked by AI were already vulnerable to the huge caches of secret zero days held by intelligence agencies and organized crime around the world. Which we know exist.
Insecure is insecure. These systems never were secure.
"We". Who "is" we? This isn't about you. or me. there was never a we. Before you open your mouth or put your pen down, wouldn't you think of the shareholder value for once?
Yeah, seriously. AI has been the single biggest technological improvement to my life in the last 20 years. If I could choose between smartphones and AI, I'll take AI any time of the day. Smartphones (in combination with social media) had a worse impact on society than AI has had and are not worth their advantages to me.
The same applies to most of humanity’s other inventions, such as transportation, weapon, electricity... Ultimately, these technologies serve as instruments for rivalry, deciding the fate of the strong versus the weak.
“I think [Mark] Zuckerberg and [Elon] Musk and Sam Altman, they understand the risks, and they don’t give a shit. They’d rather have the money. That’s my belief.”
What's risk for people valued close to or over trillion dollars? Risk is meaningless when corporations get too big to fail.
As is the case with any technology in the hands of careless people, but unfortunately we can't wait for society to catch up morally to the technical standards, innovation is organic and breakthroughs have no timing.
"It might turn out we were better off not opening Pandora's box" says man who helped open Pandora's box.
That's a touching sentiment or whatever but the nature of the thing is that it will inevitably be opened and that it cannot be closed thereafter. So (I think) we'd all be better off putting our efforts towards predicting the likely outcomes and figuring out how to make the best of them.
To be honest, I think it depends on the environment. In the past, if you didn't come out of a good university, your access to knowledge was limited. But these days, in terms of access to knowledge, I think things have clearly improved, whether or not AI is illegally sharing knowledge.
Looking back now, was it really happier before AI existed? I'm not sure. In the past, when I wanted to find out whether I was right or wrong about some piece of knowledge or information, I'd contact several people and, far from getting an answer, I'd often be told not to bother them. But AI, in that respect, is the best teacher I can think of.
I think the times are continuously getting better. It's just that there's a problem of power concentration among the people who handle it. Still, if AI is a tool of that nature, then I suppose that can happen.
But considering what access to knowledge was like before AI came along, I think having AI is better.
Knowledge was already extremely accessible in the age of the Internet, but it required time to read and learn. AI is making that same knowledge applicable without the prerequisite of learning.
Most people are not interesting in learning for learning's sake, they are motivated to learn something they can apply (see: kids complaining about how learning Math in school doesn't transfer to useful skills). Why learn about architecture and technical debt, if you can tell the machine to make it faster?
I don't think you're wrong, but fundamentally we're in an age of information overload, and this may be the new normal. In other words, the environment where you had to know things deeply and become an expert, like in the past, may be different from an environment where you know many things broadly and combine them.
I basically don't think you're wrong. However, books and reading have their information fixed at the point of publication, whereas depending on where you live or your community, the information you get buried in differs. Because I was in an Asian country, it was actually very hard for me to synchronize my programming skills with the latest technology in the US. The frameworks commonly used in my country are stuck at roughly 15 years ago. The time lag between US technology and my mainstream technology is about 5 years. In my area, it's about 15 years.
Anyway, thinking about your opinion, I think there are both valid points and dark points. I think learning for its own sake is important.
I agree that people should have an attitude of learning for its own sake, but I don't think it's mandatory. I think it should vary depending on the environment. Still... I do think having that attitude is a good thing. I like your idealism, senior.
However, that holds for people who are middle class or above and can survive. Because rich kids can picture the future and are given the right to love learning. But if you're poor, that's very hard. And when you grow up in such a poor family, there's an emphasis on it having to lead to practical skills.
And even if someone loves learning for its own sake, I think quite a lot of people love the honor that comes from learning. But that too can be called loving learning itself. It's purely a matter of individual values.
I don't think you're wrong, senior, but I also wonder whether the old ways are right in today's era of excessive information. It's just that because our lives and the environments surrounding us are different, we came to different conclusions. I upvoted because I agree with your thoughts. But there are points where I differ. Have a good day.
This is tangential encouraged by your point and other forays here. But one does wonder what arguments politicians (the power lies with them) understand when they make their pronouncements. The UK has for instance, prior to the new administration, totally and vociferously supported net zero. From the point of view (a fact for many but not for all informed authorities) that climate change is caused by CO2 and we are headed for a disaster - Net Zero is a logical step to trying to save humanity and the whole ecosystem. Now (November) I read that the UK powers that be, appear to have put it aside in actions at least. Countdown to Net Zero doesn't matter any more; open the oil fields, get rid of subsidies for photovoltaic and wind - and other measures. My point is not a political jibe but a stance I suspect to apply in other areas. Is climate change going to be a catastrophe or not? They should make up their minds.
In East Asian countries, certification systems are still maintained, but in reality they're practically meaningless. If a certification system exists, people will take an interest in CS and work harder at it, or so you'd think—but in reality, the exam questions end up stuck in the old style. In other words, there's a big gap between the exams and the actual field.
Because a certification has to be fixed at a particular point in time. Korea's certification exams start with the 23 GoF patterns and go on to rote memorization of countless patterns and layered terminology.
The problem is that the way you learn for this certification is the old way, so someone who gets the certification ends up writing legacy code.
But if you always update the questions to the latest, then nobody can pass the certification. So, having seen how the certification system works in East Asia, I don't think it's all that good
It might turn out we were better off without nuclear weapons, but so far I’m happy we haven’t had another world war (in the traditional “total war” sense).
No way to tell until after, though. So far, mutually assured destruction is working. It works until it doesn’t.
Same with AI. Thus far we’re better off with it than without it. That may change.
I mean, look what we did with fossil fuels. Some people are even still willingly burning them, even though the data is clear that our use of them has already started the total destruction of our civilization.
Is there a name for this, a technology that is so insanely useful that there is no way we won’t exploit it, but where the worst case scenario is total collapse, and impossible to determine well in advance?
(Fossil fuels I suppose don’t count as even the energy companies knew unambiguously decades ago that large scale use would eventually bake the entire agricultural system out of existence.)
>so far I’m happy we haven’t had another world war
The reason for that was more that the 3-4 great empires involved in the previous two were exhausted, Japan was defeated, China was in no shape, and US vs USSR had no reason to push for one, since the Cold War era gave them plenty of breathing room, roughly half the world each.
Both the USSR and US fought multiple wars directly against the proxies of the other. They weren't simply hanging back and letting others do the dirty work. In most any other era one of those would have become a full blown war between the two. Both sides pushed the line of not directly involved and then both sides looked the other way versus escalating. Nukes made the alternative too much of a risk for both sides.
The scene is bleak, silent, desert-like with a few dead trees bleached white by unfiltered UV. Nothing moves. In the distance a cave, and from that cave
No it's a completely serious question. The goal has always been to replicate human/biological thinking processes in software, and now that this has been more or less achieved he's suddenly not happy about it. There's been plenty of opportunity to actually think about "what if all of this actually works" but they chose not to.
"Could" does a lot of work in these kinds of claims.
We need to build it as fast as possible because it could cure all disease. Ignore whether that's possible, how it might happen, whether all we're missing to do it is more intelligence, etc.
The only way I see us curing all disease is to throw out all the names we've given to collections of symptoms. What disease? I don't see no disease, you just have a sore throat, runny nose, and mild fever.
> We need to build it as fast as possible because it could cure all disease
Hell no. We need to risk pumping out even more CO2, causing more environmental disaster, ruining entire human communities, just to cure diseases? You might as well say it's better to be in prison because you're guaranteed three meals a day. The logic of some pro-AI people is baffling.
I have to spend extra time just trying to find AI-free things, AI-free writing, AI-free communities, and people who aren't addicted to AI. And get noticed/get a job by having to push through all this AI static. And having to interact with AI-based customer service agents that take 10x as long to resolve the problem as a human being. And having to wait a bit longer because there are 10x more 'Are You Human' Captchas slowing down website loading. And having to download OS updates that are 2x as large because they contain AI models. And having to double-check everything because there's so much more hallucinated crap out there.
The other day I hired an accountant and even he sent me an AI answer that sounded right but was actually bullshit.
In short, it takes a LOT more time just to find some sanity now.
I mean, it could if "we" "just" got out of globalized financial capitalism, changed everything about our political, economical, financial, social systems, etc.
Only 14 years old and accelerationists think we can just push the capitalism pedal further and somehow get social changes going their way
I don't think accelerationists think we can do anything. The thought process of the accelerationist is partly fueled by the 'excitement' of betting and playing musical chairs and hoping to be the first ones out holding a bag of money and partly fueld by the delusion that technological advancement is always good at any level of complexity.
If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money. These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong.
The carelessness is also clear in how AI is rolled out. It's being presented as a disruptive technology, as is everything else presented by modern tech companies. The goal is a very large payout in a very short period of time, it's not to make anything better, or worse, either is just a side effect.
Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset. Same for writing, it's just a better version of Grammarly, it's fine, or as a tool in drawing software. The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
> The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
I feel like you’re not acknowledging the inevitability of the technology being developed further. You’re also ignoring competition and the basic fact that people do research and push the boundaries of their field naturally.
If the major frontier AI labs had decided, as you said, to make a fancy autocomplete and a better Grammarly and then simply stop and coast, do you really believe that every AI researcher and entrepreneur in the entire world would look at the situation and decide that the status quo is where it should end?
Humans don’t work that way. People would get ideas and want to push the limit. If the frontier labs stopped developing their technology, other companies would be created to carry on. Academic researchers would push the limits further. It would be discovered that the AI could go further and we’d get back to this same eventuality with a different set of people leading it.
I also disagree with your characterization of this as a get rich quick scheme for AI leaders who want to make a lot of money and then be done. I think this characterization feels truthy if your only conception of startups is as a gamble to retire early, but the people who build and run these companies actually do like building and running companies. If the AI leaders and executive management you’re talking about were separated from their jobs today, even with billions of equity, most of them would be out starting new companies next month because that’s what people like that do. I’m not saying this to “glaze” them or whatever I’ll surely be accused of in the comments, I’m stating some facts about this space. You don’t need to look far to find examples of this already happening.
> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset.
I'm not rooting for AI companies or hustlers that are out to ride the wave and get rich.
However your claim is just hyperbole. I've used it in cross-cutting projects to extend protocol on desktop, iot device, which involves holding premise long enough to co-develop and test. It is also fantastic in debugging multi-threaded systems.
This is an interesting theme that I see a lot: Some people are firmly anchored to an idea of AI capabilities that is a year or more behind the frontier.
Some times it comes from people who try out ChatGPT free tier or the Google AI summary and think it represents the state of the art.
More recently I suspect it’s from people who build their opinion based on being contrarian and cynical: They dislike the AI people and distrust everything they say. If the AI people say it’s changing the world, then the cynical and contrarian belief is to assume that the technology is actually very bad and they’re just lying to make a lot of money and retire.
These takes were more understandable in the earlier phases of AI when it really was very limited, but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
> Had LLMs first been presented as a "better IntelliSense"... very few people would have been upset.
Presenting AI as something it isn't doesn't change the reality which is like the Wait But Why thing https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
That was written in 2015 before LLMs were much of a thing and has been fairly spot on so far. This is not about LLMs.
There are a huge number of people involved in AI outside of the major AI companies both in smaller AI companies and lots of independent developers. They aren't having much success bringing AI to the masses and this is because they don't care enough about money and they aren't selling their ideas aggressively enough. There is this odd tendency to undersell achievements because they weren't motivated by the Right Stuff.
The AI companies are as much being pushed along by investors as they are in the driving seat, but the push is in the direction of getting real products in front of people and experimenting with this miraculous tech everywhere it could possible work.
What would be the Right Stuff?
Ignoring the implementation details, if we treat the whole global technology world as one single AI system made up of AI companies, independent developers/researchers, and end users, what would be the training goal and evaluation?
> There are a huge number of people involved in AI outside of the major AI companies
How many, approximately?
An excellent comment. It's very true - there's a deep sickness here which causes people to flee from the inherent value (relatively modest) and the longer, more tedious work of building a sustainable company around it.
> there's a deep sickness here which causes people to flee from the inherent value (relatively modest)
Value is simply the mutual understanding that something should cost a certain amount, and the system we've built emphasizes the need for profit above all else. When a set of companies comes along with a new, impressive technology claiming that they can automate away the white collar workforce, they're trying to capture as much value as they can.
It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
> It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end
I don't agree that "I am just a market actor behaving rationally" somehow absolves you of any moral duty
I'm not an ethicist or anything but I have studied quite a lot of ethical frameworks and I've never encountered one that I think would agree with this
> Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
I didn’t expect a full throated defense of mugging and burglary on HN this morning, but here we are.
I enthusiastically agree. I've been on this ride since GPT 2 and 3, I did start out looking at the technology in a neutral or positive light. I did not think the stewards of this technology would be so irresponsible, its consequences would rival the Stalinist purges.
No matter what practical benefits this technology might bring, if this runaway marketing campaign isn't stopped, in just a couple decades we'll see what the complete collapse of civil society looks like, a world where the institutions no longer see value in maintaining an educated democratic populous, and can relax back into the comfort of feudalism.
The AI formula is out of the bag. The Manhattan Project is over, trinity has happened.
If it weren’t OAI and Anthropic, it would be others.
If it weren’t US companies, it would be others.
It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).
Look at what happened with fossil fuels. Even if the entire global west totally ceased use, there are many countries that simply will not stop because of local utility. AI is the exact same way.
> If it weren’t US companies, it would be others.
There is something deeply rotten about the US mindset though, them being American is definitely a problem.
> Look at what happened with fossil fuels.
The west had a 100 years head start, on top of that these countries are producing everything you consume, don't start pointing fingers. Chinese current co2 per capita is the same as the US in 1900
I remember reading about the development of neural networks in a magazine from the 1980s.
Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did, and the Chinese followed in.
Had AI been invented in the 20th century, it would have been kept secret and used only by state actors to facilitate genocides at mass scales. But we live in different times.
How can "AI" facilitate genocide? Like what might chatgpt have suggested that would have made the nazis more effective or the khmer rouge.
> Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did
Uh? In what universe are openai and anthropic not shadow groups?
Are you misunderstanding on purpose? OpenAI and Anthropic let the public use their AI models. Shadow groups would mean that people wouldn't even know about the existence of AI.
> If it weren’t OAI and Anthropic, it would be others.
Doing what? Rushing to make trillion dollar companies, rather then developing the tech and their companies in a sustainable way? I don't even know what in the original comment this would be a response to.
Rushing towards creating machines that can give you vast amounts of military, political or economic power without regard to it's effects on the old order.
The particular form of the pursuit of power practiced by OAI and Antrophic - the creation of trillion dollar companies, is one of the most benign in recorded history. It's so unnaturally benign, that I doubt it's going to survive for much longer.
Yes, the cat is out of the bag as to how to create this type of AI.
But that doesn't mean that everyone has to use it.
We're only starting to see the beginning of the negative impact of AI to the job market and society. If if turns out that allowing most jobs to be replaced by AI is bad for society and the economy (super high unemployment - nobody buying more then necessities), then wouldn't it be smarter to not do that?
If that's the case, then each country can decide for themselves.
Dario's "Machines of loving grace" UBI/foodstamp "utopia" sure doesn't sound like a recipe for success.
Which job are worth doing and which ones you'd rather the AI do? AI should enable things that were previously impossible because of time constraints. If time is money and we were previously trading with money, we will get back to trading time. Manufacturing time for instance. Compute times. For commodities, that might be automated to some extent, I am not sure. The friction in exchanges is where the money is made and the friction here may be at the interface with the real world still.
I’m sorry, but the AI workers are as much to blame.
Or have you forgotten that Sam Altman was ousted by individuals that shared your concerns, and it was the OpenAI employees that rebelled against the board and wanted Altman back?
That HN post was infamous and nearly took down this site.
"he warned it could trigger a large-scale incident this year or next, such as a temporary blackout of financial or power systems."
I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.
Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.
So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.
We are starting to get a taste. Credit Mutuel Arkea just admitted having been hacked. Over 100 millions gone.
Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.
However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.
Blockchain dapps were getting hacked long before AI. Not a signal
If people's pensions & life savings are flushed out the back door (literally), rest assured attention WILL be paid.
Don't ignore the very real human cost this would bring. Every day the power is out, more vulnerable people would be harmed(including death) directly from not having electricity for essentials or from systemic failures like in food distribution and storage. Financial systems being down could also have extreme impacts on some people, but maybe easier to mitigate via human compassion.
Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.
I agree that people only really respond when something goes wrong badly enough that its impossible to ignore.
I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.
It would be a lesser evil rather than good.
This sort of thing was going to happen, with or without AI (state sponsored hackers, the US or China taking down systems they control, or whatever).
The later it happens, the more we will be down the path that lets it happen, and the worse the consequences. We are getting gradually worse consequences: AWS outages taking down websites, GCP outages taking down banks, attacks on fridges disrupting good supplies...
Its better for us to change path when something happens that causes dozens of people die rather than wait until its millions.
> people will die.
I don't think that's a good thing. But if we DON'T do anything, then even more poeple are likely to suffer. So while people dying is not a good thing, a thousand people dying is better than a million people dying because we don't do anything.
Same thing about burning your hand. It's not a good thing to burn your hand, but burning your hand once when you're young probably saves you from a more serious accident later on.
You’re describing a weapon, and if AI managed to do that (I believe we are not far away), then this would probably increase the investment not decrease it.
Would it? I don't think so. Not if the weapon is an indiscriminate one and makes everyone worse off. Yes, some people (maybe military generals) might support it. But there would be near universal hatred for it, if people wake up one day and find their bank accounts not working or their eletricity not working.
This is why I think AI is so stupid. It's just too powerful for anyone to control, especially with the inertial mass of capitalistic society powering it through the incremental, short-term advantages it gives.
> Not if the weapon is an indiscriminate one and makes everyone worse off.
Nuclear weapons are indiscriminate and make everyone worse off if used. And everybody has them. Your optimism is unfounded.
> Nuclear weapons are indiscriminate and make everyone worse off if used.
The difference is most people have not experienced any immediate effects of nuclear weapons. But people will experience the immediate effects of AI taking down the electricity grid, for instance.
"most people" are in the set of things the trillionaires don't give a shit about. Their market distortion has already made PCs unaffordable and electricity prices rise, among other adverse effects. What else has to happen so you take notice?
Of course, we are. Back in the late 80’s early 90’s, no one working on ML was aspiring to develop systems which would be used to generate the kind of AI crap which plagues us today.The good is being smothered by the absolutely soul-crushing pointless.
I hear you and i find it devastating, too. On top of that, the negative effects on the planet for having this technology vomit this soul-crushingly pointless crap is just depressing.
AI exposes the Sturgeon's law (90% of everything is crap), and amplifies it much more, and anything good is drowned out. Most of human-generated popular music is slop. Most of human generated code is buggy, unmaintainable and full of debt. Social media and youtube have always been dominated by click-bait and mediocre time-wasting crap.
Before AI I was already being smothered by soul crushing pointless junk made by humans.
You’re placing blame on the wrong spot. It’s algorithms and social media incentive systems not AI. Our digital media incentives addictive minimum effort chum and slop. AI just makes it easier to make.
My main fear is that the integrity of everything will be compromised, but everyone will keep acting as if it's not. I feel like it's already somewhat the case but I'm worried about it getting worse.
My prediction is that some people will be able to use ASI to create essentially infinite money inside their bank accounts (through hacking); the prices of assets will grow to ridiculous sums as the people with infinite money buy up assets, but yields on all assets would soon approach 0%; still, one asset is going to grow at a faster rate than all other assets (E.g. an AI company backed by blockchain/tokenized shares that are impossible to hack) and everyone is going to start piling their infinite money into that asset; every other asset will crash to $0 because of the rapid, exponential price acceleration of the top asset making every other asset unattractive by comparison. Then fiat money would cease to exist and shares of the top AI company will be used as money instead. Then prices of assets would stabilize again; denominated in the new currency... Then after that, we will finally be able to start innovating with AI.
I suppose, but precious metals and cryptocurriencies would still exist, and would probably be more immune to power struggles and hacking than stocks.
Not to say that they're unhackable or unstealable, but merely that they dont have a central weak point that if broken, destroys the whole system
You might be right. The weakness of current popular cryptocurrencies is the complexity and size of their code. Many above 100k lines of code. That's risky. Though the benefit is that they are somewhat heterogeneous so it may be hard to find a vulnerability which affects more than 50% of nodes; and this is a hard requirement for compromising the integrity of that system.
Traditional finance has the benefit that they can more easily reverse malicious transactions, but if hacks become too common, it will be impossible to correct fast enough and in a reliable way. With traditional banking, a hacker only needs to hack a few nodes to compromise the integrity of the entire system.
So Hinton doesn't trust the tech leaders but then the consequence of having governments censor LLMs is that only tech leaders and governments have unlimited access to AI.
If someone manages with the help of AI to hack into a vital system, then the system was already in a critically bad state beforehand. If developers can't use AI to check the code for security flaws, then security gets worse and not better.
I would posit that systems being hacked by AI were already vulnerable to the huge caches of secret zero days held by intelligence agencies and organized crime around the world. Which we know exist.
Insecure is insecure. These systems never were secure.
"We". Who "is" we? This isn't about you. or me. there was never a we. Before you open your mouth or put your pen down, wouldn't you think of the shareholder value for once?
Speak for yourself - I'm in AI heaven.
Programming has never been this awesome.
Yeah, seriously. AI has been the single biggest technological improvement to my life in the last 20 years. If I could choose between smartphones and AI, I'll take AI any time of the day. Smartphones (in combination with social media) had a worse impact on society than AI has had and are not worth their advantages to me.
Can't agree more - some of us are just happily achieving our life goals in record time thanks to AI!
The people in wall-e were in AI heaven.
The same applies to most of humanity’s other inventions, such as transportation, weapon, electricity... Ultimately, these technologies serve as instruments for rivalry, deciding the fate of the strong versus the weak.
> What's risk for people valued close to or over trillion dollars?
Mortality.
[delayed]
Such a smart guy so late for the party. Thanks to him and others who understand a lot about tech, we gave monster a weapon to destroy us.
I still don't know how Karpathy still works with those folks. I really think the guy is great but he makes me wonder
Why are the using the Ukrainian transliteration of Chernobyl now
As is the case with any technology in the hands of careless people, but unfortunately we can't wait for society to catch up morally to the technical standards, innovation is organic and breakthroughs have no timing.
"It might turn out we were better off not opening Pandora's box" says man who helped open Pandora's box.
That's a touching sentiment or whatever but the nature of the thing is that it will inevitably be opened and that it cannot be closed thereafter. So (I think) we'd all be better off putting our efforts towards predicting the likely outcomes and figuring out how to make the best of them.
AI winter has arrived.
Said AI "experts" for the last 10 years.
Nobody has said that. The winter will probably come, but it's not at all clear when. First the bubble must burst.
almost every story that has ever been written about a world with AI has posited that humanity was better off without AI.
I think we did good inventing it and it's inevitable. Lets use it well.
modern day Mary Shelley
To be honest, I think it depends on the environment. In the past, if you didn't come out of a good university, your access to knowledge was limited. But these days, in terms of access to knowledge, I think things have clearly improved, whether or not AI is illegally sharing knowledge.
Looking back now, was it really happier before AI existed? I'm not sure. In the past, when I wanted to find out whether I was right or wrong about some piece of knowledge or information, I'd contact several people and, far from getting an answer, I'd often be told not to bother them. But AI, in that respect, is the best teacher I can think of.
I think the times are continuously getting better. It's just that there's a problem of power concentration among the people who handle it. Still, if AI is a tool of that nature, then I suppose that can happen.
But considering what access to knowledge was like before AI came along, I think having AI is better.
Knowledge was already extremely accessible in the age of the Internet, but it required time to read and learn. AI is making that same knowledge applicable without the prerequisite of learning.
Most people are not interesting in learning for learning's sake, they are motivated to learn something they can apply (see: kids complaining about how learning Math in school doesn't transfer to useful skills). Why learn about architecture and technical debt, if you can tell the machine to make it faster?
I don't think you're wrong, but fundamentally we're in an age of information overload, and this may be the new normal. In other words, the environment where you had to know things deeply and become an expert, like in the past, may be different from an environment where you know many things broadly and combine them.
I basically don't think you're wrong. However, books and reading have their information fixed at the point of publication, whereas depending on where you live or your community, the information you get buried in differs. Because I was in an Asian country, it was actually very hard for me to synchronize my programming skills with the latest technology in the US. The frameworks commonly used in my country are stuck at roughly 15 years ago. The time lag between US technology and my mainstream technology is about 5 years. In my area, it's about 15 years.
Anyway, thinking about your opinion, I think there are both valid points and dark points. I think learning for its own sake is important. I agree that people should have an attitude of learning for its own sake, but I don't think it's mandatory. I think it should vary depending on the environment. Still... I do think having that attitude is a good thing. I like your idealism, senior.
However, that holds for people who are middle class or above and can survive. Because rich kids can picture the future and are given the right to love learning. But if you're poor, that's very hard. And when you grow up in such a poor family, there's an emphasis on it having to lead to practical skills.
And even if someone loves learning for its own sake, I think quite a lot of people love the honor that comes from learning. But that too can be called loving learning itself. It's purely a matter of individual values.
I don't think you're wrong, senior, but I also wonder whether the old ways are right in today's era of excessive information. It's just that because our lives and the environments surrounding us are different, we came to different conclusions. I upvoted because I agree with your thoughts. But there are points where I differ. Have a good day.
The internet pre-AI did far more to make knowledge accessible than AI has done. Its useful, but its not the critical thing.
This is a bit like Rudolf Diesel saying maybe we're better off without airplanes. The field has long since left this guy behind.
If the resulting issues are correct it doesn't matter whether the field left the guy behind or not.
He would still be right and the field practioners would be wrong.
So pumping CO2 into the atmosphere is a good thing and if someone does a backflip by recognising their mistakes that is a bad thing?
We should regulate data centers to require fully renewable power (solar/wind/hydro + battery).
This is tangential encouraged by your point and other forays here. But one does wonder what arguments politicians (the power lies with them) understand when they make their pronouncements. The UK has for instance, prior to the new administration, totally and vociferously supported net zero. From the point of view (a fact for many but not for all informed authorities) that climate change is caused by CO2 and we are headed for a disaster - Net Zero is a logical step to trying to save humanity and the whole ecosystem. Now (November) I read that the UK powers that be, appear to have put it aside in actions at least. Countdown to Net Zero doesn't matter any more; open the oil fields, get rid of subsidies for photovoltaic and wind - and other measures. My point is not a political jibe but a stance I suspect to apply in other areas. Is climate change going to be a catastrophe or not? They should make up their minds.
I guess capsule network haven't improved ...
I actually laughed out loud.
I do find capsules interesting though.
What is the point of this quote? Or any of his quotes for that matter.
We should introduce regulation so only credentialed software engineers are allowed to use AI.
I would rather see the onus on the company owners to be responsible for what they sell. Serious penalties for seriously bad products.
Absolutely! And software engineer should be an hereditary title as well.
In East Asian countries, certification systems are still maintained, but in reality they're practically meaningless. If a certification system exists, people will take an interest in CS and work harder at it, or so you'd think—but in reality, the exam questions end up stuck in the old style. In other words, there's a big gap between the exams and the actual field.
Because a certification has to be fixed at a particular point in time. Korea's certification exams start with the 23 GoF patterns and go on to rote memorization of countless patterns and layered terminology.
The problem is that the way you learn for this certification is the old way, so someone who gets the certification ends up writing legacy code.
But if you always update the questions to the latest, then nobody can pass the certification. So, having seen how the certification system works in East Asia, I don't think it's all that good
It might turn out we were better off without nuclear weapons, but so far I’m happy we haven’t had another world war (in the traditional “total war” sense).
No way to tell until after, though. So far, mutually assured destruction is working. It works until it doesn’t.
Same with AI. Thus far we’re better off with it than without it. That may change.
I mean, look what we did with fossil fuels. Some people are even still willingly burning them, even though the data is clear that our use of them has already started the total destruction of our civilization.
Is there a name for this, a technology that is so insanely useful that there is no way we won’t exploit it, but where the worst case scenario is total collapse, and impossible to determine well in advance?
(Fossil fuels I suppose don’t count as even the energy companies knew unambiguously decades ago that large scale use would eventually bake the entire agricultural system out of existence.)
World war 3 is already underway, and it is predominantly a fifth generation war. A war for people's minds. In such a theatre AI may as well be a nuke.
>so far I’m happy we haven’t had another world war
The reason for that was more that the 3-4 great empires involved in the previous two were exhausted, Japan was defeated, China was in no shape, and US vs USSR had no reason to push for one, since the Cold War era gave them plenty of breathing room, roughly half the world each.
> roughly half the world each.
Both the USSR and US fought multiple wars directly against the proxies of the other. They weren't simply hanging back and letting others do the dirty work. In most any other era one of those would have become a full blown war between the two. Both sides pushed the line of not directly involved and then both sides looked the other way versus escalating. Nukes made the alternative too much of a risk for both sides.
oh word?
The scene is bleak, silent, desert-like with a few dead trees bleached white by unfiltered UV. Nothing moves. In the distance a cave, and from that cave
... no shit Sherlock
9 minutes. Was that enough time to read the article?
My reader mode says 6–7 minutes, so, yes.
TFA could have been two sentences:
He warns that AI might cause serious problems: power failures, job losses, and greater social inequality.
He believes AI might improve our lives, but governments must make sure it is safe.
Did you look at it? It wasn’t that long.
I personally feel reading the article before commenting affects the purity of my vibe before posting
vibe comments!
We should have not left the trees. You cannot disrupt banking and power if they do not exist. Checkmate!
The ghost of Douglas Adams has entered the chat
So why did he start to develop it in the 1970s?
Why did he start a research direction that 50 years later "might turn out" to not be a good idea?
That's a relatively silly question, don't you think?
No it's a completely serious question. The goal has always been to replicate human/biological thinking processes in software, and now that this has been more or less achieved he's suddenly not happy about it. There's been plenty of opportunity to actually think about "what if all of this actually works" but they chose not to.
Sounds like sour grapes. He worked on it for a long time but he doesn't own a trillion dollar company. Missed the boat?
We are not but we need a better approach, https://www.ethicalacceleration.org
> It could help cure diseases, end boring work and give us more free time.
Seriously? It seems to be causing more problems, giving us more boring work, and less free time.... sounds like absolute bullshit.
"Could" does a lot of work in these kinds of claims.
We need to build it as fast as possible because it could cure all disease. Ignore whether that's possible, how it might happen, whether all we're missing to do it is more intelligence, etc.
The only way I see us curing all disease is to throw out all the names we've given to collections of symptoms. What disease? I don't see no disease, you just have a sore throat, runny nose, and mild fever.
> We need to build it as fast as possible because it could cure all disease
Hell no. We need to risk pumping out even more CO2, causing more environmental disaster, ruining entire human communities, just to cure diseases? You might as well say it's better to be in prison because you're guaranteed three meals a day. The logic of some pro-AI people is baffling.
How has AI given you less free time?
I have to spend extra time just trying to find AI-free things, AI-free writing, AI-free communities, and people who aren't addicted to AI. And get noticed/get a job by having to push through all this AI static. And having to interact with AI-based customer service agents that take 10x as long to resolve the problem as a human being. And having to wait a bit longer because there are 10x more 'Are You Human' Captchas slowing down website loading. And having to download OS updates that are 2x as large because they contain AI models. And having to double-check everything because there's so much more hallucinated crap out there.
The other day I hired an accountant and even he sent me an AI answer that sounded right but was actually bullshit.
In short, it takes a LOT more time just to find some sanity now.
AI is shit.
That's the whole point of the site. Did you read it. AI should not cause harm, direct or indirect.
Of course I read it. But my thesis is that one cannot avoid that. Causing harm, whether direct or indirect, is the fundamental nature of AI.
It can be avoided, humans control AI still. Humans are the ones causing harm with the help of humans.
I mean, it could if "we" "just" got out of globalized financial capitalism, changed everything about our political, economical, financial, social systems, etc.
Only 14 years old and accelerationists think we can just push the capitalism pedal further and somehow get social changes going their way
I don't think accelerationists think we can do anything. The thought process of the accelerationist is partly fueled by the 'excitement' of betting and playing musical chairs and hoping to be the first ones out holding a bag of money and partly fueld by the delusion that technological advancement is always good at any level of complexity.
You’re a cultist
How did you come to that conclusion?