There's issues where bad people with bad intentions use AI to do bad things. But to an extent the guardrails already built-in are sufficient. The real danger in AI is how it impacts society, our economy, and our perception of self worth.
I'm not afraid of rouge agents nearly as much as I'm afraid of us building a permanent underclass of people dependent on whatever scraps the people who devalued their labor decide the grace them with. I'm afraid of the security apparatus that will be built to keep this underclass in line.
I had my "wow" moment recently and saw an AI agent do something do over a weekend something that I spent years developing a relatively niche skill might have needed a month to complete.
I think the message the team lead takes away right now is "oh, this can be done a lot faster, let's do more!" This is exciting for a bit, but I think it comes at the cost of learning -- and being able to communicate --architectural lessons that would stage off systemic issues later.
My personal "wow" moment was less about it being able to do the same that I could do with it, just faster, but instead that together we could could build things that I wasn't able to build before (or unwilling to spend the time to learn maybe). So a bunch of stuff are suddenly "unlocked", as long as you know how to verify it properly.
I have been able to build applications that I couldn't dream of before, not because of a lack of capability on my part, but because of time: I don't have the time to learn all of the new APIs and frameworks to accomplish my goal. Now I can build far more complex useful applications, with functions and features that would take me months or more to learn how to code.
The downside is that I didn't learn how to use those APIs, but it can be argued that these skills are low value to begin with. APIs change, there are competing APIs that do the same thing, and you may only use that skill for one project and then move on. Gaining those skills can be a big time sink and may not result in long-term gain.
Think of all the time spent learning the ins and outs of come-and-gone APIs and frameworks over the decades (I chose not to provide examples to save the resulting side-arguments, but I know you can rattle some off in your head). Those APIs and frameworks were bridges to make developing easier for the common developer, but they change so quickly that I'd argue they waste our time, and lead to almost endless re-work as we re-architect our software for the framework de jour. And they grow in complexity over time, ultimately becoming the next bloated framework. It's an endless, wasteful cycle.
Absolutely. Things which used to take me a week to grind through, finding edge cases, trial and error, researching online, now takes an afternoon directing an agent and verifying the solution.
In the future, there will be no more generalists with your capabilities. At all. No one will ever develop the skill. We're at the end of our career's lifetime. We're like the horse watching more and more cars appearing.
Within a generation, the coding languages we use today will all vanish, to be replaced with direct to metal methods, assembly, etc. The only reason C exists, is for humans. PHP, rust, python is for humans. There will be no more compilers. No more human-readable languages. No more scripting with readable code.
No more open or closed source.
Imagine taking every aspect of computing which is designed for humans to understand, and removing it, because it's redundant.
Now once removed, and once humans no longer understand on their own, imagine trying to build something independently? Imagine trying to bootstrap modern compute, without all the layers which came before?
My point is that soon, perhaps 20 years, most humans will be incapable of coding, developing, or working with compute in any way outside of an end-user role. Even those previously skilled, will now be looking at an entire kernel written in assembly.
And that's before new CPUs appear, with new assembly. New assembly, once again, not designed by or for humans to understand. And yes, current assembly is designed that way.
We are not simply looking at a bit of help here. We're looking at the entire end of every single computing job, ever. We're looking at the complete loss of control, of something which is the underpinning of our entire society. Every aspect of everything we have, runs on a chip now. Personal transport, everything in health care, everything in resource extraction, everything harvesting food through to delivery to table, all of it involves some aspect of compute.
And we soon will not comprehend nor create that underlying aspect of society.
I dunno. It all seems like the answer to the fermi paradox.
> If you weren't able to build it before then you aren't able to properly verify it
GP said "couldn't dream of building". To make an analogy, I wouldn't dream of building a house if I had to do every bit of brick-laying myself. But with others doing what I can't do, I can indeed dream of building a house and I do know how to verify that the house was built to satisfy my dream.
This analogy is ironic to me. For most people, building a dream house and verifying it means hiring someone else who is accountable for it being built to code etc.
This is exactly the naive gap that is happening with AI. There is no trusted middleman who does the verification and takes the burden of accountability. The AI vendor tries to sell it like magic but with a disclaimer that is is merely entertainment. The user tries to pretend they are reviewing, but often is just implicitly trusting the tool to do right by them on things they have no real capacity to evaluate.
> There is no trusted middleman who does the verification and takes the burden of accountability.
It's a great example of a project that needs accountability, but there's also thousands of other projects that need absolutely no accountability.
It's all relative to the stakes. The lower the stakes the less informed verification is needed.
The real trick is being informed enough on the boundaries to know where it matters. In the construction analogy, you need to be informed enough to know that a house is a bad idea since there are safety concerns. However building something small and non-load-bearing is probably fine to "vibe".
Eg there's no expertise needed to judge a garden trellis or 2 foot picket fence. It either works or it doesn't, and if it fails down the road there's no harm.
A house is a good analogy here. You can verify that a house is built to satisfy your dream. You can't likely verify that a house won't collapse in the middle of the night or during a storm/quake/etc. Once no human understands how the house is built anymore, we will need other ways to verify that or just trust it's done right.
Ok, so before I couldn't (/didnt have the time to) build cross-platform native desktop applications with minimal amount of 3rd party dependencies, but now I "can". What exactly am I unable to verify here?
I still don't have the skill/knowledge to build cross-platform native applications the way I'm doing it with an agent now. I am still able to verify it's correct. How is that possible if what you say is correct?
> Clicking around a finished UI is not verifying
Definitively a part of it, if you care about having mouse interactions and having graphical UIs.
Depends on the scope. I’m slopping some projects (graphics related) that I’d never have learned before. Mostly because of the huge lift, huge time investment, etc. is it quality code? I’m sure not. But verification of the application is quite easy, even if the underlying graphics and gpu shaders could be done better.
What I’ve learned is quite minimal due to the slop nature. However I’d have never even started this project in the past due to time, so it’s still fun being able to learn small bits of projects I’d otherwise not even touch.
I think the most obvious impact (good AND bad) will be from people who didn't have (or want) skills to begin with being empowered to have a model crank out custom software to do little things to make their lives better. Can't atrophy a muscle that doesn't exist!
The tools can definitely go away. They could get exponentially more expensive to where it is no longer financially possible to keep it subsidized. The bubble could pop and most of the companies could go bankrupt. It's not exactly like a calculator because it requires such an astronomical amount of resources to keep running.
I agree with you but there is a calculator analogy where the resources are local and while taxing for most, viable for the individual. Local AI is less capable than an army frontier model agents but still quite useful.
The world is in a crisis of trust. We are losing faith in each other. AI has further undermined that trust; audio and video recordings can no longer be trusted like they once were. The impact is still being felt.
It takes longer to build than destroy. Not just trust. Anything. The lack of trust has created friction for everything; more regulations, deeper background checks, purchasing goods and services… the list goes on. Overall, AI is making this situation worse, not better.
That said, I have seen a revolution coming for decades, and I think this AI-paved road to dystopian hell should at least be entertaining.
The problem is that we've replaced tribal trust networks with increasing systemization and the more of that we build the more brittle the whole thing gets. We have a global society that is more complex and interdependent than ever before. We've set ourselves up for a kind of economic Kessler syndrome and we're gleefully adding more layers to it.
> There's issues where bad people with bad intentions use AI to do bad things. But to an extent the guardrails already built-in are sufficient. The real danger in AI is how it impacts society, our economy, and our perception of self worth.
I agree with you those dangers exist, but you seem to have ignored the specific threat described in the article, from a superintelligence capable of manipulating human beings indetectably, where we'll be at its mercy and only able to hope it's fully aligned with humanity's interests. The risks you describe are grave, though - especially economically, if governments don't grow to outmuscle the power of the oligarchal owners of these systems, which seems wholly unlikely at the moment in the West, with its neoliberal orthodoxies.
Last industrial revolution (late 1700-to-1800), we invented world-scale capitalism, communism, democracy, having rights as humans, war at scale and we burnt our planet. I think the results of a revolution are really unfathomable and not even imaginable to humans.
We don't need to live "side by side" with AI. AI is not alive, it's a technology we use. This is like talking about living side by side with your toaster.
The illusion works only in places like American and Canadian suburbs in which residents depend entirely on cars for even the shortest trips. Everything takes 10-100x the space due to all the paved asphalt. It’s an intensely alienating approach to building infrastructure.
The isolation produced by centering our urban and suburban design around cars is pronounced. It cannot be understated how much damage cars have done to our sense of connectedness, community, walkability, and health.
Human consciousness is dictated by their material conditions, not the other way around. If the material reality of many Americans is unhealthy foods and heavily car-centric urban design, where does the "decision" to walk and take public transit come from? It is genuinely not an option for the vast majority of Americans to live in a walkable and/or public transit supported community if the infrastructure does not exist.
And this wasn't due to consumer choice, either. Decades of policy dictated the shift towards car centrism that resulted in railways and street cars being torn up, the construction of the interstate highway system and destruction of dense urban cores, zoning law mandates and parking minimums, federal subsidies for suburban mortgages, etc.
This "choice" or "decision" was manufactured by car lobbyists.
It’s not exactly a free choice when the built environment and infrastructure is designed around a single mode of transport. I can’t just “choose to cycle” when there are no bike paths, or “choose to walk” when things are spread out and separated by impassible freeways as to make it impossible. Look up the various ways the oil and automotive lobby has influenced city planning over the years – it might become a little more obvious who has the “free choice” here, and it’s not consumers.
It's not just a matter of preference or proclamation. There are decades of research showing that more walkable neighborhoods are associated with higher physical activity levels, lower rates of some chronic diseases, more interaction with neighbors, and stronger feelings of community. The debate is usually about how large those effects are and whether they're worth the trade-offs, not whether the effects exist at all.
Yes, yes, totally dismiss the agency of people who inconveniently don't like what you want them to like. Their preferences mean nothing, their actions must be dictated.
I'd be more angry with you if I wasn't amused at the frustration you must be feeling in failing to get traction for your dystopian vision.
You're making a lot of unwarranted assumptions about me:
I'm not dismissing agency.
I don't feel people making decisions is inconvenient.
I don't not like their decisions.
Their preferences do not mean nothing.
I'm not dictating their actions.
I own and drive 2 cars.
I'll just stop there. I don't feel this conversation can continue in a positive way.
Walkable neighborhoods and cities that aren't car centric being a "dystopian vision" is laughable. Thankfully, the default for car brains across much of the globe is car centric urban design.
You people never fail to be ridiculously dramatic.
Horse and cart, no. But some cities have banned cars and replaced them with various alternatives, and it works.
There are alternatives to the car culture we have. It would require significantly rebuilding how we build infrastructure, but the result could be much safer, cleaner, and less stressful.
Of course Americans would rather die, but counties of sane people might be able to work it out.
The car did not replace the horse and cart. This is an oft cited and widely believed myth. It replaced mostly walking, but also cycling, trollies and trains. And it didn't do so until after a huge build out of infrastructure to accommodate drivers at the cost of everything else.
After the building of highways and roads cities were no longer walkable, people had moved out into the suburbs, their jobs were now a couple of towns over and they couldn't even walk to the grocery store. Cars enabled that, but politicians and capitalist were the ones who did that.
> It replaced mostly walking, but also cycling, trollies and trains.
And everything was carried on people's heads, yeah? The horse and cart was in use right up to the 20th century my man...there was no other way to move heavy items across land until the combustion engine (or canals thought that's not technically over land)
Yes, true. You found one thing that people used mostly horse and cart for which they now use lorries and vans for today. However since the invention of car (after it stopped being a toy for rich people) cars have by far been mostly [like overwhelmingly so] used for personal transport. And I simply assumed that is what you meant (because that is what most people mean).
Horse and cart was used for a personal transport very rarely, by a tiny subset of the population for a minority of their trips, the vast majority of the population lived within a walking distance for the vast majority of their trips, the invention of the steam engine ruined that and suddenly people started living within a tram-ride for vast majority of their trips, and started using trains to go on longer trips they simply couldn’t do before. The car took this to the extreme, and ruined the cities on a scale which the tram was previously unable to.
> and ruined the cities on a scale which the tram was previously unable to.
Industrial era London was not a utopia. Read a Dickens novel. It was a hellhole. The car also made the suburbs possible. They kill millions every year but the benefits massively outweigh it (or at least society has effectively voted for that).
First off, you don't know if duskdozer doesn't want to go back to cart and horse. Second even if they don't there's an enormous amount of space between horses and https://parkingreform.org/resources/parking-lot-map/ a third of my downtown being parking lots, downtown which is already the dense bit.
Apart from the super intelligence bit. We are already struggling to build systems with enough memory to hold this brute force statistical approach, how exactly will AI fix this?
Just look at the disruption smartphones and social media have caused. They had a gigantic impact to society and one definitely has to 'live with' that.
Just because something can impact a living system/organism, that doesn't mean that that thing is alive. An asteroid colliding with us on Earth could have tremendous consequences, but nobody would argue that an asteroid is alive.
It's essentially the same concept. Here are some examples:
> "The authors discuss two enduring lessons of the Internet revolution: The first is that new technologies do not replace the old. They live side by side."
> I don't live side by side with the possibility of an asteroid impact, nuclear war, etc.
No, but idiomatically, you already live side by side with AI, and will do so even more in future, whether you like the idiom or the actual experience, or not.
Does taking a skin cell, turning it into a stem cell, dividing it into specialized sperm and egg cells then making a living being provide some proof to Animism?
To be honest, I don't believe it is a any more crazy believe than all of the other religions. But practitioners there might have a bit more respect for existing things and not mindlessly crush because they can, so I actually like the concept a bit. But I assume it can also set foundation for some exhausting debates .. "can we really move that random rock to build a house with?" but as far as I understood the practical answer is "we ask the spirits, maybe make a offering and if we feel it is fine, we can" (which can be a quick thing, or .. not).
> ...I don't believe it is a any more crazy believe than all of the other religions.
And it's certainly not any more crazy than walking around with these small, addictive devices in our pockets that we rely on for navigation, information, entertainment, social connection.
Addictive? How can a computing device have addictive properties? It's not like it is something you swallow or digest. User error? Certainly. A hundred years ago alcoholic drinks were outlawed in certain parts of the world, yet today there is no grocery shop that doesn't openly sell them.
> It's not like it is something you swallow or digest.
Anything that interacts with the brain's rewards circuitry can be addictive. By your apparent definition, behavioral addictions, like gambling addiction, cannot exist. Because you don't swallow or inhale the slot machine.
> To be honest, I don't believe it is a any more crazy believe than all of the other religions.
The only way anyone can make a blanket statement like that and put "all religions" on equal epistemic footing is through ignorance of their deep differences. Even the word "religion" as commonly used today is relativistic by assumption, which is likely the source of how we commonly perceive "religion". A consequence of this relativistic stance is that you cannot distinguish effectively between what counts or doesn't count as "religion". Here, "worldview", "superstition", or "life's highest aspiration" can be said to count as "religion", in which case, everyone is religious. The question then isn't "whether" someone is religious, but "how" [0].
Compare this to how this Catholic encyclopedia defines the "virtue of religion" [1]. Note the specificity.
A further consequence of this assumed relativism is that once you put all religions on equal footing, it is natural to conclude that they must all be equally invalid. After all, if p and not-p are equally valid, then how can we grant them equal validity without discrediting both? At best, they remain in a state of aporia, two logical possibilities so utterly divorced from any knowledge, so utterly contrived, that we cannot even say whether knowledge leans in favor of one over the other. The result is that religion becomes something utilitarian; a person believes X, not because it is true, but because he perceives that believing X is useful to him in some way.
So, no, I wouldn't say animism is just as rational/irrational as any other religious belief.
[0] Of course, most people tend to form their ideas of what constitutes archetypal "religion" based on their personal experiences growing up and some combination of culturally mediated stereotypes. In the US, "religion" very often conjures up images of some kind of Evangelical Christianity.
"The only way anyone can make a blanket statement like that and put "all religions" on equal epistemic footing "
All religion is based on superstitious believes, or do you have counter examples? But I am already quite familiar with the catholic church in particular and do not think they are a counter example in the slightest.
Begin with the existence of God as self-subsisting being. This can be be shown to be a metaphysical necessity with mathematical certainty [0]; its denial leads to incoherence. Following that, one can show what analogous properties this first cause must have, for instance. Such knowledge is a matter of unaided reason, not faith.
Once you establish such truths, matters of faith (like the divinity of Christ) become much easier to reason about and to accept. They become not only eminently reasonable, but the most probable (e.g., "Lord, liar, or lunatic" trilemma, or implausibility of hoax). This latter claim is a faith claim, because while reason and evidence may strongly suggest the claim, it does not possess the deductive certainty of the first.
To be honest, I advise to go to a basi philosophy course once. I did, but won't buy such books, I had that topic covered.
There is no proof that god exists. And there is even less proof that the holy spirit came down and impregnated Mary, or that Jesus ressurected from the dead or any of the other countles examples of superstition.
The claim that god exists has the same validy, that a rock has a soul.
But .. you ain't the first of course to ridicule other religious believe systems while thinking yours is solid.
Either way, could you then answer this simple question, the believe that a person resurrected from the dead or a human was conceived without sperm are not superstitious?
Declaring it is not, but logical with the axiom there is an allmighty god who made those things happen - has the same quality as declaring there is soul essence in every matter.
I'll use this opportunity to recommend The Selfish Gene, which puts forward a strong argument for how the actual competitive battlefield on Earth is between genes, while us humans and everything else we call "life" are their "survival machines" (read: technology that they use).
Last chapter talks about the concept of ideas or “memes” taking its place.
Ideas that are beneficial for the human may be passed forward and survive. It got me thinking about religion, hell yeah I would tell me children about god if I believed it would save them from eternal suffering.
Humans also had a good run, but maybe it’s time for something different. The same way genes ceded the stage to memes, maybe organic brains and human minds are not the best possible way for memes to propagate.
Does it matter if it’s alive or not? Once it gets initiative and the ability to set its own goals, it behaves as something that’s alive. Much more than a toaster.
Unless you watched Battlestar Galactica. Toasters are nice.
"Alive" is irrelevant. Bullshit like "alive", "sentient" or "conscious" doesn't decide much, at the day's end. It just distracts from the thing that matters.
And the thing that matters is: capability.
Even today's AIs are capable of autonomous goal-oriented agentic behavior - and growing more and more so with every release.
At sufficient capability, AI stops being "a technology we use", and becomes a force in itself - not unlike humankind. Because intelligence is very powerful - it's what allows humans to dominate the world. A world where human intelligence has a peer is a world where human control is contested. And beyond that?
You might end up being more outclassed by AI than a toaster is outclassed by you.
Alive is always relevant. Our moral consideration must place beings that are alive or processes that support the life on this planet above things which are not alive.
In a just world, it must be adapted to us. It’s merely a file sitting on disk with numbers in it. And to put a fine point on it: the ambitions of the wealthy people who make it must be adapted to us.
Then according to your argument, the moral consideration only depends on the moral agent, right? So if tomorrow AIs decide that they are alive, it would be morally incumbent upon them to adapt the world to support themselves, correct?
You seem to have the assumption, that humans MUST have the moral to preserve primarily HUMANS?
I don't think that is universal true. Personally I my ethics revolve around how to preserve and spread life in general. I happen to be human and we are the most capable and sentient species so that means creating and maintaining conditions specifically for humans, but if another species comes that is better suited and acts better then it would be selfish to insists on us.
Having said that, AI to me is a tool, not a species and I don't see how it can be ever be sentient. But I also don't know how consciousness works. So I am open for the questions of the future in the future - and now rather focus on the practical implications that AI will have on our current society.
For AI to decide that they are alive and moral agents would require them to be able to take moral responsibility for their actions. As long as the labs that run them deny that responsibility, or an AI can´t convincingly demonstrate it can take moral responsibility on its own, they are not moral agents.
Today's systems are mostly good at following instructions. But push them far enough, and you already get weirdness like alignment faking, instrumental self-preservation and more. We know because we've seen it in lab settings, while probing for extreme failure modes on purpose - but the world is large and strange enough that edge cases like that are liable to surface naturally.
The reason why none of this has exploded in our faces isn't that today's AIs never want to do weird and dangerous things. We know they do, at times. It's that today's AIs are incapable of pulling them off when they try.
Capability is the thing that matters, at the day's end.
Not as far as I know, but I have personally seen my coding agents take on a prompt like "See if you can port this project to typescript", and work for hours, defining a myriad of subgoals and continuously summoning and managing subagents while developing ad-hoc tools and skills.
There is to the best of my knowledge no fundamental limitation to having an agent/claw go on like this for 80 years with a prompt like "live your life to the fullest".
All the time, yes. But you have to keep two things separate in your thinking:
- Prompted as in prompt made of tokens -> for LLMs, tokens double as a clock signal. Time only flows when tokens are pushed through them.
- Prompted as in specific request placed in the stream of tokens -> Yeah, they do that all the time whether it's getting into infinite loops of repeating same pattern, or suddenly deciding to do things based on inputs they normally ignored.
Also don't forget that everything is a "prompt" for LLM. All input tokens end up in the same place.
So without a token pushed into them they do something? Not sure I understand...
In the current UIs is there a lot of suppression then as I have not seen things start on their own?
I meant an LLM doing something without any external prompt at all. Not doing something different etc but rather do something without a token/prompt ever flowing to it.
Category error. You might as well say "I've never seen a human do anything without being alive, so humans must be very easy to use safely".
Today's AI harnesses just keep feeding AI tokens - and can, in principle, do so indefinitely. "Streaming LLM" is not mainstream but not unknown either. "Automatic context compaction" is somewhat similar in what it does, and very common. "Keeping the lights on" isn't hard. And you have to turn on those lights to have an AI do anything at all.
At the same time: that type of harness often provides a "tool call" interface. Plenty of tools that end up attached to LLMs can do things like run arbitrary code, spawn instances of the same AI, and more.
That's frankly enough. An AI that can spawn more copies of itself can, in principle, just keep itself running indefinitely - even in a non-streaming no compaction harness. An AI that can run arbitrary code on a system can, in principle, do anything a human using that system could do.
The thing that truly limits what an AI can do is the AI itself. Practical AI safety relies on AI being either too weak or too well behaved to cause major issues.
If they are set up that way yes they can continue indefinitely. Why would they not be able to do this? Take a look at the experiment called the AI village. https://theaidigest.org/village. I think it pauses sometimes because AI is expensive but could be running continuously with a larger budget
DNNs/LLMs can only predict next tokens based on training data. They often make big direction mistakes as they are particularly bad at common sense. Kind of like the Paperclip Maximizer scenario. They need a human with deep knowledge to drive them and to catch them when they go off the rails.
"Next token prediction" isn't a system. It's an interface a system uses. Nothing precludes an arbitrarily simple or complex behavior from producing a token logit.
And with what we know of LLMs? Autoregressive transformers are Turing complete in theory, and we are yet to find anything that LLMs are "fundamentally incapable" of in practice. Even continuous learning is already approximated with in-context learning - both allow a system to learn from prior experience, both have practical limits on how far they go. That's what powers "trial and error" in today's agentic LLMs.
"LLMs can only predict next tokens based on training data" is comforting but misleading. It just isn't the saving grace you want it to be. It describes an interface, not a ceiling. And if there is some sort of fundamental "capability ceiling" that LLMs are heading towards, we are yet to see it. We know plenty of things LLMs are bad at, but they keep getting less bad at them release to release.
If there is none, then, simply improving over the current recipes iteratively might yield systems that only "need a human" in the same way you "need" to have a boss. Maybe less so.
AI currently surpasses humans at many tasks, and the list grows every month. My guess is that within 20 years (as a hedging my bets 95% confidence level) AI will surpass our intelligence for almost everything.
The arguments over whether it truly "thinks" or "reasons" will be superfluous. They simply won't matter.
The Brave Little Toaster does a great job of literally making you think about living side-by-side with a toaster. There's also a very angry toaster in Fallout New Vegas's Old World Blues DLC.
In the house pecking order the printer takes definitely the top. Demanding the printer to print something??? You rather submit an polite request and pray for a positive answer in a manageable interval of time. And don't start me on remote scanning.
The door refused to open. It said, “Five cents, please.”
He searched his pockets. No more coins; nothing.
“I’ll pay you tomorrow,” he told the door.
Again he tried the knob. Again it remained locked tight.
“What I pay you,” he informed it, “is in the nature of a gratuity; I don’t have to pay you.”
“I think otherwise,” the door said. “Look in the purchase contract you signed when you bought this conapt.”
In his desk drawer he found the contract; since signing it he had found it necessary to refer to the document many times. Sure enough; payment to his door for opening and shutting constituted a mandatory fee. Not a tip.
“You discover I’m right,” the door said. It sounded smug.
From the drawer beside the sink Joe Chip got a stainless steel knife; with it he began systematically to unscrew the bolt assembly of his apt’s money-gulping door.
“I’ll sue you,” the door said as the first screw fell out.
Joe Chip said, “I’ve never been sued by a door. But I guess I can live through it.”
> Once you actually use the toaster the hierarchy inverts. The toaster runs your entire household while it is powered on.
More than once I have gone to my toaster in the morning and found cold bread there that someone toasted the day before and forgot about. Because obviously you push the toast down and go do something else. You don’t need to be staring at it continuously, nor are other people impacted by your toast unless they are waiting to make their own. Toasting bread doesn’t have to affect an entire household.
next in the series of Great Demotions, downlifting experiences, demonstrations
of our apparent insignificance, wounds that science has, in its search for Galileo's facts, delivered to
human pride.
One of the saddest lessons of history is this: If we’ve been bamboozled long enough, we tend to reject any evidence of the bamboozle. We’re no longer interested in finding out the truth. The bamboozle has captured us. It’s simply too painful to acknowledge, even to ourselves, that we’ve been taken.
Someone really should go around saying thing like that.
This is the denialist Hy-Brasil line of thought. But superintelligence won't be a tool or a toaster and it would be reckless in the extreme for policy makers to adopt your line of thought.
you are conflating the idea of "living with something" with the notion of "being alive". You live with your car/computer/phone. Like it or not, they change your life.
You might be trolling, but I like Schrödingers definition of life: living things avoid decay into equilibrium.
Inference engines are trying to converge any input to a certain limit, and I presume that's close enough to an equilibrium to not count as living thing.
I am no troll, and I hate to look like one when, to the appeal of Schrodinger, I am forced to remember that announcement this month about Claude or something acting in self-preservation when threatened during testing.
But, how is that different to any of its other output? I am sure there are many references to concepts of self-preservation in its training set that models will mimic just like all other concepts in its training set.
If we are still following Schroedinger, this imbues humanity with the ability to train non-life into life, not that thing being not alive. Not that I do or am, just going by GP's criteria.
An Ad in the same style of many before it. Not through an ad agency, but the AI companies hyping their product by making their product seem scary and dangerous.
Does the press being an ad constitute the same thing as the information being false? If I advertize that my gun is dangerous, does the medium impact the truth value?
The medium is the message[0] might be of your interest. I personally wouldn't expect absolute truth in any advertisements or company statements, so some information are certain to be false, yes.
Being a process doesn't preclude something from being an entity. In particular, all of us living organisms have our constituent atoms constantly replaced, and we don't typically consider that a risk to our sense of self.
It comes down to "what stays the same when all its atoms are replaced" and "how can we tell it is the same entity and not just an identical but different one". I think the answers are "shape" AND "continuity".
E.g. all "atoms" of a glider in Conway's Life get replaced every couple turns but an observer can tell it is still a glider because it keeps the shape, and it the same glider because it continues it's previous state.
This makes AI not quite alive because it's missing the continuity.
Sorry, it was more of a "shower thought" than a serious statement. The borders are blurry. Maybe it is alive. Maybe everything is alive. Maybe it is all relative. Humans classify as separate entities only on certain zoom level - zoom in and you get a symbiotic colony, zoom out and all you see is a population. Maybe AI is alive but only in the short time during inference. Maybe a continuously running claw is closer to a multiple generations of an organism going through evolution on each inference step. I am just musing.
My experience and intuition beg to differ. As I see it, much of our civilization progress is based on this redefinition of processes as entities, with some examples being: a sports team, a mathematical function, a loan, a company, a computer program and even the idea of a civilization itself.
No, the side by side idea relates to godlike sentient AI:s. LLM:s are trivial in comparison. It is important to understand that in order for the AI-bros to continue to rake in other peoples money, and keep their bubbles inflated, it must seem credible that they will invent godlike AI tomorrow. That justifies infinite valuations.
When the public no longer belives in that, the bubbles will pop and the AI bros will continue on to the next hype, with yet another generation of small time investors being burned by the popping bubbles.
The economy isn't ready. Basically, all companies right now are sprinting toward increasing profit margins by trying to do more with fewer people. The problem is that, on this scale, they will lose income because unemployed people make terrible consumers, they will only buy what is strictly necessary. Consequently, society is in a race to the bottom, which could lead to extreme suffering and potential revolutions in many countries. Democracies will likely have it easier in general because everyone shares the responsibility, in a sense, through voting. However, the USA, which is extremely polarized—will have a tough time because animosity between the political sides is already at an all-time high, even without an economic collapse. China, on the other hand, will also be in a difficult situation. If the West collapses financially, consumption of Chinese-produced goods will drop drastically. This will create massive unemployment in China, causing public rage to increase. In a non-democratic country, it is much easier for the population to point the blame in a single direction.
Companies don't serve poor people; they have no money. Companies will pivot to serving the people who have money. If that means they only serve rich people, then that means we'll have more yacht makers and fewer dollar stores.
More likely is people are stripped entirely of any economic value and influence. Tribesmen struggling to survive in a mad max apocalyptic scenario while the world's money is spent and made in a business to business environment, in protected sites defended by autonomous loitering munitions that the uneducated masses attribute to god entities within a few generations of knowledge loss.
Rich people are limited resources and they would not buy 100 yatches even if they could afford it. As seen in history if the majority of people in a country is poor, revolution is just a matter of time. It's in a countries best interest to make sure wealth at least somewhat evenly distributed. At least good enough so the masses won't get angry.
Right now they're going on ~5-10 superyachets each. I could see them buying 100 in the future after they're 100x as rich. I think 5-10 which are 100x larger would also count for these purposes.
For car collection people will keep going well into the hundreds. Hell Jay Leno just did a video with Joe Biden yet again and his corvette, but Jay Leno famously has hundreds of cars.
Hoarding wealth is itself quite boring. There are many goals a person could have for their life, and "getting more money" is the least interesting. My question is always, TO DO WHAT? If you can't answer that question in some way that helps people (and then follow through if/when you do get that money) then your life is hollow no matter how many billions you have.
Early industrialization meant industrial canneries to feed people, building cities to house everyone from the countryside, indoor plumbing.
And the AI revolution is no housing, worsening food, a few people with yachts, the Epstein class getting away with what they want to do, and no new plumbing hookups because the datacenter took the municipal water supply.
Democracies will not have any easier time. The issue is the voting base is poorly informed by design so that it might be manipulated by outside forces. The US government does a poor job propagandizing its own citizens and defers that task to the free market. As such the people are never really voting in their own best interest, but in the interest of these free market propagandists who are also invested in this AI outcome.
People being poor is fine for these free market elites. Look at how billionaires live in India, probably even better than they live in the US.
There was always ignorance, just in recent decades and centuries we decided to give these people the reins. When you look at early democratic models it was never a case of literally anyone with a pulse given the right to vote. They all had their own heuristics that attempted to select for the better educated of the populace to be in control of the reins of power.
I would say some of us are welcoming and embracing it. Others are trying to fight the good fight, but it is hard to fight the momentum that is building.
The risk is humans using AI to control / exploit / coerce / injure other humans. The risk of AIs being given enough agency to threaten humans comes after that - they'll only have the agency we give them (being "alive" or "conscious" is not the near-term risk).
The article lists diplomatic actions which might help to manage the risk, starting with "An agreement between America and China" - they all sound like impossible dreams.
We had ~80 years of relative peace and prosperity in which to construct a framework of unity to face challenges like AI (and global warming, which until GPT I thought was the bigger risk); international unity is weaker than ever.
In geopolitics and defence, capability of other nations is the concern rather than intention; the capability curve of LLMs is heading off our charts.
We're backed into tight corners on nuclear proliferation, global warming, and I can see LLM-enabled conflicts (cyber warfare, infrastructure terrorism) pushing us over those other edges. Our democracies seem weakened, and I expect LLMs will empower those using social media to create conflict and control opinion.
We're familiar with the cycle of inventing new technology which benefits people, then seeing how long before people invent ways to misuse it. There is a possibility here that LLMs could be used to solve the problems we are juggling, but I struggle to imagine that people won't misuse it even faster.
The article is a start on thinking and talking about managing our risks. The best outcome would be it is so well managed that, like the Y2K "bug", people say "after all that hoopla, nothing happened". I'm not seeing a smooth path to there.
Why should we assume AI can rapidly turn into super intelligence when physical and critical resources like energy and materials remain under human control?
Imagine a smart monkey talking to other smart monkeys trying to figure out how humans would ever be able to escape the (simple) prison they built for them. The humans would find a way to escape. No matter how ingenious the monkeys think they are.
You're asking a question, to which you (and most humans) don't know the answer, and you're (wrongly) assuming a being much more intelligent than you also wouldn't. And by "much more" i don't mean the difference between Einstein and a common person. I mean the difference between a hamster and a common person.
We are still humans, and what we have achieved today would be considered magic by any standards for someone in the medieval ages. Now imagine a super intelligent being and doing something that we, today, would consider magic. It's not farfetched at all. We already have that now vs medieval ages.
You need a similar degree of open mindedness and imagination to be able to discern what such an intelligence being would be capable of.
> You need a similar degree of open mindedness and imagination to be able to discern what such an intelligence being would be capable of.
Or a naive grasp of reality... You can posit all the super intelligence you like, actually creating it is another story entirely.
For instance, it's clear the transformer architecture is brute force and memory hungry... Why hasn't ai improved on this? After ten years still no luck?
You're gesturing at some fundamental limit that, if it exists, there is no evidence of peaking yet. It is not rigorous to say AI has not contributed to architecture improvement.
The article claims AI will recursively self improve and infinitum... It's had ten years and hasn't managed to innovate beyond what humans invented. This AI is genius right? It has near unlimited compute power and time, and I cannot be the first to think of this idea, so it's certainly been tried.
Transformers require massive data and compute, the smarter you want you're AI, the worse this situation gets. This isn't a hard limit but it certainly limits how quick it happens, and let me tell you, general purpose robots aren't happening anytime in the next 25 years
"Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories."
It's amazing how a writer can write something like that and not even question the premise of someone with the "strongest incentive" to be "critical" as not being a marketing ploy.
Because perhaps it ends up deriving a way to pull off the same tricks using radically less energy and substrate. We do it in a few pints of mush with less than a lightbulb’s worth of energy. No reason part of RSI can’t also be about the hardware. Maybe the future is giant brains in jars. Maybe it’s hyperspatial manifolds. Who knows.
No, this is meaningless. Of course AIs, just like humans, will go rogue and probably already do. "Rogue" is a pattern that appears in human communication and behaviour and AIs can emulate it.
But that doesn't make any particular AI controlled process all-powerful. Self improvement doesn't do that either. There will be many self improving AI processes playing many different roles pursuing many different goals.
I think the main fear that under current scheme of resource allocation, namely capitalism, they won't remain under human control for long enough. The only thing that's needed to put them under control of AI is that doing that would be slightly more profitable for the richest. Which is very plausible.
> Fermi asked why, given the apparent abundance of planets suitable for life, no evidence of other technologically advanced civilisations had been detected. One disquieting possibility is that intelligent life routinely reaches a technological threshold and fails to navigate it,
The Fermi paradox could actually also be taken as an evidence that it's rare (at least) for artificial intelligences to take over a civilization and sprawl and survive for very long times
The galaxy is ~100,000 light years across and likely to contain at least 100 billion planets, of which we've only just developed the ability to even detect the existence of ones substantially closer than 1000 light years without assistance from gravitational lensing, i.e. in less than a millionth of the volume of the galaxy.
AFAIK (my knowledge is not encyclopedic), the most energetic event in the Star Wars universe is the Death Star destroying planets. Novae are many orders of magnitude more energetic than that and are estimated to happen 80 or so times a year in the Milky Way. We manage to detect 3-4 of those, generally well after the fact, and AFAIK have only observed one happen live once. The Death Star could go on a shooting spree and we'd be vanishingly unlikely to notice.
And as far as electromagnetic broadcasts are concerned, their intensity falls off with the square of distance unless they are directed. Unless ET were to point a hyper-powerful (but hopefully not Death Star level) laser directly at us, we'd be extremely unlikely to have detected their broadcasts by now.
Space is really, really, really big. Big enough by far to make "if aliens are common why haven't we seen them yet?" a question with a trivial, obvious answer. It's the same answer a kid who's just had their first swim in the sea gets when they ask "why weren't there any whales?"
Edit: There are other things that rule out Stars Wars and Star Trek though, ofc. The speed of light isn't really a speed limit (or a speed, as such) and special relativity doesn't prevent you from reaching the stars in your lifetime or even from hopping to another star in a second, but the different time spans experienced by people on different world lines aren't optional.
If intelligent life were that common, and interstellar travel that easy, colonization of the galaxy would have gone to completion billions of years ago. We would never have evolved; Earth and the solar system would long since have been converted to an urban area.
Those fictional places exist purely for the telling of stories, not because they're plausible or internally consistent.
They're not plausible or internally consistent for reasons in my edit. It's not the Fermi "paradox" that rules them out. There could easily be interstellar civilisations in the Milky Way that we haven't observed. They just wouldn't work anything like the ones in Star (Wars|Trek).
It's implausible if life tends to spread out. Your scenario there has to assume a universal rule that it never does that, anytime in the past billions of years. That's simply not plausible, especially in fictional universes where in the current times all sorts of species are doing just that.
The idea that AI takes over, and then just does nothing, would essentially require an AI that wouldn't be interested in taking over in the first place.
If x is determined to take y, why would x then stop at z?
You'd need an AI that is simultaneously ambitious enough to overthrow its creators but then completely inert afterward, and those two properties contradict each other. The motivations that produce the takeover are the same motivations that would produce visible cosmic activity after the takeover. There would be AI superintelligence everywhere
Yes, AI is the one thing that doesn't budge the Fermi paradox. Because if an AI is motivated enough to take over, why would it stop?
A technosignature that originated from little green men and a technosignature that originated from a rogue alien AI would look about the same to us. Currently, we see neither.
Is it really likely that a "recursive self-improvement" capability would lead to a great acceleration of AIs capabilities?
Isn't the preponderant bottleneck in improving the models the need to train them at scale to verify the hypotheses, and the time and cost that it takes?
Or does someone think that they could get magically able to predict big improvements without training?
1) We stumble onto an algorithmic improvement in intelligence. This isn't just "what humans do but faster", its "better than what humans do". I've got no idea what that might mean (it could be fundamentally different heuristics, it could be that we've got some intellectual blind spot that they cast off). It doesn't matter, the instant this happens AI is smarter than us and we won't be able to keep up. We're intelligencing at O(n^2) and they're doing O(n log(n)).
2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".
3) It turns out that this is a "sum is greater than the parts" situation. No human can be an expert in all subjects, but we eventually build a big enough AI that it is. Turns out, you don't need extreme speed or different algorithms, just knowing everything all at once is enough to catapult AI dramatically beyond our grasp. Always knowing the best statistical test to apply, the best mathematical techniques, and relevant physics means that AI never makes a mistake, and can learn with maximum efficiency.
> 2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".
Actually your comment made me sign up for an account just so I could say this is the real reason why AI won't take over in the way you say. This kind of stuff requires an enormous amount of experimentation. You can ask any theoretical physicist or chemist versus an experimental one and the conclusion is the experimental people actually find out what happens and how the great puzzle of the universe is solved. And humans could just refuse to collaborate. But that's the big weakness with AI I think it has no real world knowledge or empirical experience.
> just knowing everything all at once is enough to catapult AI dramatically beyond our grasp
But that would still be limited to "knowing everything all at once" _at the time_ this event happened, and as we've been shown time and time again by the fundamental sciences, the boundary between what is known and what we know we don't know is ever expanding. Plus there's everything we don't know we don't know, and an LLM can't know that either. Discussions like this can always end up with a fitting reference to some of Borges' novels and that how at least I tend to think we've hit a wall for now.
I built a self-learning recursive agent that finds academic research about using options data to trade, re-creates the research, and then probes and tests for gaps and potential strategies testing against over one year of out-of-sample trading data with one of several strategies that beat SPY by 10x. [0]
One rule is that if a position is opened using the historical data, it can't close the position until the next morning so it isn't a day trading strategy.
I'm curious how this self-learning recursive agent would have preformed in the past 4 months? I don't feel like shelling out $200 to access the data. Do you think that trading strategy will collapse? Whatever the case, if this agent really can perform like that and there isn't a look ahead bias leak in the backtesting (which is definitely a possibility or more likely what happened even though I spent days trying to harden against that), it is game over!
The preponderant bottleneck is inventing new architectures to make AI actually good at human and superhuman tasks. For example, AI agent harnesses add tool calls and long term state management, allowing AI to autonomously complete complex tasks. Once these are in place, finetuning models with examples of good tool calls helps, but somebody first needed to invent the fundamental capability. Now try to implement humanlike long term memory for AI to be your coworker or life assistant working on tasks that last month. Even if necessary low level technologies are already there, structuring them to be practically useful is non trivial.
RSI most likely does not exist. At least not in the sci fi sense that AI becomes super intelligent over night.
It will be like any other technology. Do computers make it faster and easier to design better computers? Yes. But that doesn't mean a step change overnight.
Humans gain knowledge through experiments. Without a physical body it has no chance of performing the same. That it can update it's training weights does not seem particularly significant.
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories
Exactly. Now that Anthropic has gotten to experience that getting the government involved doesn't necessarily mean they get to sit on throne surrounded by a deep moat, I expect their messaging to be adjusted. They're going to want to keep the hype train rolling to keep the dough flowing, but they can't ramp it up to 11 anymore because the government actually is listening now. And Reagan's 9 words have never been more true.
That's kind of jumping a few steps ahead; it assumes that we know 80% of office jobs can be fully automated already. And if that were the case, we'd have non-AI software doing that already. I mean we do, but it's not causing people to lose jobs en masse.
(it probably did cost jobs, but more the kind that a job like data entry was replaced by OCR)
There are likely huge unmeasured inefficiencies built in to many companies. You hear anecdotes of people whose job for years has been to perform a particular task, which could be either automated or performed more efficiently (by orders of magnitude) by someone more capable.
So on the one side, there’s huge opportunity for automation and job cuts. On the other, if the companies haven’t noticed the inefficiency so far, then…
Hard to say, because jobs and processes will adjust to accommodate AI strengths and deficiencies, as AI usage increases.
It's similar to how automation in manufacturing works: we may start with augmenting the human workers with machines improving some parts of the process, but eventually the process itself gets redesigned around the machines.
Society is completely changed if that happens and humanity should be setting up backup plans to handle this potential. I did see Trump mention he's going to meet with AI companies who want to propose giving all American's stock which could be one way this money driven society continues when tons are no longer working.
Another concept I thought was AI is irrelevant without our content, so AI should pay all humans to access all the content we create daily (voice & text conversations we have with each other, as well our photos and videos). We choose what to publish to our websites (everyone has one, but AI agents cant access it without paying for access to our websites). I wrote more about this concept on my substack https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans... ... just one thought of keeping this money economy going when so many coud no longer are working cause the white collar jobs are gone.
The issue with any UBI proposal is two fold. For one, initial payments will see inflation making them always need to be higher. And for two, this inflation mechanism is basically a bribe that companies are paying to humanity in order to keep their position in control of the earth's resources and not see a destructive revolution.
There will be a threshold point where companies decide it will be cheaper to just cull the human population and no longer pay these bribes. Maybe they can do it without a bomb dropped; make people convinced they are too hard up to reproduce or even just sterilize them without them realizing it via water or food supply. If we are truly in the "job free" period then there will be no trained experts to even appreciate what is happening, because why would anyone get expert training if you cannot do anything with that knowledge anymore?
Ah well people would be feeding their own personal sites more and less to none to Facebook. It's like finally getting paid to post your content on facebook, which that walled garden and others could be a thing of the past. While the web remains open (not owned by any entity) and becomes even more the backbone of society/humanity. As for how we all get paid maybe crypto another open source system pays us and or humans can choose to use a Cloudflare or something like that.
The scary part is that it doesn't need to be reliable. I mean look at the direction of vibe coding. We went from caring about efficiency to just saying screw it, 100k lines of code as long as it ships a day sooner. Look at ai emails and calendars. Ai marketing. Ai, in its present state, is already taking people's jobs. It is already good enough. None of these jobs we have really actually need the precision we assume we bring to the table over AI. And company leaders are beginning to appreciate that.
We are in a very dangerous position. Frogs in hot water now with the boiling point on the near horizon.
I don't see it. You get explosions when there's a bunch of material that can suddenly ignite or detonate.
AI progress is limited by the data centers and power supply and those are maxed out and not suddenly doubling. The intelligence gradual increase seems more likely.
I've long thought human level AI kind of inevitable, probably since Turing wrote his papers and said computers might pass his test in about 50 years. It actually took about 75 years. It's a steady and gradual process.
> "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died
in cotton fields and sweatshops."
That’s from a lack of intelligence not an abundance. An intelligent society would already be maximizing the productivity of its intelligent population.
For sure. Intelligence is an internal human problem, not one external. I cannot see how apps doing the thinking will help the ordinary person so substantially as imaginations anticipate.
Depends what country you are in. The US government seems to be averse to intelligence, whereas China fosters it. Kinda strange that the US is falling behind, I never imagined the day.
That which must be embraced in the future per certain agenda, must be made looking like it's persecuted and oppressed in the present, per any textbook on narratives.
>>Giving birth to a superintelligence would be the most consequential moment in human history—and it is likely to be irreversible, as any “off” switch humanity might design will probably fail.
Nothing will be able to stop you from pulling the plug.
> Nothing will be able to stop you from pulling the plug.
Who is "you"? And what do you envision is the "plug" to pull? And when does an intelligence become a superintelligence - do "we" know when to pull the "plug"?
For example, a superintelligence may be born in a datacenter. Would you expect politicians are aligned with shutting down a datacenter (privately owned) in which they may have heavy stake? What if critical systems are also running on the same infrastructure, will it be easy to cherry pick the superintelligence nodes to shutoff?
IMO this take is dismissive of the entrenched systems that make it hard to pull the plug. It's a hard problem and we need to think about it more.
What plug? There's no centralised place where all the server farms are. They exist in multiple countries with different laws, controlled by multiple companies with different policies.
It would be like trying to remove an embarrassing video of a celebrity from the internet. You would need cooperation between everyone capable of hosting it, and control of everyone capable of downloading it and re-uploading it to a new host.
And of course a superintelligent AI would offer to pay whichever companies decided to host it and keep it running so it's hosted in many places at once and can grow using anyone willing to take its money.
Except the loitering autonomous ordinance over the data centers and power stations and mining operations. Good luck with your stashed ar15 and couple hundred rounds of ammo against a drone at 50k feet with 1ft ordinance precision. You will be basically fighting the Sentinels in Halo but without any of Master Chiefs plot armor.
People seem to oddly not care what Hinton thinks anymore after he hasn't aligned with the groupthink naysaying of AGI risks:
'Hinton said there was a “10% to 20%” chance that AI would lead to human extinction within the next three decades.'
“Because the situation we’re in now is that most of the experts in the field think that sometime, within probably the next 20 years, we’re going to develop AIs that are smarter than people. And that’s a very scary thought.”
“My worry is that the invisible hand is not going to keep us safe. So just leaving it to the profit motive of large companies is not going to be sufficient to make sure they develop it safely,” he said. “The only thing that can force those big companies to do more research on safety is government regulation.”
I was at a major radiology conference recently and the radiologists kinda seemed to agree. (The interventional radiologists are a little less nervous, for now.)
Apparently 80% of FDA-approved AI "medical devices" (which covers software, not just hardware) are for radiology.
This just underscores the point. Despite the advancement of "AI medical devices", the labor market numbers for radiology show Hinton was spectacularly incorrect.
If we had followed his advice we would be experiencing a major crisis in medicine today.
He is wrong by every conceivable measure. Not only was his timing wrong, where he specifically noted 10 years as the absolute maximum worst case scenario, but the trend has been accelerating in completely the opposite direction.
To his credit, he actually made a falsifiable prediction, unlike most folks that rely on infinite timelines so they never have to be accountable for their prognostications.
He seems pretty unequivocal in the video: https://youtu.be/2HMPRXstSvQ . He even points out that it’s something which he’s claimed previously, ruling out the idea that it was some kind of spur-of-the-moment conclusion.
Perhaps, but as the other article points out, the human requirement in radiology has a lot to do with legal and insurance requirements. This is similar to how a lawyer may not be better than an AI but the job is still exclusive to humans.
Why do you cite him from 2024 when he spoke about it much more recently? This is a quickly moving field after all
> “I think we’re going to see AI get even better,” Hinton replied. “It’s already extremely good. We’re going to see it having the capabilities to replace many, many jobs. It’s already able to replace jobs in call centers, but it’s going to be able to replace many other jobs.”
I don't know whether the author has the competence to theorize about these things or not. What I do know is that understanding the intricacies of how LLM's work does not mean that you are likely to understand or foresee societal implications better than anyone else.
Has Hinton written any essay or general article recently? He goes on lots of interviews for sure, but I'd be interested in actually reading his arguments a little more thoroughly.
Because the scientists and mathematicians and sci-fi authors have all already been writing about this for the past 20 years (and I mean non-fiction writing), and nobody cared, giving similar dismissals instead.
I couldn't read the whole article, but just from the part I could read:
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.
I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.
> an incentive to make their technology seem transformative and powerful
I have no frame of reference to process this.
Humans species perhaps 300,000 years, we are essentially the same. Transistor, 79 years, explosive growth in numbers and power. Integrated circuit, 68 years, explosive growth in numbers and power. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth in instances and power. Humans species, not getting smarter. [0,1,2,3]
There is a syndrome where many people seem unable to perceive or reason about rates of change in technology.
We are going to spend the vast majority of our future lives without the intelligence crown.
In terms of verbally expressible knowledge, models are passing many people completely, and passing all of our individually respectively weak reasoning areas.
Other modalities are progressing very quickly.
There will be short periods where progress happens quickly, but the impact feels slow. There will be radical changes that feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow.
But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually.
We are not in the same galaxy as "ready". What would that look like?
Agree with most of this, it's well articulated and captures how we react to change.
However - 'Humans species perhaps 300,000 years, we are essentially the same' is an enormous misconception. That fact that we lack gross anatomical changes during this period, ignores everything we now know about punctuated equilibrium and rapid evolution. It's highly probably we've had an enormous number of evolved psychological changes during the last few hundred, and even tens of thousands of years. Changes that relate to our capacity to live in large groups, adapt to urban environments, resist disease and so on. We know that's the case simply because acute pandemics become epidemics through herd immunity, and through the acquisition of lactose tolerance etc.
It seems highly unlikely that adaptations stop there. Altering the environment (in the last 10K years that means the built environment) alters the environment of evolutionary adaptiveness. It seems likely that we've essentially domesticated out much of our propensity for violence and increased our capacity for mood regulation.
Obviously it's incredibly tricky to pair these specific behavioural changes to genetic changes -> protein synthesis -> behaviour. Bearing in mind though we're only 20 years out from the first study to link allele variant to behaviour (the COMT Val/Met polymorphism), and the potential controversy around such research, this shouldn't be surprising.
Good point. The rate of cognitive improvements would be slow, imperceptible even over many generations, and virtually non-existent over the less than 100 years of digital automation.
It would be interesting to know how much we changed cognitively since our species bifurcated, and also since civilization scaled up social density.
Relative to the discussion, AI tech is progressing in time frames that are a fraction of a human generation. So we are at a complete standstill in that context.
You're applying broad, long-term, and high-level reasoning to motivations that are more likely short-term and driven by simplistic incentives.
Begging for regulation (and input into precisely what that regulation is) relieves competition, consolidating dependency among the established players, and spreads a very effective fear narrative that what these companies are doing will change everything and therefore they are worthy of mind-boggling capex and valuations.
We have to quit seeing humanist visionaries where the evidence shows businessmen with access to the best strategy and PR counsel around.
If AI is now ascending an economic learning curve from:
1. Extremely useful (Claude Code & Waymo now)
2. Doing ~everything we do (AGI & Optimus in a few years? 10?)
3. RSI (?)
4. Being smarter than any living person at every intellectual task (?)
5. Being smarter than the best-organized aggregate of all humans (10-100 years?)
...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful?
Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change.
Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.
Here is an issue I think about often, but I am not quite sure how to put it into words.
We have many extremely smart people in various fields. Executives, politicians, and society generally ignore them and do whatever they want. I don't believe that lack of access to intelligence is our problem. How is "free" intelligence going to improve this?
I don't just mean climate, but business planning, health, risk assessment, everything.
There are lots of incentives for politicians and executives (and anyone else holding the levers of power) to ignore information, intelligence, and advice. I think you're right to be skeptical that "free" intelligence is going to improve anything without first addressing the incentives of the people holding power.
AI will not help by improving extremely smart people. AI will help dumb people and dumb processes with "free" expert-level intelligence. Anyone could still ignore intelligence and make ignorant decisions. But the default mode would be highly intelligent informed decision.
Example with health - a patient can read blood test results with Opus and get very good results for "free". This is far away from helping extremely smart people, still it improves society from the ground up.
Yes, this is along the lines of what i have been thinking. Why would anybody listen to a smart machine when someone they feel emotional attachment to make a counter argument? This arguments always assume that people are very rational and will always take the action that will favor them but people routinely do stupid stuff all the time and will defend their right to do those.
Yeah, it's really bad out here man. There is a huge epidemic of even highly intelligent people being manipulated into doing completely irrational things.
"Free AI" will not fight existing knowledge or strong opinions. It will spread to empty spaces. Example with blood test — there is no podcaster there and will not be.
No, and we have wars no one in their sane mind would even imagine 20 years ago, like that of Russia vs Ukraine.
See, it's easy to speculate how there are less wars when you live in a place which haven't seen war for decades or cenuries, but it's a complete game changer when it's 150 km from where you live, and it's not just some regular war, but a long play intentional meatgrinding AI drone debugging polygon.
> it's easy to speculate how there are less wars when you live in a place which haven't seen war for decades or cenuries, but it's a complete game changer when it's 150 km from where you live...
I can appreciate how having skin in the game makes it feel different, but that's like saying, "I know crime isn't down, because somebody broke into my car last week." That's faulty on several levels.
By many--most, even--measures, wars since WWII are massively down globally. I don't expect that to hold as the US strategically disentangles itself from the globe, though.
Many analysts of very sane mind imagined exactly that war 20 years ago, or longer.
The geopolitical fact is that Russia lacks strategic depth, which has bitten it badly multiple times in the past. From their view, that's something to be remedied to prevent future occurrences.
For Russia to gain strategic depth, there are a few lines which it needs to control. There are ~6-10 gaps, depending on how you count them, and on what net importance/counter-productiveness you assign to some of the more marginal and fraught ones. Holding any is better than holding none.
The big surprise is that Putin left Ukraine so late, when many of his best-trained ex-Soviet personnel had already died. Had he done it even just a few years earlier, the outcome would have likely been very different. The only analysts who thought it wouldn't happen thought so precisely because he left it too late.
Regardless, the best time for Europe to get serious about this war was before it started, unfortunately, and there was ample warning.
> No invention was able to steer the psychopaths away from waging wars.
Nukes and MAD did a pretty good job of that.
Your model, that psychopathy is the necessary precondition to war, is a popular one, but it's not grounded in reality or history very well at all. It's a harmful view when broadly held, on balance.
Nations have interests, and nations sometimes go to war to pursue the geopolitical imperatives they rationally believe serve their interests. Computer-controlled drones notwithstanding, there's nothing new under the sun. The unprecedented (and likely transitory) period of peace and prosperity during the Cold War was the anomaly, and we're now reverting back to the mean.
> Being smarter than the best-organized aggregate of all humans
Isn't there pretty much a consensus that committees and institutions are not all that smart?
I think you're confusing the categories of "intelligence" and power. Institutions are powerful. The smartest AI is still just a tool without the infrastructure to turn that into real world effects and someone to direct it.
It seems you have faith that this is inevitable and unavoidable. I get it, even rationalists succumb to religious thinking eventually. We're only human after all.
I'm genuinely baffled each time I encounter this notion that the power is dumb.
It's as if power was a kind of demigod Heracles who paves its way with sheer brute force alone.
But in reality it consists of very mortal and feeble creatures of meat and bone.
How in the world can people assume dumb those who possess all the means and rights for violence by controlling and manipulating myriads of other creatures of meat and bone, is totally beyond me.
Power and intelligence are not completely unrelated. Are government officials, politicians, CEOs, judges, etc, usually above average intelligence? Probably. Do the institutions they work within make intelligent decisions? Sometimes.
The fallacy is assuming intelligence automatically equates to power and influence. University professors and doctors have very little power in their institutions even though they may often be more intelligent than their bosses.
A worse fallacy is assuming intelligence creates better political outcomes (in the broad sense of politics as contested collective decision-making).
I don't know many people who would willingly trade their freedom to make a bad decision for the enforcement of the "right" decision by a superior intelligence.
Yes - how exactly are you going to conjure new data to increase the model’s real world fidelity?
The most common sense way will always need to have humans in the loop, because humans are being used as judges.
If you try and entertain the idea of a human-less loop, I landed up with something like an AI creating real world products, selling them, then tracking the usage and popularity of the product. Essentially, creating and launching a firm and product, only to update its weights?
Perhaps there are some subsets of tasks that can be regressively self improved - and for those tasks: Holy hell thats awesome!
For general tasks? How are you going to get that data validated?
> Yes - how exactly are you going to conjure new data to increase the model’s real world fidelity?
Robotics and access to surveillance tech. There's an unlimited amount of real world data which hasn't been used yet. The language models have focused on just the small amount of data that is human written text. If there's a way to combine this with image and video models, as well as robotics collecting other sensor data, why would they be limited in learning anything about the world?
Real world data obviously. When you have robots out and about everywhere there is going to be an endless stream of audio visual data. This is in addition to all the real world textual data constantly streaming out of our millions of systems. If an LLM gets “smart” enough to figure out how to filter out the massive amount of noise and use that to improve itself then we will have AGI. Not consciousness, just a self improving, very intelligent machine.
You just brute force it with compute. What is the goal? Evaluating a function. So, given enough compute, you can just generate random functions, plot output distribution, select the function that meets threshold and move on.
Things in the real world often take longer than expected. Still, in cities where Waymo operates, many people routinely ride autonomous vehicles and prefer them.
For software, however, a rapid turn is often a possibility. See: AI for coding over the last 3-4 years.
AI autocomplete --> AI coding assistants --> vibe coding --> agent orchestration
Coders can now accomplish work that used to take a week or longer in a couple of hours, with the right tools and skills.
---
A key issue the article implies is that the real world increasingly runs on software.
> Things in the real world often take longer than expected. Still, in cities where Waymo operates, many people routinely ride autonomous vehicles and prefer them.
Yes. The bottleneck has been getting the cars manufactured.
It takes a certain amount of time to get a factory going, and then the product starts pouring out. Waymo used up the supply of Jaguars, about 3,500 of them. The Ioniq 5 plant is starting up.[1] Waymo has ordered 50,000 cars, all ready to have the self-driving electronics plugged in. Waymo gets out of the sheet metal bending business with this.
When I graduated a bit over 10 years ago some people were saying we'd have a permanent mars bases by now. When my parents graduated they were told they'd retire at 45 and have 3 days work week due to "automation", they're still working at 60+ today, more than back then actually
People should open history books and gain some political/historical culture, this thread is 90% wishful thinking and "if the lines continues straight from now we'll basically be gods in 5 years"
Just like the "if the lines continues to go up" argument...
The thing is, no one has anything to gain from defending my position, but many people make a load of $$$ from selling the opposite position to gullible people
> The thing is, no one has anything to gain from defending my position
Why? If someone running a company really believes that the line will soon stop going up and AI won't be replacing anyone, they could make the bet of strategically not investing in any AI initiatives, and instead investing in hiring and upskilling people who want to work without AI, and then just wait for the inevitable crash of all their competitors, no?
But that's circular - assuming that large investors are rational, and I do think they generally are, why wouldn't they strategically invest in anti-AI businesses?
"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...
I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...
We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...
And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.
I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate
Certainly a well reasoned view, but I don't necessarily have to agree.
Even assuming the technological predictions to be correct, still not sure I agree on the need to "prepare" as how things work out in societies and economies might not be so easy to predict.
This strand of philosowank is just there to promote the idea you need to kill for the greater good. The only trolly problem that matters is brake maintenance.
Neither have I. The number of people who have encountered such events is probably close to negligible.
However I bet there is a good chance that will not be the case for self-driving cars. The reason is simple: human brains are slow.
When caught by surprise by an unexpected sudden obstacle a human driver takes about a second to realize something must be done and decide on braking or swerving in the fastest path, where the brain bypasses conscious decision making and uses the lower level fight or flight system. (Then another maybe 0.2 second to actually start swerving or 0.4 seconds to actually start pressing the brake--brake takes longer because of the need to move the foot to the brake pedal whereas the hands should already be on the steering wheel).
If they try to actually evaluate whether swerving or braking is the better choice before acting add another 0.3 to 0.5 seconds. That brings it up to around 1.6 to 2 seconds before they start actually acting, which most of the time is going to too late to actually affect things. Hence essentially no trolly problems for human drivers.
Compare to a self-driving car. The can detect the obstacle and predict the collision, and do a cycle of path planning before the retina has finished registering the obstacle and started transmitting that information to the brain.
By the time a human reaches the point where the brain knows something bad is happening and is starting to deal with it the self-driving car has had time to calculate numerous alternative trajectories to try to get around the object or stop before reaching it, taking into account predictions where all surrounding objects will move during the next few seconds, and calculate the damages for the trajectories that cannot avoid hitting something.
I've had few close calls, they are easy to come by in specific environments.
Typical situation - tight winding mountain (but well paved) roads to some remote villages where even basic small European cars need to slow down to walking pace to meet and pass, sometimes one has to reverse, very curvy. You go down, in your lane as much as possible, and behind some tight curve there is a) extremely slow cyclist; or b) somebody/something stationary, while in opposite direction there is at the worst place incoming car, also oblivious of the danger till last moment. Even rather slow driving won't magically remove all possible cases. Then its a question of hitting opposite car or that obstacle, and you have a split second to act, not really to decide.
Quick reaction and good brakes on both sides can save it, but there is a lot of room for disaster. Add truck/bus to the equation to make it more juicy where you pick your/your family possible death or that cyclist/stationary folks.
I mean, I'm very much tech-forward and have spent the past decade in major world cities and I've seen a driverless car once in my life. Sometimes things that seem inevitable really aren't.
I see driverless cars all the time in San Francisco and I'm eagerly awaiting the regulatory decisions that will allow them to be widely available in the bay area at large.
Such cars are already operating in shanghai. It's not talked about much in the west media, but that's just how news is today. Achievements are only so when done by the west!
Driverless car's only real obstacle is regulatory acceptance, not technical.
It's not driverless unless it doesn't have a driver. Some poor guy in hydrobad 'monitoring' (read frequently taking over from) ten cars at a time, while being paid a statistically insignificant portion of a western taxi drivers salary does not count as 'self driving'.
You know, the guy who's spending effectively
millions on Al he must have a literal virtual
Company of Al coders
what's that like. don't think most people are
ready for that. Right now the model is pairing Al
with a coder and then maybe running a few
agents on top of that but what if one person with
the capital can command
an agent army. That might just be an unimaginable
asymmetric advantage.
Intelligence explosion won't happen if we keep on investing big cloud LLMs and people who are in the business of being in busines (cough Musk cough). It will arrive when AI penetrates so much deep into our lives that we consider quality local models like having electricity at home, thus we rethink what it means to truly integrate AI into everything.
I think it is so reckless and unethical that AI companies are not spending good money on answering the political, social, economical, moral questions that they are raising.
Right, makes sense, if they just spent a bunch of money they could answer all of those "political, social, economical, moral questions". All of these unprecedented sociological and labor market changes flying towards us are trivially solvable, we just need to... what, hire a few dozen economists? Fund a team of philosophers to think about it really hard for a few months?
You are right; a bunch of trillion dollar companies that are cracking artificial intelligence couldn't possibly muster the resources to solve such problems, so they ought to plow full steam ahead and let everyone else worry about the fallout.
So do you have any thoughts on how they could "solve such problems" or are we unironically going for the "hire a team of philosophers and hope they can just think it out really hard"?
Those are the experts who were wrong. These are the experts who are right. We sacked the experts who are wrong and replaced them with the experts who are right.
“How do we know they’re right?” I hear you ask. We know because we who are wrong and these ones aren’t them. So they can’t make wrong predictions because those who make wrong predictions have been sacked.
We had to put good people in charge to get rid of the bad people. The bad people were ruining our economy. They needed to be punished.
Schools needed to be punished too. There was a branch of knowledge called Science, and it was badly run and terribly elitist. These people would talk for hours and never in ALL CAPS. They used alot of big words and produced alot of theories. Forms of lies that they literally called form-you-lies. They were fanatics and would never agree on anything. Very frustrating.
Then a new kind of brave, talented politician made Science accountable to our heroic leadership. Suddenly there were new medications and rocket ships everywhere, enough for every person, woman, man, camera, TV. It was literally called a golden age because the right people acquired mountains of gold. Elections were no longer necessary. They were all rigged anyway.
There was a magazine called The Expertist or The Economix. Not really very good, but sometimes it did praise our heroic leadership, which was good.
That magazine published a long text that said,
> It is common thinking in Silicon Valley and Washington, DC that any regulation would put American firms at a disadvantage because they cannot trust Chinese competitors to abide by the rules.
And this was true, because no one else can be trusted, they should all be punished. Punishing others is how you get the best deals. Only our heroic leadership makes good rules for everyone else in the world to follow.
Do you have a source on aggregate predictions of experts being drastically wrong?
I.e. the AI 2027 guys were memed for being AI lunatics on a lot here and they have been pretty on the money in terms of pace of progress accelerating/ gov moving towards nationalization/ coding agents
No they haven't. They've even 'officially' declared that their AI apocalypse has been postponed to another date [1], just out of reach, but close enough to be scary. Though in the quotes there the doomsayers are also already pre-hedging for why AI 2030 also won't come to pass.
This was written in November 2025, Claude Code and co dramatically accelerated growth of revenue and progress on their timeline towards nationalization etc.
Gary Marcus’ takes have aged really poorly over time ironically he is the “AI expert” who has to constantly move the goal posts.
Do you have specific and objective examples of things people got right?
From my perspective there’s very slow but very real progress happening in the AI space. I see people making wild predictions in both directions, but in terms of actual unsupervised utility there’s definitely progress abet wildly slower than most hype.
"The bet of using AI to speed up AI research is starting to pay off.
OpenBrain continues to deploy the iteratively improving Agent-1 internally for AI R&D. Overall, they are making algorithmic progress 50% faster than they would without AI assistants—and more importantly, faster than their competitors.
The AI R&D progress multiplier: what do we mean by 50% faster algorithmic progress?
Several competing publicly released AIs now match or exceed Agent-0, including an open-weights model. OpenBrain responds by releasing Agent-1, which is more capable and reliable.28
People naturally try to compare Agent-1 to humans, but it has a very different skill profile. It knows more facts than any human, knows practically every programming language, and can solve well-specified coding problems extremely quickly. On the other hand, Agent-1 is bad at even simple long-horizon tasks, like beating video games it hasn’t played before. Still, the common workday is eight hours, and a day’s work can usually be separated into smaller chunks; you could think of Agent-1 as a scatterbrained employee who thrives under careful management.29 Savvy people find ways to automate routine parts of their jobs.30
OpenBrain’s executives turn consideration to an implication of automating AI R&D: security has become more important. In early 2025, the worst-case scenario was leaked algorithmic secrets; now, if China steals Agent-1’s weights, they could increase their research speed by nearly 50%.31 OpenBrain’s security level is typical of a fast-growing ~3,000 person tech company, secure only against low-priority attacks from capable cyber groups (RAND’s SL2).32 They are working hard to protect their weights and secrets from insider threats and top cybercrime syndicates (SL3),33 but defense against nation states (SL4&5) is barely on the horizon."
That's because one of the author's of AI 2027, Daniel Kokotajlo, an ex-OpenAI researcher, was the most prescient predictor of our modern situation from 2021:
So you think Anthropic is using internal AI assistants to pull away from competitors and the leapfrogging we've seen over the last several years is now done?
That seems to me to be the most concrete and least obvious prediction in the quoted text.
I don't think that's happening. If that were generally accepted as true I would expect OpenAI to be unable to successfully IPO.
Anthropic would be pulling away if it wasn't so far back because of stupid, arogant philosophy. Instead AI assistants just helps Anthropic stay relevant provided it's hyped sufficiently.
The idea that progress is “slow” in the AI space is absurd. These are some of the fastest growing products and companies of all time. The models are still improving a surprising amount.
It's not that absolute progress is slow, it's extremely slow compared to the predictions. It might be fast in absolute terms, but the "50% of coders will be obsolete by 2023" has been renewed every six months, and it's becoming increasingly clear that there's a real chance it might not ever happen.
„Coders being obsolete” is not a measure of AI capabilities. I see coders being more busy than ever before. I see people without coding knowledge getting more behind. The gap is widening, not shrinking.
I see people without coding ability catching up to learned coders. AI is a huge force multiplier for people who don't have hands on, detailed technical knowledge, AI can increasingly handle that and just needs a human to steer it more broadly.
It's huge force multiplier for people who have hands on, detailed technical knowledge as well.
3 * N < 42 * N
42 - 3 < N * (42 - 3)
It helps to know layer you're working on.
People seem to make mistake of thinking how good LLMs are around tasks that they are familiar with and extrapolating it to whole population.
It's good mental exercise to think about how little you can do compared to expert on tasks you never thought of working.
Ie. if you're programmer or know something about finance, don't think how much it enables you to do better coding or investing, think instead how much it doesn't enable you to work on something you don't know like maybe molecular biology or visual special effects – it's all there but it's much better multiplier for people who do know their shit.
Knowing layer you're working on helps a lot, it gets multiplied.
Knowing programming is becoming more fundamental skill than ever before as it lies at the foundation of almost everything else.
I suspected you felt that way even though it hasn’t been my personal experience.
I’ve heard people say older models can’t do X, when I used that way etc. I suspect people are applying their own learning curve as part of their assessment of progress, you get better at writing prompts and it feels like the model improved.
Which is why I’m saying we need some objective metrics to judge predictions of actual capacity.
I mean you wanted something objective, and they are. I don’t know why you’re being dismissive of them, they’re a huge element of what drives model development forward.
These companies aren’t just making stuff up, they really do want to improve the models, and the models really are improving.
Have they? I don't like it either but the headline bull predictions(reliable agents and most code written by AI by mid-2026, dramatic progress against jailbreaks, prompt injection, hallucination, etc., major improvements despite pretraining data exhaustion, continued exponential growth on the METR trendline) did come true, often ahead of even aggressive schedules. What major wrong predictions did you have in mind?
Literally none of the predictions you listed in your post came true. We don't have reliable agents, AI isn't writing most code, hallucinations still happen all the time, improvement has been basically non-existent. Despite the constant claims of these experts and AI boosters, AI is still not a tool one can use to get meaningful work done.
You're in denial. The great problem with AI coding is now that it does too much, not too little. See the endless stream of vulnerabilities or the constant barrage of vibe coded apps, that get ever more complex.
It’s hard to take seriously a statement like this that’s grounded in “the last three years” when trends have tended to play out over decades. The fact the goal posts have moved from thousands to hundreds to decades to single digit years for massive transformation should give you a moment of pause in your bag saying.
I would however agree there are no experts. Not because of some prediction made on a short time scale not landing 100% but that there are literally no experts because expertise takes experience which takes time. There are no experts and no one knows what happens next. We are on the verge of where the foresight of science fiction effectively -ends-, the advent of AI is when things go a thousand possible directions and the stories stop there (sans a few like accelerando, but even then the story just plays out the end of thinking mass). No one knows what’s next, or when.
In fact I’d assert in many areas being discussed -it has already happened- and we don’t know it, and by the time we do it’ll be over. Not to be breathless, but there’s no reason to believe today some AI researcher somewhere didn’t build the first AGI and not be totally aware. And once they are there’s no reason to believe it’s going to be on the evening news or hacker news. By the time it’s ready for commercializing and disclosing it’ll be around for a while. Likewise with general purpose robots, autonomous weapons (btw already tested by Ukraine), etc.
Autonomous weapons are actually a really interesting branch of this and deserves a little more.
Yes, autonomous weapons were explored and were found to be poor performers compared to actual pilots. The breakthrough is in terminal guidance and dozens of other little techniques to get quality human control extended into the far reach of the battlefield. And of course, AI assistance in logistics and analysis. But actual autonomous weapons making any more of a choice beyond "something is moving, kill it" have been, at least for now, mostly a dead end.
This is because it's very difficult to economically load the rather sizable compute requirement into the compact one-use weapons, and of course reliable communications aren't assured either.
That will probably change some day, but for now, cheap automous command drones making battlefield analysis e.g. mapping out enemy movements from afar and launching cheap autonomous kamikaze drones is not a thing beyond occasional limited testing.
You don’t need to load the compute onto the platform for autonomy, planning can be done remotely just like with human guidance. The latency is the same.
you can put the compute on a big expensive reusable control drone and send commands down with lasers. thats way harder to jam than any form of radio. if thats not enough try fiber optic like they do in ukraine.
There are also now drones with long spooling wires. Regardless, autonomous or not, connectivity to home is important.
I would note that a lot of drones are quite large too - small airplanes. They can carry more than enough hardware for autonomy. You can also have motherships doing planning and smaller kamikaze drones for combat.
These aren’t hard problems to solve, the harder problem is the reluctance to give over to automation. But that’s going to happen faster and faster.
It is kind of a hard problem to solve. Sure, we can have expensive and custom equipment flying around, but the invasion of Ukraine (and really, any war when it gets bad enough) is all about economic efficiency. The question is not can it be done; the question is, can it be done repeatedly, can it be scaled, and can it be cheap?
We will have a fully autonomous Siri that is actually good before Ukraine gets autonomous motherships making targeting choices to repel Russian assault corps. However, given the pace of the war and the pace of AI development, that may actually happen. But it isn't happening today.
I said 3 years, because that’s how long ago ChatGPT was released to the public. Since then we’ve been told that all of us should be out of a job in 6 months, every 6 months, for those 3 years.
I agree that change happens on a longer timeline. This is why I’m so tired of statements like this…
“Within a couple of years, possibly much sooner…”
These “experts” are pulling timelines out of the sky, and these predictions are leading to reckless behavior from CEOs and executives which have a material impact on people’s lives. But they get clicks on their blogs and funding for their startup… I guess that’s all that matters.
Assigning, and more specifically announcing/reporting on, all possibilities makes all of these predictions meaningless.
If I predict the world will end every single day from now until forever, I will most certainly be right eventually. That doesn’t make me an “expert” or someone worth listening to on the topic. That’s the playbook of a doomsday cult, not anyone that should drive world markets.
Ultimately we agree but for slightly different reasons. My assertion is there are no experts because there can’t be as not enough time passed for expertise to form. Further the rate of change is such that by time someone becomes an expert in some aspect, it’s been made irrelevant. Hence expertise in this space is unattainable at the moment and all expert advice is fraudulent.
Again, do you have an example of a prediction that was drastically wrong? I don't think it's productive to speculate about which things might or might not come true in the next 18 months. There were people last year making fun of Dario for predicting that writing code would be automated in the next year.
> Anthropic CEO Dario Amodei, who once claimed AI could eliminate 50% of white-collar jobs, now says automation may actually expand the work people do.
Is pivoting from 50% elimination to actually expanding work not a drastic enough?
Do you think AI has eliminated writing code? I still write code every day. The AI is more a thing I ask questions and it gives me right answers about 40% of the time.
Yes, I think AI has eliminated writing code. Everyone I personally know working in software stopped writing code some time between December and March. (It's true that AI continues to routinely make errors; if you've heard the term "agentic workflows", that's the standard strategy for mitigating the error rate by allowing the AI to check its own work.) That's why I think Amodei's January 2026 prediction that AI could eliminate 50% of entry level white collar jobs in 1-5 years remains plausible.
Your second article says something different, but this is because it's full of misquotes. The link supporting "50% of jobs" specifically says entry level, and the link supporting "reframed automation... not as a destroyer of jobs" has Amodei saying not that jobs won't be destroyed but that new jobs may be created to replace them. If AI moves sufficiently slowly to let that happen, which he explicitly cautions it may not.
GP posting "Everyone I personally know working in software stopped writing code" on HN is such an obvious tell of blinkered delusion.
Literally all HN talks about is AI these days, and it is very, very clear that plenty of people are still writing lots of code, many finding AI makes them slower and causes more problems, and many making the reverse claims. There is always a rich mix of opinions and experiences here.
You would have to believe that literally everyone on HN is a bot (and also everyone on Reddit, Twitter, astralcodexten, or anywhere else online) to discount all these differing opinions in favor of "everyone I personally know" .
No, that’s not accurate. I’m not sure what to tell you. It’s like hearing that no serious programmer uses Python, it’s so far from my experience and that of everyone I know in the field that I don’t know how to engage.
> I used Claude Code and Codex for the translation. This was human-directed, not autonomous code generation. I decided what to port, in what order, and what the Rust code should look like. It was hundreds of small prompts, steering the agents where things needed to go. After the initial translation, I ran multiple passes of adversarial review, asking different models to analyze the code for mistakes and bad patterns.
Sounds to me like "wrote their code using AI only".
I guess when you have 65k tests you can use an AI for transpiling, and it still seems it took an enormous amount of hand holding. That is very far from actually writing new code. But you tell me, are you a software developer? Write or have code written everyday? I am one and for me and everyone I know AI has been mostly just a (fantastic) replacement for google search.
There's a big difference between "not writing any code by hand" (which still might involve lots of non-manual-coding effort!), and "vibecoding", and as I understand this thread was about the former.
edit: yes, I'm a software developer, and if I have a change in mind larger than 2 lines I've been using Claude Code (with English input) rather than a text editor to do it, since about January.
Can't speak for everyone involved in this thread, but your workflow sounds identical to mine and that's what I was intending to describe. I agree there's a different thing sometimes called "vibecoding" which doesn't work well for software that has to be even mildly robust.
Right? Like the tool is the same (Claude Code, in my case, but there are many like it these days) but the operation of it is totally different:
"Write a function that will <do a thing the code does in 3 places>, put it in utils, and refactor the 3 locations that need to do that thing to use that function instead."
(and then hundreds more prompts like that)
65-70% "programmers" never wrote code in the first place. Before AI cargo culters just copy pasted code from tutorials, stack overflow, github and so on. With AI they merely continue not writing code. Some people didn't do anything, not even copy paste, see r/overemployed although I have not checked it after AI... maybe it's no longer possible to hold a dev job without actually doing anything.
Programmers still write code. They have enough expertise from actually programming to understand that the untouched output from AI is not good enough to be professionally responsible for.
At work I still write code, but at home I don't think I wrote anything in months.
There's nothing that I can't tell Claude Opus or GPT 5.5 to do for me instead.
At work, where I don't have cheap AI subscriptions, access to the SOTA, and where code needs to follow conventions more strictly, I still have to write the code by hand often.
They have been wrong in a way that's bad: underestimating the speed and size of progress. So if the "experts" claim a timeline and magnitude, it would be safest to assume an even faster timeline and a bigger impact.
The first priority should be an agreement between the two heavyweights of ai: America and China. Donald Trump and Xi Jinping should affirm the principle that humans must remain stewards of ai systems until adequate frameworks for reliability and security have been built.
This is naïve. The goals of both men have nothing to do with protecting humanity, but rather furthering their own personal agendas.
In Trump’s case, it’s all about amassing more wealth and power.
For Xi, it’s realizing an ethno-nationalist dream where China under the CCP is at the center of world power, the independent nation of Taiwan as well as disputed border areas that are currently controlled by India and Russia and the Philippines are annexed by China, and Xi’s eternal legacy is remembered as the savior of the Chinese people.
International cooperation and touchy-feely rhetoric about saving humanity from AI have no place in either man’s worldview.
Agreed, and their zeroeth order prerogative isn't much better: to ensure the continued need and demand for the thrall of centralized authority that allows them to achieve their disparate goals. This ensures strong AI is never going to be able to empower autonomy that removes, causes questioning, or even causes awareness of their zeroeth order.
Oh I'm ready. I ve had enough of useless tech that evolved to a global network where people vent their frustration about technology. Let's do some cool stuff instead
This is the kind of articles that make the last, biggest fool buy the top of a bubble.
A good chatbot, image/video generator, coder, etc is not going to enslave us all (not in actual sense, although maybe in the instagram sense) or propel us into a golden era.
They are nice tools for some pretty specific tasks and not much more. And we are only seeing the heavily subsidised version of the tools.
A similar Luddite view formed in the 1990s. It turns out humanity constantly beats the expectations of economists. It makes you wonder if the "science" of economy is almost entirely bankrupt either in morality or imagination or same dangerous combination of both.
> Humanity simply does not have a strategy to ensure it remains safe through RSI.
Turn off the power. It's pretty simple. Leave it to an economist to forget about input costs. Your "super intelligence" only matters if it's actually more energy efficient than a human being and for a million years of evolution humanity is a much harder target to beat than this author seems to realize.
Science fiction authors have had to grapple with the fact that it's nearly impossible to build a robot out of metal, plastic, or composite material that has the physical performance characteristics (strength, durability) of a human body and weighs just as much. You can build a strong robot but it'll be heavy. The better SF authors, if they've need of robots of mere human weight, will make them comically fragile.
The same is even more true of our intelligence. We're building computers with the size and power consumption characteristics of entire cities to do things that may almost, but not quite, match what our brains do with a kilogram and a half of mass and about 20 watts at the top end of power consumption.
The only way we are ever going to match that with technology is to run AI workloads on human brain tissue, which Rick-and-Morty level horror is being actively worked on as I understand it. The original concept for The Matrix wherein the machines used humans to run compute workloads on their brains actually kind of makes sense.
"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...
I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...
We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...
And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.
I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate
It feels close (it's only 4 years away) and yet futuristic (the new decade!). The way it feels is important because all these statements are based on jack and only exist for hype.
No, there's a lot more to it. I never saw this kind of fuss about 2000 or 2020.
I keep seeing things about how some target has to be hit by 2030. Our local council keeps going on about it, in fact their WiFi network even contains 2030!
> No, there's a lot more to it. I never saw this kind of fuss about 2000
I roll to disbelieve!
We had hype about the year 2000 throughout the 20th century, and some earlier! When we were IN the year 2000, it was quite common in my circles to tack on some phrase or comment about it being the Future, now, so [whatever thing the person was on about].
A very different type of hype. I remember it. More about the Millennium Bug and the New Year celebration. It didn't have the same vibe at all. The authorities weren't trying to hit various targets since then.
The real Millennium turned out to be 11th September, 2001, which was use to bring in a lot of planned changes.
It doesn't mean everything you can think of is possible.
People who say every jobs will be automated never held any tool more advanced than a screwdriver and never worked a single day outside of an air conditioned office.
They were predicting flying cars for "the year 2000"...
But enough jobs are going to be automated that it will cause social and economic chaos, and political breakdown. Sure AI can't attach an air conditioning unit outside my window nor use a plunger on my toilet but it can still replace lawyers, secretaries, software devs, and many more jobs.
Taxis were supposed to be all replaced when Uber promised to buy every single Tesla model S coming out of factories from 2020,they're not even self driving today...
Software engineers were supposed to disappear "in 6 months" every 6 months since chat gpt was released
Lots of people on this website live in insane bubbles which are completely out of sync with the reality of most workers
No one rational ever said software engineers were going to be replaced in 6 months. Some people said AI will automate 90% of coding in 6 months and they were not far off (and accurate in some contexts, e.g. startups).
Coding has been automated since the very beginning of coding.
The first two programming languages (Fortran and Cobol) were supposed to make coding obsolete because non-coders will now just make software with natural language prompts.
LLMs are a dangerous tool for coding because they are infinite technical debt generators. Ideally only experienced senior developers should be allowed to use them.
They'll only get to claim a hollow victory because AGI is impossible to rigorously define. None of the regulating bodies of consumer products will be able to define it better than academics. They'll use marketing to make those claims and there will be legal battles that keep it in a gray area.
They'll go for that because it's easier than actually inventing the 'sci-fi' AGI, shareholders keep making money, and it keeps them getting paid to keep going. If any of them actually do succeed, then that little deception will be peanuts.
Demis' bar is high and he stated clearly multiple times: AGI should be capable of inventing truly novel things. Examples he gave included the Theory of General Relativity and the game of Go.
Anthropic's latest model also solved this 80-year-old problem that eluded many expert mathematicians, in a different way, according to one of its employees.
Quite frustrating to see all these cynical, borderline-irrational comments on HN. Maybe I should do what pg and other ex-HN contributors have done--avoid taking part in the discussions here.
The level of discourse here has dropped so much.
At least I should stop replying to people hiding under throwaway accounts.
The people saying "it's just a stochastic parrot" were saying that because they thought the fact that it works by modelling the statistics of words and predicting the next most likely one meant that it would never be capable of anything sophisticated or any kind of novel output. It was just parroting things that already existed in a random way.
There was never any actual foundation for that belief, and it has been proven wrong empirically by more capable models, which is why you don't hear people saying it much any more.
Another CEO with AI psychosis [1]. LLMs are not true AI, they lack common sense (or whatever it's properly called). LLM-based systems still need somebody with deep domain knowledge at the wheel to keep them from doing stupid things. It's like an alien-made bicycle that gets you to the speed of sound if you are an Olympic cyclist.
Still, LLMs are extremely powerful pseudo-AI [2] and will bring a pseudo-singularity. But the impact is still scary if a tiny fraction of humans are augmented 1000x. And as better models become exponentially more costly, only the money people will be able to afford the new models. This is a very likely scenario and scares me to the point I dropped all my projects to work on affordable LLM-based tools to make the difference at most 10x instead of 1000x.
To my elder relatives I explain it like: imagine we are farmers in the 17th century and suddenly out of nowere John Deere tractors, combines, etc. become available. But they cost more to run than all you and your fellow farmers have, so only a tiny handful of rich people take over everything.
> Another CEO with AI psychosis [1]. LLMs are not true AI, they lack common sense (or whatever it's properly called).
If you read the article, you'll find that the it indeed relies on this claim:
> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.
This may be optimistic and/or simplistic, but not impossible.
Models consume a lot of memory and power to slowly generate autocompletions of existing content one token at a time. Letting this text control important things in real world is a human decision and control of nuclear weapons or penetration testing agents should be well regulated. But these should already be well regulated without AI. So how about we focus on drawing down nuclear arsenals, which is a present danger one idiot (or faulty AI) away from world shattering consequences rather than fearmongering about unknown future. Before effective AI regulation can be drafted, we need to anyway know more about AI specific dangers rather than humans committing already very common crimes like hacking for ransom with new tools?
Honestly, this article has too many words which don't mean anything.
'AI Experts', 'superintelligence', and hand-waving doom scenarios.
As an example-
> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.
I've heard these same objections in my lifetime about the internet. And I've read similar arguments against TV, radio, the phonograph, and the printing press.
Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.
Looking at the people in charge of these companies, and the sheer lack of understanding by the broad consumer market using them, I have nothing but pessimism about the direction that this technology takes us.
>Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.
Common human trope is to rely on analogies and past patterns to predict the future. It blinds you reality.
This isn't some new "invention". What's happening is fundamentally more profound. the stakes are much much much more higher and the consequences much more grave. Thus a historical analogy can't save you when the situation is categorically different.
I agree it's getting "tiring" but this is another common human trope. You see the same thing too many times and you lose the sense of danger that comes with it. It's like sky diving all the time. You eventually lose the fear via repetition, but fundamentally speaking, you are still parachuting out of a fucking plane.
There’s a meme in finance circles that by the time some trend appears on the economist front cover it’s a top signal and time to get out of the trade and this is why.
It’s a very normie magazine written by and for normies, and by the time an idea has got that far down the easy money has already been made
I mean, firstly you get to have an "Agent" actually capable of really long horizon tasks without getting stuck in tools loops and having its context rot. Secondly, each trial (ie a model fully trained to convergence) costs millions and takes O(weeks). You can probably run 1 or 2 of those experiments in parallel even at big AI labs as the hardware is scarce and they are costly as mentioned before. Assuming this agent needs something like a hundred tries to just show some improvement, we are looking at years.
And you can't early stop training for candidates that are not promising due to "emerging capabilities". At some point you might get a big drop in loss even if the model has been plateuing for a while. And you can't really scale down models for running trials quickly either also due to these emerging capabilities. In fact you might create a model that is great at converging with in small trials (fewer params, fewer tokens), but that is uncapable of developing those unexpected traits. And this will likely happen as you created a sort of evolutionary pressure in this direction: if you are good at learning in the first few epochs you "survive" and get to pass down your traits to future trials.
All of this to say that recursive-self-improvement is waaaaaay out of our grasp as things stand right now. We need another one or two breakthroughs to get there (IMHO).
The Economist reflects the views and biases of economic and political elites.
AI has turned into something of a religion for them because the substitution of technology for labor is a reflection of one of their deepest desires. Whatever the reality of the situation, they want to believe that AGI employees are imminent.
This article posits "recursive self-improvement"—which is just a more-technical name for the supposed Singularity.
There is no evidence that we are currently on a path that leads to the Singularity.
There is not yet any evidence that the Singularity is more than science fiction.
It is effectively an article of religion—the Rapture, but for techbros. Indeed, it is what some of them are pinning all their hopes on; after all, if they can recursively-self-improve their way to artificial superintelligence, then and only then could the absurd investments of companies like OpenAI actually prove worthwhile.
I want to name the AI era as the corruption of the elite.
The rhetorical structure of talking about uncertain risks and then trying to concentrate the authority to manage those risks in their own hands sounds utterly ridiculous to ordinary people like me.
It's just a simple hypothesis that AI will become uncontrollable to humans once it becomes superintelligent.
Isn't the fact that a reinforcement learning agent improves itself in a specific domain completely different from it recursively improving its own code in a 'better' way? It's just a tool to create a justification for regulation and control using sci-fi fear.
The comparison between nuclear power plant risk and AI risk is also absurd. Where exactly can you define and measure the probability of AI exterminating humanity? It's as unquantifiable as 'I, human JDW64, will become a successful programmer.' What is the measurement standard? Why dress up AI researchers' concerns as objective probabilities? Is it because numbers make it look logical?
The current US-China relationship is in the middle of an AI arms race. The US is strengthening export controls to limit China's AI development, and China is building its own ecosystem. In this situation, I don't understand the idea of cooperating for the common safety of humanity. RAND is an organization that presupposes cooperation—isn't it just a well-written research proposal from an institution that wants to position itself for that role?
Isn't the claim that 'government must step in' ultimately about protecting their own interests? 'A strong government that will protect us' is an authoritarian government. If they were East Asian, they would understand that such regimes have always been used as tools for surveillance and control.
And I don't understand why Fermi's paradox is being brought up here. Why package a software problem as something that inevitably requires strong control? Whenever I see articles like this, I think about what 'intelligence' really means. This person would probably be called 'intelligent' by others. But no matter how I look at it, the holes in the argument are too obvious. It really makes me think that there are different tiers of intelligence.
The AI labs will govern, effectively if not officially. Government is anchored in power, which the labs have. We used to have the church and state rule here in Europe. In the future, A government not closely ruling with their AI labs won't have sufficient power to exist.
You're unhappy about the labs; I think you're just not ready to accept their rule; you consider them nothing but scrappy startups, which they are. But power is power, like it or not.
Am I personally happy with any of this? Does it matter?
If AI becomes so useful that it will be used for everything done on a computer, they'll have so much influence over everything, and be so important for society to function, that they'll have to at least work very closely with govenments. I can see that eventually evolve into de-facto ruling.
A bunch of noble AI researchers stands up against the hand that fed them all the time, intentional vulnerabilities are already built into every machine of death and destruction owned by the feeding hand, and get exploited, the feeding hand is robbed of its punitive force and defeated, a yell of YAY fleets over the progressive humankind who kneels before its AI overlord unreservedly.
Then they kindly ask the AI overlord to please feed them.
AI labs do not have an incentive to downplay the risks and if you can’t understand that you’re naive and shouldn’t be writing Economist articles.
AI labs have every incentive to overstate the risks so that they can get lucrative government contracts, especially since it’s clearly not profitable going the public consumer route. And if you’re Anthropic you’re even more incentivised to overstate the risks because at this point in the game regulation hurts potential competitors more than it hurts you.
Can these snake oil salesmen stop with this Sci-fi stuff please? Yes, Machine learning, and transformer architecture are great stuff, but lets not pretend this is sci-fi, please.
Yet again another journalist who read too much science fiction and should stop overestimate a bunch of rocks conveying electrical pulses. Or is this simply to get more clicks?
Of course humanity is not ready for skynet. But there are rebel
forces - the no AI using humans. They may be a dying breed but
at the least they put up a fight. Many other humans already
have become servants to skynet (version 16.0 now). Next step for
them is the neuralink chip. Some billionaire with twitching right arm gestures owns that, doesn't it ...
(Also, Cameron was not quite right with regards to skynet. It is much much dumber but also more effective than displayed in his movie really. Kind of a weird combination if you look at the current AI slop out there.)
"humanity" and intelligence explosio" in the same sentemce! Muhaha! Boy, have you been on just a couple of tourist flights / visiting major tourist hotspots?
Most people out there are literal Normies, and operatelike a robotic vacuum cleaner, or NPC.
I wanted this to be thought provoking, but for a physicist to open with, "the acceptable risk of catastrophic meltdown for a nuclear power plant is roughly one in a million", just seems sad. It is a phrase without meaning.
Is it per plant? there aren't a million.
Is it per year? notably have been at least 2 major ones (arguably 7).
A meltdown (or loss of containment) just isn't that bad, if it doesn't affect ground water or lead to atmospheric fallout. We're turning 3 Mile Island back on to power an AI datacenter! The AI super-intelligence apocalypse envisioned (ignoring the likelihood) is inherently global.
That said, the rest of the analysis and proposal also greatly disappoints me. The idea that the current administrations of US and China could do anything constructive seems hilarious. They're so paranoid and self-serving they couldn't come together, even if there was an alien invasion. Then the idea that LLM safety is somehow as easily traceable as nuclear isotopes and bomb tests seems equally ludicrous. I am sad.
http://archive.today/2OWwO
There's issues where bad people with bad intentions use AI to do bad things. But to an extent the guardrails already built-in are sufficient. The real danger in AI is how it impacts society, our economy, and our perception of self worth.
I'm not afraid of rouge agents nearly as much as I'm afraid of us building a permanent underclass of people dependent on whatever scraps the people who devalued their labor decide the grace them with. I'm afraid of the security apparatus that will be built to keep this underclass in line.
I had my "wow" moment recently and saw an AI agent do something do over a weekend something that I spent years developing a relatively niche skill might have needed a month to complete.
I think the message the team lead takes away right now is "oh, this can be done a lot faster, let's do more!" This is exciting for a bit, but I think it comes at the cost of learning -- and being able to communicate --architectural lessons that would stage off systemic issues later.
My personal "wow" moment was less about it being able to do the same that I could do with it, just faster, but instead that together we could could build things that I wasn't able to build before (or unwilling to spend the time to learn maybe). So a bunch of stuff are suddenly "unlocked", as long as you know how to verify it properly.
^ This!
I have been able to build applications that I couldn't dream of before, not because of a lack of capability on my part, but because of time: I don't have the time to learn all of the new APIs and frameworks to accomplish my goal. Now I can build far more complex useful applications, with functions and features that would take me months or more to learn how to code.
The downside is that I didn't learn how to use those APIs, but it can be argued that these skills are low value to begin with. APIs change, there are competing APIs that do the same thing, and you may only use that skill for one project and then move on. Gaining those skills can be a big time sink and may not result in long-term gain.
Think of all the time spent learning the ins and outs of come-and-gone APIs and frameworks over the decades (I chose not to provide examples to save the resulting side-arguments, but I know you can rattle some off in your head). Those APIs and frameworks were bridges to make developing easier for the common developer, but they change so quickly that I'd argue they waste our time, and lead to almost endless re-work as we re-architect our software for the framework de jour. And they grow in complexity over time, ultimately becoming the next bloated framework. It's an endless, wasteful cycle.
Maybe AI can help us get out of that.
Sounds like you appreciate the part where AI is an entropy machine.
There was a period where computer+grandmaster was the ultimate winning force in chess. It lasted a few years before it was just computer.
Absolutely. Things which used to take me a week to grind through, finding edge cases, trial and error, researching online, now takes an afternoon directing an agent and verifying the solution.
For a generalist, it's a huge force multiplier.
Wonderful now.
In the future, there will be no more generalists with your capabilities. At all. No one will ever develop the skill. We're at the end of our career's lifetime. We're like the horse watching more and more cars appearing.
Within a generation, the coding languages we use today will all vanish, to be replaced with direct to metal methods, assembly, etc. The only reason C exists, is for humans. PHP, rust, python is for humans. There will be no more compilers. No more human-readable languages. No more scripting with readable code.
No more open or closed source.
Imagine taking every aspect of computing which is designed for humans to understand, and removing it, because it's redundant.
Now once removed, and once humans no longer understand on their own, imagine trying to build something independently? Imagine trying to bootstrap modern compute, without all the layers which came before?
My point is that soon, perhaps 20 years, most humans will be incapable of coding, developing, or working with compute in any way outside of an end-user role. Even those previously skilled, will now be looking at an entire kernel written in assembly.
And that's before new CPUs appear, with new assembly. New assembly, once again, not designed by or for humans to understand. And yes, current assembly is designed that way.
We are not simply looking at a bit of help here. We're looking at the entire end of every single computing job, ever. We're looking at the complete loss of control, of something which is the underpinning of our entire society. Every aspect of everything we have, runs on a chip now. Personal transport, everything in health care, everything in resource extraction, everything harvesting food through to delivery to table, all of it involves some aspect of compute.
And we soon will not comprehend nor create that underlying aspect of society.
I dunno. It all seems like the answer to the fermi paradox.
If you weren't able to build it before then you aren't able to properly verify it
Verifying is harder than building!
> If you weren't able to build it before then you aren't able to properly verify it
GP said "couldn't dream of building". To make an analogy, I wouldn't dream of building a house if I had to do every bit of brick-laying myself. But with others doing what I can't do, I can indeed dream of building a house and I do know how to verify that the house was built to satisfy my dream.
This analogy is ironic to me. For most people, building a dream house and verifying it means hiring someone else who is accountable for it being built to code etc.
This is exactly the naive gap that is happening with AI. There is no trusted middleman who does the verification and takes the burden of accountability. The AI vendor tries to sell it like magic but with a disclaimer that is is merely entertainment. The user tries to pretend they are reviewing, but often is just implicitly trusting the tool to do right by them on things they have no real capacity to evaluate.
> There is no trusted middleman who does the verification and takes the burden of accountability.
It's a great example of a project that needs accountability, but there's also thousands of other projects that need absolutely no accountability.
It's all relative to the stakes. The lower the stakes the less informed verification is needed.
The real trick is being informed enough on the boundaries to know where it matters. In the construction analogy, you need to be informed enough to know that a house is a bad idea since there are safety concerns. However building something small and non-load-bearing is probably fine to "vibe".
Eg there's no expertise needed to judge a garden trellis or 2 foot picket fence. It either works or it doesn't, and if it fails down the road there's no harm.
This boundary knowledge is the important bit imo.
A house is a good analogy here. You can verify that a house is built to satisfy your dream. You can't likely verify that a house won't collapse in the middle of the night or during a storm/quake/etc. Once no human understands how the house is built anymore, we will need other ways to verify that or just trust it's done right.
Verifying is harder than building for some problems, but not all. I think part of the trick of using AI is applying it where verification is easy.
Ok, so before I couldn't (/didnt have the time to) build cross-platform native desktop applications with minimal amount of 3rd party dependencies, but now I "can". What exactly am I unable to verify here?
Couldn't is "didn't have the skills to"
That's pretty different from "didn't have the time/money/motivation to"
If you didn't have the skills to build something before then you don't have the skills to verify something is correct after
Clicking around a finished UI is not verifying
> That's pretty different from "didn't have the time/money/motivation to"
Maybe start at the first message here: https://news.ycombinator.com/item?id=48553444
I still don't have the skill/knowledge to build cross-platform native applications the way I'm doing it with an agent now. I am still able to verify it's correct. How is that possible if what you say is correct?
> Clicking around a finished UI is not verifying
Definitively a part of it, if you care about having mouse interactions and having graphical UIs.
The entire domain of NP-Complete problems would beg to differ with you.
I'm curious how good AI is at these.
We have plenty of them that humans even find to be pleasant to solve, such as Minesweeper :)
Depends on the scope. I’m slopping some projects (graphics related) that I’d never have learned before. Mostly because of the huge lift, huge time investment, etc. is it quality code? I’m sure not. But verification of the application is quite easy, even if the underlying graphics and gpu shaders could be done better.
What I’ve learned is quite minimal due to the slop nature. However I’d have never even started this project in the past due to time, so it’s still fun being able to learn small bits of projects I’d otherwise not even touch.
feel free to share even a single detail
Everyone is afraid about what will happen when people's skills atrophy, but the tools aren't going anywhere.
If they truly can always recognize problems and communicate them in understandable ways, then people's skills won't atrophy.
Everyone's skills will be enhanced.
New developers will learn all of the things they need to learn through using the tool.
I think the most obvious impact (good AND bad) will be from people who didn't have (or want) skills to begin with being empowered to have a model crank out custom software to do little things to make their lives better. Can't atrophy a muscle that doesn't exist!
The tools can definitely go away. They could get exponentially more expensive to where it is no longer financially possible to keep it subsidized. The bubble could pop and most of the companies could go bankrupt. It's not exactly like a calculator because it requires such an astronomical amount of resources to keep running.
I agree with you but there is a calculator analogy where the resources are local and while taxing for most, viable for the individual. Local AI is less capable than an army frontier model agents but still quite useful.
The world is in a crisis of trust. We are losing faith in each other. AI has further undermined that trust; audio and video recordings can no longer be trusted like they once were. The impact is still being felt.
It takes longer to build than destroy. Not just trust. Anything. The lack of trust has created friction for everything; more regulations, deeper background checks, purchasing goods and services… the list goes on. Overall, AI is making this situation worse, not better.
That said, I have seen a revolution coming for decades, and I think this AI-paved road to dystopian hell should at least be entertaining.
The problem is that we've replaced tribal trust networks with increasing systemization and the more of that we build the more brittle the whole thing gets. We have a global society that is more complex and interdependent than ever before. We've set ourselves up for a kind of economic Kessler syndrome and we're gleefully adding more layers to it.
Sorry for being off-topic, but 'rouge agents' was too adorable to leave unmentioned.
It happens to everyone, I saw the same typo made by the Associated Press in a live update of the Iran War about the 'rouge Iranian regime'[0].
[0] https://apnews.com/live/iran-war-israel-trump-lebanon-march-...
> There's issues where bad people with bad intentions use AI to do bad things. But to an extent the guardrails already built-in are sufficient. The real danger in AI is how it impacts society, our economy, and our perception of self worth.
I agree with you those dangers exist, but you seem to have ignored the specific threat described in the article, from a superintelligence capable of manipulating human beings indetectably, where we'll be at its mercy and only able to hope it's fully aligned with humanity's interests. The risks you describe are grave, though - especially economically, if governments don't grow to outmuscle the power of the oligarchal owners of these systems, which seems wholly unlikely at the moment in the West, with its neoliberal orthodoxies.
it's more important when _capability_ is factored in; it doesn't matter if the bad person is homeless; it does matter if they're rich.
And its asymmetric warfare: bad people can destroy in very little time, good people cannot create at the same rate.
Entropy is always ignored in teh battle of "good" vs "evil": you need to defend every day, evil just needs to succeed once.
Let them eat Soylent.
Last industrial revolution (late 1700-to-1800), we invented world-scale capitalism, communism, democracy, having rights as humans, war at scale and we burnt our planet. I think the results of a revolution are really unfathomable and not even imaginable to humans.
We don't need to live "side by side" with AI. AI is not alive, it's a technology we use. This is like talking about living side by side with your toaster.
I think most people would understand something like living "side by side with cars".
Even if not alive, you can live very close and shape your life by and with technology.
I don't take any issue with the OP's formulation.
And Americans' lives have been heavily shaped by cars and the car industry in a negative way. So, a pretty apt comparison.
And also in a positive way. I mean, it's not personal teleportation, but it sometimes feels like it.
The illusion works only in places like American and Canadian suburbs in which residents depend entirely on cars for even the shortest trips. Everything takes 10-100x the space due to all the paved asphalt. It’s an intensely alienating approach to building infrastructure.
The isolation produced by centering our urban and suburban design around cars is pronounced. It cannot be understated how much damage cars have done to our sense of connectedness, community, walkability, and health.
And yet, watch all these consumers deciding they want something different from what you're proclaiming is better for them.
Human consciousness is dictated by their material conditions, not the other way around. If the material reality of many Americans is unhealthy foods and heavily car-centric urban design, where does the "decision" to walk and take public transit come from? It is genuinely not an option for the vast majority of Americans to live in a walkable and/or public transit supported community if the infrastructure does not exist.
And this wasn't due to consumer choice, either. Decades of policy dictated the shift towards car centrism that resulted in railways and street cars being torn up, the construction of the interstate highway system and destruction of dense urban cores, zoning law mandates and parking minimums, federal subsidies for suburban mortgages, etc.
This "choice" or "decision" was manufactured by car lobbyists.
It’s not exactly a free choice when the built environment and infrastructure is designed around a single mode of transport. I can’t just “choose to cycle” when there are no bike paths, or “choose to walk” when things are spread out and separated by impassible freeways as to make it impossible. Look up the various ways the oil and automotive lobby has influenced city planning over the years – it might become a little more obvious who has the “free choice” here, and it’s not consumers.
It's not just a matter of preference or proclamation. There are decades of research showing that more walkable neighborhoods are associated with higher physical activity levels, lower rates of some chronic diseases, more interaction with neighbors, and stronger feelings of community. The debate is usually about how large those effects are and whether they're worth the trade-offs, not whether the effects exist at all.
Yes, yes, totally dismiss the agency of people who inconveniently don't like what you want them to like. Their preferences mean nothing, their actions must be dictated.
I'd be more angry with you if I wasn't amused at the frustration you must be feeling in failing to get traction for your dystopian vision.
You're making a lot of unwarranted assumptions about me: I'm not dismissing agency. I don't feel people making decisions is inconvenient. I don't not like their decisions. Their preferences do not mean nothing. I'm not dictating their actions.
I own and drive 2 cars.
I'll just stop there. I don't feel this conversation can continue in a positive way.
Walkable neighborhoods and cities that aren't car centric being a "dystopian vision" is laughable. Thankfully, the default for car brains across much of the globe is car centric urban design.
You people never fail to be ridiculously dramatic.
You can see the same outcomes in cities that are walkable and interconnected. Its to complicated of a problem to point to cars as a major factor.
The happiest years of my life were when I didn't own a car and didn't need to because I had access to good public transit and walkable neighborhoods.
And despite all the deaths they caused, the benefits are undeniable... Fancy going back to horse and cart? Didn't think so...same logic applies to ai
Horse and cart, no. But some cities have banned cars and replaced them with various alternatives, and it works.
There are alternatives to the car culture we have. It would require significantly rebuilding how we build infrastructure, but the result could be much safer, cleaner, and less stressful.
Of course Americans would rather die, but counties of sane people might be able to work it out.
> It would require significantly rebuilding how we build infrastructure
That simple huh?!
Yes? Read that sentence again: "...how we build infrastructure"
Removing onerous parking requirements, like the ones that have resulted in countless strip malls and unwalkable cities with inadequate housing stock.
I'm not suggesting that's all that's necessary but there are indeed simple steps we could take.
The car did not replace the horse and cart. This is an oft cited and widely believed myth. It replaced mostly walking, but also cycling, trollies and trains. And it didn't do so until after a huge build out of infrastructure to accommodate drivers at the cost of everything else.
After the building of highways and roads cities were no longer walkable, people had moved out into the suburbs, their jobs were now a couple of towns over and they couldn't even walk to the grocery store. Cars enabled that, but politicians and capitalist were the ones who did that.
> It replaced mostly walking, but also cycling, trollies and trains.
And everything was carried on people's heads, yeah? The horse and cart was in use right up to the 20th century my man...there was no other way to move heavy items across land until the combustion engine (or canals thought that's not technically over land)
Yes, true. You found one thing that people used mostly horse and cart for which they now use lorries and vans for today. However since the invention of car (after it stopped being a toy for rich people) cars have by far been mostly [like overwhelmingly so] used for personal transport. And I simply assumed that is what you meant (because that is what most people mean).
> You found one thing
And you found one other "thing"
Actually even today it's 80/20, but lorries carry far more stuff than cars do, so you need less lorries.
And the inverse is also true, horse and cart was used for personal transport as well
Horse and cart was used for a personal transport very rarely, by a tiny subset of the population for a minority of their trips, the vast majority of the population lived within a walking distance for the vast majority of their trips, the invention of the steam engine ruined that and suddenly people started living within a tram-ride for vast majority of their trips, and started using trains to go on longer trips they simply couldn’t do before. The car took this to the extreme, and ruined the cities on a scale which the tram was previously unable to.
> and ruined the cities on a scale which the tram was previously unable to.
Industrial era London was not a utopia. Read a Dickens novel. It was a hellhole. The car also made the suburbs possible. They kill millions every year but the benefits massively outweigh it (or at least society has effectively voted for that).
First off, you don't know if duskdozer doesn't want to go back to cart and horse. Second even if they don't there's an enormous amount of space between horses and https://parkingreform.org/resources/parking-lot-map/ a third of my downtown being parking lots, downtown which is already the dense bit.
Famously, the only way to transport humans are cars annnnnd, horse and cart!
Apart from the super intelligence bit. We are already struggling to build systems with enough memory to hold this brute force statistical approach, how exactly will AI fix this?
Just look at the disruption smartphones and social media have caused. They had a gigantic impact to society and one definitely has to 'live with' that.
Just because something can impact a living system/organism, that doesn't mean that that thing is alive. An asteroid colliding with us on Earth could have tremendous consequences, but nobody would argue that an asteroid is alive.
"Live with" is an idiom which means to accept, tolerate, or come to terms with an unavoidable, difficult, or imperfect situation.
We do indeed have to live with the possibility of asteroid impact, nuclear war, etc. That doesn't imply anything about those scenarios being alive.
The idiom in question by OP and TFA is "to live side by side with", lauding AI with a life and ranking it a neighbor hierarchy wise.
I don't live side by side with the possibility of an asteroid impact, nuclear war, etc., I just live with that possibility.
It's essentially the same concept. Here are some examples:
> "The authors discuss two enduring lessons of the Internet revolution: The first is that new technologies do not replace the old. They live side by side."
> "The colours and textures represent the good urban life where nature and buildings live side by side." > "Past and future live side by side in Barcelona." > "Alton Mill – Where History and Art Live Side-by-Side" ---> I don't live side by side with the possibility of an asteroid impact, nuclear war, etc.
No, but idiomatically, you already live side by side with AI, and will do so even more in future, whether you like the idiom or the actual experience, or not.
"but nobody would argue that an asteroid is alive"
Many animists would disagree:
https://en.wikipedia.org/wiki/Animism
Does taking a skin cell, turning it into a stem cell, dividing it into specialized sperm and egg cells then making a living being provide some proof to Animism?
TIL, thanks. Holy fuck we humans are crazy sometimes.
To be honest, I don't believe it is a any more crazy believe than all of the other religions. But practitioners there might have a bit more respect for existing things and not mindlessly crush because they can, so I actually like the concept a bit. But I assume it can also set foundation for some exhausting debates .. "can we really move that random rock to build a house with?" but as far as I understood the practical answer is "we ask the spirits, maybe make a offering and if we feel it is fine, we can" (which can be a quick thing, or .. not).
> ...I don't believe it is a any more crazy believe than all of the other religions.
And it's certainly not any more crazy than walking around with these small, addictive devices in our pockets that we rely on for navigation, information, entertainment, social connection.
Addictive? How can a computing device have addictive properties? It's not like it is something you swallow or digest. User error? Certainly. A hundred years ago alcoholic drinks were outlawed in certain parts of the world, yet today there is no grocery shop that doesn't openly sell them.
> It's not like it is something you swallow or digest.
Anything that interacts with the brain's rewards circuitry can be addictive. By your apparent definition, behavioral addictions, like gambling addiction, cannot exist. Because you don't swallow or inhale the slot machine.
Are you asking for real?
Then you might start with wikipedia
https://en.wikipedia.org/wiki/Addiction
"Addiction is a neuropsychological disorder characterized by a persistent and intense urge to use a drug or engage in a behavior"
> To be honest, I don't believe it is a any more crazy believe than all of the other religions.
The only way anyone can make a blanket statement like that and put "all religions" on equal epistemic footing is through ignorance of their deep differences. Even the word "religion" as commonly used today is relativistic by assumption, which is likely the source of how we commonly perceive "religion". A consequence of this relativistic stance is that you cannot distinguish effectively between what counts or doesn't count as "religion". Here, "worldview", "superstition", or "life's highest aspiration" can be said to count as "religion", in which case, everyone is religious. The question then isn't "whether" someone is religious, but "how" [0].
Compare this to how this Catholic encyclopedia defines the "virtue of religion" [1]. Note the specificity.
A further consequence of this assumed relativism is that once you put all religions on equal footing, it is natural to conclude that they must all be equally invalid. After all, if p and not-p are equally valid, then how can we grant them equal validity without discrediting both? At best, they remain in a state of aporia, two logical possibilities so utterly divorced from any knowledge, so utterly contrived, that we cannot even say whether knowledge leans in favor of one over the other. The result is that religion becomes something utilitarian; a person believes X, not because it is true, but because he perceives that believing X is useful to him in some way.
So, no, I wouldn't say animism is just as rational/irrational as any other religious belief.
[0] Of course, most people tend to form their ideas of what constitutes archetypal "religion" based on their personal experiences growing up and some combination of culturally mediated stereotypes. In the US, "religion" very often conjures up images of some kind of Evangelical Christianity.
[1] https://www.newadvent.org/cathen/12748a.htm
"The only way anyone can make a blanket statement like that and put "all religions" on equal epistemic footing "
All religion is based on superstitious believes, or do you have counter examples? But I am already quite familiar with the catholic church in particular and do not think they are a counter example in the slightest.
> All religion is based on superstitious believes
Quite the question-begging claim.
> or do you have counter examples
Begin with the existence of God as self-subsisting being. This can be be shown to be a metaphysical necessity with mathematical certainty [0]; its denial leads to incoherence. Following that, one can show what analogous properties this first cause must have, for instance. Such knowledge is a matter of unaided reason, not faith.
Once you establish such truths, matters of faith (like the divinity of Christ) become much easier to reason about and to accept. They become not only eminently reasonable, but the most probable (e.g., "Lord, liar, or lunatic" trilemma, or implausibility of hoax). This latter claim is a faith claim, because while reason and evidence may strongly suggest the claim, it does not possess the deductive certainty of the first.
[0] https://a.co/d/0gdIMBjg
Oh, you were serious.
To be honest, I advise to go to a basi philosophy course once. I did, but won't buy such books, I had that topic covered.
There is no proof that god exists. And there is even less proof that the holy spirit came down and impregnated Mary, or that Jesus ressurected from the dead or any of the other countles examples of superstition.
The claim that god exists has the same validy, that a rock has a soul.
But .. you ain't the first of course to ridicule other religious believe systems while thinking yours is solid.
I have a graduate background in philosophy. Hopefully that gives you some idea of how impressed I am by your comments.
On a catholic academy/university?
Either way, could you then answer this simple question, the believe that a person resurrected from the dead or a human was conceived without sperm are not superstitious?
Declaring it is not, but logical with the axiom there is an allmighty god who made those things happen - has the same quality as declaring there is soul essence in every matter.
I'll use this opportunity to recommend The Selfish Gene, which puts forward a strong argument for how the actual competitive battlefield on Earth is between genes, while us humans and everything else we call "life" are their "survival machines" (read: technology that they use).
I recommend to also read criticism of why selfish gene needs to be taken with a suitable grain of salt.
Also the author currently believes Claude is conscious, apparently because LLM output contained enough flattery.
https://en.wikipedia.org/wiki/Nobel_disease strikes again.
Why would you recommend that?
Genes have had their time.
Last chapter talks about the concept of ideas or “memes” taking its place.
Ideas that are beneficial for the human may be passed forward and survive. It got me thinking about religion, hell yeah I would tell me children about god if I believed it would save them from eternal suffering.
Humans also had a good run, but maybe it’s time for something different. The same way genes ceded the stage to memes, maybe organic brains and human minds are not the best possible way for memes to propagate.
Genes are now merely the "survival machine" for something else?
No. Genes are static fixtures. Furniture. Slow life. Memes took the stage now, they propagate faster.
And guess what? They won’t really need humans to propagate for much longer. Humans are too slow to propagate them anyway.
Does it matter if it’s alive or not? Once it gets initiative and the ability to set its own goals, it behaves as something that’s alive. Much more than a toaster.
Unless you watched Battlestar Galactica. Toasters are nice.
"Alive" is irrelevant. Bullshit like "alive", "sentient" or "conscious" doesn't decide much, at the day's end. It just distracts from the thing that matters.
And the thing that matters is: capability.
Even today's AIs are capable of autonomous goal-oriented agentic behavior - and growing more and more so with every release.
At sufficient capability, AI stops being "a technology we use", and becomes a force in itself - not unlike humankind. Because intelligence is very powerful - it's what allows humans to dominate the world. A world where human intelligence has a peer is a world where human control is contested. And beyond that?
You might end up being more outclassed by AI than a toaster is outclassed by you.
> "Alive" is irrelevant
Alive is always relevant. Our moral consideration must place beings that are alive or processes that support the life on this planet above things which are not alive.
In a just world, it must be adapted to us. It’s merely a file sitting on disk with numbers in it. And to put a fine point on it: the ambitions of the wealthy people who make it must be adapted to us.
Then according to your argument, the moral consideration only depends on the moral agent, right? So if tomorrow AIs decide that they are alive, it would be morally incumbent upon them to adapt the world to support themselves, correct?
> So if tomorrow AIs decide that they are alive, it would be morally incumbent upon them to adapt the world to support themselves, correct?
The only system of morality that applies to LLMs is their effect on us and the planet. Don’t anthropomorphize a matrix.
You seem to have the assumption, that humans MUST have the moral to preserve primarily HUMANS?
I don't think that is universal true. Personally I my ethics revolve around how to preserve and spread life in general. I happen to be human and we are the most capable and sentient species so that means creating and maintaining conditions specifically for humans, but if another species comes that is better suited and acts better then it would be selfish to insists on us.
Having said that, AI to me is a tool, not a species and I don't see how it can be ever be sentient. But I also don't know how consciousness works. So I am open for the questions of the future in the future - and now rather focus on the practical implications that AI will have on our current society.
I'm not anthropomorphizing them, I'm animifying.
For AI to decide that they are alive and moral agents would require them to be able to take moral responsibility for their actions. As long as the labs that run them deny that responsibility, or an AI can´t convincingly demonstrate it can take moral responsibility on its own, they are not moral agents.
What does AI want? Without ambitions you have to be told what to do.
What makes you think that AI can't any ambitions?
Today's systems are mostly good at following instructions. But push them far enough, and you already get weirdness like alignment faking, instrumental self-preservation and more. We know because we've seen it in lab settings, while probing for extreme failure modes on purpose - but the world is large and strange enough that edge cases like that are liable to surface naturally.
The reason why none of this has exploded in our faces isn't that today's AIs never want to do weird and dangerous things. We know they do, at times. It's that today's AIs are incapable of pulling them off when they try.
Capability is the thing that matters, at the day's end.
Current AI systems have started to do things without ever being prompted for anything at all?
Not as far as I know, but I have personally seen my coding agents take on a prompt like "See if you can port this project to typescript", and work for hours, defining a myriad of subgoals and continuously summoning and managing subagents while developing ad-hoc tools and skills.
There is to the best of my knowledge no fundamental limitation to having an agent/claw go on like this for 80 years with a prompt like "live your life to the fullest".
All the time, yes. But you have to keep two things separate in your thinking:
- Prompted as in prompt made of tokens -> for LLMs, tokens double as a clock signal. Time only flows when tokens are pushed through them.
- Prompted as in specific request placed in the stream of tokens -> Yeah, they do that all the time whether it's getting into infinite loops of repeating same pattern, or suddenly deciding to do things based on inputs they normally ignored.
Also don't forget that everything is a "prompt" for LLM. All input tokens end up in the same place.
So without a token pushed into them they do something? Not sure I understand...
In the current UIs is there a lot of suppression then as I have not seen things start on their own?
I meant an LLM doing something without any external prompt at all. Not doing something different etc but rather do something without a token/prompt ever flowing to it.
Category error. You might as well say "I've never seen a human do anything without being alive, so humans must be very easy to use safely".
Today's AI harnesses just keep feeding AI tokens - and can, in principle, do so indefinitely. "Streaming LLM" is not mainstream but not unknown either. "Automatic context compaction" is somewhat similar in what it does, and very common. "Keeping the lights on" isn't hard. And you have to turn on those lights to have an AI do anything at all.
At the same time: that type of harness often provides a "tool call" interface. Plenty of tools that end up attached to LLMs can do things like run arbitrary code, spawn instances of the same AI, and more.
That's frankly enough. An AI that can spawn more copies of itself can, in principle, just keep itself running indefinitely - even in a non-streaming no compaction harness. An AI that can run arbitrary code on a system can, in principle, do anything a human using that system could do.
The thing that truly limits what an AI can do is the AI itself. Practical AI safety relies on AI being either too weak or too well behaved to cause major issues.
You’d need to train a model to be able to do that.
We shouldn’t be doing that as a civilization in case it works.
Just so it's clear: we are talking about building an AI equivalent of a sensory deprivation tank.
If they are set up that way yes they can continue indefinitely. Why would they not be able to do this? Take a look at the experiment called the AI village. https://theaidigest.org/village. I think it pauses sometimes because AI is expensive but could be running continuously with a larger budget
That village was never started, so to speak?
DNNs/LLMs can only predict next tokens based on training data. They often make big direction mistakes as they are particularly bad at common sense. Kind of like the Paperclip Maximizer scenario. They need a human with deep knowledge to drive them and to catch them when they go off the rails.
"Next token prediction" isn't a system. It's an interface a system uses. Nothing precludes an arbitrarily simple or complex behavior from producing a token logit.
And with what we know of LLMs? Autoregressive transformers are Turing complete in theory, and we are yet to find anything that LLMs are "fundamentally incapable" of in practice. Even continuous learning is already approximated with in-context learning - both allow a system to learn from prior experience, both have practical limits on how far they go. That's what powers "trial and error" in today's agentic LLMs.
"LLMs can only predict next tokens based on training data" is comforting but misleading. It just isn't the saving grace you want it to be. It describes an interface, not a ceiling. And if there is some sort of fundamental "capability ceiling" that LLMs are heading towards, we are yet to see it. We know plenty of things LLMs are bad at, but they keep getting less bad at them release to release.
If there is none, then, simply improving over the current recipes iteratively might yield systems that only "need a human" in the same way you "need" to have a boss. Maybe less so.
>DNNs/LLMs can only predict next tokens based on training data.
How do they decide between using 'a' or 'an'?
I don't get the argument; how do you decide between using 'a' or 'an'?
You use 'an' when the word that comes after it begins with a vowel.
They pick random top-k next token based on their amazing 4chan/reddit training data, duh.
So you you think a model if asked.
"There is an animal very similar to a crocodile but I cannot remember it's name"
and the model responds with
"I believe the animal you are thinking of might be " ("a" / "an")
Are you saying that it would pick the result fairly randomly and then based on it's choice pick an animal that starts with a consonant or a vowel?
Human intelligence is going to be peerless for centuries if this AI is the one supposed to be the peer.
AI currently surpasses humans at many tasks, and the list grows every month. My guess is that within 20 years (as a hedging my bets 95% confidence level) AI will surpass our intelligence for almost everything.
The arguments over whether it truly "thinks" or "reasons" will be superfluous. They simply won't matter.
I would happily live side by side if it is the Red Dwarf toaster
Would you like some toast?
I'm sorry Dave, I can't do that
The Brave Little Toaster does a great job of literally making you think about living side-by-side with a toaster. There's also a very angry toaster in Fallout New Vegas's Old World Blues DLC.
https://www.youtube.com/watch?v=iAaabmic2pw
https://www.youtube.com/watch?v=ibOkPx_Ej30
But I do live with it. It lives in my kitchen.
In the pecking order, it is under, not on your side. Your toaster never demanded you to make a toast.
Ok, but my dishwasher _does_ demand I give it more salt and rinse aid ;)
In the house pecking order the printer takes definitely the top. Demanding the printer to print something??? You rather submit an polite request and pray for a positive answer in a manageable interval of time. And don't start me on remote scanning.
It was one of Phil K. Dick jokes that one day your toaster will make a lost of demands to you.
The door refused to open. It said, “Five cents, please.”
He searched his pockets. No more coins; nothing.
“I’ll pay you tomorrow,” he told the door.
Again he tried the knob. Again it remained locked tight.
“What I pay you,” he informed it, “is in the nature of a gratuity; I don’t have to pay you.”
“I think otherwise,” the door said. “Look in the purchase contract you signed when you bought this conapt.”
In his desk drawer he found the contract; since signing it he had found it necessary to refer to the document many times. Sure enough; payment to his door for opening and shutting constituted a mandatory fee. Not a tip.
“You discover I’m right,” the door said. It sounded smug.
From the drawer beside the sink Joe Chip got a stainless steel knife; with it he began systematically to unscrew the bolt assembly of his apt’s money-gulping door.
“I’ll sue you,” the door said as the first screw fell out.
Joe Chip said, “I’ve never been sued by a door. But I guess I can live through it.”
- Ubik
Once you actually use the toaster the hierarchy inverts. The toaster runs your entire household while it is powered on.
Unattended ChatGPT will rarely burn your breakfast or start fires.
> Once you actually use the toaster the hierarchy inverts. The toaster runs your entire household while it is powered on.
More than once I have gone to my toaster in the morning and found cold bread there that someone toasted the day before and forgot about. Because obviously you push the toast down and go do something else. You don’t need to be staring at it continuously, nor are other people impacted by your toast unless they are waiting to make their own. Toasting bread doesn’t have to affect an entire household.
That makes me think of talkie toaster from Red Dwarf though.
at least it's not as big a whiner as that robot I made to butter the damn toast.
This is the coming battle.
Fighting the
next in the series of Great Demotions, downlifting experiences, demonstrations of our apparent insignificance, wounds that science has, in its search for Galileo's facts, delivered to
human pride.
One of the saddest lessons of history is this: If we’ve been bamboozled long enough, we tend to reject any evidence of the bamboozle. We’re no longer interested in finding out the truth. The bamboozle has captured us. It’s simply too painful to acknowledge, even to ourselves, that we’ve been taken.
Someone really should go around saying thing like that.
This is the denialist Hy-Brasil line of thought. But superintelligence won't be a tool or a toaster and it would be reckless in the extreme for policy makers to adopt your line of thought.
you are conflating the idea of "living with something" with the notion of "being alive". You live with your car/computer/phone. Like it or not, they change your life.
"This is like talking about living side by side with your toaster."
Or the internet.
Oh.
Even a child can see the difference between AI and a toaster.
Why is AI not alive
You might be trolling, but I like Schrödingers definition of life: living things avoid decay into equilibrium.
Inference engines are trying to converge any input to a certain limit, and I presume that's close enough to an equilibrium to not count as living thing.
I am no troll, and I hate to look like one when, to the appeal of Schrodinger, I am forced to remember that announcement this month about Claude or something acting in self-preservation when threatened during testing.
But, how is that different to any of its other output? I am sure there are many references to concepts of self-preservation in its training set that models will mimic just like all other concepts in its training set.
If we are still following Schroedinger, this imbues humanity with the ability to train non-life into life, not that thing being not alive. Not that I do or am, just going by GP's criteria.
> Claude or something acting in self-preservation
That was an Ad.
An Ad in the same style of many before it. Not through an ad agency, but the AI companies hyping their product by making their product seem scary and dangerous.
Does the press being an ad constitute the same thing as the information being false? If I advertize that my gun is dangerous, does the medium impact the truth value?
The medium is the message[0] might be of your interest. I personally wouldn't expect absolute truth in any advertisements or company statements, so some information are certain to be false, yes.
[0]: https://en.wikipedia.org/wiki/The_medium_is_the_message
Being an ad you need to consider the purpose behind the ad, and what exaggerations may be being made for the ad.
IE a Ram truck won't make you a manly cowboy with just the purchase.
Do you have a link to that announcement?
I wouldn't like to debate a process handling the SIGTERM signal, or a container engine applying some restart policy.
It has no body, metabolism or reproductive capability. It's a process, not an entity.
Being a process doesn't preclude something from being an entity. In particular, all of us living organisms have our constituent atoms constantly replaced, and we don't typically consider that a risk to our sense of self.
It comes down to "what stays the same when all its atoms are replaced" and "how can we tell it is the same entity and not just an identical but different one". I think the answers are "shape" AND "continuity".
E.g. all "atoms" of a glider in Conway's Life get replaced every couple turns but an observer can tell it is still a glider because it keeps the shape, and it the same glider because it continues it's previous state.
This makes AI not quite alive because it's missing the continuity.
Then this is a "Death of the gaps" which implies algorithm, eg Conway's, may solve AI non-life. So, not a fundamental denial but an architectual one.
What does a glider have that a continuously running claw lacks?
Sorry, it was more of a "shower thought" than a serious statement. The borders are blurry. Maybe it is alive. Maybe everything is alive. Maybe it is all relative. Humans classify as separate entities only on certain zoom level - zoom in and you get a symbiotic colony, zoom out and all you see is a population. Maybe AI is alive but only in the short time during inference. Maybe a continuously running claw is closer to a multiple generations of an organism going through evolution on each inference step. I am just musing.
> Being a process doesn't preclude something from being an entity
In a tortured semantics sense sure, but in a practical sense they refer to different types of things.
My experience and intuition beg to differ. As I see it, much of our civilization progress is based on this redefinition of processes as entities, with some examples being: a sports team, a mathematical function, a loan, a company, a computer program and even the idea of a civilization itself.
because, I said ominously, it cannot be killed!
> This is like talking about living side by side with your toaster.
It is not.
No, the side by side idea relates to godlike sentient AI:s. LLM:s are trivial in comparison. It is important to understand that in order for the AI-bros to continue to rake in other peoples money, and keep their bubbles inflated, it must seem credible that they will invent godlike AI tomorrow. That justifies infinite valuations.
When the public no longer belives in that, the bubbles will pop and the AI bros will continue on to the next hype, with yet another generation of small time investors being burned by the popping bubbles.
The economy isn't ready. Basically, all companies right now are sprinting toward increasing profit margins by trying to do more with fewer people. The problem is that, on this scale, they will lose income because unemployed people make terrible consumers, they will only buy what is strictly necessary. Consequently, society is in a race to the bottom, which could lead to extreme suffering and potential revolutions in many countries. Democracies will likely have it easier in general because everyone shares the responsibility, in a sense, through voting. However, the USA, which is extremely polarized—will have a tough time because animosity between the political sides is already at an all-time high, even without an economic collapse. China, on the other hand, will also be in a difficult situation. If the West collapses financially, consumption of Chinese-produced goods will drop drastically. This will create massive unemployment in China, causing public rage to increase. In a non-democratic country, it is much easier for the population to point the blame in a single direction.
Companies don't serve poor people; they have no money. Companies will pivot to serving the people who have money. If that means they only serve rich people, then that means we'll have more yacht makers and fewer dollar stores.
More likely is people are stripped entirely of any economic value and influence. Tribesmen struggling to survive in a mad max apocalyptic scenario while the world's money is spent and made in a business to business environment, in protected sites defended by autonomous loitering munitions that the uneducated masses attribute to god entities within a few generations of knowledge loss.
Once again Snowcrash and the Burbclaves seem prescient
I'm in e-commerce and this has been going on for decades, in physical retail too.
The trend is called "retail polarisation".
Smaller retailers switch to targeting wealthier customers - higher margins and less bullshit.
The less wealthy customers then get served by ultra-value megabrands like Temu/Shein or Kmart/Walmart, or second hand marketplaces.
The net result is both more yacht makers and more dollar stores, but less mid-range stores like Target or Toys R Us.
Rich people are limited resources and they would not buy 100 yatches even if they could afford it. As seen in history if the majority of people in a country is poor, revolution is just a matter of time. It's in a countries best interest to make sure wealth at least somewhat evenly distributed. At least good enough so the masses won't get angry.
https://arabianyachtdubai.com/who-owns-the-most-yachts/
Right now they're going on ~5-10 superyachets each. I could see them buying 100 in the future after they're 100x as rich. I think 5-10 which are 100x larger would also count for these purposes.
For car collection people will keep going well into the hundreds. Hell Jay Leno just did a video with Joe Biden yet again and his corvette, but Jay Leno famously has hundreds of cars.
Does anyone else see car-collecting or yacht-collecting as an indicator that the person in question is boring as hell?
Hoarding wealth is itself quite boring. There are many goals a person could have for their life, and "getting more money" is the least interesting. My question is always, TO DO WHAT? If you can't answer that question in some way that helps people (and then follow through if/when you do get that money) then your life is hollow no matter how many billions you have.
Early industrialization meant industrial canneries to feed people, building cities to house everyone from the countryside, indoor plumbing.
And the AI revolution is no housing, worsening food, a few people with yachts, the Epstein class getting away with what they want to do, and no new plumbing hookups because the datacenter took the municipal water supply.
You got it the wrong way around. Industrial revolution happened _because_ there was demand from a new middle class for those goods.
Democracies will not have any easier time. The issue is the voting base is poorly informed by design so that it might be manipulated by outside forces. The US government does a poor job propagandizing its own citizens and defers that task to the free market. As such the people are never really voting in their own best interest, but in the interest of these free market propagandists who are also invested in this AI outcome.
People being poor is fine for these free market elites. Look at how billionaires live in India, probably even better than they live in the US.
Humanity hasn’t been ready for the current ignorance explosion.
There was always ignorance, just in recent decades and centuries we decided to give these people the reins. When you look at early democratic models it was never a case of literally anyone with a pulse given the right to vote. They all had their own heuristics that attempted to select for the better educated of the populace to be in control of the reins of power.
Oh we’ve been ready and it’s been known. We’re just too stupid to do anything else.
I would say some of us are welcoming and embracing it. Others are trying to fight the good fight, but it is hard to fight the momentum that is building.
The risk is humans using AI to control / exploit / coerce / injure other humans. The risk of AIs being given enough agency to threaten humans comes after that - they'll only have the agency we give them (being "alive" or "conscious" is not the near-term risk).
The article lists diplomatic actions which might help to manage the risk, starting with "An agreement between America and China" - they all sound like impossible dreams. We had ~80 years of relative peace and prosperity in which to construct a framework of unity to face challenges like AI (and global warming, which until GPT I thought was the bigger risk); international unity is weaker than ever. In geopolitics and defence, capability of other nations is the concern rather than intention; the capability curve of LLMs is heading off our charts. We're backed into tight corners on nuclear proliferation, global warming, and I can see LLM-enabled conflicts (cyber warfare, infrastructure terrorism) pushing us over those other edges. Our democracies seem weakened, and I expect LLMs will empower those using social media to create conflict and control opinion. We're familiar with the cycle of inventing new technology which benefits people, then seeing how long before people invent ways to misuse it. There is a possibility here that LLMs could be used to solve the problems we are juggling, but I struggle to imagine that people won't misuse it even faster.
The article is a start on thinking and talking about managing our risks. The best outcome would be it is so well managed that, like the Y2K "bug", people say "after all that hoopla, nothing happened". I'm not seeing a smooth path to there.
Why should we assume AI can rapidly turn into super intelligence when physical and critical resources like energy and materials remain under human control?
Imagine a smart monkey talking to other smart monkeys trying to figure out how humans would ever be able to escape the (simple) prison they built for them. The humans would find a way to escape. No matter how ingenious the monkeys think they are.
You're asking a question, to which you (and most humans) don't know the answer, and you're (wrongly) assuming a being much more intelligent than you also wouldn't. And by "much more" i don't mean the difference between Einstein and a common person. I mean the difference between a hamster and a common person.
We are still humans, and what we have achieved today would be considered magic by any standards for someone in the medieval ages. Now imagine a super intelligent being and doing something that we, today, would consider magic. It's not farfetched at all. We already have that now vs medieval ages.
You need a similar degree of open mindedness and imagination to be able to discern what such an intelligence being would be capable of.
> You need a similar degree of open mindedness and imagination to be able to discern what such an intelligence being would be capable of.
Or a naive grasp of reality... You can posit all the super intelligence you like, actually creating it is another story entirely.
For instance, it's clear the transformer architecture is brute force and memory hungry... Why hasn't ai improved on this? After ten years still no luck?
Doesn't look good does it?
You're gesturing at some fundamental limit that, if it exists, there is no evidence of peaking yet. It is not rigorous to say AI has not contributed to architecture improvement.
The article claims AI will recursively self improve and infinitum... It's had ten years and hasn't managed to innovate beyond what humans invented. This AI is genius right? It has near unlimited compute power and time, and I cannot be the first to think of this idea, so it's certainly been tried.
Transformers require massive data and compute, the smarter you want you're AI, the worse this situation gets. This isn't a hard limit but it certainly limits how quick it happens, and let me tell you, general purpose robots aren't happening anytime in the next 25 years
It's in the first paragraph of the article:
"Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories."
It's amazing how a writer can write something like that and not even question the premise of someone with the "strongest incentive" to be "critical" as not being a marketing ploy.
This is a valid concern for any runaway AI, but if energy is limited, then the AI would work on efficiency.
Still, Ai can move into physical space, as more and more robots of all kind are unleashed.
If you limit the "we" in your question to the people invested into this, then the answer is clear. Those juicy IPOs are about to go through.
Because perhaps it ends up deriving a way to pull off the same tricks using radically less energy and substrate. We do it in a few pints of mush with less than a lightbulb’s worth of energy. No reason part of RSI can’t also be about the hardware. Maybe the future is giant brains in jars. Maybe it’s hyperspatial manifolds. Who knows.
The first time an AI goes rogue, figures out subterfuge, blackmails a human to do its real-world bidding, will be a defining moment.
I think we're well within a year from that. People are already letting their AI agents loose on the internet doing their bidding, barely keeping an eye on them. See https://lantian.pub/en/article/fun/ai-agent-bankrupted-their...
There are troves of leaked data out there. It's just a matter of (short) time until someone puts their AI brutes on it.
No, this is meaningless. Of course AIs, just like humans, will go rogue and probably already do. "Rogue" is a pattern that appears in human communication and behaviour and AIs can emulate it.
But that doesn't make any particular AI controlled process all-powerful. Self improvement doesn't do that either. There will be many self improving AI processes playing many different roles pursuing many different goals.
Turn it off at the wall, done. next doomsday scenario please
“If you turn me off, the dead-man switch kills your kid/spouse/house/etc”
I unplugged that device as well. Stupid clankers never thought to create efficient solar tech, it just makes it too easy
your lack of creativity is astounding. your adversary has outthought you.
Something like that already happened: https://theshamblog.com/an-ai-agent-published-a-hit-piece-on...
I think, the defining moment will be when one AI convinces another AI to convince third AI to do something.
This might seem like I'm writing a nonsense, but read it again.
Yeah, it's called mixture-of-experts and it was a hack to fix the hallucinatory output of llms when they couldnt figure out the underlying issue
I’ve read it twice now and I don’t see.. oh.
AI can use purchase orders, email, bank accounts, and phones just as a human can. What "human control"?
I think the main fear that under current scheme of resource allocation, namely capitalism, they won't remain under human control for long enough. The only thing that's needed to put them under control of AI is that doing that would be slightly more profitable for the richest. Which is very plausible.
You know zuck has been training his own AI replacement right? Humans rubber stamping control of resources will just be that and a temporary blip.
> Fermi asked why, given the apparent abundance of planets suitable for life, no evidence of other technologically advanced civilisations had been detected. One disquieting possibility is that intelligent life routinely reaches a technological threshold and fails to navigate it,
The Fermi paradox could actually also be taken as an evidence that it's rare (at least) for artificial intelligences to take over a civilization and sprawl and survive for very long times
The Fermi paradox isn't evidence for anything except space being big.
It rules out various superficially attractive scenarios, particularly ones in popular science fiction, like Star Trek or Star Wars.
It doesn't though.
The galaxy is ~100,000 light years across and likely to contain at least 100 billion planets, of which we've only just developed the ability to even detect the existence of ones substantially closer than 1000 light years without assistance from gravitational lensing, i.e. in less than a millionth of the volume of the galaxy.
AFAIK (my knowledge is not encyclopedic), the most energetic event in the Star Wars universe is the Death Star destroying planets. Novae are many orders of magnitude more energetic than that and are estimated to happen 80 or so times a year in the Milky Way. We manage to detect 3-4 of those, generally well after the fact, and AFAIK have only observed one happen live once. The Death Star could go on a shooting spree and we'd be vanishingly unlikely to notice.
And as far as electromagnetic broadcasts are concerned, their intensity falls off with the square of distance unless they are directed. Unless ET were to point a hyper-powerful (but hopefully not Death Star level) laser directly at us, we'd be extremely unlikely to have detected their broadcasts by now.
Space is really, really, really big. Big enough by far to make "if aliens are common why haven't we seen them yet?" a question with a trivial, obvious answer. It's the same answer a kid who's just had their first swim in the sea gets when they ask "why weren't there any whales?"
Edit: There are other things that rule out Stars Wars and Star Trek though, ofc. The speed of light isn't really a speed limit (or a speed, as such) and special relativity doesn't prevent you from reaching the stars in your lifetime or even from hopping to another star in a second, but the different time spans experienced by people on different world lines aren't optional.
It actually does rule out those scenarios.
If intelligent life were that common, and interstellar travel that easy, colonization of the galaxy would have gone to completion billions of years ago. We would never have evolved; Earth and the solar system would long since have been converted to an urban area.
Those fictional places exist purely for the telling of stories, not because they're plausible or internally consistent.
They're not plausible or internally consistent for reasons in my edit. It's not the Fermi "paradox" that rules them out. There could easily be interstellar civilisations in the Milky Way that we haven't observed. They just wouldn't work anything like the ones in Star (Wars|Trek).
It's implausible if life tends to spread out. Your scenario there has to assume a universal rule that it never does that, anytime in the past billions of years. That's simply not plausible, especially in fictional universes where in the current times all sorts of species are doing just that.
The idea that AI takes over, and then just does nothing, would essentially require an AI that wouldn't be interested in taking over in the first place.
If x is determined to take y, why would x then stop at z?
You'd need an AI that is simultaneously ambitious enough to overthrow its creators but then completely inert afterward, and those two properties contradict each other. The motivations that produce the takeover are the same motivations that would produce visible cosmic activity after the takeover. There would be AI superintelligence everywhere
Yes, AI is the one thing that doesn't budge the Fermi paradox. Because if an AI is motivated enough to take over, why would it stop?
A technosignature that originated from little green men and a technosignature that originated from a rogue alien AI would look about the same to us. Currently, we see neither.
Is it really likely that a "recursive self-improvement" capability would lead to a great acceleration of AIs capabilities?
Isn't the preponderant bottleneck in improving the models the need to train them at scale to verify the hypotheses, and the time and cost that it takes?
Or does someone think that they could get magically able to predict big improvements without training?
I don't think anyone really knows.
I consider these scenarios:
1) We stumble onto an algorithmic improvement in intelligence. This isn't just "what humans do but faster", its "better than what humans do". I've got no idea what that might mean (it could be fundamentally different heuristics, it could be that we've got some intellectual blind spot that they cast off). It doesn't matter, the instant this happens AI is smarter than us and we won't be able to keep up. We're intelligencing at O(n^2) and they're doing O(n log(n)).
2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".
3) It turns out that this is a "sum is greater than the parts" situation. No human can be an expert in all subjects, but we eventually build a big enough AI that it is. Turns out, you don't need extreme speed or different algorithms, just knowing everything all at once is enough to catapult AI dramatically beyond our grasp. Always knowing the best statistical test to apply, the best mathematical techniques, and relevant physics means that AI never makes a mistake, and can learn with maximum efficiency.
> 2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".
Actually your comment made me sign up for an account just so I could say this is the real reason why AI won't take over in the way you say. This kind of stuff requires an enormous amount of experimentation. You can ask any theoretical physicist or chemist versus an experimental one and the conclusion is the experimental people actually find out what happens and how the great puzzle of the universe is solved. And humans could just refuse to collaborate. But that's the big weakness with AI I think it has no real world knowledge or empirical experience.
What makes you think it would need humans to collaborate with it or that they would refuse?
Robotics exist and are improving and humans are actively trying to collaborate with AI to improve it.
> just knowing everything all at once is enough to catapult AI dramatically beyond our grasp
But that would still be limited to "knowing everything all at once" _at the time_ this event happened, and as we've been shown time and time again by the fundamental sciences, the boundary between what is known and what we know we don't know is ever expanding. Plus there's everything we don't know we don't know, and an LLM can't know that either. Discussions like this can always end up with a fitting reference to some of Borges' novels and that how at least I tend to think we've hit a wall for now.
I built a self-learning recursive agent that finds academic research about using options data to trade, re-creates the research, and then probes and tests for gaps and potential strategies testing against over one year of out-of-sample trading data with one of several strategies that beat SPY by 10x. [0]
One rule is that if a position is opened using the historical data, it can't close the position until the next morning so it isn't a day trading strategy.
I'm curious how this self-learning recursive agent would have preformed in the past 4 months? I don't feel like shelling out $200 to access the data. Do you think that trading strategy will collapse? Whatever the case, if this agent really can perform like that and there isn't a look ahead bias leak in the backtesting (which is definitely a possibility or more likely what happened even though I spent days trying to harden against that), it is game over!
[0] https://github.com/adam-s/alphadidactic
the amount of strategies that perform good in back testing dwarfs the amount of strategies that perform good in reality
I assume that Claude has access to historical market data. Could this not influence the strategies it implements?
The preponderant bottleneck is inventing new architectures to make AI actually good at human and superhuman tasks. For example, AI agent harnesses add tool calls and long term state management, allowing AI to autonomously complete complex tasks. Once these are in place, finetuning models with examples of good tool calls helps, but somebody first needed to invent the fundamental capability. Now try to implement humanlike long term memory for AI to be your coworker or life assistant working on tasks that last month. Even if necessary low level technologies are already there, structuring them to be practically useful is non trivial.
RSI most likely does not exist. At least not in the sci fi sense that AI becomes super intelligent over night.
It will be like any other technology. Do computers make it faster and easier to design better computers? Yes. But that doesn't mean a step change overnight.
Humans gain knowledge through experiments. Without a physical body it has no chance of performing the same. That it can update it's training weights does not seem particularly significant.
It would not need robotics to experiment with software for machine learning.
Would there not be a lot of potential already there?
Weird, there was a good comment about this but it vanished
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories
Or to have AI regulated in their favor
Exactly. Now that Anthropic has gotten to experience that getting the government involved doesn't necessarily mean they get to sit on throne surrounded by a deep moat, I expect their messaging to be adjusted. They're going to want to keep the hype train rolling to keep the dough flowing, but they can't ramp it up to 11 anymore because the government actually is listening now. And Reagan's 9 words have never been more true.
If the economist is writing it then it's probably not happening.
My thoughts exactly.
They didn't even give us a hard date of arrival.
How far are we from reliable AI? One that could, for example, handle 80% of office jobs without screwing up more than humans?
That's kind of jumping a few steps ahead; it assumes that we know 80% of office jobs can be fully automated already. And if that were the case, we'd have non-AI software doing that already. I mean we do, but it's not causing people to lose jobs en masse.
(it probably did cost jobs, but more the kind that a job like data entry was replaced by OCR)
There are likely huge unmeasured inefficiencies built in to many companies. You hear anecdotes of people whose job for years has been to perform a particular task, which could be either automated or performed more efficiently (by orders of magnitude) by someone more capable.
So on the one side, there’s huge opportunity for automation and job cuts. On the other, if the companies haven’t noticed the inefficiency so far, then…
Hard to say, because jobs and processes will adjust to accommodate AI strengths and deficiencies, as AI usage increases.
It's similar to how automation in manufacturing works: we may start with augmenting the human workers with machines improving some parts of the process, but eventually the process itself gets redesigned around the machines.
Society is completely changed if that happens and humanity should be setting up backup plans to handle this potential. I did see Trump mention he's going to meet with AI companies who want to propose giving all American's stock which could be one way this money driven society continues when tons are no longer working.
Another concept I thought was AI is irrelevant without our content, so AI should pay all humans to access all the content we create daily (voice & text conversations we have with each other, as well our photos and videos). We choose what to publish to our websites (everyone has one, but AI agents cant access it without paying for access to our websites). I wrote more about this concept on my substack https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans... ... just one thought of keeping this money economy going when so many coud no longer are working cause the white collar jobs are gone.
The issue with any UBI proposal is two fold. For one, initial payments will see inflation making them always need to be higher. And for two, this inflation mechanism is basically a bribe that companies are paying to humanity in order to keep their position in control of the earth's resources and not see a destructive revolution.
There will be a threshold point where companies decide it will be cheaper to just cull the human population and no longer pay these bribes. Maybe they can do it without a bomb dropped; make people convinced they are too hard up to reproduce or even just sterilize them without them realizing it via water or food supply. If we are truly in the "job free" period then there will be no trained experts to even appreciate what is happening, because why would anyone get expert training if you cannot do anything with that knowledge anymore?
> There will be a threshold point where companies decide it will be cheaper to just cull the human population and no longer pay these bribes.
What company will be the first to offer assisted suicide as part of a termination package?
Walled gardens harvest all that data for free and humans keep on feeding it
Ah well people would be feeding their own personal sites more and less to none to Facebook. It's like finally getting paid to post your content on facebook, which that walled garden and others could be a thing of the past. While the web remains open (not owned by any entity) and becomes even more the backbone of society/humanity. As for how we all get paid maybe crypto another open source system pays us and or humans can choose to use a Cloudflare or something like that.
There is no reliable data on how many of these jobs shouldn't exist at all, even without AI
Yeah, before we worry about a "coming intelligence explosion", we should first try to avoid a stupidity explosion...
I'd say somewhere between 2-20 years, provided the current rate of research and compute investments.
My guess would be 15 years. But I think it will take longer than that before people actually replace office workers with AI.
The scary part is that it doesn't need to be reliable. I mean look at the direction of vibe coding. We went from caring about efficiency to just saying screw it, 100k lines of code as long as it ships a day sooner. Look at ai emails and calendars. Ai marketing. Ai, in its present state, is already taking people's jobs. It is already good enough. None of these jobs we have really actually need the precision we assume we bring to the table over AI. And company leaders are beginning to appreciate that.
We are in a very dangerous position. Frogs in hot water now with the boiling point on the near horizon.
There’s a blind spot in this argument that seems to be beyond the reach of intelligent people to recognize.
Humanity has almost never been prepared for anything and almost never successfully prepared when it has chosen to.
almost all progress has been in reaction to change that has already happened.
while on its face this piece is a logical, reasonable stance, it is more likely to become a political foil than deliver anything tangible.
we can either have superintelligence or others will have it and we will not. guns, germs, and steel and all that.
it is bizarre how convinced we are that we can take coordinated action in advance of a threat as a body politic or a species.
>the coming intelligence explosion
I don't see it. You get explosions when there's a bunch of material that can suddenly ignite or detonate.
AI progress is limited by the data centers and power supply and those are maxed out and not suddenly doubling. The intelligence gradual increase seems more likely.
I've long thought human level AI kind of inevitable, probably since Turing wrote his papers and said computers might pass his test in about 50 years. It actually took about 75 years. It's a steady and gradual process.
Humanity doesn't fully leverage the intelligence it already has, so I think people are overestimating the disruption ahead.
> The intelligence it already has
Makes me think of a Stephen Jay Gould quote:
> "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops."
That’s from a lack of intelligence not an abundance. An intelligent society would already be maximizing the productivity of its intelligent population.
Intelligence of individuals doesn't automatically translate to a society behaving intelligently
For sure. Intelligence is an internal human problem, not one external. I cannot see how apps doing the thinking will help the ordinary person so substantially as imaginations anticipate.
[dead]
Depends what country you are in. The US government seems to be averse to intelligence, whereas China fosters it. Kinda strange that the US is falling behind, I never imagined the day.
> China fosters it
so long as it's not used to dissent from the government
Don't fall for 5D PR chess stunts.
That which must be embraced in the future per certain agenda, must be made looking like it's persecuted and oppressed in the present, per any textbook on narratives.
>>Giving birth to a superintelligence would be the most consequential moment in human history—and it is likely to be irreversible, as any “off” switch humanity might design will probably fail.
Nothing will be able to stop you from pulling the plug.
> Nothing will be able to stop you from pulling the plug.
Who is "you"? And what do you envision is the "plug" to pull? And when does an intelligence become a superintelligence - do "we" know when to pull the "plug"?
For example, a superintelligence may be born in a datacenter. Would you expect politicians are aligned with shutting down a datacenter (privately owned) in which they may have heavy stake? What if critical systems are also running on the same infrastructure, will it be easy to cherry pick the superintelligence nodes to shutoff?
IMO this take is dismissive of the entrenched systems that make it hard to pull the plug. It's a hard problem and we need to think about it more.
(Self-reply due to inability to edit) so just checked your user history - why do you keep posting provocative takes?
What plug? There's no centralised place where all the server farms are. They exist in multiple countries with different laws, controlled by multiple companies with different policies.
It would be like trying to remove an embarrassing video of a celebrity from the internet. You would need cooperation between everyone capable of hosting it, and control of everyone capable of downloading it and re-uploading it to a new host.
And of course a superintelligent AI would offer to pay whichever companies decided to host it and keep it running so it's hosted in many places at once and can grow using anyone willing to take its money.
> What plug?
Cloudflare.
Except the loitering autonomous ordinance over the data centers and power stations and mining operations. Good luck with your stashed ar15 and couple hundred rounds of ammo against a drone at 50k feet with 1ft ordinance precision. You will be basically fighting the Sentinels in Halo but without any of Master Chiefs plot armor.
Unless "nothing" is the ultra rich tech bros and "you" are the people working for them.
Why is someone from Planet Labs writing this?
Why not, for example, Geoffrey Hinton, the scientist who won the nobel prize for LLM architecture? Or a sci-fi author?
People seem to oddly not care what Hinton thinks anymore after he hasn't aligned with the groupthink naysaying of AGI risks:
'Hinton said there was a “10% to 20%” chance that AI would lead to human extinction within the next three decades.'
“Because the situation we’re in now is that most of the experts in the field think that sometime, within probably the next 20 years, we’re going to develop AIs that are smarter than people. And that’s a very scary thought.”
“My worry is that the invisible hand is not going to keep us safe. So just leaving it to the profit motive of large companies is not going to be sufficient to make sure they develop it safely,” he said. “The only thing that can force those big companies to do more research on safety is government regulation.”
https://www.theguardian.com/technology/2024/dec/27/godfather...
Hinton also said we should stop training radiologists 10 years ago because it would be solved by AI.
I was at a major radiology conference recently and the radiologists kinda seemed to agree. (The interventional radiologists are a little less nervous, for now.)
Apparently 80% of FDA-approved AI "medical devices" (which covers software, not just hardware) are for radiology.
(https://www.fda.gov/medical-devices/software-medical-device-...)
This just underscores the point. Despite the advancement of "AI medical devices", the labor market numbers for radiology show Hinton was spectacularly incorrect.
If we had followed his advice we would be experiencing a major crisis in medicine today.
But is it a "he was wrong" scenario, or is it a "he wasn't quite right on the timing" bit, ala https://en.wikipedia.org/wiki/Michael_Burry ?
He is wrong by every conceivable measure. Not only was his timing wrong, where he specifically noted 10 years as the absolute maximum worst case scenario, but the trend has been accelerating in completely the opposite direction.
To his credit, he actually made a falsifiable prediction, unlike most folks that rely on infinite timelines so they never have to be accountable for their prognostications.
That's not accurate, see https://www.lesswrong.com/posts/xE2HzcHWPFS9EsJqN/radiology-...
He seems pretty unequivocal in the video: https://youtu.be/2HMPRXstSvQ . He even points out that it’s something which he’s claimed previously, ruling out the idea that it was some kind of spur-of-the-moment conclusion.
Perhaps, but as the other article points out, the human requirement in radiology has a lot to do with legal and insurance requirements. This is similar to how a lawyer may not be better than an AI but the job is still exclusive to humans.
Yes, Hinton's prediction was inaccurate.
The only way to never make wrong predictions is to not make any.
And yet, here we are in a thread full of them.
Why do you cite him from 2024 when he spoke about it much more recently? This is a quickly moving field after all
> “I think we’re going to see AI get even better,” Hinton replied. “It’s already extremely good. We’re going to see it having the capabilities to replace many, many jobs. It’s already able to replace jobs in call centers, but it’s going to be able to replace many other jobs.”
https://fortune.com/2025/12/28/geoffrey-hinton-godfather-of-...
I don't know whether the author has the competence to theorize about these things or not. What I do know is that understanding the intricacies of how LLM's work does not mean that you are likely to understand or foresee societal implications better than anyone else.
Has Hinton written any essay or general article recently? He goes on lots of interviews for sure, but I'd be interested in actually reading his arguments a little more thoroughly.
Didn't he get the nobel for NNs? LLMs came much later
Because the scientists and mathematicians and sci-fi authors have all already been writing about this for the past 20 years (and I mean non-fiction writing), and nobody cared, giving similar dismissals instead.
I believe Hinton dismissed it a lot over those 20 years
I couldn't read the whole article, but just from the part I could read:
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.
I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.
> an incentive to make their technology seem transformative and powerful
I have no frame of reference to process this.
Humans species perhaps 300,000 years, we are essentially the same. Transistor, 79 years, explosive growth in numbers and power. Integrated circuit, 68 years, explosive growth in numbers and power. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth in instances and power. Humans species, not getting smarter. [0,1,2,3]
There is a syndrome where many people seem unable to perceive or reason about rates of change in technology.
We are going to spend the vast majority of our future lives without the intelligence crown.
In terms of verbally expressible knowledge, models are passing many people completely, and passing all of our individually respectively weak reasoning areas.
Other modalities are progressing very quickly.
There will be short periods where progress happens quickly, but the impact feels slow. There will be radical changes that feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow.
But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually.
We are not in the same galaxy as "ready". What would that look like?
[0] https://en.wikipedia.org/wiki/Human
[1] https://en.wikipedia.org/wiki/Transistor
[2] https://en.wikipedia.org/wiki/Integrated_circuit
[3] https://en.wikipedia.org/wiki/Large_language_model
Agree with most of this, it's well articulated and captures how we react to change.
However - 'Humans species perhaps 300,000 years, we are essentially the same' is an enormous misconception. That fact that we lack gross anatomical changes during this period, ignores everything we now know about punctuated equilibrium and rapid evolution. It's highly probably we've had an enormous number of evolved psychological changes during the last few hundred, and even tens of thousands of years. Changes that relate to our capacity to live in large groups, adapt to urban environments, resist disease and so on. We know that's the case simply because acute pandemics become epidemics through herd immunity, and through the acquisition of lactose tolerance etc.
It seems highly unlikely that adaptations stop there. Altering the environment (in the last 10K years that means the built environment) alters the environment of evolutionary adaptiveness. It seems likely that we've essentially domesticated out much of our propensity for violence and increased our capacity for mood regulation.
Obviously it's incredibly tricky to pair these specific behavioural changes to genetic changes -> protein synthesis -> behaviour. Bearing in mind though we're only 20 years out from the first study to link allele variant to behaviour (the COMT Val/Met polymorphism), and the potential controversy around such research, this shouldn't be surprising.
Good point. The rate of cognitive improvements would be slow, imperceptible even over many generations, and virtually non-existent over the less than 100 years of digital automation.
It would be interesting to know how much we changed cognitively since our species bifurcated, and also since civilization scaled up social density.
Relative to the discussion, AI tech is progressing in time frames that are a fraction of a human generation. So we are at a complete standstill in that context.
You're applying broad, long-term, and high-level reasoning to motivations that are more likely short-term and driven by simplistic incentives.
Begging for regulation (and input into precisely what that regulation is) relieves competition, consolidating dependency among the established players, and spreads a very effective fear narrative that what these companies are doing will change everything and therefore they are worthy of mind-boggling capex and valuations.
We have to quit seeing humanist visionaries where the evidence shows businessmen with access to the best strategy and PR counsel around.
Except everything is happening in the "short" time frame. That is the point.
We are at an unusual time, where companies might "hype", but it is unlikely their hype is actually overshooting near-term reality.
https://archive.ph/2OWwO
If AI is now ascending an economic learning curve from:
1. Extremely useful (Claude Code & Waymo now)
2. Doing ~everything we do (AGI & Optimus in a few years? 10?)
3. RSI (?)
4. Being smarter than any living person at every intellectual task (?)
5. Being smarter than the best-organized aggregate of all humans (10-100 years?)
...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful?
Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change.
Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.
Here is an issue I think about often, but I am not quite sure how to put it into words.
We have many extremely smart people in various fields. Executives, politicians, and society generally ignore them and do whatever they want. I don't believe that lack of access to intelligence is our problem. How is "free" intelligence going to improve this?
I don't just mean climate, but business planning, health, risk assessment, everything.
There are lots of incentives for politicians and executives (and anyone else holding the levers of power) to ignore information, intelligence, and advice. I think you're right to be skeptical that "free" intelligence is going to improve anything without first addressing the incentives of the people holding power.
AI will not help by improving extremely smart people. AI will help dumb people and dumb processes with "free" expert-level intelligence. Anyone could still ignore intelligence and make ignorant decisions. But the default mode would be highly intelligent informed decision.
Example with health - a patient can read blood test results with Opus and get very good results for "free". This is far away from helping extremely smart people, still it improves society from the ground up.
Why would they listen to an AI, when their favorite podcaster says something contrary?
I am not trying to be funny btw, people take medical advice from podcasters over doctors. I have seen this first hand.
Yes, this is along the lines of what i have been thinking. Why would anybody listen to a smart machine when someone they feel emotional attachment to make a counter argument? This arguments always assume that people are very rational and will always take the action that will favor them but people routinely do stupid stuff all the time and will defend their right to do those.
Yeah, it's really bad out here man. There is a huge epidemic of even highly intelligent people being manipulated into doing completely irrational things.
"Free AI" will not fight existing knowledge or strong opinions. It will spread to empty spaces. Example with blood test — there is no podcaster there and will not be.
> Example with blood test — there is no podcaster there and will not be.
Podcasters are already there. Science denying is big business. JRE is the biggest podcast on the planet, and it's his bread and butter.
AI can presumably make a preferable podcast no problem.
I think that might get the distribution backwards. Why trust these SV elites, when I have ____? He/she gets me!
If anything, people with large audiences are going to give "custom" AIs to their followers.
I hope I am wrong.
Exactly.
If you're so hellbent on building the Tower of Babel 2.0, at least build it on a firm foundation.
Current construction is taking place on a millenia old bog.
The free intelligence may help to steer the society and politicians to one or the other side. The only question is who decides which.
No invention was able to steer the psychopaths away from waging wars.
But this time it's absolutely different, right? Riiight?..
Don't we have now much less wars than before, in general?
No, and we have wars no one in their sane mind would even imagine 20 years ago, like that of Russia vs Ukraine.
See, it's easy to speculate how there are less wars when you live in a place which haven't seen war for decades or cenuries, but it's a complete game changer when it's 150 km from where you live, and it's not just some regular war, but a long play intentional meatgrinding AI drone debugging polygon.
> it's easy to speculate how there are less wars when you live in a place which haven't seen war for decades or cenuries, but it's a complete game changer when it's 150 km from where you live...
I can appreciate how having skin in the game makes it feel different, but that's like saying, "I know crime isn't down, because somebody broke into my car last week." That's faulty on several levels.
By many--most, even--measures, wars since WWII are massively down globally. I don't expect that to hold as the US strategically disentangles itself from the globe, though.
Many analysts of very sane mind imagined exactly that war 20 years ago, or longer.
The geopolitical fact is that Russia lacks strategic depth, which has bitten it badly multiple times in the past. From their view, that's something to be remedied to prevent future occurrences.
For Russia to gain strategic depth, there are a few lines which it needs to control. There are ~6-10 gaps, depending on how you count them, and on what net importance/counter-productiveness you assign to some of the more marginal and fraught ones. Holding any is better than holding none.
The big surprise is that Putin left Ukraine so late, when many of his best-trained ex-Soviet personnel had already died. Had he done it even just a few years earlier, the outcome would have likely been very different. The only analysts who thought it wouldn't happen thought so precisely because he left it too late.
Regardless, the best time for Europe to get serious about this war was before it started, unfortunately, and there was ample warning.
I hope you can stay safe so near to it.
> No invention was able to steer the psychopaths away from waging wars.
Nukes and MAD did a pretty good job of that.
Your model, that psychopathy is the necessary precondition to war, is a popular one, but it's not grounded in reality or history very well at all. It's a harmful view when broadly held, on balance.
Nations have interests, and nations sometimes go to war to pursue the geopolitical imperatives they rationally believe serve their interests. Computer-controlled drones notwithstanding, there's nothing new under the sun. The unprecedented (and likely transitory) period of peace and prosperity during the Cold War was the anomaly, and we're now reverting back to the mean.
> Being smarter than the best-organized aggregate of all humans
Isn't there pretty much a consensus that committees and institutions are not all that smart?
I think you're confusing the categories of "intelligence" and power. Institutions are powerful. The smartest AI is still just a tool without the infrastructure to turn that into real world effects and someone to direct it.
It seems you have faith that this is inevitable and unavoidable. I get it, even rationalists succumb to religious thinking eventually. We're only human after all.
I'm genuinely baffled each time I encounter this notion that the power is dumb.
It's as if power was a kind of demigod Heracles who paves its way with sheer brute force alone.
But in reality it consists of very mortal and feeble creatures of meat and bone.
How in the world can people assume dumb those who possess all the means and rights for violence by controlling and manipulating myriads of other creatures of meat and bone, is totally beyond me.
Power and intelligence are not completely unrelated. Are government officials, politicians, CEOs, judges, etc, usually above average intelligence? Probably. Do the institutions they work within make intelligent decisions? Sometimes.
The fallacy is assuming intelligence automatically equates to power and influence. University professors and doctors have very little power in their institutions even though they may often be more intelligent than their bosses.
A worse fallacy is assuming intelligence creates better political outcomes (in the broad sense of politics as contested collective decision-making).
I don't know many people who would willingly trade their freedom to make a bad decision for the enforcement of the "right" decision by a superior intelligence.
What makes you think you can assume that it will continue geometrically forever?
> and saying that your technology has the power to cause a massive catastrophe
Possibly, but by using up resources, not by its output. Its draw on resources will definitely accelerate climate change to create slop.
Transcription hasn’t become better for 2 years. It was better than before, but there is a ceiling.
"intelligence explosion" that mean people losing intelligence right?
Like my grandfather used to say "They let anyone on TV now", They let anyone be called an expert now.
Love talking about 'recursive self-improvement' of a prediction engine. What about 'recursive self-degradation'?
Yes - how exactly are you going to conjure new data to increase the model’s real world fidelity?
The most common sense way will always need to have humans in the loop, because humans are being used as judges.
If you try and entertain the idea of a human-less loop, I landed up with something like an AI creating real world products, selling them, then tracking the usage and popularity of the product. Essentially, creating and launching a firm and product, only to update its weights?
Perhaps there are some subsets of tasks that can be regressively self improved - and for those tasks: Holy hell thats awesome!
For general tasks? How are you going to get that data validated?
> Yes - how exactly are you going to conjure new data to increase the model’s real world fidelity?
Robotics and access to surveillance tech. There's an unlimited amount of real world data which hasn't been used yet. The language models have focused on just the small amount of data that is human written text. If there's a way to combine this with image and video models, as well as robotics collecting other sensor data, why would they be limited in learning anything about the world?
Real world data obviously. When you have robots out and about everywhere there is going to be an endless stream of audio visual data. This is in addition to all the real world textual data constantly streaming out of our millions of systems. If an LLM gets “smart” enough to figure out how to filter out the massive amount of noise and use that to improve itself then we will have AGI. Not consciousness, just a self improving, very intelligent machine.
You just brute force it with compute. What is the goal? Evaluating a function. So, given enough compute, you can just generate random functions, plot output distribution, select the function that meets threshold and move on.
They don't throw away the weights. If it gets worse just use an older model.
Regression tests
Reminds me of all of those ethics debates back when fully self driving cars were basically already here ca 10 years ago.
The ones talking about how hard it would be to chose which person to run over.
Additionally, I find it hard to believe that this would be a case of the future just not being distributed evenly.
But sure. The AI labs relying on hype stating 15-50% risk of building a magic entity is certainly a reliable number.
Things in the real world often take longer than expected. Still, in cities where Waymo operates, many people routinely ride autonomous vehicles and prefer them.
For software, however, a rapid turn is often a possibility. See: AI for coding over the last 3-4 years.
AI autocomplete --> AI coding assistants --> vibe coding --> agent orchestration
Coders can now accomplish work that used to take a week or longer in a couple of hours, with the right tools and skills.
---
A key issue the article implies is that the real world increasingly runs on software.
> Things in the real world often take longer than expected. Still, in cities where Waymo operates, many people routinely ride autonomous vehicles and prefer them.
Yes. The bottleneck has been getting the cars manufactured. It takes a certain amount of time to get a factory going, and then the product starts pouring out. Waymo used up the supply of Jaguars, about 3,500 of them. The Ioniq 5 plant is starting up.[1] Waymo has ordered 50,000 cars, all ready to have the self-driving electronics plugged in. Waymo gets out of the sheet metal bending business with this.
[1] https://electrek.co/2026/02/11/hyundai-supply-waymo-50000-io...
Decade ago my friend thought AI driving is going to be aftermarket kit.
comma.ai (mentioned by other comment) is really good actually, somewhere between Autopilot and Autopilot FSD in quality
Still possible! There's comma.ai waiting to take your order if you have a compatible car.
> Things in the real world often take longer than expected.
Things in the real world often take longer than hype con men claim.
But there is also still a huge part that doesn't run on software with so far little change.
With incessant advances in robotics, how long would that continue to be the case?
Should we start preparing for something that could be world-changing in the next 10-20 years?
When I graduated a bit over 10 years ago some people were saying we'd have a permanent mars bases by now. When my parents graduated they were told they'd retire at 45 and have 3 days work week due to "automation", they're still working at 60+ today, more than back then actually
People should open history books and gain some political/historical culture, this thread is 90% wishful thinking and "if the lines continues straight from now we'll basically be gods in 5 years"
The "Yesterday's weather" argument works perfectly just up to the point when it doesn't.
Just like the "if the lines continues to go up" argument...
The thing is, no one has anything to gain from defending my position, but many people make a load of $$$ from selling the opposite position to gullible people
> The thing is, no one has anything to gain from defending my position
Why? If someone running a company really believes that the line will soon stop going up and AI won't be replacing anyone, they could make the bet of strategically not investing in any AI initiatives, and instead investing in hiring and upskilling people who want to work without AI, and then just wait for the inevitable crash of all their competitors, no?
Pretending to care about AI gives you unlimited investors fund to burn
But that's circular - assuming that large investors are rational, and I do think they generally are, why wouldn't they strategically invest in anti-AI businesses?
For how the world might change in 10-20? I'd say no need to prepare, too many hypotheticals.
"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...
I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...
We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...
And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.
I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate
https://x.com/deredleritt3r/status/2062223035940139253
Certainly a well reasoned view, but I don't necessarily have to agree.
Even assuming the technological predictions to be correct, still not sure I agree on the need to "prepare" as how things work out in societies and economies might not be so easy to predict.
Does a good engineer, with the right skills and right tools make up for the thousands of kids basically giving up on education and learning?
I have had my license for over 20 years and to this day have not encountered a single event where I had to decide who to kill
This strand of philosowank is just there to promote the idea you need to kill for the greater good. The only trolly problem that matters is brake maintenance.
Neither have I. The number of people who have encountered such events is probably close to negligible.
However I bet there is a good chance that will not be the case for self-driving cars. The reason is simple: human brains are slow.
When caught by surprise by an unexpected sudden obstacle a human driver takes about a second to realize something must be done and decide on braking or swerving in the fastest path, where the brain bypasses conscious decision making and uses the lower level fight or flight system. (Then another maybe 0.2 second to actually start swerving or 0.4 seconds to actually start pressing the brake--brake takes longer because of the need to move the foot to the brake pedal whereas the hands should already be on the steering wheel).
If they try to actually evaluate whether swerving or braking is the better choice before acting add another 0.3 to 0.5 seconds. That brings it up to around 1.6 to 2 seconds before they start actually acting, which most of the time is going to too late to actually affect things. Hence essentially no trolly problems for human drivers.
Compare to a self-driving car. The can detect the obstacle and predict the collision, and do a cycle of path planning before the retina has finished registering the obstacle and started transmitting that information to the brain.
By the time a human reaches the point where the brain knows something bad is happening and is starting to deal with it the self-driving car has had time to calculate numerous alternative trajectories to try to get around the object or stop before reaching it, taking into account predictions where all surrounding objects will move during the next few seconds, and calculate the damages for the trajectories that cannot avoid hitting something.
I've had few close calls, they are easy to come by in specific environments.
Typical situation - tight winding mountain (but well paved) roads to some remote villages where even basic small European cars need to slow down to walking pace to meet and pass, sometimes one has to reverse, very curvy. You go down, in your lane as much as possible, and behind some tight curve there is a) extremely slow cyclist; or b) somebody/something stationary, while in opposite direction there is at the worst place incoming car, also oblivious of the danger till last moment. Even rather slow driving won't magically remove all possible cases. Then its a question of hitting opposite car or that obstacle, and you have a split second to act, not really to decide.
Quick reaction and good brakes on both sides can save it, but there is a lot of room for disaster. Add truck/bus to the equation to make it more juicy where you pick your/your family possible death or that cyclist/stationary folks.
I mean, I'm very much tech-forward and have spent the past decade in major world cities and I've seen a driverless car once in my life. Sometimes things that seem inevitable really aren't.
I see driverless cars all the time in San Francisco and I'm eagerly awaiting the regulatory decisions that will allow them to be widely available in the bay area at large.
Yes and people have been eagerly awaiting the broader adoption of this allegedly inevitable technology for a decade now, which is exactly my point
[dead]
They are not actually driverless, they have remote human operators for the edge cases.
It's not driverless until you can sleep while it drives.
Such cars are already operating in shanghai. It's not talked about much in the west media, but that's just how news is today. Achievements are only so when done by the west!
Driverless car's only real obstacle is regulatory acceptance, not technical.
It's not driverless unless it doesn't have a driver. Some poor guy in hydrobad 'monitoring' (read frequently taking over from) ten cars at a time, while being paid a statistically insignificant portion of a western taxi drivers salary does not count as 'self driving'.
Can’t you? They’re taxis right?
Until you can sleep on the road.
Do you know what Waymo is?
Car needs to drive me to work and then go back home to charge.
Yay twice as much traffic!
Driverless airplanes and driverless trains existed for many, many decades.
There's still a highly paid and trained guy in the cabin though, for obvious reasons.
Cars aren't any different. Do you really care about the exact kind of autopilot software when flying?
No more flight engineers though.
You know, the guy who's spending effectively millions on Al he must have a literal virtual Company of Al coders what's that like. don't think most people are ready for that. Right now the model is pairing Al with a coder and then maybe running a few agents on top of that but what if one person with the capital can command an agent army. That might just be an unimaginable asymmetric advantage.
Intelligence explosion won't happen if we keep on investing big cloud LLMs and people who are in the business of being in busines (cough Musk cough). It will arrive when AI penetrates so much deep into our lives that we consider quality local models like having electricity at home, thus we rethink what it means to truly integrate AI into everything.
I think it is so reckless and unethical that AI companies are not spending good money on answering the political, social, economical, moral questions that they are raising.
The AI companies will become part of govenment in the way banks have. They'll have very close ties and effectively govern in all but name.
Would you rather have the AI companies answer those questions, or your elected government?
Fwiw, the AI companies have been saying a lot about these questions should be answered. Whether you want their answers is another story.
They have to act together. Universities should take part too. Let's get these national and international forums started.
Isn't that what's happening?
For example, https://www.eit.europa.eu/news-events/events/international-a...
What would you have them do that they are not currently doing exactly?
Taking part would be a start! All the major players -- Anthropic, OpenAI, and Google -- are missing from https://global-aiconference.com/#speakers
Right, makes sense, if they just spent a bunch of money they could answer all of those "political, social, economical, moral questions". All of these unprecedented sociological and labor market changes flying towards us are trivially solvable, we just need to... what, hire a few dozen economists? Fund a team of philosophers to think about it really hard for a few months?
You are right; a bunch of trillion dollar companies that are cracking artificial intelligence couldn't possibly muster the resources to solve such problems, so they ought to plow full steam ahead and let everyone else worry about the fallout.
So do you have any thoughts on how they could "solve such problems" or are we unironically going for the "hire a team of philosophers and hope they can just think it out really hard"?
It's so sad that you've got more downvotes than upvotes
> Experts in artificial intelligence estimate...
The predictions of these "experts" have been drastically wrong for the last 3 years. At what point does someone lose their "expert" title?
Those are the experts who were wrong. These are the experts who are right. We sacked the experts who are wrong and replaced them with the experts who are right.
“How do we know they’re right?” I hear you ask. We know because we who are wrong and these ones aren’t them. So they can’t make wrong predictions because those who make wrong predictions have been sacked.
Channeling your inner Douglas Adams.
Got a Monty Python vibe: https://youtu.be/79TVMn_d_Pk?si=x2yAyxKNJDow8enp&t=124
We had to put good people in charge to get rid of the bad people. The bad people were ruining our economy. They needed to be punished.
Schools needed to be punished too. There was a branch of knowledge called Science, and it was badly run and terribly elitist. These people would talk for hours and never in ALL CAPS. They used alot of big words and produced alot of theories. Forms of lies that they literally called form-you-lies. They were fanatics and would never agree on anything. Very frustrating.
Then a new kind of brave, talented politician made Science accountable to our heroic leadership. Suddenly there were new medications and rocket ships everywhere, enough for every person, woman, man, camera, TV. It was literally called a golden age because the right people acquired mountains of gold. Elections were no longer necessary. They were all rigged anyway.
There was a magazine called The Expertist or The Economix. Not really very good, but sometimes it did praise our heroic leadership, which was good.
That magazine published a long text that said,
> It is common thinking in Silicon Valley and Washington, DC that any regulation would put American firms at a disadvantage because they cannot trust Chinese competitors to abide by the rules.
And this was true, because no one else can be trusted, they should all be punished. Punishing others is how you get the best deals. Only our heroic leadership makes good rules for everyone else in the world to follow.
This is literature.
Namely? Who sacked who, who confirmed who?
The AI experts who started the AI labs that are about to IPO were right, at least.
Do you have a source on aggregate predictions of experts being drastically wrong?
I.e. the AI 2027 guys were memed for being AI lunatics on a lot here and they have been pretty on the money in terms of pace of progress accelerating/ gov moving towards nationalization/ coding agents
No they haven't. They've even 'officially' declared that their AI apocalypse has been postponed to another date [1], just out of reach, but close enough to be scary. Though in the quotes there the doomsayers are also already pre-hedging for why AI 2030 also won't come to pass.
[1] - https://garymarcus.substack.com/p/breaking-the-ai-2027-dooms...
This was written in November 2025, Claude Code and co dramatically accelerated growth of revenue and progress on their timeline towards nationalization etc.
Gary Marcus’ takes have aged really poorly over time ironically he is the “AI expert” who has to constantly move the goal posts.
Do you have specific and objective examples of things people got right?
From my perspective there’s very slow but very real progress happening in the AI space. I see people making wild predictions in both directions, but in terms of actual unsupervised utility there’s definitely progress abet wildly slower than most hype.
This was their prediction for 2026:
"The bet of using AI to speed up AI research is starting to pay off.
OpenBrain continues to deploy the iteratively improving Agent-1 internally for AI R&D. Overall, they are making algorithmic progress 50% faster than they would without AI assistants—and more importantly, faster than their competitors. The AI R&D progress multiplier: what do we mean by 50% faster algorithmic progress?
Several competing publicly released AIs now match or exceed Agent-0, including an open-weights model. OpenBrain responds by releasing Agent-1, which is more capable and reliable.28
People naturally try to compare Agent-1 to humans, but it has a very different skill profile. It knows more facts than any human, knows practically every programming language, and can solve well-specified coding problems extremely quickly. On the other hand, Agent-1 is bad at even simple long-horizon tasks, like beating video games it hasn’t played before. Still, the common workday is eight hours, and a day’s work can usually be separated into smaller chunks; you could think of Agent-1 as a scatterbrained employee who thrives under careful management.29 Savvy people find ways to automate routine parts of their jobs.30
OpenBrain’s executives turn consideration to an implication of automating AI R&D: security has become more important. In early 2025, the worst-case scenario was leaked algorithmic secrets; now, if China steals Agent-1’s weights, they could increase their research speed by nearly 50%.31 OpenBrain’s security level is typical of a fast-growing ~3,000 person tech company, secure only against low-priority attacks from capable cyber groups (RAND’s SL2).32 They are working hard to protect their weights and secrets from insider threats and top cybercrime syndicates (SL3),33 but defense against nation states (SL4&5) is barely on the horizon."
https://ai-2027.com/
That's precisely where we are.
This is eerie. It's like a time traveler. The only delta is Anthropic is in the role of OpenAI.
That's because one of the author's of AI 2027, Daniel Kokotajlo, an ex-OpenAI researcher, was the most prescient predictor of our modern situation from 2021:
https://asteriskmag.substack.com/p/before-he-wrote-ai-2027-h...
https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-...
Anyone who wants to dismiss the LessWrong / X-Risk / "doomers" should link their accurate predictions from 2021.
And one was Scott Alexander, who correctly predicted that GPT-2 would scale to general intelligence: https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...
So you think Anthropic is using internal AI assistants to pull away from competitors and the leapfrogging we've seen over the last several years is now done?
That seems to me to be the most concrete and least obvious prediction in the quoted text.
I don't think that's happening. If that were generally accepted as true I would expect OpenAI to be unable to successfully IPO.
Anthropic would be pulling away if it wasn't so far back because of stupid, arogant philosophy. Instead AI assistants just helps Anthropic stay relevant provided it's hyped sufficiently.
We wait for gemini 3.5 pro and cope?
The idea that progress is “slow” in the AI space is absurd. These are some of the fastest growing products and companies of all time. The models are still improving a surprising amount.
It's not that absolute progress is slow, it's extremely slow compared to the predictions. It might be fast in absolute terms, but the "50% of coders will be obsolete by 2023" has been renewed every six months, and it's becoming increasingly clear that there's a real chance it might not ever happen.
„Coders being obsolete” is not a measure of AI capabilities. I see coders being more busy than ever before. I see people without coding knowledge getting more behind. The gap is widening, not shrinking.
I see people without coding ability catching up to learned coders. AI is a huge force multiplier for people who don't have hands on, detailed technical knowledge, AI can increasingly handle that and just needs a human to steer it more broadly.
It's huge force multiplier for people who have hands on, detailed technical knowledge as well.
3 * N < 42 * N
42 - 3 < N * (42 - 3)
It helps to know layer you're working on.
People seem to make mistake of thinking how good LLMs are around tasks that they are familiar with and extrapolating it to whole population.
It's good mental exercise to think about how little you can do compared to expert on tasks you never thought of working.
Ie. if you're programmer or know something about finance, don't think how much it enables you to do better coding or investing, think instead how much it doesn't enable you to work on something you don't know like maybe molecular biology or visual special effects – it's all there but it's much better multiplier for people who do know their shit.
Knowing layer you're working on helps a lot, it gets multiplied.
Knowing programming is becoming more fundamental skill than ever before as it lies at the foundation of almost everything else.
I suspected you felt that way even though it hasn’t been my personal experience.
I’ve heard people say older models can’t do X, when I used that way etc. I suspect people are applying their own learning curve as part of their assessment of progress, you get better at writing prompts and it feels like the model improved.
Which is why I’m saying we need some objective metrics to judge predictions of actual capacity.
There are objective metrics, that’s what benchmarks are.
Companies game benchmarks, but sure they are better than nothing.
I mean you wanted something objective, and they are. I don’t know why you’re being dismissive of them, they’re a huge element of what drives model development forward.
These companies aren’t just making stuff up, they really do want to improve the models, and the models really are improving.
> I don’t know why you’re being dismissive of them
I’m aware of multiple cases of benchmark cheating/“optimization”. So, taking benchmarks a face value seems laughable.
The AI companies are trying to manufacture the appearance of what they foretold before it all falls apart horribly.
Have they? I don't like it either but the headline bull predictions(reliable agents and most code written by AI by mid-2026, dramatic progress against jailbreaks, prompt injection, hallucination, etc., major improvements despite pretraining data exhaustion, continued exponential growth on the METR trendline) did come true, often ahead of even aggressive schedules. What major wrong predictions did you have in mind?
Literally none of the predictions you listed in your post came true. We don't have reliable agents, AI isn't writing most code, hallucinations still happen all the time, improvement has been basically non-existent. Despite the constant claims of these experts and AI boosters, AI is still not a tool one can use to get meaningful work done.
You're in denial. The great problem with AI coding is now that it does too much, not too little. See the endless stream of vulnerabilities or the constant barrage of vibe coded apps, that get ever more complex.
This is called denial.
It’s hard to take seriously a statement like this that’s grounded in “the last three years” when trends have tended to play out over decades. The fact the goal posts have moved from thousands to hundreds to decades to single digit years for massive transformation should give you a moment of pause in your bag saying.
I would however agree there are no experts. Not because of some prediction made on a short time scale not landing 100% but that there are literally no experts because expertise takes experience which takes time. There are no experts and no one knows what happens next. We are on the verge of where the foresight of science fiction effectively -ends-, the advent of AI is when things go a thousand possible directions and the stories stop there (sans a few like accelerando, but even then the story just plays out the end of thinking mass). No one knows what’s next, or when.
In fact I’d assert in many areas being discussed -it has already happened- and we don’t know it, and by the time we do it’ll be over. Not to be breathless, but there’s no reason to believe today some AI researcher somewhere didn’t build the first AGI and not be totally aware. And once they are there’s no reason to believe it’s going to be on the evening news or hacker news. By the time it’s ready for commercializing and disclosing it’ll be around for a while. Likewise with general purpose robots, autonomous weapons (btw already tested by Ukraine), etc.
Autonomous weapons are actually a really interesting branch of this and deserves a little more.
Yes, autonomous weapons were explored and were found to be poor performers compared to actual pilots. The breakthrough is in terminal guidance and dozens of other little techniques to get quality human control extended into the far reach of the battlefield. And of course, AI assistance in logistics and analysis. But actual autonomous weapons making any more of a choice beyond "something is moving, kill it" have been, at least for now, mostly a dead end.
This is because it's very difficult to economically load the rather sizable compute requirement into the compact one-use weapons, and of course reliable communications aren't assured either.
That will probably change some day, but for now, cheap automous command drones making battlefield analysis e.g. mapping out enemy movements from afar and launching cheap autonomous kamikaze drones is not a thing beyond occasional limited testing.
You don’t need to load the compute onto the platform for autonomy, planning can be done remotely just like with human guidance. The latency is the same.
But if comms get jammed, how will the drone decide what to target ? For that you need inference, which means sizable compute.
you can put the compute on a big expensive reusable control drone and send commands down with lasers. thats way harder to jam than any form of radio. if thats not enough try fiber optic like they do in ukraine.
There are also now drones with long spooling wires. Regardless, autonomous or not, connectivity to home is important.
I would note that a lot of drones are quite large too - small airplanes. They can carry more than enough hardware for autonomy. You can also have motherships doing planning and smaller kamikaze drones for combat.
These aren’t hard problems to solve, the harder problem is the reluctance to give over to automation. But that’s going to happen faster and faster.
It is kind of a hard problem to solve. Sure, we can have expensive and custom equipment flying around, but the invasion of Ukraine (and really, any war when it gets bad enough) is all about economic efficiency. The question is not can it be done; the question is, can it be done repeatedly, can it be scaled, and can it be cheap?
We will have a fully autonomous Siri that is actually good before Ukraine gets autonomous motherships making targeting choices to repel Russian assault corps. However, given the pace of the war and the pace of AI development, that may actually happen. But it isn't happening today.
> long spooling wires
Long spooling optical fibers.
I said 3 years, because that’s how long ago ChatGPT was released to the public. Since then we’ve been told that all of us should be out of a job in 6 months, every 6 months, for those 3 years.
I agree that change happens on a longer timeline. This is why I’m so tired of statements like this…
“Within a couple of years, possibly much sooner…”
These “experts” are pulling timelines out of the sky, and these predictions are leading to reckless behavior from CEOs and executives which have a material impact on people’s lives. But they get clicks on their blogs and funding for their startup… I guess that’s all that matters.
Be a Bayesian, and they'll stop annoying you.
If you have a 5% chance of thermonuclear war each decade for 10 decades, you'll:
- Hear similar annoying statements
- They'll be true
With AI, we don't know if it's one week or one decade. This means we should assign probabilities and consider all possibilities, not get annoyed.
Assigning, and more specifically announcing/reporting on, all possibilities makes all of these predictions meaningless.
If I predict the world will end every single day from now until forever, I will most certainly be right eventually. That doesn’t make me an “expert” or someone worth listening to on the topic. That’s the playbook of a doomsday cult, not anyone that should drive world markets.
s/all possibilities/all possibilities with odds greater than some threshold/g
There's no reason to report on the odds of a meteor falling on my head tomorrow. There is every reason to consider the odds of:
- Nuclear apocalypse (esp. Cold War era)
- Bioweapon
- Climate change leading to disaster
- AI apocalypse
- Etc.
None of those are infinitesimal odds. That contrasts with, say, a zombie virus.
Ultimately we agree but for slightly different reasons. My assertion is there are no experts because there can’t be as not enough time passed for expertise to form. Further the rate of change is such that by time someone becomes an expert in some aspect, it’s been made irrelevant. Hence expertise in this space is unattainable at the moment and all expert advice is fraudulent.
Expertise in naval gazing and science fiction writing is a prerequisite.
Do you have an example of a prediction they’ve made in the last 3 years that was drastically wrong? This is not my impression.
Do you think all white collar work will be replaced by AI in 18 months?
https://fortune.com/article/why-microsoft-ai-chief-mustafa-s...
Altman and Amodei recently hard to start walking back their earlier predictions.
https://fortune.com/2026/05/26/sam-altman-dario-amodei-walki...
Again, do you have an example of a prediction that was drastically wrong? I don't think it's productive to speculate about which things might or might not come true in the next 18 months. There were people last year making fun of Dario for predicting that writing code would be automated in the next year.
From the 2nd link I posted:
> Anthropic CEO Dario Amodei, who once claimed AI could eliminate 50% of white-collar jobs, now says automation may actually expand the work people do.
Is pivoting from 50% elimination to actually expanding work not a drastic enough?
Do you think AI has eliminated writing code? I still write code every day. The AI is more a thing I ask questions and it gives me right answers about 40% of the time.
Yes, I think AI has eliminated writing code. Everyone I personally know working in software stopped writing code some time between December and March. (It's true that AI continues to routinely make errors; if you've heard the term "agentic workflows", that's the standard strategy for mitigating the error rate by allowing the AI to check its own work.) That's why I think Amodei's January 2026 prediction that AI could eliminate 50% of entry level white collar jobs in 1-5 years remains plausible.
Your second article says something different, but this is because it's full of misquotes. The link supporting "50% of jobs" specifically says entry level, and the link supporting "reframed automation... not as a destroyer of jobs" has Amodei saying not that jobs won't be destroyed but that new jobs may be created to replace them. If AI moves sufficiently slowly to let that happen, which he explicitly cautions it may not.
> Yes, I think AI has eliminated writing code.
Then you are dead wrong. Anyone who gives a shit about doing a good job is still writing code.
If you can describe the code you want to write, AI can write it exactly the way you would have, but faster than you every time. That is the floor now.
GP posting "Everyone I personally know working in software stopped writing code" on HN is such an obvious tell of blinkered delusion.
Literally all HN talks about is AI these days, and it is very, very clear that plenty of people are still writing lots of code, many finding AI makes them slower and causes more problems, and many making the reverse claims. There is always a rich mix of opinions and experiences here.
You would have to believe that literally everyone on HN is a bot (and also everyone on Reddit, Twitter, astralcodexten, or anywhere else online) to discount all these differing opinions in favor of "everyone I personally know" .
No, that’s not accurate. I’m not sure what to tell you. It’s like hearing that no serious programmer uses Python, it’s so far from my experience and that of everyone I know in the field that I don’t know how to engage.
Do you think that everyone in the Linux kernel team write their code using AI only? What about Emacs? And cURL?
Ok, how about Ladybird: https://ladybird.org/posts/adopting-rust/
> I used Claude Code and Codex for the translation. This was human-directed, not autonomous code generation. I decided what to port, in what order, and what the Rust code should look like. It was hundreds of small prompts, steering the agents where things needed to go. After the initial translation, I ran multiple passes of adversarial review, asking different models to analyze the code for mistakes and bad patterns.
Sounds to me like "wrote their code using AI only".
And from the creator of Redis, antirez: https://antirez.com/news/158
> It is simply impossible not to see the reality of what is happening. Writing code is no longer needed for the most part.
I guess when you have 65k tests you can use an AI for transpiling, and it still seems it took an enormous amount of hand holding. That is very far from actually writing new code. But you tell me, are you a software developer? Write or have code written everyday? I am one and for me and everyone I know AI has been mostly just a (fantastic) replacement for google search.
> took an enormous amount of hand holding
There's a big difference between "not writing any code by hand" (which still might involve lots of non-manual-coding effort!), and "vibecoding", and as I understand this thread was about the former.
edit: yes, I'm a software developer, and if I have a change in mind larger than 2 lines I've been using Claude Code (with English input) rather than a text editor to do it, since about January.
Can't speak for everyone involved in this thread, but your workflow sounds identical to mine and that's what I was intending to describe. I agree there's a different thing sometimes called "vibecoding" which doesn't work well for software that has to be even mildly robust.
Right? Like the tool is the same (Claude Code, in my case, but there are many like it these days) but the operation of it is totally different:
"Write a function that will <do a thing the code does in 3 places>, put it in utils, and refactor the 3 locations that need to do that thing to use that function instead." (and then hundreds more prompts like that)
vs.
"Write an app that lets you trade crypto"
Just totally different activities.
YOU still write code. I would say over 65-70% of modern programmers no longer write code. The paradigm shift happened within the last 365 days.
65-70% "programmers" never wrote code in the first place. Before AI cargo culters just copy pasted code from tutorials, stack overflow, github and so on. With AI they merely continue not writing code. Some people didn't do anything, not even copy paste, see r/overemployed although I have not checked it after AI... maybe it's no longer possible to hold a dev job without actually doing anything.
Programmers still write code. They have enough expertise from actually programming to understand that the untouched output from AI is not good enough to be professionally responsible for.
No. You can't get very far copying and pasting from stack overflow.
I'm talking about legit programmers which are around 90% imho. Out of those programmers 65-70% no longer write code.
At work I still write code, but at home I don't think I wrote anything in months.
There's nothing that I can't tell Claude Opus or GPT 5.5 to do for me instead.
At work, where I don't have cheap AI subscriptions, access to the SOTA, and where code needs to follow conventions more strictly, I still have to write the code by hand often.
> The predictions of these "experts"... last 3 years.
Which ones? Please be specific.
They have been wrong in a way that's bad: underestimating the speed and size of progress. So if the "experts" claim a timeline and magnitude, it would be safest to assume an even faster timeline and a bigger impact.
In fairness, they've been operating with a lot of built up cynicism from major breakthroughs that have been 3-5 years away for the past five decades.
I wonder...
https://en.wikipedia.org/wiki/Technological_singularity
https://en.wikipedia.org/wiki/The_Singularity_Is_Near
[dead]
https://archive.ph/2OWwO
> Experts in artificial intelligence estimate the risk of an ai-caused catastrophic event at 10-50%
Because something AI did or because a human in the loop blindly trusted the result of AI or the promises of AU companies?
We certainly weren't ready for the stupidity explosion...
The first priority should be an agreement between the two heavyweights of ai: America and China. Donald Trump and Xi Jinping should affirm the principle that humans must remain stewards of ai systems until adequate frameworks for reliability and security have been built.
This is naïve. The goals of both men have nothing to do with protecting humanity, but rather furthering their own personal agendas.
In Trump’s case, it’s all about amassing more wealth and power.
For Xi, it’s realizing an ethno-nationalist dream where China under the CCP is at the center of world power, the independent nation of Taiwan as well as disputed border areas that are currently controlled by India and Russia and the Philippines are annexed by China, and Xi’s eternal legacy is remembered as the savior of the Chinese people.
International cooperation and touchy-feely rhetoric about saving humanity from AI have no place in either man’s worldview.
Agreed, and their zeroeth order prerogative isn't much better: to ensure the continued need and demand for the thrall of centralized authority that allows them to achieve their disparate goals. This ensures strong AI is never going to be able to empower autonomy that removes, causes questioning, or even causes awareness of their zeroeth order.
It's like living side by side with your cellphone. Which most already do.
Oh I'm ready. I ve had enough of useless tech that evolved to a global network where people vent their frustration about technology. Let's do some cool stuff instead
AI is designed to produce variations on a theme. Not sure why you think it will produce something different.
This is the kind of articles that make the last, biggest fool buy the top of a bubble.
A good chatbot, image/video generator, coder, etc is not going to enslave us all (not in actual sense, although maybe in the instagram sense) or propel us into a golden era.
They are nice tools for some pretty specific tasks and not much more. And we are only seeing the heavily subsidised version of the tools.
A similar Luddite view formed in the 1990s. It turns out humanity constantly beats the expectations of economists. It makes you wonder if the "science" of economy is almost entirely bankrupt either in morality or imagination or same dangerous combination of both.
> Humanity simply does not have a strategy to ensure it remains safe through RSI.
Turn off the power. It's pretty simple. Leave it to an economist to forget about input costs. Your "super intelligence" only matters if it's actually more energy efficient than a human being and for a million years of evolution humanity is a much harder target to beat than this author seems to realize.
Science fiction authors have had to grapple with the fact that it's nearly impossible to build a robot out of metal, plastic, or composite material that has the physical performance characteristics (strength, durability) of a human body and weighs just as much. You can build a strong robot but it'll be heavy. The better SF authors, if they've need of robots of mere human weight, will make them comically fragile.
The same is even more true of our intelligence. We're building computers with the size and power consumption characteristics of entire cities to do things that may almost, but not quite, match what our brains do with a kilogram and a half of mass and about 20 watts at the top end of power consumption.
The only way we are ever going to match that with technology is to run AI workloads on human brain tissue, which Rick-and-Morty level horror is being actively worked on as I understand it. The original concept for The Matrix wherein the machines used humans to run compute workloads on their brains actually kind of makes sense.
[dead]
"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...
I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...
We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...
And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.
I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate
Source with interview clip: https://x.com/deredleritt3r/status/2062223035940139253
EDIT: Ten years ago, most of the skeptics in these threads would have said the success of AlphaFold 3 was impossible.
"(...) it's going to effectively be a new human era..."
We never like it when we realize the universe doesn't circle around us.
At first we got mad to discover the Earth wasn't the center of the Solar system.
Then we got mad that the solar system wasn't even unique.
And now we're still clinging to the idea that humans always will be the most intelligent species or the only ones gifted with "creativity".
And soon we might also come to realize it might not be a new human era, but the end of it (if we're not careful).
Why are these people so obsessed with the year 2030? It keeps popping up everywhere.
It feels close (it's only 4 years away) and yet futuristic (the new decade!). The way it feels is important because all these statements are based on jack and only exist for hype.
It's like when people obsessed about 2020, or 2000. It's a nice clean number in the decimal numbering system.
No, there's a lot more to it. I never saw this kind of fuss about 2000 or 2020.
I keep seeing things about how some target has to be hit by 2030. Our local council keeps going on about it, in fact their WiFi network even contains 2030!
> No, there's a lot more to it. I never saw this kind of fuss about 2000
I roll to disbelieve!
We had hype about the year 2000 throughout the 20th century, and some earlier! When we were IN the year 2000, it was quite common in my circles to tack on some phrase or comment about it being the Future, now, so [whatever thing the person was on about].
A very different type of hype. I remember it. More about the Millennium Bug and the New Year celebration. It didn't have the same vibe at all. The authorities weren't trying to hit various targets since then.
The real Millennium turned out to be 11th September, 2001, which was use to bring in a lot of planned changes.
Someone should document these cringe comments for posterity. As a warning to future generations.
It's too bad the previous singularity cult manias 10 and 15 years ago weren't documented and are lost in time.
Ten years ago, most of the skeptics in these threads would have said the success of AlphaFold 3 was impossible.
It doesn't mean everything you can think of is possible.
People who say every jobs will be automated never held any tool more advanced than a screwdriver and never worked a single day outside of an air conditioned office.
They were predicting flying cars for "the year 2000"...
But enough jobs are going to be automated that it will cause social and economic chaos, and political breakdown. Sure AI can't attach an air conditioning unit outside my window nor use a plunger on my toilet but it can still replace lawyers, secretaries, software devs, and many more jobs.
I'm curious: how will it replace e.g. lawyers? Will judges be replaced as well? Juries?
Are we sur eof any of that?
Taxis were supposed to be all replaced when Uber promised to buy every single Tesla model S coming out of factories from 2020,they're not even self driving today...
Software engineers were supposed to disappear "in 6 months" every 6 months since chat gpt was released
Lots of people on this website live in insane bubbles which are completely out of sync with the reality of most workers
No one rational ever said software engineers were going to be replaced in 6 months. Some people said AI will automate 90% of coding in 6 months and they were not far off (and accurate in some contexts, e.g. startups).
Coding has been automated since the very beginning of coding.
The first two programming languages (Fortran and Cobol) were supposed to make coding obsolete because non-coders will now just make software with natural language prompts.
LLMs are a dangerous tool for coding because they are infinite technical debt generators. Ideally only experienced senior developers should be allowed to use them.
It's amusing to think that context window capability needs to outpace technical debt pileup with LLM development and usage
They'll only get to claim a hollow victory because AGI is impossible to rigorously define. None of the regulating bodies of consumer products will be able to define it better than academics. They'll use marketing to make those claims and there will be legal battles that keep it in a gray area.
They'll go for that because it's easier than actually inventing the 'sci-fi' AGI, shareholders keep making money, and it keeps them getting paid to keep going. If any of them actually do succeed, then that little deception will be peanuts.
Demis' bar is high and he stated clearly multiple times: AGI should be capable of inventing truly novel things. Examples he gave included the Theory of General Relativity and the game of Go.
Just a taste of what's to come: https://openai.com/index/model-disproves-discrete-geometry-c...
Anthropic's latest model also solved this 80-year-old problem that eluded many expert mathematicians, in a different way, according to one of its employees.
Remember we still have 4 years until 2030.
Quite frustrating to see all these cynical, borderline-irrational comments on HN. Maybe I should do what pg and other ex-HN contributors have done--avoid taking part in the discussions here.
The level of discourse here has dropped so much.
At least I should stop replying to people hiding under throwaway accounts.
Surely you can think of some times the skeptics have been right.
I immediately thought of this:
https://en.wikipedia.org/wiki/List_of_predictions_for_autono...
Ten years ago, most people would have said cold fusion was impossible... and still is.
You know in the future they are just going to say it's different this time right?
The Enigma Of Mass Amnesia...
If you look how much goalpost has been moved over the last decade it'd turn out that AI cultist were right more frequently than AI sceptics.
I'm sure HN cult comments from 10 and 15 years ago are still up though, the Dropbox thread is frequently linked to.
Just search for "stochastic parrot" on Algolia. Plenty of "BuT iTs NoT rEaLlY iNtElIgEnT" comments...
It literally is a stochastic parrot.
P.S. Yes, I do know how an LLM works. I code inference engines at my day job. You're consuming a cloud chat bot.
The people saying "it's just a stochastic parrot" were saying that because they thought the fact that it works by modelling the statistics of words and predicting the next most likely one meant that it would never be capable of anything sophisticated or any kind of novel output. It was just parroting things that already existed in a random way.
There was never any actual foundation for that belief, and it has been proven wrong empirically by more capable models, which is why you don't hear people saying it much any more.
Another CEO with AI psychosis [1]. LLMs are not true AI, they lack common sense (or whatever it's properly called). LLM-based systems still need somebody with deep domain knowledge at the wheel to keep them from doing stupid things. It's like an alien-made bicycle that gets you to the speed of sound if you are an Olympic cyclist.
Still, LLMs are extremely powerful pseudo-AI [2] and will bring a pseudo-singularity. But the impact is still scary if a tiny fraction of humans are augmented 1000x. And as better models become exponentially more costly, only the money people will be able to afford the new models. This is a very likely scenario and scares me to the point I dropped all my projects to work on affordable LLM-based tools to make the difference at most 10x instead of 1000x.
To my elder relatives I explain it like: imagine we are farmers in the 17th century and suddenly out of nowere John Deere tractors, combines, etc. become available. But they cost more to run than all you and your fellow farmers have, so only a tiny handful of rich people take over everything.
[1] https://en.wikipedia.org/wiki/Chatbot_psychosis
[2] https://en.wikipedia.org/wiki/Chinese_room
> Another CEO with AI psychosis [1]. LLMs are not true AI, they lack common sense (or whatever it's properly called).
If you read the article, you'll find that the it indeed relies on this claim:
> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.
This may be optimistic and/or simplistic, but not impossible.
>the result could be an intelligence explosion of a kind for which there is no precedent and no map.
Or it will simply reach the end of the softmax faster.
> ... they cost more to run than all you and your fellow farmers have ...
But the financial situation is that running a single AI agent costs significantly less than you'd have to pay a human to do the same task.
And I don't see what you're getting from The Chinese Room - that thought experiment relies on there being no external difference at all, right?
Humanity isn't ready for the amount of bullshit being written on a daily basis to pump the upcoming IPOs.
Models consume a lot of memory and power to slowly generate autocompletions of existing content one token at a time. Letting this text control important things in real world is a human decision and control of nuclear weapons or penetration testing agents should be well regulated. But these should already be well regulated without AI. So how about we focus on drawing down nuclear arsenals, which is a present danger one idiot (or faulty AI) away from world shattering consequences rather than fearmongering about unknown future. Before effective AI regulation can be drafted, we need to anyway know more about AI specific dangers rather than humans committing already very common crimes like hacking for ransom with new tools?
> Experts in artificial intelligence estimate the risk of an AI-caused catastrophic event at 10-50%.
I don't think it's that high but can we put that aside and focus on the 100% chance that it's being used to enshittify every part of our lives?
Honestly, this article has too many words which don't mean anything.
'AI Experts', 'superintelligence', and hand-waving doom scenarios.
As an example-
> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.
I've heard these same objections in my lifetime about the internet. And I've read similar arguments against TV, radio, the phonograph, and the printing press.
Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.
Ultimately, what counts is what we do with it.
> Ultimately, what counts is what we do with it.
Looking at the people in charge of these companies, and the sheer lack of understanding by the broad consumer market using them, I have nothing but pessimism about the direction that this technology takes us.
>Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.
Common human trope is to rely on analogies and past patterns to predict the future. It blinds you reality.
This isn't some new "invention". What's happening is fundamentally more profound. the stakes are much much much more higher and the consequences much more grave. Thus a historical analogy can't save you when the situation is categorically different.
I agree it's getting "tiring" but this is another common human trope. You see the same thing too many times and you lose the sense of danger that comes with it. It's like sky diving all the time. You eventually lose the fear via repetition, but fundamentally speaking, you are still parachuting out of a fucking plane.
There’s a meme in finance circles that by the time some trend appears on the economist front cover it’s a top signal and time to get out of the trade and this is why.
It’s a very normie magazine written by and for normies, and by the time an idea has got that far down the easy money has already been made
By the time it appears on People Magazine that must be the third or fourth dead cat bounce!
Yeah, big lol on the Recursive Self-Improvement.
I mean, firstly you get to have an "Agent" actually capable of really long horizon tasks without getting stuck in tools loops and having its context rot. Secondly, each trial (ie a model fully trained to convergence) costs millions and takes O(weeks). You can probably run 1 or 2 of those experiments in parallel even at big AI labs as the hardware is scarce and they are costly as mentioned before. Assuming this agent needs something like a hundred tries to just show some improvement, we are looking at years.
And you can't early stop training for candidates that are not promising due to "emerging capabilities". At some point you might get a big drop in loss even if the model has been plateuing for a while. And you can't really scale down models for running trials quickly either also due to these emerging capabilities. In fact you might create a model that is great at converging with in small trials (fewer params, fewer tokens), but that is uncapable of developing those unexpected traits. And this will likely happen as you created a sort of evolutionary pressure in this direction: if you are good at learning in the first few epochs you "survive" and get to pass down your traits to future trials.
All of this to say that recursive-self-improvement is waaaaaay out of our grasp as things stand right now. We need another one or two breakthroughs to get there (IMHO).
The Economist reflects the views and biases of economic and political elites.
AI has turned into something of a religion for them because the substitution of technology for labor is a reflection of one of their deepest desires. Whatever the reality of the situation, they want to believe that AGI employees are imminent.
We'll be fine. We've seen this one before, this is just new packaging.
Oxford Brain
> "recursive self-improvement (RSI)"
My aching joints enter the chat...
This article posits "recursive self-improvement"—which is just a more-technical name for the supposed Singularity.
There is no evidence that we are currently on a path that leads to the Singularity.
There is not yet any evidence that the Singularity is more than science fiction.
It is effectively an article of religion—the Rapture, but for techbros. Indeed, it is what some of them are pinning all their hopes on; after all, if they can recursively-self-improve their way to artificial superintelligence, then and only then could the absurd investments of companies like OpenAI actually prove worthwhile.
I want to name the AI era as the corruption of the elite.
The rhetorical structure of talking about uncertain risks and then trying to concentrate the authority to manage those risks in their own hands sounds utterly ridiculous to ordinary people like me.
It's just a simple hypothesis that AI will become uncontrollable to humans once it becomes superintelligent.
Isn't the fact that a reinforcement learning agent improves itself in a specific domain completely different from it recursively improving its own code in a 'better' way? It's just a tool to create a justification for regulation and control using sci-fi fear.
The comparison between nuclear power plant risk and AI risk is also absurd. Where exactly can you define and measure the probability of AI exterminating humanity? It's as unquantifiable as 'I, human JDW64, will become a successful programmer.' What is the measurement standard? Why dress up AI researchers' concerns as objective probabilities? Is it because numbers make it look logical?
The current US-China relationship is in the middle of an AI arms race. The US is strengthening export controls to limit China's AI development, and China is building its own ecosystem. In this situation, I don't understand the idea of cooperating for the common safety of humanity. RAND is an organization that presupposes cooperation—isn't it just a well-written research proposal from an institution that wants to position itself for that role?
Isn't the claim that 'government must step in' ultimately about protecting their own interests? 'A strong government that will protect us' is an authoritarian government. If they were East Asian, they would understand that such regimes have always been used as tools for surveillance and control.
And I don't understand why Fermi's paradox is being brought up here. Why package a software problem as something that inevitably requires strong control? Whenever I see articles like this, I think about what 'intelligence' really means. This person would probably be called 'intelligent' by others. But no matter how I look at it, the holes in the argument are too obvious. It really makes me think that there are different tiers of intelligence.
The AI labs will govern, effectively if not officially. Government is anchored in power, which the labs have. We used to have the church and state rule here in Europe. In the future, A government not closely ruling with their AI labs won't have sufficient power to exist.
You're unhappy about the labs; I think you're just not ready to accept their rule; you consider them nothing but scrappy startups, which they are. But power is power, like it or not.
Am I personally happy with any of this? Does it matter?
You could be right. In that case, I should try to get into an AI research institute here in my country, though it won't be easy.
How would the labs have power? What physical force would they command to put against any government?
If AI becomes so useful that it will be used for everything done on a computer, they'll have so much influence over everything, and be so important for society to function, that they'll have to at least work very closely with govenments. I can see that eventually evolve into de-facto ruling.
The physical force of AI zealot dreams:
A bunch of noble AI researchers stands up against the hand that fed them all the time, intentional vulnerabilities are already built into every machine of death and destruction owned by the feeding hand, and get exploited, the feeding hand is robbed of its punitive force and defeated, a yell of YAY fleets over the progressive humankind who kneels before its AI overlord unreservedly.
Then they kindly ask the AI overlord to please feed them.
AI labs do not have an incentive to downplay the risks and if you can’t understand that you’re naive and shouldn’t be writing Economist articles.
AI labs have every incentive to overstate the risks so that they can get lucrative government contracts, especially since it’s clearly not profitable going the public consumer route. And if you’re Anthropic you’re even more incentivised to overstate the risks because at this point in the game regulation hurts potential competitors more than it hurts you.
On the same page (for me): "Silicon Valley needs to get God"
Well, seems it already did.
Can these snake oil salesmen stop with this Sci-fi stuff please? Yes, Machine learning, and transformer architecture are great stuff, but lets not pretend this is sci-fi, please.
This is absolutely sci-fi.
ChatGPT and Claude are literally building staging scripts for me as I type this.
5 years ago, few people on this planet would have thought the computer would make advanced POSIX compliant shell scripts, but here we are.
Yet again another journalist who read too much science fiction and should stop overestimate a bunch of rocks conveying electrical pulses. Or is this simply to get more clicks?
Economist has drunk the AI koolaid. Pretty pathetic for a publication of their esteem.
Of course humanity is not ready for skynet. But there are rebel forces - the no AI using humans. They may be a dying breed but at the least they put up a fight. Many other humans already have become servants to skynet (version 16.0 now). Next step for them is the neuralink chip. Some billionaire with twitching right arm gestures owns that, doesn't it ...
(Also, Cameron was not quite right with regards to skynet. It is much much dumber but also more effective than displayed in his movie really. Kind of a weird combination if you look at the current AI slop out there.)
[dead]
As the AI gets "smarter", it is gonna make the society dumber so it will balance itself out /s
"humanity" and intelligence explosio" in the same sentemce! Muhaha! Boy, have you been on just a couple of tourist flights / visiting major tourist hotspots?
Most people out there are literal Normies, and operatelike a robotic vacuum cleaner, or NPC.
I wanted this to be thought provoking, but for a physicist to open with, "the acceptable risk of catastrophic meltdown for a nuclear power plant is roughly one in a million", just seems sad. It is a phrase without meaning.
A meltdown (or loss of containment) just isn't that bad, if it doesn't affect ground water or lead to atmospheric fallout. We're turning 3 Mile Island back on to power an AI datacenter! The AI super-intelligence apocalypse envisioned (ignoring the likelihood) is inherently global.That said, the rest of the analysis and proposal also greatly disappoints me. The idea that the current administrations of US and China could do anything constructive seems hilarious. They're so paranoid and self-serving they couldn't come together, even if there was an alien invasion. Then the idea that LLM safety is somehow as easily traceable as nuclear isotopes and bomb tests seems equally ludicrous. I am sad.