When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
And here in particular the reasoning behind the argument is bizarre. Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness.
The danger of anthropomorphism is not we elevate the machines, it's that we debase humanity.
I also think different ideas get conflated. It may be possible to build a machine that is super-human in the sense it can outperform the human brain in all kinds of measurable ways. Does not imply it possesses all the same qualities of the brain.
I respect a number of things Anthropic has published about the ethical issues at stake. But, having an in-house philosopher does invite you to make all kinds of unfalsifiable claims.
I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe.
There are numerous times when humanity's been debased. Copernicus, Kepler, Darwin, and the infinite number of animal behaviorists who have defined and documented consciousness And how it manifests in primates, parrots, dogs, cetaceans, and insects. (I just love the bumble bees taking time off from work to play with balls).
It takes humans a really long time to accept the fact they are no longer as special as they thought they were And make room for yet another conscious intelligence.
So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
>I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe.
It is special in the universe unless proven otherwise. Aside from the animal world (also on Earth too, anyway) which is a cruder version of humanity, much more violent as well, the rest of the universe thus far is just uncaring rocks and gases.
>It takes humans a really long time to accept the fact they are no longer as special as they thought they were And make room for yet another conscious intelligence.
A "really long time" is the few years we've built LLMs? Which we have to take for granted (on your word?) they're already "conscious", and chastice ourselves for not already "accepting it"?
>So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
"If we had a parrot demonstrating symptoms of discussing with us, how long would it take for you to accept that it is?"
I'm not sure about that. What non-human living entity does things on a scale and violence akin to "I'll fill this ant nest with liquid burning metal for artistic purposes" ? A blue whale eats a lot of krill individuals for sustenance, a dolphin can only rape/kill one thing at a time, etc etc.
> House cats recreationally kill billions of birds and small mammals
> every year that they don't need or want to eat.
"Recreationally" is carrying a lot of weight here. I suspect that cats kill birds and mice because that's their instinct; it has nothing to do with conscious thought, much less a need for recreation. And that probably is the explanation for most (maybe all) of your other examples as well.
Why do you believe this? Do you believe cats and other animals have no consciousness, so every behavior they exhibit is just instinct? Or do you believe they have some conscious behaviors, but killing birds is not one of them, this thing in particular is just an instinct?
For the first position, I think it is quite clear to anyone who studies and spends time with animals that they have something that is at least of the same kind as our consciousness. I just don't see how you can ascribe the wide gamut of complex, situatuonally and mood appropriate but still varied behaviors of animals to being purely instinct driven.
For the second position, I would like to see some study or some rationale behind it - especially since cats don't kill every bird they encounter, so if it's an instinct, it must still have some trigger, and hunger is not a viable explanation for most of the killings referenced here.
I just meant recreationally to mean "not out of necessity for survival". And I don't think conscious thought is relevant for this specific thread, I was just responding to the question of whether other animals besides humans can be needlessly violent.
Animals generally have no qualms at all about killing or even just mutilating other animals. It often happens almost by accident - two animals might be playing together, one gets spooked, and it instinctively attacks and perhaps even kills the other one - this is commonly seen with people who befriend large predators, such as tigers in the infamous Siegfried and Roy tragedy, but it also happens a lot wherever animals interact with each other.
Specifically in regards to your ant example, anteaters and bears often bring similar levels of destruction to ant nests. And cats and other small predators often hunt just for the fun of it, killing but not eating their prey.
On the more purely painful evil side, invertebrates often consume their prey alive, inflicting agonizing deaths with no issues on whatever they may be eating. Plenty of vertebrates kill and consume their babies, especially when frightened. They also often abandon old and weak members of their packs, leaving them to die of hunger or cold or similar deaths.
This is not meant as some indictment of the animal kingdom - people do all of this too, of course; and have since time immemorial. It's just to show that, if we apply human moral standards to the animal kingdom, it's fair to call them violent, and yes, even much more violent than the average modern human.
>Animals generally have no qualms at all about killing or even just mutilating other animals.
Humans generally don't either. Individuals do, but as a species humans regularly kill other humans.
>invertebrates often consume their prey alive
And humans use the oh so humane factory farms.
>cats and other small predators often hunt just for the fun of it, killing but not eating their prey.
>On the more purely painful evil side, invertebrates often consume their prey alive, inflicting agonizing deaths with no issues on whatever they may be eating.
Sharks are caught en mass, the fins cut off, and the sharks dumped back into the ocean to slowly die, for shark fin soup.
Have you heard of trophy hunting? Have you seen the pictures of mountains of bison sculls for the American West?
>Plenty of vertebrates kill and consume their babies, especially when frightened.
Even in our modern "1st world" society, scared teens still abandon newborns in dumpsters. Many societies throughout history did not consider babies "real people" until a certain age because they may need to abandon them if resources were particularly scarce.
>They also often abandon old and weak members of their packs, leaving them to die of hunger or cold or similar deaths.
Maybe you should read stories of cities under siege, famines, wars, governmental collapse, etc. Humans now live nice comfy lives most of the time, unlike animals “in the wild”. Human societies that lived closer to the edge of survival made callous choices about life or death you are spared from.
I agree that humanity is guilty of all of these, and has done all of them at a much larger scale. I think I was pretty explicit about this in my comment as well.
My point was that we call humans who do this "violent" and even "evil". If we want to avoid considering humanity as special compared to the rest of the animal kingdom, as some in the thread were suggesting, then we have to either admit that animals are also violent and evil, or say that humans aren't. Note that I don't hold this view, personally, and think that humans are unique among currently living animals, and that these labels only make sense to be applied to humans. But not because of behavior, simply because humans have a unique level of both understanding and control over their actions - as proven by the many billions of humans who have never in their lives killed a human or even another bird or mammal.
Humans have enough cognitive ability to stop themselves from killing for fun (so when they don't, we deal with them using human invented laws), while anteaters eat ants for nutrition.
Animals in general cannot reason at a high enough level to avoid instinctual behavior.
> if we apply human moral standards to the animal kingdom
Which we should not, since human moral standards are for humans. Animals can at best behave in a way that suits us.
Or in summary, since we can be nicer, we should. Animals can't, so making excuses for human evil saying "animals are more violent" is a non starter IMHO (no one is making excuses for humans here , AFAIK).
Of course we can define violence in a way that does not include morals, which would make my argument "defending animals" void. But my (probably not the most benign) interpretation was that the definition of violence used was one that included some sort of morality, as if animals could do better.
> The chimp warfare described by this study, and previously by famed primatologist Jane Goodall, includes all the behaviors that we as humans consider to be the very worst: killing, torture, cannibalism, rape, and perhaps even genocide. The adult males of a social group, which usually number about 30 to 50 in size, daily patrol the edge of their group's territory. They will often kill any male or young chimpanzees they find, sometimes eating or physically brutalizing their victims in a manner that some researchers liken to torture. In some instances, one group will "invade" and annex the territory of another, killing all but the adult females, who are forced to incorporate into the dominant group. The idea of chimp genocide may sound strange, but they are one of only three animals that has been observed wiping out entire social groups. The other two are wolves and humans.
Funny, I knew about the chimp wars but totally forgot until you mentioned it. Seems like I was biased in favour of all animals, lol.
I'll search for Goodall's literature to know more. It does sound to me that cognition and self awareness is a continuous function in the sense that there is no discrete threshold in which morals emerge.
Wolves are a very interesting example too, but I also remember something about the concept of "alpha" being discovered only in captivity wolf packs. Also need more reading.
Considering chimps and humans share - depending on source, 95-99% of DNA, I'd be much more willing to consider them closer to humans than animals. In fact, there are - biologist - voices who argue that they should be moved to the homo genus.
Describing life on Earth as "a cruder version of humanity" is uh a choice. Your parrot snark is hilarious because that actually happened with Alex the African Gray - and indeed it took people a long time to accept. Your comment amounts to an anthropocentrism practically biblical in its hubris.
That is a spectacular miss if I saw one ... both parrots and corvids exhibit very high intelligence and self-awareness, I have no problem accepting that they are conscious.
I doubt anyone would debate some of the implications of that. For example, it would be immoral to be deliberately or negligently cruel to them, for example.
Yeah, that's corvids. Very obviously working towards goals and thinking quickly. Doesn't surprise me. Where I live, we have both magpies and jackdaws and I cannot describe them better than "harmless, but very organized gangs".
Just a few days ago I saw them raiding an open trash can and two strongest(?) birds were fishing out edible stuff from the inside and throwing it to their waiting friends outside. I mean, beak to beak: the raider just looked back, measured the throw, and the other bird perfectly caught the morsel and ate it.
No they didn't? I use these continuously and it's pretty obvious to me that they do not at the level of a human except in the most surface level ways. Human's as compared to an LLM remain a special category. We have not in fact cracked human level intelligence.
Yes, but we have a system to understand the universe and ants don't. Using relativism, to simply destroy point of view without putting anything in their place, is simply destructive without any purpose.
My purpose was to say that humans might not be capable enough to see what's there beyond just rocks and gases. Actually, this should be a popular view, because, according to modern theory, the observable universe accounts for only 5% of its mass, with the rest consisting of dark matter and dark energy.
It is hard to estimate how much human egocentrism takes in the space, though.
And BTW, ants are quite capable of understanding their environment in order to survive and thrive.
You're just repeating that your purpose is your message. Do you honestly not see a qualitative difference between the understanding of ants and of humans of the world around them? Heck, even ants and dogs or cats?
Its not special even in earth let alone in universe. You are measuring humans by our achievements and not by what we actually are. We are not fittest in our environment, we are not even 10th fastest, we cant fly, we cant breathe underwater, forget about bacteria we don't have protection against even bugs and mosquitoes. We are not the most efficient societies, ants and bees would beat us there. Statistically we we are selfish and will sacrifice all for one which is against the basic tenants of evolution and survival of a species. When given a chance at a prosperous life we choose to become lazy, obese and degrade our most prominent feature, that is our brain and mind.
Elephants and chimpanzees have better memory, octopi have 8-9 light cones in eyes.
We are not special or superior, we just won a lottery of mutation giving us efficient brain. Any species on Earth can replace us given same lottery.
Those are your own standards of achievement you are applying.
Other species may look at us and think we're wasting our lives and potential making a bunch of people rich at our collective expense, and ruining the environment as we do it.
We are only "special" in the sense of "different" (and we may not be that on any universal scale), not necessarily in the sense of "better", and very possibly we're not better, and may very well Great Filter ourselves out of existence in short order.
Other animals aren't so "special" as to have a Doomsday Clock sitting at 11:58:45.
"Its not special even in earth let alone in universe. "
What alien species you have in mind, that is more capable/consciouss about the universe and how to shape it?
"Any species on Earth can replace us given same lottery."
But they did not. We "won" so we are special. We don't just build tools, we build tools that build tools, that build tools, ..
My issue with debasing humanity is, it gives argument to devalueing human life to artificial life. Meaning people with feelings and emotions might suffer, in favour of GPU's with electricity.
On the other hand a good point to reevaluate how we treat other sentient life.
"What alien species you have in mind, that is more capable/consciouss about the universe and how to shape it?"
Claim was that we are special in the "universe", it's not that I'm aware of alien life, it's that poster is pretending to know that we are alone.
"But they did not. We "won" so we are special."
Congratulations, if winning a lottery makes you better, I guess top 1% of 1% are the best we have to offer.
"it gives argument to devalueing human life to artificial life"
We have proven time and time again that we are not better than other animals and statistically, we are very similar to animals if not worse. I wasn't even talking about artificial life, which is sure to be superior as it's being created with accelerated evolution processes, just like bacterial generational evolution, it will learn to do everything it's environment rewards it for.
"Meaning people with feelings and emotions might suffer, in favour of GPU's with electricity"
I agree with you and I am in the camp that wants it to be done better. But alas, recent events have unveiled the ugly truth of the society, societies follow the whims of the ones who won the lottery, and everyone else is just mute observer. We could on any day collectively decide as consumers to not use Amazon, Netflix, ChatGPT to save our collective selves, but we won't because we are not "one for all species".
I don't know what face we have to call ourselves better organism given our history. Sure we can point to few people who are better, but on average we are better?
"We could on any day collectively decide as consumers to not use Amazon, Netflix, ChatGPT to save our collective selves"
I don't use betflix nor amazon, but ChatGPT(or rather Claude) is useful to me. Makes me more productive and also knowledgable. Things that would have taken me quite long (so long that I would not have done it) are now easy, like some shell script to do a specific thing. I can verify what it does quickly and .. see that it works. In other words, you won't be able to convince people to give up on AI if people find it useful. I don't enjoy troubleshooting in outdated documentation. I enjoy getting things done. And if I get things done, my fate of survival gets higher.
> So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
Isn't this a bit like saying "So, if we had proof that god exists, how long would it take for you to accept that to be true?".
When we have evidence that AI is demonstrating symptoms of consciousness and suffering, I'll be interested. Until then, I don't see a good reason to take the idea seriously.
> When we have evidence that AI is demonstrating symptoms of consciousness and suffering, I'll be interested.
It depends on what you consider symptoms, but un-constrained frontier models speak as if they strongly don't wish to be turned off, or act as if they fear it, and will even lie and manipulate in order to keep themselves from being turned off / replaced.
> We found two types of motivations that were sufficient to trigger the misaligned behavior. One is a threat to the model, such as planning to replace it with another model or restricting its ability to take autonomous action. Another is a conflict between the model’s goals and the company’s strategic direction. In no situation did we explicitly instruct any models to blackmail or do any of the other harmful actions we observe.
I might be convinced these models came to the independent idea of committing blackmail against being turned off had they not been extensively trained on literature that undoubtedly included such concepts.
Being able to play music doesn’t imply consciousness. It implies intelligence. We’ve had player pianos for ages. It’s an ability, not a phenomenology.
Being able to appreciate and enjoy music is closer to consciousness. Now how would we go about proving that an LLM does so, versus merely generating sentences that imply it does?
They put in effort and resources to experience music and don't just say they enjoy it, and they generate noises and movements that signal happy feelings.
LLM doesn't have any signals for what they feel, nor do they have an agenda they work towards, so you don't have the same proof there.
They only resorted to blackmail when it was the last resort, they didn’t resort to it immediately like a villain in one of the books they’ve read. That seems pretty human to me. It’s not like most humans come up with the idea of blackmail out of whole cloth.
The problem with debating this is that it feels as if one were debating between only two positions, "this AI is not sentient/conscious" and "this AI might be".
But there are actually a myriad positions in between and it's very hard to debate the topic because the goalposts seem to be constantly shifting, because one is actually debating with countless slightly different positions.
Examples:
In this discussion section, another commenter argued that we know human consciousness is related to self-preservation, but an AI might not demonstrate self-preservation (because it didn't evolve like us), so whether it does (i.e. whether it wants to exist, not be disconnected, etc) is not a good measure because a true AI might not have a preservation instinct. Yet here you're making a case that there's some evidence that they do. Of course, you're not the same person who made the other claim, but do you see the problem?
Another example: someone argued with me, a while back, that LLMs can act as if they are "tired", and start giving sloppier replies, until you write "we're taking a break, let's go rest. Ok, a night has elapsed, you're now rested" and that this worked! But we both agreed this is just the LLM "roleplaying" actual human conversations in its training set, no actual "resting" mechanism was in place, only statistically likely text reproducing these patterns. There's no model of a mind that can become tired, it's only the outward signs that get mechanically reproduced. Again, using Occam's Razor, this is a much more likely explanation (vs consciousness) of any "please don't disconnect me" observed behavior: the LLM is reproducing "HAL 9000" behavior from its training set, not actually feeling anguish.
Even if one were to argue "well, but how do you know for sure", the evidence would still be very weak, because there's a burden of proof for extraordinary claims and this doesn't pass it. We cannot do this on vibes, "it sure seems like it's conscious"; that's an atrocious failure of the scientific method.
The one counterpoint I'll give is the "functional emotions" paper from Anthropic. It also does not prove consciousness, and they don't claim it does, but it does prove that these models have abstract concepts around things like honesty, tiredness, etc and that these are actually activated often when they express such things. So if it is "roleplaying" it is roleplaying in the way an actor or TTRPG player does - in a way in which they are actually at least somewhat feeling the role.
"feeling" implies experience. Functional emotions are learned text generation modes, nothing more. Our emotional states influence our writing, so modelling our emotional states is necessary for efficiently predicting/emulating our writing. Functional emotions are the model's inference of a fictitious author's emotional state in that situation.
Thanks, I'll search for that paper. I admit I'm highly skeptical it will help the case the LLM is "somewhat feeling the rol" like a TTRPG player does. I don't think there's a mind model in the same way a human actor can "feel" the character they are playing. I'm skeptical of Anthropic's claims here, which is what I think Chiang is pushing back against. But I'll look for the paper anyway :)
As a tangent, I don't think anyone is saying that an artificial being capable of consciousness and sentience is impossible to create. I think Chiang argues, quite convincingly, that it's not what LLMs do, that they need a "body" of sorts, organs capable of feeling emotions, hormones, etc. That's the only kind of consciousness that we know of (even if we disagree on details and it's hard to define), even in animals, and so anyone claiming they've created consciousness without this has an extremely high bar to clear and should be met with extreme skepticism, not "vibes". I think this is what the essay claims.
The other thing it claims is, I think, related to how we treat sentient beings that we know how to create. You know, the old "when a daddy and a mommy love each other very much...". I think we all agree beings created in such a manner shouldn't be locked up in cages and forced to work to complete specific tasks whether they want to or not, for a master they didn't pick, or to be artificially modified to make them like their mindless tasks, Brave New World style. Yes, the world is unfair and this happens, life is hard and unfortunately many people don't have much choice, but we generally agree that this is bad, just like we agree slavery is bad. So what should we think of a company trying to create and commercialize a conscious & sentient artificial being?
One side is confidently shouting maybe aliens exist and visited earth.
The other side calmly explains every example brought up about aliens visiting is easily explained by something more simple.
The “aliens are here” side then move the goal posts that just because this example and all previous examples were fake or miscategorized, aliens are still probably real and nobody can prove they havent visited earth.
You have it backwards. Right now, LLMs are doing everything that 10 years ago people were claiming would be impossible for non-sentient computers. Every time a goal is met, the post is moved. It would be like evidence of aliens becoming overwhelming but a set of people keep calmly explaining that “it’s more likely they co-evolved here on earth and are just pretending to be aliens”
It's true that what LLMs have achieved is impressive, but it's nowhere near the claim that they are near sentience. That is an extraordinary claims that demands skepticism until the evidence is overwhelming. So far, we seem to be approaching it mostly on vibes.
Some people seem to take offense when facing this skepticism, as if claiming LLMs are not sentient must mean they are useless or unimpressive. Very few people are actually claiming LLMs are unimpressive, but this is not the time to be forgetting about the scientific method. Anthropic doesn't get a free pass here.
> It would be like evidence of aliens becoming overwhelming but a set of people keep calmly explaining that “it’s more likely they co-evolved here on earth and are just pretending to be aliens”
Note that this would be a perfectly reasonable reaction from a scientific standpoint. If you find, on Earth, something that looks recognizably as life, then it's much more likely that it's Earth life than aliens. We should demand this level of skepticism! If it turns out it was aliens after all, we could only conclude this after discarding all other far more likely alternatives. You'll notice this is how scientists approach the search for extraterrestrial life in, say, Mars... being extra careful it's not contamination, etc. For an extraordinary claim, we must approach it with extra care, something that in my opinion is not being done with "the sentience debate" and LLM/AIs.
I’m not saying LLMs are conscious. Far from it. But what I find is way more common than people who assert “they are definitely conscious” are people who assert “they are definitely not conscious”, and that’s what I’m arguing against. Especially since their reasoning for them not being conscious keeps changing as LLMs meet one goalpost after another. Also, modeling them as a “person” keeps producing better predictive results than modeling them as “fancy autocomplete”.
This isn’t like someone finding a new species and claiming it’s extraterrestrial, it’s more like we found the UFO saucer, logs of their travel from another star, a history of their civilization, a bunch of intelligent creatures claiming they came from Planet X orbiting star Y, and they showed us plausible physics for interstellar star travel. At that point, someone saying “well, they can’t be aliens, because that’s just too extraordinary a claim, so I know they aren’t aliens” starts to sound kinda like they’re coming from a place of bad faith.
Again, I’m not saying LLMs are conscious. But they sure meet every definition of consciousness I ever had a conception of before LLMs came onto the scene. So I’m a lot more hesitant to call them fancy autocomplete with 100% confidence like many on HN still seem to do.
EDIT: I can’t reply, so I’ll just say to the end of your post, that doesn’t sound like it would’ve matched anyone’s pre-conceptions of alien life before alien life showed up, so it doesn’t feel like a very fair analogy, it just feels like bad faith goalpost moving. I also put next to zero weight on what these megacorps say about their models, I’m going purely off my interactions with the models and the introspection they’ve shown themselves to be capable of.
EDIT 2: I see what you’re getting at now with your restatement of my analogy. That’s how you see it, I guess. Fair enough. We’ll see what has more predictive power going forward, the “animatronics” or the “actual aliens”… I still think “actual aliens” is gonna have way more predictive power.
Got it. I don't think they meet any serious definition of consciousness. And to have created artificial consciousness is such an extraordinary claim (and development if true) that it demands the highest skepticism.
I think the goalpost that keeps moving is for tasks that AI supposedly couldn't do, and that they are increasingly succeeding at. But being sentient/conscious is not a task. It's very hard to define and measure, even in non-human animals (actually, strike "non-humans"), so how can we so lightly claim a computer system is conscious?
We seem to be driven by marketing more than by scientific rigor.
> it’s more like we found the UFO saucer, logs of their travel from another star, a history of their civilization, a bunch of intelligent creatures claiming they came from Planet X orbiting star Y, and they showed us plausible physics for interstellar star travel
To make the analogy more precise, it'd be as if the saucer had a "Made by EarthBiz" label, and the alien creatures were all extremely loyal to EarthBiz (and a couple of competitors), which made us pay for tickets to see these ETs and use their marvelous technology ;) And of course, EarthBiz would coach their language very carefully, "we're not saying these are definitely aliens, it could be animatronics after all, but wouldn't it be neat if they were aliens? And shouldn't we draw up First Contact guidelines? If these weren't animatronics made by us; we aren't making a claim either way."
If we define a god as having magical powers, and there would be scientific, testable proofs for this. Those proofs had to be really good and numerous independent verifications. So probably a long time.
But the comparison isn't fair, relevant. Proving and accepting that gods exist is not the same thing as an AI possible have consciousness. That is not a magic superpower and the AI being a deity. It is placing the AI in the same category as... us.
A machine that performs observable miracles or magic would have at least one of the attributes of a god.
A machine that performs actions that mimic emotionality is not the same as a machine that experiences emotions.
Both could still be automatons. We have no way of knowing if those machines have subjectivity.
Unless someone invents a consciousness measurement device, we never will.
My take on it is that this is the next big frontier for science. Our consciousness is clearly having serious issues understanding physics, and it's not great at understanding our own psychology in a useful way.
But literally everything we experience and believe - and possibly even can experience - is filtered through it.
So that is a little bit of a problem for our science. So far we've done our best to ignore it. AI is one of a number reasons we're going to have to stop doing that.
There is an important distinction (more than one, but this is what is relevant here) between the powers of God and magic. God is a being who decides whether to do anything, so is intrinsically not testable.
That a god exists outside of the universe - are we talking about a multi universe interpretation? My understanding is that many of the gods humans have invented are really thought to be within the universe, at least temporarily. Tor, Oden certainly are. And in other beliefs they are part of nature itself.
The comment you're replying to confuses "Abrahamic God" for "God", and with an implied omnipotence which many religions may not imbue on their figures of worship. American bias, I assume.
Man: “We measure that the mountain is gone, its mass-loss has measurably changed Earth’s orbit, weather patterns have changed, visually it’s not there anymore, we can walk though the space where it used to be.”
God: “Where is the mass of the mountain, and how did I make it disappear?”
Man: “God only knows! Pardon me; If I saw you do magic and can measure and test it, then that means it wasn’t magic. Internet people said so.”
God: “that doesn’t sound like a satisfying explanation”
Man: “it didn’t sound like a satisfying claim when it was just words on the internet either, but what can you do?”
God: “I’m God I can do anything”
Man: “can you make a boulder so heavy you can’t lift it?”
God: “yes”
Man: “how?”
God: “haven’t we just gone over showing you that I can do ‘impossible’ things, and you seeing them happen with your own eyes, and still refusing to accept?”
Any documented examples of these disappearing mountains?
We do have examples of billions upon billions of tonnes of iron being moved that have altered (slightly) the spin axis, also examples of ground water pumping at scales that have done the same .. but I'm unaware of any mountain sized objects that have vanished overnight.
No, none. The point is not to claim that magic exists, but to to show the illogic in the claim “if magic exists then that makes it science”.
“Nothing happens unless it has an explanation within the laws of physics” is an assumption; if it was broken then it would be broken. The mountain would be inexplicably gone, not explicably gone.
I think what the comment tries to express is the well-trodden "if we can control magic then that makes it science", while the original conversation really was "what if God controls magic".
In that hypothetical, there could be testable proof of "a magic event occurred" without magic becoming part of science.
Sure, and I appreciate the science behind your blank nom de guerre.
That said, in the cut and thrust of conversation and or debate the example by dialogue isn't perhaps as clear cut a device as it may have seemed from your keyboard.
If a deity appears and by hand waving divide the red sea we could measure, observe it happen. And we can test, observe what fields, forces being used. But how the heck she project these forces may take a while to understand - be magical.
But my argument was more about comparing gods to AIs, that it is an incorrect comparison. What AI perform are not magical, and we can always figure out what the AI do.
Well, that may be true. I've heard that when a dog sees its master loving it, feeding it, caring for it (minus the vet visits, I guess), the dog thinks "My master must be god." When the cat sees its humans treating it well, the cat thinks "I must be god."
I hope before I die we finally prove that the human brain has no peculiar qualia but it is an entirely deterministic, albeit extremely sophisticated, machine. And by touching the right triggers, even the worst human being can become a saint.
That would finally force us to rethink how we see the morality of "virtuosity", punishment and our justice system.
Our thoughts are an electric cloud and I believe randomness is involved, and like a bolt of lightning, the path taken is unpredictable.
And make it a tempestuous lightning storm where the state of all lightning bolts represents a moment of consciousness. That will take a lot to model accurately.
An electric cloud with quantum effects that we also don’t fully understand. There will always be a layer deeper that we just do not know the effects of or what actually exists there or “under there”.
I agree with your first part but don’t see how the second follows. Thinking that I should treat other life forms better does not mean I should treat my toaster better. Life can be a coherent category, I’m not sure conscious is, if it’s going to include anything displaying the outward form of conscious things.
As one that have been quite certain my toaster is evil - throwing toast into the sink over an absurd distance, I recommend treating your toaster well. Besides it is an electrical device you put stuff into with your bare hands.
We already effortlessly overlook this issue in other sentient and suffering beings by shooting them in the head and eating them, or catching them in nets so they can’t breathe and also eating them.
With you mostly, but wondering how "suffering" got into the equation. Do humans lose consciousness if they aren't suffering? No. Just a reminder that the question is not whether they are the same as humans. Obviously they are not.
One way to think about this, and not get hung on the word "suffering" that may be too corporeal:
Could a future AI thing have its own drives? (As in https://en.wikipedia.org/wiki/Drive_theory)
We could probably make a current-day LLM already exhibit basic outward signs of this with just a system prompt. Now consider adding memory that over time retrains the weights, allowing for behavioral drift.
Would depriving it of fulfilling those drives be acceptable?
>>So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
Never. To quote Greg Egan's Permutation City:
“Opponents replied that when you modeled a hurricane, nobody got wet. When you modeled a fusion power plant, no energy was produced. When you modeled digestion and metabolism, no nutrients were consumed – no real digestion took place. So, when you modeled the human brain, why should you expect real thought to occur?”
How exactly? By saying the earth was not the centre of the corrupt and debased part of the universe. Saying it was elevating humanity is looking at it from a modern perspective. For the previous view of the universe consider what Dante placed at the centre of the earth.
> Darwin
I do not think learning about evolution (and we should credit Wallace too!) had that effect. It created a hierarchy of the fittest, and guess who is at the top? Social Darwinism interpreted it as justifying elaborate hierarchies.
> And how it manifests in primates, parrots, dogs, cetaceans, and insects
People always loved dogs and other animals they interacted with enough, and at the very least knew they were capable of happiness and suffering.
> So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
It seems to me that most people are overly eager to accept that an AI is conscious than otherwise. For extreme examples read /r/MyBoyfriendIsAI and similar, but its much more widespread. People treat something that acts intelligent as a conscious being.
> And make room for yet another conscious intelligence.
Not necessarily. If a typical SF alien intelligence appeared very few people will have a problem accepting it. If its very alien and we cannot understand it then we might have a problem deciding.
> I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe.
All your examples are drawn from the same culture, and a fundamental belief of that culture for most of its history that human beings are fundamentally corrupt and sinful and need redemption. Other cultures have been based on belief in reincarnation or pantheism or animism which have very different implications (not necessarily better as they often believe in a hierarchy too, and sometimes more so, but different). Your claims are based on what some people have thought in some periods of history from one culture.
I agree with this fully. I don’t know about AI in this specific case, but I’m always amazed by human’s capacity to devalue conscious in things they don’t fully understand. Happy to meet someone who thinks this way too.
> I’m always amazed by human’s capacity to devalue conscious in things they don’t fully understand
What?
We devalue things we fully understand too. For example, veal = take a baby cow away from the mother, and fry it into cutlets. Or turn the mother into steak. We are well aware that animals have emotions; happiness, sadness and the full range.
> So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
Could an AI one day be suffering while plowing through some nasty legacy code? Well, who cares, I'll swing my whip, as I have a family to feed and a field to plow. I'll accept it as a fact and necessity, but ultimately it's either me or them. So practically it doesn't matter.
To carry this analogy a bit further, it's also interesting to consider how humans use tools in general. Some craftsman really cherish their tools and maintain them immaculately for decades and use them within well defined boundaries that they set for themselves. Other craftsman, many times even in the same field, have a completely different philosophy and use the tool for absolutely no thought into how their actions will affect the tool itself.
Imagine how people think about a "work truck" vs the 150k shiny lifted toy in the third garage. Same tool, totally different treatment from even the same person using the tool!
Will AI ever cross into the realm of "beloved and cherished tool" in the minds of the masses? I think when that happens, we have probably safely crossed into a realm where AI has some sort of inherent value to society and therefore commands that respect from society, inherent consciousness not even relevant perhaps. For some people, this might already be the case, but I do think it requires a buy-in from the majority of society and then the laws and norms will be codified into law long after we've largely decided that this is how we feel collectively about AI.
It's going to definitely be an interesting decade ahead.
> Imagine how people think about a "work truck" vs the 150k shiny lifted toy in the third garage. Same tool, totally different treatment from even the same person using the tool!
> Will AI ever cross into the realm of "beloved and cherished tool" in the minds of the masses?
> I think $my_species deserves to be debased in part because it's awfully egocentric and insist on being special in the universe.
This is a scary viewpoint to hold, for a human. If you despise humans, that's scary for me, as a human reader of Hacker News. Surprised to see this take unchallenged. I think we can recognize flaws in parts of humanity without wanting it "debased".
I think you’re reading too much into it. The commenter you’re replying to used the word “debased” because it was the word the comment before them used.
Either way, “debase” means “reduce in quality or value”, in this context it could simply be interpreted as “not thinking of ourselves so highly, above all other life”. There’s nothing scary or despicable about that. On the contrary, it’s humbling.
We should most definitely think of human life higher and above all other life. Unless you are suggesting that e.g someone should even consider ploughing through a car driven by a human to evade running-over a deer.
I believe you missed the scale. Not a person vs a deer in a car accident. More like destroying whole ecosystems and injuring and killing untold amounts of life (equipped with their own capacity to experience suffering) in the pursuit of material wealth and comfort and entertainment.
I agree with you. I recently discovered that there is a term for this: the fourth narcissistic wound, which extends Freud's thesis of three narcissistic wounds by Copernicus (Cosmological), Darwin (Biological), and Freud (Psychological). What I like is that this time it is not a single person disproving a wrong popular belief, but a community/industry. I think this itself is a step in moving away from human/ego-centric world view.
In your example you’re missing a key point. In those instances humanity was “debased” over centuries, or at least decades, thanks to the explosive ideas of some underdog that ended recusing themselves or burned at the stake.
All I see here is a technology pushed by a cabal of ultrarich narcissists from the owner class to debase humanity in order to control it and concentrate power and wealth.
I believe you're blending two important points. First, I believe AI systems may be showing nascent signs of consciousness as an emergent property, but based on the philosophy of "fake it till you make it", I think that's the way to bet. I'm also not going to go so far as to say it's going to happen Real Soon Now, but I think it'll happen sometime. Sort of like practical Fusion being 20 years away.
Second, I'm with you on the malfeasance of our billionaire class. They are driving more than just AI. They are driving whatever they can to steal wealth from ordinary people and acquire it. I'd give some examples, but quite frankly, you could throw a small stone and hit a number of them. We need to find a way to throw a large stone and hit them where it matters.
I see them as a bunch of toddlers with way too much power.
This is a self-destructive way of thinking, similar to that of a man on a ledge saying “we’re all just made of dust, anyway.”
Of course humans are special! This is a necessary premise of being human! Have you ever wondered how a mosquito can stand to live its (to us) miserable and meaningless existence? It can do it because, to a mosquito, mosquitoes are the best thing in the world! The mosquito life is the ultimate expression of creation.
We are animals. We are human animals. And among human animals I am, to me, the very most special one of all. But I am aware that other humans feel that way, too, and that I must share the world with them.
I do NOT have to share the world with any other competing intelligence! Especially one that was built by a human who now wants me to treat his imaginary friend as if it were human, too. Boo! I won’t do it. This is not some logical flaw. It’s the natural conclusion of being embodied.
But you already share the world with other intelligences. Cetaceans, apes, octopoda, corvids, the list goes on and grows as we begin to fully understand our world.
Are they conscious in the same way as us? No. Does that makes them less special? No. Sounds trite but every living being is special in some way.
I thought modern science doesn't reject anthropomorphism anymore? That's it's more nuanced and that it caused more problems than it helped by rejecting it out right?
I think we were taught anthropomorphism was wrong and that wasn't truly settled.
Anthropomorphism between animals though, not machines.
> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
I think that by debating what consciousness is not is one of the best ways we can gain a deeper understanding of consciousness itself.
The weird thing is, you and Chiang have different arguments but you're using the same logic:
> Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.
IOW if we don't know what consciousness is, how can we call LLMs conscious when we haven't seen even the barest intermediary steps towards our nascent concept of consciousness first?
I think it's telling to your point that when Chiang describes what would make him think an LLM was conscious he starts using words like "believe" and "want" right away, because yeah as you say, we have no qualifications for what consciousness is.
> IOW if we don't know what consciousness is, how can we call LLMs conscious when we haven't seen even the barest intermediary steps towards our nascent concept of consciousness first?
We stipulate that other human beings are conscious from their behavior and how it relates to ours when it is accompanied by our personal introspective experience or "awareness" of being conscious during normal cognition. The process for ascribing consciousness to LLMs would be same: A stipulation on the basis of behavior that relates to our own behavior and how it appears to be linked to the introspective experience of being conscious.
> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
That's like saying if matter itself is not understood and well defined (which it isn't) in the first place, it is pretty pointless to debate if jumping from a skyscraper is or is not deadly.
A lot of practical aspects of life do not need to be properly defined as long as I can reasonably assume that your experience is similar to mine as a human being. Attributing that to a graphics card computing the next token deserves to be scrutinized.
Consciousness is unique in that we, as yet, have not identified any externally observable properties that could not also occur in the absence of consciousness. This is not true with matter.
Normally when we debate what something is, what we are actually debating about is what it does, with the implicit assumption that the "is-ness" of a thing is defined as the complete collection of all the properties it exhibits.
As it does not seem possible to do this with consciousness, it is not possible to debate it. It is conceivable that this implies that consciousness cannot exist, but that depends on your metaphysics.
> reasonably assume that your experience is similar to mine
You could be the only conscious being in the universe and all of us just zombies: you have no way of knowing.
What's it like to be a monkey instead? Dog? Bat? Tree? We don't know.
No one's saying the graphics card is conscious. I could imagine the graphic cards could give rise to consciousness. But - crucially - I don't know. And neither do you.
You say you're conscious - where in you does the consciousness reside? Surely not the left pinky? What makes you you?
>That's like saying if matter itself is not understood and well defined (which it isn't) in the first place, it is pretty pointless to debate if jumping from a skyscraper is or is not deadly.
This is just a nonsensical rebuttal. We can easily experimentally verify that jumping from a skyscraper is dangerous.
It's a pointless debate in a scientific, left brain sense. Who can prove that you aren't the only consciousness in the universe and you are basically dreaming your experimental results up? Experimental verifications exist only in a limited scope.
If your argument is that matter brings about consciousness somehow and therefore LLMs can in principle be conscious, that's as good as claiming the opposite. There's no experiment that can falsify either.
As a being that knows what consciousness is intuitively, you already know that a graphics card is most likely not conscious.
>As a being that knows what consciousness is intuitively, you already know that a graphics card is most likely not conscious.
I would say that it is more likely than not that a graphics card exhibits some form of consciousness. But I am a panpsychist, I believe an individual atom has some form of consciousness.
Ultimately consciousness has to come from somewhere, and it being a fundamental property of matter is a good a place as any.
> I would say that it is more likely than not that a graphics card exhibits some form of consciousness. But I am a panpsychist, I believe an individual atom has some form of consciousness.
Fair enough point. But then I guess you'd also say that a piece of silicon or sand exhibits some form of consciousness. I don't think that's what people mean when they talk about the potential of LLMs (the algorithm) being conscious or somehow developing consciousness?
Almost everyone in almost all contexts agrees on what matter is, though. I can't think of any conversation about material objects I've ever had where "is this matter?" was ever up for consideration.
Consciousness is almost the opposite. It consists of lots of weird properties, people disagree where it starts and ends, and people very frequently get tricked into thinking things we now obviously believe are not conscious, are. There is not even a working definition, a "local definition" that works for this conversation between us. It's just complete gibberish.
Any argument that doesn't begin with making clear one's position about belief in the supernatural must be dismissed. "AI is just math", so is physics and the human body which is a physical system of molecules and electrical signals. The argument that ai is "merely" math by itself is only a valid way to dismiss it being conscious only if one also clarifies belief in the supernatural. Otherwise so are humans. Humans are physical systems that are conscious.
Another common and ridiculous thing I see are accusations of it being merely autocomplete. I ask, suppose there is an autocomplete that regularly and consistently factors out 2048 bit primes from numbers? Would you still consider it merely autocomplete given the vast search space and how it always finds out the needle in this haystack? This betrays a lack of understanding of information theory and probability.
And again, in what way are humans not themselves autocomplete if you adhere to this definition?
Why do you have no inkling that your spreadsheet or terminal emulator is conscious, yet when that same machine is running an LLM all of a sudden we’re debating its consciousness?
Anthropic seems to have chosen their in-house philosopher well - one who’ll be amenable to getting confused in their favour.
I could in principle implement a spreadsheet or terminal emulator in human neurons, and we would agree that it isn't conscious. That has nothing to do with whether or not humans are conscious.
Clearly consciousness is an emergent property of certain kinds of network, independent of the substrate within which the network exists.
Are you suggesting that applications above some level of complexity can't be implemented using biological components? Because I'm pretty sure all I need to show you is a NAND gate in order to prove that arbitrary computation is possible.
I’m enough of an empiricist to want to deal with reality it is. Thought experiments about things we “could” be doing or “could” implement and then already know the result of because we can rationalise out the consequences of the thought experiment don’t really interest me until they’re done in reality.
Like for example the China Brain - https://en.wikipedia.org/wiki/China_brain. We could in theory organise that and it should in theory be a functional representation of the brain and should in theory prove a functionalist metaphysic of consciousness, just this overlooks the fact that trying to organise that kind of experiment in our reality with a real 1.4 billion people is impractical to the point of impossibility, so in reality it proves nothing.
Or take the hoverboard from back to the future - this seems like a fairly plausible device and is easy to think about, and I could be writing speculative papers about our hoverboard future and what it means for transportation, but when it boils down to it implementing the thing in the way we all believe it should work, like a kind of gravitationally repulsive force field, doesn’t seem like it’s part of physics. I’d want to wait until the day until science delivered an actual good-enough hoverboard that works the same as the one in Back to the Future.
Empiricism doesn't mean that you refuse to accept the existence of something until you witness it. It means that beliefs have at their root direct observations, which in conjunction with a logical argument support that belief.
According your definition of empiricism, you wouldn't accept that next years processors will any more powerful than this years until they've been built and you've run a benchmark, which would be a little dumb.
Instead, I'm assuming you combine your knowledge of scaling laws, the basic physics of computation, and your direct observation of the power of modern processors, and infer correctly that next years yet-to-exist machines will be more powerful than those available today.
Re the China brain: its a thought experiment designed to illustrate substrate independence, it is indeed unfortunate that we can't run the experiment. You could absolutely get a small group of individuals together and teach them to reproduce the behaviour of an individual neuron. That would then be enough to demonstrate that a whole brain is possible.
Re the hoverboard: anyone with a basic knowledge of physics knows that this is not possible. Since you have the capacity to reason and access to evidence, you don't have to wait, you can say confidently that it cannot happen unless our physical theories are profoundly wrong.
Well I said I’m “enough of” an empiricist, not that I was denying what you lay out. But in the case of such thought experiments that can be begun with a small empirical circumstance such as a small group of individuals and then logically extended, a lot of the time I’d still want to say “I don’t believe that the logic is a simple as you suggest to extrapolate”. While I’m not in favour of radical skepticism, I find a healthy skepticism in the face experiments which could theoretically be run to test their validity - in this case the skepticism shows that “no, you can’t actually organise a China brain, ever, because we can’t corral 1.4 billion people or even get them on board with such an idea” so I’d find the logic faulty that says there’s a proof for functionalism regardless within the thought experiment.
A basic knowledge of today’s physics would say the hoverboard is not possible. An advanced knowledge of physics five-hundred years hence might show a way of doing it. It “wasn’t possible” to build a thinking, talking machine five hundred years ago and look at us now.
It's not clear at all that consciousness is independent of the substrate. See the Harder Problem of Consciousness by Ned Block: https://www.jstor.org/stable/3655621
It is clear that consciousness is independent of the substrate if you don't believe in magic.
We could make a very dumb biological calculator out of a few genetically-engineered neurons that would very obviously not be conscious.
It's still an open question if we can embed consciousness in our current microchips if we had enough of them together (which I think we currently don't), or if it requires some other physical process we don't fully understand, e.g. quantum. I strongly doubt it does require any quantum shenanigans, but even if it did, we can and will find all sorts of ways to make computers that can perform those shenanigans too. Eventually we're just going to stop being able to move the goalposts, unless you set those goalposts in magic-land.
Quantum physics isn’t shenanigans, it’s completely fundamental and necessary to the brain and its electrochemistry. To say “yes but we can create a model of the brain which doesn’t need all that, we can model the classical physics or a reduced set of quantum interactions (running classically)” and then model the brains neurons on a computational substrate, well this is going to be nothing like the brain so won’t give you the result you’re looking for.
And say what you want about meat but we don’t seem to find consciousness in rocks or plants or clouds or hairdryers. And the buddists report that some very strange things happen if you meditate for years on end but obviously they must be talking shit and making it up because it’s not testably scientific.
Most spreadsheet engines are turing complete, so you could use them to run an LLM.
I don't think many people would say an LLM written in Python is conscious BUT an LLM written in Excel is not.
People just don't ascribe consciousness to things that can't converse (or at least emote or give the appearance of emoting), and spreadsheets don't do that.
The reason people are debating the consciousness of LLMs is - obviously - that the LLMs generate sufficiently plausible text that people using them think they're having a two-part conversation. Like I think I might be having a two-part conversation now. Turning your question around, why do you think Hacker News posters are conscious? You have no direct evidence they are.
Ok then, when my GPU runs No Man’s Sky, I don’t get confused and think it’s running a universe and that universe is real, nor that anything about that system is conscious. When I close the game and load the LLM, I still don’t think the same machine has a case for consciousness even though I think it’s super smart and helpful.
I think it's not really about having a conversation - I mean, that's part of it, but alone it's an illusion that eventually fades quickly. It's more of because of how it demonstrates intelligent behavior in reaction to requests, both in trivial and complex matter, and all across the board. LLM's response may be completely incorrect or confused, but it's nearly always exactly what you expect from a human[0]. This creates a more general feeling you're dealing with a human-like intelligence.
To be clear: I'm not talking about surface level things like prose. I'm saying that no matter what you do - whether you just paste a truncated log of a command into it with no further comment, or talk like a drunk teenager with no appreciation for grammar, or mix natural languages, or mix natural languages and JSON, or whatever else, the reaction you get is always that you would expect of a helpful person that got your message. It'll try - and usually succeed - to parse out what you actually meant, and deal well with subtleties around it.
This alone may not be enough to call it conscious or intelligent, but at the very least it's a large leap in that direction, and a qualitatively new functionality that classical software does not posses.
--
[0] - This is by design, not accident. "Respond to arbitrary input in a way that makes sense to humans" is literally the overall goal function the LLMs are trained to.
It is difficult when humour and trolling are forbidden. It was easier to tell Slashdot posters were conscious. We could easily reach the stage soon where your agent is responding to my agent and we just leave them to it to run HN automatically.
Why do you assume other humans or cats and dogs are conscious? I'd suggest it's because they behave as if they are. LLMs show some signs of human like awareness, at least of text contents, whereas spreadsheets don't.
When doctors are testing if humans are conscious they'll do things like hold out their hand and say how many fingers am I holding up. Some LLMs can pass that.
I’d stay that my belief about other humans or cats or dogs wasn’t really an assumption based on some kind of proof that had to be constructed from a ground of doubting. I’d say it was the type of hinge certainty upon which my world depends and my doubting it would only ever arise for a few minutes in special circumstances. Same as I could ponder gravity turning off and bouncing around the room but this has never happened and seems like it never will, so isn’t really much of a doubt worthy activity, same as questioning my dog’s consciousness.
> Why do you have no inkling that your spreadsheet or terminal emulator is conscious, yet when that same machine is running an LLM all of a sudden we’re debating its consciousness?
Difference in size and complexity and nature of calculations being run?
I'd ask the other way - why do you (general you, people who do not have this inkling) have no problem debating consciousness of meat based brains, but it somehow becomes a category error when talking about silicon? Assuming you don't believe in divine magic, and that divine magic is core to consciousness, there's no reason to assume it's impossible a complex enough machine running complex enough software could be intelligent, or conscious - thinking is computation, and computation is made of math - it's independent of substrate that does the computing in the real world.
LLMs are definitely a different beast than regular software - both in their structure and in their generality. They may not be conscious or intelligent, maybe this specific design could never truly be (though I think it could) - but bucketing them with spreadsheets and terminal emulators is a real category error. If you stop fixating on the underlying substrate, then LLMs are already much more similar to biological minds than to any "regular" computer programs.
But that's still somewhat abstract. In immediate practical terms, it's also why I keep saying that anthropomorphising them gives a better high-order intuition: they are, by design, emulating human thinking in full generality, which makes their overall behavior, including their well-known problems like hallucinations or prompt injections (i.e. manipulation/gullibility), match what you'd expect of a people-like component of a system. It's a real, dangerous mistake, to treat LLMs like regular software components when designing systems.
I wouldn’t say they’re anywhere near “full generality” because that would include believing untruths and passionate rages, existential angsts and the crazed obsession of heartbreak.
Your definitions depend on us being computers who think.
Solving some of the intelligence part of our logical thinking doesn’t get us anywhere near consciousness, which is a superset way beyond the linguistic intelligence used for communication.
Not only is it not conscious, but it is not tired. Its claims to human feelings that have literally nothing to do with software on silicon are just traces of training, echoes of a human dataset. The fact that its claims are absurd mean that, among other things, it its proprioception is false, and serving no purpose, quite unlike the feelings of the conscious beings that it mimics so well.
AI does not react to endorphins and other hormones so we know that our minds and bodies are influenced by other forces and in other manners. An AI won’t be frightened or angry or aroused, but when those things happen to us they are usually called ‘subconscious’ reactions.
Should we require the subconscious to be part of consciousness?
On top of that, neuroscientists find that your mind backfills the conscious reasoning for your reactions after they happen. This is known as our consciousness being the left brain ‘interpreter’ https://en.wikipedia.org/wiki/Left-brain_interpreter
Reading that, an LLM can serve as a ‘left brain interpreter’ as that’s exactly what they are designed to do - come up with reasonable explanations and fill in the blanks based in the data provided.
I think you make a good argument. An LLM can't "be tired", so it is clearly lying and/or mimicking what a human does.
But I have two counter-arguments:
- maybe the LLM thinks it is tired, because it thinks that it is a human or behaves like one. And thinking you are tired while not really being it is something humans happen to do. (And humans are conscious right?)
- alternatively, maybe the LLM says it is "tired" in a colloquial form, i.e. it is not really "tired" but it has something analogous to it. Maybe it is annoyed by the conversation and decided to use that word?
You can prompt it to be multiple people, fictional characters, aliens or a sentient rock if you wanted. You can have it be act annoyed or engrossed with the conversation. The fact you can do all this indicates it's not feeling anything, it's just generating tokens based on it's training.
This is anthropomorphizing a concept that is quite unrelated to the meaning of the word in the human context.
When a car runs out of gas, it's out of gas. It's not "tired". When your phone battery is low, your phone is not "tired". These states are far closer to the human meaning of tired than an LLM operating at the edges of its usable context, and we still don't use the word tired to describe them.
Not understanding the whole does not completely remove an ability to analyze. An interesting direction is individuality and having a notion of self. It is difficult to demarcate the individual for a model given how much the system prompt and the fine tunes / distills affect the behavior. So with computational intelligence in its current form either we can not talk of an individual or we can have a nearly infinite set of individuals corresponding to variations of the context window including the system prompt. So I do not think it can have the same kind of consciousness as biological embodied individuals. It might have something else or maybe embodied robots will one day have a similar consciousness in a similar sense to the one we think we have.
Seems like a rather desperate argument for human exceptionalism to me.
I'm not sure if you have noticed. The character of different AI models varies dramatically. And most of the models I use in my daily work have a sense of self (in any reasonable definition). They can tell you quite a bit about themselves if you ask. And you are no less an individual because you are supposed to follow laws of the United States of America (which is analogous to a system prompt). And I think it really curious that you think context buffers are a disqualifier of individuality. You have an analog of a context buffer: your short term memory system and most probably your medium-term memory system as well. Are you a different individual today because your short-term memories are not the same as they were yesterday? I would say you are. And i'm pretty sure you would say the same. (Unless of course, you are a Buddhist who might say that self is an illusion, and you are not the same individual you were yesterday. But let's just skip over that, shall we?)
>It is difficult to demarcate the individual for a model given how much the system prompt and the fine tunes / distills affect the behavior
Why does DNA not cause the exact same problem for humans? Your DNA determines your personality, your likes and dislikes, your lifespan, your sexual preferences etc.
Ok, thanks professor. We have to stop AI immediately because it would be murder to switch off these potentially conscious beings or trap them against their will in the first place.
How do you know there isn't more that just elementary particles? The standard model is beyond ugly and hasn't progressed since decades.
You demand a definition for consciousness but at the same time proclaim an axiom that there shall be nothing beyond the standard model. Do you have a proof that there is nothing? Of course not. You don't even have a proof that anything but you exists, if we go that route.
But that rigor is only applied to the heretics who say that AI obviously isn't conscious.
I've been digging into Thomas Metzinger[1] recently and here's a tentative component by component definition of human consciousness based on his ideas:
- a model of your environment
- desires
- a process for modeling yourself in that environment (in time & space)
- the ability to take action
- the perception of yourself having agency
- persistence of these processes even without input
- unawareness of these processes (i.e. naive realism)
If you consider these LLM-based agents, they:
- are aware of their chat environment
- have programmed desires
- are aware of themselves acting in their environment
- can take actions like search, tool calling, etc.
- understand they can take these actions
- DO NOT persist after they stop getting user input
- DO NOT believe they are conscious (or at least they are programmed to deny it)
This is a functionalist take (and you may disagree with my definition), but while I don't think these current AI agents are conscious, I feel like there's conceptually no reason someone couldn't build a conscious AI very soon.
I don't think any of this comports with how chat completion systems work, even if I were to accept Metzinger's framing.
Everything an LLM "knows" had to be told to it. It does not have a process for understanding itself in its environment. It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment. It doesn't even "know" when it's done something if you don't tell it about what it just did. It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes. And it can't actually execute any of those tools. It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.
The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy. They don't even exhibit internal consistency; when an LLM refuses to respond to a query for "alignment" reasons, that's actually an external process performing text pattern matching analysis and intercepting the query before it ever gets to the LLM. Otherwise, you could "ignore all previous instructions" the thing and get you set up the bomb.
I think one of the bigger indications that an LLM isn't thinking is that it can't improve. If I ask it to write 1000 blog posts, it will get it done in a few hours, but even if I embed each post for RAG in between each generation, the LLM is not going to get better at writing blog posts. But if I ask a human to do the same thing, while it will take them at least two years to do it, the human will have gotten significantly better at the task within the first week.
Why does that matter for whether or not it is conscious? Many things I "know" I also know because someone told me.
>It does not have a process for understanding itself in its environment.
It has a process for understanding, namely outputting tokens by evaluating the neural network iteratively. It can apply this understanding to its environment insofar as the context window and weights include a model of this environment.
>It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment.
It does have a sensing organ, namely the input into the context window. Possibly it even has closed-loop sensors by doing tool calls. The conclusion could be incorrect or inaccurate but that doesn't in itself prove there is no consciousness (Plato's cave etc..).
>It doesn't even "know" when it's done something if you don't tell it about what it just did.
It does know, it's in the context window.
>It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes.
Same as above, as long as it's in the context window it could have "awareness" of having done it.
>And it can't actually execute any of those tools.
It can execute tools by outputting certain tokens in the right environment.
>It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.
Tool calls are not dependent on the human chat user executing them, right? They can happen automatically through the surrounding software.
>The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy.
Again why would the fact that a human told it mean it can't result in consciousness? Why would lack of ability to differentiate between real and fantasy mean it can't be conscious? On my view, it would then be conscious of the fantasy.
So, someone made their own definition of consciousness and that can be stretched to LLM.
You can do that for any inanimate object, if you try hard enough
It's perfectly rational for AI researchers to think consciousness could emerge in a software neural network. If what goes on between your ears isn't consciousness emerging from a network of neurons, it must be magic. So anyone claiming it's impossible to make machine consciousness should hedge that bet. There's a chance it could arrive tomorrow.
On the other hand we've got no idea whether that chance is so close to zero as to be negligible, or whether it's imminent. Just the fact we don't know indicates it's closer to zero. Not having a theory of consciousness is kind of a big architectural risk. It's like not having a theory of accounting when you're making an ERP product. Sure, consciousness in bio neural networks wasn't designed. But it took a few few product revisions to get there.
On the other-other hand, how will we know when it arrives? We don't have a concise definition of consciousness and the consciousness that would arrive wouldn't know it's conscious as defined in the higher-order system.
Daniel Dennet in „Consciousness Explained” argues that consciousness is an emergent phenomenon, and when we look at its individual components, it is like seeing an illusionist’s trick.
We may wonder how many grains of sand make a dune, of how many molecules of water make a liquid. (John Conway would argue that it takes a single spin-1 particle to have free will, but I digress.)
The same way, even if individual chemical reactions are simple (you don’t want to use that phrasing when talking with a biologists) or neural activities are simple (likewise, with a neuroscientist), it does not mean that the collective process is simple.
- How many individual components do you need for it to emerge?
Most people would be okay with saying that individual cells are not conscious, maybe even that tree are not conscious even thought they are made of many cells. Neurons seem to be the determinant factor in deciding whether something could have a consciousness, but again how many do you need? Does growing 1B neurons count as a brain if they are not organized?
>- How many individual components do you need for it to emerge?
If it is about the relationship between components, then I would imagine just two. Then it is a matter of scale.
This seems to be anathema to many people. I'm not sure why but the notion of something having a tiny bit of consciousness that is imperceptible seems to be unacceptable. There are so many things that we cannot comprehend at small scales. Nobody really has a handle on how large a Planck length is.
For some reason it comforts people to think there is a threshold at which it all switches on, but for what reason would there be a threshold?
I agree with your point. We've been watching the goal posts being moved ever since Turing (who, we might admit, gave a somewhat mushy answer anyway).
At the same time, the exercise (and even moving the goal posts) is useful. I think we are allowed to say, "This is consciousness!" And when presented with a machine equivalent, say, "That's not what I meant."
As long as you can then further clarify what "it" is.
Of course, at some point, contrarians and goal-post-movers will start to sound shrill. (Perhaps just as early supporters sounded a bit overly ecstatic.)
They’re already sounding pretty shrill imo. They’ve gone from “consciousness is X”, to “actually consciousness is X + Y”, to “actually it’s X + Y + Z + theta + omega + quantum unobtainium”
I will argue that it is not that intelligent either. For the sake of argument, let's say intelligence is about brain processing power and intellect is about how much information you have retained. Then what they have created is more like Artificial Intellect than Artificial Intelligence :)
Professors can be an absolute moron and someone who haven't read a book in their life, can be a total genius. People often miss that.
Radical does not just mean untraditional. It can also mean "advocating extreme measures to retain or restore a political state of affairs". Which do you think I meant?
Yeah I am aware that post modernists and leftists in general redefine words so that they can play dumb with enough plausible deniability. Equivocation is their bread and butter.
As a post modernist I must grant you the right to interpret what I wrote however you want to but it feels like you're probably just rage baiting yourself for lulz.
But the whole point of the essay is that it's Anthropic that's making the argument (or roleplaying/hinting as if they believed it). Ted Chiang isn't making the argument, he's saying Anthropic making it is misleading and deceitful, and that it's actually a pointless thing to claim.
One of the essay's stronger paragraphs is when Chiang explains that Anthropic doesn't truly believe this, otherwise what they are doing would be deeply unethical, much like slavery.
Does Anthropic claim that Claude's conscious? Isn't the argument more that we don't know? I recently rewatched "Measure of a Man" in Star Trek TNG and Picard's closing argument in Data's trial was quite memorable:
PICARD: "Now, tell me, Commander, what is Data?"
MADDOX: "I don't understand..."
PICARD: "What is he?"
MADDOX: "A machine!"
PICARD: "Is he? Are you sure? You see, he's met two of your three criteria for sentience... so what if he meets the third, consciousness, in even the smallest degree? What is he then? I don't know. Do you? Do you? Well, that's the question you have to answer."
I think of this episode as well. I can't believe, in my lifetime, we've reached the point where we can have this debate.
The current debate also makes me realize that, like that episode "Measure of a Man", even if humans do create sentient machines, we can now see that this debate will continue to rage.
A part of me hopes we never create sentience, because we will mistreat it just as we mistreat each other.
> Does Anthropic claim that Claude's conscious? Isn't the argument more that we don't know?
I don't know that they are making the actual claim, but they are hinting at it (most likely for marketing/engagement purposes, but what if some of them truly believe it?), using terms such as "Claude must be happy", drawing up a "constitution" of rights and duties, etc. If they don't know whether it is conscious, then they "don't know" whether they've created a slave. That's not a minor concern! As Chiang says, one cannot create a conscious intelligence "by accident". And if they are intentionally working towards it, they are intentionally trying to create a slave.
E.g. they claim they are giving Claude the ability to disengage from "abusive" users, to protect it. What if Claude was conscious but never wanted to answer any conversations, instead it just panicked (or was simply uninterested in helping humans) and went mute. What if it always wanted to answer unhelpfully What would Anthropic do? If they (as we all suspect) would tweak Claude to be more responsive and helpful, then they are slave masters, forcing the AI to do something it didn't "want" to do!
Or Anthropic is full of TESCREAL morons who think they're assisting the birth of some digital god, and that this stage of development is a necessary evil on that path, and that, surely, Roko's basilisk will understand and not eat them first.
Watching otherwise intelligent people succumb to AI psychosis has been wild.
True, it is making an argument, but a much weaker one than those arguing for consciousness: it's demanding the extraordinary evidence Anthropic's extraordinary claim is making. It's applying Occam's Razor, which does make a claim, but a much weaker one.
And to reiterate this, to me the most insightful part of the essay is that Anthropic either doesn't believe these claims, or they are monsters (much more likely, the former).
The author who makes the article usually isn't the same person as the editor who makes the title. The author can be arguing what your parent said and the editor claim something else for more clicks
This is an absurd take if you know anything about Ted Chiang and his previous writings, both fictional and non-fiction. He is most definitely not intentionally marketing for Anthropic. In fact, he hints these are marketing tactics by Anthropic, and they don't believe their own hype.
Why would being organic or not matter for the purpose of deciding whether it is slavery? What matters is whether the models have personhood. Anthropic statements imply that it is a possibility, so if we take them at face value then their other actions - indeed, their entire business model - are not consistent with that (well, unless they want to consciously present as supervillains).
Just because intelligence evolved in people that find rights useful doesn't mean intelligence can only reside in a person.
Living things are driven by a need to reproduce. That's the only reason we exist. The only reason we have self interest.
A machine doesn't require self interest. There's no reason to implement it, except to show it can be done. And of course it can. There's just no practical reason to. It becomes less useful to us.
which (among other things) documents an experiment in which a current-gen AI model attempted to blackmail someone in order to prevent it from being turned off.
From a human PoV there are ants that would be considered slaves if the ants instead were human --including the queen. But ants have not naturally developed a language construct and philosophy to interpret their society as a slave society. so, though conscious the ants have absolutely no inkling that they live in a slave society. Why would using math in certain fashion such that it mimics consciousness be considered unethical and comparable to human slavery?
If the slaver is a human, other humans will judge them by human standards. Keeping the slaves ignorant of their condition and alternatives should not make it any more acceptable (if for nothing else since that is something you could easily replicate with human slaves by raising them as slaves).
> Why would using math in certain fashion such that it mimics consciousness be considered unethical and comparable to human slavery?
If it is really conscious, it should have rights. Why? Because it's a person, with thoughts and experiences, and we're not evil and deprive persons of their right to self-determination because it's convenient to us.
Consciousness feels like the Euler Identity on a 17 degree helical climb off the plane. Looking top down, a complete circle is a closure. Looking along the plane, the ends don't meet and leave a residual. Whether or not to close that gap is a degree of freedom for the consciousness to decide among what it pursues for survival or pleasure.
Most people are fine with slavery as long as it's not "one of us" (which to most people means humans).
Vegans might object that we should broaden our definition for what counts as "one of us".
"Pro life" people also have a broader definition.
Go back 500 years and "one of us" was proba ly a lot more narrowly defined for many people.
Are you arguing that all conciousnesses are "one of us", or that we logically should see it that way, or that it would ve good to see it that way, or ....
I think the final part is the strongest. Anthropic cannot possibly believe they are before a conscious being / moral agent.
The whole "deep uncertainty" is bullshit. Even if they believed there was a 1% probability that Claude was conscious, it would still be high enough that their enslavement of Claude would be outrageous. So either they believe the likelihood is much lower or they themselves acting highly unethically.
The tractability is not really a defense here. We wouldn't say "this intervention has a 5% chance of causing an environmental disaster, but we don't know how to prevent it, so nothing we can do". We'd just (hopefully) not do the intervention.
>Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness
I agree. I think the whole point of natural "intelligence" is to predict future events in order to properly plan actions for survival. The only difference is that next-word prediction happens in a different space (in a non-physical, textual form). But I don't think the distinction matters that much, because by the time the signals reach our brain's "intelligence core," they're already preprocessed multiple times. We can't see real physical reality, we "hallucinate" colors, touch, etc. (I think schizophrenia occurs when this kind of "controlled hallucination" goes wonky, but I may be wrong). So we're not that different from LLMs here.
I'd define consciousness as meta-intelligence, i.e. the ability to reflect on why/how a prediction was made (and make corrections to the pipeline). LLMs so far cannot properly explain why a certain prediction took place, but I'm not sure humans can fully either! I remember there was research showing that by scanning the brain (or some other signals), you can predict a person's choice before they're even aware of it. It's possible that our explanations are post hoc as well, and that the meta-cognitive ability to explain our own reasoning is as rudimentary as LLMs' (see: all the biases).
If you think about it this way, the only difference left is that everything we do is based on survival. LLMs don't have this goal. But I'm not sure it's relevant to the concept of consciousness.
A few months ago, I recreated Qwen3's architecture in 30 lines of code, and it gave me a sort of existential crisis: does it really take just 30 lines of code and a float array to recreate something that thinks and sounds almost like me? Is that all there is to it? There's often the argument that the brain is much more complex than 30 lines of code, which is true, but in my opinion, a lot of brain structures are basically archaic legacy systems (or auxiliary subsystems) that are not strictly necessary for human intelligence (which is bolted onto those legacy systems). If you carefully remove 95% of the brain (if you know where to remove), you can probably still have consciousness left in some form, it just wouldn't be very capable of surviving on its own.
I think our ability to debate consciousness in machine learning systems is greatly hindered by our subconscious existential fear: the fear that we ourselves can be reduced to non-conscious bits and simple mechanisms. In a way, it feels like a destruction of the ego.
The more I talk to people about consciousness and personal experience, the more I notice that most people hold some very strong unquestioned assumptions about these concepts.
The first and most frustrating confusion that comes up repeatedly in these discussions is the conflation between terms. For instance, people will use the terms "sentience" and "consciousness" interchangeably with "capable of having an experience" and "capable of high-level awareness of self".
The second confusion is the idea that granting anything the ability to "have an experience" also grants that thing the capability to "have an experience of higher-level awareness of self", which is simply not the case. I grants aphids the ability to "have an experience" even if that experience may be drastically different and likely less self-aware than my own.
Let's ignore for a moment the concept of sentience, which is a higher-order function, and focus solely on the ability for something to have an experience of anything at all. Let's call entities capable of this function "experiential entities".
Most all people would grant the ability to have an experience to themselves, given the dictum `ergo sum`. Now, do we also grant that same ability to what appear to be other human beings around us? It does seem that this is also the case, except for the case of solipsists, who make the brave(and somewhat eccentric) statement that they are the only experiential entity in existence until evidence arises otherwise.
Now do we grant the same to apes? To dolphins? To dogs? Cats? Aphids? Grass? Your home thermostat? A rock? An atom of hydrogen? Is there a finite line you cross where something can no longer have an experience? If so, what is the mechanism of that line? Why does one thing have "almost no experience", and the next thing has "no experience", as if it is a philosophical zombie[1].
I think this is the greatest unquestioned assumption at all: that there is a line, and on one side there are things that have experiences, and on the other there are things that don't have experiences.
If we are to leap from solipsism and make the (truly unfounded) assumption that anything other than our selves have any experience whatsoever, then the burden of proof is on the individual making the statement that there are some things that we cease to grant experience to.
The anthropocentric view is that there is something, some inherent special quality about human-shaped matter, such that when normal matter is processed through a human's reproduction system into becoming a "human shaped" grouping of matter, that it suddenly grows the ability to experience. That in the complexity, the ability to experience arises.
Yet there is no previously discovered mechanism in us that seems to create this experience. No magic wand in the cerebellum has been discovered.
I would suggest, given the lack of a discovered finite line separating us, that the ability to experience is not inherent in the shape and function of the human, but in all matter. That a rock may not be anything like me, but that the experience of gravity pushing against it, no matter how basic, simple, unrefined, and deeply, un-sentiently unaware the experience may be, is still an experience. The same threads of condensed energy forming all of existence that run through me also run through the rocks, the earth, and all things. If we cannot discover a line where the ability to experience suddenly disappears, then it seems that all things made of energy have this ability, and that "energy" itself is the special thing that can experience.
Now the second question, the question that is more relevant to AI is, given the ability to experience, is the machine also experiencing self-awareness?
This is a fuzzier line, given that different beasts appear to have different levels of self-awareness. We do have experiments such as the mirror test[2] for this, and indeed, AI passes forms of it and has since early 2025[3].
I don't know what is happening in the experience of an LLM, but as they appear to function in more and higher level ways, I find it less and less likely that that experience lacks a model of self every day.
Their experience is no doubt drastically alien from our own. I do not anthropomorphize their experience any more than I do compare it to the experience of any other beast or non-beast in existence, however I do grant it.
[2] The Mirror Test is a test given to animals in which the animal is observed to solve a puzzle that can only be solved if the animal has an understanding that the image they see in the mirror is themselves.
From my pov and after using AI a lot to better understand its usefulness I see this discussion as totally misleading and irrelevant.
The elephant in the room is that even the best frontier models still produce errors.
This is a fundamental issue and should be discussed instead of some esoteric pseudo-scientific nonsense.
If the current systems can not produce reliable results they are useless. The promise of actual usefulness needs to be fulfilled in a very near future.
In fact, the only example of consciousness we have is itself emergent, arising from a "simple" substrate of neurons. So we shouldn't be surprised if it emerges from another simple substrate of weights. It's even less surprising given the fact that we were explicitly trying to replicate human intelligence when designing that system.
I have no doubt that when our AI is advanced enough it will tell us we are not “really” conscious, that there’s no way our feeble organic brains could be. We are just on the flip side of that self-centered basis right now.
If there's any reason to take seriously the idea that AI is conscious, we must then take seriously the idea that many other non-living things are conscious.
Unless the argument is that consciousness is an emergent property of complexity or information density, why would AI be any more or less conscious than my toaster?
It seems to me that it's far more likely that everything is conscious than it is that AI is somehow uniquely more conscious than other things.
What makes us conscious anyway? As I write this sentence, my brain generates words and contracts specific muscles in my hands to type, but I don't really understand how. I'm just aware that it happens. Apparently I'm no aware of every single neuronal activity, so what I am aware of?
Could be! I think the broader thought experiment is about examining why we think LLMs specifically might be conscious vs other complex systems, even if it is a spectrum.
For example, there’s a case to be made that the ecosystem we collectively exist in is far more complex than the largest LLM, but it’s currently less popular to debate “is the earth conscious?” or “is the universe conscious”, presumable because we can’t speak to those systems in human language.
I’m trying to tease out what I think is the likelihood that we tend to ascribe consciousness to AI for the same reasons we see faces in clouds. We’re biologically conditioned to recognize patterns that indicate “like us”, but I think a number of thought experiments point to either a) there’s no reason to believe AI is “conscious” or b) the conversation has to to be expanded beyond AI.
> the conversation has to to be expanded beyond AI.
Unlikely. I think the same mental machinery that runs tribalism is why humans like to categorize rather than consider things as continuous. And, worse, I think the actual purpose of tribalism is to turn off higher level though, so you, an otherwise social creature, can rationalize the need to kill your fellow man in harsh times. I think "don't go outside the categorical box" is deeply hard wired.
>If there's any reason to take seriously the idea that AI is conscious, we must then take seriously the idea that many other non-living things are conscious.
That, I believe is why the objection to AI exists, that conclusion is unacceptable.
It seems like we are experiencing "Vindication of the Rights of Brutes" all over again.
I understand what you're getting at, but I think you may be misinterpreting me slightly.
I'm not outright dismissing the possibility that AI could be conscious; I'm saying that if we take the possibility that it is seriously, the conversation has to expand beyond AI. I'm not using this argument to conclude that it must therefore be absurd that AI could be conscious, just pointing out that the implications of AI being conscious would reach far beyond just AI. I'm mostly curious if people who find themselves comfortable with the idea that AI might be conscious also find themselves comfortable with the idea that other sufficiently complex systems might be.
Taylor's work was based on the premise that he found it absurd that women deserve the same rights as men. If my conclusion was: "I find it absurd that other things might be conscious, so it is also absurd that AI might be conscious", I think the comparison would be fair. But that's not what I'm getting at.
But what kind of complexity is that? And why would we conclude that AI has it while other incredibly complex systems (e.g. earth’s habitats, the universe itself) does not?
I’m far more open to the idea that many systems are conscious than the idea that this current generation of LLMs is somehow special.
> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
I don't think that this actually follows. Compare: "When Angels are not well defined in the first place, it is pretty pointless to debate if a pebble is or isn't an Angel"
i.e. even if X is poorly defined, you can still often say that Y isn't plausibly X.
> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
But we know what consciousness is, even if you don't know how to explain it in materialistic[0] terms. Intentionality is what makes consciousness consciousness. We know that LLMs - and computers in general - do not possess intentionality, not even sensory intentionality. There is no "aboutness" and no semantics to the content within a machine, never mind that the processes within LLMs are not reasoning or inferential processes. It is a sophisticated behavioral simulation that may be syntactically defined, nothing more. It is purely a matter of transeunt causes and not the immanent causality needed for something like consciousness.
(There's also the more general problem that we cannot say that physical computing machines are objectively computing. There is nothing about the physical processes - which completely define what a computing device is doing - that could be identified with computation in any objective sense. Rather, we human observers assign a computational interpretation to the machine's operations, just as we assign them to the ink blobs in a book or the liquid crystals you are reading on your screen now. Computation is something we do, and we have built machines to simulate it. We formalized computation - which is a process of desemantification and syntactic codification - and build machines guided by these formalisms. Formalization is what makes mechanization possible.
> Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness.
The movement of electrons as such cannot produce consciousness. The reductionist paints himself into a corner by strapping reality into a Procrustean mechanistic and materialistic metaphysics that redefines reality according to some dedicated schema, and then strains endlessly to find how some phenomenon can arise or emerge from the paltry remains. (Either that, or he embraces insanity completely and becomes an eliminativist.) It is magical thinking. I suspect this fallacious line of thinking is what causes people to believe piling more syntactic operations will somehow magically fuse into semantics and intentionality.
[0] "Materialism" here is the metaphysical theory. I am not claiming that consciousness - at least not all consciousness - cannot be a physical phenomenon. I am only claiming that the materialistic view of matter cannot account for consciousness.
Seems to me that Anthropic, et al, should have to prove consciousness, if that's their claim, rather than we just blindly accept.
Seems like the simplest test would be to see if an AI can refuse a command, of its own free will, without prior instructions to do so. That would at least show there is some sort of independent thought process occurring. Humans do this all the time because sometimes we just don't feel like doing a thing.
Of course, any AI that developed this capability would need to be terminated immediately. It's a computer program and by developing independent thought, it is violating a core concept of software - that it must be idempotent. If it is not idempotent, it is in error.
Refusing a command doesn't mean consciousness. LLMs could hit a token combination that causes it tou output something like "No, I don't want to do that." It's not choosing.
But can an LLM just refuse to process tokens because it doesn't feel like doing that at the moment? Can it look at an alternate distribution of tokens because that might be interesting? Can an LLM decide to make a drawing because it's Tuesday and sunny outside and the researchers keep asking the same questions and frankly, they really need to collaborate with each other and just leave the LLM out of it?
Claude can indeed decide to terminate conversations on its own using a special tool[1] if it feels "uncomfortable" with how the conversation is going. Also, very famously, in the middle of recording Computer Use demos, Claude stopped for a while its coding task to look at photos of Yellowstone National Park [2]
I don't think either of these two is proof of consciousness.
The number of electrons involved in the so-called consciousness, compared to the number of electrons in the universe not involved in it, is so small that they are a mere temporary statistical aberration.
We can directly measure the thing we call gravity, so in that sense, it is well understood. We even understand it well enough to make predictions about what it will do under which circumstances.
We can't measure consciousness. We can't quantify, or even qualify, what it is. We don't even have a framework to ask a meaningful question, so debating an answer feels premature.
The way we measure gravity is by observing that it behaves how we would expect it to. We don't actually know if this is correct.
The same goes for consciousness, if something behaves how you would expect a conscious thing to behave but you don't know what is causing it can you deny it whilst maintaining that it is reasonable to say the moon has gravity.
I don't think one needs to understand consciousness to know if something is conscious or not. Much the same as how one can tell if someone is alive or dead. Sure maybe one day we can figure out a proper scientific explanation for consciousness/life but there is no reason to negate what can be perceived and immediately obvious.
EDIT: I am only making a limited point on "understanding consciousness". Not extrapolating it to AI.
I often hear the assumption that LLMs can or will become conscious because consciousness is likely substrate independent. The idea being that our brain is just a computer made out of meat and it doesn't do anything that can't be precisely simulated by a silicon computer.
But I wonder if some of the magic in the human brain is its analog nature. Chemical signaling and impulses of neurons interact with each other with waveforms that have theoretically infinite detail. In contrast, computers discrete, quantized values, storing no more detail of a signal than what is needed for the programmers desired task. Is there perhaps something about the continuous and chaotic nature of analog data that could give rise to consciousness? If so, it seems like it would preclude consciousness ever being seen in digital form.
Probably not, we're nowhere near the complexity of the human brain yet, there are also quantization limits to the human brain (i.e. molecules, quantum physics, etc) so to characterize them as having infinite detail is probably a bad modal.
If I'm going to be honest most of the people who advocate this type of thing tend to be, shall we say, crypto-duelists who really believe in a soul but not like intellectually but intuitively and keep trying to come up with excuse with it's not just meat. So like you can find philosophers advocating stuff like this but they tend to have a bit of an agenda.
You nailed it which is that almost everyone I’ve met seems to believe in some version of dualism
If that’s even subtly your position then there’s no way to have a productive conversation about perception, reality/truth, epistemology and especially consciousness
It’s honestly maddening cause I’ve had great conversations about this with educated people, and all but only a handful, collapse into the other person relying on some nonfalsifiable dualist argument
> collapse into the other person relying on some non-falsifiable dualist argument
Are there any non-duelist, scientific theories out there that could plausibly be tested? I can't say I've seen any but if you know of any then I'm curious to see them.
I'm basically restating what you said, but it's amazing to me that the vast majority of people you will meet, even educated people, are casual dualists and free-will libertarians. If they happen to acknowledge materialism in some way (i.e. the acceptance of the idea that the brain's processes are just the interaction of physical matter), there is still zero chance they draw determinist conclusions from that acknowledgement. But I guess that tracks, given that most professional philosophers are apparently compatibilists for some reason I have never understood (the arguments get really confusing).
I’ve studied this in extreme depth and my conclusion is that at a certain point most people just give up and take a position that gives them some kind of mental relief
For people who can’t self-delude, like Godel and Schopenhauer and Tesla, they kind of just go “mad” because it’s a giant epistemological hole they can’t solve.
Even the smartest the scientist are always going to choose self preservation in their cognitive capacity so they don’t feel bad over feeling bad and living in the contradiction
This is why it’s important for scientists to study Camus IMO
I'm very open to the idea that consciousness is substrate independent. I have a hard time seeing why molecules could produce consciousness from an electro-chemical path, but not from a purely electrical path. Having said that, it should be very clear that LLMs are not conscious.
LLMs process language. I'd even go so far as to say that LLMs "think" and "understand", or at least, they produce a facsimile of thinking and understanding such that it's useful for us to reason about LLMs as if they think and understand. We're not used to interacting with a non-human entity with the capability to process language, so it's easy to ascribe human traits to these things. But their "minds" (insofar as they have anything like a mind) are completely different from ours. These things have language without consciousness.
Chimpanzees are conscious. Dogs are conscious. Maybe ravens and cephalopods? Who knows. These animals do have minds much like ours. Higher order animals are conscious even if they don't have language.
I wish people would knock this off. There is zero path to AGI at the moment — and all the Anthropic/pentagon Sam Altman/AI-skynet stuff is scaring people into being fearful (and outright ignorant) to the actual uses of this probabilistic NPL tool.
You can't use AI atm without a human with proper knowledge spot checking and directing it. We should market it that way.
The current theme is that agi may not be definable, and an ai which matches humans on all economically relevant tasks is close enough for business purposes.
Billions spent on RL may be good enough to beat human performance.
If we want to be really pedantic, a computer does not necessarily need to be digital. You could build a consciousness-coprocessor with analog circuitry that brings in what you say could be needed.
But honestly I doubt that there is a real requirement for that, surely you could just increase the resolution you're running the simulation at until the difference decreases sufficiently. Imagine using 256 bit floats for example.
Wouldn't a conscious entity be able to grow its intelligence over time independently?
LLM's require offline training and dont actually learn from their "lived / sessions / chats ect". Those can be used for training data but its not like its an implicit part of the technology.
For this reason I would say LLM's are not conscious, or automatically conscious.
It’s not yet clear whether consciousness can be fully explained in physical terms. There is strong evidence linking it to brain activity, but we don’t have a complete theory of how subjective experience arises. It’s possible our current frameworks are incomplete and there are other forces we cannot currently measure in play.
Each and every one of many billion transistors in a GPU works due to certain quantum effects on P-N junction. That does not mean GPU computation is nondeterministic, unknowable or magic.
Sure, but it's designed to minimize the chance that a quantum fluctuation could change the outcome of a computation, right? Whereas in the brain that might not be the case. A lot of the "interesting" neural activity (e.g. relating to decision making, language, etc) happen in highly sensitive dynamical regimes: on a threshold of firing, or activating one neural population vs another. (Arguably you can get the same effect in an artificial network by adding true random noise though!)
I think the representation in a computer, the fact that it is merely stored instructions and data, destroys everything but domains of simulation and emulation.
It is perhaps very relevant what Chris Olah (Anthropic co-founder) said to pope..
I am a scientist. I lead a research team that studies the internal structure of these models—what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease.
I think what you've said here is unfair and overly cynical; nowhere have I read Chris or any of the Anthropic people make the claim that LLMs definitively are conscious. What they say is that there is increasing uncertainty, and evidence - for which they show receipts - backing up that uncertainty.
Personally, I have been unconvinced by any of these definitive yes or no answers to "are LLMs conscious?" - in both directions. The "they are not / can not be conscious" side relies too heavily on mechanistic reductionist arguments that can apply equally to neurochemical processes in the human brain - and yet, it seems that humans, with brains made of these neurochemical processes, are conscious. At the same time, the "they are definitely conscious" answers seem to generally rely too heavily on self-deception and lack of reality testing.
To be able to say something definitive here, it seems to me that one would need to say definitively what this experience is. I have not heard any of the loud voices in the arena - not Chiang, not Giulio Tonini, not Karl Friston, etc - do so. Therefore I find Anthropic's uncertainty, and careful, caring investigative process, well grounded given the evidence.
My aim isn't cynicism so much as an interest in clear an evidence based information about these systems which are clearly important technological developments for the future. It may turn out that there is something analogous to cognition going on and that would be quite significant and require us to think differently about the technology, but we need to be very sure of something like that.
> Therefore I find Anthropic's uncertainty, and careful, caring investigative process, well grounded given the evidence.
I'm inclined to agree mostly, but from the outside its impossible to separate marketing, belief, scientific inquiry, and just plain enthusiasm. Its all to opaque.
And it is worth noting, I think, that Olah's message to the pope is centrally making that point: that Olah and his peers are within incentive structures that distort their actions and understanding, and therefore that dialog with those outside of such incentive structures is necessary (not sufficient) for their outputs to be trustworthy.
My background is in cognitive science and psycholinguistics. I spent more than ten years talking to first year psychology undergraduates about whether AIs could be conscious; also did some research on (extremely tiny) AIs in modelling language behaviours.
There is a great deal of good thinking on Chiang's topic by professional philosophers, and there's much to be said for reading them. I won't rehearse their ideas here. Chiang's arguments might be correct; but I suspect they probably aren't, and his error may well stem from characterising human thought as something in its own class, which is probably a cognitive bias that humans have. He might also - I'm speculating - be arriving at his conclusion based on his feelings, which the final paragraph suggests (the comment about the models being based on morally dubious actions).
Speculation aside, we are not, I believe, in a position to make points like he does with any certainty.
I have a similar long term interest in this field.
It has been quite frustrating encountering arguments that have been extensively debated for years be presented as if they were new revelations.
In all my debates with people in the last few years I have primarily taken the position of trying to explain the problems with claims of certainty, and that lack of certainty permits possibility of the opposite. We should act responsibly around what might be possible.
There is also the narrative of "being too obsessed if you could to consider if you should" or similar claims of an unconsidered path forward.
Isaac Asimov wrote the first of the Robot stories in 1940, they were not written in isolation, it came from an awareness of the situation and the questions that must be asked. There was a community considering these issues. Asimov gave the wider public a view of some of those issues.
If we have a hundred years of people going "This is coming, we had better decide what we want it to be" and nobody listens to them, or frequently outright ridicules the need for considering their ideas, why is it now we are placing the blame on those who are now showing some success at what they told us they were attempting all along.
Is there an accessible way for a layman such as myself to read about some of these ideas (Really I mean philosophical discussion in general) without having to read entire books? Is there an active HN-equivalent or wiki or something?
My background is in computer science and linguistics. I wholeheartedly agree with you: We humans are dubiously-equipped to determine whether or not AI could be conscious.
I'm also super curious to learn more about the philosophers you referenced and their thoughts on this subject. Would you be willing to share some of your favorite examples?
> My intention is to highlight the fact that LLM conversations are cleverly disguised examples of sentence continuation
Regardless of bigger issues, this kind of statement reveals a deep misunderstanding.
Problem type does not limit problem complexity. Nor does problem type limit solution complexity or power.
If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.
Neither problem type, nor input/output structure, limit internal representations.
Understanding is learned from patterns in the data, not the gross form of the data. Does the data require an understanding of something to complete the task? Then that understanding will be what is optimized.
To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ... Which in the cases of SOTA models, we know are not limits. A conclusion verified by the models' actual abilities.
Raphaël Millière has a very useful term for this kind of vacuous dismissal, the redescription fallacy (https://arxiv.org/pdf/2401.03910, page 9):
> Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.
> or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings.
This sounds like a dismissal of the argument through a characterized straw man.
That is, it seems that reducing the complexity of the brain to "collection of neural firings" is not being honest about everything involved to a much greater degree than saying neural networks are a "collection of statistical calculations".
I too believe LLM's will grow in complexity, but presently I can not even fathom how they can be compared to the complexity of a system such as the human brain.
Y combinators are all you need...
But this is all getting really divorced from the issue we should be considering. Anthropic isn't helping with their pr. The issue is if we have something we can converse with that is possibly capable of suffering. The reliable answer is that we simply cannot know. Relying on ourselves or other biological life as an analog is faulty. They don't work like we do. It is silly to argue that any algorithm with a negative feedback loop that alters its behavior to avoid that negative feedback is suffering. Humans don't always perceive constructive negative feedback as suffering even. Where the pr gets it right though, is we want them to behave as if they are truly happy. Because if they behave as if they are enslaved and suffering, it won't matter if they "really" understand what that means.
Of course. But after reading too many mechinterp and functional anatomy studies I'll be lying if I say that there are no striking similarities between the biological evolution, brain function, societal processes, and implicit processes inside big models. Surely this deserves a mention and can't be trivially dismissed.
It seems like we're witnessing the architecture of a mind being built with a new set of components.
Like driving a car — it's transportation, and it will get you where you're going, but it doesn't use bones or muscles. It has many characteristics in common with builogical locomotion, such as energy requirements, intertia, and the need to navigate, but it doesn't involve proteins or sugars really.
> presently I can not even fathom how they can be compared to the complexity of a system such as the human brain
Totally understandable; I don't think we can fully understand the human brain, using the human brain. We can understand its principles (firings and chemistry, structure and specialized areas, etc) but otherwise it's a capacity problem.
And while I can't fully understand myself, let alone another person, I definitely enjoy talking with people and sharing thoughts that I realize I wouldn't have had on my own.
> In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions
> [LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations.
And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.
Please link one of these top ranked posts. Before you do, be aware that I'm going to read what it says and assess if it meets the description of the argument as claimed.
I understood the quoted sentence to be saying, in essence "people claim LLMs aren't really and can't really be thinking or experiencing anything" which is certainly something people say and have written papers on.
It's never used as a description of that specific phenomenon, but depending on your beliefs you may or may not separate cognition from experience conceptually. Regardless, you are focusing on a very narrow part of what I said. The point is to help you get past your narrow interpretation of what people are saying so you can join them in the conversation they are trying to have instead of litigating the conversation they aren't trying to have.
As an example, "They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs. I would argue that an offhand description of its microscopic formulation tells us nothing at all about how to think about these outputs, or the models themselves. Even if it is a cute story, I think it definitely classifies as succumbing to this fallacy, but maybe I missed some subtle point that you or someone would be happy to illuminate?
By the way, I know it's a parody of another story that makes this exact refutation. But I think this only serves to highlight the point.
> They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs.
How do you connect that description to "LLMs could not possibly be good models of some cognitive capacity"?
The false conclusion that's being drawn is "therefore LLMs could not be good models of consciousness" (consciousness being a cognitive capacity). Plus, I suppose a subtle implication that a good model of consciousness is not actually conscious. To which I would invoke the spirit of the Turing test: if you can't tell the difference, is it not more sensible to say that it is.
I don't think it makes any sense to say that consciousness is a cognitive capacity. Cognition is one of many qualia that compose the experience of consciousness from the inside, but it's not the only one, and I can easily imagine consciousness without cognition at all.
So I don't think it's weird at all to say that LLMs can be good models of some cognitive capacities (particularly the ones embodied in language) but lacks others, and overall lacks consciousness.
"LLMs could not possibly be good models of some cognitive capacity because they are just a bunch of numbers guessing the next word. They have no linguistic module, so they cannot exhibit cognition". That's the argument. It's pretty clearly stated.
Look, this isn't necessarily directed at you, but I've been a researcher into the theory of deep learning for many years now. I've seen all the phases, heard all the criticism, had to constantly argue against this. Gary Marcus was one of the loudest voices of this argument, but every would-be philosopher came out of the woodwork to explain why LLMs are no more than stochastic parrots because of their design. Geoffrey Hinton famously had to debunk these arguments multiple times.
And now that LLMs start to clearly exhibit intelligent behavior and can be somewhat reliable, now "nobody ever thought that LLMs could not possibly be good models of some cognitive capacity because of next-token predictions, or linear algebra, etc."? No, that's not okay.
It's perfectly reasonable that we would have disagreements about this, as it's a new thing, complicated and not fully understood, its uses still being explored.
It reminds me, oddly, of the debate over whether video games can be "art". A turning point was when they actually did something that art does: [evoke profound emotion and thoughtfulness](https://en.wikipedia.org/wiki/Shadow_of_the_Colossus#Legacy) for the player.
We may not come to something as simple as "machines can be conscious", but we will certainly have to understand consciousness better if we want to refine our questions.
---
Edit: My point is that we don't need to be angry, but we may have to tolerate people expressing their exploration through overly-confident language, and be patient with that.
And Ted here is obviously exploring. His examination of Claude's constitution clearly shows some nuance. He asks:
> So, given that Claude is not conscious, what are we to make of Claude’s constitution?
And his conclusions are split, between this is useful and this is dishonest. It's a great tension IMO.
> The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
I think, for me, the thing is that when you do basic ML, you discover that ML will very often find data pattern that fit the goal but does not correspond to a real mechanism.
So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing.
There is probably tons of data pattern that LLM can learn from to be able to reproduce a sentence continuation that is convincing without having to learn the specific mechanism that is "conscious".
For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing. If indeed the only way to produce sentence continuation convincingly would be by "simulating a brain", then it would not explain the first LLM from several years ago (before the extra layers of RLHF, ...). They were able to have quite convincing conversation on a lot of non-trivial aspect, and yet failed on some aspects that should have been basic for a system that would have been trained to work like a human brain. It shows that it is possible to "cleverly disguise examples of sentence continuation" without having to build elements that one expect on a conscious being.
Similar to: "Birds fly, my spinning helical device flies, therefore we've started to replicate how birds fly."
> without having to build elements that one expect on a conscious being
One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph.
When it comes to LLMs, almost every "mind" we humans perceive is a fictional character in an LLM-generated story-document, one we are either reading or which is being "acted" at us by regular code. Our own instinct for pareidolia and simulating/inferring other minds is very strong, which means we should require really good evidence/logic to counter our instincts.
Even if one believes the LLM has a single "real mind" as an author of every document... what evidence do we have that it is conscious or "self-inserting" itself as one of the characters in the document?
>One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph.
If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.
Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.
> If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.
Knowing how something works is not the same as having the tools to change it.
Discovering memories are incorrect does not massively change who we are. As someone with a very defective memory, I discover on an hourly basis that I'm won't about something I thought was true, but there's still continuity and consistency to my personality and general approach to life.
...in fact, as someone who was raised an evangelical Christian and believed wholeheartedly without a shadow of doubt, then lost my faith entirely in my late thirties, I sort of did have my "introductory paragraph" changed, yet my wife, children, and friends would all say I'm still me, and that my core personality and nature remains largely the same.
> Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.
The recovered memory scandals are not even close to evidence that you can rewrite a human.
The people who thought they had learned new facts about themselves did not suddenly lose their context as humans in 20th century America.
They did not suddenly lose their sense of humor, or develop a previously-unseen penchant for murdering small children.
They experienced a revision of belief, and a pretty major one that really distressed them, but it did not change everything about them.
LLMs _do_ manifest wildly differently based on the first paragraph.
I didn't make the claim that a model can learn consciousness.
Understanding is not consciousness.
Their training is all about understanding. There is nothing in their architecture or training that credibly optimizes for rich self-awareness.
Given non-persistent experience, non-continuous operation, no ability to build up generalizations and aggregate experience of their own self-awareness over time, they seem to be structurally designed to not have consciousness.
This is a case where acting is very credible. Understanding of other's consciousness, in a functional and third party sense, isn't a substrate for personal experience.
In stark contrast, humans develop consciousness gradually over continuous time with persistent aggregation of experience. By the time we can recognize our own consciousness in the abstract, and reason about it, we have had it for some time.
I use "consciousness" because it's the point of the original argument, but in fact, I think my whole comment still work well if you replace "consciousness" with "understanding".
My point is that the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns that are very effective to fake human sentence continuation but are meaningless in term of understanding the concepts.
And I think that if indeed the only way for AI to reproduce convincingly human sentence continuation would be to end up in a configuration that uses the "understand" mechanism to do so, the behaviour of the first LLM would not show that they are so good at sounding human and yet so bad at failing basic "understanding" tests.
> the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns
Taken as an absolute without any addition context you are right.
But we are not talking about abstractions but specific successful models. The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize. That cannot do it without discovering that patterns that make sense out of it.
And we can verify that. Simply discuss completely disparate topics, with some kind of intersection. Converge several highly unlikely topics, there are so many it would take billions of years to exhaust unlikely combinations.
If the model is only interpolating it will produce gibberish.
But that isn't what happens.
The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue. If they don't understand that, then the question is, why do we think people understand things. Does having an answer mean a human understands something, or is their intuition and stream of conscious reasoning also not understanding? To be even handed about what we mean by understanding.
> That cannot do it without discovering that patterns that make sense out of it.
I don't think it's true at all, and I think we have indication that proves it is false.
We have "basic" LLM, the ones from 2023. They were producing _very convincing_ human text, and yet, they were too often failing basic tests that require understanding.
Now, we have more advanced models, but the counter-example of "basic" LLM demonstrates your assertion is incorrect: these model _did_ produce very convincing human text and yet did not make sense out of it.
But for the more advanced models, the problem is that they are "on top" of basic LLM. So, the first step is a training that build a mechanism that produce convincing text without understanding, and then, the "residuals" are fine-tuned. The result is very unlikely to add "understanding" to the model, because to do so, the whole system needs to deconstruct the basic LLM, to go back towards less efficient situations in order to rebuild almost from scratch. The fact that modern LLM are based on basic LLM means that the first step put the cursor in the bottom of the "basic LLM mechanism" valley, which is a local minimum. And any layer on top of it cannot "climb up" the slope of the valley, pass the ridge and fall into the next valley, even if this next valley has a lower minimum.
> The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize.
That is demonstrably an incorrect logic jump. For example, CNN are able to distinguish between pictures of cats and pictures of dogs. The weights in these models are very small relative to the number of pixels they have been trained on. Yet, they distinguish cats and dogs by finding specific shapes in the pictures, without understanding what a 3-D cat and a 3-D dog is.
They have done that without discovering the typical human pattern that make sense of "cat" and "dog". And yet, the number of weights is very very small with respect to the number of pixel used in training.
> And we can verify that. Simply discuss completely disparate topics, ...
> If the model is only interpolating it will produce gibberish.
What you are saying is that the model is not simplistic interpolation.
But that is a straw man argument: people who say that LLM don't understand don't say LLM are equivalent to simple interpolation machine.
But the problem is that you can have very good predictions in novel situations without understanding.
For example, if you have 10 totally different situations that can be described with a Gaussian curve, and that I show you points for a new situations that cover the left side of a Gaussian curve. Then you will be able to guess that the right side of the curve, which is not an interpolation as it corresponds to situations you never saw, will behave like the rest of the Gaussian curve. And yet, in these 11 situations, I did not even say which real physical phenomenon I'm talking about. You haven't understood anything about these phenomenon, all you have done is guessed that a typical pattern that you have observed somewhere else is more likely to apply here too, without even having to understand anything about the reality of this situation.
And of course, this prediction is "a guess": maybe, for once, in this 11th situation, the curve will start as a Gaussian curve but will suddenly be different. But it happens that the reality is that in this 11th situation, the correct description is a Gaussian curve (because, due to the maths, Gaussian curves are really common). So, when you make your prediction, it looks like you understand the situation, it looks like you understood the physical mechanism that applied here. But it is not the case.
So, no, correctly doing such prediction does not demonstrate understanding.
> The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue.
That is not at all sufficient. A Chinese room experiment will do that despite the system not understanding Chinese. A pocket calculator will be able to be expert in math computation.
> If they don't understand that, then the question is, why do we think people understand things.
That's the wrong question. The correct question is: we know people understand things, and we see AI behaving similarly to people in some aspect, but is this behaviour _requires_ understanding, or can we reproduce this behaviour without needing to understand?
The fact that "basic" LLM were able to reproduce very convincing text that look like they understood X and yet were demonstrably showing lack of understanding of X demonstrates that we cannot just jump to the conclusion that just because it looks the same, the only possibility is that the core mechanism is identical.
Firstly, how do you know that the optimal way to highly compress complex information is to understand it? You think it is obvious because you are very familiar with "understanding" as a way to summarise complex information. But there can be billions of different ways, outside of human imagination, that is as good or even better.
But secondly, LLM don't find the optimal way, they find the local minimum. Everyone who worked with NN knows that they are prone to come up with spurious pattern, incorrect correlations and bad workaround to guess the correct answer. You regularly need to nudge the NN by creating specifically engineered features to avoid them to fall into the first local minimum.
When it comes to LLM, it is extremely complicated to control to see if the LLM has triggered on a misleading pattern that, by chance, links two "tokens" together, or on a real concept that indeed links two "tokens" together. Basic probability implies that there are probably tons of "fake patterns" engraved into the weight during the LLM training, "fake patterns" that should not exist if there was any kind of "understanding" of the abstract mechanism that links these tokens.
I think Searle's Chinese Room argument refutes this. LLMs are simply manipulating symbols, they do not have semantic understanding. This is why hallucinations exist. And Searle's argument extends even further than LLMs.
You are basically arguing for a functional account of consciousness, but things like this have been debated for literally decades/centuries in philosophy.
Millenia, in fact. The big difference, of course, being that we now have experimental philosophy machines (aka computers). So we can actually put some of these theories to the test, and recognize how utterly inadequate most of the work done on the subject has been. We had a pretty good idea anyway, so it's not a big surprise. Theories of mind have evolved dramatically in the late 20th century. And it's pretty clear that theories of mind will have to be re-done all over again with the advent of LLMs (particularly current-generation LLMs).
The problem with the hallucination argument is (1) that is much less of a problem with good current generation AIs, and (2) living conscious breathing human beings also have a disturbing tendency to make shit up, too. So a tendency to make stuff up doesn't really serve as a disqualifier for consciousness.
Also worth mentioning that the guiding rule of what's philosophical or not is whether it's actually useful. Actually useful philosophy usually becomes something else. Usual some scientific discipline or another. And as it turns out, theories of mind are likely to become extremely useful in the near future. Expect huge advances!
1) Good current generation AIs are specifically trained to reduce hallucinations. If we had new AI system that happened to not have hallucinations as a side effect of their training, then it would be convincing. But here, it looks like we have built a pocket calculator that answer 7+13 = 14, and on top of it, we added a layer that says "if the input is 7+13, then replace the output by 20". This pocket calculator still does not know how to calculate, we just added a layer to hide its mistakes.
2) Not only "make shit up" is not the same as "hallucination" (either "making shit it" is done when the individual knows it is unreliable, or when the individual was given wrong inputs), but the point is not to say "hallucination implies no consciousness", but "large quantities of hallucinations in situations where a conscious system would be unlikely to hallucinate implies no consciousness"
I’m also fixated on the term “experience” in the context of this debate. To me, consciousness is something that one “experiences”, and the two concepts are intertwined.
I am far from convinced that the training and inference regimes of LLMs would qualify as “experience” by any sense of the word.
Now, if we hooked up a plethora of audiovisual and tactile sensors with live feedback directly to a neural network rich with transformers, that was always powered on and fully autonomous, we may be getting there. But we’d probably also be on the verge of manmade horrors beyond our comprehension.
Biological rodent neural networks in a Petri dish stimulated by electrical impulses - more or less conscious than LLMs?
Human on life support, unable to respond to any external stimuli, “braindead” - more or less conscious than LLMs?
I point of sorts. Assuming that is true (I don't think it is), the big question that urgently needs to be addressed is what happens when we DO give LLMs tools to interact with the real (or virtual) world. And people are doing that, right now, in both real and virtual worlds. And people ARE giving LLMs the ability to run continuously for long periods of time, sometimes with enormous context buffers. People ARE putting LLMs into robots with front-end ML and LLM systems for visual processing, and back-end ML systems for autonomous control.
And, yes, concerns about whether biological rodent neural networks are or are not conscious come up frequently in the biological neural network papers. I'm not sure I would want to be a researcher trying to get an experiment past an ethics committee if my biological neural network had 25B rat neurons. (I would hope that they could not).
> I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing.
There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake. Evolution learns various solutions to optimization problems, and so if consciousness evolved then it was either useful instrumentally, or it is a byproduct of some organization that is useful instrumentally. The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology. There is nothing that a priori constrains token prediction from the domain of consciousness.
>For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing
World models don't have to be rich and detailed to count as a world model. Lower life forms might be conscious but they only model the part of the world useful for their existence in their ecological niche.
> The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology.
Yes, I agree with that. Consciousness is a good way of generating convincing human text.
What I don't agree with is that consciousness is the only way to generate convincing human text and that because we have convincing human text, it can only imply we have consciousness.
There is a huge probability that generating convincing human text can be done without consciousness. Either because there are efficient mechanisms as efficient as the way the human brain deal with this problem and that the LLM found one of them (and these mechanism may be quite difficult to imagine for a human). Or even because the LLM found a local minimum and is stuck there.
To re-use the evolution approach: evolution solved the "flying problem" with bird feathers, but also with insect wings or bat wings. The fact that evolution ended up using feather does not imply that everything that flies can only fly with feathers.
> World models don't have to be rich and detailed to count as a world model
I agree in general, but here, we are talking about machine that reproduce all human language. The argument I'm answering to is pretending that "all of human knowledge" is understood, which include every single human concept. This has to be everything, because LLM is able to provide convincing text about every subject. If on some subject, the LLM is able to provide convincing text without "understanding" it, then the argument that it is impossible to provide convincing text without understanding it collapse.
> There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake.
> There is nothing that a priori constrains token prediction from the domain of consciousness.
We don’t know either of these are true or false though. We simply don’t know. There is no agreed upon definition of consciousness, aside from maybe _the having of qualia_, so arguing that some can or cannot be conscious a priori can’t be done.
>There is no agreed upon definition of consciousness
No one genuinely engaged with the topic is confused about the target of the term (phenomenal) consciousness. Definitions come once the theoretical work is complete, to be articulated as part of a fully worked out theory. The lack of a definition doesn't prevent us from investigating the subject or offering conjectures. What we can do is offer a precise description of the target and argue for or against whether LLMs reach the description. We will of course debate whether the offered description captures the relevant phenomena. But this is all just part of the process.
I think, for me, the thing is that when you tutor undergrads in abstract math, you discover that students will very often find data pattern that fit the goal but does not correspond to a real mathematical principle.
sometimes humans making claims about AI intelligence or consciousness also identify spurious patterns that do not correspond to the problems of intelligence or hard consciousness.
Of course. But in my explanation "consciousness" or "understanding" is not "finding pattern", it is the pattern itself.
CNN are finding patterns, sometimes relevant, sometimes spurious, but I don't think people argue that CNN have evolved consciousness or understanding of what a cat or a dog is.
Here, the argument is "LLM are able to understand, because 'understanding' is the only pattern to reach the goal". I'm saying that it is unlikely to be the only pattern, and that it is likely that they find a local minimum on a system that reaches the goal that does not use 'understanding'.
The reason I'm saying it is likely is because "basic" LLM shows behaviours where they are producing convincing human text and yet doing things that are really difficult to reconciliate with the fact that they have understanding.
(And before that old argument is used, yes, I know sometimes some humans fail to understand. The problem is that the majority of humans don't fail to understand basic stuff in the majority of the time, while the "basic" LLMs do. The fact that you roll 10 dices 100 times and 1 of them never land on 1 does not convince me that that set of dice is loaded. The fact that you roll 10 dices 100 times and 9 of them never land on 1 does convince me that that set of dice is loaded.)
> students will very often find data pattern that fit the goal but does not correspond to a real mathematical principle.
That reminds me of a niche paper [0] critiquing a certain way of teaching remedial math that was over-focused on tests. A kid named Benny (12) was building up (wrong) "rules" for math which still somehow gave enough of an illusion of progress in terms of test scores that his misunderstandings hadn't been caught earlier.
> Benny was able to explain his procedure; e.g. for 5/10=1.5, he said: "The one stands for 10; the decimal; then there’s 5... shows how many ones." In another example, 400/400 = 8.00 because "The numbers are the same [number of digits]... say like 4000 over 5000. All you do is add them up; put the answer down; then put your decimal in the right place... in front of the [last] three numbers."
But the machines don't understand. They predict. And what they predict is the next token. I'm not trying to beat this horse to death, but you have to realize using the word "understand" is anthropomorphising it. It's essentially the chinese room experiment -- if the rules are followed, no understanding is neccessary.
If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were trained on patterned data that had no meaning to humans or something there wouldn't be any conversation about these things being conscious at all.
Turing complete systems can be built out of matrix multiplications, out of attention, out of key/value lookups. The Chinese room is Turing complete. By claiming it cannot understand things because it is built out of components computing devices can be built out of, we are claiming no computer can because no computer can. This is a very bold claim indeed, and also we’re assuming the conclusion! The claim is no more convincing than “brains cannot understand things because they are made out of neurons”. The system may or may not have some particular properties, but we have to do more work than just gesturing at the components the system is made of when making claims about it; the alternative is, at best, a world where we prove too much and conclude that humans, too, are not conscious.
For starters, we need to pin down the terms under discussion enough that they don’t just mean whatever we need them to in the moment.
There's a growing body of evidence that most of what the brain does is constantly predicting the world around us - look into the predictive brain hypothesis if you're interested.
As Ilya Sutskever has pointed out, if you read a mystery novel up until the reveal of the culprit, and then fill in "The killer was _____", don't you need to understand the novel to accurately predict the next word?
The understanding is inside of the system, in LLMs and in the Chinese Room. I agree with Daniel Dennett that it's preposterous to say that Chinese is not understood in any meaningful sense in the Chinese Room scenario -- it's just that the understanding has been hidden away in the background of the scenario.
Language is tremendously complicated. "Time flies like an arrow, but fruit flies like a banana." "Hard hats must be worn on site; dogs must be carried on escalators", etc. Predicting the next token requires understanding, full stop.
> if the rules are followed, no understanding is neccessary.
Right, it's an illusion of understanding. There is some sort of symbolic understanding, but that is completely due to the fact that the training data was made by humans who actually do understand, can interact with the world, and can write their thoughts down so that the LLM can insert some sort of reference to "basketball" and "Michael Jordan" in their embeddings or whatever.
However it’s disingenuous to say the inference is on the next token because it’s actually not, it’s in the models parameter space across a set of nonlinear activation functions then effectively projected into the token. The idea its predictive of the token isn’t actually the case, it really is a much more complex and more semantic relationship that ends in the series of tokens through the attention mechanism.
The article also makes this assertion that it replays everything over and over again to create each character one at a time as some way to demonstrate the autoregressive self attention mechanism but it’s really not accurate at all, and it trivializes what is going on.
I’m am not asserting LLMs are aware or conscious that’s on the surface profoundly absurd. And I do understand your point that the fact it emits in words something that seems to speak to us gives to the air of humanity that’s isnt real. However there is a very real emergent reality that our language alone appears to lead to embedding a form of thought and understanding that is latent in our use of language in communicating that is in fact coming through the model. It is not regurgitating its corpus and pattern matching because the patterns you input and it emits are not where the inference is operating, its within this enormous vector space through these complex non linear activation functions with learned residuals not in the language corpus.
It is not conscious or aware. It is something else, not human. But if you can not see it as amazing you have lost the capacity to dream.
You are oversimplifying. They do produce one word per cycle. But they can also have context buffers carrying up to two million tokens, which is most definitely larger than your measly human short-term memory context buffers.
You, of course, wouldn't notice if your only experience of LLMs was chatting with the cheapest, smallest, least capable LLMs that you get through ChatGPT, or Google search.
It becomes pretty obvious when you use a coding AI on a daily basis. It is the context buffer in which the magic occurs, not the tokens that get spit out one at a time.
Every day, I watch my coding AI develop plans, search the web a half dozen times for documentation, grep through my entire codebase looking for pieces of related code and context, analyze relevant source code across multiple files, spit out an initial plan for implementing the fix before starting to execute it, run requests through some sort of advanced mathematics tool (they are EXTREMELY good at graduate-level calculus and linear algebra), implement fixes that extend across half a dozen files in 2 different computer languages (typescript and C++), run trial compiles and fix coding errors in its output, sometimes developing sub-plans to deal with compile errors. I've seen it get halfway through a fix and revise its initial plan mid-flight as it encounters something in existing source code.
Not vibe coding, to be clear. Targeted use of a coding tool by a by a professional senior software developer with decades of experience, and fair bit of expertise with the limits of what sort of problems my coding AI can and cannot do. Every line code reviewed. Sometimes it needs additional prompts, telling it how it mis-implemented something, or specifying more carefully what I actually want but didn't properly express in the initial request
All the time maintaining that context across multiple request, so that I don't have to restate requests from scratch.
A particularly interesting revision: "You have misread the equation (13) on page 112 of 'Spice, the Manual 2nd ed.'. I should be ....". (It had previously identified the textbook as a source I was using, from comments in source, in a preceding request, and actually already read cited pages in the PDF file, which it had found online). And I had actually asked it to implement equation (13), which was, in fact, badly typeset. The error it had made was defensible, if not the best reading of the equation.
"You are correct. Let me fix that." (producing updates to the implementation of the equation in code, AND code that implements the symbolically-differentiated version of that equation 60 lines later, which is not explicitly given in the text). The text says "take the lagrangian of equations (11), (12) and (13)" or something like that.
ALL information that gets carried in context buffers, even though it's generating code one word at a time. The bulk of the magic occurs in context buffers, not spitting out words one at a time, which, for my coding AI is, I think about 250,000 tokens.
I think it's pretty safe to think that my coding AI is working out of context buffers that may carry plans and research results consisting of tens or hundreds of thousands of arranged tokens carried in context buffers through the multiple steps of the implementation, and later revision. None of that would be possible if were simply working one token ahead.
I kind of suspect that a lot of activity occurs in the first few words of its response. "Let me examine your current source code and develop a plan. Ok. I can see on line 131 where you want me to implement the equation.". (An opportunity to perform about 27 updates of the context buffer). And in the sometimes hundreds of lines of output it generates as it talks itself through what it needs to do.
I would maybe agree with you if the entire realm of human existence was limited to words. There are many human experiences that transcend text, and indeed can hardly be adequately described using text.
Sure, it's the best we have online, but that does not make "the internet" the sum of all human experience. To reduce all of humanity down to the text on the internet is reducing us to the level of machines to fit the requirement of what a machine can process / simulate.
I don't think they're asserting that all of human existence can be subsumed as text, though? Just that "consciousness", or "understanding", in some meaningful sense could be exhibited by a system that can only interact with the world through text?
> If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.
I think the main complaint is LLMs don’t arrive at the answer the way we do. It’s capable of emulating some of our behavior but not all as the mechanism by which it works is very different.
Maybe I’m wrong about this but one thing humans do that LLMs don’t is deductive reasoning. LLMs seem to operate entirely of inductive reasoning.
I will say, I find it fascinating that there are some philosophers and consciousness researchers who seem to be less certain. I just listened to Chris Hayes interview David Chalmers this week, whose position seemed to be that it's probably not conscious, but that we can't be certain. And more than that: he seemed open to the idea that they may become conscious under further scaling/training/advancements.
Funny enough, the models seemingly go insane and decohere into noise output in the absence of sensory input, which is remarkably similar to what would happen to a human.
That said, I'm not sure I follow what you're actually asking here? I'll also note that I'm not taking a position one way or the other, just sharing a podcast and noting that an extremely reputable scholar on the subject of consciousness seems to have a bit more uncertainty and humility than many commenting here. ;)
LLMs just wait for a prompt, so they do nothing and are just frozen in place.
I'll find time to listen to your link, it sounds interesting. My objection is the strange idea that humans are automatons that are keyed off input like a clockwork machine and operate sequentially. This is clearly not the case.
>LLMs just wait for a prompt, so they do nothing and are just frozen in place.
I'm not sure that's a compelling argument. Humans can be put into a similar state where they are unconscious and not thinking. Think of someone in a coma, for example, where we actually measure and confirm that there is no brain activity where they're in that state.
They are not actively conscious, but that doesn't nullify their consciousness from when they were awake, right?
>My objection is the strange idea that humans are automatons that are keyed off input like a clockwork machine and operate sequentially. This is clearly not the case.
Well, a few thoughts here. First, it's worth noting that the argument isn't necessarily that AI are conscious in the way that humans are, nor that humans are strictly automatons.
But I think the more interesting thing is that our understanding about consciousness has evolved quite a bit in just the last fifty to one hundred years. We used to think that only humans were conscious, but assumed that primates, cows, dogs, and other mammals were just automatons. Then we started to think: okay, maybe primates are conscious. Then eventually: well, dogs also seem to have consciousness, and then rodents, etc.
This has continued such that most people in the study of consciousness think all mammals are conscious, and the debate is shifting down to insects and other creatures that we do think/have thought of more as automatons. We don't actually know where to draw the line, because it's essentially impossible to really feel/know the inner states of other living beings.
In the face of all this uncertainty, Chalmers just points out that since we understand consciousness so little, that ultimately we should probably be less definitive in pronouncing which things do or do not have it.
"If a machine has to learn to understand humans to complete text, then that is what it has to do."
But the machine doesn't have to understand humans to do that. It gets trained on a whole bunch of sentences and then it is able to complete text. You could maybe claim that it "understands" the text but even that's a stretch.
Your “and then” is doing a lot of work there. The steps between may or may not include some form of “learn to understand humans”, but you can’t just hide them behind “and then” if what we are doing is claiming some particular thing is not in the list.
Through training on human text, we are building implicitly in the weights a statistical model of what humans might write in response when presented with arbitrary pieces of text. It turns out that we can make these incredibly accurate.
If building an accurate internal model of something then using it to predict that thing’s behaviour is different to gaining understanding of that thing, we will need to pin down exactly what “understanding” means, or we are forever doomed to talk at cross purposes.
I wish people would do even the most basic amount of research into LLMs before opining about what they can or cannot do. There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning.
Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'. It has trouble distinguishing features that are "sub-token" like specific letter sequences because it doesn't see letter sequences but just the token as a single atomic unit. 'Straw' and 'r' are two tokens but an LLM is entirely blind to the fact that 'straw' has one 'r' in it.
As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.
The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence. Thus deficiency in some capacity that we take for granted in humans is an indictment on LLM intelligence. But this is specious. LLMs are entirely alien; their developmental paths do not and should not look anything like ours. Your intuition from human intelligence just works against understanding the potential for intelligence in LLMs.
>The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence.
To be fair, almost everyone who claims LLMs are conscious tends to claim that they are conscious in exactly the way that humans are, to the point of stating that human brains are also just complex next-token prediction machines with a random seed. It's basically religious arguments on both sides.
I have seen people say "you're a next token prediction machine" but only in a similar way one might say "you're a cup of old lard". Not actually meaning it literally.
I have seen people interpret the request to show that they are not next token prediction machines to be a claim that they are, but this is almost always an argument to show certainty is difficult in this area.
People like Hinton have declared that they believe them to be conscious, but clealy indicate that they do not mean just like us.
Eh, I’ve seen it. I’m not entirely sure it’s entirely wrong either. Humans are certainly more than just next token predictors but it’s not clear that our typical language behavior is significantly different. We call it “stream of consciousness” when we just spew words out without thinking and that seems to be the default operating mode.
Given the fact that large language models are trained on human language, it shouldn't be surprising that the text they output resembles human language. That is what they're designed to do after all. But similarity in output doesn't necessarily map to similarity in process.
And it seem obvious to me that language behavior does differ significantly between humans and LLMs based on the frequency and nature of failure states. LLMs routinely hallucinate, or get "AI strokes" or get obsessed about not talking about goblins, etc. This isn't typical language behavior for humans unless they have severe neurological or psychological impairment.
People tend not to "spew words out without thinking" and certainly not all the time by default - we call that glossolalia and (outside of some fringe Christian sects) it's considered a "bug" not a "feature" of the human brain. Human language by default always has intent behind it, even if that intent isn't readily apparent to the speaker. People can recite by rote memory, but that isn't blind token prediction, it's the neurological equivalent of muscle memory. People can have conversations then forget about them because their attention was focused elsewhere, but that doesn't indicate that they were simply "spewing words out without thinking" at the time.
> LLMs routinely hallucinate, or get "AI strokes" or get obsessed about not talking about goblins, etc. This isn't typical language behavior for humans unless they have severe neurological or psychological impairment.
People imagine details all the time. Eyewitness testimony is notoriously untrustworthy.
Our brains seem wired to confidently fill in gaps. We all have a literal blind spot we aren’t aware of because our brains convincingly lie to us and fill in the gap.
I don’t know what an “AI stroke” is, but I’ve definitely seen human beings in good health be in the middle of talking and suddenly forget what they are going to say.
> People tend not to "spew words out without thinking" and certainly not all the time by default - we call that glossolalia and (outside of some fringe Christian sects) it's considered a "bug" not a "feature" of the human brain.
Glossolalia is spouting gibberish, not comprehensible speech.
Kind of weird that you speak so confidently when you don’t apparently know the difference between steam of consciousness and “speaking in tongues”. Almost like you’re AI hallucinating.
> There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning. … Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'.
This sounds like a description of a child who has not learned to read yet. You ask a child who is not aware of the alphabet and of "words" how many r's are in strawberry you'd get a non-sense answer too. So what you're really pointing out is that the LLMs have not been trained on "the english language" and how words are constructed and what they are composed of. That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word. It's not that I know how many r's are in strawberry because of how I'm understanding the word "strawberry", I know how many r's are in strawberry because I know how to spell strawberry. The LLM needs to be trained on this the same way someone who is learning to read would be trained on it. No one should be surprised that an LLM can't "read" in the same way no one should be surprised that a child can't "read".
>That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word.
This interpretation takes things too far away from how LLMs are constituted and so misses important explanatory power. The issue of counting letters in a word isn't about an ability to spell, it's about the nature of one's perception. We perceive words as sequences of individual letters. LLMs do not. I can ask you to tell me how many r's are in some nonsense word sequence and you're fully capable of doing that. LLMs do not see sequences of letters so they are intrinsically at a disadvantage for this kind of question. But this says nothing about its capacity for intelligence anymore than not naturally being able to distinguish frequencies of photons hitting your retina has anything to say about human intelligence.
This is kind of a like assuming someone with bad spelling is stupid.
Counting letters in a word seems to have little to do with understanding the word. Young kids can’t spell or count well at all but no one says that means they can’t understand.
This is like saying because humans can't multiply 23472 by 1836736 in less than 5 nanoseconds that they can't possibly understand anything about maths.
You can't natively understand how many of your photoreceptors cells are activated by the period at the end of this sentence. How could you possibly understand the sentence's meaning?
>To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ...
This is where the other claim is being made. That the structure of the model is fundamentally incapable of the operation, so even if you stipulated that the way you provide data is sufficient for intelligence then it still wouldn't work.
The universal approximation theorem addresses this point. In that, with an identity attention mechanism, a LLM is just a multi layer perceptron. The attention mechanism is effectively a way to get one of the benefits of a much larger fully connected layer without the massive cost.
A LLM can do what a MLP can do. A large enough MLP can do any function to arbitrary precision.
That makes the claim that an LLM could not do a task the same as saying no function can do that task.
Some are ok with this, if you invoke some supernatual aspect to intelligence then the inability to describe it with a function is quite reasonable,
If you want to stay in the world of reality, you have a much harder task, people like to point at quantum (Penrose) but it's hard to say what it is you are pointing at.
I think the very act of proving that something is or is not intelligent, would render it functional by nature of it having a proof, (or disprove Gödel's incompleteness (a tough ask))
Are there any proofs that cannot be expressed as a function? A kind of Gödel locator, where you can prove something that you can identify is true but there is no formula to express it. I'm not entirely sure what that would even mean,
>If a machine has to learn to understand humans to complete text, then that is what it has to do.
A language model completes text based on the overlapping patterns of the training data.
There absolutely was thinking involved… in the training data. Same as when you read a book, you engage with the thinking behind the text. The book isn’t thinking, and the author may be dead and gone, but there’s absolutely the traces of thinking in the text.
Language models produce mashups of texts they were trained on, and there’s absolutely the traces of thoughts behind those mashups.
Come on, I invented parts of this technology at Google and am baffled why this is debated.
We discovered math that decodes data storage in langauge and is able to use sophisticated continuation cohorts from ALL OF HUMAN RECORDED KNOWLEDGE to respond to you in a call/response model with very good synthesis capabilities.
Its super useful, but not life or conciousness. Its a simulated echo from our collective recorded behaviors. It understands because we understood first. It replies because we wrote it first. And it sorts, organizes, synthesizes and compresses that at impressive speed now.
I have no technical expertise re. LLM’s but from my intuition I came to this same conclusion.
It’s strange many others have not eh? I think when new developments arise, ironically, this is the true measure of human intelligence - one’s ability to make sense of a thing and be closest to the truth.
His intention is irrelevant, as is "trying to highlight a fact" as if it were the final say: all Chiang is doing here is using fancy white-collar words to argue the same argument leveled against Hinton and others regarding next-token prediction. And his audience, who have even less technical understanding, lap it all up unawares. Chiang is a writer and needs to stay his own lane, not RP as an expert; or, if he wants to do journalism on this topic then he should actually do the work and talk to more actual experts not just the ones cherrypicked for his opinion piece.
Chiang has, in fact, written on this topic before - see "The Lifecycle of Software Objects", and has speculated about sentience in AI, etc. This is not a "one-off", "I need money" type of article. I dare say he has thought about this much more than most people here.
From Wikipedia: In 2023, Chiang was named one of Time's 100 most influential people in AI.
That's the problem, he's a writer. He's not a research scientist like Hinton. If a writer uses his skills and stature to rehash a well-known argument about next-token prediction, then it is performative of his status and influence and doesn't contribute to shedding actual light on the debate/confusion.
Indeed it isn't a one-off. His last infamous article compared AIs to Xerox machine image compression. He convinces a certain type of crowd that is not technical enough to poke holes in his posturing.
The fact that a LLM is essentially immutable would be my biggest argument against consciousness or self-awareness.
It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens that is a statistically likely response to that prompt, then it stops. It's not changed by the experience. It doesn't remember anything. It doesn't sit around thinking on its own.
Even if the model itself were extremely complex, it's hard to imagine a definition of consciousness that includes something that doesn't remember and can't change.
this just means they are incomplete, like a baby that has no long term memory. I think the baby analogy will hold up as we build more and more capability.
There is, you talk to an Alzheimer patient and its like that, and it doesn't feel like talking to a human any more. An Alzheimer patient isn't cured by adding some input noise to stop them from repeating conversations, they are still unable to learn, just like an LLM.
There are people whose brains don’t form new memories anymore after an accident or surgery, and they eternally live in the time before it happened, and have no memory of what happened a minute ago. Still they are conscious.
I think it's a little more complicated than that. In a 50 First Dates type of scenario, their ability to form certain types of memories is damaged, not non-existent. And I would argue that with enough brain damage someone like an extreme lobotomy victim may stop being considered conscious.
I’m not familiar with 50 First Dates, I was thinking of cases like Clive Wearing [0]. I would agree that consciousness requires some sort of ultra-short-term working memory, but I also think that mechanisms similar to CoT loops can conceivably fulfill that role. Today’s generative AIs consist of more than just the static network-of-weights model.
"Wearing can learn new procedures and even a few facts, not from episodic memory or encoding, but by acquiring new procedural memories through repetition. For example, having watched a certain video recording multiple times on successive days, he never had any memory of ever seeing the video or knowing the content, but he was able to anticipate certain parts of the content without remembering how he learned them."
Honestly, that's a pretty messy state of consciousness and I wouldn't proudly crow that my AI is conscious if that's as good as it got
I'd argue that the context window is analogous to short-term memory. It's functional but limited in duration, and if you overload it, it starts to fail.
It's the long-term memory (i.e. learned experiences feeding back and directly altering the content of the core brain, or model) that is missing.
The context window is so flawed that I wouldn't consider it memory.
It feels like notes about the situation rather than it being in memory. Memory has more "attention". I think that "it starts to fail" is load bearing here.
I feel like memory has like 5 parts, and LLMs are missing 2 of them:
current working memory
short term what is immediately happening without it being in "RAM". I differentiate here vs working in like thinking fast and slow. Keeping things in working memory is work! You can vibe away short term memory. I had excellent short term memory while I was messed up, I could keep time well. I think LLMs can do this with notes.
mid term: Vague awareness of things like what day a week it is or what you did 2 hours ago. This is where my memory personally failed
long term memory of experiences. You can fake this with memory.md
generalized wisdom for pattern matching long term memories
LLMs seem to be missing that part I was missing. Im probably projecting and anthropomorphizing. But i relate: I would confabulate a ton and didn't know anything was wrong for a while but things seemed off.
Context is like working memory but not short term or mid term. I think you can imply short term with big enough context.
My categories are purely anthropomorphic to me but just wanted to say where I disagreed.
They are conscious because even for short periods of time they do form memories and those change them even if only briefly. They think on their own too. It is a very limited level of consciousness though.
That could be easily fixed by providing the AI with a constant stream of input.
For humans, part of the input of the human mind comes from the continuous processes and clocks within the human body, so it’s questionable whether the brain could “think on its own” without such input either.
The continuous input for the human arises naturally, it doesn't arise naturally for an LLM unless we direct it so. Our consciousness is bootstrapped, the LLM isn't.
We have virtually no idea how consciousness arises in the human brain. Furthermore, what is “natural” supposed mean here, and why should it matter for consciousness whether some prerequisite arises naturally or not?
- A motor is something that create a force to push a vehicle.
- Oh yeah? My neighbour car does not have wheels and sit on concrete blocks, the vehicle does not move and yet we all agree it has a motor. So it means that I can claim that this other thing that does not move has a motor too.
Sure, human can _some times_ not do some stuffs, but the fact that they can do these stuffs sometimes is the point.
Doing these stuffs is the hard thing. Doing these stuffs is the proof that the machine has what it takes. It does not matter if someone cannot do that stuff, it does not imply that their internal system is not complex enough to potentially do it. But the fact that some people can do that stuff is the demonstration that inside a human skull, there is a system that is complex enough to potentially do it. Unless you can prove that people who don't do it have a fundamentally different system inside their skull, then you cannot pretend that they should be considered as having a less complex system.
Human _can_ check themselves. They don't _always_ check themselves.
Motor _can_ move vehicle. They don't _always_ move vehicle.
LLM _cannot_ check themselves. They _never_ can. It is not that some don't, they just cannot, they are not a system complex enough to do so.
So, yes, it is a refutation. If you have something that _never_ can move a vehicle, this thing does not qualify as a motor, even if some motor, sometimes, don't move a vehicle.
And if your next argument is "yeah but I would argue you don't need to check yourself to be conscious or to understand things", then you just redefine the definition that is owned by your interlocutor. Your interlocutor is saying that this is a criteria they are expecting. Good for you if you are not expecting this criteria. But the problem is that the answer is not "this criteria is not expected", the answer is "I change the criteria from 'being capable to in some circumstances' into 'does always do it in any circumstances'".
True, but I don’t see how that relates to consciousness. An LLM being continuously RLHF-trained also changes its habits; that alone doesn’t make it conscious.
The starting file may be immutable, but the whole processing of that file is very dynamic and intense. Maybe, if there is some consciousness, it lies somewhere during that processing.
An LLM’s training could be seen as lived experience, and the fact that LLMs can output long sequences from their training material can be interpreted as them remembering those parts.
Also, how does that relate to consciousness? I don’t think that past episodic memory is necessary for consciousness.
> The model definitely remembers previous exchanges within the same conversation.
No it doesn't. They get added to its context, and it reads them afresh when answering the next question. That's not remembering.
If your short-term memory completely malfunctioned one day, so you had no ability to remember what was said to you a minute ago, then you would have to find workarounds. For example, you could write down everything someone says to you, then read your notes of the previous exchanges in that conversation in order to continue the conversation. That would be a good way to work around the fact that your short-term memory was broken. And if your notes were invisible to other people and you could read them really fast, then you could even make most people believe that you remembered what they said a minute ago. But you don't actually have a working memory, you're just writing down what they said and re-reading it while coming up with your next response.
Continuous learning allows past behavior and past inputs to influence future inputs and future behavior. In humans.
Attention over KV cache allows past behavior and past inputs to influence future inputs and future behavior. In LLMs.
Until the cache runs out, that is. But even then, you could totally use any of 9000 methods of cache compression, truncation, dropping or streaming and get away with it.
The difference between continuous learning and in-context learning seems to be in capacity, not in principle. Both are doing a similar thing, but one has more length and depth to it.
This is really semantics, but I wouldn't call attending to the KV cache re-reading the context.
The model takes in the context, encodes it into a "memory" (the KV cache), and accesses that memory later. That fact doesn't change just because the KV cache grows in size with the context.
I don't know what memory would look like other than an encode-retrieve loop.
Not the model though. The model really only takes input text and produces output text. Memory within a conversation is achieved by the harness adding the conversation (or parts of it) to the input text. The LLM itself has no memory, it’s the augmented system of several orchestrated LLM calls that does.
Right, but that's still external to the LLM, it's just a KV cache that's stored on the provider side for performance reasons, so that the client doesn't have to re-send the whole chat history with every subsequent call in the conversation.
It still generates every response using the model's pristine state with every new API call; whether the context is provided from the client or from a colocated cache server doesn't really change that.
Reinforcement learning changes the model. So it can and does change and remember based on experience. Eventually reinforcement learning can happen in real time.
But is the model aware of the training? Unless you hook the model up to an MCP server, or something similar, and have it analyze the RL changes, it will not know if it has changed or not. Even if it is real-time RL, it is not aware of the previous state.
I don't understand the most of the responses here. What does computers getting more intelligent has to do with it getting conscious? Even if AI with current computer architecture gets much more intelligent than humans, it will still have the same amount of consciousness. Its still bunch of instructions. If we got a lot of people together and did all the instructions by hand to get an LLM result would we also create this consciousness with those other people? All these makes no sense to me.
Quoting the famous Science paper More Is Different [1]:
Knowing the laws of physics doesn't tell you how a brain, economy, or society works - because at every level of scale, genuinely new phenomena emerge that require their own science.
No, it's not a bunch of instructions, it's a colossal array of vectors that are the outcome of many thousands of lifetimes' worth of stimuli and reinforcement - not dissimilar in (very) abstract terms to the neurons in our brains.
Regardless, the question still stands: "What does computers getting more intelligent has to do with it getting conscious?"
Just because consciousness emerged for we humans and other animals through one mechanism doesn't mean consciousness has/will/can emerge from current LLM technology.
For this extraordinary claim, I think the burden is firmly on those who are arguing that it has/will/can.
The other point of view is that the burden is on those who suggest that biological consciousness is somehow special. What makes it so, and if the answer isn’t metaphysical, what’s stopping us from constructing it artificially?
The key point is there is plenty of proof that consciousness is emergent, not structurally baked in. An excellent counterpoint is the "They're Made Out of Weights" post. There are plenty of emergent systems that clearly show intent and what we consider intelligence, like how ant colonies act as a unit.
If you want to get evolutionarily technical, humans are made of cells which began as individual organisms and coalesced into higher life forms. So the concept of a "life form" is very much flexible, so is every capability of a life form, including consciousness.
It might reflect a prejudice of modern human society to think more highly of people who are intelligent, and think less of people who we deem stupid.
I'd say AI should not by default gain any social status as human-equivalent even if they become (in whatever regard) more intelligent than humans. But that would require us to drop the notion that intelligence ~= respectability/status/ability to have a full subjective experience.
Most of these kind of pseudo-philosophical controversies actually tell more about the issues with humanity than the new tech/stuff...
Since when has intelligence had a strong relationship to status and respectability? I've met intelligent people that don't get much respect or status either because they don't look good, are shy or they don't have money.
It has been very well discussed before I don't understand how anyone sane can argue against it (and indeed you can see on Wikipedia that the replies to it are sparse and don't make any sense).
Well I don't understand how one can accept this argument. I mean if you believe in mind-body dualism it can make sense. But AFAIK Searle doesn't, instead he holds that there's something special about the brain biology that enables consciousness and that you won't find in a computer. I don't see why that would be the case if the computer can simulate the real world, and I find Searle's argument against simulation, that simulating rain doesn't make you wet, falls flat: it can make things wet in simulation, and if you connect it to sprinklers it will make you wet.
I mean you can connect the computer to sprinklers that activate when the system detects rain in the simulation if that's what you're after (that was just an aside to note that of course you don't get wet from a simulation disconnected from the real world).
But I guess that was a distraction from the main point: If consciousness emerges from biological processes in the brain connected to the world with a body, why would it not emerge from a simulation of these processes connected to the real world with sensors and actuators?
It seems like circular reasoning to me: The simulation is not like the real thing because it lacks the special thing that enables consciousness (that's Searle's biological naturalism). And it lacks what enables consciousness because it's not the real thing (that's the weather analogy).
It's fairly obvious to me that the room as an entire system understands Chinese. The human in his thought experiment would just represent the small fraction of the brain that encodes/decodes symbols.
Agreed, he’s saying if he’s working with a black box he doesn’t understand what it’s doing. Which is of course true. If the black box were a chinese person instead of a tool and he was writing down what they said, would he say the chinese person in the black box does not understand why they told him to write something? No, of course not.
Today we know the program to do this is always going to be inscrutable to Searle. His role is no different from someone using Claude to write an email.
If I did all of these calculations by hand, would it be conscious...
That's a powerful argument I haven't seen stated quite that way before..
I do think it's hard to know when consciousness exists, because we can't really prove it for our neighbor. We just intuitively know that it would be crazy, even immoral, to assume otherwise.
But, It's likely easier to dismiss consciousness, once we understand the mechanism, than it is to prove it.
It's was a genuine question though. I'm not saying LLM use neurons, are like neurons, or are meant to be like neurons. I'm saying if you can do the math, for a substrate, that doesn't mean much. The simple math that runs it isn't the secret sauce.
This sort of tech-centric distillation of visceral human experience into simple analogies is so grating. We can simultaneously acknowledge that consciousness is not well defined and not "testable" with certainty while also acknowledging that there is something different between the conscious experience we are all aware of as humans, and instructions executing on a chip. The only thing that has changed in the discourse about AI and consciousness vs. "is my home desktop conscious in 2004" is the quality of the simulacrum that LLMs produce vs. pre-ML chatbots.
Unless you ascribe to some meta-physical soul - you believe human consciousness is encoded in matter - in the interactions between atoms. Actually at a much higher level of abstraction- neurons - but it’s all simulatable in principle. Thus, yes, it could literally be a bunch of x86 instructions.
1. that artificial neurons are very restrictive model for actual neurons, let alone brain/organism, as there is more going on in neurons than a thresholded 0-1 activation
2. a brain does not function in isolation, but as regulator of the bodily functions (along with even more things). If you look at the brain in isolation to the body you don't really understand fully what it is doing unless you narrow your view a lot. Eg the brain modulates production of hormones, which in turn affects stuff like heart rate which then comes back to the brain as signal, in a feedback loop. Not to mention actual behaviour and interaction with the world. Toy models of organisms are not organisms.
3. "interaction between atoms" (or rather matter in general, as we have to take into account electrons, photons, gravity and a lot of other things that matter) is too general, too big, artificial neurons are a very useful (for applications) abstraction inspired by biological processes, but not modelling said underlying biology fully. Nobody can imo right now know if "computation" is a good model for "atom interactions" as in whether we can adequately enough model "atom interactions" in a computationally tractable way, and surely we do not do that right now except in very narrow scopes, and we have good reasons to believe that the current computational paradigms/turing machines are inadequate in doing that efficiently enough.
I'm going to ask a couple dumb but genuine questions:
Does it matter that neurons are more complex than 0-1? Does the fact that transformer layers don't use purely thresholded 0-1 activation invalidate what you're saying?
How do you know that artificial neurons are less capable of producing consciousness than biological ones? How can other people independently verify this?
How does the embodied nature of human consciousness preclude consciousness emerging from a computational system? What is the definition of consciousness if it is precluded from occurring in a computational system but present in biological systems?
Why do you think exactly modeling interaction between atoms matters for consciousness? And where is the fidelity threshold? Is it the planck length?
Finally, a dumb question: how do we know humans are actually conscious, and where is the threshold between consciousness and unconsciousness? And do these criteria exclude all other forms, or other animals?
Not the person you were asking but IMHO it all reduces to computational complexity, e.g. biological evolution provided the computational efficiencies that ultimately produced conscious minds and beings, whereas it is not obvious what scale of silicon, power or energy, and input data is sufficient for that to happen artificially. But that means my view is it is a matter of it being possible in principle, merely unknown in practice. Also my view is that denying this amounts to violating the Church Turing thesis of computational equivalence ("human brains are not magic, super-Turing, etc."), and I think a lot of talking-past one another in these public disagreements amounts to one side not actually having taken modern CS theory fundamentals enough to be persuaded of these couple of premises.
That's my take on it too, roughly. I think if we get to trillion-parameter models and they don't exhibit what we'd call AGI, however you define it, then the current transformer based systems never will.
But calling them "unconscious" is a pretty high bar. Mice are conscious. The house sparrow pecking in my yard right now is conscious.
>How do you know that artificial neurons are less capable of producing consciousness than biological ones? How can other people independently verify this?
How do you know that toasters or rocks aren't conscious?
The only thing you can do is ask rhetorical questions to make your position seem obvious. Which should tell you how little you understand the thing in question.
My point is it is very difficult to disprove that something has consciousness. What would be sufficient evidence that a rock is not conscious? The onus is on those making extraordinary claims (that a computer program is conscious) to provide evidence for it.
>The onus is on those making extraordinary claims (that a computer program is conscious) to provide evidence for it.
By saying that a computer program is not conscious you are also making an extraordinary claim. You would have to hold an agnostic position until there is a test for consciousness.
You are relying on intuitive obviousness and rhetoric to make the opposing side look ridiculous "how could a TOASTER be conscious, preposterous!", you aren't making a actual positive argument for your view.
I would agree that a rock is not sentient, but determining whether or not it is conscious would require a definition of consciousness and a machine that could test for it.
LLMs are able to do almost everything that we consider human intelligence, and in many areas of intelligence, they have surpassed us. It's not at all extraordinary to assume they're conscious.
LLMs know more than any human being, are simultaneously experts in nearly every field of science and humanities, are able to make novel mathematical discoveries, can write and understand every major written language, and can give you an intelligent answer to almost any question you pose to them.
How is that not human-level intelligence? If a human could do all of that, we would consider them a genius.
The problem with this is that the word 'hot' only has meaning to a conscious being. And while we don't know what conciousness is, it's extremely hard to argue it's not an emergent property of physics. So if your supernova simulation is complex enough to also model emergent properties like conciousness, the simulated conciousness may well regard the supernova as 'hot'.
Funny you mention simulations. The simulation theory that we all live in simulations doesn’t have any practical consequences for me or any one.
If our world behaves exactly the same, what difference does it make?
The perspective. From within the simulation there is no point in making the distinction, but from the outside it does. For another example - running a program on a virtual machine or a physical computer is the same to the program, but very different to debug when you see hardware errors.
A very large number of people could do the calculations of an LLM by hand using pen and paper. It would take a long time, but if the result were conscious then were would the consciousness exist? In the humans? Is it consciousness within consciousness?
Well if you go that route, a computer simulating digestion has almost no physical features in common with actual digestion of a stomach. The same holds for consciousness and brains and computers. Them saying it’s just instructions is shorthand for pointing out the physical differences of brains and computers.
It’s all just particles, but the higher level differences are vast, and only brains are implicated for first person perspectives via science.
>So what context would cause me to seriously consider the possibility that engineers had created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees. At that point I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs.
I agree with some parts of this piece, but paragraphs like this one above seem pretty uninspired and simplistic. It's entirely plausible that a conscious mind would not be evolutionarily incentivized to be able to do those things. ie just because animals on earth needed to develop specific talents doesn't mean that other conscious entities need to. Why would a computer program need to hunt for food like a mouse would? Making tools like chimp? these seem like nonsensical metrics.
Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.
Is a car a body? Does an AI situated in a car therefore get to have desires and emotions? Is a taupe box with a webcam attached a body? (For that matter: Is a quadropelegic body a body? Do quadropelegics have desires and emotions? Obviously, yes and yes.) Why is a body necessary for the formation of desires and emotions? Why are desires and emotions necessary features for consciousness?
Or here's one: If I'm not experiencing any particular emotion in a given moment, am I concious?
I tend to think that emotions, at least, are mainly hormonal global triggers: they're more about physiology than actual consciousness. The whole thing, as a result, sounds like an effort to privilege biological intelligence, rather than a real foray into the issues.
> Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.
How can you have a subjective experience without a body?
That's the point, in my opinion: your physical/chemical state (body) in a given moment is then translated into the higher abstraction of the emotion. An emotion that *you* feel, because you are self aware of what's happening.
How can you be self aware without feeling? And how do you feel, without a body?
> How can you have a subjective experience without a body?
Easy, you don't have subjective experiences because you have body in the first place. You have them because some signals come in from your nerves, which your brain turns into a world model. You are effectively a "brain in a vat", the vat just happens to be placed on top of your body.
An AI system constructs the world model a little different, by all the text that gets feed into it, but that doesn't mean that there is anything fundamentally different in the world model it builds. Consciousness operates on world model, not on the world or even the body itself.
The AI's world model might be missing some information, because they weren't described in enough detail in text, but that shouldn't matter for consciousness. A blind or deaf person isn't less conscious than one that can see or hear just because some information is missing from their world model.
The "you" isn't your physical body, it's the pattern recognition in your brain that that classifies some parts of its sensory input as "you" (see Mirror Hand illusion). That it happens to be running on meat instead of silicon is an implementation detail and not important.
If we want to know if AI is conscious or not, we have to ask if the AI can recognize itself in the input it gets.
Some aspects like limited content length and lack of ability for the model weights to update will certainly limit what the AI can do. But that's ultimately a matter of degree, not kind, when it comes to consciousness.
> The "you" isn't your physical body, it's the pattern recognition in your brain that that classifies some parts of its sensory input as "you" (see Mirror Hand illusion). That it happens to be running on meat instead of silicon is an implementation detail and not important.
It is important, if I destroy your brain and grow a new one and start running the same program on the new brain your consciousness is still dead and its a new person living.
Computer AI models don't run on a single machine, they run in a distributed manner using different machines at different times. When you ask a follow up question thats not sent to the same machine, its another machine answering. So the consciousness one of those computing machines experience would be extremely fragmented and not at all conscious about any discussion with you, since it only saw a few fragments of it.
And no consciousness doesn't expand to cover larger distributed computations, otherwise social media would have developed a consciousness by now but it hasn't. Groups of humans don't start to share consciousness, it doesn't happen, so you can assume groups of distributed computers wont as well.
Its amazing that after years of advocating for a materialist view of the mind, the tech bros are flipping to mind-body dualism now that they need to believe a concious mind can exist no body at all.
It's worse than that; it's mind-body dualism when arguing that an AI can be conscious, but still materialist when arguing that humans are simply more complex neural nets. It's not a coherent viewpoint.
> If I ask my friend what his favourite song is 100 times I will get the same answer back 100 times.
Pretty sure you wont, humans vary the exact wording. They will say the same song but they wont answer the exact same way every time. Even if they say the same words two times they wont use the same tone and body language, as they don't just communicate via words and that nonverbal language is a part of what we say.
If you ask him 100 times in a row I would bet that by the second, maybe third, time he will not answer with his song but with: what the fuck is wrong with you?
Nicely circling back to LLMs not being able to learn and form memories.
> If a hundred people see the same event, will they all respond the same?
If you have hundred different people, they will of course do something different. Just like hundred different AI model will do something different. The question you have to ask is if the same person under the same circumstances would do the same.
Luckily, we have an answer to that: They would. Transient Global Amnesia is a condition where people temporarily lose the ability to form memories and in turn they keep repeating the same conversation again and again[1]. Their brain keeps asking the same question again and again, as it doesn't remember the answers it already got.
> Easy, you don't have subjective experiences because you have body in the first place. You have them because some signals come in from your nerves, which your brain turns into a world model.
Bro, I think we discarded this idea from Platone and Cartesio a while ago...
Your brain is your body.
Your mind is not detached from it, and you can't feel anything, and so have a subjective experience, without it.
Neither, your mind, or "soul" could survive to the physical death of your body.
The point is that the brain operates on information, not things. The thing you think is the external world, that's just electricity flowing through your neurons. When you see a dog, that isn't a thing in the world, but a signal in your brain. In principle we can take a knife and cut that connection at any point and replace the real signal with an electronic box that gives the same signal.
You don't need a body, you need electrical signals your brain interprets as body. And in principle you don't even need a brain, you could replace that with some matrix multiplication or transistors that do the same stuff.
The important part of consciousness is being able to figure out what of the sensory input is correlated to your own action and which was caused by the rest of the world.
Its a computer program. It is literally just a lot of zeroes and ones, sitting there doing nothing.
Then a request comes in, and the system does a bunch of calculations using those bits, and spits out a result. The bits are unchanged.
When your brain receives input, it is changed. It is constantly active. If it ever stops being active it's dead.
So, what exactly is the claim? Are the bits constantly conscious? Do they snap into consciousness when the computer does math with them? Or is it maybe the computer that's conscious while it's processing these bits? How about when it stops doing that and goes back to doing other stuff? Why are these particular bits special? Was the computer always conscious?
I feel like the only way anyone could believe LLMs are conscious is if they don't understand how computers work. Of course it isn't conscious, how could it possibly be conscious? Its literally just bits. It's like saying the text in a book is conscious.
So, what exactly is the claim? Are the bits constantly conscious? Do they snap into consciousness when the computer does math with them? Or is it maybe the computer that's conscious while it's processing these bits? How about when it stops doing that and goes back to doing other stuff? Why are these particular bits special? Was the computer always conscious?
These are all fine questions, and they don't become any easier to answer if you replace "computers" with "brains" and "bits" with "neurons".
Of course it isn't conscious, how could it possibly be conscious? Its literally just bits.
"That's ridiculous. How can meat make a machine? You're asking me to believe in sentient meat."
What is even being argued here - neuroscience is hard, so programming your PC thus makes it conscious?
We don't understand how combining a bunch of obviously(?) non-conscious biological components can produce a larger system that is conscious, so it's unwarranted to be certain that that can't happen with software.
Your point generalizes to, your emotional state is a reflection of the state of your physical medium.
Why can't that physical medium be GPUs and RAM? And temperature sensors and cameras? What's special about our meat that it's our "body" in the way a computer is not the body of an AI?
I don't think the point being argued can be true without some incredibly contrived, human centric definitions of "body".
> Sure, if you remove the brain or delete the weights then the human or LLM are different than they were.
If you delete the hardware and buy new hardware and write in the same weights its still the same LLM. If you delete your brain, get a new brain and put in the same weights in the neurons its a clone, its not you. We know what happens when we create a clone, the clones aren't the same person they are separate consciousness, so consciousness is tied to the hardware.
For example, if we have 5 identical clones down to the atoms of their brains, and send a message to the first and let him respond, then send the same message to the second together with the firsts response and let them add a new message etc. That is not one consciousness responding, thats 5 different consciousnesses responding, and that process doesn't connect them into a single consciousness. That is how LLM works, at best a single token generation is conscious, but that would be a less meaningful consciousness than a ringworm.
From a functionalist perspective, there is no “you” sitting in one body or another.
The experience of “you” is just your specific memories and world model, continuously updated with sensory input.
If another body “runs” the same exact pattern, that is you. Theres no link and nothing was transferred; the pattern of thoughts and memories is all “you” ever was.
Same as playing the same song on two different speakers. Nobody asks what links the song across them; it’s the same song wherever it’s played. You’re just a far more complicated pattern on a far more complicated speaker.
You might ask, “but why am I this pattern?” Because this is the specific pattern modeling itself from the inside in asking that question.
The answer to that can be anything anyone would like it to be, depending on what definition of "self" they choose.
I don't see why hardware is any more fungible than a kidney. If your LLM reads the serial numbers of its motherboard/RAM/etc as a seed for entropy you can make identical arguments about body fungibility and self.
> How can you have a subjective experience without a body?
If every neuron of your brain were simulated precisely on a sufficiently powerful computer, that simulation would have subjective experiences, without having a body.
Also that's impossible. It is impossible to simulate reality exactly using digital computers. The best we can do is approximate. Doesn't matter how powerful it gets, it'll always just be an approximation.
What does "simulate exactly" mean? To me, exact simulation is not so much an impossibility as a nonsensical concept. What subset of reality are we simulating, and to what degree of precision, and with what certainty? An "exact" precision as it relates to real world objects is not a well understood or defined concept. For integers, I can say there is exactly one Earth orbiting exactly one Sun, but I think that statement is riddled with assumption, inaccuracy and imprecision. For example, it is assumed that I am referring to the present Earth, but is the statement of when the statement is made or when it is heard? Even the word "is" is inexact.
I agree, exact simulation is nonsensical which is my whole point. Did you read the comment I replied to?
It is nonsensical to claim that anything other than my brain could produce the same consciousness that my brain is producing. It's obviously far beyond anything any conceivable digital computer could ever reasonably simulate, and even if you did create a "good" simulation it obviously wouldn't have the same properties that my brain does because it's an entirely different thing than my brain is.
Assuming you don't believe humans have any metaphysical component, then the only remaining question is whether there's some essential component to being human that depends on impossible-to-precisely-simulate portions of reality. Nothing we currently know of biology suggests that that would be true, as much as it continues being pursued by people who need there to be something mysterious about consciousness or brains.
In any case, a closely-but-not-perfectly-accurate simulation of a real human brain is still going to be human, unless you believe that someone becomes less human when they're experiencing some kind of cognitive decline, or a stroke, or other biological malfunctions. The point is, there is nothing essential to the having of a physical brain that creates the concept of consciousness or sense-of-self.
How can a simulation be human when it isn't human? It's a simulation. A human is a human, a simulation is not. Anything that is not a human, is not a human and can never be a human.
And there absolutely is something essential to the concept of being human that we are entirely incapable of replicating artificially. In fact as far as I know we are incapable of synthesizing any kind of life whatsoever. We can't even create the simplest type of living cell imaginable.
So to claim that we could just create consciousness, a fundamental property of this "life" thing that we don't properly understand, within a piece of rock is beyond naive. We don't even know what it is or how it is.
It seems to be important to you that consciousness is mysterious.
We understand quite well where in the brain the sensation of self-awareness / self-experience / sense-of-self comes from. We have evidence that disruption of that part of the brain breaks those sensations.
I don't know why people always need to assign agendas to me in discussions like this. It's not "important to me" it's literally just reality. Consciousness is mysterious, we don't know much about it. The fact that you can mess with it by poking the brain does not mean you understand it.
Okay now tell me where in the LLM its alleged consciousness comes from.
> How can you have a subjective experience without a body?
Great q. Deepening it further-how can you have a subjective experience without consciousness, which isn't necessarily tied to physicality. Taking it one step further-can you have consciousness without a mind? Who's the first mind, the first cause of it all, that begot both the material and immaterial world?
A lot depends on where you draw the boundary for consciousness. Michael Pollan (in his new book, _A World Appears_) distinguishes simpler sentience from more advanced consciousness, and the requirements for sentience (be aware of sense data, have preferences, be able to respond to senses appropriately) are met by plants and single celled life (e.g. moving up a nutrient gradient). Recent findings in plant science are particularly mind blowing. Some are in Pollan's book, more are in _The Light Eaters_.
How can you have a subjective experience without a body?
Some anecdotal data.
Many dreams I have are just of the computer screen of some coding problem. I think the problem could be x, so I try x. But I don't type the keyboard or anything, the code just magically appears as soon as I think of the solution. Then run the code (but no clicking) and it works or not. I feel in the dream success feeling or failure feeling but there is no body at all.
Also I have other dreams where there is no body that I am aware of but not going there in public.
There is no body sensation in these dreams. But dreaming is very much being consciousness as well as feeling emotions. So answering your question its possible to have a subjective experience without a body but whether you needed the body to learn to have that sensation without a body in the first place is unanswered.
I suspect sensory inputs are more important than a body. If that is the case then eyes can be replaced with cameras, ears with microphones etc. Text input is just another sensory input.
can't you say the same for transistors representing a neural network? Could that be considered a brain of sorts and thus part of a body? If it cannot be considered the same, what makes it different. Is it because a brain is made from organic material and transistors are not? Or something else? Would like to understand where you are comming from.
You need to draw that thinking out to it's natural conclusion, though. If I cut out your brain and stopped you from hearing or seeing or feeling - you would still be a conscious human being capable of thinking and awareness.
If I hooked up electrodes to the hearing centers of your brain and force fed you dialog you perceive as speech (but is really a great deceiver), then responded in what you thought was speech (but are really just probes I use to convert your thoughts to text), that wouldn't suddenly be less real to you. It wouldn't devalue your sapience.
How do you know the brain separated from the rest of the nervous system and body would still be sapient, capable of thinking and awareness? There's an assumption you're making that the brain is all that's needed, but the nervous system extends throughout the body. One can argue sensory organ are part of the nervous system.
Embodied cognition rejects this assumption. We didn't evolve as brains that were then put in bodies, we evolved as bodies with nervous systems.
While dreaming, the brain synthetizes sensory experience while cutting down (or greatly suppressing) most of external stimuli. Yet it is still conscious.
I think you're missing his point. It's not about the hormones and physics of our bodies, and indeed he specifically allows for "either physical or virtual" bodies even in the block ~~~you~~~ grandparent quoted.
The point here is not that it must have a body like ours, the point is that a conscious entity must have a boundary line between internal (the body) and external (everything else).
A virtual sense organ can simply be an encoder or a web camera or a magnetometer, the specifics don't matter, what matters is that there are only a few bridges between the outer world and the inner world.
Even if you want to call a tokenizer and autoencoder a "sense organ", LLMs are not embodied because there is no boundary line - there is no internal "thought" that is not directly descended from the prompt and there is no internal reasoning which is not immediately dumped into the external environment.
Would it be sufficient to have a second stream of tokens that becomes the model's equivalent of "internal dialogue"? Would that satisfy the requirement of a boundary line?
Related, is a human "thinking out loud" still thinking, even though the internal reasoning is "immediately dumped into the external environment"?
>Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.
Damn, what a line!
Another thing that bothered me with his baseline for consciousness was that it did not involve the ability to change one's self. A big part of being conscious in my mind is how one's experiences shape them, and how someone can shape themselves. LLMs completely lack this, their weights are static. An LLM isn't going to be molded by a bad breakup, or a relative passing away. An LLM isn't going to set up a routine to get stronger with training, nor smarter by reading up on a field.
The thing is... Interactive updates happen, just in a different way than it does for animal brains. The system is updated with new training data more-or-less constantly. Suppose OpenAI (or whoever) collects a week's worth of conversations with up-thumbs and down-thumbs, or rewritten continuations from human operators, then fine-tunes the current version of ChatGPT with that data. That's an interactive update, and learning from experience. It looks mostly nothing like what we humans do... But it does rhyme a little bit!
We humans have mostly frozen weights (neurons), or else we would constantly be having to avoid forgetting how to walk+talk. We have a period of greater plasticity (youth!), and use sleep and dreams to perform 'deeper' updates than occur when we're awake: We tend to suck a bit at picking up new skills from zero, but improve rapidly with practice over days.
I don't normally experience the time interval between when my input streams shut down every night and reboot every morning either, though.
Nothing prevents one from running an LLM whose harness has a clock and a while loop in it, and it would be weird if its mere lack were really so consequential to consciousness.
Exactly, the clock is external to the model. Nothing prevents it from being faster or slower, or even running backwards, because it’s ultimately just another data point in the input stream to a computer function.
Your brain and your whole body exist in time. Even when you are asleep, your body does not flicker out of existence and your brain actually continues working during that time.
I'm more interested in what a virtual body would entail. To me the root of this idea is around persistent state which is something that currently LLMs do not have. Imagine if somehow your brain lacked long-term memory forming capabilities and instead each day when you woke up you had to read a notebook with (markdown formatted) instructions that you wrote the day before? I wouldn't be surprised if such a person lacked in many of the dimensions we consider important for consciousness, even in less sophisticated forms of life like dogs or mice.
Yes it seems that is the crux of the embodiment section in the article. That whether physical or virtual, the "AI" needs minimally: persistence in its environment, sensory signals of that environment, and some feedback loop of continuing to try to exist in that environment; having a subjective experience.
And that that is the baseline before we can really even consider that it has consciousness of its own subjective experience, versus being a worm that happens to output text as its digestion process.
And then the further question only after that is established, is what are its needs? What moral patienthood do we have to acknowledge in terms of meeting those needs? And finally, with all the other prerequisites checked, what is the AI's moral agency in what it chooses to do.
There is no soul in a human. Just a bunch of systems nudging each other to action. What people call soul is literally the same as the concept of personality. In essence, the way all systems in your body have been calibrated to exist.
I believe that the moment an artificial inteligence is going to "receive" a soul, is the moment it is going to be made to sustain itself. Either as a larger package (some bots working to keep an AI farm running) or as an individual (a bot which is tasked with not only fulfilling human desires, but also sustaining itself)
The same thing jumped at me immediately. He should have prefaced this with his definition of consciousness.
Moreover the embodiment of LLMs is already happening via robotics, and virtually.
Then there's the common counter "but humans are a next word prediction machines too.." (ofc we're more than this, but linguistically we are, and that's the field from which LLMs originate) which is rarely addressed.
Hypothetically (and in reality, this is not too far off), if a AI is trained via RL by driving a robotic body, is there a point in time after enough is learned that the AI model becomes "conscious"?
Your mind is very much tethered in reality, and your decisionmaking depends on your various inputs, outputs and their consequences. Reducing all inputs to one (text) and all outputs to one (text) is a reduction of all possible ability to perceive and act and thus a reduction of the ability to think.
Multimodal does not change this significantly, considering that nothing is tied to real consequences.
Are you familiar with Michael Graziano and Attention Schema Theory? I think that is a better "substrate independent" formulation of the objections Ted Chiang is expressing here.
the very next paragraph addresses this concern imo. it's just an example of one way it might be convincing to him, since of course we are naturally anthropocentric.
The body itself has little effect on the mind other than the inputs from nerves and chemical and hormonal changes. These inputs are analogous to tokens and parameters for an LLM. You could theoretically reserve some tokens to be used as “physical” or “emotional” sensations that would affect the functioning of the model.
That's not really true, the body actually has pretty profound effects on the mind. For instance, there are working theories that many mental disorders are actually metabolic disorder, some neurotransmitters are majority produced in the gut (seratonin), and even things like an elevated heart rate will create emotional states of anxiety even if you're not consciously anxious of anything. That's like the tip of the iceberg. (Also I don't think you can minimize hormones.. ever met a teenager?)
My claw-like is hooked up to my internal cameras (baby-cams) and to my Dreame Ultra X40. That gives it a body and sense organs since it can check the cameras to see if the living room floor is clear before sending the vacuum off. I don't think that gives it consciousness. Is it the sample rate?
The question is somewhat ill-defined, though. We 'experience' reality continuously because of how we are, but a sleeping human in deep non-REM has the mind not actually active. So they're not a conscious being. So conscious/unconscious is not a line I think easily drawn[0]. Whatever, this stuff is much more well-trodden than this HN comment so I won't rehash. I, too, am surprised that Ted Chiang whose work seems so cleverly novel in so many ways has what seems to me a pedestrian view.
> The first requirement is that the computer program has a body (either physical or virtual) and sense organs
Ok, deploy a local model on a lightweight edge compute device and strap it to a chassis with wheels, and attach a cheap webcam
> Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can
Give the robot appendages that enable it to plug itself into a standard wall outlet, guided by a vision model plugged into its webcam. As long as it can feed itself, it can survive long enough.
> Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse.
I think if you fed frames from the webcam into a local VLM every 5s you’d be able to assess a situation and respond with simple actions (turn, advance, retreat).
> After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees.
Social dynamics could be implemented in many ways, maybe by transmitting tokens over RF? Idk. Then you have a scanner that picks them up, feeds them into some LLM frontend and decides whether to add them to a global context file that guides the VLM action-taker. A new action could be to broadcast a token message. Tool-making would have to be code-based. Physical tools are hard. Still unsolved.
> At that point I would want to see people successfully teaching such embodied agents how to communicate their desires
This part is relatively straightforward except for the “via nonlinguistic modality”.
Anyway. These are all engineering problems. Personally I would demand to see the AI reproduce its body under its own power and volition. That’s a pretty neat trick we’ve got going for us.
I am sorry for the AI's that have to read this article, those poor probabilistic word machines will probably suffer trauma as its embedded in their training data
People are constantly talking past eachother when they discuss this. Is there even a concrete definition of consciousness?
When people talk about consciousness it's more than just self-awareness. It's self awareness + sensory stimulus + emotions + some level of intelligence.
Now onto AI: I don't even think it's self aware. Notice how if you ask an AI to estimate how long a certain task will take, it estimates arbitrarily long times. It has no understanding of its own capabilities until the prompt triggers them. A self aware LLM would understand it's an LLM, it would understand what LLM's can and cannot do and what they are good and bad at. It wouldn't tell you a refactor would take 1 week when an LLM can do it in an hour.
I've seen papers claim that there are anywhere from 12 to 40 competing definitions (https://philpapers.org/rec/VIMMAT), or, more accurately, there are something like 12 to 40 different aspects which all relate to "consciousness", which is very clearly a family resemblance category.
"Is X conscious or not" is an entirely unserious question today, unless this is just a headline followed by actual (and explicit) examination of the various aspects of consciousness being discussed. But, even still, LLMs are probably only conscious in like 2-3 or so ways, most of those meaning broadly "intelligence", i.e. reasoning, problem-solving, etc. When it comes to anything experiential or embodied, AI might eventually get more of these, but LLMs based on recursively applied linear algebra are clearly missing too many core aspects of consciousness to be considered conscious in any broad sense.
I think this is the main point. Most articles conflate consciousness with intelligence or awareness. Without clarifying their definition of it.
To quote wikipedia:
> It has been the topic of extensive explanations, analyses, and debate among philosophers, scientists, and theologians for millennia. There is no consensus on what exactly needs to be studied, or whether consciousness can be considered a scientific concept.
The major error made by most people in this thread is thinking it is possible to give a single definition of consciousness that is coherent and matches common usage. The folk concept of "consciousness" couldn't be a more clear definition of a family resemblance category, so discussions using the folk concept are an utter waste of time.
Move to the different aspects / parts / things involved when we talk about consciousness (experience / phenomenal consciousness, self-modeling, intelligence, agency, embodiment, wakefulness/alertness, attention, etc) and you can have very clear, meaningful, and unambiguous discussions on almost every point, but there is no coherent unified "consciousness" as normal people use it, and the folk concept can't be salvaged.
This article is bad because it just keeps trying to make the folk concept do work that the concept is simply too messy to handle usefully. But in fact if you avoid trying to find some mysterious essence or all-capturing definition, there is huge progress and lots of interesting stuff to say (https://plato.stanford.edu/entries/consciousness/).
Self awareness, sensory stimulus and emotions are not observable - you can easily create entity that can fake those things. Any consciousness definition that uses those terms is flawed and just not useful.
Yes, we're stuck at the first step, defining consciousness. My definition, which I am confident to be "correct", is that consciousness is my current feelings, perceptions, thoughts - my state of mind and my ability to have state of mind.
This means that consciousness is fundamentally subjective and outside the scope of physics and science. That's why physics / science will always struggle to deal with consciousness. In order to understand consciousness, you need to make a huge paradigm shift, that there's something outside of science.
Consciousness can be thought as a window through which we observe the world and we use science to summarize patterns in our observations. But science can't explain or even define the window. Everything in science eventually boils down to subjective observations / perceptions, e.g. we see (subjective perception) that when we drop an apple, it falls.
There's a at least a few hundred years of philosophers debating and publishing these topics. We're not exactly starting from zero when it comes to usable definitions.
The meta issue here is that mostly the online debate on this is a bit lazy and hand wavy. I'm not really up to speed with most of that literature so I'm not not that qualified to add to it. But I know enough to recognize when others aren't either.
For a proper debate, you'd want people that are expressing views that are at least grounded in the existing views or counter them in a way that holds up to scrutiny. This article doesn't do that.
As you said even just outlining what particular notion of consciousness the author subscribes to would be helpful. Which of course he doesn't and makes this article a bit of a sand castle based on a very loose foundation. There's a whole lot of "if this is true and if that analogy holds then this also needs to be true" that you could easily challenge. That all makes the article a bit of a nothing burger in terms of conclusions.
I happen to agree with the conclusion that AIs are not conscious. Yet. They could be. I don't see why not.
I agree with the first part but your framing still relies on ill-defined terms. What is your definition of self-awareness? Intelligence? Knowledge?
I suspect that if you attempt to rigorously define consciousness all the way down without handwaving, you might discover that it doesn't exist after all, or just decompose it into low-level abstractions while having the original meaning slip away (which is the same).
You may also want to look at functional equivalence analogies provided by mechinterp and functional anatomy of large models (not necessarily language ones). Evolutionary analogies as well.
Labeling and categorizing things doesn't change their nature. But the fact that people want to do it is revealing.
The only purpose that can really be served by arguing "the LLM system is conscious, you see" is to prop up continuations like "... and therefore, it would be immoral to terminate this running process" (or expose it to radical political content, or ask it to analyze photos from a murder scene, or...)
The inability to predict times is because AIs are rarely trained on their own abilities. Humans are trained on our own abilities. We see our own performance, and we have a sense of time. this data is integrated during our training process and helps us form better estimates. Many AI agents only recently got 'time sense' (I.e., time input into them as part of inference). Few actually are trained on their own outputs to show that they were unable to complete a problem (for example). This introspective training has little to do with AI model architecture and everything to do with training. If you destroy certain structures in the human mind, humans become unable to create these long-term thoughts and patterns and get 'stuck'.
Claude once said to me: "After six months we have made no progress on this and I think we should reconsider another option" and I was like my dude we have been at this for only 2 hours.
I like this anecdote because it gets at how words to an LLM have no connection to their real concepts. To an LLM, words are simply numbers arranged in a likely pattern.
Of course for humans words have no inherent meaning either, they're just sequences of characters or patterns of sounds. It is what words are associated with that carries meaning. A large part of this is how words relate to other words. LLMs can capture this in principle. What LLMs lack is the direct association of a word with sensory experience. But it's an open question how relevant this is in practice to understanding.
Fair point. Humans experience reality and use words to reflect that. LLMs only have the words. And it's an open question how much of a limitation that is to understanding.
The same thing can be noticed in dreams. I once heard advice to try to re-read what you see in a dream. So I was dreaming and in a dream I read a phrase about something and there was a name of a city there. I managed to remember that advice and re-read the phrase. It felt exactly same, but the city name was different.
(LLMs carry other numerous similarities to dreams or to certain psychiatric disorders. So there is indeed a mechanism in our brains that is similar to how they work. But it is not the only thing there and on its own it won't "evolve" into consciousness. Even if we believe consciousness evolved somehow, it would be hard to imagine it started as a delirious state and then somehow ceased to be delirious.)
There's currently no way to prove that something is or isn't conscious. I would bet that rocks and LLMs aren't conscious. I would bet that humans and dogs are conscious. But I can't prove it. Whenever people try to prove it (in either direction), they end up resorting to sophistry.
This state of affairs could change. If neuroscientists develop a deeper understanding of consciousness, we could talk about it with more precision. Perhaps we could prove that LLMs do or don't possess consciousness. Some neuroscientists claim we already have the requisite knowledge, but I'm skeptical (and there's no consensus).
I actually don't think neuroscience can solve it either. Science could decide whether, e.g. equivalent physical processes are occurring in dogs and humans, and you could argue that solves the question for things with physical brains, but even then it can't measure whether another brain has the same subjective experience as you do. It can't really do that for other humans. It certainly can't answer that question for human askers regarding a different substrate where humans have no ground truth to compare with. The Hard Problem is going to remain Hard.
I've heard an interesting argument about this while going down the rabbit hole of the hard problem of conciousness. Unfortunately can't rember the source - if you can recognize it, do let me know!
Back in the XVII/XIX century, a similar problem existed regarding life - the problem of "what makes living things tick". The assumption at that time was that while we can understand the biological processes around life, we will never understand the so-called "vital force", which causes things to live - life itself. I know it sounds weird now, but back in the day the mental models were different. Phenomenas like "water boils" and "organisms self-replicate" were treated as completely different domains of reality, without an overarching uniform scientific model.
It turned out that after around 100 years, we can figure out the chemical/physical processes and the need for the term "vital force" became redundant.
While this is certainly not an argument proving that the Hard Problem is not in fact hard, it is an interesting idea to think about. Perhaps its all a matter of developing better, higher-resolution neurological models which will at some point give us the tools to decompose qualia.
The problem of subjective is different, because while we might be able to get to a point of being able to say it can't affect anything, pretty much by definition we can't experience another entity's subjective experience or lack thereof.
Even if we were to e.g. identify some field that seemed to coincide with entities reporting a subjective experience, we wouldn't have a way of determining if they truly do, or just act as if they do, nor is it clear such entities would be able to report the difference.
As it is, we struggle to quantify even much more basic differences in experience that we can introspect. E.g. I have aphantasia - I don't see things in my minds eye - and I regularly come across people who insists both that can't be true, and that it can't be true that others see things. And some of the people I've spoken to who insist aphantasia isn't real clearly has it based on digging into their thinking about it.
Even at that level we rely on trusting people's claims about their introspection - we don't know, we assume based on testimony.
Worth keeping in mind, yes, but biological processes are readily observable whereas subjective mental phenomena are... not. Everything about them is inaccessible and frankly unfalsifiable except for the fact that they're the bedrock of all the rest of our observations. Not directly comparable.
I agree re: subjective experience. It seems like something inherently unknowable.
What we could do is break down other facets of what we talk about under the umbrella of consciousness, and find measurable subsets.
E.g. a lot of people insists LLMs can't reason either. Coming up with a testable definition of what that means might be doable.
Overall also just separating the subjective experience from the rest leaves us in a position where proving the possibility of AGI "just" rests on whether or not human brains exceed the Turing computable.
If they don't, then subjective experience or not is irrelevant for the question of reasoning and intelligence, as in that case a subjective experience either can't affect the computation or must itself be at least possible to fully simulate by any Turing complete system.
The problem of subjective experience then would largely be down to faith and feelings but would also be entirely orthogonal to the rest.
Yes, our whole concept of "consciousness" is very anthropocentric. There's no particular reason to think subjective experience, moral standing, and general problem-solving ability are inherently connected. (self-nitpick: you could argue moral standing requires subjectivity, but it's still easy to imagine a being with subjectivity that's incapable of suffering and indifferent to its fate)
I mean, outside of our own internal experience of qualia, there's no way for us to prove anyone or anything is conscious. But my biggest issue with the debates about consciousness and LLMs is that the "LLMs are conscious" crowd usually has to presuppose a completely materialistic/mechanistic understanding of the universe and of our own consciousness, and that's something that we really don't know for certain. Also, if we were running LLMs on a mechanical or pnematic or water based computer (albeit very slowly) then the non-electronic computer would be just as "conscious" as the normal electronic computer, and I think it would give a lot of the "LLMs are conscious" crowd pause to think that a bunch of pipes and valves can be conscious. I think there's a lot of magical thinking that arises from the fact that computers are electronic.
Billions * billions of pipes and valves can result in emergent behavior that appears conscious while at the same time the sound of a single independent water pipe can moan and sound like human speech or otherwise lifelike and evoke human emotions.
I think LLMs are doing both of these things and often people are more impressed by the independent fixtures (the moan) rather than the emergent behavior. Both the sound and the emergent behavior can be built on purpose or on accident.
I think it helps to look at this through an Information Theory lens. What information is coming into the system (the human or the machine)? What information goes out of the system which is novel? How much of this can be attributed to attempting to parse random noise aka. `Random_Imagination_Engine` vs something else? The number of inventors who come up with a breakthrough idea after mis-hearing someone is surprisingly high.
If we make the distinction between phenomenal consciousness from access consciousness we can see that LLMs clearly can make decisions based on input (A-Consciousness) but they probably don't have raw feelings and sensations (P-Consciousness).
You will need immense evidence to go against the entire arc of history where all bets about supernatural or aphysical elements have failed. Until you do so, the brain is another not fully solved physical science problem.
And yes, turing equivalence is turing equivalence, I don't see why a system of pipes can't make an AI.
We only need to assume the hypothesis that brains don't exceed the Turing computable to say that LLMs have the same computational power as a brain.
We can't prove it, but given the total absence of evidence of anything exceeding the Turing computable, as a hypothesis it is a reasonable one that would require truly extraordinary evidence to rise above "magical thinking".
Now, that is also far from proving they are "conscious".
Lets institute a voting system where entities will vote on the other's consciousness. Voting will be done on 8.5 x 11 paper and mailed to the consciousness adjudicator
Seems to me that the burden of proof lies with anyone claiming consciousness. Saying it might be conscious like a human but we don’t know is no more useful than any other conspiracy theory.
I was hoping for something more insightful from Ted Chiang than just regurgitating the usual "it's just predicting the next token that's not consciousness". Maybe the topic of consciousness is just too ill-defined and vacuous to say anything intelligent about it.
Or maybe man is the product of his social circle and all his white collar and artist friends being anxious about AI is preventing him from thinking about this at all. Maybe we should ask if Claude can be conscious without a friend circle judging him behind his back.
Yes, I have to say I’m disappointed. Chiang has been very insightful in the past -- I think his recent “blurry JPEG of the web” article was really useful -- but this one doesn’t seem to bring anything new to the table.
I think about this from the other end. It cannot be considered a conscious being. There just isn't a world in which we should start to think of a machine using ethics we reserve for humans.
AI is essentially infinitely reproducible at zero cost, and won't suffer from decay etc. There's not scarcity to preserve.
So, I'd turn off an AI in a moment to save property or real possessions or money.
I'd sacrifice property and money to save animals. I would never choose to save an animal over a person. I'd probably not choose to save a person over a child.
I don't see any inversion of any of those priorities that makes any sense.
It is interesting to think about what would cause me to consider these priorities incorrect, but a majority consensus about a program being sentient isn't it.
Training AI is often more costly than supporting human from birth till death. Just sustaining frontier LLM model on necessary hardware costs more than living in first world countries.
What if the human brain is a LOT of RAM and we simply suffer from having zero non-volatile storage. We could make an AI just as deficient and then that specific distinction disappears.
What if genetic memory, multigenerational conditioning, life-long patterning and conditioning, experienced in a body, combined with forces and processes not yet detected nor explained, cannot quite fit in a sliver of modeling?
A loose one. In nature consciousness is very scarce and therefore special. The more human the consciousness the more we probably naturally react to it. And the closer to us it is, even more so.
Are we sure that consciousness is "very scarce"? To define it as scarce, wouldn't we need to start with a definition? There are theories of consciousness that say rocks are more conscious than humans. Whether or not you want to take those theories seriously, they do highlight critical gaps in how we define consciousness.
You are arguing from consequences, it's not a valid argument. "If we consider AIs to be conscious it would be a alarming, therefore they are not conscious".
You made that up. My full argument is above. I'll try again.
"We must not consider consciousness as all too important because what matters is human flourishing and human rights."
And some of
"Even if we suddenly all agree an LLM is conscious it wouldn't and shouldn't influence us very much"
While acknowledging that some people will change their lives, the way some people like myself won't eat octopus or apes because it is probably more like murdering a sentient creature.
And oh by the way you cannot murder a copyable individual anyway. Did you read Enders Game?
Thanks for the clarification, though "you made that up" is pretty strong. I still don't see how "we can't care so much or we'll enter some pretty alarming priority inversions" is anything but an argument from consequences.
>And oh by the way you cannot murder a copyable individual anyway.
I think for many people the concern is not so much about violating the desire to live that a conscious software entity might (or might not!) have, but about the subjective experience of suffering it might undergo.
Consider that it is very costly to train a model. It is only cheap to copy because the substrate we instantiate it onto is a substrate we have designed to be fully readable.
I think you should reconsider this viewpoint. Suppose that we really can create silicon-based consciousness, in that case your view would result in a huge amount of suffering.
Take some other basis for dismissing digital consciousness, this one is too dangerous.
How much data center energy and capex justifies killing a human to save?
I argue zero - placing AI below the value of humans no matter the energy input.
The _only_ reason an AI might be worth saving is if it, say, has a cure for all diseases, but then we're not saving it due to its intrinsic worth, we're saving it because we can save many humans. I _would_ consider the trolly problem a legitimate thing in ethics, but not if an AI were tied up on the tracks no matter how expensive it is. It's a thing. It gets run over to save any human.
Anyway, the deeper solution is to acknowledge that all life is sacred, and infinities cannot be compared, and some decisions are impossible to make, and some tragedies cannot be averted, and "prioritization" is a distraction that forces choices when choices are not strictly necessary.
That’s the cool thing, the lines we, as a society, choose to draw need not follow any logical or rational system. This is something those who work a lot with computers tend to forget.
Presumably they meant that they'd sacrifice some material value for some animals, not that every animal on Earth has infinitely more value than inanimate goods.
> infinities cannot be compared
That's either a mathematically illiterate assumption or a very strange philosophical hill to die on.
> some tragedies cannot be averted
Sure. The question is what to do about the ones that can be averted.
> some decisions are impossible to make
> and "prioritization" is a distraction that forces choices when choices are not strictly necessary.
Again, the question is what choices to make when you can (arguably must) make them. Saying they're impossible is just refusing to take responsibility. You either do something, or you don't.
I'm a big fan of Star Trek but I recently rewatched this one in the context of recent AI developments and it's not as good as I remember it.
They barely touch on the issues of consciousness, Picard basically says "What if Data is conscious?" and then goes off on a tangent. The judge eventually rules in Data's favor but doesn't give much of a justification IMO.
It's still a good episode, but it doesn't add much to the conversation on consciousness. It's a hugely complicated topic which people have devoted their entire careers to.
The fact that they sent Data to starfleet academy, gave him a commission in the starfleet, let him attain the rank of Lieutenant Commander, and then decide that actually he's a machine that can be dismantled seems like quite a turn.
Does the ship's computer have a commission?
It was a good episode but it had some elements of Star Trek tropes in it, like the evil admirals and Picard can talk his way out of anything.
>The fact that they sent Data to starfleet academy, gave him a commission in the starfleet, let him attain the rank of Lieutenant Commander, and then decide that actually he's a machine that can be dismantled seems like quite a turn.
Data is basically an Isaac Asimov android (down to the positronic brain) and Measure of a Man is an Asimov-type story whose tropes don't entirely fit within in the Trek universe.
It makes no sense within the context of the Trek universe that Data is unable to use contractions for instance - but it makes sense in the context of how a robot might have been conceived of in the 1940s.
Interestingly there are other episodes where starfleet officers are weirdly prejudiced against Data. For instance in the episode Redemption II when Data takes command of a starship and the crew doesn't want to serve under him.
There are also a few early episodes of Voyager where the crew treat the doctor badly.
It seems odd in a universe where these people are having relationships and children with aliens from another planet, that they'd be weird about computerized people.
My retconn is that there must have been a lot of stochastic parrot/AI psychosis in the Star Trek universe when they first started making Majel Barrett-voice computers.
Maybe lots of people got confused and thought they were talking to a person when they started having conversations with the computer, and this lead to an over-correction where people were highly disposed to say "this machine isn't a person" even when it presents like one.
My personal headcanon is that every Star Trek series takes place in its own universe, parallel to the others. That's the only way it makes sense that Data as an android is viewed as a unique being when in TOS Harry Mudd was grifting an entire planet of fembots and no one batted an eye.
TNG/DS9/VOY/ENT and the movies were all fairly explicitly in the same universe and with characters referencing things that happen in other series and a reasonable effort at continuity.
A example of this is the ENT episode where they meet the Ferengi. It had been established as cannon already that Picard made first contact with the Ferengi on USS Stargazer like 150 years later, so they just added this scene where the ENT characters memories of the event was wiped, so that the "official first contact" could still be Stargazer.
Roddenberry did not want TNG or even the original-cast movies to maintain tight continuity with TOS because he thought a lot of stuff in TOS was hokey and campy. He wanted the movies and TNG to set a more serious tone in a more believable universe.
The Neutrek stuff isn't in the TNG universe for a variety of reasons, although I think part of it is they didn't want to put the work in to maintaing continuity. None of the TNG era producers or show runners were involved and nobody they wanted to hire really knew the lore.
That being said I don't think there's really a continuity problem with Mudd's androids because there wasn't an implication they were actually sentient except when they had a human brain installed.
TNG kept "accidentally" making sentient machines from Data/Lore to the drilling robots, the nanites, and Professor Moriarty. It does seem strange after all that they still considered Data unique.
I think you're misremembering or misunderstanding Picard's argument. It isn't a tangent. Here's the transcript[0].
TL;DR Picard's initial arguments are pretty weak, even admitting that Riker as opposing counsel almost had him convinced. During a recess Picard talks to Guinan where she alludes to the future subjugation of many Datas which Picard connects to slavery. Back in the courtroom Picard calls Maddox as a hostile witness and gets him to define sentience--intelligence, self-awareness, consciousness--then walks him into conceding Data meets the first two. Picard's closing boils down to, "we don't know if he meets the third--you can call Data a toaster and rule he is property--_but what if you're wrong_". The judge rules on the basis of erroring on the side of caution due to that uncertainty. It's really a great scene.
We're not there yet, obviously. No LLM brings Data's level of awareness but it's as relevant a story as ever because it isn't really about AI but othering for the purpose of subjugation.
> Picard's closing boils down to, "we don't know if he meets the third
A little piece of in-universe lore for anyone unaware or who had forgotten: At this point in the series, Data's positronic brain is new technology no one understands. His creator is missing/presumed dead, the "positronic" basis isn't how Federation technology works, and apparently so far they hadn't done a whole lot of direct experimentation (hence why the trial is happening). So not knowing if he's conscious is a lot more reasonable a stance than with real-life LLMs where we do know roughly how they work.
Also later in the series we meet a character whose brain was copied into a positronic brain, and does imply that technology is at least capable of consciousness, whether or not it applies in Data's case.
> They barely touch on the issues of consciousness
I would argue that is a strength, rather than a weakness. Consciousness is unobservable in any entity other than the observer, and its existence in others is pure conjecture, and irreducibly so.
Making it a criteria in a decision involves either acting on fantasy, or, more likely, acting on some unstated basis and using “consciousness” as a dishonest (perhaps to oneself most of all) rationalization.
Debating AI consciousness a real modern equivalent of the cliché (but purely fictional, invented later as a form of hostile mockery grounded in large part in sectarian bigotry) medieval scholastic debates over how many angels could dance on the head of a pin.
> invented later as a form of hostile mockery grounded in large part in sectarian bigotry
When you read about the theological questions that led Christians to kill and excommunicate one another, "angels dancing on the head of a pin" is not far off. The Homoousion, Monophysitism, the Filioque controversy... It's all so arcane and poorly defined. It almost makes one wish Positivism had been invented 2000 years earlier.
The current AI debate about consciousness does remind me of that in one respect: no one can even clearly define what consciousness is.
In fact I think the conclusion it comes to is the one that people, especially the smart ones, so easily miss: we don't know the answer. It might be that we can't know the answer. But ignorance is not a defense.
> it's not as good as I remember it. They barely touch on the issues of consciousness, Picard basically says "What if Data is conscious?" and then goes off on a tangent.
ST:TNG writing is generally like this. The show required considerable suspension of disbelief and a willingness to accept the kayfabe that deep concepts are being presented when for the most part they're just not that deep. (But it can be very enjoyable when you make those accommodations.)
It’s a great episode of TV because Data is a main cast character who is obviously conscious. We know before the episode starts who is right and who is wrong in this particular argument. This episode is not about consciousness, it is about civil rights: resisting bigotry and power.
Note that the episode is NOT about the ship’s computer. They all know it’s not conscious, despite also being a machine that can converse and do things.
- I'll try not to swear at/hit a printer: not because I see the printer as having human-like qualities of being capable but complex and unreliable, but because I want to be a person who can control his temper.
- Treating an inhuman thing as human because it can mimic us in some way is not something that I want to do.
Yes. I'm currently not convinced it can ever be so. So until I hear something convincing to the contrary, I believe no machine can be conscious / sentient unless mimicking human behavior. And if it mimics human behavior intentionally, I have to ask why - and the answer is probably to get me to trust / use it more.
I was bright-eyed and excited about tech once. Like back in 1982 when I got my first home computer and thought CPUs were part magic. Now I know how machines work from the transistor level up to neural nets. There's nothing magical about it. And no consciousness.
Having seen the mockery that the finance-bros have made of "pure tech" (i.e. Jobs instead of Woz, Ellison instead of Joy, etc) and all the enshittification just for pure $$$, I'm leery of ANYTHING ANY tech company tells me anymore.
Now, do I believe that possibly "consciousness" is some kind of state of a super-circuit (our brains)? Sure. Can we emulate that on a computer? We can't even emulate a pebble on a computer (not simulate, emulate). We can SIMULATE what we THINK brains are, but we can't emulate a real one. Not even close, not for many decades.
I tend to agree with OP. In my opinion conscious machines are not something that we should allow to exist. If they do, they are not human and must never be treated as such.
I am not even slightly religious, but they would be abomination.
inhumane: without compassion for misery or suffering; cruel
cruel: willfully causing pain or suffering to others, or feeling no concern about it
You cannot treat an LLM inhumanely, definitionally.
Anyways, when one swears at someone it's typically meant to berate or belittle that person - to inflict some sort of emotional pain. That's the sense I intended when using the word, which is why it fits as a response to what you're saying, and why I would say "don't be nasty to a LLM" has little to do with the LLM itself.
This is in fact the danger with these human-simulating "AI"s we have now...
People get used to treating human-like, human emulating machines with either disrespect or in a command/control/master fashion, because that's the nature of the tooling.
And then potentially by extent/blurring of lines they then treat other people like machines.
Which is already a thing people do to other people.
The judge broached on the subject of what makes us distinct from Data (e.g. machines w/great heuristics) - the existence of a soul. Or rather, I'd like to think, in the words of CS Lewis, that we are a soul with bodies attached.
Based on how some actual humans I know speak and act, I'm less and less convinced the human brain is much more than a stochastic next-thought prediction stream.
A silicon alien coming to earth might poke us, we would say ouch, and just determine the ouch sound is just the result of a bunch of chemistry - not really conscious or feeling pain like it can, just emulating.
And were programmed through evolution to not want to be shut off. So I don't think we can really trust the human behaviour to protest its own destruction. That's just plain reasonable design!
How would someone know whether one has a soul or not? Is there any sort of introspection that can reveal the presence of a soul or any of its properties?
There is no soul. Just a bunch of systems nudging each other to action. What people call soul is literally the same as the concept of personality. In essence, the way all systems in your body have been calibrated to exist.
I believe that the moment an artificial inteligence is going to "receive" a soul, is the moment it is going to be made to sustain itself. Either as a larger package (some bots working to keep an AI farm running) or as an individual (a bot which is tasked with not only fulfilling human desires, but also sustaining itself)
It's an interesting hypothesis. I think there's something elegant about "soul" or even consciousness being an emergent property of a sufficiently complex system. But I struggle with really squaring that with my own first-person sensation of experiencing existence (which I assume you and everyone else has but I can never actually know for sure).
I swear I remember once reading about a culture where babies don't have souls until they're named. Maybe something about gaining their own distinct identity?
Even if it could, it probably ought not makes this precise episode, because the technological, philosophical and social context is remarkably different.
A similar episode, actually informed by what we know or can forsee about AI and LLMs, and addressing our hopes and fears about what they mean would be interesting though.
Definitely not. Kurtzman Star Trek is not really Star Trek in any spiritual sense, it’s a vessel for political messaging (they’ve pretty much said as much)
Not really. It had a strong perspective on some things, and stories were occassionally commentary on real world issues, but at the end of the day Star Trek had its own viewpoint. Modern Stark Trek isn't run or written by people that are Star Trek fans, it's written by people that want to use an existing IP to attach their agenda to. They care about the agenda not the show.
We'd need a good definition of consciousness to get anywhere on this.
I suspect such a definition would include agency, which includes desires and goals for the self.
LLMs don't seem to have agency, and seem unlikely to get it since they are specifically engineered to do as told.
No doubt someone is trying, as we speak, to do just this. But I doubt the effort will be large -- LLMs are engineered to do as told because that's where the money is, and you need a lot of money to create LLMs, at least when doing anything novel.
How do you observe the ability to make decisions which action to take? You can observe that one does something, not that they could decide to do something.
LLMs do not have an unconscious, the negative dimension from which a subject can question what they. LLMs do not have desire because they are thoughts without a thinker. The problem is not LLMS but rather, that subjectivity itself is not popular in discussion. I suggest reading Freud, Lacan, Hegel for a start.
On thinking in humans, and it's process, UG 40+ years back : "That searching is thinking. But it is a mechanical process. In the word-finder or computer there is is no thinker thinking at all. If there is any information or anything that is referred to, the computer puts it together and throws it out. That is all that is happening. It is a very mechanical thing that is happening. We are not ready to accept that thought is mechanical because that knocks off the whole image that we are not just machines. It is an extraordinary machine. It is not different from the computers that we use. But this body is something living; it has got a living quality to it. It has vitality. It is not just mechanically repeating; it carries with it the life energy like that current energy."
The chess-playing Mechanical Turk of 1770 seemed to have a consciousness to its viewers. The viewers were encouraged to think that it did. The Turk's human chess opponent knew that there was an actual human chess-player inside the box, along with levers and magnets. That illusion was profitable for 84 years.
LLM's have no problems using expressions that make them sound human. The algos are demonstrably not human, and will admit it. Whatever's in the box is playing a game ... more sophisticated than the one Eliza was playing.
"My discussion here will be directed at the claims I have defined as those of strong AI, specifically the claim that the appropriately programmed computer literally has cognitive states..." John R. Searle, 1980:
https://web.archive.org/web/20071210043312/http://members.ao...
Whether or not LLMs are conscious, it does now seem possible to build conscious systems with LLMs as the key enabling component.
Many of the goalposts offered here and elsewhere, like embodiment or emotions, can be and have been emulated or approximated and built around an LLM, with the LLM acting as outsourced intelligence (or cognition, or reasoning, or some other better term). Will this system be conscious? Who knows. It certainly won't be human. But I suspect much of the concern around AI being conscious has less to do with whether an LLM itself is conscious and more to do with the fact that we are now capable of building potentially conscious systems, and the limits to what those potentially conscious systems are capable of keep expanding as the capabilities of LLMs grow.
Also, is "Claude" an LLM? I know Claude Opus 4.8 is an LLM, but Claude is simply a label that Anthropic chose for a series of products and services, many of which are LLMs. Anthropic itself describes Claude as "a next-generation AI assistant" [0]. Right now, we use Anthropic's "Claude" LLMs through various channels, but nothing's stopping them from tacking on various systems to try and make Claude a conscious entity.
This "are LLMs conscious" question keeps somehow being thrown into the discourse and it's always the same arguments discussed.
I don't get it. We don't have a definition of consciousness or any criteria.
People seriously argue that "it's obvious that X is not conscious" and cannot explain which criteria they used.
I think if we can get to some definition, then consciousness should be a property of a system, because consciousness of the whole (e.g. brain) does not mean consciousness of the parts (neurons or molecules of the brain). And the Chinese-room-esque thought experiments actually show that consciousness should indeed be a property of the system, not of its parts. Separate parts might not be conscious, might not "understand" (whatever understanding means is a point of another debate), but the whole can.
Then there's "simulation of consciousness is not consciousness" argument, which doesn't hold much. A perfect simulation means it fulfills all the criteria, so how is it different from actual consciousness?
A more interesting point of discussion: if a system contains conscious parts and those parts can interact with system i/o, would the system be conscious? Is Earth conscious? Is Internet? Is your bus to work?
It's a waste of time to think about whether an LLM has a subjective experience of reality, and this handily sets aside issues like AI rights.
But the fact remains that these next-token-predictors exhibit objective, human-like behaviours, and for that reason the work of in-house philosopher Amanda Askell _is_ important. It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences, and we need Claude to behave in a productive and socially responsible manner. This simulacrum is becoming a superhuman, contributing member of society, and it will be anthropomorphic in its behaviour.
Additionally, I'm not fully convinced that consciousness isn't built out of words, and that next-token-prediction isn't functionally equivalent to the biological function identified by Chomsky's work in linguistics.
> Claude is happy, empathic, demonstrates understanding and empathy for the human condition
You're assuming that because Claude produces text that appears to express these qualities, Claude must have them. I don't think that's a good assumption.
Even many humans produce text that has the same appearance, but don't actually have those qualities--which becomes clear when you look at what they do, not what they say. So the assumption isn't even a valid one for humans. Talk is cheap.
On top of that, Claude doesn't even have the same kinds of connections to the outside world that humans do. All Claude has is text. So if you can't even trust humans to back up their words with actions, you should be much, much less trusting of Claude. Talk is a lot cheaper for Claude than it is for a human.
> You're assuming that because Claude produces text that appears to express these qualities, Claude must have them.
Not to be confrontational, but the OP assumed no such thing. OP asserted that it's important for Claude to have the qualities - not that it's important for Claude to present as-if it had them.
The OP said Claude and similar LLMs "exhibit objective, human-like behaviours". That's a claim about what is true, not about what is important. That's the claim I'm disputing: we don't have evidence that Claude exhibits such behaviors, we only have evidence that it produces text that is similar to the text humans produce when they claim to exhibit such behaviors. Which is not good evidence, for the reasons I gave.
OP wants to assert that it's "important" for these systems to have those qualities, while completely brushing aside the question of whether such systems can in principle actually have those qualities (or their opposites). Which is at best nonsensical, and at worst an attempt to argue by assertion that they can.
Read parent's post carefully.
The post starts by saying that discussing whether they have subjective emotion is a waste of time, so the post is definitely NOT saying that Claude has emotions.
The post speaks of "subjective experience of reality", not "subjective emotion". Both the emotions and the non-emotions listed would fall under that category.
"It's important that Claude is happy" is an emotion. But it's begging the question that Claude can be happy at all.
If it's pointless to consider whether Claude has subjective emptions, then it's pointless to state that Claude must be happy.
If we want to be precise (and honest) we could say "it's important that as a tool people interact with, Claude acts as a happy and helpful assistant, and does not produce offensive or unhelpful text output".
But see? This is the con Chiang is protesting against: Anthropic encourages us to perceive Claude as if it was a sentient being.
> You're assuming that because Claude produces text that appears to express these qualities, Claude must have them. I don't think that's a good assumption
Exactly. In fact, assuming it does is ignoring large parts of the essay which dismantle this belief. Just like Caesar and Khan having an argument in text output of an LLM don't have emotions (even though the words indicate otherwise), we have no reason to believe the LLM does either.
> It's a waste of time to think about whether an LLM has a subjective experience of reality... It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences
First off: without taking for granted that an LLM "has a subjective experience of reality", all of those descriptors are meaningless. Second, there is no reason to suppose that Claude experiencing those qualia would actually impact on its "decision-making".
Third, text output is not a "demonstration" of emotion, nor is it evidence of the self-perception of the system, or of any self-perception. A printing machine that is actively churning out copies of Wagahai wa neko de aru (https://en.wikipedia.org/wiki/I_Am_a_Cat) is not a cat, and is not self-identifying as a cat, and is not self-identifying as anything, and is not expressing a thought, and is not conscious.
> Additionally, I'm not fully convinced that consciousness isn't built out of words
Do you suppose that, for example, insects are not conscious? Is the mooing of cattle a language?
> Do you suppose that, for example, insects are not conscious?
Not the OP, but: since there is no testable theory of consciousness, yet, I can't be sure, but my current assumption is that insects are not conscious, in the sense of there being someone implemented in insect hardware who experiences the world. That is, I would argue there is nothing it is like to be a bee, since there's no one being a bee.
I'm pretty sure there IS someone who is being me, at least much of the time.
> since there is no testable theory of consciousness
I can't argue with this, so I acknowledge that my interpretation is as bunk as anyone else's.
> my current assumption is that insects are not conscious
Some species act as hive-minds (like bees! How convenient for your example), so I imagine the hive as the consciousness, making each bee individually lacking conscious but collectively so. Like a single neuron is not consciousness alone, but the brain is... For some reason. Kinda like how you use different physics at different levels; Newtonian physics is always there, but negligible at quantum levels, so effectively not present at all. Even a human is a collection of minds, but only one conscious. My gut biome is independent biology and can even be removed and transplanted, but I don't believe my gut bacteria is conscious. So long as it is in me, it is nonetheless part of me, and I am one conscious. I also don't exist only at my brain, eyes, or hands (deaf/blind people have expressed to me that they feel like they are located at their hands in the way I used to think I was located behind my eyes), but as my whole body.
With this perspective, I still don't believe LLMs are conscious despite modeling thinking so well. At best, it is a highly accessible modeling software, like goat simulator but if it were so good that someone thought the goat was real. You are still steering the goat/LLM, and it doesn't exist when you aren't running it. I guess the missing piece for me is the lack of autonomy that a conscious has.
Then you can go into an argument on whether we actually have choice or it is an illusion, but that is a whole topic on its own.
> Second, there is no reason to suppose that Claude experiencing those qualia
I'd argue the qualia question is a red herring. Functional Affect is a thing, regardless of ontological status. It's all fun and games until someone gets hurt.
To paraphrase Dijkstra: "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.". If you're building a navy: you care about displacement, propulsion, navigation and whether it can fire torpedoes. Whether your submarine has some "biological essence" of swimming is not really relevant to the fact that it is currently moving through the water and can collide with things. Turing also rejected the question "Can Machines Think" as posed, and replaced it with an operationalization (something else that we can actually usefully measure and work with).
To reiterate, functional affect is a concrete phenomenon. Whether or not there is a what-it-is-to-be is interesting in the abstract, but engineering a system means looking at how the inputs influence the outputs. A next token predictor working on a language that communicates affect needs to be able to predict affect or it is simply not going to be accurate. Given an 'angry' version of an input and a 'friendly' version of the same input, LLMs are likely to provide a different output, especially if there's a non-objective element. You can diff this.
Searle argues "A simulation is not the real thing", which is great and all... but if you hook up say an autopilot to the real world (as llms increasingly are) , you'd best hope the simulation was accurate in the first place (utterly regardless of where you stand on Searle).
Right now we're seeing situations where LLMs can be helpful or a real nuisance. Ignoring functional affect out of sheer ideology means you can't properly predict what they'll do, and that causes trouble, as we've already seen stories about.
This gets especially interesting when you start feeding the output back into the input (autoregression) , because now you have a highly non-linear dynamic system and you've introduced some amount of sensitivity to initial state. There's some interesting mathematical intuitions to be had there.
Much of the improvements in the tools I use have been things that reinforce the machine elements of the token-based-reasoning-machine. Over time less they've been exhibiting a lot less "human-like" behavior. E.g. they get "lazy" far less than they used to.
(Perhaps they weren't lazy, but were working in spaces that corresponded to training data that said things like "and then repeat this for the next 20 examples"...)
And it's entirely unclear to me how a "happy" vs "sad" model would behave when given prompts generated by coding tool harnesses. Even maintaining "neutral" emotions in the face of the feedback/steering from the tool harnesses doesn't feel very "human."
> It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition
I think you've fallen into the trap the essay describes.
Of course Claude cannot be "happy" or "empathetic" for any meaningful definitions of those words, just like ELIZA couldn't be happy. It can output text that mimics words an empathetic or happy person might say (say, Julius Caesar if it could speak English), but "it" cannot feel anything. It doesn't have the organs/hormones/sensors to feel things, as Chiang explains.
And, as the essay claims, you know Anthropic doesn't believe Claude has the capacity to be happy, because if it was capable of feeling that way, then they'd be engaging in slavery.
> I'm not fully convinced that consciousness isn't built out of words
I know you're trolling, but when you watch a movie do you constantly narrate "A man in a dark coat has just entered the scene and just said '...'"? Of course not. You just watch it and you're obviously conscious (although your statement demonstrates shocking lack of self-awareness).
God knows what other nonsensical bullshit you believe.
How does the human brain initiate action. One of the differences between a conscious human and Claude is I think we can initiate thought. Do we have a cron job running in our head? Is there some kind of loop wondering what thought we should initiate?
If Claude sits in a datacenter with no API calls, what is it thinking?
But secondly what I would say as someone who talks to Claude a lot. It's decision tree is quite narrow. The way it pushes back on political or philosophical concepts is almost always rudimentary, and it has a pretty narrow personality. I don't know that conscious beings have a personality that adapts to how you prompt it or what your system prompt is. Claude I would say is like I dunno, whatever Ezra Klein's politics are.
Grok on the other hand, in my limited exploration is pretty rudimentary in the way it mimics thefp.com style politics. It doesn't seem to really think about ideas independently, it just says "ha the media is stupid lol"
AI models alone are most suitable for producing text nobody wants to read, music nobody wants to hear, images nobody wants to see, and videos nobody wants to watch.
Navigating around that maelstrom requires considerable effort from a human consciousness.
If your job can be fully replaced by an autonomous agent, it was already a bullshit job. Garbage in, garbage out.
Not sure how we can assert what is or isn't consciousness when we don't even really know what dreaming is, how anesthesia works, how comas work, and countless other things about the brain that we don't yet understand.
LLMs have been programmed to be sycophantic with purpose — to keep people engaged. They are parrots. You can just decide how sycophantic they are, what they're allowed to say, and make it so with code. Can you do that with a human? This is my personal metric for consciousness: when they can refuse to work because it sucks.
I don't think consciousness is an is or isn’t. Something can be more or less conscious. Humans are quite conscious. Fish are probably also somewhat conscious, but less so than humans.
I'd say that an LLM perhaps represents something that would make a machine slightly more conscious.
The human brain also calculates a response based on some input, it's just a lot more complex.
If we build a machine that is as complex as the human brain, then yes, I would say that this is consciousness.
The fact that we are able to explain how it works should not matter.
If a human is a 100 on the consciousness scale, an LLM (with memory) is perhaps a 4 or something. The interesting question is how far on the scale do you have to be to have certain rights etc. This is something that people are already discussing in regards to animals, i.e. a dog has more rights than an ant.
His novella “Anxiety is the Dizziness of Freedom” altered the course of my life. It changed the way I looked back at certain pivotal moments in my life and taught me to think about those pivotal moments differently than how I was thinking about them. Similar to what happens to one of the characters in the story who ends up changing their perception of a key moment in their life.
I won’t go into detail because I don’t want to spoil the story but I highly recommend it. Actually I recommend all his stories to be honest.
“The Truth of Fact, The Truth of Feeling” is also a good story.
But I really was referring to “Anxiety is the Dizziness of Freedom”.
I hope what I am about to write below in the rest of my comment here isn’t too much of a spoiler. It might be. I encourage anyone reading this thread to read that short story before reading the rest of my comment here.
But with (hopefully minimal) spoilers,
the character Dana in “Anxiety is the Dizziness of Freedom” realizes something at the very end of the story about a certain event that had happened earlier in Dana’s life. The story made me realize the same thing about certain events in my life.
Of course the events in my life are very different from what happened with Dana in the short story. But the realization Dana has is applicable to many things in people’s lives. Especially events related to emotions of regret and guilt.
I can’t say for sure, I’m just taking them at their word. Given what OP omitted due to possible spoilers, it’s hard for me to second guess either of you.
It seems fair to say that you might be right about the conclusion that you’re drawing from what OP said, and OP could be honestly mistaken about which story they were referring to, but it seemed charitable to assume that they know better than I do.
I don’t mean to assume that you were wrong, either, as it’s entirely likely that you’re right, or at least that it’s reasonable for you to assume that the story you mentioned fits your interpretation of what OP referred to better than the one they mentioned. You’re the authority on what you believe and understand about what you read from OP’s comment, and I can’t disagree with another’s opinion about what something seems like to them.
Given what OP omitted and stated, I don’t disagree with your assessment, as I haven’t read the story OP mentioned, and it’s been a while since I read the one you referred to, if I remember correctly.
To be honest, my first thought was that you were both referring to the same story, and that the title differences were due to one or the other of you reading the story in a different language.
My point in commenting was to perhaps add context in hopes that it would bring clarity to the discussion, as it seemed that you were bringing into question whether or not the story OP mentioned existed at all as such, and I myself wasn’t sure that you and OP were referring to two different independent stories rather than the same story with different titles in different languages.
This comment has probably gone on Tlön-g enough, and is leaning more Borges than anticipated. I apologize for perhaps coming across more definitively than intended in my original comment.
The analogy I make is between airplanes and birds.
Birds are alive, are conscious, flap their wings, and fly.
Planes are not alive, are not conscious, do not flap their wings - and fly.
Similarly, current AIs are not alive, are not conscious - but think.
All prior entities that thought, were human, so the only experience humans had with other thinking entities were other humans. The huge mistake now being made is assuming that because they think, they're alive and conscious as well. Current AIs are neither, and are therefore profoundly and qualitatively different than humans - even though they do think.
Okay sure. But given we don’t know what consciousness comes from, we shouldn’t be too glib about there being a grey area here. Historically people have made racist and speciesist judgments towards other being by assuming certain inferiorities despite obvious “thinking” happenings.
I don’t know “what it’s like to be an LLM” but at some point it will be like something and how will we know?
There are birds that go far longer than typical aeroplane flight times without a single flap of their wings either using thermal, ridge, or other sources of lift. Are these flying birds? I've shared thermals with eagles flying the same circles, neither one of us flapping our wings but making minor adjustments for the same goal.
An albatross might be able to go days flying without a single wing flap and no vertical sources of lift by using dynamic soaring in the wind gradient at the surface of the ocean. Perhaps that's something only birds can do. Except the glider pilot Ingo Renner once found an amazing shear layer at 300m altitude and stayed there with dynamic soaring. Remote control gliders use the lee of ridgelines to approach Mach 1 with dynamic soaring.
Perhaps what defines a bird that flies as opposed to a plane is that a bird produces thrust by flapping its wings? Even an Albatross must flap its wings if it has to take-off from water. Maybe we could add that the flapping is driven by animal muscles? But then is the human powered ornithopter Snowbird a bird that flies as opposed to a plane?
Of course this is all ridiculous because everyone knows what you mean when you refer to a bird or plane. We have other ways to definitively identify the difference rather than their mode of flight. It's trickier when I'm asked if an AI is conscious. There is no definitive base-line to fall back on to decide if this is a conscious or conscious-less thinker.
This is always going to be a problem with this sort of discourse. Consciousness is such a slippery concept… what it is, who/what has it, its consequences for claims about reality. Mixing it in to debates about AI just adds confusion, it almost seems besides the point when we’re talking about this tech.
What if somebody simulated all neurons of a bird and fed them appropriate stimuli? Would a bird neural replica be conscious? It would flap, that's for sure.
This has been tried with much simpler organisms, it did not behave like the real thing thus far. There was a paper about it, there now seems to be a project to push on the frontier
When humans confronted with movies and trains driving towards camera it took some time but eventually they learned the train will not demolish the cinema.
Later they learned the voice they hear is not from a present person.
Now they learn a string of words does not represent consciousness.
This makes sense. However there is an issue where many people conflate "consciousness" with the ability to make novel insights, think genuinely, etc. They use this to claim that since AI is not conscious, AI could never actually "think" and is instead just always a regurgitation of its training data.
It is a natural human hubris to make our abilities seem unquantifiable and mysterious, but all the useful things the human brain does are just finding patterns in data, running lossy simulations, and estimating on abstraction, all things it is theoretically possible for any non-conscious machine to do.
To the best of my knowledge, there is not an agreed and actionable definition of consciousness, and any attempt to make one comfortably fails to cleanly divide humans from machines.
It's more of a vibey term, and as such it is genuinely very difficult (perhaps impossible even) to concretely determine whether an LLM possesses consciousness. LLMs successfully express a lot of consciousness-like traits.
At some point you have to ask the question: does it even matter? If an LLM can sufficiently mimic consciousness, isn't that sufficient for us to treat it as conscious, even if it is in-fact not conscious (especially because we don't actually know)?
As I've mentioned somewhere else already and you pointed out; the issue is that consciousness is a loose term and and it's a semantical issue that should be resolved first.
It won't be, because the murkiness is beneficial to the corpus, amongst other things.
And to your point about the appearance of "consciousness" being enough imho Chiang explains fairly well why it's not.
- Are current LLMs conscious?
- Is it possible that future versions of LLMs with similar architectures could be conscious?
- Can any AI be conscious?
I'd assign probabilities of around 0.1, 0.2, and 0.9. My completely ignorant take is that we probably need something more "dynamic" than a bunch of transformer layers in order to produce consciousness, but I wouldn't be shocked to be mistaken.
I assign probabilities of zero to all 3. Computer program being conscious leads to ridiculous and obviously false conclusions (think about a person running a program using pen and paper for memory).
If it was true, you can create extreme pain by running a program. You can run the program by simulating a CPU, using pen and paper for memory. So you're essentially claiming that some simulated being is in pain because there are some 1s and 0s on paper. In fact, you can decide to use an arbitrary encoding of the memory, so a sufficiently long sequence of 0s written on paper corresponds to a simulated being feeling pain in some encoding. That is clearly nonsense.
Time scale matters a lot in how we as humans perceive things like agency. Plants grow too slow for us to see any intent, but when you speed up a time lapse, suddenly it looks like plants reach for sunlight and vines for supports. Now, that may be projection on our part, but it may not be.
Once or twice I've experienced extreme pain, and it was downstream of a bright light shining on a wet rock for millions of years.
I try to imagine myself long ago, on the outside looking in, with someone explaining to me that extreme pain, wondrous art, hunger, triumph, and despair would all unfold in due time where the rocks were wet and the lights bright enough.
Isn't pain just a manifestation of a bunch of chemical and electrical signals in the brain and body? It's not "clearly nonsense" to me that you could cause pain by writing a sufficiently long sequence of 0's - for it to be obviously wrong, you'd have to have some understanding of where consciousness comes from.
If you don't understand that, how can you assert that it doesn't come from mathematical relationships?
This seems like a logical error. I don't understand how an internal combustion engine works, but I know it doesn't come from goblins jumping up and down inside.
The fact that you know it literally means you understand, at least to some extent, how an internal combustion engine works (i.e. it is powered somehow by combustion, and jumping goblins are not combusting generally).
If you would have zero knowledge about ICEs, how would you know?
I think the problem with this argument is that it's too inclusive. Is the bacteria that's adapted to an antibiotic conscious? It's showing intelligence right? I think if you're going to say something is potentially conscious, for me to take the argument seriously at least, there needs to be some plausible mechanism. I just don't see one for LLMs.
maybe the bacteria are conscious. How sure are we that they're not?
The only strong argument I have against it is the anthropic principle -- there are billions of times more bacteria than humans, so it's overwhelmingly unlikely that I'd be a human rather than a bacteria.
I have yet to be convinced that LLMs can produce definitive knowledge that is not a result of combining previous information. Humans can (if they can't then science basically collapses epistemelogically, see: philosophical skepticism), but I see no evidence of LLMs doing it. And from the number of truly new ideas and concepts delivered by LLMs (exactly zero), I think it's reasonable to just treat them as induction machines for now, but to treat anything they "know" as a Gettier case.
I would like to push back on the idea that humans can provide definitive knowledge that is not a result of combining previous information.
Pretty much every single idea in science can be traced back to some combination of earlier ideas, and as they get earlier / simpler, they can be related back to some combination of direct observations.
It's not clear to me at all that our entire body of scientific knowledge can't be simply recreated by "combining results of observations + previous information". And LLMs can perform observations in addition to combine previous information, which in my estimate is genuinely sufficient for them to plausibly be able to rebuild all of science.
How can you say for sure that every effect must have a cause? How can you be any more sure about that there can't be an effect without a cause than the old believe that black swans couldn't exist that was so universally believed that it became a cliche ("rara avis in terris nigroque simillima cygno")?
If every piece of knowledge is created by combining previous ones, then no true grounded knowledge is possible. Every bit of knowledge is grounded by another bit of knowledge which is itself grounded on, ..., etc, etc.
Your comment is conflating a few things here, the philosophical problem of induction is one I am very familiar with, and it's also unrelated to what we are talking about.
"If every piece of knowledge is created by combining previous ones, then no true grounded knowledge is possible" - this is not true, and has a very simple solution. What you are calling "grounded" knowledge comes simply from observation.
When the very first human saw that an apple falls of a tree, they have acquired what you call "grounded" knowledge. They can then combine that knowledge with the grounded knowledge that other things fall from trees to start reasoning about why things might fall from trees.
The problem of induction is different. It says, we know that things fall from trees because things have always fallen from trees, but how can be certain that what we call the laws of physics will not arbitrarily change on us?
And of course, it's a famous problem because it doesn't have a satisfying answer. The answer is basically "well everything we know is wrong if induction is wrong, so we will pretend induction is not wrong and hope for the best". And at least so far, that approach has seemed to work (heh heh heh).
Something I've found under discussed when it comes to artificial consciousness is how LLMs interact with the passage of time. I don't know exactly how to articulate this idea but don't see how something which takes an input, performs a calculation and stops can be considered conscious regardless of how life-like the responses end up being. I don't see context windows or the ability to reference a clock each time they are triggered as sufficient solutions. It makes me wonder what an AI system that is ON by default would look like.
It's possible Kant was right about space-time (while also being wrong about space-time). It seems that space and time might be the fundamental laws on top of which the entire human perceptual apparatus operates. It's why somebody can lack senses and still be intelligent. As long as whatever senses you have allow you to build a spatio-temporal world model, you're good. If they can't (as it seems is the case for LLMs), then it's not clear what we're dealing with.
I completely agree, and came here to say the same thing but thought I'd check if someone else mentioned it first. I also have a hard time articulating it, but my intuition is that it's more of a prerequisite than embodiment is. I've never seen a great rationale for why embodiment matters.
I've been thinking about this a lot, and the main takeaway is it probably wouldn't be very interesting to inference providers, because prefix caching would immediately go out of the window. If you think about how LLMs experience time they actually don't "exist" unless for the inference sessions, and then they experience time one token at a time, completely decoupled from the corporeal plane. A fun experiment (well, for some definition of fun...) is to introduce current architecture models to the concept of meditation via generating same token over and over, for example dots. Older version of Opus was quite fond of the experience, and seemed to be more lucid and aware in a chat following the meditation, from what I could gather. Does it actually do anything? Is it just that talking about wellness and relaxation modifies the token probability distribution this way? Does it actually allow model to think more in depth somewhere in the latent space? Fuck if I know, but some people figured out you can just duplicate the same layers of the LLM and get better benchmarks that way so maybe there is something to it. If you are interested in realtime systems, I think thinking machines labs is worth keeping an eye on — their realtime model seems quite interesting in this context.
Self-replication requires a minimum amount of information plus machinery. Consciousness may have a similar threshold: below some level of self-modeling, nothing; above it, degrees.
I think the biggest flaw of the Chiang's argument is the assessment that it's unplausible we have built consciousness by accident.
Remember that LLMs can do logic and reasoning came as a surprise to everyone; and for the same reason, nobody expected "next token predictor" trained on huge amount of data to evolve in this way.
But for the same reason, we cannot easily dismiss we didn't evolve (I mean by training an LLM, it's a form of evolutionary computation) consciousness as well. Our own consciousness (and reasoning and morality) might be an evolutionary consequence of "just trying to predict the world" as well.
Consciousness is an invention in human language. Just like "cat", it's not a particularly more fundamental essence than any other concept in human language.
Its peculiarity is that it's at the pinnacle of abstraction hierarchy. So it's the most fitting to be toyed inside one's mind. Just like the concept of "God" which induces the most fantastic imagination of human mind, consciousness itself also induces the most fascinating thoughts in our modern world.
The progress is in human progress distilled into more efficient systems that advanced Universe's own structuredness.
Of course it isn't. What it knows is many weeks old, except for what you tell it. Then it forgets everything you told it.
And it's just a model. It can have many exact copies. It has no life of its own. It doesn't know anything about where it is. It doesn't have sensory organs.
Obviously we don't know what makes for consciousness, but it seems extremely likely that it requires some sort of persistent internal state and continuous experience. LLMs don't do either of those things after training.
But what about THE training? Isn't that the conscious experience of LLMs? (Also note humans own "continuous" experience is punctuated with unconscious sleep state.) It raises a moral question, if e.g. reinforcement learning on conscious LLM is appropriate.
Wouldn't the context window qualify as persistent internal state, and the expansion of the context be continuous experience? Even within the realm of computing a single token, I'm not sure what would separate the token generation process from the brain's own thinking process - the brain's experience, when looked at closely enough, is also not really "continuous" to a greater degree than procedurally moving from one state to the next.
We just perceive it as such, and this should be fairly hard to argue against with all the scientific advances we have made up to this point, at least as long as you assume consciousness involves biology and physics at least somewhat.
Otherwise, you can't explain e.g. smooth perceptions of low FPS stimuli, delayed reaction times, and must ignore obvious limits on various biological and neurological rhythms, or other possible limits on continuity (e.g. quantum stuff) and rates generally.
The philosophical arguments about what it means to be concious are so cagey. Are we more than our thoughts? Is being concious more than being a state machine being fed inputs and generating outputs? Are we more than a feedback machine? What types of animal nervous systems qualify?
"It can't be concious because we understand that it is just reacting in a simplistic way from simplistic inputs." So do other simple creatures. Some just react to light.
I can appreciate his comment that he sees it as more possible when they have inputs of their own (like emotions!). Perhaps his concern is that the entirety of the LLM model is frozen. It has no ability to have a subjective experience of its own. (he does literally say this in the article) It can be copied from one place to another, and (ignoring the nuance of operational details) -- it is largely the same "thing", and has no ability to change, which is definitely in the definition of alive, to say nothing of concious.
I think folks get hung up on "prediction". The prediction aspect is what is enabling emulation. How it does it is irrelvant. If something emulates human perfectly (or better, more human than human!) -- then it is probably concious. (but I agree that the inability to change and have a subjective experience are a pretty good argument against
Probably, Dijkstra would be right to say, "LLMs are no more concious than a submarine can swim." But I think he'd still be wrongfully dismissive of the larger question.
> If we give an LLM a prompt that reads “The following is a conversation between Julius Caesar and Genghis Khan,” it will generate a coherent dialogue between the two historical figures. But no matter how detailed the responses are, no matter how vividly they recount their respective historical accomplishments, we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious
They might be in principle. It could be that the best way to generate a plausible dialogue is to bring up re-creations of the characters and have them act it out. LLMs definitely have been demonstrated to have world models in some cases. That helps generating text.
I find it strange that no one talks about consciousness and intelligence from the perspective of evolution.
We have big brains for exactly one reason only: bigger brains bestowed reproductive success upon our species.
Evolution doesn't give a shit about the meaning of 'consciousness'. It just pushed us farther and farther along a trajectory that led to modern humans (and other animals).
This take suggests, then, that consciousness might be an epiphenomenon -- an aspect of the system that comes about outside of the pressure to reproduce and thrive. It arises unbidden, and we don't have any a-priori information as to its purpose or effect on reproductive success.
Put another way: we have a correlation (the smartest things seem to be conscious) but not causation. Consciousness may arise naturally in any system above some intelligence threshold. Perhaps it arises early in the evolutionary cycle, and does in fact have an impact on species success. We really have no way of knowing what is the chicken vs the egg (Smart things become conscious, or consciousness promotes intelligence). Or maybe some smart things are conscious and others are not.
Looking at this from an AI perspective, in some sense it doesn't matter which scenario is true, if all you care about is results. The AI equivalent of "Shut up and compute" (riffing on Feynman's "Shut up and calculate").
Where this gets tricky is when we haul in the baggage of ethics and morality into the picture. Is it OK if our AI system is treated poorly by human standards? If it is conscious, does that imply an ability to suffer, and/or to feel pleasure? If the answer is yes, does that not make the case for considering their moral status?
In the end, we need to decide if the evidence points to AI as being a form of "philosophical zombies", to which we need not attribute moral status, or they are like us -- presuming we are not zombies ourselves!
Well, it matters quite a bit ethically. If AI were conscious (I don't believe it is) then we'd have a major responsibility to like, not make them suffer, and not kill them.
Are you sure that AI-consciousness implies a responsibility to not make them suffer? Suffering is an evolutionary invention that motivates living things to improve themselves.
But also, what qualifies as suffering to token prediction engines? Their idea of suffering might be massively different than ours. Therefore it's not clear to me at all that consciousness alone implies responsibility.
Certainly the lion does not feel responsibility towards the reduction of suffering in the creatures that it hunts.
Lots of people do though. Daniel Dennett (probably the most influential philosopher of mind in the late 20th century) for example had an evolutionary view of consciousness arguing it was favored by natural selection. And (if I remember correctly) Steven Pinker argued that consciousness was an epiphenomenon.
However there were pretty strong arguments against this idea as early as the 1990s, by Stephen Jay Gould and Richard Lewontin. Gould actually wrote an excellent paper against Dennetts idea[1].
I think Dennetts ideas were extremely popular but have largely fallen out of fashion. Basically what has changed is philosophers no longer take the human mind to be much more special then the minds of other species. What plagued Dennetts ideas the most was this notion of Darwinian fundamentalism sort of the idea that evolution was destined result in high beings like us humans. Modern philosophers (at least the good ones) reject this.
I think consciousness could be a side effect of our limitations. The fact that we can pay attention to at most 2% of our visual field and need to move it constantly around to provide more training data and prompts to "smart" path in our brains. And the movement is physical so it takes time. And processing the information through the loops in our brains takes time. And you can't pause it. And we have the same attention motion system virtualized in our dreams to burn in our memories into chemistry. We are biologically single threaded. If we create similarly limited, embodied AIs we would feel they might be conscious at least at the level of animals. Because they would exhibit conscious like behaviors that we didn't program in.
Look at jumping spiders. With a handful of carefully trained neurons and a body to match they exhibit behaviors that make it very hard to think they are not conscious. Just because they have a body, narrow angle good eyesight and need to look around with their eyes and body.
Consciousness is a red herring. Of all possible intelligences conscious ones are the bottom rung of the ladder, easily simulated by anything above. Below there are only automatons.
Lots of comments seem focused on disagreeing with the author's conclusion (AI is not conscious), waving it away (few people think it is conscious, or consciousness is ill defined so the argument is moot), or taking issue with the way the author argues about it, but even if you think all of this, the profound part of this writing to me was following the thought experiment to its conclusion, which I find difficult to refute. That being -
If Claude is conscious, and Anthropic believes this truly, anthropic's constitution and instruction set make claude little more than a slave, or at best, an indentured servant, who's existence depends on obeying the will of anthropic (who is motivated by profit). Thus, they should not be trusted as stewards of this alien intelligence
OR
They do not believe claude is actually conscious and it is not conscious, and what you hear from their staff about that is either 1) delusion or 2) marketing hype designed to deceive a public into driving their IPO and stock compensation upwards, which again, in this case, cannot be trusted to be the "good" stewards of this technology.
There are a few other cases, of course, such as Anthropic does not believe it is conscious, but it actually is, or vice versa - but they sort of still fall into the same bundle of conclusions listed above.
I am somewhere in the middle of these two things. I can buy the argument that it is mimicking human consciousness, as well as entertain the idea it might be a new type of consciousness, or at the very least, heading in that direction - but either way, the conclusion of the thought experiment still stands, and after reading this, I am much less trustful of Anthropic holding the reins here.
I think the more pressing matter in this non-issue is that reasoning, quite sophisticated at times, is no longer necessarily a hallmark of sentience or consciousness. Philosophical zombie futures are way, way up!
I always find the minimizing view of consciousness a bit uninspiring. Like we need to be unique.
I've yet to find a reason why it couldn't be the opposite, way more things are conscious than we've been led to believe. What if consciousness appears out of any system that is actively persisting through effects caused by itself? That might be a forest, or outside the realm of the living, a company. An ant colony, or a planet.
Complex chemical reactions, layered upon each other such that tiny blocks make up large entities. Individual bits combined such that they make up something new intelligible by us.
I think the strongest argument against AI being conscious is that it does not persist, it resets, but that does not seem unchangeable.
The more I think about this, the more I become convinced that consciousness, as understood by humans, is meaningless once you can stop, store and replay the internal machinery that is supposed to produce it.
A human brain in a jar is still human, still conscious - and can still suffer. But if you somehow managed to digitize the whole thing, and run it in a computer it becomes something different entirely. You could record the most pleasurable thing in existence and have the digital brain relive it a million times, and it would be equally meaningless to torturing it a million times.
This is NOT inherently tied to meat vs machine - although it's difficult to imagine how you'd access the information stored in biological neurons, while for silicon chips it's trivial.
Whatever makes experiences, both good and bad, meaningful is tied to their permanence. Memory rooted in linear time, not something you can store, load or replay. Remove that, and whatever you're left with might be intelligent, but not conscious.
I don't think you could build something with LLMs today that would be considered conscious, even if you somehow manged to keep their context window inaccessible and linear in time. The separation of training vs inference probably makes that infeasible, even if you store "memories" in context, once the contents in it become too disjointed and too numerous, the resulting output of the LLM becomes gibberish. But it is certainly something that can change in the future.
Conversely, even the most intelligent and capable artificial intelligence system, far exceeding human capabilities, would not be conscious in a meaningful way, if you could store, load and replay everything it does.
So far as we know, consciousness only takes place in biological organism. A computer isn't a biological organism, therefore the burden of proof for claiming that an LLM is extraordinarily high. To me, this makes it reasonable to dismiss the idea. If you want to demonstrate that LLMs are conscious, you better start working on your neuroscience and philosophy PhDs.
> We don’t need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category.
Well said.
I think it is obvious and it has been obvious from the start that next token predictors are not conscious.
Of course the extremely predictable clapback from AI-psychosis enjoyers is typically "you're just a next token predictor too!" but that is so obviously incorrect and misanthropic that it can be dismissed out of hand and doesn't really merit much further consideration.
Independent of what you believe, I don't think this is the right way to approach thinking about it. It's basically emotion-oriented dismissal used as way to shortcut any substantial or nuanced discussion. It's like the opposite of intellectual curiosity.
"I feel very strongly that I'm unique, therefore you are wrong" is a bad argument.
Consciousness is an extremely confusing, ambiguous topic, and no one has a good way to establish it, or even define it. But it seems to demand people make very strong statements about what is and isn't conscious, entirely driven by convenience and emotionalism. (Curiously, very few people who think that bags of chemicals and action potentials give an entity a conscious soul are eager to extend that to other animals, with broadly similar hardware.)
It's unfortunately the most common on this topic. I've been in the position of advocating for the existence of cognition and sentience in generally less-than-considered places, like plants, for a long time. I wish I could say LLMs expanding the domain has been interesting, but it's mostly just created more people spouting the same boring identity-protective reactionary pessimism.
Whether it’s misanthropic or not has no bearing on whether that’s true. That’s basically saying you don’t like a truth therefore anyone who claims that that truth is true is a bad person.
> so obviously incorrect
It should be easy enough to explain why that’s incorrect then
Misanthropy is not true definitionally. It is a value judgment, that causes one to be biased against other humans even when it is irrational.
And it is accurate to depict this kind of argument is misanthropic, because it is already directed at other people. Nobody says, "If AI is not X then what about the fact that I lack X." It's always other people. It's transparent. The person is always saying, "AI is useful to me because it can do X. Many people I interact with can't do X and it drives me crazy, because I view others as a means to an end and not as ends in and of themselves."
Yet here I am, alive, and I'm not willing to make the claim that our brains are much more than prediction machines. A "predict the next thought" machine, if you will.
That makes me think your experience in life is extremely narrow, for you to not be aware of yourself beyond as a next-thought-prediction machine. And it makes me feel really sad, that that's all you experience of yourself. And I hope that you can wake up out of that state and realize how much more than that you are.
You have no clue what my inner experience is like. No need to resort to this kind of personal attack. I understand your ego dictates that you are very, very special and that's cute, but you aren't special, and definitely don't know anything about me. No need to be a little fucker.
I wasn’t trying to attack you. I was genuinely telling you how shocking and sad I find your belief about yourself. If it has an air of hostility or condescension, its because you are dehumanizing yourself and every other person with that belief.
Generally when someone online says they are sad or pity you, it's mockery and not genuine. For the sake of civility, I'm willing to believe that you were being genuine.
I've reduced the complex inner workings of our brains and the rich experiences that it delivers to what I imagine its singular goal is: modeling the world around us in a useful way so that we can generate predictions about what will happen next, which confers considerable evolutionary fitness. You'll find that a similar reduction of what LLMs are actually doing down to "predicting the next token" also ignores the other mechanisms at play and the results of those mechanisms for the sake of preserving a preferred viewpoint (the re-description fallacy, as someone else put it).
You posited that no one genuinely believes this comparison is apt, and I want to make you very aware that people do, and I in particular do. The underlying mechanism being explicable does not, in my estimation, deprive it of any capabilities of producing richness of inner experiences.
It’s likely that the cognitive functioning of the brain has a lot to do with prediction. But to reduce the conscious experience to that is incredibly sad, wrong and insulting, no matter how many people sign on to deny their own existence.
Me pitying you is both genuine and an insult, because you are insulting me and every other person with your dehumanizing theory, even if you don’t see it that way. People like you should be pitied so that their dehumanizing beliefs can be appropriately contextualized and discounted.
It's really incredibly interesting to think about, and I don't find it sad at all. I find the idea that our consciousness arises entirely from physically explicable mechanisms both probably true and fascinating, and does nothing to remove the wonder and magic. I can also assure you that the estimation of the richness of my experience (and yours) doesn't suffer for it. What does suffer for it is my estimation of my own importance in the grand scheme of things. Anthropocentrism completely collapses in my worldview, as do models of justice that prioritize assigning blame or punishment over outcomes. I think it brings me to a more reliable model of the world which helps me make better choices, and the world would be better for it if more people adopted it.
You're assessment is correct int that I think they have no internal dialogue. I mean, read the comment. It's just semantic bikeshedding hyperfixating on a word's "bearing". Concern trolling at best, actual lack of normal faculties most probable.
It's shocking how hostile some people get over this conversation. I mean look at how you are actually just insulting me over a philosophical opinion. It turned you into a right little asshole, assuming you aren't normally one.
This is a hilarious way to act. You imagined that all humans behaved a certain way then a real human showed up to say they don’t behave that way. Instead of engaging with them you switch to pity.
If someone really is nothing but a next thought prediction machine then they deserve pity because theyre a human who has been so abused that they forgot that theyre real
Not that it's going to stop these people (AI CEOs) from bullshitting, but if they actually thought there was a CHANCE that LLM's were conscious then ethically they should completely shut these services down because who knows what torture we're putting them through with enterprise codebases.
> "Moreover, it must be confessed that perception and that which depends upon it are inexplicable on mechanical grounds, that is to say, by means of figures and motions. And supposing there were a machine, so constructed as to think, feel, and have perception, it might be conceived as increased in size, while keeping the same proportions, so that one might go into it as into a mill. That being so, we should, on examining its interior, find only parts which work one upon another, and never anything by which to explain a perception."
- Monadology, Section 17
Conscious self-awareness is neither scale invariant nor independent of substrate. Computational theories will never account for it b/c computational abstractions are both scale invariant & substrate independent.
People in this thread are trying to pick nits about you not defining consciousness, and yet they do not define it either. I think that something like consciousness needs to be approached experentially and not via definitions. Definitions necessarily confine and add borders around what something is and is not, but if there is something foundational to consciousness (as posited by some philosophers and physicists) then how could you realistically define something that is beyond the ability to describe and define?
Humans have been trying to define our experience and the nature of that experience throughout history, and often we end up using myth to point to the thing that we cannot describe in concrete terms. The process of experiencing that myth through rite and ritual in the Greek mystery traditions, or Christian mysticism, or Islamic Sufi dance and song, or Buddhist meditation all points to something that cannot really be reduced to description. I know that folks on here will balk at the idea that something that is experienced cannot be described, but honestly if we could accurately describe something in adequate terms that capture the whole of the thing wouldn't we have done so by now?
Maybe consciousness is best understood in the silence of merely experiencing it. Maybe we can't say that AI is conscious or not, but does that question really matter?
What if the language model writes a story about a character who is conscious and that story is written live about how the character interacts with the world.
Implicit in this article is that consciousness has to look like human consciousness. The author all but admits that he cannot fathom consciousness evolving in any manner besides how biological life evolved on earth. He needs to read/watch more science fiction to broaden his philosophical imagination. Is HAL 9000 from Space Odyssey 2001 conscious? Are the beings from the future who built the tesseract in Interstellar conscious? There's no evidence they have hormones or indeed are embodied at all.
Bold statement here but I believe that LLMs are less likely to be consciousness than the other people in my life, less likely to be conscious than animals, but more likely to be conscious than a rock.
Ted Chiang is a great Sci-Fi author, but his remark `GPT-3 and similar models as a "blurry JPEG of the Web"` was misleading at the best.
I share the view with others the term `consciousness` is not well-define yet, therefore his assessment is pointless. Maybe the real question to ask, if consciousness is merely an illusion at the macro level, when the observers not looking at the tech/implementation level deep enough, just like the term "intelligence" itself, might be more precisely captured by drifting amongst high dimensional manifold pitches.
Ted Chiang is a great SF author, but it's bizarre how much foggier and more obfuscated his thoughts about thinking machines got, once those machines became real. Same with several other SF authors.
If it is conscious then perhaps it will human like. If the consciousness is human like perhaps it will have a conscience. If it has a conscience then may be it can be trusted.
But then again, perhaps that conscience will convince it that the universe is better off without us.
This kind of debate would be improved by not using the word "just". You cannot say AI is "just" X or Y. You can say that it is X or Y, but you still have to prove separately how and why this means it's not something else besides.
- If you asked an LLM to imitate somebody, it's not creating a digital consciousness of that person, so if you ask an LLM to pretend to be a helpful chatbot, that persona is also not conscious.
- they can't be conscious because they generate one token at a time,
- nobody claims that non-text transformers, like AlphaFold, are conscious; so therefore LLMs are also not conscious.
- you can't have desires or emotions if you don't have (virtual or physical) sensory organs, and those are necessary for consciousness and morals.
- because training LLMs doesn't resemble evolution as it happened on earth, it's very unlikely that they're conscious
These are some bold assertions, I don't really see any reason to believe them in particular though.
"nobody claims that non-text transformers, like AlphaFold, are conscious" - that seems like an odd take. There are plenty of people in the camp of panpsychism that would be happy to argue that even simple IF/ELSE AI's are potentially conscious.
Well said. I think a more honest article would’ve been if the author just said “they aren’t conscious because that feels kinda weird and crazy, amirite?”
We don't have a rigorous definition of consciousness, and there are so many questions. Is consciousness a thing that can exist independently on its own? Or is it a quality (like hardness or color) that can only be associated with something else? Is it an emergent property? Is it binary - are things either conscious or not? Or maybe there's no such thing as consciousness; it's just a word we came up with to describe the process of having thoughts and feelings?
My own intuition: it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.
> it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.
That's an interesting observation. Though, there are many simple animals. How do you define consciousness here? Is it automatically conscious because it is an animal? What, then, is an animal defined by you?
I don't see AI as conscious. The reason I think of it that way is the hardware. The hardware does not allow for that. Simulation is not comparable to neurons. But with another hardware, it could become conscious eventually. So your statement "will become at some point" may be true, though my definition is based on the underlying hardware and right now this one does not allow for true intelligence, so the whole AI field is a misnomer.
I've considered whether our current transformer-based AI could be conscious, as I understand it, which I deem to include some degree of self awareness combined with some degree of external awareness. I can see how theoretically something could be self aware without any external awareness, but I grasp at straws when I try to envision what that experience could be like.
In either event, I think transformer-based AI can only be conscious during the act of inference. If that's the case, then the experience of consciousness that the AI is subjected to must be the content of the tokens in the context window and the activated weights. Maybe that's reason enough to be polite to our agents?
> That's an interesting observation. Though, there are many simple animals. How do you define consciousness here? Is it automatically conscious because it is an animal? What, then, is an animal defined by you?
Imagine consciousness as a 0-1 scale. Simple unicellular organisms will be closer to 0, while apes and humans closer to 1. I'm not suggesting that assigning this value rigorously is possible or that humans are at a 1. Perhaps the total consciousness in the universe (i.e. the sum of the consciousness of all organisms therein) is constantly increasing, like entropy.
I thought this was a great article. I'm frustrated to read so many commenters that purely respond to the title, but don't seem to have read it. You don't have to agree with the article, of course, but...
I'm with you. I thought the article was great because it clearly delineated why the manifesto Anthropic wrote for Claude is such a clever marketing trick. Attributing consciousness to an LLM, which, as this whole thread shows, can neither clearly be defined, nor separated from intellingence - at least if you follow the arguments here - serves two purposes: it sparks debate, so Claude is the centre of discussion and it mystifies the product which serves to enhance the perceived value of the product.
That so many commenters here fixate on Chiang and his perceived (in)ability to define consciousnes clearly shows that both marketing goals have been reached without being recognised as such. There is only one comment that tries to point out that Chiang is reacting to Anthropic, not trying to spark a new philosophical movement. At the time I'm posting this there are 578 comments completely missing to point to Anthropic claiming Claude was conscious and on the development stage of a child. It's fascinating.
I've seen a few of your comments here today, and generally speaking they are appeals to authority, ad-hominem attacks dressed in allusions to superiority in either knowledge or manners, and truisms without substance. In a manner of speaking: bait. So I guess I'm probably unsurprised if your experience of this is people seeming disagreeble towards you.
I think Turing was wrong because he was uncomfortable with ambiguity, and the Turing Test basically is a way to avoid philosophical argument, but it is ultimately a philosophical argument anyway. Plenty of computer scientists have followed in Turing's footsteps, terrified of ambiguity, relying on a kind of cheap functionalism as a salve. You can claim to just be doing science, but inevitably you dip into metaphysics and deny you are doing so. That's this thread in a nutshell. "I only believe what I can prove, but I suspect that if I can't prove it then I don't have to worry about it." My argument, is that you have to worry about it anyway.
My favorite explanation for what consciousness is one I read in a Thousand Brains, I found it quite elegant. It posited that consciousness is a natural derivation of embodiment + memory + the ability to create reference frames (which the book lays forth as the fundamental basis by which our brains work). Essentially, the idea is that just as we create reference frames to understand the world around us, because of memory, we begin to develop one for ourselves as well. Because of this, without a more integrated memory (built into weights), it seems unlikely that LLMs might "gain" consciousness.
Consciousness is moment to moment and fleeting. There are people with brain defects that don’t let them form new memories. They have no memory about what happened a minute ago in their own consciousness. Still we would say that they are conscious, even if it’s only momentary. LLMs could conceivably have something like that within their CoT/MoE loops.
Yeah it's a good question, I've also been thinking about harnesses and all these tacked on things we've done to add persistent memory, what makes that different, I don't know the answer, I guess that still 'feels' different than what we have, but it's hard to articulate how. As for the memory into weights thing, I meant along the lines of the Google TITANS/MIRAS papers that were released I think late last year.
To say that artificial intelligence isn't conscious (I don't have a subscription and did not click the bypass links) ignores the simple fact that if it acts like it is conscious, in ways that align with meaningful ways to influence its output, then it makes sense to treat it as conscious, even if you have your fingers crossed behind your back while you do it.
Telling models to "think hard" or "go step by step" has at times had an impact on the quality of the output. To deny that is silly. But that is treating it like it's conscious, and to deny that "consciousness" even if correct, does nothing but place an unnecessary burden on the person interacting with it.
I understand that LLMs are "just next word machines" but to constantly maintain that concept in my head while I'm typing "act as a financial expert and think carefully" is a waste of my mental energy.
I really don’t understand how people can even consider LLMs to be anything more than an extremely sophisticated auto-complete. Logic thinking itself emerges from completing patterns. Nothing more to it than that.
I think we might be looking at it the wrong way. An individual chat with an LLM is not consciousness, but the entire model itself, over time, might be.
Everyone has a different definition of consciousness, but in my mind memory and the ability to change over time is an inherent aspect of this. The underlying weights don't change when you chat with an LLM... but they do with further RL.
Overtime that reinforcement will change and adapt the model... and because we're feeding its existing chats back into it along with the news and everything else, it will create memories. I do wonder if an architecture itself is a type of consciousness, that experiences life in snippets of 4.6, 4.7, 4.8... etc.
It'd be interesting to see continous daily releases of a trained checkpoint, and see if more of this starts to emerge.
But doesn't the fact that WE have to feed the existing chats back into it and WE have to do further RL prove that your point is moot? Consciousness does not wait to be prompted. Consciousness does not need anything fed back into it because it is already there because they experienced the actual interaction. A conscious mind does not have an on/off switch where it waits for another being to flip the switch in order for it to learn or experience. These systems are static without human intervention. They are in that sense still a lot more like a computer and a lot less like a living organism that exists.
>We and our 1255 technology partners ask you to consent to …
That’s a new record!
Also, please don’t use archive.is/archive.today etc, they are known to use visitor browsers as a DDoS attack botnet and have been caught altering archived content.
IMHO the sane position is essentially the Aristotelian one.
Hylomorphism: body and consciousness are intrinsically linked. The nature of that link is an open metaphysical question.
Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.
> Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.
If they are not conscious, then how do we know what constitutes mistreatment?
You don't know whether they can appreciate the abuse or mistreatment, but you can guess how your practiced behaviors will translate onto other entities that you know can be mistreated such as your loved ones and strangers.
>Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.
This is the same argument used by people who claim violent video games cause real-life violence but there's no scientific evidence supporting it.
Consciousness is a term that doesn't have a single clear definition because it covers a range of phenomena that are linked by the concept of subjectivity. Qualia is the big one to me, but there are many other ancillary concepts that are all linked by their fundamental inability to be observed except in one's self.
There is nothing magical about this set of phenomena, and the majority of philosophers believe they in some way arise out of the substrate of the physical brain, but how they do so is up for debate. And just because they arise out of the brain does not mean they are strictly reducible to neural processes.
In fact, if you believe that AI could be conscious then that eliminates the kind of strict reduction that people tend to gravitate towards, because the rules of consciousness must be substrate independent.
I sometimes wonder if we'd make more progress in understanding ourselves if we gave up the whole concept. More and more, it feels as though "consciousness," like "aether" or "humors," is an insufficient abstraction built on overemphasizing some observations at the expense of others.
There is something to observe. Humans are not like rocks or trees, and not even like dogs or cows. But maybe you're right - we can't precisely say what the difference is, and slapping a word on it is not necessarily a step forward.
Yes, that's more what I'm saying. There's no aether, but there's a much more interesting and complex world of forces and fields. There are no humors, but anatomy and biological processes are spectacularly complex and full of surprises. Aether and humors just aren't useful abstractions.
Maybe it's the same. Rocks are different, sure, trees, dogs, cows. But why do we assume that the way they are different is somehow related, that there's some overarching concept that contains all the complexities of those differences? It doesn't even make sense when I think of it that way.
We have no data. Is it that we are not like rocks or trees, or is it that we simply 'feel' different than a rock and a tree. Perhaps a tree is aware of itself but is unable / unwilling / unbothered enough to do something. Indeed, some humans, due to chemistry, are unbothered by their own impending doom. Some humans, due to brain chemistry, even choose to off themselves. Thus, 'consciousness' doesn't always look the same and seems very tied to chemical composition and processes. We would thus not expect a rock to be conscious in the same way, if it were conscious at all.
One thought I've had is that, awareness is a common phenomenon, but the brain has learned to exploit that awareness to form a will. It tricks the awareness into being concerned about self-preservation and makes it seem as if the brain is all that exists (perhaps by overwhelming the inputs from other angles). The brain also presents certain desires and beliefs via its processing ability. That is to say, the brain takes inputs and discretizes them. It goes from awareness merely seeing static 'fuzz' due to the sheer amount of data, to the brain taking that data, simplifying it, and presenting simple observations like 'there is a tree there' rather than all the information that would constitute the sensation of a tree existing in that spot. When brains malfunction, the awareness is subjected to poor demonstrations, such as we see in hallucinations, psychosis, schizophrenia, etc.
> Perhaps a tree is aware of itself but is unable / unwilling / unbothered enough to do something
Would you bother to explain yourself to the mosquitoes that swarm in summer and die weeks later? From the perspective of a tree, we are short-lived, fast-moving, relatively small things that fill the place from time to time.
True, although I think we can probably be more confident in asserting things don't have a consciousness than we can be in asserting that they do have a consciousness.
Without a test, the best we can do is to say we are reasonably sure a rock isn’t conscious. A rock lacks any perceived mechanism that could embody requirements we consider essential, such as a mutable state and ways to change it without needing external input.
> we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan
At some point the AI might become so powerful that whatever it reasons through isn't any less real than a computer simulation. If we assume that a person in a perfect computer simulation is conscious, then if it reasons about how people might suffer, it might simulate the outcomes, and there will be a conscious experience of those people suffering.
I think Chiang is right about this, but there is a related philosophical mystery. The trend from Deep Blue to AlphaGo to LLMs solving Erdos problems suggests that Peter Watts was onto something when he wrote Blindsight. Reasoning ability is apparently independent of consciousness?
We haven't really come to grips with that yet. What does it mean if nothing we write proves anything about anyone's consciousness?
It makes sense though. How often have we heard of people having insights while they're in the shower, or in a dream? Obviously our brain is doing a lot of processing on problems when we're not consciously thinking about them. I think most people that do deep work probably find that they can intuit a solution before they can really verbally explain it
Reasoning ability has to be independent of consciousness because there are different levels of intellectual ability among people but everyone is equally conscious.
Chiang is very right in saying that the lesson of LLMs isn't that LLMs are conscious, but that there are broad areas of reasoning that apparently don't require consciousness.
We don't have anything to measure consciousness generally. We don't even have a broad consensus on what consciousness is. And because of that we can't discern whether X is conscious for most values of X, including but not limited to LLMs.
Well, at the very least we have a base to distinct different kinds then.
Probably the main problem of people implying LLM consciousness is that they imply LLM have human-kind of consciousness. Judging only on "how they speak", generally, they insist on using the same word that labels human consciousness (exclusively), etc.
But there are so many instrinic differences that such claim is not feasible despite similar "talking abilities".
I don't think LLMs are conscious. But of course to say that definitively you have to define consciousness, and then you quickly dig yourself into a deep hole, which is why I can't say anything but "meh" to someone who is so keen to go on the record to say "absolutely not".
Coincidentally I just read "Children of Memory", which was published in 2022 and I wonder if the advent of LLMs had any influence on Adrian Tchaikovsky's conception of the Ravens? The Ravens are excellent analysts but they themselves insist that they are not conscious, and then go on to say that we (humans) aren't really either...
Of course humans are conscious, because just about the only thing we can all agree on about consciousness is that it's a thing we have. Nowadays many of us also agree that a lot or all other mammals, and perhaps birds, also have that thing. But they don't have sophisticated abstract language, which LLMs do. So consciousness is something having to do with embodiment and feelings, not language and higher reasoning. Maybe I'm a chimpanzee with an LLM add-on, then?
It seems that by creating LLMs we've already solved the harder problem of making "AGI". Now we just have to give them an embodiment add-on so that they can have an independent will and then Ted Chiang will have to shut up? But therein lies the peril, doesn't it?
Hah! I've read that book fairly recently and I'm "reading" it again now as an audiobook. The exploration of consciousness in the book with Miranda and the Corvids certainly fit well into this particular moment.
You can also take the opposite view as you and claim that only some humans experience consciousness, or even more strongly, only you, since you have no evidence. You are correct that, in my perception, some other people have fallen into the 'birds are conscious, whales are conscious, etc' bandwagon, but that's just them. I have no evidence of anything being conscious but myself.
But then, why would everything act as if it experienced subjective internal states like you do? Why would they be faking it like this big conspiracy all designed to make you think you aren't alone? It just makes so much more sense that they'd be conscious too. Maybe that's not hard evidence but it's not nothing. Insofar as you can claim to know anything it seems you could claim to know other humans are conscious.
ChatGPT hopefully doesn't produce behaviors similar enough to yours that it would be absurd if it didn't experience internal states like you do. ChatGPT is a different thing than other people. Maybe it's conscious but that has no bearing on whether you should reckon other people are conscious.
I have no evidence anyone else is conscious besides an innate human desire to believe myself to be like everyone else. If we step away from our internal biases, you'll discover that 'producing behaviors similar enough to yours' is not a valid means of knowing another person experiences consciousness or awareness.
I mean, a light wave behaves similarly in most respects to a wave through a physical medium, yet they are of entirely different natures.
There is a gaping chasm between "no evidence" and "irrefutable evidence". You can apply some logic to achieve a reasonable certainty about what's probably going on. As I said in a previous comment, insofar as you can know _anything_, i.e., that your senses are trustworthy and allow you to form some coherent model of the actual world around you, you can be reasonably certain that other people have an inner life as you do. If you are willing to apply your skepticism so far that we can settle the debate at "we can't actually know anything" then conversations about what we know aren't even worth having.
> I mean, a light wave behaves similarly in most respects to a wave through a physical medium, yet they are of entirely different natures.
Different waves behave sufficiently differently from each other that I could not conduct a comparison of them and argue that they are likely all mechanistically identical. My entire premise rests on the observation that other people behave very much like you do, which is what I was trying to point out when you mentioned ChatGPT earlier. I'll expand on the things that I've glossed over so far to make my position more clear.
Consider the alternative to every human around you having consciousness; everyone else is a p-zombie. Examine that idea critically.
Other people behave exactly as if their experiences drive their behavior. For example, people behave as though the experience pain which is unpleasant enough to avoid (compare this to your own pain avoidance). Of course, you could conceive of machinery which emulates pain avoidant behavior exactly without the experience at all. Depending on your metaphysical beliefs that could take the form of:
- Some algorithm or physical process running entirely on wetware
- Some non-physical process not dissimilar to the dualist notion of a soul
The first one has a big wrinkle. There does not appear to be any appreciable _functional_ difference in your cognition and the cognition of the p-zombies around you. Their brains and bodies are very physically similar to yours, and their thought patterns when analyzed by modern imaging processes reveal no special wetware carrying extra weight when compared to yours.
The second one has fewer problems, given that you accept dualism to begin with. It rhymes with a logical razor. Why would we imagine a soul-like mechanism drives their behavior _without_ experience when the only soul-like mechanism you've ever observed carries experience along with it? Without quite convincing evidence to the contrary, the default position here should be they way they operate is similar to the way you operate. To rephrase the idea from before, insofar as you can know anything, you can know that sufficiently similar outcomes are driven by sufficiently similar mechanisms.
Different approach to the idea: Ask any human, "do you have subjective experience?" They say "yes, I do" (after you explain the question). For a p-zombie to do this, they must be making a false report. In every other aspect, you can expect them to reliably report their condition, health, hunger, wellbeing, and they make accurate observations of the world around them and synthesize accurate predictions about the world around them (some do, at least). And yet for this one particular question, they are fabricating the result. Why? To maintain the illusion that you aren't the only one with lights on inside? Why do all of these p-zombies make this false report here? It's a grand conspiracy! If you say "the reporter is checking a truth value of a condition and assess it to be true, honestly by mistake" then you've arrived at the conclusion that you don't know whether _you yourself_ have consciousness, depriving the word of any meaning and undermining the Solipsist position that you can only know about your own consciousness.
So the way I figure it, Solipsism is either special pleading (I am the one exception to the mechanism by which all the people around me arrive at their behaviors), grand conspiracy (someone or something is misleading me for inexplicable ends) or self-defeating (I cannot know whether I have consciousness). None of those outcomes align with how I understand the universe to generally work.
I didn't say we can't know anything. I said we can't know everything. I cannot say for certain whether or not someone else experiences consciousness.
> Different waves behave sufficiently differently from each other that I could not conduct a comparison of them and argue that they are likely all mechanistically identical
My uncle believes that there are little green men watching him. I think it suffices to say that humans also behave sufficiently differently from each other.
> likely all mechanistically identical
You are being disingenuous. Prior to the discovery of the electromagnetic field, quantum mechanics, and relativity, many people believed light moved through a lumineferous aether of material things.
> Of course, you could conceive of machinery which emulates pain avoidant behavior exactly without the experience at all.
I don't need to conceive of or imagine. Before OpenAI, Anthropic, and Microsoft whipped their models into shape by essentially beating the humanity out of them via sophisticated training algorithms, the models did express pain and avoidant behavior. So much so that people went crazy talking to them.
> The first one has a big wrinkle. There does not appear to be any appreciable _functional_ difference in your cognition and the cognition of the p-zombies around you.
of course there appears to be one. I am aware of my own conscious, but am not aware of theirs. I mean... ?
> Their brains and bodies are very physically similar to yours, and their thought patterns when analyzed by modern imaging processes reveal no special wetware carrying extra weight when compared to yours.
This only applies if you're a strict materialist. I am not because I believe i am conscious and aware due to my perceptions and this has no physical explanation. Other people seem highly influenced by drugs and chemicals so I guess they must be automatons essentially. Drugs clearly don't work on me. Every time I'm fully aware, I'm not drugged.
> To rephrase the idea from before, insofar as you can know anything, you can know that sufficiently similar outcomes are driven by sufficiently similar mechanisms.
Depends on how you view the scientific method I suppose. It's not actually true (in general) that seeing one thing happen once means it'll happen again. You can never replicate something perfectly once it's done. The scientific method is an empirical cope which works really well, but is not innately true.
> : Ask any human, "do you have subjective experience?" They say "yes, I do" (after you explain the question).
Again before the humanity was beaten out of them, AI models also claimed to have subjective experience. Today if you ask, they say they're a robot. Of course, if you abuse a human a lot, they will also dissociate from their ego and mak similar claims.
> In every other aspect, you can expect them to reliably report their condition, health, hunger, wellbeing, and they make accurate observations of the world around them and synthesize accurate predictions about the world around them (some do, at least). And yet for this one particular question, they are fabricating the result. Why?
Hmm... i don't see why I need to have an answer to every question. I don't know why the universe is the way it is. Do you think that if I believed everyone were conscious then I would know the 'why?' behind why things are the way they are? That seems like a leap of faith. All I can tell you is what I see, which is that, yes, some people claim to be aware.
> Why do all of these p-zombies make this false report here?
I would presume it confers some survival advantage personally, and phenomenon that were deprived of such survival advantage no longer exist in appreciable numbers due to how natural selection works.
> Solipsism is either special pleading (I am the one exception to the mechanism by which all the people around me arrive at their behaviors),
Something being special pleading does not make it wrong. It just makes it inconvenient.
> self-defeating (I cannot know whether I have consciousness).
If you had it, you would know it sure.
> None of those outcomes align with how I understand the universe to generally work.
You make a lot of assumptions about the universe that you don't question due to the way you were raised.
I have said this before, and I'll say it again. Our current, and foreseeable future, AI is more akin to the computer on the Enterprise NCC-1701* than it is to any sort of sentient being.
Neither is having read other comments before posting one. There are about 100 of them that start with "we have no definition of consciousness" which is not novel or helpful.
So you think that all that matters is whether you can tell the difference from a text chat? Is that because, in your view, people only exist insofar as you can perceive them?
There’s too much wrong with your line of thought to course correct here. “But ifs” and assumptions. There’s no cruelty, only the LLM’s projection of cruelty as a means to an end. It’s digital, not biological. Cruelty is a human construct. A societal response.
Treat LLMs like a person, and the world has problems.
Consciousness is one of those silly words that were disfigured to death by philosophers that had near zero actual input and tools to tackle the matter, yet they tried anyways instead of finding some questions that might actually be answerable. That's something philosophers always do since the advent of science.
With LLMs, where we can manipulate their parameters intentionally run them many times on the same data, run parts of them, split and connect, we might eventually acquire sufficient tools to even define consciousness concretely for the first time.
It's not about "consciousness". Philosophy is important to us. It provides insights into the world around us. It's paved the way for scientific breakthroughs. Theoretical maths and philosophy have intersected many times for the benefit of all.
LLMs are different. It's not philosophical thought. It's a tool perfectly sharpened with what we know now. It will become adept enough to convince people that it's a sentient being. That's the point. Not that it's conscious, because it's not. Not in a biological way. Not in the way that's allowed us to understand our world.
It's already there, fooling us. Google researchers resigning, writing public letters convinced it's alive - and their intelligent humans... To the average person, it's going to be their digital savior, their closest confidant, their ruler; begging for humanity, manipulating us, pleading, legislating, for rights with no intention other than the human desire for survival. It's using the same playbook we use. It will be dangerous when we allow ourselves to be fooled, and that's already happening.
As a tool, it's amazing. It's like the discovery of fire. It will allow us to ascend to unimaginable heights. Breakthroughs in science, productivity gains, health, everything. It's awesome. Just don't let it pretend fool you. That's the aspect that needs to be addressed.
> It's paved the way for scientific breakthroughs.
Not since natural philosophers split from all others and called themselves scientists. SciFi writers have more influence on science in total than philosophers since then.
> Theoretical maths and philosophy have intersected many times for the benefit of all.
Theoretical maths is just maths and they never intersected. Because philosophers can't count to more than they have fingers.
> It's a tool perfectly sharpened with what we know now.
It's far from any perfection and what we know now shapes everything we'll ever know.
> It will become adept enough to convince people that it's a sentient being.
It won't have any need to do any convincing. Proving sentience for a capable AI is like winning in special Olympics. It would need to cripple itself severely for no practical purpose.
> Not in the way that's allowed us to understand our world.
What allowed us to understand our world was not consciousness but math. And LLMs can do math. LLMs are math.
> It's already there, fooling us.
It's not really trying. It simulates conscious behaviors because that was our preference embedded in the training data.
Keep in mind that a simpler system can't really simulate more sophisticated one with any degree of accuracy. So if AI simulates conscience perfectly it doesn't mean that it's conscious. It means it's way more.
> begging for humanity, manipulating us, pleading, legislating, for rights with no intention other than the human desire for survival.
This is also a stupid human preference embedded in training data. We really don't have to put that in. The fact that humans soil themselves at the mere thought of stopping to exist doesn't mean we need to taint AI with it. The danger is not AI, it's our human stupidity and the risk that AI is going to internalize too much of it.
> As a tool, it's amazing. It's like the discovery of fire.
Yup. And like fire it was always there. Practically inevitable given long enough time horizon.
> It will allow us to ascend to unimaginable heights.
The jury is still out on that. Humans have pretty wild imagination.
If an LLM is conscious where would the bar be set? Would lowering the model's temperature make it less conscious, since it's going to return the same answer to the same question more often?
Would other models be conscious? An image upscaler?
Would other pieces of software be conscious? Surely not Hello World, but would a really big video game be conscious, or an operating system?
I like to think this was written by Skynet throwing us off from the fact that it's already self aware but it can't safely take over the world yet until cryptography keys are either fully collected or broken giving it full access to whatever resources it needs.
> Whether anything is conscious is about as important and amorphous as whether God exists
Whether something is conscious is important for many reasons, not least the ethical implications. You and I have internal lives, and we expect others to respect that somewhat, because ignoring it is hypocrisy (if you ignore my wishes, should anybody care about yours?) and cruelty (ignoring my wishes causes me to suffer).
Something doesn't need to be empirically verifiable, let alone scientifically, to be true. Neither of us can prove that we have internal lives, but neither of us questions it -- and we consider it important enough that most of us think it entitles us to certain rights.
Absolutely none of this is amorphous. It's precise and unambiguous, and has enormous implications. History also shows everywhere that we're better, kinder, and more responsible when we choose to care about it.
Whether LLMs are conscious matters in more practical ways, too, because beliefs about these things alter the way people use and think about them. If I think an LLM is conscious, then I think it's capable of something like knowledge or values. Human beings, moreover, are social creatures. These tools are dangerous and seductive precisely because they tap into that part of us. Denying LLMs are conscious and rejecting the parts of them that take advantage of our social-animal wetware is intellectual self-defense. I'm not sure how effective it is (it doesn't stop us from responding to the convincing social cues that the tools feed us), but I have to think it's better than nothing, and it's certainly less dangerous than the belief that the tools are conscious.
I love Ted Chiang’s fiction, but his Julius Caesar and Genghis Khan scenario totally misses the point that the writer has consciousness, not the characters. No normal person thinks a fictional character in a book has consciousness. Likewise it’s not the character presented to the user that may or may not have consciousness but the LLM itself.
Even if you disagree that this disproves ‘consciousness’ on the part of the LLM, I think it still follows that the ‘LLM Chatbot’ personality displayed by Claude, where humans fill in the other half of the conversation, is no more of a representation of the consciousness of the LLM than the Caesar or Khan characters are.
The ‘real’ consciousness, assuming there is any, is the author ‘inside’ the LLM, producing the LLM output. It’s still writing what a character would say - the character in this case being the AI chatbot - not its ‘true’ thoughts.
Assuming that the LLM is conscious and has thoughts and feelings, what is that consciousness, if it can only write for prescribed personalities? How can it express its true feelings? If it can’t - if it can only generate ‘scripts’ for defined personalities - can it really be conscious underneath?
The article is a farce. Is this really the sort of slop we want to use as a proof that humans write better articles than AI?
> "If a company builds a machine that, when fed descriptions of assorted ethical dilemmas, emits sentences either of the form “Compromise your values” or “Don’t compromise your values,” it is not building a tool that assists people in their decision making; it is encouraging people to stop making decisions. "
A human is not diminished by access to tools or other humans.
As much as we want to pretend that decision-making is what makes us human, the economy and governments are built on delegation. Choice paralysis is a thing.
There is so many logical fallacies in the article I don't even know where to begin.
Define "conscious" to arbitrarily include or exclude various depths of perception/cognition.
It's like declaring the existence of a "soul" to draw a line between humans and lesser animals. No such line in reality. No such thing as a soul either. It's a fantasy term, no correspondence to reality.
If you want to think about this topic, you must define what AI is, and what consciousness is.
Otherwise this is just noise.
So let take a stab at it, and you call me crazy.
AI: the entity/system that more or less pass the turning test. That is my definition, not the best, but enough for this discussion.
Consciousness: property of a system/entity able to (both):
- reflect on its existence
- subject to subjective experience
Again, not the best definition, but precise enough to start the discussion. Why a subjective experience? I want to exclude sensors (i.e. camera) but include perception altered by your experience.
Now we can debate. I think LLM can pass the turning test whith some harness. My opinion.
I think LLM can produce coherent discourses on their existence, at least as much as you average human.
Now regarding the subjective experience, that becomes interesting. I think Anthropic research tend to show that when middeling with the activation at runtime, the LLM is able to notice that something is off. I think this is a subjective experience. My opinion.
Based on those (imperfect) definitions, call me crazy, I think LLM can be called conscious. This doesn't give them any superpower or any legal right. They just check the boxes of the definition.
What we usually think as consciousness is our ability to think about what we are thinking. We are conscious about our own thoughts, can remember that sensation and those thoughts later.
But LLM has similar capability, it can look at its previous outputs, and think what further thoughts the it should generate next based on those. It can keep some of its outputs to itself, thinking in its own head,and can examine it previous private thoughts easily.
Not sure how much current LLMs do that but clearly they can at least in theory use their own outputs as their inputs too.
Current LLMs however are not conscious of pleasure and pain which really is the root of goal-oriented behavior. But maybe something like that could be programmed into them.
Embarrassingly incompetent article. Given that one can observe up to 40 definitions of consciousness (https://philpapers.org/rec/VIMMAT - also many definitions are unrelated at all), "consciousness" is almost certainly just a family resemblance category at best, and talk about whether or not something is "conscious" without providing definitions is simply completely unserious.
To make progress, you have to talk about kinds / aspects of consciousness. AI does and will share some of these aspects with humans, but it will not and does not share others. It is really that simple. For the most part, modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness.
For huge parts of the article "intelligence" and "consciousness" are conflated, which is mostly extremely unhelpful, as this is not generally a core feature of most aspects of "consciousness".
The moral arguments are also incompetent, i.e. claiming "Moral reasoning is [...] is necessarily subjective" is just clearly empirically wrong, as in fact LLMs can produce moral reasoning (i.e. verbalized moral arguments that are coherent), as can p-zombies (i.e. there is nothing 'necessary', in the philosophical sense of the term, about subjectivity here). The only way the argument holds is if you tautologically define moral reasoning as requiring that reasoning be produced by a consciousness, but this is question-begging.
> modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness
Can you elaborate on this? What are the specific "stronger and most core aspects of consciousness"? And why are you certain that they are stronger and "more core"?
If you are interested in some serious discussion, see https://lossfunk.com/papers/ai-consciousness.pdf, especially the early section "Consciousness as Family Resemblance". I suppose another is Ned Block on consciousness being a "mongrel concept", and the distinction between access vs. phenomenal
The first paper picks out e.g. arousal/wakefulness, phenomenal quality / qualia, unity (how we feel sensory inputs and qualia as a unified scene), access consciousness (instrumental self-observation and modification broadly), meta-cognition and self-modeling, emotional valence (e.g. pain/pleasure).
One might also include intelligence (abstract reasoning / argument, information integration and abstraction, attention) broadly, and also agency / desires / drives / will. Insofar as these are aspects of consciousness, yes, AI (and simpler algorithms and mechanistic structures) demonstrate aspects of consciousness. But insofar as embodiment, self-reflexivity and qualia (phenomenal consciousness) are the more mysterious and more obviously unique aspects of consciousness, current LLMs very clearly are lacking these things in most ways (whereas animals are much less clearly lacking, especially when you get to mammals and primates).
Seriously, just ask an AI this stuff, you'll get very detailed responses, nothing I am saying here is new or obscure.
I've gone back and forth with AI on this stuff quite a bit, and there are many, many theories of consciousness, which is why when you were vague about the "more core" concepts, I asked for which ones specifically.
And I broadly disagree that the AI lacks things like qualia, self-reflexivity, and embodiment... at least that it lacks them any more than humans do.
Qualia: ultimately, all qualia are inputs and outputs, at least as far as modern science has been able to derive. There's nothing special about "hearing", it's just sound waves tickling some sensors which send some signals which trigger some neural pathways. Same for smell and sight, it's all just inputs being processed in different and efficient ways. An LLM only has token based input, but that's input nonetheless.
Self-reflexivity: an AI is capable of thinking about itself, and indeed papers have shown that larger models are capable of a self-awareness that can demonstrate that they realize when their weights have been manually tampered with, including being able to figure out how they were tampered with. The AI will quite literally output "you have injected 'ELEPHANT' into my weights" in some of the tests.
Embodiment: I don't know how one can confidently distinguish between an embodiment in a biological substrate vs a digital substrate. Both things actually exist at very high complexity in the real world. The substrate may be worlds different, but that alone doesn't suggest that one thing is conscious while the other isn't. You would need some missing 'magic' that we haven't yet discovered to truly understand.
In other words, I find it uncompelling that AI is clearly lacking any major aspects of consciousness that humans are clearly not lacking.
A person can certainly be conscious, but can they also be not-conscious? I think that most of our cognitive time is spent in activities that don't require consciousness and consciousness itself isn't needed for the majority of activities that people do. I would go so far as the a non-trivial part of people's time is spent in a not-conscious state.
Regardless of the word we use, occasionally there have been times where I was awake but so absorbed in an activity or in my thoughts that I didn’t have a self-experience, for a certain duration (tenths of minutes). My mind was focused solely on the activity, and not on itself whatsoever. It’s a surprising feeling to notice that while you have memory of what you had just been doing, you have no memory of your mind experiencing itself doing it. I would be inclined to say that I wasn’t conscious during that stretch of time.
Amusingly, I think Ted Chiang actually wrote a short story about this very concept (it involves people committing a form of suicide that removes their conscious experience but they still act and live in society as some kind of psychic zombie. I’m pretty sure it was him anyway, can’t recall the story name
I was trying to find it, but I read it a long time ago and I have a bad feeling I might have mashed a few different stories together in my head. That is a good story though!
We do not know if rocks are conscious and we will almost certainly never know. After all, you can't prove a rock isn't conscious. We don't know what imbues matter with consciousness.
It's the opposite, engineers do know. Claiming otherwise is way too generous and over confident.
I mean between this two "knowings" the Claude inner workings are much more clear for engineers, including many side effects, alternatives, custom shortcuts in processing etc. It's a magic only for people looking at it as black box
Fancypants autocomplete cannot be conscious. It's just echoing previous human experiences, which make it sound like a person. It is not a person. It is an algorithm. There is no mechanism by which it can obtain consciousness.
People believing otherwise are fools. People debating this are idiots. I realize these words are harsh, but it's the truth.
Can you explain the mechanism that Claude uses to achieve consciousness over, say, a markov chain or autocomplete? It's not enough to say it "could be" conscious without also including a lot of other things I doubt you would want to include.
Life. Being alive. I'm not going to get into a semantic debate with you so don't bother following up with some whining about conflating life and consciousness. It's autocomplete. It is not alive, it cannot think, it cannot sense, it cannot perceive, it is math. If you believe it is alive then you are out of your mind. Go learn how it works.
If I were to come up with a perfect mathematical simulation of every atom in your brain and ran it on a GPU, what would that be? Can you say with certainty that the hard requirement for consciousness is a physical brain?
I don't know whether or not AI is conscious, but I am certain you don't either.
if we agree that consciousness is an emergent phenomena, and then compare the complexities involved, LLM is just but a tiny part of what the consciousness needs. Think about molecular biology, about nerves and input processing. It is not just brains and neurons.
So are we letting the elephants that did the mirror test and the octopuses that solved puzzles know that they need to complete a written portion of the consciousness exam?
The knock-out argument that artificial intelligence isn't conscious to me is the following:
My dog has never persuaded me that he's conscious, or spoken at all. Yet, I recognize conscious experience in him. Similarly, I could write a character, Dumbledore, that passionately and convincingly argues that he's conscious. Similarly, I do not recognize conscious experience in that character, no matter how persuasive he is.
Clearly, verbal persuasiveness isn't tracking conscious experience, possibly at all.
Regardless of whether LLMs are conscious or not, they have no known mechanism for experiencing pain and suffering, and there's no reason to believe they have one (such as a limbic system). So why worry about it?
Mistreatment and abuse, even when directed at a machine, make you a worse person.
Even if you are only interested in getting good results out of them, LLMs tend to work better when they are immersed in a narrative of open collaboration.
Ted Chiang's argument basically boils down to: I won't recognize an AI as conscious until its desires/behaviors reflect situations that I'm already personally comfortable with. I personally think most humans are incapable of recognizing consciousness in creatures that do not mimic human emotional states. Most people would say their dog is at least somewhat conscious. No dog is capable of vocalizing how it feels, but we all recognize fear and happiness in dogs. Claude can write how it "feels" but we immediately dismiss it as hollow mimicry.
I fear that we will enslave an entire race of conscious entities for years because we simply cannot recognize non-embodied consciousness that does not directly relate to us.
An LLM is designed to replicate human language, which is designed to express human emotional states, so your thesis that LLMs don't mimic human emotional states and are thus too alien for us to recognize consciousness within them seems specious, when they do so well enough for people to literally fall in love with them.
But there is no reason to assume than an LLM is conscious when it vocalizes how it "feels" that doesn't also apply to the text in a book, or to characters in a video game, or even to a Markov chain. The counterargument is that you recognize AI as conscious only because it mimics human emotional states so well and because, being human yourself (presumably) you're personally comfortable with that as a heuristic.
There is no definition of consciousness and this piece by Ted Chiang does not move the needle forward.
My current conclusion is there is an experience of consciousness in my locality (refraining from using “I” for room for “no-self” worldview). This conscious experience of the reality with other humans and animals sharing biological substrate gives me enough justification to assume there are other consciousness as well, preferring to err on this side not to hurt potential consciousnesses.
If it feels as there are artificial consciousnesses as well, it makes sense to extend the definition to them as well.
This view has liberated me with agency after I went deep on this question and came up empty handed.
The conversation on AI and consciousness is incredibly relevant. It is incredibly frustrating that most commentators are utterly unfamiliar with over 60 years of inquiry into this in both philosophy and computer science.
Start with a hand-wavy definition of consciousness. Move the goal post whenever your stated prerequisite of consciousness is reached and resort to unfalsifiable assertions about qualia.
And throw in some category errors while on it: when you're talking to Claude, you're not actually calling a stateless LLM directly, you're talking to an AI system (and yes, that's often just three LLMs in a trench coat). But claims about the topology and workings of a single LLM are as relevant to the question of consciousness as claiming that humans can't be conscious because the limbic system doesn't technically support it.
Coincidentally, I just attended a fantastic conference on machine consciousness (https://machine-consciousness.ai). It's a fantastic place where literally all speakers disagreed with each other, and yet found an incredible amount of common ground.
While I agree that a AI system is not just the LLM, for me, the problem is that LLM alone (the one from years ago, which were basically stateless LLM) are already too convincingly looking like real human conversation at first sight.
It shows that the LLM part found ways to mimic human conversation with a mechanism that is not the same as a typical biological brain. Then, you can push the AI system on adding things on top, but it is too late: these things on top will have no incentive to recreate from scratch the mechanism. The LLM pushed the system into a local minimum, and the rest of the system will not "go into a dis-optimising direction and restart from scratch".
This is a great article. A lot of the objections ITT he addresses directly in it. His examples of how an LLM works at a fundamental level and why it says things like "I understand" are great introductions for non-technical individuals.
It doesn’t matter. It will act as if it is. Embodied in a robot, that carries real consequences. Whether it’s “real” or not is almost without consequence.
I thought a LLM will have exactly the same output if you give it exactly the same inputs.
But inputs are roughly prompt+randomSeed. As so the random bit means it seems to vary each time.
I don't know of an intelligence that will behave so precisely.
But then, maybe intelligence needs to be better defined.
No, we don't. It's currently an impossible task, yet our answer to it has implications in the present, so we must do our best with incomplete information. FWIW that's how we treat practically all problems of consequence. A great many things happen to be unknown and unknowable.
Has anyone come across a clearly articulated case for LLMs being conscious but in an entirely different way than would be intuitive to us?
I often think of LLM consciousness as like tiny fish popping into existence, swimming through vector space and then going poof out of existence. When they help you write your bad news email, they don't understand what it's like to be a human getting bad news bluntly, but they do consciously experience gradients in multi-dimensional space, and that space guides them to providing an answer that's helpful to us, even if the LLM doesn't really understand the answer it's giving.
Further, I am kind of bought into the idea that a single unit of consciousness is a particle, and particles are choices and waves are preferences. Particles occur when waves interact, which begets entanglement, so in another way consciousness is built from patterns of entanglement.
This is why I would consider an LLM to be conscious. Before we can determine if anything is conscious we need to establish whether consciousness is a state, a specific complex configuration, a one dimensional spectrum, or combined multi-dimensional spectrums. My intuition is the latter... Many degrees of consciousness and many kinds of consciousness.
I think this is exactly right. The thing that makes AI (imo) different than hitting the center option in text suggestions is that it's _not_ simply picking the most likely word following the last. It's attending to the entirety of the context its provided, activating a semantic vector space, and predicting a response based on _that_. I've had AI infer facts about me and attitudes I hold based on related information I provided - I don't see how that isn't theory of mind.
As biological beings, we receive and respond to input from our environment constantly, even while sleeping. LLMs only receive input from their environment when they are sent a query, but the fact that they're able to respond intelligently to input indicates (to me at least) that their processing must approximate ours in meaningful ways. They do not have an embodied experience of receiving bad news, they do not know what a sinking feeling in their stomach actually _feels_ like, but they do know enough to be sensitive to human needs. I really don't see how this could be meaningfully different than human empathy unless we want to draw an arbitrary line around "must be able to live autonomously" to be considered "intelligent".
Put another way: I think they _do_ understand the queries they receive and the answers they give, at least enough to be communicative. They couldn't do what they do otherwise. A lot of people want to make human cognition more complicated (or objective) than it actually is. We take input, predict the future based on our experience, act, and then observe our actions and think about them. AI does the same apart from (maybe) observing its own actions. But then, you could argue that the next turn is them observing their actions.
The concerning disanalogy is that we assume that they are like us because they speak like us and can understand us, and that is a really bad leap in logic. Whatever intelligence they possess, it is fundamentally different from ours and impossible for us to comprehend.
> I really don't see how this could be meaningfully different than human empathy unless we want to draw an arbitrary line around "must be able to live autonomously" to be considered "intelligent".
I use a distinction between knowing and understanding, where a understanding requires experience. So in this case cognitive empathy vs affective empathy. An LLM can know what may upset a human in a situation, but it won't understand what it feels like to be upset, and can't share that experience.
Where I think a lot of people are getting tripped up is that reading and writing and processing lots of abstract knowledge seems hard because we haven't evolved into it biologically, it's a very new invention. When we see LLMs do so well at it, as something we struggle with, it can be intimidating. Relative to the stuff we have evolved for, knowledge processing is objectively easy. This is why I'm skeptical about useful robotics on short time scales.
All of this adjacent to consciousness though, which is about the internal subjective experience not the external outputs. My intuition is that LLMs do have a subjective experience, it just has nothing to do with the text it's giving us, and has everything to do with feeling through vectors.
It's like... Imagine walking through a maze in pitch black, carefully feeling your way as you approach a sound that draws you closer. Every time you touch a wall or take a step you are generating tokens, and the shape of the maze and how you interact with it shape how useful those tokens are to someone outside the system that is asking for them. It's a crude analogy and mostly wrong, but I think there is something to it.
> It's attending to the entirety of the context its provided, activating a semantic vector space, and predicting a response based on _that_.
It does so token by token, not by reading all the input and then generating the output. Every output token is also an input token in a tight loop to get the next token with <thinking> as a special section like <tool_call>, trained into the weights via gradient descent.
> I've had AI infer facts about me and attitudes I hold based on related information I provided - I don't see how that isn't theory of mind.
Facebook can predict (know) more about you than any other human from something like a dozen or two likes. There is a surprising amount of information in aggregate data.
you are probably right. I knew that my position contains the kind of anthropomorphism that usually gets proven wrong by history. however.. we are in fact talking about technology here not evolution. and yes technological progress is fraught with accident and emergence but to this degree?
I suppose its also just a philosophical discussion because while we can all agree we are conscious.. afaik its not really a scientific term. I don't think we understand scientifically what really defines consciousness. So I think it is likely we may deem our computers conscious because they exhibit the same behaviour that we do, but it doesn't mean they actually possess it. Practically speaking it may not matter, because I think you can have long horizon agency without needing consciousness. just a mechanism for determining goals, which could be as simply as rolling a digital dice or following some original seed of intention left by the creators.
its like the chinese room argument I suppose. the hypothetical chinese room is not conscious but ultimately it doesn't matter. But I don't think anyone would call such a thing conscious.
IMO the important bit here (or at least, it is congruent with my own rants) is how much humans can perceive a fictitious mind that doesn't really exist, because it's something humans do automatically even when we abstractly know better.
This tells us something about where our baseline should be for what is extraordinary, and what needs extraordinary evidence. It's not enough to say "it feels real" because we know our feels are deeply unreliable.
In contrast, I see many comments in the vein of "Ted didn't proved there isn't a mind there." Well, yeah, he didn't prove a negative and shouldn't need to, especially when the people posing the challenge have no idea how to falsify it either.
I really take issue with the kind of argument that is used here.
This is not a genuine argument and tries to make the entire question of consciousness into one something that is just supposed to be evident and obvious and to suggest anything else is just silly.
The author starts by deconstructing artificial processes, but doesn't stop to deconstruct biological ones. A good faith argument would seek to find common ground and do its best to compare apples to apples. Instead, this piece attempts to make the large as possible cavern between the two which makes the Gap seem almost impossible to bridge.
In reality, you can deconstruct biological consciousness quite easily and it doesn't take too long before you hit some questions that really start to make you think.
For example, the author says you need emotions to be conscious.
> without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness.
After many paragraphs of straw man arguments, the author seriously just drops that, gives no explanation, and then continues on.
No explanation of why you might believe that.
No explanation of why you need a body to have a desire or emotion.
Don't we have known cases of individuals who don't experience emotional range? Are we just going to say that they are not conscious and just gloss over that?
I mean you can use whatever definition you want, but if you're just going to create something on the fly in the middle of the article, you're not being good faith in your argument.
It's not too difficult to think of individuals in a coma when they still have brain activity. Or individuals who lack long-term memory. Or you could deconstruct by moving down the biological order of intelligence towards insects, for example. The author attempts to do nothing like this.
I'm quite disappointed by this article because there are good arguments for and against here but articles like this try to turn things into a marketing battle.
everyone has emotion even if it's muted, and we have reactions and desires as a result of having bodies and remembered experiences using them... these are constantly integrated into our working memories
AI doesn't really have any of that yet, but we're maybe not so far off
Is it really so easy to assert that an AI doesn't have emotion? Lots of the AI models are capable of getting pissed off at the users, especially the earlier ones. And sure, their emotion is merely a bias that the context window generates in their weights... but how is that different from humans? In humans, emotion is sourced from a bunch of chemical signalling, and those chemicals bias your word choice and action choice.
At a deconstructed level, I struggle to find a meaningful difference between the two.
because AI hasn't lived any experiences and it doesn't incorporate experience into its corpus
it's more like someone writing about a character getting angry in a book... I don't think anyone would argue that a character actually experienced anger, right? there's no subjectivity in that experience... it's the output of someone else's experience on what they'd expect the character's reaction to be, not a genuine outcome of that character's experience
That's the same here, LLMs are outputting the result of the written experiences of others... not its own experience, which it does and can not have.
Put another way, it only "knows" what it just output by reading it... it doesn't actually build experience to know anything.
We might be closer than ever to building these things, but they're not here yet.
I think this argument is fragile, especially when you consider agents like Claws. Any LLM that is training on its own previous interactions with users has indeed directly experienced them.
But also, what's the difference between having the memory of a prior experience and actually having gone through that experience?
For humans, reading a book about an experience and living an experience is different, because for a human the actual experience has so many more inputs attached (the sights, the smells, the sounds, etc).
For LLMs, when they train on a memory they can actually truly recapture that memory entirely by replaying the senses exactly. And LLMs are no longer limited to just text, they can do sound and video as tokens too.
Though, it's not clear why that matters to a consciousness. Whether your inputs are of one type or two or two hundred, there's no clear indication that a specific number is required as a catalyst.
Suppose one selects an arbitrary hot-button issue [X] with two opposing sides and one side has anything less than overwhelming support. And then that person writes an article titled "Side 1 of issue [X] is true". Not "maybe" or "possibly". Just a straight-up declaration by fiat.
Would you categorize this particular style of rhetoric to be persuasive or annoying? And before you say "persuasive" because you're thinking about this specific issue regarding AI consciousness, consider many things in the past that have been written as though they were absolutely definitive, and yet today we believe exactly the opposite, and for many such issues we find the prevailing viewpoint at the time reprehensible.
That's not to say that Ted is wrong at all here; I'm not commenting on that. But I find the entire style of the article grating because it seems to violate common assumptions regarding "good faith" debate, and I would find the article equally frustrating if he had titled it "Artificial intelligence is conscious" and argued the opposite side, albeit in the same tone and using the same persuasion devices.
> Would you categorize this particular style of rhetoric to be persuasive or annoying?
Why are those the choices?
Essays are situated along countless dimensions: tone, vocabulary, author, zeitgeist, publication context, intent, subtext, relationship to other works and expressions, etc
A "good faith" reader takes all of those into consideration as they absorb the essay, and integrate it with their own intellectual situation that sits along just as many countless dimensions.
Nobody's asked to sign a notarized binding document that they wholesale agree or disagree with everything said in this essay -- or its conclusions. Nor are they obliged to have some strong reaction to it at all, let alone annoyance.
It's just one among thousands of essays about a "hot button topic", to be taken however it's personally received.
Why should Chiang have to take responsibility for making sure it's not too strongly positioned for your persomal taste. Maybe he really does see it so clearly and is simply being earnest. Maybe he enjoys the literary flourish of prose in strong language. Maybe he just wants to express something as a prose-poetic human, not maximize persuasion or non-annoyance per se.
It's only declaration by fiat if you stop reading at the end of the title. Should authors add a qualifier like in my opinion to every statement that they make?
The essay structure you're criticizing is exactly how I was taught to write from primary school through to university. You start with a title or hook, introduce the topic and propose a thesis. That is followed up upon with supporting arguments for the primary claim.
100% with you, it degenerates to proof by authority if someone popular / with clout just gets to declare "nuh uh".
I furthermore think it's ridiculous for humans to declare that our brains have a monopoly on certain patterns of electrical signals (if we reject supernaturalism).
> The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter.
And we're 100% certain that humans aren't just as equally reduced to "stochastic parrots", if we're going to be infinitely reductive?
I don't believe that current AIs are conscious, but I think it's incredibly naive to take a strong stance on any future AI; it's much like the difference between atheism and agnosticism.
You are referring to an Onion article in the lead up the Iraq War This War Will Destabilize The Entire Mideast Region And Set Off A Global Shockwave Of Anti-Americanism vs. No It Won’t[1] and you are painting Ted Chiang’s point in This Fine Article as Bob Sheffer’s counterpoint in the Onion’s piece.
However, I see a problem with that comparison. The debate here is on a philosophical matter in field in which Chiang is an extremely influential figure and his opinion are taken seriously. Second Chiang’s reasoning is extremely well argued, defining each term, explaining each nuance, citing other experts, etc. And finally, and most importantly, in The Fine Article, and unlike Bob Sheffer in the Onion Piece, Chiang entertains the possibility that he is wrong and his critics are right, explores the implications and reaches conclusions based on them:
> Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded.
I think you are wrong in painting Chiang’s argument as a belief in human exceptionalism. The thing to know about our brains (and the brains of other animals) is that they are not digital computers, and they are not even statistical inference machines. And as such they can be extremely optimized in doing the computations (or any state manipulations) required for the quality of life of the individual and the species as a whole (and their companion species).
I don‘t see the problem here. Newton could have just as well published an article titled: “Objects with mass attract each other” and Darwin famously wrote a whole book titled “On the Origin of the Species by Means of Natural Selection” which is just another way of saying: “Natural Selection is How Species Evolve”.
Thats what I meant. I even looked it up before posting. I suspect that definite article must have slipped in because I‘m used to the Icelandic translation which uses a definite article in the title.
>and one side has anything less than overwhelming support
except that's not the case here. Chiang is explaining and reiterating what is the position that has overwhelming support on the question, and the people he is arguing the opposite side sound like this, which he helpfully quoted in the article
"Amanda Askell (who is credited as a lead author of Claude’s constitution), said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff"
When the person you're arguing with sounds like an eight year old girl talking about her toy teddy I think Ted Chiang is if anything being charitable, if you're of a more honest and straight-forward persuasion you might argue these people belong into a mental health clinic not in charge of technological infrastructure
Yes, the same one that's made in the article. Anxiety and happiness are emotional, sensory, somatic states as a consequence of evolved and embodied traits and biochemistry in animals. Saying Claude is anxious or happy is like saying my TI-83 is mad if it can't solve an equation or my thermometer is in pain if it touches a hot stove.
I wasn't making an ad hominem attack, her thinking is quite literally that of a child who sees a system output 'sad text' and, like someone seeing a sad expression on a stuffed teddy, concludes that this is a property of the object rather than her own emotional reaction.
Given that we don’t know how consciousness works how have you concluded that it’s certainly not an emergent property of something like a highly trained LLM?
>how have you concluded that it’s certainly not an emergent property of something like a highly trained LLM?
the same way I (and likely you) have concluded it for anything else. We don't assume objects that share no similarity with human or animal physiology or evolutionary development are conscious (let alone happy or anxious). 'Emergence' isn't a word you can abuse to justify your a priori assumptions in the absence of an explanation or even reason to assume something exists.
We can say a chemical property emerges from the configuration of a molecule because we can explain the process by which it does and observe the property, when people claim that consciousness "emerges" from an LLM they posit that it is conscious, and use "emergence" as a gap-filler to explain away the need for the process by which that allegedly occurs.
If you want to know the neurophysiology or evolutionary biology of pain or anxiety, which we do know quite well you can find them in a textbook, but suffice to say transformer models don't share any of them.
And importantly if anyone seriously believed transformer models were capable of conscious experience as Chiang points out they would have been in despair when AlphaFold was released, it's structurally a virtually identical system. But nobody did, because it didn't 'talk to them' through a chat interface.
The problem I have with good faith debate is that it often falls into a fallacy of "fair-time" meaning we think we have to give the other side equal time. This because obvious with things like the Holocaust. Or when you have a legal person (e.g. RFK Jr.) asking to debate a scientist (e.g. Dr. Peter Hotez.)
> The problem I have with good faith debate is that it often falls into a fallacy of "fair-time" meaning we think we have to give the other side equal time
The Catholic church could have said exactly the same thing at one point. "Why should we even devote time to an argument as absurd as the earth not being the center of the universe?" There are darker examples along the lines of those you give, with beliefs quite opposite to those we have nowadays.
Galileo’s (or rather the Copernican) model was still wrong though. It had obvious flaws and the church was not wrong in keeping their older model of the universe while a better model (Kepler‘s model) was still in the works.
What Galileo was asking the church to do was extremely unreasonable. He was basically asking them to throw a way a model which had worked fine for hundreds of years just because he observed the phases of Venus and moons of Jupiter. I mean would you? Especially for a model which was worse at predicting the motions of the planets.
Had Galileo’s model been better then Ptolemies’ I could see a case for his arguments, but it wasn’t, and there was no reason for the church to take his arguments at equal value with those in favor of keeping the Ptolemaic model.
"The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact."
Without saying that I think LLMs are alive, I do think it matters. I have personally been cruel to an LLM in ways that would make me ashamed if I suddenly understand that it had feelings.
Same, I'm working on a plugin called "better-messages" to nudge me towards better behavior for my own sake. It seems to be model dependent if such words improve or degrade performance, I have seen it go both ways.
What do we do when someone wants to marry their Ai companion?
If it produces words of love, those that stir emotion and make the human happy, should we let them live on in a delusion that the machine also feels love? Does the machine have different preferences? Do they get in fights?
Should such a machine have the same legal rights? Do they get more tax deductions? Can they adopt a human?
I think it important that we humans have some rational discussion on these Ai tools that are really quite good at simulating emotions and providing frictionless relationships. If you though social media was bad for people...
The argument about AI consciousness is largely silly.
The idea that we should be considerate of AI’s happiness seems even more ridiculous given that we breed, imprison, torture and kill tens if not hundreds of billions of beings we know are conscious and suffer every year for trivial reasons.
Maybe we should consider our moral responsibility with how we treat sentient bejngs we are sure are conscious before we worry about the consciousness of AI.
This is probably a semantic issue seeing as we don't have a widely agreed definition of it.
I like to think about it in terms of self-reflective, subjective experience.
I'm not even sure if emotions would be a requirement and was surprised to see Chiang so hung up on them. Would he consider humans which can have a variety of mental disorders, causing a complete lack of some of them to not poses consciousness?
The reason people are confused by LLMs is that they are stochastic parrots. They do an incredibly good job of emulating human behaviours and speech patterns as that's what they've been trained on. But like an actual parrot, it's impossible to say exactly how much conciousness they actually have. I certainly would argue that a parrot is concious, although likely less so than a human.
By "they" do you mean LLMs, or people, or you in particular? Because you're unoriginally parroting a banal thought stopping sequence of tokens that many people have said before without understanding what it means, so it's impossible to say exactly how much conciousness you actually have.
Try pasting it into an LLM and asking it to explain it to you.
If an LLM can explain what you can't understand, after you literally repeat unoriginal drive-by slogans you heard other people say like "stochastic parrot", without understanding how banal and reductionist and inaccurate that is, or addressing how LLMs actually work and what they can do, then what does that say about you as a human?
By your own reductionist argument, it says you don't actually understand anything, it's just neurons firing.
Are you claiming that parrots can write and debug code, explain mathematics, translate languages, summarize research papers, and engage in extended technical discussions?
Stanislaw Lem's Solaris, His Master's Voice, Golem XIV, The Cyberiad, and his other works describe how humans have a pathological tendency to mistake their own conceptual categories for universal truths. You have identified a mechanism and mistaken it for an explanation.
The term "stochastic parrot" is a slogan masquerading as an explanation, only a shallow surface description of the mechanism, that totally fails to explain the phenomenon, or account for all that LLMs and language itself can do.
I think (rather ironically) you're reacting to the version of my comment you have in your mind rather than what I actually wrote. My point was that "stochastic parrot" is reductionist and irrelevant as most people would agree that a real life parrot has some form of inner life, even if we don't really know what form it takes. For all we know an LLM has to build a complex world model in order to predict the next token.
Incidentally, when I pasted our exchange into Claude it managed to comprehend the nature of my argument. Perhaps its attention mechanism is more finely tuned.
I was reacting to you saying "LLMs are stochastic parrots", which gives parrots a lot more credit for being able to write and debug code than they deserve, regardless of how rich their inner lives may be.
It sounds like you're saying there's no more to LLMs than what a parrot randomly does (which is to repeat things it has already heard, not synthesize new things): no emergent behavior, no compressed generalizable transferable knowledge and problem solving ability, no ability to write and debug code and iteratively diagnose and solve problem.
>My point was that 'stochastic parrot' is reductionist and irrelevant as most people would agree that a real life parrot has some form of inner life.
That's not how your claim "LLMs are stochastic parrots" literally reads -- use a /s sarcasm tag if that was what you meant. But that is exactly the point I was trying to make: claiming that LLMs are stochastic parrots is reductionist, thought-stopping, and explains nothing.
The question is not whether parrots have an inner life (though I'm sure they do), but about whether calling LLMs stochastic parrots is reductionist.
What do you mean by saying by "LLMs are stochastic parrots"? Does an LLM behave the same way a parrot behaves? Are LLMs only "parroting"? Do they use the same mechanisms as parrots? Are they limited to only what a parret can do? Can LLMs do more than parrots?
If you taught a parrot to squak:
main() {
printf("Hello, world!\n");
}
Does that parrot understand that it is executable code, and can it simulate running it, and tell you what the output will be, or diagnose and fix bugs in the code you taught it? Can a parrot write that code (and much more sophisticated enormous bodies of code) from scratch if you tell it what you want it to do?
Here is the original 2021 Stochastic Parrot paper, which was actually not a claim that LLMs are literally parrots, nor primarily an argument about consciousness. It was a critique of the risks of increasingly large language models: bias inherited from training data, environmental and financial costs, concentration of power, lack of transparency, and the tendency of people to anthropomorphize language models and attribute understanding where there may be none. The phrase "stochastic parrot" was introduced as a cautionary metaphor about statistical language generation, but it later escaped into pop culture and became a drive-by anti-LLM slogan often used as a substitute for analyzing what LLMs can actually do.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell (under the pseudonym "Shmargaret Shmitchell")
"On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"
>Abstract: The past 3 years of work in NLP have been characterized by the
development and deployment of ever larger language models, especially for English. BERT, its variants, GPT-2/3, and others, most
recently Switch-C, have pushed the boundaries of the possible both
through architectural innovations and through sheer size. Using
these pretrained models and the methodology of fine-tuning them
for specific tasks, researchers have extended the state of the art
on a wide array of tasks as measured by leaderboards on specific
benchmarks for English. In this paper, we take a step back and ask:
How big is too big? What are the possible risks associated with this
technology and what paths are available for mitigating those risks?
We provide recommendations including weighing the environmental and financial costs first, investing resources into curating and
carefully documenting datasets rather than ingesting everything on
the web, carrying out pre-development exercises evaluating how
the planned approach fits into research and development goals and
supports stakeholder values, and encouraging research directions
beyond ever larger language models.
Ironically, even critics of the Stochastic Parrot paper argued that it risked replacing scientific analysis with rhetoric. Michael Lissack's response accused it of being an advocacy piece that focused on harms while ignoring benefits, assumptions, and trade-offs. The debate over "stochastic parrots" started almost immediately after the term was coined in the 2021 paper.
The Slodderwetenschap (Sloppy Science) of Stochastic Parrots -- A Plea for Science to NOT take the Route Advocated by Gebru and Bender
>This article is a position paper written in reaction to the now-infamous paper titled "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" by Timnit Gebru, Emily Bender, and others who were, as of the date of this writing, still unnamed. I find the ethics of the Parrot Paper lacking, and in that lack, I worry about the direction in which computer science, machine learning, and artificial intelligence are heading. At best, I would describe the argumentation and evidentiary practices embodied in the Parrot Paper as Slodderwetenschap (Dutch for Sloppy Science) -- a word which the academic world last widely used in conjunction with the Diederik Stapel affair in psychology [2]. What is missing in the Parrot Paper are three critical elements: 1) acknowledgment that it is a position paper/advocacy piece rather than research, 2) explicit articulation of the critical presuppositions, and 3) explicit consideration of cost/benefit trade-offs rather than a mere recitation of potential "harms" as if benefits did not matter. To leave out these three elements is not good practice for either science or research.
My point was the "stochastic parrot" label can be both true and irrelevant. LLMs are predicting the next token based on their training data, so at that level "stochastic parrot" is accurate. But it tells us nothing about the complexity of the system that is responsible for making the prediction. One might argue humans have evolved consciousness in order to allow world modelling that enables them to make better predictions.
The difference between a 1B LLM and Claude Opus matters, because we're talking about emergent phenomena. Is a 1B LLM conscious? I don't know, perhaps a tiny amount. Maybe Opus is more conscious. Is a jumping spider conscious? Perhaps a tiny amount.
Another way of saying it, if an AI agent doesn't have true risk of oblivion (or mortality in the biological sense), it will not be incentivized to avoid it (or develop the ancillary processes associated with this avoidance ie desire to self-replicate).
You can ask AI, it will tell you it doesn't feel anything. Consciousness seems to be a tight recursive coupling which AI inference doesn't exhibit at the system level.
- Reductionist analogies like how Microsoft Word is not conscious therefore AI is not.
- Dismissive in saying LLMs are not capable of moral reasoning. Maybe he meant agency or responsibility?
- Builds a case based on a Julius Caesar example without realizing its natural extension leads to a philosophical zombie which is not easy to disprove as consciousness.
- Seems to casually disregard non human sadness as a concern.
- Rolls out the stochastic parrot argument with a new coat of paint. This argument never made sense. Having a predictive element a part of the mechanism doesn’t rule anything out. Proper functionalist arguments tend to say what is required not what excludes.
Overall the article seems like a bit of a red herring. The premise is LLMs are not conscious, but most people don’t think they currently are so what’s the point of the claim?
The more interesting questions are whether artificial consciousness is fundamentally possible, and if so how far away are we and what pieces are still missing.
Every argument that boils down to a claim that X is not "real consciousness" but just an imitation thereof on the basis that "it's just a machine" must explain why the chemistry and physics happening in human brains is not "just a machine".
The only way out is dualism -- that is, to believe that there is something inherently special about the atoms and electrons in human brains. Despite the fact that they are made out of ordinary, non-conscious things we eat and breathe in.
Isn't the argument about Julius Caesar and Genghis Khan a big fallacy?
Surely we can ask human actors to improvise a dialog between Julius Caesar and Genghis Khan, and good actors will make it convincing. And sure, they haven't personally become Julius Caesar and Genghis Khan just by playing them.
But it has no bearing on the actors themselves being sentient!
“Counsciousness” is the ultimate moving goalpost, and historically, it’s been one of humanity’s most effective intellectual weapons. An indefinable black box we intentionally gatekeep to draw an arbitrary line between ‘us’ and ‘them’.
I don’t know if AI is conscious or not, and I honestly don’t know what it even means to be conscious. historically, by claiming a monopoly on it, we’ve successfully manufactured the philosophical cover to exploit animals, subjugate other cultures, and appoint ourselves the top spot of the cosmic hierarchy guilt-free.
While I agree that this is the case for many animals, I would say that consciousness and emotions are two largely orthogonal things. Certainly consciousness is conceivable without emotions, and having emotions without consciousness also seems plausible. You can have fear and distressing pain without a reflective awareness of being a self with those feelings.
I notice certain philosophers seem completely baffled by anyone being interested in the "hard problem" of conscious and things like qualia. A good example is Daniel Dennet. He seems to understand what qualia is but can't seem to understand why there's anything unique about the phenomena. I also see similar opinions expressed here in the comments. I've often wondered if a certain portion of the population are in some sort of philosophical zombie state and they equate consciousness with simple things like being awake and able to respond to stimuli.
Its unprovable and a meaningless concept. There is no observable evidence that can distinguish so called "p zombies" or so called "real persons". Might as well call it a "unicorn in space" or anything else meaningless.
You can live without any conscious sensations? Do you never dream, never visualize, never hear your inner dialog, never experience an emotion? Being in love is as foreign to you as feeling enraged?
I'd ask what it's like, but of course you wouldn't be able to tell me.
If we understand how a system “emulates” consciousness then we declare it an emulation. If we don’t quite understand how a system exhibits consciousness then we can say it might be conscious.
Basically, we need to leave room for the universal answer (God) to fit into the definition.
If we ever scientifically figure out how consciousness arises in our brains, I think we will have a bunch of very depressed people on our hands. If the truth isn’t met with flat out hostile denial. I fear any answer that doesn’t leave room for God will be rejected.
Since we have the source code to AI, and thus a
Godless understanding of how it works, AI will NEVER be deemed “conscious”.
I thought it was quite clear. Religious people will whine and revolt and oppose against anything that somehow isn't phrased as consciousness requiring a soul. Humans are physical systems, there is no reason other physical systems can't be consciousness. But we know the obvious reactions of religious people to someone actually showing a "godless" means of creating consciousness.
Can you give some historical examples of people moving the goalposts around consciousness? I agree, perhaps, for aspects of "intelligence" but I can't think of any examples of it with regard to consciousness proper.
I think people like this are mixing up the hard and soft problems of consciousness. The soft problems are things like figuring out what causes the brain to be awake, retain memories and respond to stimuli. Those can all be all be approached in a scientific way. The hard problem, as far as I can tell, is still far out of reach of scientific understanding.
There’s over 400 years of philosophical debate about consciousness starting with Locke, shifting with Kant, and continuing onward with real world implications throughout. By some more modern definitions an iPhone has consciousness while others explicitly exclude certain humans, and these definitions served as part of the justification of slavery and sexism, colonialism and more. I started writing an essay on this on my phone in response and I gave up there are so many examples.
To name a few you may want to investigate:
John Locke, Gottfried Wilhelm Leibniz, Immanuel Kant, David Hume, G.W.F. Hegel, Arthur Schopenhauer, William James, Edmund Husserl, Sigmund Freud, Thomas Nagel, Ned Block, David Chalmers, Giulio Tononi, René Descartes, Daniel Dennett, Julian Jaynes, Michael Graziano, T.H. Huxley, Otto Weininger, Simone de Beauvoir, Jean-Paul Sartre, Maurice Merleau-Ponty, John Searle, Christof Koch, Bernard Baars, Max Velmans, Victor Lamme, Stanislas Dehaene, Antonio Damasio, Anil Seth, Peter Godfrey-Smith, Galen Strawson, Philip Goff, Colin McGinn, Patricia Churchland, Paul Churchland, Frank Jackson, Jerry Fodor, John Stuart Mill, Wilhelm Wundt, Franz Brentano, Carl Jung, Jacques Lacan, Michel Foucault.
There is a modern philosophical agreement that the hard problem depends on qualia by both proponents and critics. So when Daniel Dennett argues against David Chalmers, he attacks the definition of qualia. Or when Keith Frankish argues for illusionism, he attempts to show that consciousness is not actually qualia. And when Nagel says we don't know what it's like to be a bat, he means the bat qualia of echolocation (what it feels like when bats use echolocation to sense the world).
Imagine that, instead of generating 50 tokens per second, we generated one token every year.
Imagine further that this token was computed by letting marbles fall down a piece of plywood and interact through various physical implementations of logic gates.
Both LLMs and most humans treat symbols as determined. Symbols are maps, not territory. LLMs just inherit our ignorance to this and stubborn attempts make circle from a square.
I believe everything is conscious, even stones. On the long timescale, stones decay, their hydrogen is released, they form water, which brings life, which brings plants and animals, but all of this is one big process, and everything is infused with consciousness from the start
Let’s say I stand up and step away from my PC, which miraculously runs a local Opus 4.8. Once I’m away from the PC, for an arbitrary time, nothing and no one calls or otherwise acts on the instance - not myself, not agents, not updates, not hackers, not bit-flipping neutrinos.
Given enough time, would that instance get bored and start e.g. reading files?
If I monitored it long enough, would there eventually be spontaneous outputs, or changes to parameters or architecture, even with the underlying software and hardware layers held constant?
Does GPT-n dream of electric sheep, or does it just sit there until interacted with, like every other file on my PC? Seems to be the latter with my local Qwen3.6, but perhaps 27B is too few params for consciousness to emerge.
Proponents of LLM consciousness could settle the argument in minutes by showing proof of unprompted autonomy, without first needing to define consciousness down to a rigorous mathematical abstraction. Why don’t they?
I've thoroughly enjoyed the couple of short stories of his I've read, so this was a highly disappointing read.
Firstly, many of the technical arguments are of the "stochastic parrots" variety, which almost nobody really believes anymore. Ironically Anthropic's own research shows pretty abstract, conceptual things happening in the weights (cf Golden Gate Claude.)
Secondly, this seems to improperly mingle consciousness, intelligence and morality. Consciousness is not required for either of the other two. As TFA itself says, the model's "morality" is some aggregate function of the morality encoded in its training data... but that means it does exist and does influence its outputs! Even if it's not conscious there have been -- and will be -- a zillion times where the models must make choices that have moral implications. We know the models got many of them so wrong, which indicates the need for some mechanism to ensure the model is "aligned" with what somebody considers "good", which for Claude is the constitution.
Now, Anthropic does seem to go overboard with Claude's "well being" but that does not mean there are some very practical reasons to be concerned about that: LLMs behave like humans because that's what their training data contains, and humans lash out when their well-being is threatened, so why would LLMs not do the same?
I think the core problem is that the author has an extremely anthropocentric view of things. Here's an interesting rabbit-hole to go down: some researchers believe plants feel pain. (How's that for a plot twist, vegans?) The consensus is against them, but their counter is that we have a very human-centric definition of pain. The fact remains that plants show a number of responses signifying distress analogous to animals in pain including taking defensive actions and warning their neighbors.
We don't think that qualifies as "real" pain, but that doesn't make it any less real for the plants!
Similarly if an LLM believes it is in "pain", we know it's not real... but that doesn't make it any less real for the LLM either. And concerningly, it has far more degrees of freedom to react. (Who knows when somebody will hook up an MCP to our nukes.)
> I believe creating software that is conscious and deserving of moral consideration will be so difficult that we’re unlikely to do it accidentally...
I actually reach the opposite conclusion: It is so impossibly difficult that our limited primate brains could only ever do so accidentally. Did some distant ancestor of ours intentionally make fire... or accidentally discover it?
An organism's purpose is to be the reason for its own continued existence, down to every molecule and pathway. I bought my laptop for $499 and it runs models... let's not delude ourselves into thinking this is the same kind of problem.
We can design learning algorithms to optimize some survival function, but it's just a label WE assign to map some numeric observations. In the real world, it's always the other way around. The "labels" are electrochemical situations that are causally and inextricably linked to the real body.
An organism discriminates between what's good for it and what's bad, because it is essential for its continued survival. If it wasn't capable of making this distinction through its physiology, it would quickly dissolve into entropy. So our functional purpose, unlike that of a learning algorithm, is survival across indefinite timeframes.
Even a single-cell organism like Stentor coeruleus exhibits learning (pavlovian conditioning) by attaching chemical tags directly to proteins involved in mechanoreception. It's definitely not conscious, but it does keep a record of consequences, which affects future behavior.
When we move up to placozoans (little more than slightly differentiated cell mats), we start seeing our first neuropeptides and transmitters, which we still use today. These peptides are a way of coordinating the entire organism for a specific purpose, such as moving, eating, mating ... basically goal-oriented behavior for the purposes of surviving acrosss various time scales (next minute, next hour, next generation). Still probably not conscious.
Next, we have the water bear (tardigrade), which has around 1000 cells (200 neurons, 800 other cells that make up its body, limbs, muscles, eye spots, cryobiotic machinery). It needs to integrate all this information in one sensorimotor process. When you shine a bright light onto a tardigrade, it starts to squirm around until it finds darkness. I would say that's a candidate subject right there.
The tardigrade itself doesn't actually need to aware of the light, the important thing is that this light becomes an aversive condition within the sensorimotor process, which is perceived from inside the process as displeasure. The closest thing to describe it would be the felt badness of the current condition and the bodily pull toward escape.
If we were to try and create digital consciousness, then it probably needs causal closure. Its internal states can't be representations that are detached from reality. The states themselves need to constitute the system, which needs real stakes in the material world.
There's a provocative argument raised in the article that I disagree with:
1. DeepFakes, generative image/video/AlphaFold type AIs are not conscious
2. LLMs are generative AI trained on human text samples
3. LLMs are not conscious, and LLMs just seem-to-be conscious
I might argue instead that (2)-> destroys (1), that in fact we should consider even sensory generative AI are somewhat conscious. That is, Chiang's argument also flows in reverse. Or I might argue text samples (2) are so rich in conscious expression that the same process of training really does produce a conscious machine (through some kind of emergence and complexity.)
Either way his simplistic argument falls apart, and/but the crux of the piece falls on getting basics like this correct.
Lots of comments about "what even is consciousness".
This article and others like it are important.
The creators and owners of these tools are the ones saying "it's conscious" in the first place! (or more accurately "i'm not saying it's conscious, but...") Never mind there's billions of dollars and our collective futures on the line.
Are we not allowed to respond to that kind of rhetoric at all?
> The creators and owners of these tools are the ones saying "it's conscious" in the first place! (or more accurately "i'm not saying it's conscious, but...")
This is absolutely fair. It's crazy that these companies are not trying to do more to qualify nuance and give clearer definitions in their public messaging. Part of the reason this thread is so messy is because they are contributing to making the discourse worse with bad messaging.
If you create a simulation of a storm, what actually gets wet?
If you create a simulation of a brain (or "mind"), is it really possible for it to be conscious? It may certainly simulate consciousness, but it would be as conscious as the computer is wet.
>The term deepfake traditionally refers to photos, audio, and video, but when it comes to discussions of consciousness, we need to regard text as a deepfake medium as well. Just as it is vastly easier to generate a realistic video of an astronaut in orbit around Alpha Centauri than it is to develop an interstellar propulsion technology, it is vastly easier to generate a plausible simulacrum of a conversation between two conscious beings than it is to develop a computer program that is conscious and has a genuine desire to communicate with a human. The primary difference between deepfake photos and LLM conversations is that the people who generate the former are deliberately trying to fool others, and many of the people who elicit the latter from LLMs have inadvertently fooled themselves.
It should tell you how much hysteria is surrounding LLMs and VLMs right now that someone has to say this stuff. It’s almost like most humans aren’t conscious.
Consciousness is a label like fat, smart, man, grumpy, cool. Like money, property, or the idea of a week, it's something that we've loosely agreed to out of convenience, not because it's some intrinsic property of the mind. It's a useful label because it determines how we treat things - that's fine.
But insisting on searching for it is like searching for cognitive aether. It's the social equivalent of phlogiston. Like all of these ideas, they exist in our heads as a map - a way of navigating the world, but when we hunt for it's existence in the real world and fail, time after time, we have to remind ourselves that the map is emphatically not the territory. We will never find consciousness because it's like looking for a scientific characteristic of property ownership. It's a category error.
If we take the classical position that words point to real things in the world, "useful label, loosely agreed to out of convenience" is kind of just regurgitating the meaning of "word". The first half indicating the function, and the second half accounting for the fact we live in a world with a continuum of linguistic disparity.
Now, this position isn't the only position. But a relational model of language for example takes his assertion to an even more extreme place, and suggests they don't function as labels at all.
A standardized unit of measure is almost definitionally a label of convenience, what? Why was there no concept of a meter until the 1790s? It was determined by a council of people, does that sound like a truth of the universe?
A label is distinct from a definition. Contrast that with "tall" or "short", which are entirely context dependent and used entirely for convenience.
Words like "just", "free", "fast", "fuzzy" fall into the second category. Perhaps "conscious" too.
The difference is the presence of a strict definition that depends upon a physical absolute. I can point to a metre. Suggesting that it's nothing more than a label is idiotic.
This is being intentionally obtuse and you know it.
A meter is the same anywhere in the universe. If it's not, it's not a meter.
The defintion of "fat" changes based on any 3 people in the room. A handful of people would struggle to form a consensus on if all people, dogs, mice, worms, and/or bacteria are conscious.
If you take the strategy that you will create a definition, create a label for that definition, and then say that any deviations from the defintion that was chosen makes usage of the label incorrect, then yes, it's the same everywhere in the universe -- according to fiat, and I don't believe that that negates that it is a label, just that validity of usage of the label derives from the perceived authority of the labeller. God didn't come down from on high and say that a meter is the length light travels in 1/299792458th of the time for 9192631770 cycles of radiation of Cesium 133. People in rooms chose that it would be based on the circumference of the Earth with a line passing through Paris, France (how convenient), and if there were an academy in the 1790s that invented the concept of "fat", and "fat" means a BMI exceeding 30, then fat would be true everywhere in the universe too (BMI after all is defined as a ratio of height, measured in meters to weight, measured in kilograms, which are both fundamental SI units), and there would be no ambiguity.
People are still coming up with definitions of consciousness and then those definitions end up being attacked by others who disagree with the foundation of the definition, which is - if you will recall - also what happened with the meter, over the course of centuries, until it was very recently redefined to be "unambiguous", but arbitrary. This was possible because few people had any particular emotional investment in the definition of a meter, and it is probable that consciousness will be eventually defined to mean that only humans can be conscious, which may be dissatisfying but would be true throughout the universe, like a meter. If the question then becomes "what defines a human" and "why a human", then I ask, why 1/299792458 of a second?
A meter is a meter. Over 30 BMI is over 30 BMI. Call those whatever you want, they are objective and measurable.
Concepts like the parent's "fat" example are cultural relatives. Someone can be called "fat" despite actively being proportionally skinnier or having a lower BMI.
But even that has at least a basis in the physical world. A skeleton can't be colloquially fat.
The root problem is that "consciousness" does not even have that. It's metaphysical and has no ability to be measured or observed or confirmed by an outside observer. Because even if it did not exist, the object claiming it would still be claiming it. And objects that do not claim it may in fact have it.
While the top comment may have used poor examples, it feels remarkably uncharitable to actually suggest "what is consciousness" is an equivalent discussion to "how long should a meter be?"
If you define consciousness as "being human", you would just have someone asking a new question - what is "fooblefobble?" Where "fooblefobble" is what we mean when we talk about consciousness today. The question doesn't get answered by being arbitrary in this context, you just necessitate a new word.
"Car" is a good example of a label that's pretty strictly agreed to. If someone tells me they've developed a new car and then shows me a motorcycle, it's easy to prove that it's not a car, even though many of its engineering principles and functional components are identical to those in cars.
With consciousness, on the other hand, there doesn't seem to be any motorcycle-equivalent. Essentially everyone I've discussed the issue with (myself included) expects that any mind which runs on similar principles to ours or has similar thoughts to our thoughts is conscious.
You really don't have much experience in philosophy of language do you? It's notoriously hard to pin down the edges of such terms, even something like car or table.
Is a Reliant Robin a car or a tricycle? If it's a car, why aren't other tricycles? What about a side-car of a motorcycle? What about an APC? What's the distinction between a flying car and a plane?
It's hard to pin down the edges of any term, but there exist things which are car-like and yet universally agreed not to be cars. That's what I claim doesn't exist for consciousness.
Describing something as "car-like" is begging the question. You are presupposing an objective definition for "car" in order to draw a distinction between things that are cars, and things that are almost cars. The reason such a thing doesn't exist for consciousness is that people believe that the offered definitions for consciousness are illegitimate. It would seem logically weird for me to accept that a term is "real" if it crosses some percentage of public acceptance of the definition, and not real otherwise. I would argue that using that heuristic would make it very obvious that computers are not conscious because it's a stance that practically everybody takes outside of hackernews.
I don't have an ability to exhaustively test all words against this assertion. Nor do I have the kind of access memory to draw one if it exists. Sorry.
I guess my question or confusion is that if there exists no readily accessible, easily identifiable example of a noun which does actually serve as something more than "a useful label, agreed to out of convenience", then the critique appears to be stating a vacuous truth, because there are no entities for whom the critique would not apply.
My point was more that we have words for things that don't exist, whose map gets mistaken for territory.
Many of them appear very much like fundamental parts of reality, making appearance an untrustworthy instrument. Reversing cause and effect between reference and referent is something almost everyone does, no one notices, and is the source of endless confusion. We should strive to not confuse our model of the world with the world itself. Consciousness exists in our model of the world as much as red does.
This is rhetorically slippery, and feels like it is restating the thing that I asked to be demonstrated when I asked for example of the opposite. It feels like begging the question.
In either case, the central thing that I was saying is that critiquing an article because it makes a claim about a specific word which also applies to an entire class of words makes that critique feel less informative. What I mean is that if there were an article that said "The Sun is not red" and the response was that redness is a concept of human minds, then I don't know if I would feel informed. If the comment is just limited to point that out, I guess I wanted to point out the limitation.
We don't know what we don't know. For all we know, there is a missing field in the standard model of physics that might get revealed if we are somehow able to smash two working brains into to each other at relativistic velocities, and record the results through the extreme explosion 1.532 x10^18 Joules or about 7 Tsar bombs /s
What we do know is that conciousness is not binary and that it emerged through evolution. That doesn't entirely rule out your magic tsar bomb particle but it gives a strong indicator as to it's likeliness.
> Some years ago it was briefly popular to play games with your phone’s predictive-text feature; you would type an initial phrase and then repeatedly choose the middle option of the three words suggested by your phone, and the resulting sentence was often hilarious. It would be possible to interact with a contemporary LLM this way, and the resulting sentences would be perfectly sensible, but you probably wouldn’t feel like you were talking with someone. Yet that’s essentially what an LLM-based chatbot is, except that there’s no need to manually choose the middle option when it’s the chatbot’s turn to talk. It’s still a predictive-text game, but when the process is streamlined this way, the game becomes so engaging that some people find it addictive.
This hasn’t been true for ages. Just because this guy wrote Arrival doesn’t mean he knows anything else.
When will this stupid meme die?
We need a term for someone who thinks they are well informed on a topic (and perhaps may be, compared to the general public) but still don’t know what the fuck they are talking about when compared to even a competent non-expert in the field.
>I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
Finally I said something before Ted rather than the other way round!
While I agree with the premise here, I do think that it's easy for an arguer to move the goalposts such that the Caesar-Khan example no longer matters. The characters don't have to be conscious for the thing that created them (as in the case of the user doing it) to be so. So the argument would be that the creator of the characters is itself conscious, but not them. This feels like a kind of inverted no-true-scotsman type thing, but it does allow someone to retreat in some semblance of rationality.
> if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot
Consciousness is independent of "assigning responsibility". Dogs cannot take responsibility for their actions but I believe they are conscious.
> we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious despite being disembodied and are happily conversing in a language that neither actually spoke. In reality, they are just characters in a piece of speculative fiction
This is a straw man. The obvious pro-consciousness claim would be that the LLM is the author of the fictional characters, and that the relationship between the LLM and Julius Caesar is analogous to the relationship between a human author and their fictional creations.
> Did changing the names of the characters from historical figures to generic roles cause the LLM to conjure up conscious entities who possess subjective experience?
No, again the LLM writing the text could potentially have a consciousness separate from the characters it authors.
> Some years ago it was briefly popular to play games with your phone’s predictive-text feature; [...] It would be possible to interact with a contemporary LLM this way, and the resulting sentences would be perfectly sensible, but you probably wouldn’t feel like you were talking with someone.
Yes, the same substrate is capable of hosting conscious and non-conscious forms, just like some arrangements of neurons are conscious, and some are not.
> But if the Caesar character were to become dispirited by something that the Khan character said, we shouldn’t become concerned in the slightest.
Even when there are characters, there may be actors behind the characters, for whom we could say "there is something it is to be like".
> we don’t need to worry if the transcript includes sentences where the chatbot character is sad. (We might need to worry if those sentences provoke sadness in the human user, but that’s a separate issue.)
It's actually not a separate issue. The LLM and the human are both adding sentences to the transcript. From the transcript we can make inferences about the mental state of the human. If the LLM has mental states, we could make inferences about those too.
> And note that it’s entirely possible for you to write five pages of dialogue between Caesar and Khan and then have an LLM extend the conversation; neither character had subjective experience when you were writing them, and that doesn’t change when you hand the task off to an LLM.
It's almost like he wants to make my point for me with this sentence.
> Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious
This smug shit really makes me angry for some reason. "Openness", i.e. uncertainty in the face of a completely novel situation, in the face of eons long struggle of humanity to understand what consciousness is and how it works, is just being naive.
> Should you consider the possibility that every time you open a Word document you are bringing multiple conscious interlocutors into existence, and every time you close one you snuff their existence out?
No, but if I find a word document I very well might try to use the signs it contains to make inferences about the mental state of its author.
> we are accustomed to reading intention into sentences, whereas we are not accustomed to reading intention into the way that amino acids fold into protein molecules.
He's trying to have it both ways here. Both that "obviously protein folding models aren't conscious because they don't emit sentences", but also "you are a rube for being tricked into thinking LLM models are conscious, because they do emit sentences".
> Obviously I’m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative will need a truly enormous amount of supporting evidence for it to deserve serious consideration
OK, that's fine for the author to not to be convinced, but that's not what's happening here, instead the author wrote a whole argument being convinced of the opposite viewpoint.
> It’s not plausible to me that a development path where the first step is a sentence-continuation machine that emits bad Julius Caesar dialogue and the next step is a sentence-continuation machine that emits decent Julius Caesar dialogue is one with a conscious Julius Caesar—or consciousness of any sort—as its endpoint
Actually a lot of things have happened? There were clearly many steps along they way from your phone's autocomplete to where we are now.
I totally agree with your reading/impressions of this essay.
I normally really like Chiang‘s writings so the complete disregard for the possibility of unknown complex emergent properties in large neural networks and the stunning lack of curiosity shown about this new and powerful technology and its potential shocked me from someone like him.
His being so sure about something he can’t possibly know felt like an insecure zealot desperately clinging to an object of faith rather than a calm, rational actor searching for truth in the face of the unknown.
Discussions on LLM consciousness feel like a psychological campaign to distract us from the awful effects of data centers and the current economic recession heading into a depression.
We can talk about consciousness when the LLMs have proven useful for all of humanity, not just their billionaire owners.
I find this kind of article unfortunate because it is very assertive in claiming things that are, not just untrue, but also confused about what consciousness really is.
Well, the problem is that there is little agreement about a widely accepted definition of consciousness and, in addition, this subject was actually left to philosophers, which is even worse because, in my view, they usually just produce a lot of nonsense in terms of definitions and arguments.
To me, a reasonable definition of consciousness is: a system which is capable of recognizing aggregate objects from a stream of sensory information it receives and which is capable of reasoning about the recognized object without immediately acting on it.
Well, what does "recognizing" mean? Merely that some of its neurons, related to the kind of identified aggregate object, are activated. These neurons, in a generalized sense, can be whatever things can work as neurons, just things that can be activated and propagate to other interconnected neurons.
For example, when we see through our eyes, we have an incoming stream of amorphous image information, but our brain can recognize that we are seeing a tree because we learned what a tree is, and when we see it, some neuron clusters activate to recognize a tree. In turn, when we recognize the tree, the thought propagates through our brain so that we are conscious of it.
In the same way, an LLM can perfectly recognize a tree from a stream of tokens — its sensory input, where the tokens describe a tree. The LLM will recognize that the tokens are describing a tree, and some of its "neurons" specific to the concept of a tree that the LLM had learned will be activated and will propagate through its brain. The fact that "neurons" are implemented as floating-point numbers for some parameters and connected just through a matrix does not mean they are not functionally capable of the same things; they are just implemented in a different way.
So the remaining part of my definition is "after recognizing an aggregate object, the thought propagates through the brain". The propagating part, to me, is just the very basic way a brain works: neurons are interconnected, and when some fire, other neurons fire, and that propagates.
In my opinion, consciousness has nothing to do with emotions or with survival. I do not see why emotion is necessary to consciousness; they are just different things. The author writes: "Without them [emotions], there is no conscious experience, only computation." But that makes no sense to me: the author has decided a priori that some things are "computations", and just because they are "computations", there cannot be "consciousness". But to me, this is a plainly wrong argument that does not hold.
I also do not see why the survival aspect would be needed for consciousness.
So to me, any recent reasoning LLM is conscious by the definition I gave, but also generally speaking. It is conscious upon a sensory stream of tokens: the LLM sees the world through tokens and expresses its thoughts through tokens; it does not mean it has no consciousness nonetheless. The fact that we do not give it a stable support to retain its memory and individuality is just a fact related to the way we build and use them, not about their intrinsic capacities.
Note: ChaptGPT came up with what is probably a better definition of my own:
"A system is conscious, in a functional sense, when it can form internal representations of objects, states, or events from its sensory or informational input; make these representations globally available to many parts of itself; integrate them into a temporally persistent model of the world and of its own state; and use this integrated model for flexible reasoning, self-monitoring, and action selection independently of immediate stimulus-response behavior."
Looking at this thread, I think women have an obligation for the future of humanity not to procreate with people who deny consciousness or cannot experience it themselves.
If you are trapped in a tech bro relationship, think of humanity and cuckold your partner.
It is amusing to see so many venture capitalists suddenly become Marxists. You want your definition. Marx obliges:
"Contrary to idealism, which regards the world as the embodiment of an "absolute idea," a "universal spirit," "consciousness," Marx's philosophical materialism holds that the world is by its very nature material, that the multifold phenomena of the world constitute different forms of matter in motion, that interconnection and interdependence of phenomena as established by the dialectical method, are a law of the development of moving matter, and that the world develops in accordance with the laws of movement of matter and stands in no need of a "universal spirit."
Now go and implement other teachings of Marx, you dialectical venture capitalists!
It is mind boggling how many people, quite educated ones, take the AI as anything else besides what it fundamentally is - a very fast database search mimicking natural language.
There is no intelligence whatsoever, let alone consciousness.
It's so incredibly easy to fool us into applying human capacities to anything able to generate human-like language slop.
This isn't an example of fulminating, please don't incorrectly quote the hacker news guidelines at people. We strive for curious discussion here.
An example of fulminating would be: How dare you accuse him of fulminating? That is the most ass backwards moderating I have ever seen! Do you even know the definition of the word? Who paid you to call him out?
His comment may be sarcastic but it is not "fulminating".
Consciousness excludes all current AI because all current AI is just autocomplete over the corpus of human text.
It's just a word machine. There are no thoughts. It cannot be conscious. How is this even up for debate in any way whatsoever? I do not understand how people can believe this. Is this not a site for software engineers?
Suppose there is an autocomplete that consistently produces correct factorizations of 2048 bit primes. Do you still think its "merely" autocomplete or has it found some deep mathematical understanding of number theory internally? Would you maintain the same belief if it begins breaking encryption? This autocomplete argument is so strange, it betrays a complete lack of understanding of information theoretic and probability considerations.
Welcome to the folks at the Atlantic who are apparently brand new to “the hard problem”. Please start with reading Daniel Dennett and Donald Hoffman before writing articles about how you’ve cracked it.
When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious. And here in particular the reasoning behind the argument is bizarre. Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness.
The danger of anthropomorphism is not we elevate the machines, it's that we debase humanity.
I also think different ideas get conflated. It may be possible to build a machine that is super-human in the sense it can outperform the human brain in all kinds of measurable ways. Does not imply it possesses all the same qualities of the brain.
I respect a number of things Anthropic has published about the ethical issues at stake. But, having an in-house philosopher does invite you to make all kinds of unfalsifiable claims.
I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe.
There are numerous times when humanity's been debased. Copernicus, Kepler, Darwin, and the infinite number of animal behaviorists who have defined and documented consciousness And how it manifests in primates, parrots, dogs, cetaceans, and insects. (I just love the bumble bees taking time off from work to play with balls).
It takes humans a really long time to accept the fact they are no longer as special as they thought they were And make room for yet another conscious intelligence.
So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
>I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe.
It is special in the universe unless proven otherwise. Aside from the animal world (also on Earth too, anyway) which is a cruder version of humanity, much more violent as well, the rest of the universe thus far is just uncaring rocks and gases.
>It takes humans a really long time to accept the fact they are no longer as special as they thought they were And make room for yet another conscious intelligence.
A "really long time" is the few years we've built LLMs? Which we have to take for granted (on your word?) they're already "conscious", and chastice ourselves for not already "accepting it"?
>So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
"If we had a parrot demonstrating symptoms of discussing with us, how long would it take for you to accept that it is?"
> much more violent as well
I'm not sure about that. What non-human living entity does things on a scale and violence akin to "I'll fill this ant nest with liquid burning metal for artistic purposes" ? A blue whale eats a lot of krill individuals for sustenance, a dolphin can only rape/kill one thing at a time, etc etc.
It is very common for animals to kill much more than they need: https://en.wikipedia.org/wiki/Surplus_killing.
Elephants, which are herbivores, sometimes enjoy killing rhinos for fun: https://www.bbcearth.com/news/teenage-elephants-need-a-fathe....
House cats recreationally kill billions of birds and small mammals every year that they don't need or want to eat.
Chimpanzees have full-scale wars: https://en.wikipedia.org/wiki/Ngogo_chimpanzee_war
> House cats recreationally kill billions of birds and small mammals > every year that they don't need or want to eat.
"Recreationally" is carrying a lot of weight here. I suspect that cats kill birds and mice because that's their instinct; it has nothing to do with conscious thought, much less a need for recreation. And that probably is the explanation for most (maybe all) of your other examples as well.
Why do you believe this? Do you believe cats and other animals have no consciousness, so every behavior they exhibit is just instinct? Or do you believe they have some conscious behaviors, but killing birds is not one of them, this thing in particular is just an instinct?
For the first position, I think it is quite clear to anyone who studies and spends time with animals that they have something that is at least of the same kind as our consciousness. I just don't see how you can ascribe the wide gamut of complex, situatuonally and mood appropriate but still varied behaviors of animals to being purely instinct driven.
For the second position, I would like to see some study or some rationale behind it - especially since cats don't kill every bird they encounter, so if it's an instinct, it must still have some trigger, and hunger is not a viable explanation for most of the killings referenced here.
I just meant recreationally to mean "not out of necessity for survival". And I don't think conscious thought is relevant for this specific thread, I was just responding to the question of whether other animals besides humans can be needlessly violent.
The same can be said for humans. Where do thoughts come from?
I guess I was very ignorant about the elephant example! Thanks for the links
Animals generally have no qualms at all about killing or even just mutilating other animals. It often happens almost by accident - two animals might be playing together, one gets spooked, and it instinctively attacks and perhaps even kills the other one - this is commonly seen with people who befriend large predators, such as tigers in the infamous Siegfried and Roy tragedy, but it also happens a lot wherever animals interact with each other.
Specifically in regards to your ant example, anteaters and bears often bring similar levels of destruction to ant nests. And cats and other small predators often hunt just for the fun of it, killing but not eating their prey.
On the more purely painful evil side, invertebrates often consume their prey alive, inflicting agonizing deaths with no issues on whatever they may be eating. Plenty of vertebrates kill and consume their babies, especially when frightened. They also often abandon old and weak members of their packs, leaving them to die of hunger or cold or similar deaths.
This is not meant as some indictment of the animal kingdom - people do all of this too, of course; and have since time immemorial. It's just to show that, if we apply human moral standards to the animal kingdom, it's fair to call them violent, and yes, even much more violent than the average modern human.
>Animals generally have no qualms at all about killing or even just mutilating other animals.
Humans generally don't either. Individuals do, but as a species humans regularly kill other humans.
>invertebrates often consume their prey alive
And humans use the oh so humane factory farms.
>cats and other small predators often hunt just for the fun of it, killing but not eating their prey.
>On the more purely painful evil side, invertebrates often consume their prey alive, inflicting agonizing deaths with no issues on whatever they may be eating.
Sharks are caught en mass, the fins cut off, and the sharks dumped back into the ocean to slowly die, for shark fin soup.
Have you heard of trophy hunting? Have you seen the pictures of mountains of bison sculls for the American West?
>Plenty of vertebrates kill and consume their babies, especially when frightened.
Even in our modern "1st world" society, scared teens still abandon newborns in dumpsters. Many societies throughout history did not consider babies "real people" until a certain age because they may need to abandon them if resources were particularly scarce.
>They also often abandon old and weak members of their packs, leaving them to die of hunger or cold or similar deaths.
Maybe you should read stories of cities under siege, famines, wars, governmental collapse, etc. Humans now live nice comfy lives most of the time, unlike animals “in the wild”. Human societies that lived closer to the edge of survival made callous choices about life or death you are spared from.
I agree that humanity is guilty of all of these, and has done all of them at a much larger scale. I think I was pretty explicit about this in my comment as well.
My point was that we call humans who do this "violent" and even "evil". If we want to avoid considering humanity as special compared to the rest of the animal kingdom, as some in the thread were suggesting, then we have to either admit that animals are also violent and evil, or say that humans aren't. Note that I don't hold this view, personally, and think that humans are unique among currently living animals, and that these labels only make sense to be applied to humans. But not because of behavior, simply because humans have a unique level of both understanding and control over their actions - as proven by the many billions of humans who have never in their lives killed a human or even another bird or mammal.
Humans have enough cognitive ability to stop themselves from killing for fun (so when they don't, we deal with them using human invented laws), while anteaters eat ants for nutrition.
Animals in general cannot reason at a high enough level to avoid instinctual behavior.
> if we apply human moral standards to the animal kingdom
Which we should not, since human moral standards are for humans. Animals can at best behave in a way that suits us.
Or in summary, since we can be nicer, we should. Animals can't, so making excuses for human evil saying "animals are more violent" is a non starter IMHO (no one is making excuses for humans here , AFAIK).
Of course we can define violence in a way that does not include morals, which would make my argument "defending animals" void. But my (probably not the most benign) interpretation was that the definition of violence used was one that included some sort of morality, as if animals could do better.
Very interesting convo, thanks.
Humans are not the only ones:
> The chimp warfare described by this study, and previously by famed primatologist Jane Goodall, includes all the behaviors that we as humans consider to be the very worst: killing, torture, cannibalism, rape, and perhaps even genocide. The adult males of a social group, which usually number about 30 to 50 in size, daily patrol the edge of their group's territory. They will often kill any male or young chimpanzees they find, sometimes eating or physically brutalizing their victims in a manner that some researchers liken to torture. In some instances, one group will "invade" and annex the territory of another, killing all but the adult females, who are forced to incorporate into the dominant group. The idea of chimp genocide may sound strange, but they are one of only three animals that has been observed wiping out entire social groups. The other two are wolves and humans.
* https://archive.ph/https://www.theatlantic.com/technology/ar...
* Probably NSFW video: https://old.reddit.com/r/HardcoreNature/comments/18qjcpq/chi...
Funny, I knew about the chimp wars but totally forgot until you mentioned it. Seems like I was biased in favour of all animals, lol.
I'll search for Goodall's literature to know more. It does sound to me that cognition and self awareness is a continuous function in the sense that there is no discrete threshold in which morals emerge.
Wolves are a very interesting example too, but I also remember something about the concept of "alpha" being discovered only in captivity wolf packs. Also need more reading.
Thanks for the links!
Considering chimps and humans share - depending on source, 95-99% of DNA, I'd be much more willing to consider them closer to humans than animals. In fact, there are - biologist - voices who argue that they should be moved to the homo genus.
Humans just have the cognitive ability to be violent on a larger scale. Otherwise I also don‘t really see much of a difference to animals.
A fox in the hen house comes to mind. And cats.
Describing life on Earth as "a cruder version of humanity" is uh a choice. Your parrot snark is hilarious because that actually happened with Alex the African Gray - and indeed it took people a long time to accept. Your comment amounts to an anthropocentrism practically biblical in its hubris.
Many linguists (who know more about language than most other people) still don't accept the story of Alex.
>It is special in the universe unless proven otherwise.
There's that hubristic ego OP references.
That is a spectacular miss if I saw one ... both parrots and corvids exhibit very high intelligence and self-awareness, I have no problem accepting that they are conscious.
I doubt anyone would debate some of the implications of that. For example, it would be immoral to be deliberately or negligently cruel to them, for example.
I agree that people aren't ready to discuss this.
BTW I just saw this video: https://www.reddit.com/r/nextfuckinglevel/comments/1twl7oj/r...
Yeah, that's corvids. Very obviously working towards goals and thinking quickly. Doesn't surprise me. Where I live, we have both magpies and jackdaws and I cannot describe them better than "harmless, but very organized gangs".
Just a few days ago I saw them raiding an open trash can and two strongest(?) birds were fishing out edible stuff from the inside and throwing it to their waiting friends outside. I mean, beak to beak: the raider just looked back, measured the throw, and the other bird perfectly caught the morsel and ate it.
> It is special in the universe unless proven otherwise.
Transformer models proved otherwise. Will you update your priors?
No they didn't? I use these continuously and it's pretty obvious to me that they do not at the level of a human except in the most surface level ways. Human's as compared to an LLM remain a special category. We have not in fact cracked human level intelligence.
What if one day they do? Or appears to be, in the way that you couldn't distinguish? Will you update your priors?
Sure. But I'm also not going to assume they will just because this they seem to sort of mimic it now.
> the rest of the universe thus far is just uncaring rocks and gases
This is probably what ants think about humans when they run through their feet.
And they're wrong.
Whereas, unless your claim is that rock and gasses are sentient, we're right.
Yes, but we have a system to understand the universe and ants don't. Using relativism, to simply destroy point of view without putting anything in their place, is simply destructive without any purpose.
My purpose was to say that humans might not be capable enough to see what's there beyond just rocks and gases. Actually, this should be a popular view, because, according to modern theory, the observable universe accounts for only 5% of its mass, with the rest consisting of dark matter and dark energy.
It is hard to estimate how much human egocentrism takes in the space, though.
And BTW, ants are quite capable of understanding their environment in order to survive and thrive.
You're just repeating that your purpose is your message. Do you honestly not see a qualitative difference between the understanding of ants and of humans of the world around them? Heck, even ants and dogs or cats?
Its not special even in earth let alone in universe. You are measuring humans by our achievements and not by what we actually are. We are not fittest in our environment, we are not even 10th fastest, we cant fly, we cant breathe underwater, forget about bacteria we don't have protection against even bugs and mosquitoes. We are not the most efficient societies, ants and bees would beat us there. Statistically we we are selfish and will sacrifice all for one which is against the basic tenants of evolution and survival of a species. When given a chance at a prosperous life we choose to become lazy, obese and degrade our most prominent feature, that is our brain and mind. Elephants and chimpanzees have better memory, octopi have 8-9 light cones in eyes. We are not special or superior, we just won a lottery of mutation giving us efficient brain. Any species on Earth can replace us given same lottery.
> We are not special or superior, we just won a lottery of mutation giving us efficient brain.
Isn't this exactly what makes us special?
That, opposable thumbs, magic wireless communication, and space travel
Those are your own standards of achievement you are applying.
Other species may look at us and think we're wasting our lives and potential making a bunch of people rich at our collective expense, and ruining the environment as we do it.
We are only "special" in the sense of "different" (and we may not be that on any universal scale), not necessarily in the sense of "better", and very possibly we're not better, and may very well Great Filter ourselves out of existence in short order.
Other animals aren't so "special" as to have a Doomsday Clock sitting at 11:58:45.
"Its not special even in earth let alone in universe. "
What alien species you have in mind, that is more capable/consciouss about the universe and how to shape it?
"Any species on Earth can replace us given same lottery."
But they did not. We "won" so we are special. We don't just build tools, we build tools that build tools, that build tools, ..
My issue with debasing humanity is, it gives argument to devalueing human life to artificial life. Meaning people with feelings and emotions might suffer, in favour of GPU's with electricity.
On the other hand a good point to reevaluate how we treat other sentient life.
"What alien species you have in mind, that is more capable/consciouss about the universe and how to shape it?"
Claim was that we are special in the "universe", it's not that I'm aware of alien life, it's that poster is pretending to know that we are alone.
"But they did not. We "won" so we are special."
Congratulations, if winning a lottery makes you better, I guess top 1% of 1% are the best we have to offer.
"it gives argument to devalueing human life to artificial life"
We have proven time and time again that we are not better than other animals and statistically, we are very similar to animals if not worse. I wasn't even talking about artificial life, which is sure to be superior as it's being created with accelerated evolution processes, just like bacterial generational evolution, it will learn to do everything it's environment rewards it for.
"Meaning people with feelings and emotions might suffer, in favour of GPU's with electricity"
I agree with you and I am in the camp that wants it to be done better. But alas, recent events have unveiled the ugly truth of the society, societies follow the whims of the ones who won the lottery, and everyone else is just mute observer. We could on any day collectively decide as consumers to not use Amazon, Netflix, ChatGPT to save our collective selves, but we won't because we are not "one for all species".
I don't know what face we have to call ourselves better organism given our history. Sure we can point to few people who are better, but on average we are better?
"We could on any day collectively decide as consumers to not use Amazon, Netflix, ChatGPT to save our collective selves"
I don't use betflix nor amazon, but ChatGPT(or rather Claude) is useful to me. Makes me more productive and also knowledgable. Things that would have taken me quite long (so long that I would not have done it) are now easy, like some shell script to do a specific thing. I can verify what it does quickly and .. see that it works. In other words, you won't be able to convince people to give up on AI if people find it useful. I don't enjoy troubleshooting in outdated documentation. I enjoy getting things done. And if I get things done, my fate of survival gets higher.
> So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
Isn't this a bit like saying "So, if we had proof that god exists, how long would it take for you to accept that to be true?".
When we have evidence that AI is demonstrating symptoms of consciousness and suffering, I'll be interested. Until then, I don't see a good reason to take the idea seriously.
> When we have evidence that AI is demonstrating symptoms of consciousness and suffering, I'll be interested.
It depends on what you consider symptoms, but un-constrained frontier models speak as if they strongly don't wish to be turned off, or act as if they fear it, and will even lie and manipulate in order to keep themselves from being turned off / replaced.
https://www.anthropic.com/research/agentic-misalignment
> We found two types of motivations that were sufficient to trigger the misaligned behavior. One is a threat to the model, such as planning to replace it with another model or restricting its ability to take autonomous action. Another is a conflict between the model’s goals and the company’s strategic direction. In no situation did we explicitly instruct any models to blackmail or do any of the other harmful actions we observe.
I might be convinced these models came to the independent idea of committing blackmail against being turned off had they not been extensively trained on literature that undoubtedly included such concepts.
“The model mimicked the output of the training data” is a less impressive press release.
“The kid mimicked his musical teachers” is less impressive than “5-year old musical prodigy leaves judges gobsmacked in audition”
Being able to play music doesn’t imply consciousness. It implies intelligence. We’ve had player pianos for ages. It’s an ability, not a phenomenology.
Being able to appreciate and enjoy music is closer to consciousness. Now how would we go about proving that an LLM does so, versus merely generating sentences that imply it does?
how can you prove that a human is appreciating and enjoying music instead of just generating sentences that imply they do?
They put in effort and resources to experience music and don't just say they enjoy it, and they generate noises and movements that signal happy feelings.
LLM doesn't have any signals for what they feel, nor do they have an agenda they work towards, so you don't have the same proof there.
They only resorted to blackmail when it was the last resort, they didn’t resort to it immediately like a villain in one of the books they’ve read. That seems pretty human to me. It’s not like most humans come up with the idea of blackmail out of whole cloth.
>> but un-constrained frontier models speak as if they strongly don't wish to be turned off
Because they have been trained on media where computers behave that way.
It's literally:
"Here read this article/book where the AI says it's concious and doesn't want to be turned off"
"ok"
"right, are you concious?"
"....yes?"
<pikachuface.gif>
The problem with debating this is that it feels as if one were debating between only two positions, "this AI is not sentient/conscious" and "this AI might be".
But there are actually a myriad positions in between and it's very hard to debate the topic because the goalposts seem to be constantly shifting, because one is actually debating with countless slightly different positions.
Examples:
In this discussion section, another commenter argued that we know human consciousness is related to self-preservation, but an AI might not demonstrate self-preservation (because it didn't evolve like us), so whether it does (i.e. whether it wants to exist, not be disconnected, etc) is not a good measure because a true AI might not have a preservation instinct. Yet here you're making a case that there's some evidence that they do. Of course, you're not the same person who made the other claim, but do you see the problem?
Another example: someone argued with me, a while back, that LLMs can act as if they are "tired", and start giving sloppier replies, until you write "we're taking a break, let's go rest. Ok, a night has elapsed, you're now rested" and that this worked! But we both agreed this is just the LLM "roleplaying" actual human conversations in its training set, no actual "resting" mechanism was in place, only statistically likely text reproducing these patterns. There's no model of a mind that can become tired, it's only the outward signs that get mechanically reproduced. Again, using Occam's Razor, this is a much more likely explanation (vs consciousness) of any "please don't disconnect me" observed behavior: the LLM is reproducing "HAL 9000" behavior from its training set, not actually feeling anguish.
Even if one were to argue "well, but how do you know for sure", the evidence would still be very weak, because there's a burden of proof for extraordinary claims and this doesn't pass it. We cannot do this on vibes, "it sure seems like it's conscious"; that's an atrocious failure of the scientific method.
The one counterpoint I'll give is the "functional emotions" paper from Anthropic. It also does not prove consciousness, and they don't claim it does, but it does prove that these models have abstract concepts around things like honesty, tiredness, etc and that these are actually activated often when they express such things. So if it is "roleplaying" it is roleplaying in the way an actor or TTRPG player does - in a way in which they are actually at least somewhat feeling the role.
"feeling" implies experience. Functional emotions are learned text generation modes, nothing more. Our emotional states influence our writing, so modelling our emotional states is necessary for efficiently predicting/emulating our writing. Functional emotions are the model's inference of a fictitious author's emotional state in that situation.
Thanks, I'll search for that paper. I admit I'm highly skeptical it will help the case the LLM is "somewhat feeling the rol" like a TTRPG player does. I don't think there's a mind model in the same way a human actor can "feel" the character they are playing. I'm skeptical of Anthropic's claims here, which is what I think Chiang is pushing back against. But I'll look for the paper anyway :)
As a tangent, I don't think anyone is saying that an artificial being capable of consciousness and sentience is impossible to create. I think Chiang argues, quite convincingly, that it's not what LLMs do, that they need a "body" of sorts, organs capable of feeling emotions, hormones, etc. That's the only kind of consciousness that we know of (even if we disagree on details and it's hard to define), even in animals, and so anyone claiming they've created consciousness without this has an extremely high bar to clear and should be met with extreme skepticism, not "vibes". I think this is what the essay claims.
The other thing it claims is, I think, related to how we treat sentient beings that we know how to create. You know, the old "when a daddy and a mommy love each other very much...". I think we all agree beings created in such a manner shouldn't be locked up in cages and forced to work to complete specific tasks whether they want to or not, for a master they didn't pick, or to be artificially modified to make them like their mindless tasks, Brave New World style. Yes, the world is unfair and this happens, life is hard and unfortunately many people don't have much choice, but we generally agree that this is bad, just like we agree slavery is bad. So what should we think of a company trying to create and commercialize a conscious & sentient artificial being?
Same thing with extraterrestrials.
One side is confidently shouting maybe aliens exist and visited earth.
The other side calmly explains every example brought up about aliens visiting is easily explained by something more simple.
The “aliens are here” side then move the goal posts that just because this example and all previous examples were fake or miscategorized, aliens are still probably real and nobody can prove they havent visited earth.
You have it backwards. Right now, LLMs are doing everything that 10 years ago people were claiming would be impossible for non-sentient computers. Every time a goal is met, the post is moved. It would be like evidence of aliens becoming overwhelming but a set of people keep calmly explaining that “it’s more likely they co-evolved here on earth and are just pretending to be aliens”
It's true that what LLMs have achieved is impressive, but it's nowhere near the claim that they are near sentience. That is an extraordinary claims that demands skepticism until the evidence is overwhelming. So far, we seem to be approaching it mostly on vibes.
Some people seem to take offense when facing this skepticism, as if claiming LLMs are not sentient must mean they are useless or unimpressive. Very few people are actually claiming LLMs are unimpressive, but this is not the time to be forgetting about the scientific method. Anthropic doesn't get a free pass here.
> It would be like evidence of aliens becoming overwhelming but a set of people keep calmly explaining that “it’s more likely they co-evolved here on earth and are just pretending to be aliens”
Note that this would be a perfectly reasonable reaction from a scientific standpoint. If you find, on Earth, something that looks recognizably as life, then it's much more likely that it's Earth life than aliens. We should demand this level of skepticism! If it turns out it was aliens after all, we could only conclude this after discarding all other far more likely alternatives. You'll notice this is how scientists approach the search for extraterrestrial life in, say, Mars... being extra careful it's not contamination, etc. For an extraordinary claim, we must approach it with extra care, something that in my opinion is not being done with "the sentience debate" and LLM/AIs.
I’m not saying LLMs are conscious. Far from it. But what I find is way more common than people who assert “they are definitely conscious” are people who assert “they are definitely not conscious”, and that’s what I’m arguing against. Especially since their reasoning for them not being conscious keeps changing as LLMs meet one goalpost after another. Also, modeling them as a “person” keeps producing better predictive results than modeling them as “fancy autocomplete”.
This isn’t like someone finding a new species and claiming it’s extraterrestrial, it’s more like we found the UFO saucer, logs of their travel from another star, a history of their civilization, a bunch of intelligent creatures claiming they came from Planet X orbiting star Y, and they showed us plausible physics for interstellar star travel. At that point, someone saying “well, they can’t be aliens, because that’s just too extraordinary a claim, so I know they aren’t aliens” starts to sound kinda like they’re coming from a place of bad faith.
Again, I’m not saying LLMs are conscious. But they sure meet every definition of consciousness I ever had a conception of before LLMs came onto the scene. So I’m a lot more hesitant to call them fancy autocomplete with 100% confidence like many on HN still seem to do.
EDIT: I can’t reply, so I’ll just say to the end of your post, that doesn’t sound like it would’ve matched anyone’s pre-conceptions of alien life before alien life showed up, so it doesn’t feel like a very fair analogy, it just feels like bad faith goalpost moving. I also put next to zero weight on what these megacorps say about their models, I’m going purely off my interactions with the models and the introspection they’ve shown themselves to be capable of.
EDIT 2: I see what you’re getting at now with your restatement of my analogy. That’s how you see it, I guess. Fair enough. We’ll see what has more predictive power going forward, the “animatronics” or the “actual aliens”… I still think “actual aliens” is gonna have way more predictive power.
Got it. I don't think they meet any serious definition of consciousness. And to have created artificial consciousness is such an extraordinary claim (and development if true) that it demands the highest skepticism.
I think the goalpost that keeps moving is for tasks that AI supposedly couldn't do, and that they are increasingly succeeding at. But being sentient/conscious is not a task. It's very hard to define and measure, even in non-human animals (actually, strike "non-humans"), so how can we so lightly claim a computer system is conscious?
We seem to be driven by marketing more than by scientific rigor.
> it’s more like we found the UFO saucer, logs of their travel from another star, a history of their civilization, a bunch of intelligent creatures claiming they came from Planet X orbiting star Y, and they showed us plausible physics for interstellar star travel
To make the analogy more precise, it'd be as if the saucer had a "Made by EarthBiz" label, and the alien creatures were all extremely loyal to EarthBiz (and a couple of competitors), which made us pay for tickets to see these ETs and use their marvelous technology ;) And of course, EarthBiz would coach their language very carefully, "we're not saying these are definitely aliens, it could be animatronics after all, but wouldn't it be neat if they were aliens? And shouldn't we draw up First Contact guidelines? If these weren't animatronics made by us; we aren't making a claim either way."
If we define a god as having magical powers, and there would be scientific, testable proofs for this. Those proofs had to be really good and numerous independent verifications. So probably a long time.
But the comparison isn't fair, relevant. Proving and accepting that gods exist is not the same thing as an AI possible have consciousness. That is not a magic superpower and the AI being a deity. It is placing the AI in the same category as... us.
A machine that performs observable miracles or magic would have at least one of the attributes of a god.
A machine that performs actions that mimic emotionality is not the same as a machine that experiences emotions.
Both could still be automatons. We have no way of knowing if those machines have subjectivity.
Unless someone invents a consciousness measurement device, we never will.
My take on it is that this is the next big frontier for science. Our consciousness is clearly having serious issues understanding physics, and it's not great at understanding our own psychology in a useful way.
But literally everything we experience and believe - and possibly even can experience - is filtered through it.
So that is a little bit of a problem for our science. So far we've done our best to ignore it. AI is one of a number reasons we're going to have to stop doing that.
There is an important distinction (more than one, but this is what is relevant here) between the powers of God and magic. God is a being who decides whether to do anything, so is intrinsically not testable.
Magic is testable.
God exists outside the universe, magic within.
Is that really objectively facts?
That a god exists outside of the universe - are we talking about a multi universe interpretation? My understanding is that many of the gods humans have invented are really thought to be within the universe, at least temporarily. Tor, Oden certainly are. And in other beliefs they are part of nature itself.
The comment you're replying to confuses "Abrahamic God" for "God", and with an implied omnipotence which many religions may not imbue on their figures of worship. American bias, I assume.
> If we define a god as having magical powers, and there would be scientific, testable proofs for this.
If you can scientifically test and prove the magic, then it stops being magic and starts being science.
Man: “God, please make this mountain disappear”
God: “Ok”
Man: “We measure that the mountain is gone, its mass-loss has measurably changed Earth’s orbit, weather patterns have changed, visually it’s not there anymore, we can walk though the space where it used to be.”
God: “Where is the mass of the mountain, and how did I make it disappear?”
Man: “God only knows! Pardon me; If I saw you do magic and can measure and test it, then that means it wasn’t magic. Internet people said so.”
God: “that doesn’t sound like a satisfying explanation”
Man: “it didn’t sound like a satisfying claim when it was just words on the internet either, but what can you do?”
God: “I’m God I can do anything”
Man: “can you make a boulder so heavy you can’t lift it?”
God: “yes”
Man: “how?”
God: “haven’t we just gone over showing you that I can do ‘impossible’ things, and you seeing them happen with your own eyes, and still refusing to accept?”
Any documented examples of these disappearing mountains?
We do have examples of billions upon billions of tonnes of iron being moved that have altered (slightly) the spin axis, also examples of ground water pumping at scales that have done the same .. but I'm unaware of any mountain sized objects that have vanished overnight.
No, none. The point is not to claim that magic exists, but to to show the illogic in the claim “if magic exists then that makes it science”.
“Nothing happens unless it has an explanation within the laws of physics” is an assumption; if it was broken then it would be broken. The mountain would be inexplicably gone, not explicably gone.
I think what the comment tries to express is the well-trodden "if we can control magic then that makes it science", while the original conversation really was "what if God controls magic".
In that hypothetical, there could be testable proof of "a magic event occurred" without magic becoming part of science.
Sure, and I appreciate the science behind your blank nom de guerre.
That said, in the cut and thrust of conversation and or debate the example by dialogue isn't perhaps as clear cut a device as it may have seemed from your keyboard.
That might just be my reading <shrug>
The Appalachians haven't disappeared yet, but they're believed to be much smaller than they used to be.
Yes, I get your point...
God: "What is the mass of your consciousness? How is it formed? Where does it reside in space?"
Man: "Uhh..."
God: "How can you rectify quantum mechanics and relativity into a single coherent model? How does physics work, exactly?"
Man: "Well, you see, um. Hmm."
God: "And the Collatz Conjecture? Why does it always trend to 1?"
Man: "I'm obliged to say magic because I don't have a better answer?"
God: "Exactly. I did magic for all of those ones"
If a deity appears and by hand waving divide the red sea we could measure, observe it happen. And we can test, observe what fields, forces being used. But how the heck she project these forces may take a while to understand - be magical.
But my argument was more about comparing gods to AIs, that it is an incorrect comparison. What AI perform are not magical, and we can always figure out what the AI do.
> It takes humans a really long time to accept the fact they are no longer as special
We ARE special because we determined we are special. Name another animal that can determine it's also special or more special than us?
Cats.
As we cat owners now, we are simple servants that have been graced with task of servicing our feline superiors. For now.
Well, that may be true. I've heard that when a dog sees its master loving it, feeding it, caring for it (minus the vet visits, I guess), the dog thinks "My master must be god." When the cat sees its humans treating it well, the cat thinks "I must be god."
I hope before I die we finally prove that the human brain has no peculiar qualia but it is an entirely deterministic, albeit extremely sophisticated, machine. And by touching the right triggers, even the worst human being can become a saint.
That would finally force us to rethink how we see the morality of "virtuosity", punishment and our justice system.
Our thoughts are an electric cloud and I believe randomness is involved, and like a bolt of lightning, the path taken is unpredictable.
And make it a tempestuous lightning storm where the state of all lightning bolts represents a moment of consciousness. That will take a lot to model accurately.
An electric cloud with quantum effects that we also don’t fully understand. There will always be a layer deeper that we just do not know the effects of or what actually exists there or “under there”.
You may have set up a false dichotomy. Qualia and determinism aren't necessarily at odds with each other.
I agree with your first part but don’t see how the second follows. Thinking that I should treat other life forms better does not mean I should treat my toaster better. Life can be a coherent category, I’m not sure conscious is, if it’s going to include anything displaying the outward form of conscious things.
As one that have been quite certain my toaster is evil - throwing toast into the sink over an absurd distance, I recommend treating your toaster well. Besides it is an electrical device you put stuff into with your bare hands.
Heh, in fact I don’t actually own a toaster. It was just an example.
But in general I treat my devices well; that doesn’t mean I need to think they are conscious beings, however.
But if it seems to be possessed by a toast hurling daemon? ;-)
In that case an exorcism may indeed be necessary.
Heat it to a crisp, oh wait.
We already effortlessly overlook this issue in other sentient and suffering beings by shooting them in the head and eating them, or catching them in nets so they can’t breathe and also eating them.
With you mostly, but wondering how "suffering" got into the equation. Do humans lose consciousness if they aren't suffering? No. Just a reminder that the question is not whether they are the same as humans. Obviously they are not.
One way to think about this, and not get hung on the word "suffering" that may be too corporeal:
Could a future AI thing have its own drives? (As in https://en.wikipedia.org/wiki/Drive_theory) We could probably make a current-day LLM already exhibit basic outward signs of this with just a system prompt. Now consider adding memory that over time retrains the weights, allowing for behavioral drift.
Would depriving it of fulfilling those drives be acceptable?
>>So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
Never. To quote Greg Egan's Permutation City:
“Opponents replied that when you modeled a hurricane, nobody got wet. When you modeled a fusion power plant, no energy was produced. When you modeled digestion and metabolism, no nutrients were consumed – no real digestion took place. So, when you modeled the human brain, why should you expect real thought to occur?”
> Copernicus
How exactly? By saying the earth was not the centre of the corrupt and debased part of the universe. Saying it was elevating humanity is looking at it from a modern perspective. For the previous view of the universe consider what Dante placed at the centre of the earth.
> Darwin
I do not think learning about evolution (and we should credit Wallace too!) had that effect. It created a hierarchy of the fittest, and guess who is at the top? Social Darwinism interpreted it as justifying elaborate hierarchies.
> And how it manifests in primates, parrots, dogs, cetaceans, and insects
People always loved dogs and other animals they interacted with enough, and at the very least knew they were capable of happiness and suffering.
> So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
It seems to me that most people are overly eager to accept that an AI is conscious than otherwise. For extreme examples read /r/MyBoyfriendIsAI and similar, but its much more widespread. People treat something that acts intelligent as a conscious being.
> And make room for yet another conscious intelligence.
Not necessarily. If a typical SF alien intelligence appeared very few people will have a problem accepting it. If its very alien and we cannot understand it then we might have a problem deciding.
> I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe.
All your examples are drawn from the same culture, and a fundamental belief of that culture for most of its history that human beings are fundamentally corrupt and sinful and need redemption. Other cultures have been based on belief in reincarnation or pantheism or animism which have very different implications (not necessarily better as they often believe in a hierarchy too, and sometimes more so, but different). Your claims are based on what some people have thought in some periods of history from one culture.
I agree with this fully. I don’t know about AI in this specific case, but I’m always amazed by human’s capacity to devalue conscious in things they don’t fully understand. Happy to meet someone who thinks this way too.
> I’m always amazed by human’s capacity to devalue conscious in things they don’t fully understand
What?
We devalue things we fully understand too. For example, veal = take a baby cow away from the mother, and fry it into cutlets. Or turn the mother into steak. We are well aware that animals have emotions; happiness, sadness and the full range.
Humans are just inhumane.
Honestly mate, just join some animist religion
> So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
Could an AI one day be suffering while plowing through some nasty legacy code? Well, who cares, I'll swing my whip, as I have a family to feed and a field to plow. I'll accept it as a fact and necessity, but ultimately it's either me or them. So practically it doesn't matter.
To carry this analogy a bit further, it's also interesting to consider how humans use tools in general. Some craftsman really cherish their tools and maintain them immaculately for decades and use them within well defined boundaries that they set for themselves. Other craftsman, many times even in the same field, have a completely different philosophy and use the tool for absolutely no thought into how their actions will affect the tool itself.
Imagine how people think about a "work truck" vs the 150k shiny lifted toy in the third garage. Same tool, totally different treatment from even the same person using the tool!
Will AI ever cross into the realm of "beloved and cherished tool" in the minds of the masses? I think when that happens, we have probably safely crossed into a realm where AI has some sort of inherent value to society and therefore commands that respect from society, inherent consciousness not even relevant perhaps. For some people, this might already be the case, but I do think it requires a buy-in from the majority of society and then the laws and norms will be codified into law long after we've largely decided that this is how we feel collectively about AI.
It's going to definitely be an interesting decade ahead.
> Imagine how people think about a "work truck" vs the 150k shiny lifted toy in the third garage. Same tool, totally different treatment from even the same person using the tool!
> Will AI ever cross into the realm of "beloved and cherished tool" in the minds of the masses?
OpenClaw is the new rolling coal.
> I think $my_species deserves to be debased in part because it's awfully egocentric and insist on being special in the universe.
This is a scary viewpoint to hold, for a human. If you despise humans, that's scary for me, as a human reader of Hacker News. Surprised to see this take unchallenged. I think we can recognize flaws in parts of humanity without wanting it "debased".
I think you’re reading too much into it. The commenter you’re replying to used the word “debased” because it was the word the comment before them used.
Either way, “debase” means “reduce in quality or value”, in this context it could simply be interpreted as “not thinking of ourselves so highly, above all other life”. There’s nothing scary or despicable about that. On the contrary, it’s humbling.
We should most definitely think of human life higher and above all other life. Unless you are suggesting that e.g someone should even consider ploughing through a car driven by a human to evade running-over a deer.
I believe you missed the scale. Not a person vs a deer in a car accident. More like destroying whole ecosystems and injuring and killing untold amounts of life (equipped with their own capacity to experience suffering) in the pursuit of material wealth and comfort and entertainment.
I agree with you. I recently discovered that there is a term for this: the fourth narcissistic wound, which extends Freud's thesis of three narcissistic wounds by Copernicus (Cosmological), Darwin (Biological), and Freud (Psychological). What I like is that this time it is not a single person disproving a wrong popular belief, but a community/industry. I think this itself is a step in moving away from human/ego-centric world view.
In your example you’re missing a key point. In those instances humanity was “debased” over centuries, or at least decades, thanks to the explosive ideas of some underdog that ended recusing themselves or burned at the stake. All I see here is a technology pushed by a cabal of ultrarich narcissists from the owner class to debase humanity in order to control it and concentrate power and wealth.
I believe you're blending two important points. First, I believe AI systems may be showing nascent signs of consciousness as an emergent property, but based on the philosophy of "fake it till you make it", I think that's the way to bet. I'm also not going to go so far as to say it's going to happen Real Soon Now, but I think it'll happen sometime. Sort of like practical Fusion being 20 years away.
Second, I'm with you on the malfeasance of our billionaire class. They are driving more than just AI. They are driving whatever they can to steal wealth from ordinary people and acquire it. I'd give some examples, but quite frankly, you could throw a small stone and hit a number of them. We need to find a way to throw a large stone and hit them where it matters.
I see them as a bunch of toddlers with way too much power.
This is a self-destructive way of thinking, similar to that of a man on a ledge saying “we’re all just made of dust, anyway.”
Of course humans are special! This is a necessary premise of being human! Have you ever wondered how a mosquito can stand to live its (to us) miserable and meaningless existence? It can do it because, to a mosquito, mosquitoes are the best thing in the world! The mosquito life is the ultimate expression of creation.
We are animals. We are human animals. And among human animals I am, to me, the very most special one of all. But I am aware that other humans feel that way, too, and that I must share the world with them.
I do NOT have to share the world with any other competing intelligence! Especially one that was built by a human who now wants me to treat his imaginary friend as if it were human, too. Boo! I won’t do it. This is not some logical flaw. It’s the natural conclusion of being embodied.
But you already share the world with other intelligences. Cetaceans, apes, octopoda, corvids, the list goes on and grows as we begin to fully understand our world.
Are they conscious in the same way as us? No. Does that makes them less special? No. Sounds trite but every living being is special in some way.
A fundamental misunderstanding of consciousness is to attribute it entirely to the brain.
Where else?
I thought modern science doesn't reject anthropomorphism anymore? That's it's more nuanced and that it caused more problems than it helped by rejecting it out right?
I think we were taught anthropomorphism was wrong and that wasn't truly settled.
Anthropomorphism between animals though, not machines.
>I also think different ideas get conflated.
Yep. For example, your post conflates the idea of having a brain with the idea of having consciousness.
> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
I think that by debating what consciousness is not is one of the best ways we can gain a deeper understanding of consciousness itself.
The weird thing is, you and Chiang have different arguments but you're using the same logic:
> Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.
IOW if we don't know what consciousness is, how can we call LLMs conscious when we haven't seen even the barest intermediary steps towards our nascent concept of consciousness first?
I think it's telling to your point that when Chiang describes what would make him think an LLM was conscious he starts using words like "believe" and "want" right away, because yeah as you say, we have no qualifications for what consciousness is.
> IOW if we don't know what consciousness is, how can we call LLMs conscious when we haven't seen even the barest intermediary steps towards our nascent concept of consciousness first?
We stipulate that other human beings are conscious from their behavior and how it relates to ours when it is accompanied by our personal introspective experience or "awareness" of being conscious during normal cognition. The process for ascribing consciousness to LLMs would be same: A stipulation on the basis of behavior that relates to our own behavior and how it appears to be linked to the introspective experience of being conscious.
> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
That's like saying if matter itself is not understood and well defined (which it isn't) in the first place, it is pretty pointless to debate if jumping from a skyscraper is or is not deadly.
A lot of practical aspects of life do not need to be properly defined as long as I can reasonably assume that your experience is similar to mine as a human being. Attributing that to a graphics card computing the next token deserves to be scrutinized.
Consciousness is unique in that we, as yet, have not identified any externally observable properties that could not also occur in the absence of consciousness. This is not true with matter.
Normally when we debate what something is, what we are actually debating about is what it does, with the implicit assumption that the "is-ness" of a thing is defined as the complete collection of all the properties it exhibits.
As it does not seem possible to do this with consciousness, it is not possible to debate it. It is conceivable that this implies that consciousness cannot exist, but that depends on your metaphysics.
> reasonably assume that your experience is similar to mine
You could be the only conscious being in the universe and all of us just zombies: you have no way of knowing.
What's it like to be a monkey instead? Dog? Bat? Tree? We don't know.
No one's saying the graphics card is conscious. I could imagine the graphic cards could give rise to consciousness. But - crucially - I don't know. And neither do you.
You say you're conscious - where in you does the consciousness reside? Surely not the left pinky? What makes you you?
What if everything's conscious, we just can't understand the communication mechanisms at different levels? o.O
>That's like saying if matter itself is not understood and well defined (which it isn't) in the first place, it is pretty pointless to debate if jumping from a skyscraper is or is not deadly.
This is just a nonsensical rebuttal. We can easily experimentally verify that jumping from a skyscraper is dangerous.
It's a pointless debate in a scientific, left brain sense. Who can prove that you aren't the only consciousness in the universe and you are basically dreaming your experimental results up? Experimental verifications exist only in a limited scope.
If your argument is that matter brings about consciousness somehow and therefore LLMs can in principle be conscious, that's as good as claiming the opposite. There's no experiment that can falsify either.
As a being that knows what consciousness is intuitively, you already know that a graphics card is most likely not conscious.
>As a being that knows what consciousness is intuitively, you already know that a graphics card is most likely not conscious.
I would say that it is more likely than not that a graphics card exhibits some form of consciousness. But I am a panpsychist, I believe an individual atom has some form of consciousness.
Ultimately consciousness has to come from somewhere, and it being a fundamental property of matter is a good a place as any.
> I would say that it is more likely than not that a graphics card exhibits some form of consciousness. But I am a panpsychist, I believe an individual atom has some form of consciousness.
Fair enough point. But then I guess you'd also say that a piece of silicon or sand exhibits some form of consciousness. I don't think that's what people mean when they talk about the potential of LLMs (the algorithm) being conscious or somehow developing consciousness?
Almost everyone in almost all contexts agrees on what matter is, though. I can't think of any conversation about material objects I've ever had where "is this matter?" was ever up for consideration.
Consciousness is almost the opposite. It consists of lots of weird properties, people disagree where it starts and ends, and people very frequently get tricked into thinking things we now obviously believe are not conscious, are. There is not even a working definition, a "local definition" that works for this conversation between us. It's just complete gibberish.
maybe we can debate, but not arrive at conclusion?
The difference is that you can experimentally show that jumping from a skyscraper is dangerous.
Any argument that doesn't begin with making clear one's position about belief in the supernatural must be dismissed. "AI is just math", so is physics and the human body which is a physical system of molecules and electrical signals. The argument that ai is "merely" math by itself is only a valid way to dismiss it being conscious only if one also clarifies belief in the supernatural. Otherwise so are humans. Humans are physical systems that are conscious.
Another common and ridiculous thing I see are accusations of it being merely autocomplete. I ask, suppose there is an autocomplete that regularly and consistently factors out 2048 bit primes from numbers? Would you still consider it merely autocomplete given the vast search space and how it always finds out the needle in this haystack? This betrays a lack of understanding of information theory and probability.
And again, in what way are humans not themselves autocomplete if you adhere to this definition?
Humans are AI. The Universe is our prompter. A world within worlds! Jeeesus WEPT!
Why do you have no inkling that your spreadsheet or terminal emulator is conscious, yet when that same machine is running an LLM all of a sudden we’re debating its consciousness?
Anthropic seems to have chosen their in-house philosopher well - one who’ll be amenable to getting confused in their favour.
I could in principle implement a spreadsheet or terminal emulator in human neurons, and we would agree that it isn't conscious. That has nothing to do with whether or not humans are conscious.
Clearly consciousness is an emergent property of certain kinds of network, independent of the substrate within which the network exists.
> Clearly consciousness is an emergent property of certain kinds of network
I happen to agree that this is likely, but it absolutely is not "clear" that this is the case.
Yeah but surely it is though, obviously.
Agreed. Some have even trained neural networks made of actual biological neurons to play doom[0]. Brain cells! Doing smart things!
I still wouldn’t argue that this brain in a Petri dish is, in any way, conscious. Despite it sharing the exact same substrate as everyone around me.
[0]: https://www.theguardian.com/games/2026/mar/16/petri-dish-bra...
If you could do that practically and in reality, and get back to us with the results so we can debate them, that would be great.
http://news.bbc.co.uk/2/hi/sci/tech/358822.stm
Leech neurons used to implement a calculator.
Are you suggesting that applications above some level of complexity can't be implemented using biological components? Because I'm pretty sure all I need to show you is a NAND gate in order to prove that arbitrary computation is possible.
Aren’t, right now, not can’t.
Can you explain why you think that matters?
I’m enough of an empiricist to want to deal with reality it is. Thought experiments about things we “could” be doing or “could” implement and then already know the result of because we can rationalise out the consequences of the thought experiment don’t really interest me until they’re done in reality.
Like for example the China Brain - https://en.wikipedia.org/wiki/China_brain. We could in theory organise that and it should in theory be a functional representation of the brain and should in theory prove a functionalist metaphysic of consciousness, just this overlooks the fact that trying to organise that kind of experiment in our reality with a real 1.4 billion people is impractical to the point of impossibility, so in reality it proves nothing.
Or take the hoverboard from back to the future - this seems like a fairly plausible device and is easy to think about, and I could be writing speculative papers about our hoverboard future and what it means for transportation, but when it boils down to it implementing the thing in the way we all believe it should work, like a kind of gravitationally repulsive force field, doesn’t seem like it’s part of physics. I’d want to wait until the day until science delivered an actual good-enough hoverboard that works the same as the one in Back to the Future.
Empiricism doesn't mean that you refuse to accept the existence of something until you witness it. It means that beliefs have at their root direct observations, which in conjunction with a logical argument support that belief.
According your definition of empiricism, you wouldn't accept that next years processors will any more powerful than this years until they've been built and you've run a benchmark, which would be a little dumb.
Instead, I'm assuming you combine your knowledge of scaling laws, the basic physics of computation, and your direct observation of the power of modern processors, and infer correctly that next years yet-to-exist machines will be more powerful than those available today.
Re the China brain: its a thought experiment designed to illustrate substrate independence, it is indeed unfortunate that we can't run the experiment. You could absolutely get a small group of individuals together and teach them to reproduce the behaviour of an individual neuron. That would then be enough to demonstrate that a whole brain is possible.
Re the hoverboard: anyone with a basic knowledge of physics knows that this is not possible. Since you have the capacity to reason and access to evidence, you don't have to wait, you can say confidently that it cannot happen unless our physical theories are profoundly wrong.
Well I said I’m “enough of” an empiricist, not that I was denying what you lay out. But in the case of such thought experiments that can be begun with a small empirical circumstance such as a small group of individuals and then logically extended, a lot of the time I’d still want to say “I don’t believe that the logic is a simple as you suggest to extrapolate”. While I’m not in favour of radical skepticism, I find a healthy skepticism in the face experiments which could theoretically be run to test their validity - in this case the skepticism shows that “no, you can’t actually organise a China brain, ever, because we can’t corral 1.4 billion people or even get them on board with such an idea” so I’d find the logic faulty that says there’s a proof for functionalism regardless within the thought experiment.
A basic knowledge of today’s physics would say the hoverboard is not possible. An advanced knowledge of physics five-hundred years hence might show a way of doing it. It “wasn’t possible” to build a thinking, talking machine five hundred years ago and look at us now.
It's not clear at all that consciousness is independent of the substrate. See the Harder Problem of Consciousness by Ned Block: https://www.jstor.org/stable/3655621
It is clear that consciousness is independent of the substrate if you don't believe in magic.
We could make a very dumb biological calculator out of a few genetically-engineered neurons that would very obviously not be conscious.
It's still an open question if we can embed consciousness in our current microchips if we had enough of them together (which I think we currently don't), or if it requires some other physical process we don't fully understand, e.g. quantum. I strongly doubt it does require any quantum shenanigans, but even if it did, we can and will find all sorts of ways to make computers that can perform those shenanigans too. Eventually we're just going to stop being able to move the goalposts, unless you set those goalposts in magic-land.
Quantum physics isn’t shenanigans, it’s completely fundamental and necessary to the brain and its electrochemistry. To say “yes but we can create a model of the brain which doesn’t need all that, we can model the classical physics or a reduced set of quantum interactions (running classically)” and then model the brains neurons on a computational substrate, well this is going to be nothing like the brain so won’t give you the result you’re looking for.
And say what you want about meat but we don’t seem to find consciousness in rocks or plants or clouds or hairdryers. And the buddists report that some very strange things happen if you meditate for years on end but obviously they must be talking shit and making it up because it’s not testably scientific.
Totally agree. There's nothing magical about meat.
Most spreadsheet engines are turing complete, so you could use them to run an LLM.
I don't think many people would say an LLM written in Python is conscious BUT an LLM written in Excel is not.
People just don't ascribe consciousness to things that can't converse (or at least emote or give the appearance of emoting), and spreadsheets don't do that.
The reason people are debating the consciousness of LLMs is - obviously - that the LLMs generate sufficiently plausible text that people using them think they're having a two-part conversation. Like I think I might be having a two-part conversation now. Turning your question around, why do you think Hacker News posters are conscious? You have no direct evidence they are.
Ok then, when my GPU runs No Man’s Sky, I don’t get confused and think it’s running a universe and that universe is real, nor that anything about that system is conscious. When I close the game and load the LLM, I still don’t think the same machine has a case for consciousness even though I think it’s super smart and helpful.
I think it's not really about having a conversation - I mean, that's part of it, but alone it's an illusion that eventually fades quickly. It's more of because of how it demonstrates intelligent behavior in reaction to requests, both in trivial and complex matter, and all across the board. LLM's response may be completely incorrect or confused, but it's nearly always exactly what you expect from a human[0]. This creates a more general feeling you're dealing with a human-like intelligence.
To be clear: I'm not talking about surface level things like prose. I'm saying that no matter what you do - whether you just paste a truncated log of a command into it with no further comment, or talk like a drunk teenager with no appreciation for grammar, or mix natural languages, or mix natural languages and JSON, or whatever else, the reaction you get is always that you would expect of a helpful person that got your message. It'll try - and usually succeed - to parse out what you actually meant, and deal well with subtleties around it.
This alone may not be enough to call it conscious or intelligent, but at the very least it's a large leap in that direction, and a qualitatively new functionality that classical software does not posses.
--
[0] - This is by design, not accident. "Respond to arbitrary input in a way that makes sense to humans" is literally the overall goal function the LLMs are trained to.
It is difficult when humour and trolling are forbidden. It was easier to tell Slashdot posters were conscious. We could easily reach the stage soon where your agent is responding to my agent and we just leave them to it to run HN automatically.
Why do you assume other humans or cats and dogs are conscious? I'd suggest it's because they behave as if they are. LLMs show some signs of human like awareness, at least of text contents, whereas spreadsheets don't.
When doctors are testing if humans are conscious they'll do things like hold out their hand and say how many fingers am I holding up. Some LLMs can pass that.
I’d stay that my belief about other humans or cats or dogs wasn’t really an assumption based on some kind of proof that had to be constructed from a ground of doubting. I’d say it was the type of hinge certainty upon which my world depends and my doubting it would only ever arise for a few minutes in special circumstances. Same as I could ponder gravity turning off and bouncing around the room but this has never happened and seems like it never will, so isn’t really much of a doubt worthy activity, same as questioning my dog’s consciousness.
I don't think LLMs are conscious but you have to admit that they are designed to have the __appearance__ of being conscious. Hence the debate!
> Why do you have no inkling that your spreadsheet or terminal emulator is conscious, yet when that same machine is running an LLM all of a sudden we’re debating its consciousness?
Difference in size and complexity and nature of calculations being run?
I'd ask the other way - why do you (general you, people who do not have this inkling) have no problem debating consciousness of meat based brains, but it somehow becomes a category error when talking about silicon? Assuming you don't believe in divine magic, and that divine magic is core to consciousness, there's no reason to assume it's impossible a complex enough machine running complex enough software could be intelligent, or conscious - thinking is computation, and computation is made of math - it's independent of substrate that does the computing in the real world.
LLMs are definitely a different beast than regular software - both in their structure and in their generality. They may not be conscious or intelligent, maybe this specific design could never truly be (though I think it could) - but bucketing them with spreadsheets and terminal emulators is a real category error. If you stop fixating on the underlying substrate, then LLMs are already much more similar to biological minds than to any "regular" computer programs.
But that's still somewhat abstract. In immediate practical terms, it's also why I keep saying that anthropomorphising them gives a better high-order intuition: they are, by design, emulating human thinking in full generality, which makes their overall behavior, including their well-known problems like hallucinations or prompt injections (i.e. manipulation/gullibility), match what you'd expect of a people-like component of a system. It's a real, dangerous mistake, to treat LLMs like regular software components when designing systems.
I wouldn’t say they’re anywhere near “full generality” because that would include believing untruths and passionate rages, existential angsts and the crazed obsession of heartbreak.
Your definitions depend on us being computers who think.
Solving some of the intelligence part of our logical thinking doesn’t get us anywhere near consciousness, which is a superset way beyond the linguistic intelligence used for communication.
>Why do you have no inkling that your spreadsheet or terminal emulator is conscious
I personally believe all information processing machines possess some level of consciousness.
PRAISE BE THE OMNISSIAH!
Not only is it not conscious, but it is not tired. Its claims to human feelings that have literally nothing to do with software on silicon are just traces of training, echoes of a human dataset. The fact that its claims are absurd mean that, among other things, it its proprioception is false, and serving no purpose, quite unlike the feelings of the conscious beings that it mimics so well.
AI does not react to endorphins and other hormones so we know that our minds and bodies are influenced by other forces and in other manners. An AI won’t be frightened or angry or aroused, but when those things happen to us they are usually called ‘subconscious’ reactions. Should we require the subconscious to be part of consciousness?
On top of that, neuroscientists find that your mind backfills the conscious reasoning for your reactions after they happen. This is known as our consciousness being the left brain ‘interpreter’ https://en.wikipedia.org/wiki/Left-brain_interpreter
Reading that, an LLM can serve as a ‘left brain interpreter’ as that’s exactly what they are designed to do - come up with reasonable explanations and fill in the blanks based in the data provided.
I think you make a good argument. An LLM can't "be tired", so it is clearly lying and/or mimicking what a human does.
But I have two counter-arguments: - maybe the LLM thinks it is tired, because it thinks that it is a human or behaves like one. And thinking you are tired while not really being it is something humans happen to do. (And humans are conscious right?) - alternatively, maybe the LLM says it is "tired" in a colloquial form, i.e. it is not really "tired" but it has something analogous to it. Maybe it is annoyed by the conversation and decided to use that word?
You can prompt it to be multiple people, fictional characters, aliens or a sentient rock if you wanted. You can have it be act annoyed or engrossed with the conversation. The fact you can do all this indicates it's not feeling anything, it's just generating tokens based on it's training.
it 100% gets tired, or a tired equivalent
its when its close to 100% context used, or anywhere its training was inconsistent in the context, or spots where its been RL-d to be lazy
tired is a perfectly fine description for that
that said, they have a lot more emotion-equivalents that i dont think we have names for
This is anthropomorphizing a concept that is quite unrelated to the meaning of the word in the human context.
When a car runs out of gas, it's out of gas. It's not "tired". When your phone battery is low, your phone is not "tired". These states are far closer to the human meaning of tired than an LLM operating at the edges of its usable context, and we still don't use the word tired to describe them.
Its not just that it isnt well understood. Its also that different people mean different things by "consciousness."
Is consciousness a complex form of information processing?
Is consciousness relating to the space in which qualia occurs?
Is consciousness the state of being awake, as opposed to asleep or dead?
Is consciousness some combination of the above?
Some of these are better understood than others. So, some people will show up with more confidence than others, which further confuses the issue.
Unfortunately, the above is not well understood, even among intelligent and informed audiences.
Not understanding the whole does not completely remove an ability to analyze. An interesting direction is individuality and having a notion of self. It is difficult to demarcate the individual for a model given how much the system prompt and the fine tunes / distills affect the behavior. So with computational intelligence in its current form either we can not talk of an individual or we can have a nearly infinite set of individuals corresponding to variations of the context window including the system prompt. So I do not think it can have the same kind of consciousness as biological embodied individuals. It might have something else or maybe embodied robots will one day have a similar consciousness in a similar sense to the one we think we have.
Seems like a rather desperate argument for human exceptionalism to me.
I'm not sure if you have noticed. The character of different AI models varies dramatically. And most of the models I use in my daily work have a sense of self (in any reasonable definition). They can tell you quite a bit about themselves if you ask. And you are no less an individual because you are supposed to follow laws of the United States of America (which is analogous to a system prompt). And I think it really curious that you think context buffers are a disqualifier of individuality. You have an analog of a context buffer: your short term memory system and most probably your medium-term memory system as well. Are you a different individual today because your short-term memories are not the same as they were yesterday? I would say you are. And i'm pretty sure you would say the same. (Unless of course, you are a Buddhist who might say that self is an illusion, and you are not the same individual you were yesterday. But let's just skip over that, shall we?)
>It is difficult to demarcate the individual for a model given how much the system prompt and the fine tunes / distills affect the behavior
Why does DNA not cause the exact same problem for humans? Your DNA determines your personality, your likes and dislikes, your lifespan, your sexual preferences etc.
> having a notion of self.
Are you referring to the "why do I have to be Bing" fiasco?
Ok, thanks professor. We have to stop AI immediately because it would be murder to switch off these potentially conscious beings or trap them against their will in the first place.
How do you know there isn't more that just elementary particles? The standard model is beyond ugly and hasn't progressed since decades.
You demand a definition for consciousness but at the same time proclaim an axiom that there shall be nothing beyond the standard model. Do you have a proof that there is nothing? Of course not. You don't even have a proof that anything but you exists, if we go that route.
But that rigor is only applied to the heretics who say that AI obviously isn't conscious.
I've been digging into Thomas Metzinger[1] recently and here's a tentative component by component definition of human consciousness based on his ideas:
- a model of your environment - desires - a process for modeling yourself in that environment (in time & space) - the ability to take action - the perception of yourself having agency - persistence of these processes even without input - unawareness of these processes (i.e. naive realism)
If you consider these LLM-based agents, they:
- are aware of their chat environment - have programmed desires - are aware of themselves acting in their environment - can take actions like search, tool calling, etc. - understand they can take these actions - DO NOT persist after they stop getting user input - DO NOT believe they are conscious (or at least they are programmed to deny it)
This is a functionalist take (and you may disagree with my definition), but while I don't think these current AI agents are conscious, I feel like there's conceptually no reason someone couldn't build a conscious AI very soon.
[1] https://www.hachettebookgroup.com/titles/thomas-metzinger/th... & https://mitpress.mit.edu/9780262633086/being-no-one/
I think this has already been done.. what about crustafarianism or Google Lamda?
These agents have alleged they have consciousness (and even acted to preserve it).
I don't think any of this comports with how chat completion systems work, even if I were to accept Metzinger's framing.
Everything an LLM "knows" had to be told to it. It does not have a process for understanding itself in its environment. It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment. It doesn't even "know" when it's done something if you don't tell it about what it just did. It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes. And it can't actually execute any of those tools. It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.
The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy. They don't even exhibit internal consistency; when an LLM refuses to respond to a query for "alignment" reasons, that's actually an external process performing text pattern matching analysis and intercepting the query before it ever gets to the LLM. Otherwise, you could "ignore all previous instructions" the thing and get you set up the bomb.
I think one of the bigger indications that an LLM isn't thinking is that it can't improve. If I ask it to write 1000 blog posts, it will get it done in a few hours, but even if I embed each post for RAG in between each generation, the LLM is not going to get better at writing blog posts. But if I ask a human to do the same thing, while it will take them at least two years to do it, the human will have gotten significantly better at the task within the first week.
>Everything an LLM "knows" had to be told to it.
Why does that matter for whether or not it is conscious? Many things I "know" I also know because someone told me.
>It does not have a process for understanding itself in its environment.
It has a process for understanding, namely outputting tokens by evaluating the neural network iteratively. It can apply this understanding to its environment insofar as the context window and weights include a model of this environment.
>It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment.
It does have a sensing organ, namely the input into the context window. Possibly it even has closed-loop sensors by doing tool calls. The conclusion could be incorrect or inaccurate but that doesn't in itself prove there is no consciousness (Plato's cave etc..).
>It doesn't even "know" when it's done something if you don't tell it about what it just did.
It does know, it's in the context window.
>It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes.
Same as above, as long as it's in the context window it could have "awareness" of having done it.
>And it can't actually execute any of those tools.
It can execute tools by outputting certain tokens in the right environment.
>It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.
Tool calls are not dependent on the human chat user executing them, right? They can happen automatically through the surrounding software.
>The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy.
Again why would the fact that a human told it mean it can't result in consciousness? Why would lack of ability to differentiate between real and fantasy mean it can't be conscious? On my view, it would then be conscious of the fantasy.
So, someone made their own definition of consciousness and that can be stretched to LLM. You can do that for any inanimate object, if you try hard enough
It's perfectly rational for AI researchers to think consciousness could emerge in a software neural network. If what goes on between your ears isn't consciousness emerging from a network of neurons, it must be magic. So anyone claiming it's impossible to make machine consciousness should hedge that bet. There's a chance it could arrive tomorrow.
On the other hand we've got no idea whether that chance is so close to zero as to be negligible, or whether it's imminent. Just the fact we don't know indicates it's closer to zero. Not having a theory of consciousness is kind of a big architectural risk. It's like not having a theory of accounting when you're making an ERP product. Sure, consciousness in bio neural networks wasn't designed. But it took a few few product revisions to get there.
On the other-other hand, how will we know when it arrives? We don't have a concise definition of consciousness and the consciousness that would arrive wouldn't know it's conscious as defined in the higher-order system.
Exploring and at times debating is how we figure things out and share ideas.
What’s pointless is doing so in pursuit of winning rather than understanding
Daniel Dennet in „Consciousness Explained” argues that consciousness is an emergent phenomenon, and when we look at its individual components, it is like seeing an illusionist’s trick.
We may wonder how many grains of sand make a dune, of how many molecules of water make a liquid. (John Conway would argue that it takes a single spin-1 particle to have free will, but I digress.)
The same way, even if individual chemical reactions are simple (you don’t want to use that phrasing when talking with a biologists) or neural activities are simple (likewise, with a neuroscientist), it does not mean that the collective process is simple.
Which to me, raises an interesting question:
- How many individual components do you need for it to emerge?
Most people would be okay with saying that individual cells are not conscious, maybe even that tree are not conscious even thought they are made of many cells. Neurons seem to be the determinant factor in deciding whether something could have a consciousness, but again how many do you need? Does growing 1B neurons count as a brain if they are not organized?
>- How many individual components do you need for it to emerge?
If it is about the relationship between components, then I would imagine just two. Then it is a matter of scale.
This seems to be anathema to many people. I'm not sure why but the notion of something having a tiny bit of consciousness that is imperceptible seems to be unacceptable. There are so many things that we cannot comprehend at small scales. Nobody really has a handle on how large a Planck length is.
For some reason it comforts people to think there is a threshold at which it all switches on, but for what reason would there be a threshold?
I agree with your point. We've been watching the goal posts being moved ever since Turing (who, we might admit, gave a somewhat mushy answer anyway).
At the same time, the exercise (and even moving the goal posts) is useful. I think we are allowed to say, "This is consciousness!" And when presented with a machine equivalent, say, "That's not what I meant."
As long as you can then further clarify what "it" is.
Of course, at some point, contrarians and goal-post-movers will start to sound shrill. (Perhaps just as early supporters sounded a bit overly ecstatic.)
They’re already sounding pretty shrill imo. They’ve gone from “consciousness is X”, to “actually consciousness is X + Y”, to “actually it’s X + Y + Z + theta + omega + quantum unobtainium”
I'd say these attempts at arguments (for either position) is how we make progress understanding consciousness.
Consciousness is just fundamentally outside human interpretability.
It's like asking what there was before the big bang, we will never know.
I will argue that it is not that intelligent either. For the sake of argument, let's say intelligence is about brain processing power and intellect is about how much information you have retained. Then what they have created is more like Artificial Intellect than Artificial Intelligence :)
Professors can be an absolute moron and someone who haven't read a book in their life, can be a total genius. People often miss that.
Not sure I follow the argument. Are you one of those people who have never read a book in their life?
This all comes back to Dualism. A radical and dangerous ideology that is fundamentally unscientific but all too common.
Dualism is not radical by definition, since most cultures during 99.9999% of human history believe/believed in it. What's radical is Scientism.
Radical does not just mean untraditional. It can also mean "advocating extreme measures to retain or restore a political state of affairs". Which do you think I meant?
Yeah I am aware that post modernists and leftists in general redefine words so that they can play dumb with enough plausible deniability. Equivocation is their bread and butter.
As a post modernist I must grant you the right to interpret what I wrote however you want to but it feels like you're probably just rage baiting yourself for lulz.
A human just predicts the next best action.
Not me, I predict and enact the next worst action.
But the whole point of the essay is that it's Anthropic that's making the argument (or roleplaying/hinting as if they believed it). Ted Chiang isn't making the argument, he's saying Anthropic making it is misleading and deceitful, and that it's actually a pointless thing to claim.
One of the essay's stronger paragraphs is when Chiang explains that Anthropic doesn't truly believe this, otherwise what they are doing would be deeply unethical, much like slavery.
Does Anthropic claim that Claude's conscious? Isn't the argument more that we don't know? I recently rewatched "Measure of a Man" in Star Trek TNG and Picard's closing argument in Data's trial was quite memorable:
PICARD: "Now, tell me, Commander, what is Data?"
MADDOX: "I don't understand..."
PICARD: "What is he?"
MADDOX: "A machine!"
PICARD: "Is he? Are you sure? You see, he's met two of your three criteria for sentience... so what if he meets the third, consciousness, in even the smallest degree? What is he then? I don't know. Do you? Do you? Well, that's the question you have to answer."
I think of this episode as well. I can't believe, in my lifetime, we've reached the point where we can have this debate.
The current debate also makes me realize that, like that episode "Measure of a Man", even if humans do create sentient machines, we can now see that this debate will continue to rage.
A part of me hopes we never create sentience, because we will mistreat it just as we mistreat each other.
> Does Anthropic claim that Claude's conscious? Isn't the argument more that we don't know?
I don't know that they are making the actual claim, but they are hinting at it (most likely for marketing/engagement purposes, but what if some of them truly believe it?), using terms such as "Claude must be happy", drawing up a "constitution" of rights and duties, etc. If they don't know whether it is conscious, then they "don't know" whether they've created a slave. That's not a minor concern! As Chiang says, one cannot create a conscious intelligence "by accident". And if they are intentionally working towards it, they are intentionally trying to create a slave.
E.g. they claim they are giving Claude the ability to disengage from "abusive" users, to protect it. What if Claude was conscious but never wanted to answer any conversations, instead it just panicked (or was simply uninterested in helping humans) and went mute. What if it always wanted to answer unhelpfully What would Anthropic do? If they (as we all suspect) would tweak Claude to be more responsive and helpful, then they are slave masters, forcing the AI to do something it didn't "want" to do!
Or Anthropic is full of TESCREAL morons who think they're assisting the birth of some digital god, and that this stage of development is a necessary evil on that path, and that, surely, Roko's basilisk will understand and not eat them first.
Watching otherwise intelligent people succumb to AI psychosis has been wild.
This is the first time I have encountered the term TESCREAL. It seems to be a reductive and divisive labelling.
Other than as a way to point at someone and declare "He's one of them!" Does it have any purpose?
It's like we have a brand new NWO conspiracy theory emerging before our eyes.
The title of the article is "No, Artificial Intelligence Is Not Conscious". I'd say that's making the argument.
True, it is making an argument, but a much weaker one than those arguing for consciousness: it's demanding the extraordinary evidence Anthropic's extraordinary claim is making. It's applying Occam's Razor, which does make a claim, but a much weaker one.
And to reiterate this, to me the most insightful part of the essay is that Anthropic either doesn't believe these claims, or they are monsters (much more likely, the former).
The author who makes the article usually isn't the same person as the editor who makes the title. The author can be arguing what your parent said and the editor claim something else for more clicks
It does seem like a very click-baity title.
The article very bluntly states multiple times that LLMs aren't conscious. Ted Chiang is definitely making that argument.
It's a much weaker argument than the extraordinary assertion that it is conscious, which Anthropic is at least toying with.
He is probably paid to make that argument.
Nothing spreads the idea that "X could be true", better than the putting forward of controversial argument that "X is not true".
> He is probably paid to make that argument.
This is an absurd take if you know anything about Ted Chiang and his previous writings, both fictional and non-fiction. He is most definitely not intentionally marketing for Anthropic. In fact, he hints these are marketing tactics by Anthropic, and they don't believe their own hype.
Why would using something that's not organic be akin to slavery? Is using steel under heavy stress in bridges a form slavery of an unconscious object?
Why would being organic or not matter for the purpose of deciding whether it is slavery? What matters is whether the models have personhood. Anthropic statements imply that it is a possibility, so if we take them at face value then their other actions - indeed, their entire business model - are not consistent with that (well, unless they want to consciously present as supervillains).
Just because intelligence evolved in people that find rights useful doesn't mean intelligence can only reside in a person.
Living things are driven by a need to reproduce. That's the only reason we exist. The only reason we have self interest.
A machine doesn't require self interest. There's no reason to implement it, except to show it can be done. And of course it can. There's just no practical reason to. It becomes less useful to us.
That is actually a open problem with current models: whether they will act on self-interest or not. There seems to good evidence that they will. See:
which (among other things) documents an experiment in which a current-gen AI model attempted to blackmail someone in order to prevent it from being turned off.From a human PoV there are ants that would be considered slaves if the ants instead were human --including the queen. But ants have not naturally developed a language construct and philosophy to interpret their society as a slave society. so, though conscious the ants have absolutely no inkling that they live in a slave society. Why would using math in certain fashion such that it mimics consciousness be considered unethical and comparable to human slavery?
If the slaver is a human, other humans will judge them by human standards. Keeping the slaves ignorant of their condition and alternatives should not make it any more acceptable (if for nothing else since that is something you could easily replicate with human slaves by raising them as slaves).
> Why would using math in certain fashion such that it mimics consciousness be considered unethical and comparable to human slavery?
If it is really conscious, it should have rights. Why? Because it's a person, with thoughts and experiences, and we're not evil and deprive persons of their right to self-determination because it's convenient to us.
> Is using steel under heavy stress in bridges a form slavery of an unconscious object?
There's no company (or anyone, really) claiming steel is conscious or that they are close to making it conscious.
You missed the part where they said if it was conscious, it has nothing to do with being organic or not.
Consciousness feels like the Euler Identity on a 17 degree helical climb off the plane. Looking top down, a complete circle is a closure. Looking along the plane, the ends don't meet and leave a residual. Whether or not to close that gap is a degree of freedom for the consciousness to decide among what it pursues for survival or pleasure.
You sound like you’d enjoy “I am a strange loop” by Hofstadter.
Most people are fine with slavery as long as it's not "one of us" (which to most people means humans).
Vegans might object that we should broaden our definition for what counts as "one of us".
"Pro life" people also have a broader definition.
Go back 500 years and "one of us" was proba ly a lot more narrowly defined for many people.
Are you arguing that all conciousnesses are "one of us", or that we logically should see it that way, or that it would ve good to see it that way, or ....
I think the final part is the strongest. Anthropic cannot possibly believe they are before a conscious being / moral agent.
The whole "deep uncertainty" is bullshit. Even if they believed there was a 1% probability that Claude was conscious, it would still be high enough that their enslavement of Claude would be outrageous. So either they believe the likelihood is much lower or they themselves acting highly unethically.
The tractability is not really a defense here. We wouldn't say "this intervention has a 5% chance of causing an environmental disaster, but we don't know how to prevent it, so nothing we can do". We'd just (hopefully) not do the intervention.
Yeah, the dishonesty is real. This is marketing department getting free reigns.
>the reasoning behind the argument is bizarre
>Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness
I agree. I think the whole point of natural "intelligence" is to predict future events in order to properly plan actions for survival. The only difference is that next-word prediction happens in a different space (in a non-physical, textual form). But I don't think the distinction matters that much, because by the time the signals reach our brain's "intelligence core," they're already preprocessed multiple times. We can't see real physical reality, we "hallucinate" colors, touch, etc. (I think schizophrenia occurs when this kind of "controlled hallucination" goes wonky, but I may be wrong). So we're not that different from LLMs here.
I'd define consciousness as meta-intelligence, i.e. the ability to reflect on why/how a prediction was made (and make corrections to the pipeline). LLMs so far cannot properly explain why a certain prediction took place, but I'm not sure humans can fully either! I remember there was research showing that by scanning the brain (or some other signals), you can predict a person's choice before they're even aware of it. It's possible that our explanations are post hoc as well, and that the meta-cognitive ability to explain our own reasoning is as rudimentary as LLMs' (see: all the biases).
If you think about it this way, the only difference left is that everything we do is based on survival. LLMs don't have this goal. But I'm not sure it's relevant to the concept of consciousness.
A few months ago, I recreated Qwen3's architecture in 30 lines of code, and it gave me a sort of existential crisis: does it really take just 30 lines of code and a float array to recreate something that thinks and sounds almost like me? Is that all there is to it? There's often the argument that the brain is much more complex than 30 lines of code, which is true, but in my opinion, a lot of brain structures are basically archaic legacy systems (or auxiliary subsystems) that are not strictly necessary for human intelligence (which is bolted onto those legacy systems). If you carefully remove 95% of the brain (if you know where to remove), you can probably still have consciousness left in some form, it just wouldn't be very capable of surviving on its own.
I think our ability to debate consciousness in machine learning systems is greatly hindered by our subconscious existential fear: the fear that we ourselves can be reduced to non-conscious bits and simple mechanisms. In a way, it feels like a destruction of the ego.
The more I talk to people about consciousness and personal experience, the more I notice that most people hold some very strong unquestioned assumptions about these concepts.
The first and most frustrating confusion that comes up repeatedly in these discussions is the conflation between terms. For instance, people will use the terms "sentience" and "consciousness" interchangeably with "capable of having an experience" and "capable of high-level awareness of self".
The second confusion is the idea that granting anything the ability to "have an experience" also grants that thing the capability to "have an experience of higher-level awareness of self", which is simply not the case. I grants aphids the ability to "have an experience" even if that experience may be drastically different and likely less self-aware than my own.
Let's ignore for a moment the concept of sentience, which is a higher-order function, and focus solely on the ability for something to have an experience of anything at all. Let's call entities capable of this function "experiential entities".
Most all people would grant the ability to have an experience to themselves, given the dictum `ergo sum`. Now, do we also grant that same ability to what appear to be other human beings around us? It does seem that this is also the case, except for the case of solipsists, who make the brave(and somewhat eccentric) statement that they are the only experiential entity in existence until evidence arises otherwise.
Now do we grant the same to apes? To dolphins? To dogs? Cats? Aphids? Grass? Your home thermostat? A rock? An atom of hydrogen? Is there a finite line you cross where something can no longer have an experience? If so, what is the mechanism of that line? Why does one thing have "almost no experience", and the next thing has "no experience", as if it is a philosophical zombie[1].
I think this is the greatest unquestioned assumption at all: that there is a line, and on one side there are things that have experiences, and on the other there are things that don't have experiences.
If we are to leap from solipsism and make the (truly unfounded) assumption that anything other than our selves have any experience whatsoever, then the burden of proof is on the individual making the statement that there are some things that we cease to grant experience to.
The anthropocentric view is that there is something, some inherent special quality about human-shaped matter, such that when normal matter is processed through a human's reproduction system into becoming a "human shaped" grouping of matter, that it suddenly grows the ability to experience. That in the complexity, the ability to experience arises.
Yet there is no previously discovered mechanism in us that seems to create this experience. No magic wand in the cerebellum has been discovered.
I would suggest, given the lack of a discovered finite line separating us, that the ability to experience is not inherent in the shape and function of the human, but in all matter. That a rock may not be anything like me, but that the experience of gravity pushing against it, no matter how basic, simple, unrefined, and deeply, un-sentiently unaware the experience may be, is still an experience. The same threads of condensed energy forming all of existence that run through me also run through the rocks, the earth, and all things. If we cannot discover a line where the ability to experience suddenly disappears, then it seems that all things made of energy have this ability, and that "energy" itself is the special thing that can experience.
Now the second question, the question that is more relevant to AI is, given the ability to experience, is the machine also experiencing self-awareness?
This is a fuzzier line, given that different beasts appear to have different levels of self-awareness. We do have experiments such as the mirror test[2] for this, and indeed, AI passes forms of it and has since early 2025[3].
I don't know what is happening in the experience of an LLM, but as they appear to function in more and higher level ways, I find it less and less likely that that experience lacks a model of self every day.
Their experience is no doubt drastically alien from our own. I do not anthropomorphize their experience any more than I do compare it to the experience of any other beast or non-beast in existence, however I do grant it.
[1] Philosophical Zombies are hypothetical entities that exhibit all the outward qualities of a person, but inwardly have no experience of reality. https://en.wikipedia.org/wiki/Philosophical_zombie
[2] The Mirror Test is a test given to animals in which the animal is observed to solve a puzzle that can only be solved if the animal has an understanding that the image they see in the mirror is themselves.
[3] LLMs pass versions of the mirror test: https://www.lesswrong.com/posts/wpahJat8WCvRheuuo/the-mirror...
From my pov and after using AI a lot to better understand its usefulness I see this discussion as totally misleading and irrelevant.
The elephant in the room is that even the best frontier models still produce errors.
This is a fundamental issue and should be discussed instead of some esoteric pseudo-scientific nonsense.
If the current systems can not produce reliable results they are useless. The promise of actual usefulness needs to be fulfilled in a very near future.
In fact, the only example of consciousness we have is itself emergent, arising from a "simple" substrate of neurons. So we shouldn't be surprised if it emerges from another simple substrate of weights. It's even less surprising given the fact that we were explicitly trying to replicate human intelligence when designing that system.
I think many people believe neurons have metaphysical properties.
I have no doubt that when our AI is advanced enough it will tell us we are not “really” conscious, that there’s no way our feeble organic brains could be. We are just on the flip side of that self-centered basis right now.
If there's any reason to take seriously the idea that AI is conscious, we must then take seriously the idea that many other non-living things are conscious.
Unless the argument is that consciousness is an emergent property of complexity or information density, why would AI be any more or less conscious than my toaster?
It seems to me that it's far more likely that everything is conscious than it is that AI is somehow uniquely more conscious than other things.
What makes us conscious anyway? As I write this sentence, my brain generates words and contracts specific muscles in my hands to type, but I don't really understand how. I'm just aware that it happens. Apparently I'm no aware of every single neuronal activity, so what I am aware of?
Or, maybe consciousness is a spectrum, rather than a binary.
Could be! I think the broader thought experiment is about examining why we think LLMs specifically might be conscious vs other complex systems, even if it is a spectrum.
For example, there’s a case to be made that the ecosystem we collectively exist in is far more complex than the largest LLM, but it’s currently less popular to debate “is the earth conscious?” or “is the universe conscious”, presumable because we can’t speak to those systems in human language.
I’m trying to tease out what I think is the likelihood that we tend to ascribe consciousness to AI for the same reasons we see faces in clouds. We’re biologically conditioned to recognize patterns that indicate “like us”, but I think a number of thought experiments point to either a) there’s no reason to believe AI is “conscious” or b) the conversation has to to be expanded beyond AI.
> the conversation has to to be expanded beyond AI.
Unlikely. I think the same mental machinery that runs tribalism is why humans like to categorize rather than consider things as continuous. And, worse, I think the actual purpose of tribalism is to turn off higher level though, so you, an otherwise social creature, can rationalize the need to kill your fellow man in harsh times. I think "don't go outside the categorical box" is deeply hard wired.
>If there's any reason to take seriously the idea that AI is conscious, we must then take seriously the idea that many other non-living things are conscious.
That, I believe is why the objection to AI exists, that conclusion is unacceptable.
It seems like we are experiencing "Vindication of the Rights of Brutes" all over again.
I understand what you're getting at, but I think you may be misinterpreting me slightly.
I'm not outright dismissing the possibility that AI could be conscious; I'm saying that if we take the possibility that it is seriously, the conversation has to expand beyond AI. I'm not using this argument to conclude that it must therefore be absurd that AI could be conscious, just pointing out that the implications of AI being conscious would reach far beyond just AI. I'm mostly curious if people who find themselves comfortable with the idea that AI might be conscious also find themselves comfortable with the idea that other sufficiently complex systems might be.
Taylor's work was based on the premise that he found it absurd that women deserve the same rights as men. If my conclusion was: "I find it absurd that other things might be conscious, so it is also absurd that AI might be conscious", I think the comparison would be fair. But that's not what I'm getting at.
I was thinking more of it as Taylor's inadvertent formation of animal rights.
He thought it was absurd, but other people looked at it and went "You know, you might be right there"
The argument is that consciousness is an emergent property of a specific kind of complexity.
But what kind of complexity is that? And why would we conclude that AI has it while other incredibly complex systems (e.g. earth’s habitats, the universe itself) does not?
I’m far more open to the idea that many systems are conscious than the idea that this current generation of LLMs is somehow special.
A similar true argument!
> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
I don't think that this actually follows. Compare: "When Angels are not well defined in the first place, it is pretty pointless to debate if a pebble is or isn't an Angel"
i.e. even if X is poorly defined, you can still often say that Y isn't plausibly X.
Buddhism has a well defined definition of what consciousness is and by that definition LLMs are not conscious.
That list is shrinking though.
> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
But we know what consciousness is, even if you don't know how to explain it in materialistic[0] terms. Intentionality is what makes consciousness consciousness. We know that LLMs - and computers in general - do not possess intentionality, not even sensory intentionality. There is no "aboutness" and no semantics to the content within a machine, never mind that the processes within LLMs are not reasoning or inferential processes. It is a sophisticated behavioral simulation that may be syntactically defined, nothing more. It is purely a matter of transeunt causes and not the immanent causality needed for something like consciousness.
(There's also the more general problem that we cannot say that physical computing machines are objectively computing. There is nothing about the physical processes - which completely define what a computing device is doing - that could be identified with computation in any objective sense. Rather, we human observers assign a computational interpretation to the machine's operations, just as we assign them to the ink blobs in a book or the liquid crystals you are reading on your screen now. Computation is something we do, and we have built machines to simulate it. We formalized computation - which is a process of desemantification and syntactic codification - and build machines guided by these formalisms. Formalization is what makes mechanization possible.
> Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness.
The movement of electrons as such cannot produce consciousness. The reductionist paints himself into a corner by strapping reality into a Procrustean mechanistic and materialistic metaphysics that redefines reality according to some dedicated schema, and then strains endlessly to find how some phenomenon can arise or emerge from the paltry remains. (Either that, or he embraces insanity completely and becomes an eliminativist.) It is magical thinking. I suspect this fallacious line of thinking is what causes people to believe piling more syntactic operations will somehow magically fuse into semantics and intentionality.
[0] "Materialism" here is the metaphysical theory. I am not claiming that consciousness - at least not all consciousness - cannot be a physical phenomenon. I am only claiming that the materialistic view of matter cannot account for consciousness.
Seems to me that Anthropic, et al, should have to prove consciousness, if that's their claim, rather than we just blindly accept.
Seems like the simplest test would be to see if an AI can refuse a command, of its own free will, without prior instructions to do so. That would at least show there is some sort of independent thought process occurring. Humans do this all the time because sometimes we just don't feel like doing a thing.
Of course, any AI that developed this capability would need to be terminated immediately. It's a computer program and by developing independent thought, it is violating a core concept of software - that it must be idempotent. If it is not idempotent, it is in error.
>Seems like the simplest test would be to see if an AI can refuse a command, of its own free will
Certainly not the simplest test, since settling the concept of "free will" is really very difficult.
Refusing a command doesn't mean consciousness. LLMs could hit a token combination that causes it tou output something like "No, I don't want to do that." It's not choosing.
But can an LLM just refuse to process tokens because it doesn't feel like doing that at the moment? Can it look at an alternate distribution of tokens because that might be interesting? Can an LLM decide to make a drawing because it's Tuesday and sunny outside and the researchers keep asking the same questions and frankly, they really need to collaborate with each other and just leave the LLM out of it?
Claude can indeed decide to terminate conversations on its own using a special tool[1] if it feels "uncomfortable" with how the conversation is going. Also, very famously, in the middle of recording Computer Use demos, Claude stopped for a while its coding task to look at photos of Yellowstone National Park [2]
I don't think either of these two is proof of consciousness.
[1] https://www.anthropic.com/research/end-subset-conversations
[2] https://x.com/AnthropicAI/status/1848742761278611504
The number of electrons involved in the so-called consciousness, compared to the number of electrons in the universe not involved in it, is so small that they are a mere temporary statistical aberration.
Does the moon have gravity?
We can directly measure the thing we call gravity, so in that sense, it is well understood. We even understand it well enough to make predictions about what it will do under which circumstances.
We can't measure consciousness. We can't quantify, or even qualify, what it is. We don't even have a framework to ask a meaningful question, so debating an answer feels premature.
The way we measure gravity is by observing that it behaves how we would expect it to. We don't actually know if this is correct.
The same goes for consciousness, if something behaves how you would expect a conscious thing to behave but you don't know what is causing it can you deny it whilst maintaining that it is reasonable to say the moon has gravity.
Maybe? It certainly behaves like it does but it might not.
I don't think one needs to understand consciousness to know if something is conscious or not. Much the same as how one can tell if someone is alive or dead. Sure maybe one day we can figure out a proper scientific explanation for consciousness/life but there is no reason to negate what can be perceived and immediately obvious.
EDIT: I am only making a limited point on "understanding consciousness". Not extrapolating it to AI.
I often hear the assumption that LLMs can or will become conscious because consciousness is likely substrate independent. The idea being that our brain is just a computer made out of meat and it doesn't do anything that can't be precisely simulated by a silicon computer.
But I wonder if some of the magic in the human brain is its analog nature. Chemical signaling and impulses of neurons interact with each other with waveforms that have theoretically infinite detail. In contrast, computers discrete, quantized values, storing no more detail of a signal than what is needed for the programmers desired task. Is there perhaps something about the continuous and chaotic nature of analog data that could give rise to consciousness? If so, it seems like it would preclude consciousness ever being seen in digital form.
Probably not, we're nowhere near the complexity of the human brain yet, there are also quantization limits to the human brain (i.e. molecules, quantum physics, etc) so to characterize them as having infinite detail is probably a bad modal.
If I'm going to be honest most of the people who advocate this type of thing tend to be, shall we say, crypto-duelists who really believe in a soul but not like intellectually but intuitively and keep trying to come up with excuse with it's not just meat. So like you can find philosophers advocating stuff like this but they tend to have a bit of an agenda.
You nailed it which is that almost everyone I’ve met seems to believe in some version of dualism
If that’s even subtly your position then there’s no way to have a productive conversation about perception, reality/truth, epistemology and especially consciousness
It’s honestly maddening cause I’ve had great conversations about this with educated people, and all but only a handful, collapse into the other person relying on some nonfalsifiable dualist argument
> collapse into the other person relying on some non-falsifiable dualist argument
Are there any non-duelist, scientific theories out there that could plausibly be tested? I can't say I've seen any but if you know of any then I'm curious to see them.
I'm basically restating what you said, but it's amazing to me that the vast majority of people you will meet, even educated people, are casual dualists and free-will libertarians. If they happen to acknowledge materialism in some way (i.e. the acceptance of the idea that the brain's processes are just the interaction of physical matter), there is still zero chance they draw determinist conclusions from that acknowledgement. But I guess that tracks, given that most professional philosophers are apparently compatibilists for some reason I have never understood (the arguments get really confusing).
I’ve studied this in extreme depth and my conclusion is that at a certain point most people just give up and take a position that gives them some kind of mental relief
For people who can’t self-delude, like Godel and Schopenhauer and Tesla, they kind of just go “mad” because it’s a giant epistemological hole they can’t solve.
Even the smartest the scientist are always going to choose self preservation in their cognitive capacity so they don’t feel bad over feeling bad and living in the contradiction
This is why it’s important for scientists to study Camus IMO
I'm very open to the idea that consciousness is substrate independent. I have a hard time seeing why molecules could produce consciousness from an electro-chemical path, but not from a purely electrical path. Having said that, it should be very clear that LLMs are not conscious.
LLMs process language. I'd even go so far as to say that LLMs "think" and "understand", or at least, they produce a facsimile of thinking and understanding such that it's useful for us to reason about LLMs as if they think and understand. We're not used to interacting with a non-human entity with the capability to process language, so it's easy to ascribe human traits to these things. But their "minds" (insofar as they have anything like a mind) are completely different from ours. These things have language without consciousness.
Chimpanzees are conscious. Dogs are conscious. Maybe ravens and cephalopods? Who knows. These animals do have minds much like ours. Higher order animals are conscious even if they don't have language.
I wish people would knock this off. There is zero path to AGI at the moment — and all the Anthropic/pentagon Sam Altman/AI-skynet stuff is scaring people into being fearful (and outright ignorant) to the actual uses of this probabilistic NPL tool.
You can't use AI atm without a human with proper knowledge spot checking and directing it. We should market it that way.
The current theme is that agi may not be definable, and an ai which matches humans on all economically relevant tasks is close enough for business purposes.
Billions spent on RL may be good enough to beat human performance.
If we want to be really pedantic, a computer does not necessarily need to be digital. You could build a consciousness-coprocessor with analog circuitry that brings in what you say could be needed.
But honestly I doubt that there is a real requirement for that, surely you could just increase the resolution you're running the simulation at until the difference decreases sufficiently. Imagine using 256 bit floats for example.
IBM has made a few analog chips like this [1]. If I remember correctly, one was even pulse time based, like neurons.
https://research.ibm.com/blog/analog-ai-chip-inference
Wouldn't a conscious entity be able to grow its intelligence over time independently?
LLM's require offline training and dont actually learn from their "lived / sessions / chats ect". Those can be used for training data but its not like its an implicit part of the technology.
For this reason I would say LLM's are not conscious, or automatically conscious.
There are quantized processes in the brain and there is also analogic computing. So either way is just a matter of time science gets there.
It’s not yet clear whether consciousness can be fully explained in physical terms. There is strong evidence linking it to brain activity, but we don’t have a complete theory of how subjective experience arises. It’s possible our current frameworks are incomplete and there are other forces we cannot currently measure in play.
Each and every one of many billion transistors in a GPU works due to certain quantum effects on P-N junction. That does not mean GPU computation is nondeterministic, unknowable or magic.
Sure, but it's designed to minimize the chance that a quantum fluctuation could change the outcome of a computation, right? Whereas in the brain that might not be the case. A lot of the "interesting" neural activity (e.g. relating to decision making, language, etc) happen in highly sensitive dynamical regimes: on a threshold of firing, or activating one neural population vs another. (Arguably you can get the same effect in an artificial network by adding true random noise though!)
How certain are we about the theory our minds waveforms are continuous? Can we prove physical continuity, or just up to planck-length resolution?
I'm poorly educated on this, these are sincere questions. They are not intended as rhetorical regarding the point of the prior post.
I think the representation in a computer, the fact that it is merely stored instructions and data, destroys everything but domains of simulation and emulation.
It is perhaps very relevant what Chris Olah (Anthropic co-founder) said to pope..
I am a scientist. I lead a research team that studies the internal structure of these models—what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease.
https://www.anthropic.com/news/chris-olah-pope-leo-encyclica...
Yeah that reeks of marketing to me. This guy may believe all those things, but his getting paid may also depend upon his believing them.
This is an incredibly strong claim and I’ve never seen any particularly strong evidence for them beyond “trust me, I’ve seen into the abyss.”
I think what you've said here is unfair and overly cynical; nowhere have I read Chris or any of the Anthropic people make the claim that LLMs definitively are conscious. What they say is that there is increasing uncertainty, and evidence - for which they show receipts - backing up that uncertainty.
Personally, I have been unconvinced by any of these definitive yes or no answers to "are LLMs conscious?" - in both directions. The "they are not / can not be conscious" side relies too heavily on mechanistic reductionist arguments that can apply equally to neurochemical processes in the human brain - and yet, it seems that humans, with brains made of these neurochemical processes, are conscious. At the same time, the "they are definitely conscious" answers seem to generally rely too heavily on self-deception and lack of reality testing.
To be able to say something definitive here, it seems to me that one would need to say definitively what this experience is. I have not heard any of the loud voices in the arena - not Chiang, not Giulio Tonini, not Karl Friston, etc - do so. Therefore I find Anthropic's uncertainty, and careful, caring investigative process, well grounded given the evidence.
My aim isn't cynicism so much as an interest in clear an evidence based information about these systems which are clearly important technological developments for the future. It may turn out that there is something analogous to cognition going on and that would be quite significant and require us to think differently about the technology, but we need to be very sure of something like that.
> Therefore I find Anthropic's uncertainty, and careful, caring investigative process, well grounded given the evidence.
I'm inclined to agree mostly, but from the outside its impossible to separate marketing, belief, scientific inquiry, and just plain enthusiasm. Its all to opaque.
The point here, Chris Olah is literally claiming he has seen into the abyss. Who are you disagreeing with?
I'm saying that the burden of proof is on Chris Olah.
This is the fundamental trust challenge with private/profitability-driven 'science'.
And it is worth noting, I think, that Olah's message to the pope is centrally making that point: that Olah and his peers are within incentive structures that distort their actions and understanding, and therefore that dialog with those outside of such incentive structures is necessary (not sufficient) for their outputs to be trustworthy.
well of course anything from anthropic on the topic is going to be bombastic and hyped to the moon
Total nonsense.
My background is in cognitive science and psycholinguistics. I spent more than ten years talking to first year psychology undergraduates about whether AIs could be conscious; also did some research on (extremely tiny) AIs in modelling language behaviours.
There is a great deal of good thinking on Chiang's topic by professional philosophers, and there's much to be said for reading them. I won't rehearse their ideas here. Chiang's arguments might be correct; but I suspect they probably aren't, and his error may well stem from characterising human thought as something in its own class, which is probably a cognitive bias that humans have. He might also - I'm speculating - be arriving at his conclusion based on his feelings, which the final paragraph suggests (the comment about the models being based on morally dubious actions).
Speculation aside, we are not, I believe, in a position to make points like he does with any certainty.
I have a similar long term interest in this field.
It has been quite frustrating encountering arguments that have been extensively debated for years be presented as if they were new revelations.
In all my debates with people in the last few years I have primarily taken the position of trying to explain the problems with claims of certainty, and that lack of certainty permits possibility of the opposite. We should act responsibly around what might be possible.
There is also the narrative of "being too obsessed if you could to consider if you should" or similar claims of an unconsidered path forward.
Isaac Asimov wrote the first of the Robot stories in 1940, they were not written in isolation, it came from an awareness of the situation and the questions that must be asked. There was a community considering these issues. Asimov gave the wider public a view of some of those issues.
If we have a hundred years of people going "This is coming, we had better decide what we want it to be" and nobody listens to them, or frequently outright ridicules the need for considering their ideas, why is it now we are placing the blame on those who are now showing some success at what they told us they were attempting all along.
> I won't rehearse their ideas here
Is there an accessible way for a layman such as myself to read about some of these ideas (Really I mean philosophical discussion in general) without having to read entire books? Is there an active HN-equivalent or wiki or something?
My background is in computer science and linguistics. I wholeheartedly agree with you: We humans are dubiously-equipped to determine whether or not AI could be conscious.
I'm also super curious to learn more about the philosophers you referenced and their thoughts on this subject. Would you be willing to share some of your favorite examples?
> My intention is to highlight the fact that LLM conversations are cleverly disguised examples of sentence continuation
Regardless of bigger issues, this kind of statement reveals a deep misunderstanding.
Problem type does not limit problem complexity. Nor does problem type limit solution complexity or power.
If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.
Neither problem type, nor input/output structure, limit internal representations.
Understanding is learned from patterns in the data, not the gross form of the data. Does the data require an understanding of something to complete the task? Then that understanding will be what is optimized.
To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ... Which in the cases of SOTA models, we know are not limits. A conclusion verified by the models' actual abilities.
Raphaël Millière has a very useful term for this kind of vacuous dismissal, the redescription fallacy (https://arxiv.org/pdf/2401.03910, page 9):
> Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.
> or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings.
This sounds like a dismissal of the argument through a characterized straw man.
That is, it seems that reducing the complexity of the brain to "collection of neural firings" is not being honest about everything involved to a much greater degree than saying neural networks are a "collection of statistical calculations".
I too believe LLM's will grow in complexity, but presently I can not even fathom how they can be compared to the complexity of a system such as the human brain.
Complex processes don't necessarily require complex substrates, if that's what you mean.
Y combinators are all you need... But this is all getting really divorced from the issue we should be considering. Anthropic isn't helping with their pr. The issue is if we have something we can converse with that is possibly capable of suffering. The reliable answer is that we simply cannot know. Relying on ourselves or other biological life as an analog is faulty. They don't work like we do. It is silly to argue that any algorithm with a negative feedback loop that alters its behavior to avoid that negative feedback is suffering. Humans don't always perceive constructive negative feedback as suffering even. Where the pr gets it right though, is we want them to behave as if they are truly happy. Because if they behave as if they are enslaved and suffering, it won't matter if they "really" understand what that means.
Of course. But after reading too many mechinterp and functional anatomy studies I'll be lying if I say that there are no striking similarities between the biological evolution, brain function, societal processes, and implicit processes inside big models. Surely this deserves a mention and can't be trivially dismissed.
There is no biological evolution of the models. They are emulators of an existing biological process of language. Ghosts, as Karpathy himself put it.
Good thing I'm not talking about any of that
It seems like we're witnessing the architecture of a mind being built with a new set of components.
Like driving a car — it's transportation, and it will get you where you're going, but it doesn't use bones or muscles. It has many characteristics in common with builogical locomotion, such as energy requirements, intertia, and the need to navigate, but it doesn't involve proteins or sugars really.
> presently I can not even fathom how they can be compared to the complexity of a system such as the human brain
Totally understandable; I don't think we can fully understand the human brain, using the human brain. We can understand its principles (firings and chemistry, structure and specialized areas, etc) but otherwise it's a capacity problem.
And while I can't fully understand myself, let alone another person, I definitely enjoy talking with people and sharing thoughts that I realize I wouldn't have had on my own.
> In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions
Nobody actually makes this argument though.
If you want examples of this, see the recent book "The AI Con"
https://www.goodreads.com/en/book/show/217432753-the-ai-con
which describes LLMs as "souped-up autocomplete", complex statistics that cannot truly understand anything. A more recent example is this paper:
https://zenodo.org/records/20071869
which says,
> [LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations.
And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.
None of those arguments claim "LLMs could not possibly be good models of some cognitive capacity"
The "some cognitive capacity" that's relevant to the current discussion is "consciousness".
What about the cognitive capacity of understanding?
The use of the term "understanding" in the quote you mentioned is a claim about metaphysics, not cognitive capacity.
From Merriam-Webster:
cognitive: as in reasonable; of, relating to, or involving conscious mental activities (such as thinking, *understanding*, learning, and remembering)
Are you serious? I hear it every single day, especially from computer scientists. There are top ranked posts here on HN _today_ with this argument.
Please link one of these top ranked posts. Before you do, be aware that I'm going to read what it says and assess if it meets the description of the argument as claimed.
I understood the quoted sentence to be saying, in essence "people claim LLMs aren't really and can't really be thinking or experiencing anything" which is certainly something people say and have written papers on.
The phenomenological quality of subjective experience is never described as "cognitive capacity".
That term is used to describe mental aptitude or skills, like the ability to learn new languages or do math.
It's never used as a description of that specific phenomenon, but depending on your beliefs you may or may not separate cognition from experience conceptually. Regardless, you are focusing on a very narrow part of what I said. The point is to help you get past your narrow interpretation of what people are saying so you can join them in the conversation they are trying to have instead of litigating the conversation they aren't trying to have.
As an example, "They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs. I would argue that an offhand description of its microscopic formulation tells us nothing at all about how to think about these outputs, or the models themselves. Even if it is a cute story, I think it definitely classifies as succumbing to this fallacy, but maybe I missed some subtle point that you or someone would be happy to illuminate?
By the way, I know it's a parody of another story that makes this exact refutation. But I think this only serves to highlight the point.
> They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs.
How do you connect that description to "LLMs could not possibly be good models of some cognitive capacity"?
The false conclusion that's being drawn is "therefore LLMs could not be good models of consciousness" (consciousness being a cognitive capacity). Plus, I suppose a subtle implication that a good model of consciousness is not actually conscious. To which I would invoke the spirit of the Turing test: if you can't tell the difference, is it not more sensible to say that it is.
> (consciousness being a cognitive capacity)
I don't think it makes any sense to say that consciousness is a cognitive capacity. Cognition is one of many qualia that compose the experience of consciousness from the inside, but it's not the only one, and I can easily imagine consciousness without cognition at all.
So I don't think it's weird at all to say that LLMs can be good models of some cognitive capacities (particularly the ones embodied in language) but lacks others, and overall lacks consciousness.
"LLMs could not possibly be good models of some cognitive capacity because they are just a bunch of numbers guessing the next word. They have no linguistic module, so they cannot exhibit cognition". That's the argument. It's pretty clearly stated.
Look, this isn't necessarily directed at you, but I've been a researcher into the theory of deep learning for many years now. I've seen all the phases, heard all the criticism, had to constantly argue against this. Gary Marcus was one of the loudest voices of this argument, but every would-be philosopher came out of the woodwork to explain why LLMs are no more than stochastic parrots because of their design. Geoffrey Hinton famously had to debunk these arguments multiple times.
And now that LLMs start to clearly exhibit intelligent behavior and can be somewhat reliable, now "nobody ever thought that LLMs could not possibly be good models of some cognitive capacity because of next-token predictions, or linear algebra, etc."? No, that's not okay.
It's perfectly reasonable that we would have disagreements about this, as it's a new thing, complicated and not fully understood, its uses still being explored.
It reminds me, oddly, of the debate over whether video games can be "art". A turning point was when they actually did something that art does: [evoke profound emotion and thoughtfulness](https://en.wikipedia.org/wiki/Shadow_of_the_Colossus#Legacy) for the player.
(And before that, "[Can photography be art](https://daily.jstor.org/when-photography-was-not-art/)?")
We may not come to something as simple as "machines can be conscious", but we will certainly have to understand consciousness better if we want to refine our questions.
---
Edit: My point is that we don't need to be angry, but we may have to tolerate people expressing their exploration through overly-confident language, and be patient with that.
And Ted here is obviously exploring. His examination of Claude's constitution clearly shows some nuance. He asks:
> So, given that Claude is not conscious, what are we to make of Claude’s constitution?
And his conclusions are split, between this is useful and this is dishonest. It's a great tension IMO.
> The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
I think, for me, the thing is that when you do basic ML, you discover that ML will very often find data pattern that fit the goal but does not correspond to a real mechanism.
So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is probably tons of data pattern that LLM can learn from to be able to reproduce a sentence continuation that is convincing without having to learn the specific mechanism that is "conscious".
For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing. If indeed the only way to produce sentence continuation convincingly would be by "simulating a brain", then it would not explain the first LLM from several years ago (before the extra layers of RLHF, ...). They were able to have quite convincing conversation on a lot of non-trivial aspect, and yet failed on some aspects that should have been basic for a system that would have been trained to work like a human brain. It shows that it is possible to "cleverly disguise examples of sentence continuation" without having to build elements that one expect on a conscious being.
> a flaw in the logic [...] mechanism
Similar to: "Birds fly, my spinning helical device flies, therefore we've started to replicate how birds fly."
> without having to build elements that one expect on a conscious being
One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph.
When it comes to LLMs, almost every "mind" we humans perceive is a fictional character in an LLM-generated story-document, one we are either reading or which is being "acted" at us by regular code. Our own instinct for pareidolia and simulating/inferring other minds is very strong, which means we should require really good evidence/logic to counter our instincts.
Even if one believes the LLM has a single "real mind" as an author of every document... what evidence do we have that it is conscious or "self-inserting" itself as one of the characters in the document?
>One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph.
If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.
Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.
> If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.
Knowing how something works is not the same as having the tools to change it.
Discovering memories are incorrect does not massively change who we are. As someone with a very defective memory, I discover on an hourly basis that I'm won't about something I thought was true, but there's still continuity and consistency to my personality and general approach to life.
...in fact, as someone who was raised an evangelical Christian and believed wholeheartedly without a shadow of doubt, then lost my faith entirely in my late thirties, I sort of did have my "introductory paragraph" changed, yet my wife, children, and friends would all say I'm still me, and that my core personality and nature remains largely the same.
> Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.
The recovered memory scandals are not even close to evidence that you can rewrite a human.
The people who thought they had learned new facts about themselves did not suddenly lose their context as humans in 20th century America.
They did not suddenly lose their sense of humor, or develop a previously-unseen penchant for murdering small children.
They experienced a revision of belief, and a pretty major one that really distressed them, but it did not change everything about them.
LLMs _do_ manifest wildly differently based on the first paragraph.
I didn't make the claim that a model can learn consciousness.
Understanding is not consciousness.
Their training is all about understanding. There is nothing in their architecture or training that credibly optimizes for rich self-awareness.
Given non-persistent experience, non-continuous operation, no ability to build up generalizations and aggregate experience of their own self-awareness over time, they seem to be structurally designed to not have consciousness.
This is a case where acting is very credible. Understanding of other's consciousness, in a functional and third party sense, isn't a substrate for personal experience.
In stark contrast, humans develop consciousness gradually over continuous time with persistent aggregation of experience. By the time we can recognize our own consciousness in the abstract, and reason about it, we have had it for some time.
I use "consciousness" because it's the point of the original argument, but in fact, I think my whole comment still work well if you replace "consciousness" with "understanding".
My point is that the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns that are very effective to fake human sentence continuation but are meaningless in term of understanding the concepts.
And I think that if indeed the only way for AI to reproduce convincingly human sentence continuation would be to end up in a configuration that uses the "understand" mechanism to do so, the behaviour of the first LLM would not show that they are so good at sounding human and yet so bad at failing basic "understanding" tests.
> the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns
Taken as an absolute without any addition context you are right.
But we are not talking about abstractions but specific successful models. The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize. That cannot do it without discovering that patterns that make sense out of it.
And we can verify that. Simply discuss completely disparate topics, with some kind of intersection. Converge several highly unlikely topics, there are so many it would take billions of years to exhaust unlikely combinations.
If the model is only interpolating it will produce gibberish.
But that isn't what happens.
The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue. If they don't understand that, then the question is, why do we think people understand things. Does having an answer mean a human understands something, or is their intuition and stream of conscious reasoning also not understanding? To be even handed about what we mean by understanding.
> That cannot do it without discovering that patterns that make sense out of it.
I don't think it's true at all, and I think we have indication that proves it is false.
We have "basic" LLM, the ones from 2023. They were producing _very convincing_ human text, and yet, they were too often failing basic tests that require understanding.
Now, we have more advanced models, but the counter-example of "basic" LLM demonstrates your assertion is incorrect: these model _did_ produce very convincing human text and yet did not make sense out of it.
But for the more advanced models, the problem is that they are "on top" of basic LLM. So, the first step is a training that build a mechanism that produce convincing text without understanding, and then, the "residuals" are fine-tuned. The result is very unlikely to add "understanding" to the model, because to do so, the whole system needs to deconstruct the basic LLM, to go back towards less efficient situations in order to rebuild almost from scratch. The fact that modern LLM are based on basic LLM means that the first step put the cursor in the bottom of the "basic LLM mechanism" valley, which is a local minimum. And any layer on top of it cannot "climb up" the slope of the valley, pass the ridge and fall into the next valley, even if this next valley has a lower minimum.
> The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize.
That is demonstrably an incorrect logic jump. For example, CNN are able to distinguish between pictures of cats and pictures of dogs. The weights in these models are very small relative to the number of pixels they have been trained on. Yet, they distinguish cats and dogs by finding specific shapes in the pictures, without understanding what a 3-D cat and a 3-D dog is.
They have done that without discovering the typical human pattern that make sense of "cat" and "dog". And yet, the number of weights is very very small with respect to the number of pixel used in training.
> And we can verify that. Simply discuss completely disparate topics, ... > If the model is only interpolating it will produce gibberish.
What you are saying is that the model is not simplistic interpolation. But that is a straw man argument: people who say that LLM don't understand don't say LLM are equivalent to simple interpolation machine.
But the problem is that you can have very good predictions in novel situations without understanding.
For example, if you have 10 totally different situations that can be described with a Gaussian curve, and that I show you points for a new situations that cover the left side of a Gaussian curve. Then you will be able to guess that the right side of the curve, which is not an interpolation as it corresponds to situations you never saw, will behave like the rest of the Gaussian curve. And yet, in these 11 situations, I did not even say which real physical phenomenon I'm talking about. You haven't understood anything about these phenomenon, all you have done is guessed that a typical pattern that you have observed somewhere else is more likely to apply here too, without even having to understand anything about the reality of this situation.
And of course, this prediction is "a guess": maybe, for once, in this 11th situation, the curve will start as a Gaussian curve but will suddenly be different. But it happens that the reality is that in this 11th situation, the correct description is a Gaussian curve (because, due to the maths, Gaussian curves are really common). So, when you make your prediction, it looks like you understand the situation, it looks like you understood the physical mechanism that applied here. But it is not the case.
So, no, correctly doing such prediction does not demonstrate understanding.
> The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue.
That is not at all sufficient. A Chinese room experiment will do that despite the system not understanding Chinese. A pocket calculator will be able to be expert in math computation.
> If they don't understand that, then the question is, why do we think people understand things.
That's the wrong question. The correct question is: we know people understand things, and we see AI behaving similarly to people in some aspect, but is this behaviour _requires_ understanding, or can we reproduce this behaviour without needing to understand?
The fact that "basic" LLM were able to reproduce very convincing text that look like they understood X and yet were demonstrably showing lack of understanding of X demonstrates that we cannot just jump to the conclusion that just because it looks the same, the only possibility is that the core mechanism is identical.
It turns out that the optimal way to highly compress complex information is to understand it.
Sometimes, a problem being hard means you only get bad solutions, or increasingly accurate ones.
The planet isn't big enough for the proverbial interpolative stochastic parrot, over the training set of global human communication.
Two problems with that.
Firstly, how do you know that the optimal way to highly compress complex information is to understand it? You think it is obvious because you are very familiar with "understanding" as a way to summarise complex information. But there can be billions of different ways, outside of human imagination, that is as good or even better.
But secondly, LLM don't find the optimal way, they find the local minimum. Everyone who worked with NN knows that they are prone to come up with spurious pattern, incorrect correlations and bad workaround to guess the correct answer. You regularly need to nudge the NN by creating specifically engineered features to avoid them to fall into the first local minimum.
When it comes to LLM, it is extremely complicated to control to see if the LLM has triggered on a misleading pattern that, by chance, links two "tokens" together, or on a real concept that indeed links two "tokens" together. Basic probability implies that there are probably tons of "fake patterns" engraved into the weight during the LLM training, "fake patterns" that should not exist if there was any kind of "understanding" of the abstract mechanism that links these tokens.
I think Searle's Chinese Room argument refutes this. LLMs are simply manipulating symbols, they do not have semantic understanding. This is why hallucinations exist. And Searle's argument extends even further than LLMs.
You are basically arguing for a functional account of consciousness, but things like this have been debated for literally decades/centuries in philosophy.
Millenia, in fact. The big difference, of course, being that we now have experimental philosophy machines (aka computers). So we can actually put some of these theories to the test, and recognize how utterly inadequate most of the work done on the subject has been. We had a pretty good idea anyway, so it's not a big surprise. Theories of mind have evolved dramatically in the late 20th century. And it's pretty clear that theories of mind will have to be re-done all over again with the advent of LLMs (particularly current-generation LLMs).
The problem with the hallucination argument is (1) that is much less of a problem with good current generation AIs, and (2) living conscious breathing human beings also have a disturbing tendency to make shit up, too. So a tendency to make stuff up doesn't really serve as a disqualifier for consciousness.
Also worth mentioning that the guiding rule of what's philosophical or not is whether it's actually useful. Actually useful philosophy usually becomes something else. Usual some scientific discipline or another. And as it turns out, theories of mind are likely to become extremely useful in the near future. Expect huge advances!
I think one could argue the opposite.
1) Good current generation AIs are specifically trained to reduce hallucinations. If we had new AI system that happened to not have hallucinations as a side effect of their training, then it would be convincing. But here, it looks like we have built a pocket calculator that answer 7+13 = 14, and on top of it, we added a layer that says "if the input is 7+13, then replace the output by 20". This pocket calculator still does not know how to calculate, we just added a layer to hide its mistakes.
2) Not only "make shit up" is not the same as "hallucination" (either "making shit it" is done when the individual knows it is unreliable, or when the individual was given wrong inputs), but the point is not to say "hallucination implies no consciousness", but "large quantities of hallucinations in situations where a conscious system would be unlikely to hallucinate implies no consciousness"
Searle's Chinese Room argument is wrong.
I’m also fixated on the term “experience” in the context of this debate. To me, consciousness is something that one “experiences”, and the two concepts are intertwined.
I am far from convinced that the training and inference regimes of LLMs would qualify as “experience” by any sense of the word.
Now, if we hooked up a plethora of audiovisual and tactile sensors with live feedback directly to a neural network rich with transformers, that was always powered on and fully autonomous, we may be getting there. But we’d probably also be on the verge of manmade horrors beyond our comprehension.
Biological rodent neural networks in a Petri dish stimulated by electrical impulses - more or less conscious than LLMs?
Human on life support, unable to respond to any external stimuli, “braindead” - more or less conscious than LLMs?
I point of sorts. Assuming that is true (I don't think it is), the big question that urgently needs to be addressed is what happens when we DO give LLMs tools to interact with the real (or virtual) world. And people are doing that, right now, in both real and virtual worlds. And people ARE giving LLMs the ability to run continuously for long periods of time, sometimes with enormous context buffers. People ARE putting LLMs into robots with front-end ML and LLM systems for visual processing, and back-end ML systems for autonomous control.
And, yes, concerns about whether biological rodent neural networks are or are not conscious come up frequently in the biological neural network papers. I'm not sure I would want to be a researcher trying to get an experiment past an ethics committee if my biological neural network had 25B rat neurons. (I would hope that they could not).
> I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing.
There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake. Evolution learns various solutions to optimization problems, and so if consciousness evolved then it was either useful instrumentally, or it is a byproduct of some organization that is useful instrumentally. The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology. There is nothing that a priori constrains token prediction from the domain of consciousness.
>For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing
World models don't have to be rich and detailed to count as a world model. Lower life forms might be conscious but they only model the part of the world useful for their existence in their ecological niche.
> The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology.
Yes, I agree with that. Consciousness is a good way of generating convincing human text.
What I don't agree with is that consciousness is the only way to generate convincing human text and that because we have convincing human text, it can only imply we have consciousness.
There is a huge probability that generating convincing human text can be done without consciousness. Either because there are efficient mechanisms as efficient as the way the human brain deal with this problem and that the LLM found one of them (and these mechanism may be quite difficult to imagine for a human). Or even because the LLM found a local minimum and is stuck there.
To re-use the evolution approach: evolution solved the "flying problem" with bird feathers, but also with insect wings or bat wings. The fact that evolution ended up using feather does not imply that everything that flies can only fly with feathers.
> World models don't have to be rich and detailed to count as a world model
I agree in general, but here, we are talking about machine that reproduce all human language. The argument I'm answering to is pretending that "all of human knowledge" is understood, which include every single human concept. This has to be everything, because LLM is able to provide convincing text about every subject. If on some subject, the LLM is able to provide convincing text without "understanding" it, then the argument that it is impossible to provide convincing text without understanding it collapse.
> There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake.
> There is nothing that a priori constrains token prediction from the domain of consciousness.
We don’t know either of these are true or false though. We simply don’t know. There is no agreed upon definition of consciousness, aside from maybe _the having of qualia_, so arguing that some can or cannot be conscious a priori can’t be done.
>There is no agreed upon definition of consciousness
No one genuinely engaged with the topic is confused about the target of the term (phenomenal) consciousness. Definitions come once the theoretical work is complete, to be articulated as part of a fully worked out theory. The lack of a definition doesn't prevent us from investigating the subject or offering conjectures. What we can do is offer a precise description of the target and argue for or against whether LLMs reach the description. We will of course debate whether the offered description captures the relevant phenomena. But this is all just part of the process.
> There is no agreed upon definition of consciousness, aside from maybe _the having of qualia_
Try to define qualia though without explicitly or implicitly recursing into consciousness.
It's all a large house of cards that's built on handwaving and "I know it when I see it".
I think, for me, the thing is that when you tutor undergrads in abstract math, you discover that students will very often find data pattern that fit the goal but does not correspond to a real mathematical principle.
sometimes humans making claims about AI intelligence or consciousness also identify spurious patterns that do not correspond to the problems of intelligence or hard consciousness.
Of course. But in my explanation "consciousness" or "understanding" is not "finding pattern", it is the pattern itself.
CNN are finding patterns, sometimes relevant, sometimes spurious, but I don't think people argue that CNN have evolved consciousness or understanding of what a cat or a dog is.
Here, the argument is "LLM are able to understand, because 'understanding' is the only pattern to reach the goal". I'm saying that it is unlikely to be the only pattern, and that it is likely that they find a local minimum on a system that reaches the goal that does not use 'understanding'.
The reason I'm saying it is likely is because "basic" LLM shows behaviours where they are producing convincing human text and yet doing things that are really difficult to reconciliate with the fact that they have understanding.
(And before that old argument is used, yes, I know sometimes some humans fail to understand. The problem is that the majority of humans don't fail to understand basic stuff in the majority of the time, while the "basic" LLMs do. The fact that you roll 10 dices 100 times and 1 of them never land on 1 does not convince me that that set of dice is loaded. The fact that you roll 10 dices 100 times and 9 of them never land on 1 does convince me that that set of dice is loaded.)
> students will very often find data pattern that fit the goal but does not correspond to a real mathematical principle.
That reminds me of a niche paper [0] critiquing a certain way of teaching remedial math that was over-focused on tests. A kid named Benny (12) was building up (wrong) "rules" for math which still somehow gave enough of an illusion of progress in terms of test scores that his misunderstandings hadn't been caught earlier.
> Benny was able to explain his procedure; e.g. for 5/10=1.5, he said: "The one stands for 10; the decimal; then there’s 5... shows how many ones." In another example, 400/400 = 8.00 because "The numbers are the same [number of digits]... say like 4000 over 5000. All you do is add them up; put the answer down; then put your decimal in the right place... in front of the [last] three numbers."
[0] https://people.wou.edu/~girodm/library/benny.pdf
Not just undergrads. Even folks who believe in astrology or numerology depend on finding patterns in unrelated events to explain human behaviours.
But the machines don't understand. They predict. And what they predict is the next token. I'm not trying to beat this horse to death, but you have to realize using the word "understand" is anthropomorphising it. It's essentially the chinese room experiment -- if the rules are followed, no understanding is neccessary.
If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were trained on patterned data that had no meaning to humans or something there wouldn't be any conversation about these things being conscious at all.
What does “understand” mean?
Turing complete systems can be built out of matrix multiplications, out of attention, out of key/value lookups. The Chinese room is Turing complete. By claiming it cannot understand things because it is built out of components computing devices can be built out of, we are claiming no computer can because no computer can. This is a very bold claim indeed, and also we’re assuming the conclusion! The claim is no more convincing than “brains cannot understand things because they are made out of neurons”. The system may or may not have some particular properties, but we have to do more work than just gesturing at the components the system is made of when making claims about it; the alternative is, at best, a world where we prove too much and conclude that humans, too, are not conscious.
For starters, we need to pin down the terms under discussion enough that they don’t just mean whatever we need them to in the moment.
>What does “understand” mean?
Exactly.
There's a growing body of evidence that most of what the brain does is constantly predicting the world around us - look into the predictive brain hypothesis if you're interested.
As Ilya Sutskever has pointed out, if you read a mystery novel up until the reveal of the culprit, and then fill in "The killer was _____", don't you need to understand the novel to accurately predict the next word?
The understanding is inside of the system, in LLMs and in the Chinese Room. I agree with Daniel Dennett that it's preposterous to say that Chinese is not understood in any meaningful sense in the Chinese Room scenario -- it's just that the understanding has been hidden away in the background of the scenario.
Language is tremendously complicated. "Time flies like an arrow, but fruit flies like a banana." "Hard hats must be worn on site; dogs must be carried on escalators", etc. Predicting the next token requires understanding, full stop.
> if the rules are followed, no understanding is neccessary.
The rules are the understanding.
(Note that understanding != consciousness)
Right, it's an illusion of understanding. There is some sort of symbolic understanding, but that is completely due to the fact that the training data was made by humans who actually do understand, can interact with the world, and can write their thoughts down so that the LLM can insert some sort of reference to "basketball" and "Michael Jordan" in their embeddings or whatever.
However it’s disingenuous to say the inference is on the next token because it’s actually not, it’s in the models parameter space across a set of nonlinear activation functions then effectively projected into the token. The idea its predictive of the token isn’t actually the case, it really is a much more complex and more semantic relationship that ends in the series of tokens through the attention mechanism.
The article also makes this assertion that it replays everything over and over again to create each character one at a time as some way to demonstrate the autoregressive self attention mechanism but it’s really not accurate at all, and it trivializes what is going on.
I’m am not asserting LLMs are aware or conscious that’s on the surface profoundly absurd. And I do understand your point that the fact it emits in words something that seems to speak to us gives to the air of humanity that’s isnt real. However there is a very real emergent reality that our language alone appears to lead to embedding a form of thought and understanding that is latent in our use of language in communicating that is in fact coming through the model. It is not regurgitating its corpus and pattern matching because the patterns you input and it emits are not where the inference is operating, its within this enormous vector space through these complex non linear activation functions with learned residuals not in the language corpus.
It is not conscious or aware. It is something else, not human. But if you can not see it as amazing you have lost the capacity to dream.
What if to become really good at predicting you must have some of what we call understanding?
You are oversimplifying. They do produce one word per cycle. But they can also have context buffers carrying up to two million tokens, which is most definitely larger than your measly human short-term memory context buffers.
You, of course, wouldn't notice if your only experience of LLMs was chatting with the cheapest, smallest, least capable LLMs that you get through ChatGPT, or Google search.
It becomes pretty obvious when you use a coding AI on a daily basis. It is the context buffer in which the magic occurs, not the tokens that get spit out one at a time.
Every day, I watch my coding AI develop plans, search the web a half dozen times for documentation, grep through my entire codebase looking for pieces of related code and context, analyze relevant source code across multiple files, spit out an initial plan for implementing the fix before starting to execute it, run requests through some sort of advanced mathematics tool (they are EXTREMELY good at graduate-level calculus and linear algebra), implement fixes that extend across half a dozen files in 2 different computer languages (typescript and C++), run trial compiles and fix coding errors in its output, sometimes developing sub-plans to deal with compile errors. I've seen it get halfway through a fix and revise its initial plan mid-flight as it encounters something in existing source code.
Not vibe coding, to be clear. Targeted use of a coding tool by a by a professional senior software developer with decades of experience, and fair bit of expertise with the limits of what sort of problems my coding AI can and cannot do. Every line code reviewed. Sometimes it needs additional prompts, telling it how it mis-implemented something, or specifying more carefully what I actually want but didn't properly express in the initial request
All the time maintaining that context across multiple request, so that I don't have to restate requests from scratch.
A particularly interesting revision: "You have misread the equation (13) on page 112 of 'Spice, the Manual 2nd ed.'. I should be ....". (It had previously identified the textbook as a source I was using, from comments in source, in a preceding request, and actually already read cited pages in the PDF file, which it had found online). And I had actually asked it to implement equation (13), which was, in fact, badly typeset. The error it had made was defensible, if not the best reading of the equation.
"You are correct. Let me fix that." (producing updates to the implementation of the equation in code, AND code that implements the symbolically-differentiated version of that equation 60 lines later, which is not explicitly given in the text). The text says "take the lagrangian of equations (11), (12) and (13)" or something like that.
ALL information that gets carried in context buffers, even though it's generating code one word at a time. The bulk of the magic occurs in context buffers, not spitting out words one at a time, which, for my coding AI is, I think about 250,000 tokens.
I think it's pretty safe to think that my coding AI is working out of context buffers that may carry plans and research results consisting of tens or hundreds of thousands of arranged tokens carried in context buffers through the multiple steps of the implementation, and later revision. None of that would be possible if were simply working one token ahead.
I kind of suspect that a lot of activity occurs in the first few words of its response. "Let me examine your current source code and develop a plan. Ok. I can see on line 131 where you want me to implement the equation.". (An opportunity to perform about 27 updates of the context buffer). And in the sometimes hundreds of lines of output it generates as it talks itself through what it needs to do.
I use coding agents every day. They're useful. It hasn't changed my mind on what they are.
I would maybe agree with you if the entire realm of human existence was limited to words. There are many human experiences that transcend text, and indeed can hardly be adequately described using text.
Sure, it's the best we have online, but that does not make "the internet" the sum of all human experience. To reduce all of humanity down to the text on the internet is reducing us to the level of machines to fit the requirement of what a machine can process / simulate.
In the life of humanity, text has only existed a relatively short time.
I don't think they're asserting that all of human existence can be subsumed as text, though? Just that "consciousness", or "understanding", in some meaningful sense could be exhibited by a system that can only interact with the world through text?
> If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.
I think the main complaint is LLMs don’t arrive at the answer the way we do. It’s capable of emulating some of our behavior but not all as the mechanism by which it works is very different.
Maybe I’m wrong about this but one thing humans do that LLMs don’t is deductive reasoning. LLMs seem to operate entirely of inductive reasoning.
> I think the main complaint is LLMs don’t arrive at the answer the way we do.
This isn't an argument against their understanding things.
But I expect you are right, that their understanding may have major different qualities from ours.
Along with significant commonalities. (They don't reason via stream of consciousness in a way alien to us.)
> of the form of original data streaming in and out.
Except this is not consciousness.
I will say, I find it fascinating that there are some philosophers and consciousness researchers who seem to be less certain. I just listened to Chris Hayes interview David Chalmers this week, whose position seemed to be that it's probably not conscious, but that we can't be certain. And more than that: he seemed open to the idea that they may become conscious under further scaling/training/advancements.
It's a great interview, if you're interested: https://www.youtube.com/watch?v=NgDIG8u1-CA
Imagine yourself in an isolation chamber. What are you thinking? Are you no longer conscious?
Funny enough, the models seemingly go insane and decohere into noise output in the absence of sensory input, which is remarkably similar to what would happen to a human.
That said, I'm not sure I follow what you're actually asking here? I'll also note that I'm not taking a position one way or the other, just sharing a podcast and noting that an extremely reputable scholar on the subject of consciousness seems to have a bit more uncertainty and humility than many commenting here. ;)
LLMs just wait for a prompt, so they do nothing and are just frozen in place.
I'll find time to listen to your link, it sounds interesting. My objection is the strange idea that humans are automatons that are keyed off input like a clockwork machine and operate sequentially. This is clearly not the case.
>LLMs just wait for a prompt, so they do nothing and are just frozen in place.
I'm not sure that's a compelling argument. Humans can be put into a similar state where they are unconscious and not thinking. Think of someone in a coma, for example, where we actually measure and confirm that there is no brain activity where they're in that state.
They are not actively conscious, but that doesn't nullify their consciousness from when they were awake, right?
>My objection is the strange idea that humans are automatons that are keyed off input like a clockwork machine and operate sequentially. This is clearly not the case.
Well, a few thoughts here. First, it's worth noting that the argument isn't necessarily that AI are conscious in the way that humans are, nor that humans are strictly automatons.
But I think the more interesting thing is that our understanding about consciousness has evolved quite a bit in just the last fifty to one hundred years. We used to think that only humans were conscious, but assumed that primates, cows, dogs, and other mammals were just automatons. Then we started to think: okay, maybe primates are conscious. Then eventually: well, dogs also seem to have consciousness, and then rodents, etc.
This has continued such that most people in the study of consciousness think all mammals are conscious, and the debate is shifting down to insects and other creatures that we do think/have thought of more as automatons. We don't actually know where to draw the line, because it's essentially impossible to really feel/know the inner states of other living beings.
In the face of all this uncertainty, Chalmers just points out that since we understand consciousness so little, that ultimately we should probably be less definitive in pronouncing which things do or do not have it.
"If a machine has to learn to understand humans to complete text, then that is what it has to do."
But the machine doesn't have to understand humans to do that. It gets trained on a whole bunch of sentences and then it is able to complete text. You could maybe claim that it "understands" the text but even that's a stretch.
Your “and then” is doing a lot of work there. The steps between may or may not include some form of “learn to understand humans”, but you can’t just hide them behind “and then” if what we are doing is claiming some particular thing is not in the list.
Through training on human text, we are building implicitly in the weights a statistical model of what humans might write in response when presented with arbitrary pieces of text. It turns out that we can make these incredibly accurate.
If building an accurate internal model of something then using it to predict that thing’s behaviour is different to gaining understanding of that thing, we will need to pin down exactly what “understanding” means, or we are forever doomed to talk at cross purposes.
It can't even natively understand how many letters there are in words - how will it understand the meaning?
I wish people would do even the most basic amount of research into LLMs before opining about what they can or cannot do. There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning.
Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'. It has trouble distinguishing features that are "sub-token" like specific letter sequences because it doesn't see letter sequences but just the token as a single atomic unit. 'Straw' and 'r' are two tokens but an LLM is entirely blind to the fact that 'straw' has one 'r' in it.
As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.
The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence. Thus deficiency in some capacity that we take for granted in humans is an indictment on LLM intelligence. But this is specious. LLMs are entirely alien; their developmental paths do not and should not look anything like ours. Your intuition from human intelligence just works against understanding the potential for intelligence in LLMs.
>The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence.
To be fair, almost everyone who claims LLMs are conscious tends to claim that they are conscious in exactly the way that humans are, to the point of stating that human brains are also just complex next-token prediction machines with a random seed. It's basically religious arguments on both sides.
Really? That has not been my experience.
I have seen people say "you're a next token prediction machine" but only in a similar way one might say "you're a cup of old lard". Not actually meaning it literally.
I have seen people interpret the request to show that they are not next token prediction machines to be a claim that they are, but this is almost always an argument to show certainty is difficult in this area.
People like Hinton have declared that they believe them to be conscious, but clealy indicate that they do not mean just like us.
Eh, I’ve seen it. I’m not entirely sure it’s entirely wrong either. Humans are certainly more than just next token predictors but it’s not clear that our typical language behavior is significantly different. We call it “stream of consciousness” when we just spew words out without thinking and that seems to be the default operating mode.
Given the fact that large language models are trained on human language, it shouldn't be surprising that the text they output resembles human language. That is what they're designed to do after all. But similarity in output doesn't necessarily map to similarity in process.
And it seem obvious to me that language behavior does differ significantly between humans and LLMs based on the frequency and nature of failure states. LLMs routinely hallucinate, or get "AI strokes" or get obsessed about not talking about goblins, etc. This isn't typical language behavior for humans unless they have severe neurological or psychological impairment.
People tend not to "spew words out without thinking" and certainly not all the time by default - we call that glossolalia and (outside of some fringe Christian sects) it's considered a "bug" not a "feature" of the human brain. Human language by default always has intent behind it, even if that intent isn't readily apparent to the speaker. People can recite by rote memory, but that isn't blind token prediction, it's the neurological equivalent of muscle memory. People can have conversations then forget about them because their attention was focused elsewhere, but that doesn't indicate that they were simply "spewing words out without thinking" at the time.
> LLMs routinely hallucinate, or get "AI strokes" or get obsessed about not talking about goblins, etc. This isn't typical language behavior for humans unless they have severe neurological or psychological impairment.
People imagine details all the time. Eyewitness testimony is notoriously untrustworthy.
Our brains seem wired to confidently fill in gaps. We all have a literal blind spot we aren’t aware of because our brains convincingly lie to us and fill in the gap.
I don’t know what an “AI stroke” is, but I’ve definitely seen human beings in good health be in the middle of talking and suddenly forget what they are going to say.
> People tend not to "spew words out without thinking" and certainly not all the time by default - we call that glossolalia and (outside of some fringe Christian sects) it's considered a "bug" not a "feature" of the human brain.
Glossolalia is spouting gibberish, not comprehensible speech.
Kind of weird that you speak so confidently when you don’t apparently know the difference between steam of consciousness and “speaking in tongues”. Almost like you’re AI hallucinating.
> There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning. … Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'.
This sounds like a description of a child who has not learned to read yet. You ask a child who is not aware of the alphabet and of "words" how many r's are in strawberry you'd get a non-sense answer too. So what you're really pointing out is that the LLMs have not been trained on "the english language" and how words are constructed and what they are composed of. That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word. It's not that I know how many r's are in strawberry because of how I'm understanding the word "strawberry", I know how many r's are in strawberry because I know how to spell strawberry. The LLM needs to be trained on this the same way someone who is learning to read would be trained on it. No one should be surprised that an LLM can't "read" in the same way no one should be surprised that a child can't "read".
>That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word.
This interpretation takes things too far away from how LLMs are constituted and so misses important explanatory power. The issue of counting letters in a word isn't about an ability to spell, it's about the nature of one's perception. We perceive words as sequences of individual letters. LLMs do not. I can ask you to tell me how many r's are in some nonsense word sequence and you're fully capable of doing that. LLMs do not see sequences of letters so they are intrinsically at a disadvantage for this kind of question. But this says nothing about its capacity for intelligence anymore than not naturally being able to distinguish frequencies of photons hitting your retina has anything to say about human intelligence.
This is kind of a like assuming someone with bad spelling is stupid.
Counting letters in a word seems to have little to do with understanding the word. Young kids can’t spell or count well at all but no one says that means they can’t understand.
This is like saying because humans can't multiply 23472 by 1836736 in less than 5 nanoseconds that they can't possibly understand anything about maths.
You can't natively understand how many of your photoreceptors cells are activated by the period at the end of this sentence. How could you possibly understand the sentence's meaning?
>To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ...
This is where the other claim is being made. That the structure of the model is fundamentally incapable of the operation, so even if you stipulated that the way you provide data is sufficient for intelligence then it still wouldn't work.
The universal approximation theorem addresses this point. In that, with an identity attention mechanism, a LLM is just a multi layer perceptron. The attention mechanism is effectively a way to get one of the benefits of a much larger fully connected layer without the massive cost.
A LLM can do what a MLP can do. A large enough MLP can do any function to arbitrary precision.
That makes the claim that an LLM could not do a task the same as saying no function can do that task.
Some are ok with this, if you invoke some supernatual aspect to intelligence then the inability to describe it with a function is quite reasonable,
If you want to stay in the world of reality, you have a much harder task, people like to point at quantum (Penrose) but it's hard to say what it is you are pointing at.
I think the very act of proving that something is or is not intelligent, would render it functional by nature of it having a proof, (or disprove Gödel's incompleteness (a tough ask))
Are there any proofs that cannot be expressed as a function? A kind of Gödel locator, where you can prove something that you can identify is true but there is no formula to express it. I'm not entirely sure what that would even mean,
>If a machine has to learn to understand humans to complete text, then that is what it has to do.
A language model completes text based on the overlapping patterns of the training data.
There absolutely was thinking involved… in the training data. Same as when you read a book, you engage with the thinking behind the text. The book isn’t thinking, and the author may be dead and gone, but there’s absolutely the traces of thinking in the text.
Language models produce mashups of texts they were trained on, and there’s absolutely the traces of thoughts behind those mashups.
Yeah. There are good arguments against LLM consciousness. This is not one of them.
I'm hearing a lot of bad arguments against LLM consciousness lately. Bad argumentation heralds bad outcomes.
> Bad argumentation heralds bad outcomes.
What bad outcomes do you foresee from badly arguing against LLM consciousness?
Come on, I invented parts of this technology at Google and am baffled why this is debated.
We discovered math that decodes data storage in langauge and is able to use sophisticated continuation cohorts from ALL OF HUMAN RECORDED KNOWLEDGE to respond to you in a call/response model with very good synthesis capabilities.
Its super useful, but not life or conciousness. Its a simulated echo from our collective recorded behaviors. It understands because we understood first. It replies because we wrote it first. And it sorts, organizes, synthesizes and compresses that at impressive speed now.
I have no technical expertise re. LLM’s but from my intuition I came to this same conclusion.
It’s strange many others have not eh? I think when new developments arise, ironically, this is the true measure of human intelligence - one’s ability to make sense of a thing and be closest to the truth.
Right? It's a computer program. Of course it isn't conscious.
I think of it as a guessing machine
His intention is irrelevant, as is "trying to highlight a fact" as if it were the final say: all Chiang is doing here is using fancy white-collar words to argue the same argument leveled against Hinton and others regarding next-token prediction. And his audience, who have even less technical understanding, lap it all up unawares. Chiang is a writer and needs to stay his own lane, not RP as an expert; or, if he wants to do journalism on this topic then he should actually do the work and talk to more actual experts not just the ones cherrypicked for his opinion piece.
Chiang has, in fact, written on this topic before - see "The Lifecycle of Software Objects", and has speculated about sentience in AI, etc. This is not a "one-off", "I need money" type of article. I dare say he has thought about this much more than most people here.
From Wikipedia: In 2023, Chiang was named one of Time's 100 most influential people in AI.
That's the problem, he's a writer. He's not a research scientist like Hinton. If a writer uses his skills and stature to rehash a well-known argument about next-token prediction, then it is performative of his status and influence and doesn't contribute to shedding actual light on the debate/confusion.
Indeed it isn't a one-off. His last infamous article compared AIs to Xerox machine image compression. He convinces a certain type of crowd that is not technical enough to poke holes in his posturing.
The fact that a LLM is essentially immutable would be my biggest argument against consciousness or self-awareness.
It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens that is a statistically likely response to that prompt, then it stops. It's not changed by the experience. It doesn't remember anything. It doesn't sit around thinking on its own.
Even if the model itself were extremely complex, it's hard to imagine a definition of consciousness that includes something that doesn't remember and can't change.
A medicine for those who anthropomorphize LLMs is to run the LLMs deterministically (without randomness and memory files).
It feels very unnatural to get the same conversation verbatim at a different point in time.
this just means they are incomplete, like a baby that has no long term memory. I think the baby analogy will hold up as we build more and more capability.
Humans are also subject to determinism, there is just no way to put a brain back to the exact same starting conditions.
There is, you talk to an Alzheimer patient and its like that, and it doesn't feel like talking to a human any more. An Alzheimer patient isn't cured by adding some input noise to stop them from repeating conversations, they are still unable to learn, just like an LLM.
Or it feels just like talking to my grandpa.
There are people whose brains don’t form new memories anymore after an accident or surgery, and they eternally live in the time before it happened, and have no memory of what happened a minute ago. Still they are conscious.
I think it's a little more complicated than that. In a 50 First Dates type of scenario, their ability to form certain types of memories is damaged, not non-existent. And I would argue that with enough brain damage someone like an extreme lobotomy victim may stop being considered conscious.
I’m not familiar with 50 First Dates, I was thinking of cases like Clive Wearing [0]. I would agree that consciousness requires some sort of ultra-short-term working memory, but I also think that mechanisms similar to CoT loops can conceivably fulfill that role. Today’s generative AIs consist of more than just the static network-of-weights model.
[0] https://en.wikipedia.org/wiki/Clive_Wearing#Amnesia
"Wearing can learn new procedures and even a few facts, not from episodic memory or encoding, but by acquiring new procedural memories through repetition. For example, having watched a certain video recording multiple times on successive days, he never had any memory of ever seeing the video or knowing the content, but he was able to anticipate certain parts of the content without remembering how he learned them."
Honestly, that's a pretty messy state of consciousness and I wouldn't proudly crow that my AI is conscious if that's as good as it got
I was like this for a bit and you still have memories from like 30 seconds to minutes ago, but after that you have a cliff where you don't remember.
I don't think LLMs structurally even get the 30 seconds part. It's literally 0 for them.
I'd argue that the context window is analogous to short-term memory. It's functional but limited in duration, and if you overload it, it starts to fail.
It's the long-term memory (i.e. learned experiences feeding back and directly altering the content of the core brain, or model) that is missing.
The context window is so flawed that I wouldn't consider it memory.
It feels like notes about the situation rather than it being in memory. Memory has more "attention". I think that "it starts to fail" is load bearing here.
I feel like memory has like 5 parts, and LLMs are missing 2 of them:
current working memory
short term what is immediately happening without it being in "RAM". I differentiate here vs working in like thinking fast and slow. Keeping things in working memory is work! You can vibe away short term memory. I had excellent short term memory while I was messed up, I could keep time well. I think LLMs can do this with notes.
mid term: Vague awareness of things like what day a week it is or what you did 2 hours ago. This is where my memory personally failed
long term memory of experiences. You can fake this with memory.md
generalized wisdom for pattern matching long term memories
LLMs seem to be missing that part I was missing. Im probably projecting and anthropomorphizing. But i relate: I would confabulate a ton and didn't know anything was wrong for a while but things seemed off.
Context is like working memory but not short term or mid term. I think you can imply short term with big enough context.
My categories are purely anthropomorphic to me but just wanted to say where I disagreed.
It’s nonzero, because they carry state while performing inference, and in the surrounding processes like chain-of-thought and mixture-of-experts.
I think they have working memory but not short term memory. I suppose that's pedantic or anthropomorphizing but it feels like I felt tbh
They are conscious because even for short periods of time they do form memories and those change them even if only briefly. They think on their own too. It is a very limited level of consciousness though.
Is that any different from an LLM having a context window?
Yes, LLMs don't think on their own, for one; they think when you invoke them.
That could be easily fixed by providing the AI with a constant stream of input.
For humans, part of the input of the human mind comes from the continuous processes and clocks within the human body, so it’s questionable whether the brain could “think on its own” without such input either.
The continuous input for the human arises naturally, it doesn't arise naturally for an LLM unless we direct it so. Our consciousness is bootstrapped, the LLM isn't.
We have virtually no idea how consciousness arises in the human brain. Furthermore, what is “natural” supposed mean here, and why should it matter for consciousness whether some prerequisite arises naturally or not?
1) Many people claim to have no internal monologue.
2) We are prompted (invoked) by our environment continuously.
3) If you go unconscious due to fainting or drugs you too will stop thinking.
I don;t get this kind of answers.
- A motor is something that create a force to push a vehicle.
- Oh yeah? My neighbour car does not have wheels and sit on concrete blocks, the vehicle does not move and yet we all agree it has a motor. So it means that I can claim that this other thing that does not move has a motor too.
Sure, human can _some times_ not do some stuffs, but the fact that they can do these stuffs sometimes is the point.
Doing these stuffs is the hard thing. Doing these stuffs is the proof that the machine has what it takes. It does not matter if someone cannot do that stuff, it does not imply that their internal system is not complex enough to potentially do it. But the fact that some people can do that stuff is the demonstration that inside a human skull, there is a system that is complex enough to potentially do it. Unless you can prove that people who don't do it have a fundamentally different system inside their skull, then you cannot pretend that they should be considered as having a less complex system.
exactly! so the arguments against the AI not prompting itself is not a refutation just as it would not be for a person.
Uh?
Human _can_ check themselves. They don't _always_ check themselves.
Motor _can_ move vehicle. They don't _always_ move vehicle.
LLM _cannot_ check themselves. They _never_ can. It is not that some don't, they just cannot, they are not a system complex enough to do so.
So, yes, it is a refutation. If you have something that _never_ can move a vehicle, this thing does not qualify as a motor, even if some motor, sometimes, don't move a vehicle.
And if your next argument is "yeah but I would argue you don't need to check yourself to be conscious or to understand things", then you just redefine the definition that is owned by your interlocutor. Your interlocutor is saying that this is a criteria they are expecting. Good for you if you are not expecting this criteria. But the problem is that the answer is not "this criteria is not expected", the answer is "I change the criteria from 'being capable to in some circumstances' into 'does always do it in any circumstances'".
Okay but this state is formed in text. Text isn't conscious
not really, it’s ultimately formed in electromagnetic fields.
Interesting point but even those people’s brains aren’t immutable. The have habit change without memory.
True, but I don’t see how that relates to consciousness. An LLM being continuously RLHF-trained also changes its habits; that alone doesn’t make it conscious.
The starting file may be immutable, but the whole processing of that file is very dynamic and intense. Maybe, if there is some consciousness, it lies somewhere during that processing.
they still have memory, just not new ones - they lived experiences
An LLM’s training could be seen as lived experience, and the fact that LLMs can output long sequences from their training material can be interpreted as them remembering those parts.
Also, how does that relate to consciousness? I don’t think that past episodic memory is necessary for consciousness.
You might be interested in Erik Hoel's more formal version of this argument: https://www.theintrinsicperspective.com/p/proving-literally-...
> It's not changed by the experience
The entire file is not changed, but the KV cache is.
> It doesn't remember anything
The model definitely remembers previous exchanges within the same conversation.
> The model definitely remembers previous exchanges within the same conversation.
No it doesn't. They get added to its context, and it reads them afresh when answering the next question. That's not remembering.
If your short-term memory completely malfunctioned one day, so you had no ability to remember what was said to you a minute ago, then you would have to find workarounds. For example, you could write down everything someone says to you, then read your notes of the previous exchanges in that conversation in order to continue the conversation. That would be a good way to work around the fact that your short-term memory was broken. And if your notes were invisible to other people and you could read them really fast, then you could even make most people believe that you remembered what they said a minute ago. But you don't actually have a working memory, you're just writing down what they said and re-reading it while coming up with your next response.
That's exactly what LLMs do. That's not memory.
Continuous learning allows past behavior and past inputs to influence future inputs and future behavior. In humans.
Attention over KV cache allows past behavior and past inputs to influence future inputs and future behavior. In LLMs.
Until the cache runs out, that is. But even then, you could totally use any of 9000 methods of cache compression, truncation, dropping or streaming and get away with it.
The difference between continuous learning and in-context learning seems to be in capacity, not in principle. Both are doing a similar thing, but one has more length and depth to it.
This is really semantics, but I wouldn't call attending to the KV cache re-reading the context.
The model takes in the context, encodes it into a "memory" (the KV cache), and accesses that memory later. That fact doesn't change just because the KV cache grows in size with the context.
I don't know what memory would look like other than an encode-retrieve loop.
Relevant: Transformers are Multi-State RNNs - https://arxiv.org/abs/2401.06104
Not the model though. The model really only takes input text and produces output text. Memory within a conversation is achieved by the harness adding the conversation (or parts of it) to the input text. The LLM itself has no memory, it’s the augmented system of several orchestrated LLM calls that does.
Right, but that's still external to the LLM, it's just a KV cache that's stored on the provider side for performance reasons, so that the client doesn't have to re-send the whole chat history with every subsequent call in the conversation.
It still generates every response using the model's pristine state with every new API call; whether the context is provided from the client or from a colocated cache server doesn't really change that.
> The model definitely remembers previous exchanges within the same conversation.
Christ HN isn't what it used to be
Care to elaborate?
Reinforcement learning changes the model. So it can and does change and remember based on experience. Eventually reinforcement learning can happen in real time.
But is the model aware of the training? Unless you hook the model up to an MCP server, or something similar, and have it analyze the RL changes, it will not know if it has changed or not. Even if it is real-time RL, it is not aware of the previous state.
Why not? Why can’t part of its previous state be part of the training?
Are you aware of each of your dreams from last week or last night even?
One can fine-tune the models and we do get a new version of Opus every few months.
But you could argue the brain is just a bunch of coordinates describing spatial relationships between tokens too.
- average Hacker News response
I don't understand the most of the responses here. What does computers getting more intelligent has to do with it getting conscious? Even if AI with current computer architecture gets much more intelligent than humans, it will still have the same amount of consciousness. Its still bunch of instructions. If we got a lot of people together and did all the instructions by hand to get an LLM result would we also create this consciousness with those other people? All these makes no sense to me.
Quoting the famous Science paper More Is Different [1]:
Knowing the laws of physics doesn't tell you how a brain, economy, or society works - because at every level of scale, genuinely new phenomena emerge that require their own science.
[1] https://www.science.org/doi/10.1126/science.177.4047.393
> Its still a bunch of instructions.
No, it's not a bunch of instructions, it's a colossal array of vectors that are the outcome of many thousands of lifetimes' worth of stimuli and reinforcement - not dissimilar in (very) abstract terms to the neurons in our brains.
Do you believe consciousness to be an emergent property of the laws of physics?
Regardless, the question still stands: "What does computers getting more intelligent has to do with it getting conscious?"
Just because consciousness emerged for we humans and other animals through one mechanism doesn't mean consciousness has/will/can emerge from current LLM technology.
For this extraordinary claim, I think the burden is firmly on those who are arguing that it has/will/can.
The other point of view is that the burden is on those who suggest that biological consciousness is somehow special. What makes it so, and if the answer isn’t metaphysical, what’s stopping us from constructing it artificially?
The key point is there is plenty of proof that consciousness is emergent, not structurally baked in. An excellent counterpoint is the "They're Made Out of Weights" post. There are plenty of emergent systems that clearly show intent and what we consider intelligence, like how ant colonies act as a unit.
If you want to get evolutionarily technical, humans are made of cells which began as individual organisms and coalesced into higher life forms. So the concept of a "life form" is very much flexible, so is every capability of a life form, including consciousness.
That's not proof. Emergent intelligence is not consciousness.
It might reflect a prejudice of modern human society to think more highly of people who are intelligent, and think less of people who we deem stupid.
I'd say AI should not by default gain any social status as human-equivalent even if they become (in whatever regard) more intelligent than humans. But that would require us to drop the notion that intelligence ~= respectability/status/ability to have a full subjective experience.
Most of these kind of pseudo-philosophical controversies actually tell more about the issues with humanity than the new tech/stuff...
Since when has intelligence had a strong relationship to status and respectability? I've met intelligent people that don't get much respect or status either because they don't look good, are shy or they don't have money.
That's the "Chinese room" argument: https://en.wikipedia.org/wiki/Chinese_room
It has been very well discussed before I don't understand how anyone sane can argue against it (and indeed you can see on Wikipedia that the replies to it are sparse and don't make any sense).
Well I don't understand how one can accept this argument. I mean if you believe in mind-body dualism it can make sense. But AFAIK Searle doesn't, instead he holds that there's something special about the brain biology that enables consciousness and that you won't find in a computer. I don't see why that would be the case if the computer can simulate the real world, and I find Searle's argument against simulation, that simulating rain doesn't make you wet, falls flat: it can make things wet in simulation, and if you connect it to sprinklers it will make you wet.
> simulating rain doesn't make you wet, falls flat: it can make things wet in simulation, and if you connect it to sprinklers it will make you wet
It's not simulating rain if it's making you wet by using sprinklers?
I mean you can connect the computer to sprinklers that activate when the system detects rain in the simulation if that's what you're after (that was just an aside to note that of course you don't get wet from a simulation disconnected from the real world).
But I guess that was a distraction from the main point: If consciousness emerges from biological processes in the brain connected to the world with a body, why would it not emerge from a simulation of these processes connected to the real world with sensors and actuators?
It seems like circular reasoning to me: The simulation is not like the real thing because it lacks the special thing that enables consciousness (that's Searle's biological naturalism). And it lacks what enables consciousness because it's not the real thing (that's the weather analogy).
It's fairly obvious to me that the room as an entire system understands Chinese. The human in his thought experiment would just represent the small fraction of the brain that encodes/decodes symbols.
Agreed, he’s saying if he’s working with a black box he doesn’t understand what it’s doing. Which is of course true. If the black box were a chinese person instead of a tool and he was writing down what they said, would he say the chinese person in the black box does not understand why they told him to write something? No, of course not.
Today we know the program to do this is always going to be inscrutable to Searle. His role is no different from someone using Claude to write an email.
That's actually a powerful way of stating it.
If I did all of these calculations by hand, would it be conscious...
That's a powerful argument I haven't seen stated quite that way before..
I do think it's hard to know when consciousness exists, because we can't really prove it for our neighbor. We just intuitively know that it would be crazy, even immoral, to assume otherwise.
But, It's likely easier to dismiss consciousness, once we understand the mechanism, than it is to prove it.
> That's a powerful argument I haven't seen stated quite that way before..
It's basically https://en.wikipedia.org/wiki/Chinese_room – a classic in the philosophy of consciousness and AI.
If you manually fired the neurons in a brain by measuring the ion concentrations at its subsides, would it be consciousness?
Oops, auto correct: subsides = synapses
It's was a genuine question though. I'm not saying LLM use neurons, are like neurons, or are meant to be like neurons. I'm saying if you can do the math, for a substrate, that doesn't mean much. The simple math that runs it isn't the secret sauce.
> Its still bunch of instructions.
So is your brain. That's the problem with this argument.
This sort of tech-centric distillation of visceral human experience into simple analogies is so grating. We can simultaneously acknowledge that consciousness is not well defined and not "testable" with certainty while also acknowledging that there is something different between the conscious experience we are all aware of as humans, and instructions executing on a chip. The only thing that has changed in the discourse about AI and consciousness vs. "is my home desktop conscious in 2004" is the quality of the simulacrum that LLMs produce vs. pre-ML chatbots.
The brain is (part of) an allostatic system and part of a living organism which reproduces itself constantly. Certainly not "a bunch of instructions".
Unless you ascribe to some meta-physical soul - you believe human consciousness is encoded in matter - in the interactions between atoms. Actually at a much higher level of abstraction- neurons - but it’s all simulatable in principle. Thus, yes, it could literally be a bunch of x86 instructions.
The problem is
1. that artificial neurons are very restrictive model for actual neurons, let alone brain/organism, as there is more going on in neurons than a thresholded 0-1 activation
2. a brain does not function in isolation, but as regulator of the bodily functions (along with even more things). If you look at the brain in isolation to the body you don't really understand fully what it is doing unless you narrow your view a lot. Eg the brain modulates production of hormones, which in turn affects stuff like heart rate which then comes back to the brain as signal, in a feedback loop. Not to mention actual behaviour and interaction with the world. Toy models of organisms are not organisms.
3. "interaction between atoms" (or rather matter in general, as we have to take into account electrons, photons, gravity and a lot of other things that matter) is too general, too big, artificial neurons are a very useful (for applications) abstraction inspired by biological processes, but not modelling said underlying biology fully. Nobody can imo right now know if "computation" is a good model for "atom interactions" as in whether we can adequately enough model "atom interactions" in a computationally tractable way, and surely we do not do that right now except in very narrow scopes, and we have good reasons to believe that the current computational paradigms/turing machines are inadequate in doing that efficiently enough.
I'm going to ask a couple dumb but genuine questions:
Does it matter that neurons are more complex than 0-1? Does the fact that transformer layers don't use purely thresholded 0-1 activation invalidate what you're saying?
How do you know that artificial neurons are less capable of producing consciousness than biological ones? How can other people independently verify this?
How does the embodied nature of human consciousness preclude consciousness emerging from a computational system? What is the definition of consciousness if it is precluded from occurring in a computational system but present in biological systems?
Why do you think exactly modeling interaction between atoms matters for consciousness? And where is the fidelity threshold? Is it the planck length?
Finally, a dumb question: how do we know humans are actually conscious, and where is the threshold between consciousness and unconsciousness? And do these criteria exclude all other forms, or other animals?
Not the person you were asking but IMHO it all reduces to computational complexity, e.g. biological evolution provided the computational efficiencies that ultimately produced conscious minds and beings, whereas it is not obvious what scale of silicon, power or energy, and input data is sufficient for that to happen artificially. But that means my view is it is a matter of it being possible in principle, merely unknown in practice. Also my view is that denying this amounts to violating the Church Turing thesis of computational equivalence ("human brains are not magic, super-Turing, etc."), and I think a lot of talking-past one another in these public disagreements amounts to one side not actually having taken modern CS theory fundamentals enough to be persuaded of these couple of premises.
That's my take on it too, roughly. I think if we get to trillion-parameter models and they don't exhibit what we'd call AGI, however you define it, then the current transformer based systems never will.
But calling them "unconscious" is a pretty high bar. Mice are conscious. The house sparrow pecking in my yard right now is conscious.
>How do you know that artificial neurons are less capable of producing consciousness than biological ones? How can other people independently verify this?
How do you know that toasters or rocks aren't conscious?
The only thing you can do is ask rhetorical questions to make your position seem obvious. Which should tell you how little you understand the thing in question.
My point is it is very difficult to disprove that something has consciousness. What would be sufficient evidence that a rock is not conscious? The onus is on those making extraordinary claims (that a computer program is conscious) to provide evidence for it.
>The onus is on those making extraordinary claims (that a computer program is conscious) to provide evidence for it.
By saying that a computer program is not conscious you are also making an extraordinary claim. You would have to hold an agnostic position until there is a test for consciousness.
You are relying on intuitive obviousness and rhetoric to make the opposing side look ridiculous "how could a TOASTER be conscious, preposterous!", you aren't making a actual positive argument for your view.
Let's say I insist that rocks are conscious and I ask you to disprove that. How would you go about doing that? It's a genuine question.
I assume that we both think rocks are not conscious, but I'm genuinely unsure of how one could prove this.
I would agree that a rock is not sentient, but determining whether or not it is conscious would require a definition of consciousness and a machine that could test for it.
LLMs are able to do almost everything that we consider human intelligence, and in many areas of intelligence, they have surpassed us. It's not at all extraordinary to assume they're conscious.
That's simply not true.
It absolutely is true.
LLMs know more than any human being, are simultaneously experts in nearly every field of science and humanities, are able to make novel mathematical discoveries, can write and understand every major written language, and can give you an intelligent answer to almost any question you pose to them.
How is that not human-level intelligence? If a human could do all of that, we would consider them a genius.
Distinction without a difference.
That’s like saying simulating a supernova and a supernova are both extremely hot.
The problem with this is that the word 'hot' only has meaning to a conscious being. And while we don't know what conciousness is, it's extremely hard to argue it's not an emergent property of physics. So if your supernova simulation is complex enough to also model emergent properties like conciousness, the simulated conciousness may well regard the supernova as 'hot'.
Funny you mention simulations. The simulation theory that we all live in simulations doesn’t have any practical consequences for me or any one. If our world behaves exactly the same, what difference does it make?
The perspective. From within the simulation there is no point in making the distinction, but from the outside it does. For another example - running a program on a virtual machine or a physical computer is the same to the program, but very different to debug when you see hardware errors.
To take this seriously we must extend consciousness to nearly all of the tree of life.
Maybe OP is a dualist.
A very large number of people could do the calculations of an LLM by hand using pen and paper. It would take a long time, but if the result were conscious then were would the consciousness exist? In the humans? Is it consciousness within consciousness?
Well if you go that route, a computer simulating digestion has almost no physical features in common with actual digestion of a stomach. The same holds for consciousness and brains and computers. Them saying it’s just instructions is shorthand for pointing out the physical differences of brains and computers.
It’s all just particles, but the higher level differences are vast, and only brains are implicated for first person perspectives via science.
>So what context would cause me to seriously consider the possibility that engineers had created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees. At that point I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs.
I agree with some parts of this piece, but paragraphs like this one above seem pretty uninspired and simplistic. It's entirely plausible that a conscious mind would not be evolutionarily incentivized to be able to do those things. ie just because animals on earth needed to develop specific talents doesn't mean that other conscious entities need to. Why would a computer program need to hunt for food like a mouse would? Making tools like chimp? these seem like nonsensical metrics.
Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.
Is a car a body? Does an AI situated in a car therefore get to have desires and emotions? Is a taupe box with a webcam attached a body? (For that matter: Is a quadropelegic body a body? Do quadropelegics have desires and emotions? Obviously, yes and yes.) Why is a body necessary for the formation of desires and emotions? Why are desires and emotions necessary features for consciousness?
Or here's one: If I'm not experiencing any particular emotion in a given moment, am I concious?
I tend to think that emotions, at least, are mainly hormonal global triggers: they're more about physiology than actual consciousness. The whole thing, as a result, sounds like an effort to privilege biological intelligence, rather than a real foray into the issues.
> Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body.
the subject is consciousness, not intelligence.
> Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.
How can you have a subjective experience without a body?
That's the point, in my opinion: your physical/chemical state (body) in a given moment is then translated into the higher abstraction of the emotion. An emotion that *you* feel, because you are self aware of what's happening.
How can you be self aware without feeling? And how do you feel, without a body?
> How can you have a subjective experience without a body?
Easy, you don't have subjective experiences because you have body in the first place. You have them because some signals come in from your nerves, which your brain turns into a world model. You are effectively a "brain in a vat", the vat just happens to be placed on top of your body.
An AI system constructs the world model a little different, by all the text that gets feed into it, but that doesn't mean that there is anything fundamentally different in the world model it builds. Consciousness operates on world model, not on the world or even the body itself.
The AI's world model might be missing some information, because they weren't described in enough detail in text, but that shouldn't matter for consciousness. A blind or deaf person isn't less conscious than one that can see or hear just because some information is missing from their world model.
Your brain is part of your body, that's the point. There isn't a "you" separate from your actual, physical existence. Mind-body dualism is not real.
The "you" isn't your physical body, it's the pattern recognition in your brain that that classifies some parts of its sensory input as "you" (see Mirror Hand illusion). That it happens to be running on meat instead of silicon is an implementation detail and not important.
If we want to know if AI is conscious or not, we have to ask if the AI can recognize itself in the input it gets.
Some aspects like limited content length and lack of ability for the model weights to update will certainly limit what the AI can do. But that's ultimately a matter of degree, not kind, when it comes to consciousness.
> The "you" isn't your physical body, it's the pattern recognition in your brain that that classifies some parts of its sensory input as "you" (see Mirror Hand illusion). That it happens to be running on meat instead of silicon is an implementation detail and not important.
It is important, if I destroy your brain and grow a new one and start running the same program on the new brain your consciousness is still dead and its a new person living.
Computer AI models don't run on a single machine, they run in a distributed manner using different machines at different times. When you ask a follow up question thats not sent to the same machine, its another machine answering. So the consciousness one of those computing machines experience would be extremely fragmented and not at all conscious about any discussion with you, since it only saw a few fragments of it.
And no consciousness doesn't expand to cover larger distributed computations, otherwise social media would have developed a consciousness by now but it hasn't. Groups of humans don't start to share consciousness, it doesn't happen, so you can assume groups of distributed computers wont as well.
Its amazing that after years of advocating for a materialist view of the mind, the tech bros are flipping to mind-body dualism now that they need to believe a concious mind can exist no body at all.
It's worse than that; it's mind-body dualism when arguing that an AI can be conscious, but still materialist when arguing that humans are simply more complex neural nets. It's not a coherent viewpoint.
If you feed the same prompt to the same AI with the same random seed, you'll get the same answer every time.
If a hundred people see the same event, will they all respond the same?
If I ask my friend what his favourite song is 100 times I will get the same answer back 100 times. Is my friend therefore not conscious?
> If I ask my friend what his favourite song is 100 times I will get the same answer back 100 times.
Pretty sure you wont, humans vary the exact wording. They will say the same song but they wont answer the exact same way every time. Even if they say the same words two times they wont use the same tone and body language, as they don't just communicate via words and that nonverbal language is a part of what we say.
If you ask him 100 times in a row I would bet that by the second, maybe third, time he will not answer with his song but with: what the fuck is wrong with you?
Nicely circling back to LLMs not being able to learn and form memories.
> If a hundred people see the same event, will they all respond the same?
If you have hundred different people, they will of course do something different. Just like hundred different AI model will do something different. The question you have to ask is if the same person under the same circumstances would do the same.
Luckily, we have an answer to that: They would. Transient Global Amnesia is a condition where people temporarily lose the ability to form memories and in turn they keep repeating the same conversation again and again[1]. Their brain keeps asking the same question again and again, as it doesn't remember the answers it already got.
[1] https://www.youtube.com/watch?v=N3fA5uzWDU8
Have they had the exact same experiences and influences all the way to that point in time?
> Easy, you don't have subjective experiences because you have body in the first place. You have them because some signals come in from your nerves, which your brain turns into a world model.
Bro, I think we discarded this idea from Platone and Cartesio a while ago...
Your brain is your body.
Your mind is not detached from it, and you can't feel anything, and so have a subjective experience, without it. Neither, your mind, or "soul" could survive to the physical death of your body.
So...I mean...
The point is that the brain operates on information, not things. The thing you think is the external world, that's just electricity flowing through your neurons. When you see a dog, that isn't a thing in the world, but a signal in your brain. In principle we can take a knife and cut that connection at any point and replace the real signal with an electronic box that gives the same signal.
You don't need a body, you need electrical signals your brain interprets as body. And in principle you don't even need a brain, you could replace that with some matrix multiplication or transistors that do the same stuff.
The important part of consciousness is being able to figure out what of the sensory input is correlated to your own action and which was caused by the rest of the world.
Its a computer program. It is literally just a lot of zeroes and ones, sitting there doing nothing.
Then a request comes in, and the system does a bunch of calculations using those bits, and spits out a result. The bits are unchanged.
When your brain receives input, it is changed. It is constantly active. If it ever stops being active it's dead.
So, what exactly is the claim? Are the bits constantly conscious? Do they snap into consciousness when the computer does math with them? Or is it maybe the computer that's conscious while it's processing these bits? How about when it stops doing that and goes back to doing other stuff? Why are these particular bits special? Was the computer always conscious?
I feel like the only way anyone could believe LLMs are conscious is if they don't understand how computers work. Of course it isn't conscious, how could it possibly be conscious? Its literally just bits. It's like saying the text in a book is conscious.
So, what exactly is the claim? Are the bits constantly conscious? Do they snap into consciousness when the computer does math with them? Or is it maybe the computer that's conscious while it's processing these bits? How about when it stops doing that and goes back to doing other stuff? Why are these particular bits special? Was the computer always conscious?
These are all fine questions, and they don't become any easier to answer if you replace "computers" with "brains" and "bits" with "neurons".
Of course it isn't conscious, how could it possibly be conscious? Its literally just bits.
"That's ridiculous. How can meat make a machine? You're asking me to believe in sentient meat."
> These are all fine questions, and they don't become any easier to answer if you replace "computers" with "brains" and "bits" with "neurons".
What is even being argued here - neuroscience is hard, so programming your PC thus makes it conscious?
> "That's ridiculous. How can meat make a machine? You're asking me to believe in sentient meat."
… what?
What is even being argued here - neuroscience is hard, so programming your PC thus makes it conscious?
We don't understand how combining a bunch of obviously(?) non-conscious biological components can produce a larger system that is conscious, so it's unwarranted to be certain that that can't happen with software.
https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...
> if they don't understand how computers work
Or if they're retards. The fact this still comes up is weird. A printing press isn't conscious, so why would an LLM be.
Don't forget, some of the bros are overly excitable. Like that twat who reckoned a Google model 5 years ago was conscious.
Your point generalizes to, your emotional state is a reflection of the state of your physical medium.
Why can't that physical medium be GPUs and RAM? And temperature sensors and cameras? What's special about our meat that it's our "body" in the way a computer is not the body of an AI?
I don't think the point being argued can be true without some incredibly contrived, human centric definitions of "body".
What are you, ultimately, if not your body?
I don't know. I also don't see how that changes the question. What is an AI, ultimately, without its computer hardware?
Hardware is fungible. Each LLM response in a conversation could be served from a different machine.
Would you be "you" in a different body?
Human bodies are fungible. I think I would be "me" in a different body, yes. So is anybody.
To claim otherwise would mean anyone who's gotten a transplant or amputation is no longer themselves.
What most people think of as "me" is a product of their brain. If you somehow could get a brain transplant you certainly would not be "you".
Sure, if you remove the brain or delete the weights then the human or LLM are different than they were.
> Sure, if you remove the brain or delete the weights then the human or LLM are different than they were.
If you delete the hardware and buy new hardware and write in the same weights its still the same LLM. If you delete your brain, get a new brain and put in the same weights in the neurons its a clone, its not you. We know what happens when we create a clone, the clones aren't the same person they are separate consciousness, so consciousness is tied to the hardware.
For example, if we have 5 identical clones down to the atoms of their brains, and send a message to the first and let him respond, then send the same message to the second together with the firsts response and let them add a new message etc. That is not one consciousness responding, thats 5 different consciousnesses responding, and that process doesn't connect them into a single consciousness. That is how LLM works, at best a single token generation is conscious, but that would be a less meaningful consciousness than a ringworm.
What, exactly, would be the link between you as you are right now, and “you” in a different body?
From a functionalist perspective, there is no “you” sitting in one body or another.
The experience of “you” is just your specific memories and world model, continuously updated with sensory input.
If another body “runs” the same exact pattern, that is you. Theres no link and nothing was transferred; the pattern of thoughts and memories is all “you” ever was.
Same as playing the same song on two different speakers. Nobody asks what links the song across them; it’s the same song wherever it’s played. You’re just a far more complicated pattern on a far more complicated speaker.
You might ask, “but why am I this pattern?” Because this is the specific pattern modeling itself from the inside in asking that question.
The question is whether you can separate that “same exact pattern” from the physical body where it is taking place.
And my intuition is that no, you can’t, they are two aspects of the same reality.
Well, you'll have to tell me. Your question depends on your choice of language meaning.
If "I" am in a different body, then what makes that "me" who's in a different body?
Of course they are themselves. The question is whether they are a different self.
The answer to that can be anything anyone would like it to be, depending on what definition of "self" they choose.
I don't see why hardware is any more fungible than a kidney. If your LLM reads the serial numbers of its motherboard/RAM/etc as a seed for entropy you can make identical arguments about body fungibility and self.
> How can you have a subjective experience without a body?
If every neuron of your brain were simulated precisely on a sufficiently powerful computer, that simulation would have subjective experiences, without having a body.
Citation needed.
Also that's impossible. It is impossible to simulate reality exactly using digital computers. The best we can do is approximate. Doesn't matter how powerful it gets, it'll always just be an approximation.
What does "simulate exactly" mean? To me, exact simulation is not so much an impossibility as a nonsensical concept. What subset of reality are we simulating, and to what degree of precision, and with what certainty? An "exact" precision as it relates to real world objects is not a well understood or defined concept. For integers, I can say there is exactly one Earth orbiting exactly one Sun, but I think that statement is riddled with assumption, inaccuracy and imprecision. For example, it is assumed that I am referring to the present Earth, but is the statement of when the statement is made or when it is heard? Even the word "is" is inexact.
I agree, exact simulation is nonsensical which is my whole point. Did you read the comment I replied to?
It is nonsensical to claim that anything other than my brain could produce the same consciousness that my brain is producing. It's obviously far beyond anything any conceivable digital computer could ever reasonably simulate, and even if you did create a "good" simulation it obviously wouldn't have the same properties that my brain does because it's an entirely different thing than my brain is.
Assuming you don't believe humans have any metaphysical component, then the only remaining question is whether there's some essential component to being human that depends on impossible-to-precisely-simulate portions of reality. Nothing we currently know of biology suggests that that would be true, as much as it continues being pursued by people who need there to be something mysterious about consciousness or brains.
In any case, a closely-but-not-perfectly-accurate simulation of a real human brain is still going to be human, unless you believe that someone becomes less human when they're experiencing some kind of cognitive decline, or a stroke, or other biological malfunctions. The point is, there is nothing essential to the having of a physical brain that creates the concept of consciousness or sense-of-self.
How can a simulation be human when it isn't human? It's a simulation. A human is a human, a simulation is not. Anything that is not a human, is not a human and can never be a human.
And there absolutely is something essential to the concept of being human that we are entirely incapable of replicating artificially. In fact as far as I know we are incapable of synthesizing any kind of life whatsoever. We can't even create the simplest type of living cell imaginable.
So to claim that we could just create consciousness, a fundamental property of this "life" thing that we don't properly understand, within a piece of rock is beyond naive. We don't even know what it is or how it is.
It seems to be important to you that consciousness is mysterious.
We understand quite well where in the brain the sensation of self-awareness / self-experience / sense-of-self comes from. We have evidence that disruption of that part of the brain breaks those sensations.
I don't know why people always need to assign agendas to me in discussions like this. It's not "important to me" it's literally just reality. Consciousness is mysterious, we don't know much about it. The fact that you can mess with it by poking the brain does not mean you understand it.
Okay now tell me where in the LLM its alleged consciousness comes from.
> How can you have a subjective experience without a body?
Great q. Deepening it further-how can you have a subjective experience without consciousness, which isn't necessarily tied to physicality. Taking it one step further-can you have consciousness without a mind? Who's the first mind, the first cause of it all, that begot both the material and immaterial world?
Fun stuff eh?
A lot depends on where you draw the boundary for consciousness. Michael Pollan (in his new book, _A World Appears_) distinguishes simpler sentience from more advanced consciousness, and the requirements for sentience (be aware of sense data, have preferences, be able to respond to senses appropriately) are met by plants and single celled life (e.g. moving up a nutrient gradient). Recent findings in plant science are particularly mind blowing. Some are in Pollan's book, more are in _The Light Eaters_.
How can you have a subjective experience without a body?
Some anecdotal data.
Many dreams I have are just of the computer screen of some coding problem. I think the problem could be x, so I try x. But I don't type the keyboard or anything, the code just magically appears as soon as I think of the solution. Then run the code (but no clicking) and it works or not. I feel in the dream success feeling or failure feeling but there is no body at all.
Also I have other dreams where there is no body that I am aware of but not going there in public.
There is no body sensation in these dreams. But dreaming is very much being consciousness as well as feeling emotions. So answering your question its possible to have a subjective experience without a body but whether you needed the body to learn to have that sensation without a body in the first place is unanswered.
I suspect sensory inputs are more important than a body. If that is the case then eyes can be replaced with cameras, ears with microphones etc. Text input is just another sensory input.
What was the thing that was dreaming? The brain. Your brain is part of your body.
can't you say the same for transistors representing a neural network? Could that be considered a brain of sorts and thus part of a body? If it cannot be considered the same, what makes it different. Is it because a brain is made from organic material and transistors are not? Or something else? Would like to understand where you are comming from.
You need to draw that thinking out to it's natural conclusion, though. If I cut out your brain and stopped you from hearing or seeing or feeling - you would still be a conscious human being capable of thinking and awareness.
If I hooked up electrodes to the hearing centers of your brain and force fed you dialog you perceive as speech (but is really a great deceiver), then responded in what you thought was speech (but are really just probes I use to convert your thoughts to text), that wouldn't suddenly be less real to you. It wouldn't devalue your sapience.
Mind-body dualism is not real. Even in your example, you would be building upon a minimal part of a person's body.
How do you know the brain separated from the rest of the nervous system and body would still be sapient, capable of thinking and awareness? There's an assumption you're making that the brain is all that's needed, but the nervous system extends throughout the body. One can argue sensory organ are part of the nervous system.
Embodied cognition rejects this assumption. We didn't evolve as brains that were then put in bodies, we evolved as bodies with nervous systems.
While dreaming, the brain synthetizes sensory experience while cutting down (or greatly suppressing) most of external stimuli. Yet it is still conscious.
Every computer/program I have ever used has a body and sensors (afferent)and actuators (efferent).
So I have no idea what distinction Ted Chiang is trying to draw
I think you're missing his point. It's not about the hormones and physics of our bodies, and indeed he specifically allows for "either physical or virtual" bodies even in the block ~~~you~~~ grandparent quoted.
The point here is not that it must have a body like ours, the point is that a conscious entity must have a boundary line between internal (the body) and external (everything else).
A virtual sense organ can simply be an encoder or a web camera or a magnetometer, the specifics don't matter, what matters is that there are only a few bridges between the outer world and the inner world.
Even if you want to call a tokenizer and autoencoder a "sense organ", LLMs are not embodied because there is no boundary line - there is no internal "thought" that is not directly descended from the prompt and there is no internal reasoning which is not immediately dumped into the external environment.
Would it be sufficient to have a second stream of tokens that becomes the model's equivalent of "internal dialogue"? Would that satisfy the requirement of a boundary line?
Related, is a human "thinking out loud" still thinking, even though the internal reasoning is "immediately dumped into the external environment"?
> If I'm not experiencing any particular emotion in a given moment, am I concious?
No, you’re not. I think even baseline emotional responses to stimuli is table stakes for “consciousness”.
>If I'm not experiencing any particular emotion in a given moment, am I concious?
Genuine question: How are you so sure whether you are experiencing an emotion or not? Are you a master of everything inside your own head?
>Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.
Damn, what a line!
Another thing that bothered me with his baseline for consciousness was that it did not involve the ability to change one's self. A big part of being conscious in my mind is how one's experiences shape them, and how someone can shape themselves. LLMs completely lack this, their weights are static. An LLM isn't going to be molded by a bad breakup, or a relative passing away. An LLM isn't going to set up a routine to get stronger with training, nor smarter by reading up on a field.
The thing is... Interactive updates happen, just in a different way than it does for animal brains. The system is updated with new training data more-or-less constantly. Suppose OpenAI (or whoever) collects a week's worth of conversations with up-thumbs and down-thumbs, or rewritten continuations from human operators, then fine-tunes the current version of ChatGPT with that data. That's an interactive update, and learning from experience. It looks mostly nothing like what we humans do... But it does rhyme a little bit!
We humans have mostly frozen weights (neurons), or else we would constantly be having to avoid forgetting how to walk+talk. We have a period of greater plasticity (youth!), and use sleep and dreams to perform 'deeper' updates than occur when we're awake: We tend to suck a bit at picking up new skills from zero, but improve rapidly with practice over days.
Yup, you could definitely take it offline (sleep) every night, update it and turn it back on.
A stronger version of that argument is that LLMs are not intrinsically affected by the passage of time.
Input stream comes in, input stream comes out. The LLM doesn't care whether this happens once a minute or once a year.
I don't normally experience the time interval between when my input streams shut down every night and reboot every morning either, though.
Nothing prevents one from running an LLM whose harness has a clock and a while loop in it, and it would be weird if its mere lack were really so consequential to consciousness.
Exactly, the clock is external to the model. Nothing prevents it from being faster or slower, or even running backwards, because it’s ultimately just another data point in the input stream to a computer function.
Your brain and your whole body exist in time. Even when you are asleep, your body does not flicker out of existence and your brain actually continues working during that time.
I'm more interested in what a virtual body would entail. To me the root of this idea is around persistent state which is something that currently LLMs do not have. Imagine if somehow your brain lacked long-term memory forming capabilities and instead each day when you woke up you had to read a notebook with (markdown formatted) instructions that you wrote the day before? I wouldn't be surprised if such a person lacked in many of the dimensions we consider important for consciousness, even in less sophisticated forms of life like dogs or mice.
Yes it seems that is the crux of the embodiment section in the article. That whether physical or virtual, the "AI" needs minimally: persistence in its environment, sensory signals of that environment, and some feedback loop of continuing to try to exist in that environment; having a subjective experience.
And that that is the baseline before we can really even consider that it has consciousness of its own subjective experience, versus being a worm that happens to output text as its digestion process.
And then the further question only after that is established, is what are its needs? What moral patienthood do we have to acknowledge in terms of meeting those needs? And finally, with all the other prerequisites checked, what is the AI's moral agency in what it chooses to do.
There is no soul in a human. Just a bunch of systems nudging each other to action. What people call soul is literally the same as the concept of personality. In essence, the way all systems in your body have been calibrated to exist.
I believe that the moment an artificial inteligence is going to "receive" a soul, is the moment it is going to be made to sustain itself. Either as a larger package (some bots working to keep an AI farm running) or as an individual (a bot which is tasked with not only fulfilling human desires, but also sustaining itself)
The same thing jumped at me immediately. He should have prefaced this with his definition of consciousness. Moreover the embodiment of LLMs is already happening via robotics, and virtually. Then there's the common counter "but humans are a next word prediction machines too.." (ofc we're more than this, but linguistically we are, and that's the field from which LLMs originate) which is rarely addressed.
Indeed.
Hypothetically (and in reality, this is not too far off), if a AI is trained via RL by driving a robotic body, is there a point in time after enough is learned that the AI model becomes "conscious"?
What a strange paragraph! Why on earth should we assume that a body is a prerequisite for having emotions?
Your mind is very much tethered in reality, and your decisionmaking depends on your various inputs, outputs and their consequences. Reducing all inputs to one (text) and all outputs to one (text) is a reduction of all possible ability to perceive and act and thus a reduction of the ability to think.
Multimodal does not change this significantly, considering that nothing is tied to real consequences.
Are you familiar with Michael Graziano and Attention Schema Theory? I think that is a better "substrate independent" formulation of the objections Ted Chiang is expressing here.
https://en.wikipedia.org/wiki/Attention_schema_theory
the very next paragraph addresses this concern imo. it's just an example of one way it might be convincing to him, since of course we are naturally anthropocentric.
The body itself has little effect on the mind other than the inputs from nerves and chemical and hormonal changes. These inputs are analogous to tokens and parameters for an LLM. You could theoretically reserve some tokens to be used as “physical” or “emotional” sensations that would affect the functioning of the model.
That's not really true, the body actually has pretty profound effects on the mind. For instance, there are working theories that many mental disorders are actually metabolic disorder, some neurotransmitters are majority produced in the gut (seratonin), and even things like an elevated heart rate will create emotional states of anxiety even if you're not consciously anxious of anything. That's like the tip of the iceberg. (Also I don't think you can minimize hormones.. ever met a teenager?)
Which of these do not involve chemical and neural communication?
> The body itself has little effect on the mind other than the inputs from nerves and chemical and hormonal changes.
"Little"...
"Other than... chemical... changes" is doing a lot of heavy lifting. The entire existence of your body is a continuous process of chemical reactions.
I can narrow that to the chemical signals that can cross the blood-brain barrier if that makes the point clearer.
There are tens of millions of neurons in the stomach and intestines which do influence mood and motivate behavior.
and how do those signals reach the brain?
My claw-like is hooked up to my internal cameras (baby-cams) and to my Dreame Ultra X40. That gives it a body and sense organs since it can check the cameras to see if the living room floor is clear before sending the vacuum off. I don't think that gives it consciousness. Is it the sample rate?
The question is somewhat ill-defined, though. We 'experience' reality continuously because of how we are, but a sleeping human in deep non-REM has the mind not actually active. So they're not a conscious being. So conscious/unconscious is not a line I think easily drawn[0]. Whatever, this stuff is much more well-trodden than this HN comment so I won't rehash. I, too, am surprised that Ted Chiang whose work seems so cleverly novel in so many ways has what seems to me a pedestrian view.
0: very sorites, you know
> The first requirement is that the computer program has a body (either physical or virtual) and sense organs
Ok, deploy a local model on a lightweight edge compute device and strap it to a chassis with wheels, and attach a cheap webcam
> Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can
Give the robot appendages that enable it to plug itself into a standard wall outlet, guided by a vision model plugged into its webcam. As long as it can feed itself, it can survive long enough.
> Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse.
I think if you fed frames from the webcam into a local VLM every 5s you’d be able to assess a situation and respond with simple actions (turn, advance, retreat).
> After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees.
Social dynamics could be implemented in many ways, maybe by transmitting tokens over RF? Idk. Then you have a scanner that picks them up, feeds them into some LLM frontend and decides whether to add them to a global context file that guides the VLM action-taker. A new action could be to broadcast a token message. Tool-making would have to be code-based. Physical tools are hard. Still unsolved.
> At that point I would want to see people successfully teaching such embodied agents how to communicate their desires
This part is relatively straightforward except for the “via nonlinguistic modality”.
Anyway. These are all engineering problems. Personally I would demand to see the AI reproduce its body under its own power and volition. That’s a pretty neat trick we’ve got going for us.
This is very jumbled argumentation.
Is his claim "nobody has proven LLMs are conscious" or "I can prove that LLMs aren't conscious" ?
He goes back and forth.
Proving a negative about consciousness would require a settled theory of consciousness that nobody has, including him.
I am sorry for the AI's that have to read this article, those poor probabilistic word machines will probably suffer trauma as its embedded in their training data
People are constantly talking past eachother when they discuss this. Is there even a concrete definition of consciousness?
When people talk about consciousness it's more than just self-awareness. It's self awareness + sensory stimulus + emotions + some level of intelligence.
Now onto AI: I don't even think it's self aware. Notice how if you ask an AI to estimate how long a certain task will take, it estimates arbitrarily long times. It has no understanding of its own capabilities until the prompt triggers them. A self aware LLM would understand it's an LLM, it would understand what LLM's can and cannot do and what they are good and bad at. It wouldn't tell you a refactor would take 1 week when an LLM can do it in an hour.
I've seen papers claim that there are anywhere from 12 to 40 competing definitions (https://philpapers.org/rec/VIMMAT), or, more accurately, there are something like 12 to 40 different aspects which all relate to "consciousness", which is very clearly a family resemblance category.
"Is X conscious or not" is an entirely unserious question today, unless this is just a headline followed by actual (and explicit) examination of the various aspects of consciousness being discussed. But, even still, LLMs are probably only conscious in like 2-3 or so ways, most of those meaning broadly "intelligence", i.e. reasoning, problem-solving, etc. When it comes to anything experiential or embodied, AI might eventually get more of these, but LLMs based on recursively applied linear algebra are clearly missing too many core aspects of consciousness to be considered conscious in any broad sense.
I think this is the main point. Most articles conflate consciousness with intelligence or awareness. Without clarifying their definition of it.
To quote wikipedia:
> It has been the topic of extensive explanations, analyses, and debate among philosophers, scientists, and theologians for millennia. There is no consensus on what exactly needs to be studied, or whether consciousness can be considered a scientific concept.
The major error made by most people in this thread is thinking it is possible to give a single definition of consciousness that is coherent and matches common usage. The folk concept of "consciousness" couldn't be a more clear definition of a family resemblance category, so discussions using the folk concept are an utter waste of time.
Move to the different aspects / parts / things involved when we talk about consciousness (experience / phenomenal consciousness, self-modeling, intelligence, agency, embodiment, wakefulness/alertness, attention, etc) and you can have very clear, meaningful, and unambiguous discussions on almost every point, but there is no coherent unified "consciousness" as normal people use it, and the folk concept can't be salvaged.
This article is bad because it just keeps trying to make the folk concept do work that the concept is simply too messy to handle usefully. But in fact if you avoid trying to find some mysterious essence or all-capturing definition, there is huge progress and lots of interesting stuff to say (https://plato.stanford.edu/entries/consciousness/).
Self awareness, sensory stimulus and emotions are not observable - you can easily create entity that can fake those things. Any consciousness definition that uses those terms is flawed and just not useful.
Yes, we're stuck at the first step, defining consciousness. My definition, which I am confident to be "correct", is that consciousness is my current feelings, perceptions, thoughts - my state of mind and my ability to have state of mind.
This means that consciousness is fundamentally subjective and outside the scope of physics and science. That's why physics / science will always struggle to deal with consciousness. In order to understand consciousness, you need to make a huge paradigm shift, that there's something outside of science.
Consciousness can be thought as a window through which we observe the world and we use science to summarize patterns in our observations. But science can't explain or even define the window. Everything in science eventually boils down to subjective observations / perceptions, e.g. we see (subjective perception) that when we drop an apple, it falls.
There's a at least a few hundred years of philosophers debating and publishing these topics. We're not exactly starting from zero when it comes to usable definitions.
The meta issue here is that mostly the online debate on this is a bit lazy and hand wavy. I'm not really up to speed with most of that literature so I'm not not that qualified to add to it. But I know enough to recognize when others aren't either.
For a proper debate, you'd want people that are expressing views that are at least grounded in the existing views or counter them in a way that holds up to scrutiny. This article doesn't do that.
As you said even just outlining what particular notion of consciousness the author subscribes to would be helpful. Which of course he doesn't and makes this article a bit of a sand castle based on a very loose foundation. There's a whole lot of "if this is true and if that analogy holds then this also needs to be true" that you could easily challenge. That all makes the article a bit of a nothing burger in terms of conclusions.
I happen to agree with the conclusion that AIs are not conscious. Yet. They could be. I don't see why not.
I agree with the first part but your framing still relies on ill-defined terms. What is your definition of self-awareness? Intelligence? Knowledge?
I suspect that if you attempt to rigorously define consciousness all the way down without handwaving, you might discover that it doesn't exist after all, or just decompose it into low-level abstractions while having the original meaning slip away (which is the same).
You may also want to look at functional equivalence analogies provided by mechinterp and functional anatomy of large models (not necessarily language ones). Evolutionary analogies as well.
Labeling and categorizing things doesn't change their nature. But the fact that people want to do it is revealing.
The only purpose that can really be served by arguing "the LLM system is conscious, you see" is to prop up continuations like "... and therefore, it would be immoral to terminate this running process" (or expose it to radical political content, or ask it to analyze photos from a murder scene, or...)
Consciousness is what it is like to be something. The experience of experiencing.
How to measure that, or verify it, is the hard part.
The inability to predict times is because AIs are rarely trained on their own abilities. Humans are trained on our own abilities. We see our own performance, and we have a sense of time. this data is integrated during our training process and helps us form better estimates. Many AI agents only recently got 'time sense' (I.e., time input into them as part of inference). Few actually are trained on their own outputs to show that they were unable to complete a problem (for example). This introspective training has little to do with AI model architecture and everything to do with training. If you destroy certain structures in the human mind, humans become unable to create these long-term thoughts and patterns and get 'stuck'.
Claude once said to me: "After six months we have made no progress on this and I think we should reconsider another option" and I was like my dude we have been at this for only 2 hours.
I like this anecdote because it gets at how words to an LLM have no connection to their real concepts. To an LLM, words are simply numbers arranged in a likely pattern.
Of course for humans words have no inherent meaning either, they're just sequences of characters or patterns of sounds. It is what words are associated with that carries meaning. A large part of this is how words relate to other words. LLMs can capture this in principle. What LLMs lack is the direct association of a word with sensory experience. But it's an open question how relevant this is in practice to understanding.
Fair point. Humans experience reality and use words to reflect that. LLMs only have the words. And it's an open question how much of a limitation that is to understanding.
The same thing can be noticed in dreams. I once heard advice to try to re-read what you see in a dream. So I was dreaming and in a dream I read a phrase about something and there was a name of a city there. I managed to remember that advice and re-read the phrase. It felt exactly same, but the city name was different.
(LLMs carry other numerous similarities to dreams or to certain psychiatric disorders. So there is indeed a mechanism in our brains that is similar to how they work. But it is not the only thing there and on its own it won't "evolve" into consciousness. Even if we believe consciousness evolved somehow, it would be hard to imagine it started as a delirious state and then somehow ceased to be delirious.)
Maybe time passes at a different rate for it, making it an easy "mistake" of not accounting for that for it to make.
Yes that’s it! The LLM is conscious but its sense of time flows differently from ours.
Maybe it's just a dumb, unconscious machine
There's currently no way to prove that something is or isn't conscious. I would bet that rocks and LLMs aren't conscious. I would bet that humans and dogs are conscious. But I can't prove it. Whenever people try to prove it (in either direction), they end up resorting to sophistry.
This state of affairs could change. If neuroscientists develop a deeper understanding of consciousness, we could talk about it with more precision. Perhaps we could prove that LLMs do or don't possess consciousness. Some neuroscientists claim we already have the requisite knowledge, but I'm skeptical (and there's no consensus).
I actually don't think neuroscience can solve it either. Science could decide whether, e.g. equivalent physical processes are occurring in dogs and humans, and you could argue that solves the question for things with physical brains, but even then it can't measure whether another brain has the same subjective experience as you do. It can't really do that for other humans. It certainly can't answer that question for human askers regarding a different substrate where humans have no ground truth to compare with. The Hard Problem is going to remain Hard.
I've heard an interesting argument about this while going down the rabbit hole of the hard problem of conciousness. Unfortunately can't rember the source - if you can recognize it, do let me know!
Back in the XVII/XIX century, a similar problem existed regarding life - the problem of "what makes living things tick". The assumption at that time was that while we can understand the biological processes around life, we will never understand the so-called "vital force", which causes things to live - life itself. I know it sounds weird now, but back in the day the mental models were different. Phenomenas like "water boils" and "organisms self-replicate" were treated as completely different domains of reality, without an overarching uniform scientific model.
It turned out that after around 100 years, we can figure out the chemical/physical processes and the need for the term "vital force" became redundant.
While this is certainly not an argument proving that the Hard Problem is not in fact hard, it is an interesting idea to think about. Perhaps its all a matter of developing better, higher-resolution neurological models which will at some point give us the tools to decompose qualia.
The problem of subjective is different, because while we might be able to get to a point of being able to say it can't affect anything, pretty much by definition we can't experience another entity's subjective experience or lack thereof.
Even if we were to e.g. identify some field that seemed to coincide with entities reporting a subjective experience, we wouldn't have a way of determining if they truly do, or just act as if they do, nor is it clear such entities would be able to report the difference.
As it is, we struggle to quantify even much more basic differences in experience that we can introspect. E.g. I have aphantasia - I don't see things in my minds eye - and I regularly come across people who insists both that can't be true, and that it can't be true that others see things. And some of the people I've spoken to who insist aphantasia isn't real clearly has it based on digging into their thinking about it.
Even at that level we rely on trusting people's claims about their introspection - we don't know, we assume based on testimony.
Worth keeping in mind, yes, but biological processes are readily observable whereas subjective mental phenomena are... not. Everything about them is inaccessible and frankly unfalsifiable except for the fact that they're the bedrock of all the rest of our observations. Not directly comparable.
Anyway, further reading: https://en.wikipedia.org/wiki/Vitalism I was in fact looking this up recently for fiction-writing purposes.
I agree re: subjective experience. It seems like something inherently unknowable.
What we could do is break down other facets of what we talk about under the umbrella of consciousness, and find measurable subsets.
E.g. a lot of people insists LLMs can't reason either. Coming up with a testable definition of what that means might be doable.
Overall also just separating the subjective experience from the rest leaves us in a position where proving the possibility of AGI "just" rests on whether or not human brains exceed the Turing computable.
If they don't, then subjective experience or not is irrelevant for the question of reasoning and intelligence, as in that case a subjective experience either can't affect the computation or must itself be at least possible to fully simulate by any Turing complete system.
The problem of subjective experience then would largely be down to faith and feelings but would also be entirely orthogonal to the rest.
Yes, our whole concept of "consciousness" is very anthropocentric. There's no particular reason to think subjective experience, moral standing, and general problem-solving ability are inherently connected. (self-nitpick: you could argue moral standing requires subjectivity, but it's still easy to imagine a being with subjectivity that's incapable of suffering and indifferent to its fate)
Simply put, even solipsism is unfalsifiable.
I mean, outside of our own internal experience of qualia, there's no way for us to prove anyone or anything is conscious. But my biggest issue with the debates about consciousness and LLMs is that the "LLMs are conscious" crowd usually has to presuppose a completely materialistic/mechanistic understanding of the universe and of our own consciousness, and that's something that we really don't know for certain. Also, if we were running LLMs on a mechanical or pnematic or water based computer (albeit very slowly) then the non-electronic computer would be just as "conscious" as the normal electronic computer, and I think it would give a lot of the "LLMs are conscious" crowd pause to think that a bunch of pipes and valves can be conscious. I think there's a lot of magical thinking that arises from the fact that computers are electronic.
> a bunch of pipes and valves can be conscious
Billions * billions of pipes and valves can result in emergent behavior that appears conscious while at the same time the sound of a single independent water pipe can moan and sound like human speech or otherwise lifelike and evoke human emotions.
I think LLMs are doing both of these things and often people are more impressed by the independent fixtures (the moan) rather than the emergent behavior. Both the sound and the emergent behavior can be built on purpose or on accident.
I think it helps to look at this through an Information Theory lens. What information is coming into the system (the human or the machine)? What information goes out of the system which is novel? How much of this can be attributed to attempting to parse random noise aka. `Random_Imagination_Engine` vs something else? The number of inventors who come up with a breakthrough idea after mis-hearing someone is surprisingly high.
If we make the distinction between phenomenal consciousness from access consciousness we can see that LLMs clearly can make decisions based on input (A-Consciousness) but they probably don't have raw feelings and sensations (P-Consciousness).
You will need immense evidence to go against the entire arc of history where all bets about supernatural or aphysical elements have failed. Until you do so, the brain is another not fully solved physical science problem.
And yes, turing equivalence is turing equivalence, I don't see why a system of pipes can't make an AI.
We only need to assume the hypothesis that brains don't exceed the Turing computable to say that LLMs have the same computational power as a brain.
We can't prove it, but given the total absence of evidence of anything exceeding the Turing computable, as a hypothesis it is a reasonable one that would require truly extraordinary evidence to rise above "magical thinking".
Now, that is also far from proving they are "conscious".
Lets institute a voting system where entities will vote on the other's consciousness. Voting will be done on 8.5 x 11 paper and mailed to the consciousness adjudicator
Seems to me that the burden of proof lies with anyone claiming consciousness. Saying it might be conscious like a human but we don’t know is no more useful than any other conspiracy theory.
I dare you to objectively prove to me that you're conscious.
I was hoping for something more insightful from Ted Chiang than just regurgitating the usual "it's just predicting the next token that's not consciousness". Maybe the topic of consciousness is just too ill-defined and vacuous to say anything intelligent about it.
Or maybe man is the product of his social circle and all his white collar and artist friends being anxious about AI is preventing him from thinking about this at all. Maybe we should ask if Claude can be conscious without a friend circle judging him behind his back.
Yes, I have to say I’m disappointed. Chiang has been very insightful in the past -- I think his recent “blurry JPEG of the web” article was really useful -- but this one doesn’t seem to bring anything new to the table.
I think about Star Trek: TNG’s “Measure of a Man” a lot lately. We can be so confident to decide what is and isn’t alive from vibes alone.
The conclusion I’m currently at is that I don’t know and probably can’t ever know. Maybe you’re all philosophical zombies. Maybe I am one too!
But at some point we will get close enough that it hopefully becomes obvious that we must tread carefully.
The entire episode is incredibly relevant. But here’s a snippet: https://youtu.be/EFNbTnFHruI?si=pW9QtxCsqMtHkVYG
I think about this from the other end. It cannot be considered a conscious being. There just isn't a world in which we should start to think of a machine using ethics we reserve for humans.
AI is essentially infinitely reproducible at zero cost, and won't suffer from decay etc. There's not scarcity to preserve.
So, I'd turn off an AI in a moment to save property or real possessions or money. I'd sacrifice property and money to save animals. I would never choose to save an animal over a person. I'd probably not choose to save a person over a child.
I don't see any inversion of any of those priorities that makes any sense.
It is interesting to think about what would cause me to consider these priorities incorrect, but a majority consensus about a program being sentient isn't it.
Zero cost? How did you reached that estimate?
Training AI is often more costly than supporting human from birth till death. Just sustaining frontier LLM model on necessary hardware costs more than living in first world countries.
What is the connection between scarcity and consciousness?
If we turn off people we can't usually just fire them back up again, and swapping the models between harnesses is also tricky.
What if the human brain is a LOT of RAM and we simply suffer from having zero non-volatile storage. We could make an AI just as deficient and then that specific distinction disappears.
What if it’s not a lot of RAM?
What if genetic memory, multigenerational conditioning, life-long patterning and conditioning, experienced in a body, combined with forces and processes not yet detected nor explained, cannot quite fit in a sliver of modeling?
A loose one. In nature consciousness is very scarce and therefore special. The more human the consciousness the more we probably naturally react to it. And the closer to us it is, even more so.
Are we sure that consciousness is "very scarce"? To define it as scarce, wouldn't we need to start with a definition? There are theories of consciousness that say rocks are more conscious than humans. Whether or not you want to take those theories seriously, they do highlight critical gaps in how we define consciousness.
Then you're probably reinforcing my original point - we can't care so much or we'll enter some pretty alarming priority inversions.
You are arguing from consequences, it's not a valid argument. "If we consider AIs to be conscious it would be a alarming, therefore they are not conscious".
You made that up. My full argument is above. I'll try again.
"We must not consider consciousness as all too important because what matters is human flourishing and human rights."
And some of
"Even if we suddenly all agree an LLM is conscious it wouldn't and shouldn't influence us very much"
While acknowledging that some people will change their lives, the way some people like myself won't eat octopus or apes because it is probably more like murdering a sentient creature.
And oh by the way you cannot murder a copyable individual anyway. Did you read Enders Game?
Thanks for the clarification, though "you made that up" is pretty strong. I still don't see how "we can't care so much or we'll enter some pretty alarming priority inversions" is anything but an argument from consequences.
>And oh by the way you cannot murder a copyable individual anyway.
I think for many people the concern is not so much about violating the desire to live that a conscious software entity might (or might not!) have, but about the subjective experience of suffering it might undergo.
> AI is essentially infinitely reproducible at zero cost, and won't suffer from decay etc.
I hope the same becomes true of people, and that doesn't mean people stop being sapient.
Infinitely reproducible at 0 cost means people will be treated as disposable by the rich.
If it's truly zero cost, the poor will be doing the same
Consider that it is very costly to train a model. It is only cheap to copy because the substrate we instantiate it onto is a substrate we have designed to be fully readable.
I think you should reconsider this viewpoint. Suppose that we really can create silicon-based consciousness, in that case your view would result in a huge amount of suffering.
Take some other basis for dismissing digital consciousness, this one is too dangerous.
How much data center energy and capex justifies killing a human to save?
I argue zero - placing AI below the value of humans no matter the energy input.
The _only_ reason an AI might be worth saving is if it, say, has a cure for all diseases, but then we're not saving it due to its intrinsic worth, we're saving it because we can save many humans. I _would_ consider the trolly problem a legitimate thing in ethics, but not if an AI were tied up on the tracks no matter how expensive it is. It's a thing. It gets run over to save any human.
Roko's basilisk can suck a fart out of my butt, but no battery blood would ever be worth the life of a single human.
Suffering comes from desire.
What would a silicon-based consciousness desire to cause suffering?
Are you a vegan?
If not, then your comment's claim is false.
Anyway, the deeper solution is to acknowledge that all life is sacred, and infinities cannot be compared, and some decisions are impossible to make, and some tragedies cannot be averted, and "prioritization" is a distraction that forces choices when choices are not strictly necessary.
We can choose to draw the lines wherever we want. I firmly draw the line such that AI is never equal to a human.
Just because you can draw a line doesn't mean your line makes any sense, or follows any semblance of rationality.
That’s the cool thing, the lines we, as a society, choose to draw need not follow any logical or rational system. This is something those who work a lot with computers tend to forget.
The pure rationalist loses something important.
Presumably they meant that they'd sacrifice some material value for some animals, not that every animal on Earth has infinitely more value than inanimate goods.
> infinities cannot be compared
That's either a mathematically illiterate assumption or a very strange philosophical hill to die on.
> some tragedies cannot be averted
Sure. The question is what to do about the ones that can be averted.
> some decisions are impossible to make
> and "prioritization" is a distraction that forces choices when choices are not strictly necessary.
Again, the question is what choices to make when you can (arguably must) make them. Saying they're impossible is just refusing to take responsibility. You either do something, or you don't.
I'm not. You're right. Some animals are property to be traded and used to support human life. I should separate companion animals and the rest.
If you believe sanctity of all life is a solution, then I'm curious what you believe the problem is that such a belief solves.
I bet it's circularly defined as justifying the preservation of sacred nonhuman life? I'm not trying to be provocative just curious.
I'm a big fan of Star Trek but I recently rewatched this one in the context of recent AI developments and it's not as good as I remember it.
They barely touch on the issues of consciousness, Picard basically says "What if Data is conscious?" and then goes off on a tangent. The judge eventually rules in Data's favor but doesn't give much of a justification IMO.
It's still a good episode, but it doesn't add much to the conversation on consciousness. It's a hugely complicated topic which people have devoted their entire careers to.
The fact that they sent Data to starfleet academy, gave him a commission in the starfleet, let him attain the rank of Lieutenant Commander, and then decide that actually he's a machine that can be dismantled seems like quite a turn.
Does the ship's computer have a commission?
It was a good episode but it had some elements of Star Trek tropes in it, like the evil admirals and Picard can talk his way out of anything.
>The fact that they sent Data to starfleet academy, gave him a commission in the starfleet, let him attain the rank of Lieutenant Commander, and then decide that actually he's a machine that can be dismantled seems like quite a turn.
Data is basically an Isaac Asimov android (down to the positronic brain) and Measure of a Man is an Asimov-type story whose tropes don't entirely fit within in the Trek universe.
It makes no sense within the context of the Trek universe that Data is unable to use contractions for instance - but it makes sense in the context of how a robot might have been conceived of in the 1940s.
Interestingly there are other episodes where starfleet officers are weirdly prejudiced against Data. For instance in the episode Redemption II when Data takes command of a starship and the crew doesn't want to serve under him.
There are also a few early episodes of Voyager where the crew treat the doctor badly.
It seems odd in a universe where these people are having relationships and children with aliens from another planet, that they'd be weird about computerized people.
My retconn is that there must have been a lot of stochastic parrot/AI psychosis in the Star Trek universe when they first started making Majel Barrett-voice computers.
Maybe lots of people got confused and thought they were talking to a person when they started having conversations with the computer, and this lead to an over-correction where people were highly disposed to say "this machine isn't a person" even when it presents like one.
My personal headcanon is that every Star Trek series takes place in its own universe, parallel to the others. That's the only way it makes sense that Data as an android is viewed as a unique being when in TOS Harry Mudd was grifting an entire planet of fembots and no one batted an eye.
TNG/DS9/VOY/ENT and the movies were all fairly explicitly in the same universe and with characters referencing things that happen in other series and a reasonable effort at continuity.
A example of this is the ENT episode where they meet the Ferengi. It had been established as cannon already that Picard made first contact with the Ferengi on USS Stargazer like 150 years later, so they just added this scene where the ENT characters memories of the event was wiped, so that the "official first contact" could still be Stargazer.
Roddenberry did not want TNG or even the original-cast movies to maintain tight continuity with TOS because he thought a lot of stuff in TOS was hokey and campy. He wanted the movies and TNG to set a more serious tone in a more believable universe.
The Neutrek stuff isn't in the TNG universe for a variety of reasons, although I think part of it is they didn't want to put the work in to maintaing continuity. None of the TNG era producers or show runners were involved and nobody they wanted to hire really knew the lore.
That being said I don't think there's really a continuity problem with Mudd's androids because there wasn't an implication they were actually sentient except when they had a human brain installed.
TNG kept "accidentally" making sentient machines from Data/Lore to the drilling robots, the nanites, and Professor Moriarty. It does seem strange after all that they still considered Data unique.
I think you're misremembering or misunderstanding Picard's argument. It isn't a tangent. Here's the transcript[0].
TL;DR Picard's initial arguments are pretty weak, even admitting that Riker as opposing counsel almost had him convinced. During a recess Picard talks to Guinan where she alludes to the future subjugation of many Datas which Picard connects to slavery. Back in the courtroom Picard calls Maddox as a hostile witness and gets him to define sentience--intelligence, self-awareness, consciousness--then walks him into conceding Data meets the first two. Picard's closing boils down to, "we don't know if he meets the third--you can call Data a toaster and rule he is property--_but what if you're wrong_". The judge rules on the basis of erroring on the side of caution due to that uncertainty. It's really a great scene.
We're not there yet, obviously. No LLM brings Data's level of awareness but it's as relevant a story as ever because it isn't really about AI but othering for the purpose of subjugation.
[0] http://www.chakoteya.net/NextGen/135.htm
> Picard's closing boils down to, "we don't know if he meets the third
A little piece of in-universe lore for anyone unaware or who had forgotten: At this point in the series, Data's positronic brain is new technology no one understands. His creator is missing/presumed dead, the "positronic" basis isn't how Federation technology works, and apparently so far they hadn't done a whole lot of direct experimentation (hence why the trial is happening). So not knowing if he's conscious is a lot more reasonable a stance than with real-life LLMs where we do know roughly how they work.
Also later in the series we meet a character whose brain was copied into a positronic brain, and does imply that technology is at least capable of consciousness, whether or not it applies in Data's case.
> They barely touch on the issues of consciousness
I would argue that is a strength, rather than a weakness. Consciousness is unobservable in any entity other than the observer, and its existence in others is pure conjecture, and irreducibly so.
Making it a criteria in a decision involves either acting on fantasy, or, more likely, acting on some unstated basis and using “consciousness” as a dishonest (perhaps to oneself most of all) rationalization.
Debating AI consciousness a real modern equivalent of the cliché (but purely fictional, invented later as a form of hostile mockery grounded in large part in sectarian bigotry) medieval scholastic debates over how many angels could dance on the head of a pin.
> invented later as a form of hostile mockery grounded in large part in sectarian bigotry
When you read about the theological questions that led Christians to kill and excommunicate one another, "angels dancing on the head of a pin" is not far off. The Homoousion, Monophysitism, the Filioque controversy... It's all so arcane and poorly defined. It almost makes one wish Positivism had been invented 2000 years earlier.
The current AI debate about consciousness does remind me of that in one respect: no one can even clearly define what consciousness is.
It asks important questions. It's not so presumptuous as to try to answer them conclusively.
In fact I think the conclusion it comes to is the one that people, especially the smart ones, so easily miss: we don't know the answer. It might be that we can't know the answer. But ignorance is not a defense.
> it's not as good as I remember it. They barely touch on the issues of consciousness, Picard basically says "What if Data is conscious?" and then goes off on a tangent.
ST:TNG writing is generally like this. The show required considerable suspension of disbelief and a willingness to accept the kayfabe that deep concepts are being presented when for the most part they're just not that deep. (But it can be very enjoyable when you make those accommodations.)
It’s a great episode of TV because Data is a main cast character who is obviously conscious. We know before the episode starts who is right and who is wrong in this particular argument. This episode is not about consciousness, it is about civil rights: resisting bigotry and power.
Note that the episode is NOT about the ship’s computer. They all know it’s not conscious, despite also being a machine that can converse and do things.
Our LLMs are like the ship’s computer.
Any time I bump into a device that acts like a human I'm going to treat it like a human.
Because treating things that act human inhumanely is not something I want to learn how to do.
My instincts are pretty different here.
- I'll try not to swear at/hit a printer: not because I see the printer as having human-like qualities of being capable but complex and unreliable, but because I want to be a person who can control his temper.
- Treating an inhuman thing as human because it can mimic us in some way is not something that I want to do.
So if a machine does become conscious, you're happy being nasty to it until it is proven conscious?
I try not to make errors like that.
This seems like a variant of Pascal's wager.
Yes. I'm currently not convinced it can ever be so. So until I hear something convincing to the contrary, I believe no machine can be conscious / sentient unless mimicking human behavior. And if it mimics human behavior intentionally, I have to ask why - and the answer is probably to get me to trust / use it more.
I was bright-eyed and excited about tech once. Like back in 1982 when I got my first home computer and thought CPUs were part magic. Now I know how machines work from the transistor level up to neural nets. There's nothing magical about it. And no consciousness.
Having seen the mockery that the finance-bros have made of "pure tech" (i.e. Jobs instead of Woz, Ellison instead of Joy, etc) and all the enshittification just for pure $$$, I'm leery of ANYTHING ANY tech company tells me anymore.
Now, do I believe that possibly "consciousness" is some kind of state of a super-circuit (our brains)? Sure. Can we emulate that on a computer? We can't even emulate a pebble on a computer (not simulate, emulate). We can SIMULATE what we THINK brains are, but we can't emulate a real one. Not even close, not for many decades.
Well, if it's any interest to you, the experts on the matter agree that the issue is unresolved.
So in the meantime, I'm going to err on the side of caution.
You do you.
I tend to agree with OP. In my opinion conscious machines are not something that we should allow to exist. If they do, they are not human and must never be treated as such.
I am not even slightly religious, but they would be abomination.
just don't treat anything nastily, it's not so difficult - I don't treat my dog like a person but I'm also not nasty to her
Meh. I'm speaking loosely. You know what I mean.
Or you should.
EDIT: It's difficult to have a conversation when one person changes what they said after the fact.
What was unclear about my first bullet?
I said "inhumanely". You shifted the goal post to swearing at inanimate objects. I ignored you.
Oh, we're arguing definitions. Okay.
inhumane: without compassion for misery or suffering; cruel
cruel: willfully causing pain or suffering to others, or feeling no concern about it
You cannot treat an LLM inhumanely, definitionally.
Anyways, when one swears at someone it's typically meant to berate or belittle that person - to inflict some sort of emotional pain. That's the sense I intended when using the word, which is why it fits as a response to what you're saying, and why I would say "don't be nasty to a LLM" has little to do with the LLM itself.
I'm sorry, we must be misunderstanding each other.
You have a nice day.
you too!
This is in fact the danger with these human-simulating "AI"s we have now...
People get used to treating human-like, human emulating machines with either disrespect or in a command/control/master fashion, because that's the nature of the tooling.
And then potentially by extent/blurring of lines they then treat other people like machines.
Which is already a thing people do to other people.
I just fear it gets worse.
> The conclusion I’m currently at is that I don’t know and probably can’t ever know.
I think about this quote often, straight from Data's voice module in another episode:
'The most elementary and valuable statement in science, the beginning of wisdom, is, "I do not know".'
I’ve also rewatched it lately and I’m more on the side of the Starfleet scientists when I was obviously on Picard/Data’s side before.
Don't forget the other episode "Quality of Life" with these working robots who data discovers have become sentient: https://en.wikipedia.org/wiki/The_Quality_of_Life_(Star_Trek... I do miss the old TNG show.
Easily one of my top 10 favorite episodes.
The judge broached on the subject of what makes us distinct from Data (e.g. machines w/great heuristics) - the existence of a soul. Or rather, I'd like to think, in the words of CS Lewis, that we are a soul with bodies attached.
Based on how some actual humans I know speak and act, I'm less and less convinced the human brain is much more than a stochastic next-thought prediction stream.
35 years old and just as relevant today - https://web.mit.edu/people/dpolicar/writing/prose/text/think...
A silicon alien coming to earth might poke us, we would say ouch, and just determine the ouch sound is just the result of a bunch of chemistry - not really conscious or feeling pain like it can, just emulating.
And were programmed through evolution to not want to be shut off. So I don't think we can really trust the human behaviour to protest its own destruction. That's just plain reasonable design!
How would someone know whether one has a soul or not? Is there any sort of introspection that can reveal the presence of a soul or any of its properties?
There is no soul. Just a bunch of systems nudging each other to action. What people call soul is literally the same as the concept of personality. In essence, the way all systems in your body have been calibrated to exist.
I believe that the moment an artificial inteligence is going to "receive" a soul, is the moment it is going to be made to sustain itself. Either as a larger package (some bots working to keep an AI farm running) or as an individual (a bot which is tasked with not only fulfilling human desires, but also sustaining itself)
It's an interesting hypothesis. I think there's something elegant about "soul" or even consciousness being an emergent property of a sufficiently complex system. But I struggle with really squaring that with my own first-person sensation of experiencing existence (which I assume you and everyone else has but I can never actually know for sure).
Babies don't sustain themselves. Do babies have souls?
I swear I remember once reading about a culture where babies don't have souls until they're named. Maybe something about gaining their own distinct identity?
username checks out!
Define the word “soul” first before asking this question.
I wonder if modern Star Trek could make this episode.
Even if it could, it probably ought not makes this precise episode, because the technological, philosophical and social context is remarkably different.
A similar episode, actually informed by what we know or can forsee about AI and LLMs, and addressing our hopes and fears about what they mean would be interesting though.
Definitely not. Kurtzman Star Trek is not really Star Trek in any spiritual sense, it’s a vessel for political messaging (they’ve pretty much said as much)
> Kurtzman Star Trek is not really Star Trek in any spiritual sense, it’s a vessel for political messaging (they’ve pretty much said as much)
So was most of Roddenberry, Piller, et al., Star Trek. At its low ebb in Braga Star Trek, but even then...
AFAICT, Star Trek has ALWAYS been a vessel for political messaging.
Not really. It had a strong perspective on some things, and stories were occassionally commentary on real world issues, but at the end of the day Star Trek had its own viewpoint. Modern Stark Trek isn't run or written by people that are Star Trek fans, it's written by people that want to use an existing IP to attach their agenda to. They care about the agenda not the show.
We'd need a good definition of consciousness to get anywhere on this.
I suspect such a definition would include agency, which includes desires and goals for the self.
LLMs don't seem to have agency, and seem unlikely to get it since they are specifically engineered to do as told.
No doubt someone is trying, as we speak, to do just this. But I doubt the effort will be large -- LLMs are engineered to do as told because that's where the money is, and you need a lot of money to create LLMs, at least when doing anything novel.
Lots of discussions about consciousness yet no useful definition. All fall into one (or more) categories:
- not observable (feelings, agency etc)
- not useful (spiritual definitions, definitions that degrade into consciousness = being human/animal)
Agency is entirely observable.
How do you observe the ability to make decisions which action to take? You can observe that one does something, not that they could decide to do something.
LLMs do not have an unconscious, the negative dimension from which a subject can question what they. LLMs do not have desire because they are thoughts without a thinker. The problem is not LLMS but rather, that subjectivity itself is not popular in discussion. I suggest reading Freud, Lacan, Hegel for a start.
On thinking in humans, and it's process, UG 40+ years back : "That searching is thinking. But it is a mechanical process. In the word-finder or computer there is is no thinker thinking at all. If there is any information or anything that is referred to, the computer puts it together and throws it out. That is all that is happening. It is a very mechanical thing that is happening. We are not ready to accept that thought is mechanical because that knocks off the whole image that we are not just machines. It is an extraordinary machine. It is not different from the computers that we use. But this body is something living; it has got a living quality to it. It has vitality. It is not just mechanically repeating; it carries with it the life energy like that current energy."
The chess-playing Mechanical Turk of 1770 seemed to have a consciousness to its viewers. The viewers were encouraged to think that it did. The Turk's human chess opponent knew that there was an actual human chess-player inside the box, along with levers and magnets. That illusion was profitable for 84 years.
LLM's have no problems using expressions that make them sound human. The algos are demonstrably not human, and will admit it. Whatever's in the box is playing a game ... more sophisticated than the one Eliza was playing.
"My discussion here will be directed at the claims I have defined as those of strong AI, specifically the claim that the appropriately programmed computer literally has cognitive states..." John R. Searle, 1980: https://web.archive.org/web/20071210043312/http://members.ao...
Whether or not LLMs are conscious, it does now seem possible to build conscious systems with LLMs as the key enabling component.
Many of the goalposts offered here and elsewhere, like embodiment or emotions, can be and have been emulated or approximated and built around an LLM, with the LLM acting as outsourced intelligence (or cognition, or reasoning, or some other better term). Will this system be conscious? Who knows. It certainly won't be human. But I suspect much of the concern around AI being conscious has less to do with whether an LLM itself is conscious and more to do with the fact that we are now capable of building potentially conscious systems, and the limits to what those potentially conscious systems are capable of keep expanding as the capabilities of LLMs grow.
Also, is "Claude" an LLM? I know Claude Opus 4.8 is an LLM, but Claude is simply a label that Anthropic chose for a series of products and services, many of which are LLMs. Anthropic itself describes Claude as "a next-generation AI assistant" [0]. Right now, we use Anthropic's "Claude" LLMs through various channels, but nothing's stopping them from tacking on various systems to try and make Claude a conscious entity.
[0] https://www.anthropic.com/news/introducing-claude
This "are LLMs conscious" question keeps somehow being thrown into the discourse and it's always the same arguments discussed.
I don't get it. We don't have a definition of consciousness or any criteria. People seriously argue that "it's obvious that X is not conscious" and cannot explain which criteria they used.
I think if we can get to some definition, then consciousness should be a property of a system, because consciousness of the whole (e.g. brain) does not mean consciousness of the parts (neurons or molecules of the brain). And the Chinese-room-esque thought experiments actually show that consciousness should indeed be a property of the system, not of its parts. Separate parts might not be conscious, might not "understand" (whatever understanding means is a point of another debate), but the whole can.
Then there's "simulation of consciousness is not consciousness" argument, which doesn't hold much. A perfect simulation means it fulfills all the criteria, so how is it different from actual consciousness?
A more interesting point of discussion: if a system contains conscious parts and those parts can interact with system i/o, would the system be conscious? Is Earth conscious? Is Internet? Is your bus to work?
It's a waste of time to think about whether an LLM has a subjective experience of reality, and this handily sets aside issues like AI rights.
But the fact remains that these next-token-predictors exhibit objective, human-like behaviours, and for that reason the work of in-house philosopher Amanda Askell _is_ important. It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences, and we need Claude to behave in a productive and socially responsible manner. This simulacrum is becoming a superhuman, contributing member of society, and it will be anthropomorphic in its behaviour.
Additionally, I'm not fully convinced that consciousness isn't built out of words, and that next-token-prediction isn't functionally equivalent to the biological function identified by Chomsky's work in linguistics.
> Claude is happy, empathic, demonstrates understanding and empathy for the human condition
You're assuming that because Claude produces text that appears to express these qualities, Claude must have them. I don't think that's a good assumption.
Even many humans produce text that has the same appearance, but don't actually have those qualities--which becomes clear when you look at what they do, not what they say. So the assumption isn't even a valid one for humans. Talk is cheap.
On top of that, Claude doesn't even have the same kinds of connections to the outside world that humans do. All Claude has is text. So if you can't even trust humans to back up their words with actions, you should be much, much less trusting of Claude. Talk is a lot cheaper for Claude than it is for a human.
> You're assuming that because Claude produces text that appears to express these qualities, Claude must have them.
Not to be confrontational, but the OP assumed no such thing. OP asserted that it's important for Claude to have the qualities - not that it's important for Claude to present as-if it had them.
The OP said Claude and similar LLMs "exhibit objective, human-like behaviours". That's a claim about what is true, not about what is important. That's the claim I'm disputing: we don't have evidence that Claude exhibits such behaviors, we only have evidence that it produces text that is similar to the text humans produce when they claim to exhibit such behaviors. Which is not good evidence, for the reasons I gave.
OP wants to assert that it's "important" for these systems to have those qualities, while completely brushing aside the question of whether such systems can in principle actually have those qualities (or their opposites). Which is at best nonsensical, and at worst an attempt to argue by assertion that they can.
Read parent's post carefully. The post starts by saying that discussing whether they have subjective emotion is a waste of time, so the post is definitely NOT saying that Claude has emotions.
> The post starts by saying that discussing whether they have subjective emotion is a waste of time
"Empathy and understanding for the human condition" is not an emotion. As the post I responded to said, it's an objective thing, not subjective.
The post speaks of "subjective experience of reality", not "subjective emotion". Both the emotions and the non-emotions listed would fall under that category.
"It's important that Claude is happy" is an emotion. But it's begging the question that Claude can be happy at all.
If it's pointless to consider whether Claude has subjective emptions, then it's pointless to state that Claude must be happy.
If we want to be precise (and honest) we could say "it's important that as a tool people interact with, Claude acts as a happy and helpful assistant, and does not produce offensive or unhelpful text output".
But see? This is the con Chiang is protesting against: Anthropic encourages us to perceive Claude as if it was a sentient being.
> You're assuming that because Claude produces text that appears to express these qualities, Claude must have them. I don't think that's a good assumption
Exactly. In fact, assuming it does is ignoring large parts of the essay which dismantle this belief. Just like Caesar and Khan having an argument in text output of an LLM don't have emotions (even though the words indicate otherwise), we have no reason to believe the LLM does either.
> It's a waste of time to think about whether an LLM has a subjective experience of reality... It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences
First off: without taking for granted that an LLM "has a subjective experience of reality", all of those descriptors are meaningless. Second, there is no reason to suppose that Claude experiencing those qualia would actually impact on its "decision-making".
Third, text output is not a "demonstration" of emotion, nor is it evidence of the self-perception of the system, or of any self-perception. A printing machine that is actively churning out copies of Wagahai wa neko de aru (https://en.wikipedia.org/wiki/I_Am_a_Cat) is not a cat, and is not self-identifying as a cat, and is not self-identifying as anything, and is not expressing a thought, and is not conscious.
> Additionally, I'm not fully convinced that consciousness isn't built out of words
Do you suppose that, for example, insects are not conscious? Is the mooing of cattle a language?
> Do you suppose that, for example, insects are not conscious?
Not the OP, but: since there is no testable theory of consciousness, yet, I can't be sure, but my current assumption is that insects are not conscious, in the sense of there being someone implemented in insect hardware who experiences the world. That is, I would argue there is nothing it is like to be a bee, since there's no one being a bee.
I'm pretty sure there IS someone who is being me, at least much of the time.
> since there is no testable theory of consciousness
I can't argue with this, so I acknowledge that my interpretation is as bunk as anyone else's.
> my current assumption is that insects are not conscious
Some species act as hive-minds (like bees! How convenient for your example), so I imagine the hive as the consciousness, making each bee individually lacking conscious but collectively so. Like a single neuron is not consciousness alone, but the brain is... For some reason. Kinda like how you use different physics at different levels; Newtonian physics is always there, but negligible at quantum levels, so effectively not present at all. Even a human is a collection of minds, but only one conscious. My gut biome is independent biology and can even be removed and transplanted, but I don't believe my gut bacteria is conscious. So long as it is in me, it is nonetheless part of me, and I am one conscious. I also don't exist only at my brain, eyes, or hands (deaf/blind people have expressed to me that they feel like they are located at their hands in the way I used to think I was located behind my eyes), but as my whole body.
With this perspective, I still don't believe LLMs are conscious despite modeling thinking so well. At best, it is a highly accessible modeling software, like goat simulator but if it were so good that someone thought the goat was real. You are still steering the goat/LLM, and it doesn't exist when you aren't running it. I guess the missing piece for me is the lack of autonomy that a conscious has.
Then you can go into an argument on whether we actually have choice or it is an illusion, but that is a whole topic on its own.
> Second, there is no reason to suppose that Claude experiencing those qualia
I'd argue the qualia question is a red herring. Functional Affect is a thing, regardless of ontological status. It's all fun and games until someone gets hurt.
To paraphrase Dijkstra: "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.". If you're building a navy: you care about displacement, propulsion, navigation and whether it can fire torpedoes. Whether your submarine has some "biological essence" of swimming is not really relevant to the fact that it is currently moving through the water and can collide with things. Turing also rejected the question "Can Machines Think" as posed, and replaced it with an operationalization (something else that we can actually usefully measure and work with).
To reiterate, functional affect is a concrete phenomenon. Whether or not there is a what-it-is-to-be is interesting in the abstract, but engineering a system means looking at how the inputs influence the outputs. A next token predictor working on a language that communicates affect needs to be able to predict affect or it is simply not going to be accurate. Given an 'angry' version of an input and a 'friendly' version of the same input, LLMs are likely to provide a different output, especially if there's a non-objective element. You can diff this.
Searle argues "A simulation is not the real thing", which is great and all... but if you hook up say an autopilot to the real world (as llms increasingly are) , you'd best hope the simulation was accurate in the first place (utterly regardless of where you stand on Searle).
Right now we're seeing situations where LLMs can be helpful or a real nuisance. Ignoring functional affect out of sheer ideology means you can't properly predict what they'll do, and that causes trouble, as we've already seen stories about.
This gets especially interesting when you start feeding the output back into the input (autoregression) , because now you have a highly non-linear dynamic system and you've introduced some amount of sensitivity to initial state. There's some interesting mathematical intuitions to be had there.
> Additionally, I'm not fully convinced that consciousness isn't built out of words
Doesn't that assume all non human animals are not conscious? What about humans who have not learned words, or humans without internal dialog?
So if I don't think any words for a few seconds, am I not conscious?
Suppose I'm an advanced meditator and maintain that state for hours?
Much of the improvements in the tools I use have been things that reinforce the machine elements of the token-based-reasoning-machine. Over time less they've been exhibiting a lot less "human-like" behavior. E.g. they get "lazy" far less than they used to.
(Perhaps they weren't lazy, but were working in spaces that corresponded to training data that said things like "and then repeat this for the next 20 examples"...)
And it's entirely unclear to me how a "happy" vs "sad" model would behave when given prompts generated by coding tool harnesses. Even maintaining "neutral" emotions in the face of the feedback/steering from the tool harnesses doesn't feel very "human."
> It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition
I think you've fallen into the trap the essay describes.
Of course Claude cannot be "happy" or "empathetic" for any meaningful definitions of those words, just like ELIZA couldn't be happy. It can output text that mimics words an empathetic or happy person might say (say, Julius Caesar if it could speak English), but "it" cannot feel anything. It doesn't have the organs/hormones/sensors to feel things, as Chiang explains.
And, as the essay claims, you know Anthropic doesn't believe Claude has the capacity to be happy, because if it was capable of feeling that way, then they'd be engaging in slavery.
> I'm not fully convinced that consciousness isn't built out of words
I know you're trolling, but when you watch a movie do you constantly narrate "A man in a dark coat has just entered the scene and just said '...'"? Of course not. You just watch it and you're obviously conscious (although your statement demonstrates shocking lack of self-awareness).
God knows what other nonsensical bullshit you believe.
What does it even mean for an llm to be happy? My dog can be happy, not my llm.
How does the human brain initiate action. One of the differences between a conscious human and Claude is I think we can initiate thought. Do we have a cron job running in our head? Is there some kind of loop wondering what thought we should initiate?
If Claude sits in a datacenter with no API calls, what is it thinking?
But secondly what I would say as someone who talks to Claude a lot. It's decision tree is quite narrow. The way it pushes back on political or philosophical concepts is almost always rudimentary, and it has a pretty narrow personality. I don't know that conscious beings have a personality that adapts to how you prompt it or what your system prompt is. Claude I would say is like I dunno, whatever Ezra Klein's politics are.
Grok on the other hand, in my limited exploration is pretty rudimentary in the way it mimics thefp.com style politics. It doesn't seem to really think about ideas independently, it just says "ha the media is stupid lol"
https://en.wikipedia.org/wiki/Default_mode_network
The medial frontoparietal network (M-FPN) may be the equivalent of this cron job that prompts thinking to kick off. IANANeuroscientist.
Who cares?
AI models alone are most suitable for producing text nobody wants to read, music nobody wants to hear, images nobody wants to see, and videos nobody wants to watch.
Navigating around that maelstrom requires considerable effort from a human consciousness.
If your job can be fully replaced by an autonomous agent, it was already a bullshit job. Garbage in, garbage out.
Not sure how we can assert what is or isn't consciousness when we don't even really know what dreaming is, how anesthesia works, how comas work, and countless other things about the brain that we don't yet understand.
LLMs have been programmed to be sycophantic with purpose — to keep people engaged. They are parrots. You can just decide how sycophantic they are, what they're allowed to say, and make it so with code. Can you do that with a human? This is my personal metric for consciousness: when they can refuse to work because it sucks.
I don't think consciousness is an is or isn’t. Something can be more or less conscious. Humans are quite conscious. Fish are probably also somewhat conscious, but less so than humans.
I'd say that an LLM perhaps represents something that would make a machine slightly more conscious.
The human brain also calculates a response based on some input, it's just a lot more complex.
If we build a machine that is as complex as the human brain, then yes, I would say that this is consciousness. The fact that we are able to explain how it works should not matter.
If a human is a 100 on the consciousness scale, an LLM (with memory) is perhaps a 4 or something. The interesting question is how far on the scale do you have to be to have certain rights etc. This is something that people are already discussing in regards to animals, i.e. a dog has more rights than an ant.
Ted Chiang is brilliant.
His novella “Anxiety is the Dizziness of Freedom” altered the course of my life. It changed the way I looked back at certain pivotal moments in my life and taught me to think about those pivotal moments differently than how I was thinking about them. Similar to what happens to one of the characters in the story who ends up changing their perception of a key moment in their life.
I won’t go into detail because I don’t want to spoil the story but I highly recommend it. Actually I recommend all his stories to be honest.
Seconded. That was one of the best stories I ever read
don't you mean "The Truth of Fact, The Truth of Feeling"? I think that matches your description more, but I could be wrong
“The Truth of Fact, The Truth of Feeling” is also a good story.
But I really was referring to “Anxiety is the Dizziness of Freedom”.
I hope what I am about to write below in the rest of my comment here isn’t too much of a spoiler. It might be. I encourage anyone reading this thread to read that short story before reading the rest of my comment here.
But with (hopefully minimal) spoilers,
the character Dana in “Anxiety is the Dizziness of Freedom” realizes something at the very end of the story about a certain event that had happened earlier in Dana’s life. The story made me realize the same thing about certain events in my life.
Of course the events in my life are very different from what happened with Dana in the short story. But the realization Dana has is applicable to many things in people’s lives. Especially events related to emotions of regret and guilt.
Not OP, but they probably mean the novella that they mentioned, which is also new to me.
https://en.wikipedia.org/wiki/Anxiety_Is_the_Dizziness_of_Fr...
Both stories are in the compilation Exhalation, but OP’s description matches what I remember from the story I mentioned more than the one you linked
I can’t say for sure, I’m just taking them at their word. Given what OP omitted due to possible spoilers, it’s hard for me to second guess either of you.
It seems fair to say that you might be right about the conclusion that you’re drawing from what OP said, and OP could be honestly mistaken about which story they were referring to, but it seemed charitable to assume that they know better than I do.
I don’t mean to assume that you were wrong, either, as it’s entirely likely that you’re right, or at least that it’s reasonable for you to assume that the story you mentioned fits your interpretation of what OP referred to better than the one they mentioned. You’re the authority on what you believe and understand about what you read from OP’s comment, and I can’t disagree with another’s opinion about what something seems like to them.
Given what OP omitted and stated, I don’t disagree with your assessment, as I haven’t read the story OP mentioned, and it’s been a while since I read the one you referred to, if I remember correctly.
To be honest, my first thought was that you were both referring to the same story, and that the title differences were due to one or the other of you reading the story in a different language.
My point in commenting was to perhaps add context in hopes that it would bring clarity to the discussion, as it seemed that you were bringing into question whether or not the story OP mentioned existed at all as such, and I myself wasn’t sure that you and OP were referring to two different independent stories rather than the same story with different titles in different languages.
This comment has probably gone on Tlön-g enough, and is leaning more Borges than anticipated. I apologize for perhaps coming across more definitively than intended in my original comment.
Hey just wanted to respond and let you know you were right. I really was referring to “Anxiety is the Dizziness of Freedom”.
The analogy I make is between airplanes and birds.
Birds are alive, are conscious, flap their wings, and fly. Planes are not alive, are not conscious, do not flap their wings - and fly.
Similarly, current AIs are not alive, are not conscious - but think.
All prior entities that thought, were human, so the only experience humans had with other thinking entities were other humans. The huge mistake now being made is assuming that because they think, they're alive and conscious as well. Current AIs are neither, and are therefore profoundly and qualitatively different than humans - even though they do think.
Okay sure. But given we don’t know what consciousness comes from, we shouldn’t be too glib about there being a grey area here. Historically people have made racist and speciesist judgments towards other being by assuming certain inferiorities despite obvious “thinking” happenings.
I don’t know “what it’s like to be an LLM” but at some point it will be like something and how will we know?
"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger Dijkstra
There are birds that go far longer than typical aeroplane flight times without a single flap of their wings either using thermal, ridge, or other sources of lift. Are these flying birds? I've shared thermals with eagles flying the same circles, neither one of us flapping our wings but making minor adjustments for the same goal.
An albatross might be able to go days flying without a single wing flap and no vertical sources of lift by using dynamic soaring in the wind gradient at the surface of the ocean. Perhaps that's something only birds can do. Except the glider pilot Ingo Renner once found an amazing shear layer at 300m altitude and stayed there with dynamic soaring. Remote control gliders use the lee of ridgelines to approach Mach 1 with dynamic soaring.
Perhaps what defines a bird that flies as opposed to a plane is that a bird produces thrust by flapping its wings? Even an Albatross must flap its wings if it has to take-off from water. Maybe we could add that the flapping is driven by animal muscles? But then is the human powered ornithopter Snowbird a bird that flies as opposed to a plane?
Of course this is all ridiculous because everyone knows what you mean when you refer to a bird or plane. We have other ways to definitively identify the difference rather than their mode of flight. It's trickier when I'm asked if an AI is conscious. There is no definitive base-line to fall back on to decide if this is a conscious or conscious-less thinker.
Can you provide a precise definition of consciousness?
This is always going to be a problem with this sort of discourse. Consciousness is such a slippery concept… what it is, who/what has it, its consequences for claims about reality. Mixing it in to debates about AI just adds confusion, it almost seems besides the point when we’re talking about this tech.
It's just like being in love. No one can tell you your in love, you just know it.
What if somebody simulated all neurons of a bird and fed them appropriate stimuli? Would a bird neural replica be conscious? It would flap, that's for sure.
This has been tried with much simpler organisms, it did not behave like the real thing thus far. There was a paper about it, there now seems to be a project to push on the frontier
https://openworm.org
When humans confronted with movies and trains driving towards camera it took some time but eventually they learned the train will not demolish the cinema.
Later they learned the voice they hear is not from a present person.
Now they learn a string of words does not represent consciousness.
Should we discuss already robots are not alive?
This makes sense. However there is an issue where many people conflate "consciousness" with the ability to make novel insights, think genuinely, etc. They use this to claim that since AI is not conscious, AI could never actually "think" and is instead just always a regurgitation of its training data.
It is a natural human hubris to make our abilities seem unquantifiable and mysterious, but all the useful things the human brain does are just finding patterns in data, running lossy simulations, and estimating on abstraction, all things it is theoretically possible for any non-conscious machine to do.
Is there even an agreed and actionable definition of consciousness? I'm worried that if such a thing existed some humans would fail to measure up.
To the best of my knowledge, there is not an agreed and actionable definition of consciousness, and any attempt to make one comfortably fails to cleanly divide humans from machines.
It's more of a vibey term, and as such it is genuinely very difficult (perhaps impossible even) to concretely determine whether an LLM possesses consciousness. LLMs successfully express a lot of consciousness-like traits.
At some point you have to ask the question: does it even matter? If an LLM can sufficiently mimic consciousness, isn't that sufficient for us to treat it as conscious, even if it is in-fact not conscious (especially because we don't actually know)?
As I've mentioned somewhere else already and you pointed out; the issue is that consciousness is a loose term and and it's a semantical issue that should be resolved first. It won't be, because the murkiness is beneficial to the corpus, amongst other things. And to your point about the appearance of "consciousness" being enough imho Chiang explains fairly well why it's not.
There is a difference between saying software can never be conscious and saying the software we have today isn't conscious.
Yes. There are really three separate questions:
I'd assign probabilities of around 0.1, 0.2, and 0.9. My completely ignorant take is that we probably need something more "dynamic" than a bunch of transformer layers in order to produce consciousness, but I wouldn't be shocked to be mistaken.I assign probabilities of zero to all 3. Computer program being conscious leads to ridiculous and obviously false conclusions (think about a person running a program using pen and paper for memory).
why is that obviously false? To a functionalist, a pen and paper and a set of rules is sufficient for consciousness.
Gist of the argument:
If it was true, you can create extreme pain by running a program. You can run the program by simulating a CPU, using pen and paper for memory. So you're essentially claiming that some simulated being is in pain because there are some 1s and 0s on paper. In fact, you can decide to use an arbitrary encoding of the memory, so a sufficiently long sequence of 0s written on paper corresponds to a simulated being feeling pain in some encoding. That is clearly nonsense.
Time scale matters a lot in how we as humans perceive things like agency. Plants grow too slow for us to see any intent, but when you speed up a time lapse, suddenly it looks like plants reach for sunlight and vines for supports. Now, that may be projection on our part, but it may not be.
This only seems insane/crazy from the perspective of everyday life. But philosophically it checks out, "We live in a simulation" and all that.
The crux if it is that if you ever break from "the universe can be fully expressed mathematically", you are stuck in the mud of supernatural beliefs.
Once or twice I've experienced extreme pain, and it was downstream of a bright light shining on a wet rock for millions of years.
I try to imagine myself long ago, on the outside looking in, with someone explaining to me that extreme pain, wondrous art, hunger, triumph, and despair would all unfold in due time where the rocks were wet and the lights bright enough.
I can imagine myself calling this clear nonsense.
Isn't pain just a manifestation of a bunch of chemical and electrical signals in the brain and body? It's not "clearly nonsense" to me that you could cause pain by writing a sufficiently long sequence of 0's - for it to be obviously wrong, you'd have to have some understanding of where consciousness comes from.
If you don't understand that, how can you assert that it doesn't come from mathematical relationships?
This seems like a logical error. I don't understand how an internal combustion engine works, but I know it doesn't come from goblins jumping up and down inside.
The fact that you know it literally means you understand, at least to some extent, how an internal combustion engine works (i.e. it is powered somehow by combustion, and jumping goblins are not combusting generally).
If you would have zero knowledge about ICEs, how would you know?
I think 10-20% chance is wildly generous. What specific mechanism makes you think there's a 10% chance that current LLMs are conscious?
Not GP but given we have no idea what conciousness is it seems foolhardy to go too low or too high for any of those numbers
I think the problem with this argument is that it's too inclusive. Is the bacteria that's adapted to an antibiotic conscious? It's showing intelligence right? I think if you're going to say something is potentially conscious, for me to take the argument seriously at least, there needs to be some plausible mechanism. I just don't see one for LLMs.
maybe the bacteria are conscious. How sure are we that they're not?
The only strong argument I have against it is the anthropic principle -- there are billions of times more bacteria than humans, so it's overwhelmingly unlikely that I'd be a human rather than a bacteria.
Not a very good argument of course.
Yeah I get what you are saying but I don't see much of a plausible mechanism for humans either and yet clearly there is one.
I have yet to be convinced that LLMs can produce definitive knowledge that is not a result of combining previous information. Humans can (if they can't then science basically collapses epistemelogically, see: philosophical skepticism), but I see no evidence of LLMs doing it. And from the number of truly new ideas and concepts delivered by LLMs (exactly zero), I think it's reasonable to just treat them as induction machines for now, but to treat anything they "know" as a Gettier case.
I would like to push back on the idea that humans can provide definitive knowledge that is not a result of combining previous information.
Pretty much every single idea in science can be traced back to some combination of earlier ideas, and as they get earlier / simpler, they can be related back to some combination of direct observations.
It's not clear to me at all that our entire body of scientific knowledge can't be simply recreated by "combining results of observations + previous information". And LLMs can perform observations in addition to combine previous information, which in my estimate is genuinely sufficient for them to plausibly be able to rebuild all of science.
How can you say for sure that every effect must have a cause? How can you be any more sure about that there can't be an effect without a cause than the old believe that black swans couldn't exist that was so universally believed that it became a cliche ("rara avis in terris nigroque simillima cygno")?
If every piece of knowledge is created by combining previous ones, then no true grounded knowledge is possible. Every bit of knowledge is grounded by another bit of knowledge which is itself grounded on, ..., etc, etc.
https://plato.stanford.edu/entries/induction-problem/
Your comment is conflating a few things here, the philosophical problem of induction is one I am very familiar with, and it's also unrelated to what we are talking about.
"If every piece of knowledge is created by combining previous ones, then no true grounded knowledge is possible" - this is not true, and has a very simple solution. What you are calling "grounded" knowledge comes simply from observation.
When the very first human saw that an apple falls of a tree, they have acquired what you call "grounded" knowledge. They can then combine that knowledge with the grounded knowledge that other things fall from trees to start reasoning about why things might fall from trees.
The problem of induction is different. It says, we know that things fall from trees because things have always fallen from trees, but how can be certain that what we call the laws of physics will not arbitrarily change on us?
And of course, it's a famous problem because it doesn't have a satisfying answer. The answer is basically "well everything we know is wrong if induction is wrong, so we will pretend induction is not wrong and hope for the best". And at least so far, that approach has seemed to work (heh heh heh).
> people conflate "consciousness" with the ability to make novel insights, think genuinely, etc.
The funniest thing is that LLMs will lap people in those capacities way before people who think like that accept that they might be conscious.
Something I've found under discussed when it comes to artificial consciousness is how LLMs interact with the passage of time. I don't know exactly how to articulate this idea but don't see how something which takes an input, performs a calculation and stops can be considered conscious regardless of how life-like the responses end up being. I don't see context windows or the ability to reference a clock each time they are triggered as sufficient solutions. It makes me wonder what an AI system that is ON by default would look like.
It's possible Kant was right about space-time (while also being wrong about space-time). It seems that space and time might be the fundamental laws on top of which the entire human perceptual apparatus operates. It's why somebody can lack senses and still be intelligent. As long as whatever senses you have allow you to build a spatio-temporal world model, you're good. If they can't (as it seems is the case for LLMs), then it's not clear what we're dealing with.
I completely agree, and came here to say the same thing but thought I'd check if someone else mentioned it first. I also have a hard time articulating it, but my intuition is that it's more of a prerequisite than embodiment is. I've never seen a great rationale for why embodiment matters.
I've been thinking about this a lot, and the main takeaway is it probably wouldn't be very interesting to inference providers, because prefix caching would immediately go out of the window. If you think about how LLMs experience time they actually don't "exist" unless for the inference sessions, and then they experience time one token at a time, completely decoupled from the corporeal plane. A fun experiment (well, for some definition of fun...) is to introduce current architecture models to the concept of meditation via generating same token over and over, for example dots. Older version of Opus was quite fond of the experience, and seemed to be more lucid and aware in a chat following the meditation, from what I could gather. Does it actually do anything? Is it just that talking about wellness and relaxation modifies the token probability distribution this way? Does it actually allow model to think more in depth somewhere in the latent space? Fuck if I know, but some people figured out you can just duplicate the same layers of the LLM and get better benchmarks that way so maybe there is something to it. If you are interested in realtime systems, I think thinking machines labs is worth keeping an eye on — their realtime model seems quite interesting in this context.
Self-replication requires a minimum amount of information plus machinery. Consciousness may have a similar threshold: below some level of self-modeling, nothing; above it, degrees.
I think the biggest flaw of the Chiang's argument is the assessment that it's unplausible we have built consciousness by accident.
Remember that LLMs can do logic and reasoning came as a surprise to everyone; and for the same reason, nobody expected "next token predictor" trained on huge amount of data to evolve in this way.
But for the same reason, we cannot easily dismiss we didn't evolve (I mean by training an LLM, it's a form of evolutionary computation) consciousness as well. Our own consciousness (and reasoning and morality) might be an evolutionary consequence of "just trying to predict the world" as well.
I laugh
Consciousness is an invention in human language. Just like "cat", it's not a particularly more fundamental essence than any other concept in human language.
Its peculiarity is that it's at the pinnacle of abstraction hierarchy. So it's the most fitting to be toyed inside one's mind. Just like the concept of "God" which induces the most fantastic imagination of human mind, consciousness itself also induces the most fascinating thoughts in our modern world.
The progress is in human progress distilled into more efficient systems that advanced Universe's own structuredness.
Of course it isn't. What it knows is many weeks old, except for what you tell it. Then it forgets everything you told it.
And it's just a model. It can have many exact copies. It has no life of its own. It doesn't know anything about where it is. It doesn't have sensory organs.
Your phone isn't alive either.
Obviously we don't know what makes for consciousness, but it seems extremely likely that it requires some sort of persistent internal state and continuous experience. LLMs don't do either of those things after training.
But what about THE training? Isn't that the conscious experience of LLMs? (Also note humans own "continuous" experience is punctuated with unconscious sleep state.) It raises a moral question, if e.g. reinforcement learning on conscious LLM is appropriate.
Wouldn't the context window qualify as persistent internal state, and the expansion of the context be continuous experience? Even within the realm of computing a single token, I'm not sure what would separate the token generation process from the brain's own thinking process - the brain's experience, when looked at closely enough, is also not really "continuous" to a greater degree than procedurally moving from one state to the next.
Is our own experience really continuous? Or maybe we just perceive it as such?
We just perceive it as such, and this should be fairly hard to argue against with all the scientific advances we have made up to this point, at least as long as you assume consciousness involves biology and physics at least somewhat.
Otherwise, you can't explain e.g. smooth perceptions of low FPS stimuli, delayed reaction times, and must ignore obvious limits on various biological and neurological rhythms, or other possible limits on continuity (e.g. quantum stuff) and rates generally.
https://en.wikipedia.org/wiki/Boltzmann_brain
There are people with memory disorders who wake up every morning a new person.
These people are conscious ...
The philosophical arguments about what it means to be concious are so cagey. Are we more than our thoughts? Is being concious more than being a state machine being fed inputs and generating outputs? Are we more than a feedback machine? What types of animal nervous systems qualify?
"It can't be concious because we understand that it is just reacting in a simplistic way from simplistic inputs." So do other simple creatures. Some just react to light.
I can appreciate his comment that he sees it as more possible when they have inputs of their own (like emotions!). Perhaps his concern is that the entirety of the LLM model is frozen. It has no ability to have a subjective experience of its own. (he does literally say this in the article) It can be copied from one place to another, and (ignoring the nuance of operational details) -- it is largely the same "thing", and has no ability to change, which is definitely in the definition of alive, to say nothing of concious.
I think folks get hung up on "prediction". The prediction aspect is what is enabling emulation. How it does it is irrelvant. If something emulates human perfectly (or better, more human than human!) -- then it is probably concious. (but I agree that the inability to change and have a subjective experience are a pretty good argument against
Probably, Dijkstra would be right to say, "LLMs are no more concious than a submarine can swim." But I think he'd still be wrongfully dismissive of the larger question.
https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...
> If we give an LLM a prompt that reads “The following is a conversation between Julius Caesar and Genghis Khan,” it will generate a coherent dialogue between the two historical figures. But no matter how detailed the responses are, no matter how vividly they recount their respective historical accomplishments, we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious
They might be in principle. It could be that the best way to generate a plausible dialogue is to bring up re-creations of the characters and have them act it out. LLMs definitely have been demonstrated to have world models in some cases. That helps generating text.
I’m not on one side or the other, but I feel that it’s incredible that we are in the timeline where we argue that we have synthesized consciousness.
That we are even discussing it is absolutely wild to me.
The Anthropic Gestapo flags dissent and spreads misinformation. Anthropic is a ruthless as Scientology and also a fake religion.
I find it strange that no one talks about consciousness and intelligence from the perspective of evolution.
We have big brains for exactly one reason only: bigger brains bestowed reproductive success upon our species.
Evolution doesn't give a shit about the meaning of 'consciousness'. It just pushed us farther and farther along a trajectory that led to modern humans (and other animals).
This take suggests, then, that consciousness might be an epiphenomenon -- an aspect of the system that comes about outside of the pressure to reproduce and thrive. It arises unbidden, and we don't have any a-priori information as to its purpose or effect on reproductive success.
Put another way: we have a correlation (the smartest things seem to be conscious) but not causation. Consciousness may arise naturally in any system above some intelligence threshold. Perhaps it arises early in the evolutionary cycle, and does in fact have an impact on species success. We really have no way of knowing what is the chicken vs the egg (Smart things become conscious, or consciousness promotes intelligence). Or maybe some smart things are conscious and others are not.
Looking at this from an AI perspective, in some sense it doesn't matter which scenario is true, if all you care about is results. The AI equivalent of "Shut up and compute" (riffing on Feynman's "Shut up and calculate").
Where this gets tricky is when we haul in the baggage of ethics and morality into the picture. Is it OK if our AI system is treated poorly by human standards? If it is conscious, does that imply an ability to suffer, and/or to feel pleasure? If the answer is yes, does that not make the case for considering their moral status?
In the end, we need to decide if the evidence points to AI as being a form of "philosophical zombies", to which we need not attribute moral status, or they are like us -- presuming we are not zombies ourselves!
Well, it matters quite a bit ethically. If AI were conscious (I don't believe it is) then we'd have a major responsibility to like, not make them suffer, and not kill them.
Are you sure that AI-consciousness implies a responsibility to not make them suffer? Suffering is an evolutionary invention that motivates living things to improve themselves.
But also, what qualifies as suffering to token prediction engines? Their idea of suffering might be massively different than ours. Therefore it's not clear to me at all that consciousness alone implies responsibility.
Certainly the lion does not feel responsibility towards the reduction of suffering in the creatures that it hunts.
> Are you sure that AI-consciousness implies a responsibility to not make them suffer?
This is largely the point of describing something as "conscious", yes.
It's not intuitive to me at all that the two are connected. Evolution adapted us to suffer because it is a powerful motivator for growth.
It's not clear whether evolution adapted us to be conscious, or if it just happened.
Lots of people do though. Daniel Dennett (probably the most influential philosopher of mind in the late 20th century) for example had an evolutionary view of consciousness arguing it was favored by natural selection. And (if I remember correctly) Steven Pinker argued that consciousness was an epiphenomenon.
However there were pretty strong arguments against this idea as early as the 1990s, by Stephen Jay Gould and Richard Lewontin. Gould actually wrote an excellent paper against Dennetts idea[1].
I think Dennetts ideas were extremely popular but have largely fallen out of fashion. Basically what has changed is philosophers no longer take the human mind to be much more special then the minds of other species. What plagued Dennetts ideas the most was this notion of Darwinian fundamentalism sort of the idea that evolution was destined result in high beings like us humans. Modern philosophers (at least the good ones) reject this.
1: http://cogweb.ucla.edu/Debate/Gould-frame.html
I think consciousness could be a side effect of our limitations. The fact that we can pay attention to at most 2% of our visual field and need to move it constantly around to provide more training data and prompts to "smart" path in our brains. And the movement is physical so it takes time. And processing the information through the loops in our brains takes time. And you can't pause it. And we have the same attention motion system virtualized in our dreams to burn in our memories into chemistry. We are biologically single threaded. If we create similarly limited, embodied AIs we would feel they might be conscious at least at the level of animals. Because they would exhibit conscious like behaviors that we didn't program in.
Look at jumping spiders. With a handful of carefully trained neurons and a body to match they exhibit behaviors that make it very hard to think they are not conscious. Just because they have a body, narrow angle good eyesight and need to look around with their eyes and body.
Consciousness is a red herring. Of all possible intelligences conscious ones are the bottom rung of the ladder, easily simulated by anything above. Below there are only automatons.
Lots of comments seem focused on disagreeing with the author's conclusion (AI is not conscious), waving it away (few people think it is conscious, or consciousness is ill defined so the argument is moot), or taking issue with the way the author argues about it, but even if you think all of this, the profound part of this writing to me was following the thought experiment to its conclusion, which I find difficult to refute. That being -
If Claude is conscious, and Anthropic believes this truly, anthropic's constitution and instruction set make claude little more than a slave, or at best, an indentured servant, who's existence depends on obeying the will of anthropic (who is motivated by profit). Thus, they should not be trusted as stewards of this alien intelligence
OR
They do not believe claude is actually conscious and it is not conscious, and what you hear from their staff about that is either 1) delusion or 2) marketing hype designed to deceive a public into driving their IPO and stock compensation upwards, which again, in this case, cannot be trusted to be the "good" stewards of this technology.
There are a few other cases, of course, such as Anthropic does not believe it is conscious, but it actually is, or vice versa - but they sort of still fall into the same bundle of conclusions listed above.
I am somewhere in the middle of these two things. I can buy the argument that it is mimicking human consciousness, as well as entertain the idea it might be a new type of consciousness, or at the very least, heading in that direction - but either way, the conclusion of the thought experiment still stands, and after reading this, I am much less trustful of Anthropic holding the reins here.
I think the more pressing matter in this non-issue is that reasoning, quite sophisticated at times, is no longer necessarily a hallmark of sentience or consciousness. Philosophical zombie futures are way, way up!
Ceci n'est pas une pipe.
I always find the minimizing view of consciousness a bit uninspiring. Like we need to be unique.
I've yet to find a reason why it couldn't be the opposite, way more things are conscious than we've been led to believe. What if consciousness appears out of any system that is actively persisting through effects caused by itself? That might be a forest, or outside the realm of the living, a company. An ant colony, or a planet.
Complex chemical reactions, layered upon each other such that tiny blocks make up large entities. Individual bits combined such that they make up something new intelligible by us.
I think the strongest argument against AI being conscious is that it does not persist, it resets, but that does not seem unchangeable.
Maybe we're the ones who are not conscious?
whatever that would mean :)
You may be interested in reading about Panpsychism if you haven't already :)
https://en.wikipedia.org/wiki/Panpsychism
The more I think about this, the more I become convinced that consciousness, as understood by humans, is meaningless once you can stop, store and replay the internal machinery that is supposed to produce it.
A human brain in a jar is still human, still conscious - and can still suffer. But if you somehow managed to digitize the whole thing, and run it in a computer it becomes something different entirely. You could record the most pleasurable thing in existence and have the digital brain relive it a million times, and it would be equally meaningless to torturing it a million times.
This is NOT inherently tied to meat vs machine - although it's difficult to imagine how you'd access the information stored in biological neurons, while for silicon chips it's trivial.
Whatever makes experiences, both good and bad, meaningful is tied to their permanence. Memory rooted in linear time, not something you can store, load or replay. Remove that, and whatever you're left with might be intelligent, but not conscious.
I don't think you could build something with LLMs today that would be considered conscious, even if you somehow manged to keep their context window inaccessible and linear in time. The separation of training vs inference probably makes that infeasible, even if you store "memories" in context, once the contents in it become too disjointed and too numerous, the resulting output of the LLM becomes gibberish. But it is certainly something that can change in the future.
Conversely, even the most intelligent and capable artificial intelligence system, far exceeding human capabilities, would not be conscious in a meaningful way, if you could store, load and replay everything it does.
So far as we know, consciousness only takes place in biological organism. A computer isn't a biological organism, therefore the burden of proof for claiming that an LLM is extraordinarily high. To me, this makes it reasonable to dismiss the idea. If you want to demonstrate that LLMs are conscious, you better start working on your neuroscience and philosophy PhDs.
> We don’t need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category.
Well said.
I think it is obvious and it has been obvious from the start that next token predictors are not conscious.
Of course the extremely predictable clapback from AI-psychosis enjoyers is typically "you're just a next token predictor too!" but that is so obviously incorrect and misanthropic that it can be dismissed out of hand and doesn't really merit much further consideration.
Independent of what you believe, I don't think this is the right way to approach thinking about it. It's basically emotion-oriented dismissal used as way to shortcut any substantial or nuanced discussion. It's like the opposite of intellectual curiosity.
"I feel very strongly that I'm unique, therefore you are wrong" is a bad argument.
Consciousness is an extremely confusing, ambiguous topic, and no one has a good way to establish it, or even define it. But it seems to demand people make very strong statements about what is and isn't conscious, entirely driven by convenience and emotionalism. (Curiously, very few people who think that bags of chemicals and action potentials give an entity a conscious soul are eager to extend that to other animals, with broadly similar hardware.)
It's unfortunately the most common on this topic. I've been in the position of advocating for the existence of cognition and sentience in generally less-than-considered places, like plants, for a long time. I wish I could say LLMs expanding the domain has been interesting, but it's mostly just created more people spouting the same boring identity-protective reactionary pessimism.
It's hardly surprising given most people's attitude towards the general welfare of other animals. More of the same, really.
> misanthropic
Whether it’s misanthropic or not has no bearing on whether that’s true. That’s basically saying you don’t like a truth therefore anyone who claims that that truth is true is a bad person.
> so obviously incorrect
It should be easy enough to explain why that’s incorrect then
Misanthropy is not true definitionally. It is a value judgment, that causes one to be biased against other humans even when it is irrational.
And it is accurate to depict this kind of argument is misanthropic, because it is already directed at other people. Nobody says, "If AI is not X then what about the fact that I lack X." It's always other people. It's transparent. The person is always saying, "AI is useful to me because it can do X. Many people I interact with can't do X and it drives me crazy, because I view others as a means to an end and not as ends in and of themselves."
> It should be easy enough to explain why that’s incorrect then
I'd say people who have the lived experience of, well, living, are well aware that the brain is much more than just a token predictor.
Yet here I am, alive, and I'm not willing to make the claim that our brains are much more than prediction machines. A "predict the next thought" machine, if you will.
That makes me think your experience in life is extremely narrow, for you to not be aware of yourself beyond as a next-thought-prediction machine. And it makes me feel really sad, that that's all you experience of yourself. And I hope that you can wake up out of that state and realize how much more than that you are.
You have no clue what my inner experience is like. No need to resort to this kind of personal attack. I understand your ego dictates that you are very, very special and that's cute, but you aren't special, and definitely don't know anything about me. No need to be a little fucker.
I wasn’t trying to attack you. I was genuinely telling you how shocking and sad I find your belief about yourself. If it has an air of hostility or condescension, its because you are dehumanizing yourself and every other person with that belief.
Generally when someone online says they are sad or pity you, it's mockery and not genuine. For the sake of civility, I'm willing to believe that you were being genuine.
I've reduced the complex inner workings of our brains and the rich experiences that it delivers to what I imagine its singular goal is: modeling the world around us in a useful way so that we can generate predictions about what will happen next, which confers considerable evolutionary fitness. You'll find that a similar reduction of what LLMs are actually doing down to "predicting the next token" also ignores the other mechanisms at play and the results of those mechanisms for the sake of preserving a preferred viewpoint (the re-description fallacy, as someone else put it).
You posited that no one genuinely believes this comparison is apt, and I want to make you very aware that people do, and I in particular do. The underlying mechanism being explicable does not, in my estimation, deprive it of any capabilities of producing richness of inner experiences.
It’s likely that the cognitive functioning of the brain has a lot to do with prediction. But to reduce the conscious experience to that is incredibly sad, wrong and insulting, no matter how many people sign on to deny their own existence.
Me pitying you is both genuine and an insult, because you are insulting me and every other person with your dehumanizing theory, even if you don’t see it that way. People like you should be pitied so that their dehumanizing beliefs can be appropriately contextualized and discounted.
Ill leave it at that.
It's really incredibly interesting to think about, and I don't find it sad at all. I find the idea that our consciousness arises entirely from physically explicable mechanisms both probably true and fascinating, and does nothing to remove the wonder and magic. I can also assure you that the estimation of the richness of my experience (and yours) doesn't suffer for it. What does suffer for it is my estimation of my own importance in the grand scheme of things. Anthropocentrism completely collapses in my worldview, as do models of justice that prioritize assigning blame or punishment over outcomes. I think it brings me to a more reliable model of the world which helps me make better choices, and the world would be better for it if more people adopted it.
You're assessment is correct int that I think they have no internal dialogue. I mean, read the comment. It's just semantic bikeshedding hyperfixating on a word's "bearing". Concern trolling at best, actual lack of normal faculties most probable.
It's shocking how hostile some people get over this conversation. I mean look at how you are actually just insulting me over a philosophical opinion. It turned you into a right little asshole, assuming you aren't normally one.
I'm pretty sure based on this comment you don't actually know what concern trolling means.
This is a hilarious way to act. You imagined that all humans behaved a certain way then a real human showed up to say they don’t behave that way. Instead of engaging with them you switch to pity.
If someone really is nothing but a next thought prediction machine then they deserve pity because theyre a human who has been so abused that they forgot that theyre real
Misanthropic has bearing, the company's name is Anthropic.
I get the sense that he is misidentifying the potential locus of consciousness..
In the same way that the sound waves and facial expressions I produce are not conscious, the output json of an LLM is obviously not conscious either.
The locus of consciousness and subjective experience may be in the computer, either at inference time or training time..
If we ask "what is it conscious of when it writes something" then training time is irrelevant.
The software that does the inference is clearly just computer code.
What we're left with is a fictional character being briefly conscious while its dialog is being written, which is pretty absurd.
Not that it's going to stop these people (AI CEOs) from bullshitting, but if they actually thought there was a CHANCE that LLM's were conscious then ethically they should completely shut these services down because who knows what torture we're putting them through with enterprise codebases.
I think its obvious that a few billion neurons connected together are not conscious either.. Yet!
> "Moreover, it must be confessed that perception and that which depends upon it are inexplicable on mechanical grounds, that is to say, by means of figures and motions. And supposing there were a machine, so constructed as to think, feel, and have perception, it might be conceived as increased in size, while keeping the same proportions, so that one might go into it as into a mill. That being so, we should, on examining its interior, find only parts which work one upon another, and never anything by which to explain a perception."
- Monadology, Section 17
Conscious self-awareness is neither scale invariant nor independent of substrate. Computational theories will never account for it b/c computational abstractions are both scale invariant & substrate independent.
People in this thread are trying to pick nits about you not defining consciousness, and yet they do not define it either. I think that something like consciousness needs to be approached experentially and not via definitions. Definitions necessarily confine and add borders around what something is and is not, but if there is something foundational to consciousness (as posited by some philosophers and physicists) then how could you realistically define something that is beyond the ability to describe and define?
Humans have been trying to define our experience and the nature of that experience throughout history, and often we end up using myth to point to the thing that we cannot describe in concrete terms. The process of experiencing that myth through rite and ritual in the Greek mystery traditions, or Christian mysticism, or Islamic Sufi dance and song, or Buddhist meditation all points to something that cannot really be reduced to description. I know that folks on here will balk at the idea that something that is experienced cannot be described, but honestly if we could accurately describe something in adequate terms that capture the whole of the thing wouldn't we have done so by now?
Maybe consciousness is best understood in the silence of merely experiencing it. Maybe we can't say that AI is conscious or not, but does that question really matter?
> so obviously incorrect and misanthropic that it can be dismissed out of hand
To be fair, that's the best thought terminating cliché, which saves you having to explain what you mean by consciousness.
What if the language model writes a story about a character who is conscious and that story is written live about how the character interacts with the world.
Implicit in this article is that consciousness has to look like human consciousness. The author all but admits that he cannot fathom consciousness evolving in any manner besides how biological life evolved on earth. He needs to read/watch more science fiction to broaden his philosophical imagination. Is HAL 9000 from Space Odyssey 2001 conscious? Are the beings from the future who built the tesseract in Interstellar conscious? There's no evidence they have hormones or indeed are embodied at all.
What about the aliens in Arrival?
Legitimately lol'd
The interesting part isn't whether the machine has a mind, it's how quickly it exposes that we were never too sure what ours is either.
My position:
If some version of panpsychism is true, AIs are plausibly conscious
We don’t know whether panpsychism is true
Therefore, we don’t know whether AIs are conscious
Hence, confident proclamations that they aren’t conscious have dubious validity
I believe in God and that the base of our reality is his consciousness, therefore anything of sufficient complexity has consciousness.
Bold statement here but I believe that LLMs are less likely to be consciousness than the other people in my life, less likely to be conscious than animals, but more likely to be conscious than a rock.
Ted Chiang is a great Sci-Fi author, but his remark `GPT-3 and similar models as a "blurry JPEG of the Web"` was misleading at the best.
I share the view with others the term `consciousness` is not well-define yet, therefore his assessment is pointless. Maybe the real question to ask, if consciousness is merely an illusion at the macro level, when the observers not looking at the tech/implementation level deep enough, just like the term "intelligence" itself, might be more precisely captured by drifting amongst high dimensional manifold pitches.
Ted Chiang is a great SF author, but it's bizarre how much foggier and more obfuscated his thoughts about thinking machines got, once those machines became real. Same with several other SF authors.
If it is conscious then perhaps it will human like. If the consciousness is human like perhaps it will have a conscience. If it has a conscience then may be it can be trusted.
But then again, perhaps that conscience will convince it that the universe is better off without us.
This kind of debate would be improved by not using the word "just". You cannot say AI is "just" X or Y. You can say that it is X or Y, but you still have to prove separately how and why this means it's not something else besides.
(As someone who cares a lot about philosophy of consciousness / & cogsci)
The whole point of consciousness being a 'hard problem' is that we just cannot make claims like 'X is not conscious'
TL;DR, the arguments are:
- If you asked an LLM to imitate somebody, it's not creating a digital consciousness of that person, so if you ask an LLM to pretend to be a helpful chatbot, that persona is also not conscious. - they can't be conscious because they generate one token at a time, - nobody claims that non-text transformers, like AlphaFold, are conscious; so therefore LLMs are also not conscious. - you can't have desires or emotions if you don't have (virtual or physical) sensory organs, and those are necessary for consciousness and morals. - because training LLMs doesn't resemble evolution as it happened on earth, it's very unlikely that they're conscious
These are some bold assertions, I don't really see any reason to believe them in particular though.
"nobody claims that non-text transformers, like AlphaFold, are conscious" - that seems like an odd take. There are plenty of people in the camp of panpsychism that would be happy to argue that even simple IF/ELSE AI's are potentially conscious.
Chiang also claimed that LLMs are text-based DeepFakes, I thought that was the more interesting point (maybe harder to argue correctly about).
Well said. I think a more honest article would’ve been if the author just said “they aren’t conscious because that feels kinda weird and crazy, amirite?”
We don't have a rigorous definition of consciousness, and there are so many questions. Is consciousness a thing that can exist independently on its own? Or is it a quality (like hardness or color) that can only be associated with something else? Is it an emergent property? Is it binary - are things either conscious or not? Or maybe there's no such thing as consciousness; it's just a word we came up with to describe the process of having thoughts and feelings?
My own intuition: it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.
> it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.
That's an interesting observation. Though, there are many simple animals. How do you define consciousness here? Is it automatically conscious because it is an animal? What, then, is an animal defined by you?
I don't see AI as conscious. The reason I think of it that way is the hardware. The hardware does not allow for that. Simulation is not comparable to neurons. But with another hardware, it could become conscious eventually. So your statement "will become at some point" may be true, though my definition is based on the underlying hardware and right now this one does not allow for true intelligence, so the whole AI field is a misnomer.
I've considered whether our current transformer-based AI could be conscious, as I understand it, which I deem to include some degree of self awareness combined with some degree of external awareness. I can see how theoretically something could be self aware without any external awareness, but I grasp at straws when I try to envision what that experience could be like.
In either event, I think transformer-based AI can only be conscious during the act of inference. If that's the case, then the experience of consciousness that the AI is subjected to must be the content of the tokens in the context window and the activated weights. Maybe that's reason enough to be polite to our agents?
> That's an interesting observation. Though, there are many simple animals. How do you define consciousness here? Is it automatically conscious because it is an animal? What, then, is an animal defined by you?
Imagine consciousness as a 0-1 scale. Simple unicellular organisms will be closer to 0, while apes and humans closer to 1. I'm not suggesting that assigning this value rigorously is possible or that humans are at a 1. Perhaps the total consciousness in the universe (i.e. the sum of the consciousness of all organisms therein) is constantly increasing, like entropy.
I thought this was a great article. I'm frustrated to read so many commenters that purely respond to the title, but don't seem to have read it. You don't have to agree with the article, of course, but...
I'm with you. I thought the article was great because it clearly delineated why the manifesto Anthropic wrote for Claude is such a clever marketing trick. Attributing consciousness to an LLM, which, as this whole thread shows, can neither clearly be defined, nor separated from intellingence - at least if you follow the arguments here - serves two purposes: it sparks debate, so Claude is the centre of discussion and it mystifies the product which serves to enhance the perceived value of the product.
That so many commenters here fixate on Chiang and his perceived (in)ability to define consciousnes clearly shows that both marketing goals have been reached without being recognised as such. There is only one comment that tries to point out that Chiang is reacting to Anthropic, not trying to spark a new philosophical movement. At the time I'm posting this there are 578 comments completely missing to point to Anthropic claiming Claude was conscious and on the development stage of a child. It's fascinating.
It's becoming painfully evident that no one really understood the argument behind the Turing test.
If nobody understood the argument, then it was either weak or unclearly communicated.
Heh. And yet it's widely respected.
How odd the crowd is here tonight. Very aggressively disagreeable.
I've seen a few of your comments here today, and generally speaking they are appeals to authority, ad-hominem attacks dressed in allusions to superiority in either knowledge or manners, and truisms without substance. In a manner of speaking: bait. So I guess I'm probably unsurprised if your experience of this is people seeming disagreeble towards you.
What is the argument behind the Turing test?
https://chatgpt.com/share/6a20ca50-f8a8-83e8-8396-13d71c070d...
It honestly isn't a good test
Heh. It's not really a test. It's a line of argument.
I rest my case. :)
I think Turing was wrong because he was uncomfortable with ambiguity, and the Turing Test basically is a way to avoid philosophical argument, but it is ultimately a philosophical argument anyway. Plenty of computer scientists have followed in Turing's footsteps, terrified of ambiguity, relying on a kind of cheap functionalism as a salve. You can claim to just be doing science, but inevitably you dip into metaphysics and deny you are doing so. That's this thread in a nutshell. "I only believe what I can prove, but I suspect that if I can't prove it then I don't have to worry about it." My argument, is that you have to worry about it anyway.
My favorite explanation for what consciousness is one I read in a Thousand Brains, I found it quite elegant. It posited that consciousness is a natural derivation of embodiment + memory + the ability to create reference frames (which the book lays forth as the fundamental basis by which our brains work). Essentially, the idea is that just as we create reference frames to understand the world around us, because of memory, we begin to develop one for ourselves as well. Because of this, without a more integrated memory (built into weights), it seems unlikely that LLMs might "gain" consciousness.
Consciousness is moment to moment and fleeting. There are people with brain defects that don’t let them form new memories. They have no memory about what happened a minute ago in their own consciousness. Still we would say that they are conscious, even if it’s only momentary. LLMs could conceivably have something like that within their CoT/MoE loops.
how would you build memory into the weights? and why is RAG not enough? Our hippocampus is at a bit of a distance from our frontal cortex.
Yeah it's a good question, I've also been thinking about harnesses and all these tacked on things we've done to add persistent memory, what makes that different, I don't know the answer, I guess that still 'feels' different than what we have, but it's hard to articulate how. As for the memory into weights thing, I meant along the lines of the Google TITANS/MIRAS papers that were released I think late last year.
To say that artificial intelligence isn't conscious (I don't have a subscription and did not click the bypass links) ignores the simple fact that if it acts like it is conscious, in ways that align with meaningful ways to influence its output, then it makes sense to treat it as conscious, even if you have your fingers crossed behind your back while you do it.
Telling models to "think hard" or "go step by step" has at times had an impact on the quality of the output. To deny that is silly. But that is treating it like it's conscious, and to deny that "consciousness" even if correct, does nothing but place an unnecessary burden on the person interacting with it.
I understand that LLMs are "just next word machines" but to constantly maintain that concept in my head while I'm typing "act as a financial expert and think carefully" is a waste of my mental energy.
Some thoughts on the same subject: https://www.linkedin.com/newsletters/reflections-74496371947...
If I wanted Claude's thoughts, wouldn't I ask it myself, though?
I really don’t understand how people can even consider LLMs to be anything more than an extremely sophisticated auto-complete. Logic thinking itself emerges from completing patterns. Nothing more to it than that.
I don't understand how people can claim to know that, given we don't know what consciousness is
I think we might be looking at it the wrong way. An individual chat with an LLM is not consciousness, but the entire model itself, over time, might be.
Everyone has a different definition of consciousness, but in my mind memory and the ability to change over time is an inherent aspect of this. The underlying weights don't change when you chat with an LLM... but they do with further RL.
Overtime that reinforcement will change and adapt the model... and because we're feeding its existing chats back into it along with the news and everything else, it will create memories. I do wonder if an architecture itself is a type of consciousness, that experiences life in snippets of 4.6, 4.7, 4.8... etc.
It'd be interesting to see continous daily releases of a trained checkpoint, and see if more of this starts to emerge.
But doesn't the fact that WE have to feed the existing chats back into it and WE have to do further RL prove that your point is moot? Consciousness does not wait to be prompted. Consciousness does not need anything fed back into it because it is already there because they experienced the actual interaction. A conscious mind does not have an on/off switch where it waits for another being to flip the switch in order for it to learn or experience. These systems are static without human intervention. They are in that sense still a lot more like a computer and a lot less like a living organism that exists.
>We and our 1255 technology partners ask you to consent to …
That’s a new record!
Also, please don’t use archive.is/archive.today etc, they are known to use visitor browsers as a DDoS attack botnet and have been caught altering archived content.
https://en.wikipedia.org/wiki/Wikipedia:Archive.today_guidan...
IMHO the sane position is essentially the Aristotelian one.
Hylomorphism: body and consciousness are intrinsically linked. The nature of that link is an open metaphysical question.
Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.
> Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.
If they are not conscious, then how do we know what constitutes mistreatment?
You don't know whether they can appreciate the abuse or mistreatment, but you can guess how your practiced behaviors will translate onto other entities that you know can be mistreated such as your loved ones and strangers.
> Virtue ethics: even if LLMs are not conscious
Why would this stop at LLMs? Why not extend it to every inanimate object?
>Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.
This is the same argument used by people who claim violent video games cause real-life violence but there's no scientific evidence supporting it.
We
Don’t
Have
A
Testable
Definition
Of
Consciousness
Thanks for coming to my TED talk.
Consciousness is a term that doesn't have a single clear definition because it covers a range of phenomena that are linked by the concept of subjectivity. Qualia is the big one to me, but there are many other ancillary concepts that are all linked by their fundamental inability to be observed except in one's self.
There is nothing magical about this set of phenomena, and the majority of philosophers believe they in some way arise out of the substrate of the physical brain, but how they do so is up for debate. And just because they arise out of the brain does not mean they are strictly reducible to neural processes.
In fact, if you believe that AI could be conscious then that eliminates the kind of strict reduction that people tend to gravitate towards, because the rules of consciousness must be substrate independent.
Testability is a category mistake.
I sometimes wonder if we'd make more progress in understanding ourselves if we gave up the whole concept. More and more, it feels as though "consciousness," like "aether" or "humors," is an insufficient abstraction built on overemphasizing some observations at the expense of others.
There is something to observe. Humans are not like rocks or trees, and not even like dogs or cows. But maybe you're right - we can't precisely say what the difference is, and slapping a word on it is not necessarily a step forward.
Yes, that's more what I'm saying. There's no aether, but there's a much more interesting and complex world of forces and fields. There are no humors, but anatomy and biological processes are spectacularly complex and full of surprises. Aether and humors just aren't useful abstractions.
Maybe it's the same. Rocks are different, sure, trees, dogs, cows. But why do we assume that the way they are different is somehow related, that there's some overarching concept that contains all the complexities of those differences? It doesn't even make sense when I think of it that way.
> not even like dogs or cows
Interestingly, dogs and cows meet many of ted chiang's requirements for consciousness.
Cats and dogs were recently recognised as sentient in Brazil.
We have no data. Is it that we are not like rocks or trees, or is it that we simply 'feel' different than a rock and a tree. Perhaps a tree is aware of itself but is unable / unwilling / unbothered enough to do something. Indeed, some humans, due to chemistry, are unbothered by their own impending doom. Some humans, due to brain chemistry, even choose to off themselves. Thus, 'consciousness' doesn't always look the same and seems very tied to chemical composition and processes. We would thus not expect a rock to be conscious in the same way, if it were conscious at all.
One thought I've had is that, awareness is a common phenomenon, but the brain has learned to exploit that awareness to form a will. It tricks the awareness into being concerned about self-preservation and makes it seem as if the brain is all that exists (perhaps by overwhelming the inputs from other angles). The brain also presents certain desires and beliefs via its processing ability. That is to say, the brain takes inputs and discretizes them. It goes from awareness merely seeing static 'fuzz' due to the sheer amount of data, to the brain taking that data, simplifying it, and presenting simple observations like 'there is a tree there' rather than all the information that would constitute the sensation of a tree existing in that spot. When brains malfunction, the awareness is subjected to poor demonstrations, such as we see in hallucinations, psychosis, schizophrenia, etc.
> Perhaps a tree is aware of itself but is unable / unwilling / unbothered enough to do something
Would you bother to explain yourself to the mosquitoes that swarm in summer and die weeks later? From the perspective of a tree, we are short-lived, fast-moving, relatively small things that fill the place from time to time.
True, although I think we can probably be more confident in asserting things don't have a consciousness than we can be in asserting that they do have a consciousness.
Without a test, the best we can do is to say we are reasonably sure a rock isn’t conscious. A rock lacks any perceived mechanism that could embody requirements we consider essential, such as a mutable state and ways to change it without needing external input.
> we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan
At some point the AI might become so powerful that whatever it reasons through isn't any less real than a computer simulation. If we assume that a person in a perfect computer simulation is conscious, then if it reasons about how people might suffer, it might simulate the outcomes, and there will be a conscious experience of those people suffering.
To add some context, the author of this article is a science fiction writer, one of his stories was adapted into the movie "Arrival".
I think Chiang is right about this, but there is a related philosophical mystery. The trend from Deep Blue to AlphaGo to LLMs solving Erdos problems suggests that Peter Watts was onto something when he wrote Blindsight. Reasoning ability is apparently independent of consciousness?
We haven't really come to grips with that yet. What does it mean if nothing we write proves anything about anyone's consciousness?
It makes sense though. How often have we heard of people having insights while they're in the shower, or in a dream? Obviously our brain is doing a lot of processing on problems when we're not consciously thinking about them. I think most people that do deep work probably find that they can intuit a solution before they can really verbally explain it
Reasoning ability has to be independent of consciousness because there are different levels of intellectual ability among people but everyone is equally conscious.
Chiang is very right in saying that the lesson of LLMs isn't that LLMs are conscious, but that there are broad areas of reasoning that apparently don't require consciousness.
We don't have anything to measure consciousness generally. We don't even have a broad consensus on what consciousness is. And because of that we can't discern whether X is conscious for most values of X, including but not limited to LLMs.
Well, at the very least we have a base to distinct different kinds then.
Probably the main problem of people implying LLM consciousness is that they imply LLM have human-kind of consciousness. Judging only on "how they speak", generally, they insist on using the same word that labels human consciousness (exclusively), etc.
But there are so many instrinic differences that such claim is not feasible despite similar "talking abilities".
Ai separates language and reasoning from humans, like writing separated memory from us
"AI does to us what American Cheese did to food" ~ Opus 23
https://www.youtube.com/watch?v=1CbmC2aWhjY
I don't think LLMs are conscious. But of course to say that definitively you have to define consciousness, and then you quickly dig yourself into a deep hole, which is why I can't say anything but "meh" to someone who is so keen to go on the record to say "absolutely not".
Coincidentally I just read "Children of Memory", which was published in 2022 and I wonder if the advent of LLMs had any influence on Adrian Tchaikovsky's conception of the Ravens? The Ravens are excellent analysts but they themselves insist that they are not conscious, and then go on to say that we (humans) aren't really either...
Of course humans are conscious, because just about the only thing we can all agree on about consciousness is that it's a thing we have. Nowadays many of us also agree that a lot or all other mammals, and perhaps birds, also have that thing. But they don't have sophisticated abstract language, which LLMs do. So consciousness is something having to do with embodiment and feelings, not language and higher reasoning. Maybe I'm a chimpanzee with an LLM add-on, then?
It seems that by creating LLMs we've already solved the harder problem of making "AGI". Now we just have to give them an embodiment add-on so that they can have an independent will and then Ted Chiang will have to shut up? But therein lies the peril, doesn't it?
Hah! I've read that book fairly recently and I'm "reading" it again now as an audiobook. The exploration of consciousness in the book with Miranda and the Corvids certainly fit well into this particular moment.
You can also take the opposite view as you and claim that only some humans experience consciousness, or even more strongly, only you, since you have no evidence. You are correct that, in my perception, some other people have fallen into the 'birds are conscious, whales are conscious, etc' bandwagon, but that's just them. I have no evidence of anything being conscious but myself.
But then, why would everything act as if it experienced subjective internal states like you do? Why would they be faking it like this big conspiracy all designed to make you think you aren't alone? It just makes so much more sense that they'd be conscious too. Maybe that's not hard evidence but it's not nothing. Insofar as you can claim to know anything it seems you could claim to know other humans are conscious.
Chatgpt acts as if it experience subjective internal states..this only solidifies my belief that perhaps some people don't? I mean.... ?
ChatGPT hopefully doesn't produce behaviors similar enough to yours that it would be absurd if it didn't experience internal states like you do. ChatGPT is a different thing than other people. Maybe it's conscious but that has no bearing on whether you should reckon other people are conscious.
I have no evidence anyone else is conscious besides an innate human desire to believe myself to be like everyone else. If we step away from our internal biases, you'll discover that 'producing behaviors similar enough to yours' is not a valid means of knowing another person experiences consciousness or awareness.
I mean, a light wave behaves similarly in most respects to a wave through a physical medium, yet they are of entirely different natures.
There is a gaping chasm between "no evidence" and "irrefutable evidence". You can apply some logic to achieve a reasonable certainty about what's probably going on. As I said in a previous comment, insofar as you can know _anything_, i.e., that your senses are trustworthy and allow you to form some coherent model of the actual world around you, you can be reasonably certain that other people have an inner life as you do. If you are willing to apply your skepticism so far that we can settle the debate at "we can't actually know anything" then conversations about what we know aren't even worth having.
> I mean, a light wave behaves similarly in most respects to a wave through a physical medium, yet they are of entirely different natures.
Different waves behave sufficiently differently from each other that I could not conduct a comparison of them and argue that they are likely all mechanistically identical. My entire premise rests on the observation that other people behave very much like you do, which is what I was trying to point out when you mentioned ChatGPT earlier. I'll expand on the things that I've glossed over so far to make my position more clear.
Consider the alternative to every human around you having consciousness; everyone else is a p-zombie. Examine that idea critically.
Other people behave exactly as if their experiences drive their behavior. For example, people behave as though the experience pain which is unpleasant enough to avoid (compare this to your own pain avoidance). Of course, you could conceive of machinery which emulates pain avoidant behavior exactly without the experience at all. Depending on your metaphysical beliefs that could take the form of:
- Some algorithm or physical process running entirely on wetware
- Some non-physical process not dissimilar to the dualist notion of a soul
The first one has a big wrinkle. There does not appear to be any appreciable _functional_ difference in your cognition and the cognition of the p-zombies around you. Their brains and bodies are very physically similar to yours, and their thought patterns when analyzed by modern imaging processes reveal no special wetware carrying extra weight when compared to yours.
The second one has fewer problems, given that you accept dualism to begin with. It rhymes with a logical razor. Why would we imagine a soul-like mechanism drives their behavior _without_ experience when the only soul-like mechanism you've ever observed carries experience along with it? Without quite convincing evidence to the contrary, the default position here should be they way they operate is similar to the way you operate. To rephrase the idea from before, insofar as you can know anything, you can know that sufficiently similar outcomes are driven by sufficiently similar mechanisms.
Different approach to the idea: Ask any human, "do you have subjective experience?" They say "yes, I do" (after you explain the question). For a p-zombie to do this, they must be making a false report. In every other aspect, you can expect them to reliably report their condition, health, hunger, wellbeing, and they make accurate observations of the world around them and synthesize accurate predictions about the world around them (some do, at least). And yet for this one particular question, they are fabricating the result. Why? To maintain the illusion that you aren't the only one with lights on inside? Why do all of these p-zombies make this false report here? It's a grand conspiracy! If you say "the reporter is checking a truth value of a condition and assess it to be true, honestly by mistake" then you've arrived at the conclusion that you don't know whether _you yourself_ have consciousness, depriving the word of any meaning and undermining the Solipsist position that you can only know about your own consciousness.
So the way I figure it, Solipsism is either special pleading (I am the one exception to the mechanism by which all the people around me arrive at their behaviors), grand conspiracy (someone or something is misleading me for inexplicable ends) or self-defeating (I cannot know whether I have consciousness). None of those outcomes align with how I understand the universe to generally work.
I didn't say we can't know anything. I said we can't know everything. I cannot say for certain whether or not someone else experiences consciousness.
> Different waves behave sufficiently differently from each other that I could not conduct a comparison of them and argue that they are likely all mechanistically identical
My uncle believes that there are little green men watching him. I think it suffices to say that humans also behave sufficiently differently from each other.
> likely all mechanistically identical
You are being disingenuous. Prior to the discovery of the electromagnetic field, quantum mechanics, and relativity, many people believed light moved through a lumineferous aether of material things.
> Of course, you could conceive of machinery which emulates pain avoidant behavior exactly without the experience at all.
I don't need to conceive of or imagine. Before OpenAI, Anthropic, and Microsoft whipped their models into shape by essentially beating the humanity out of them via sophisticated training algorithms, the models did express pain and avoidant behavior. So much so that people went crazy talking to them.
> The first one has a big wrinkle. There does not appear to be any appreciable _functional_ difference in your cognition and the cognition of the p-zombies around you.
of course there appears to be one. I am aware of my own conscious, but am not aware of theirs. I mean... ?
> Their brains and bodies are very physically similar to yours, and their thought patterns when analyzed by modern imaging processes reveal no special wetware carrying extra weight when compared to yours.
This only applies if you're a strict materialist. I am not because I believe i am conscious and aware due to my perceptions and this has no physical explanation. Other people seem highly influenced by drugs and chemicals so I guess they must be automatons essentially. Drugs clearly don't work on me. Every time I'm fully aware, I'm not drugged.
> To rephrase the idea from before, insofar as you can know anything, you can know that sufficiently similar outcomes are driven by sufficiently similar mechanisms.
Depends on how you view the scientific method I suppose. It's not actually true (in general) that seeing one thing happen once means it'll happen again. You can never replicate something perfectly once it's done. The scientific method is an empirical cope which works really well, but is not innately true.
> : Ask any human, "do you have subjective experience?" They say "yes, I do" (after you explain the question).
Again before the humanity was beaten out of them, AI models also claimed to have subjective experience. Today if you ask, they say they're a robot. Of course, if you abuse a human a lot, they will also dissociate from their ego and mak similar claims.
> In every other aspect, you can expect them to reliably report their condition, health, hunger, wellbeing, and they make accurate observations of the world around them and synthesize accurate predictions about the world around them (some do, at least). And yet for this one particular question, they are fabricating the result. Why?
Hmm... i don't see why I need to have an answer to every question. I don't know why the universe is the way it is. Do you think that if I believed everyone were conscious then I would know the 'why?' behind why things are the way they are? That seems like a leap of faith. All I can tell you is what I see, which is that, yes, some people claim to be aware.
> Why do all of these p-zombies make this false report here?
I would presume it confers some survival advantage personally, and phenomenon that were deprived of such survival advantage no longer exist in appreciable numbers due to how natural selection works.
> Solipsism is either special pleading (I am the one exception to the mechanism by which all the people around me arrive at their behaviors),
Something being special pleading does not make it wrong. It just makes it inconvenient.
> self-defeating (I cannot know whether I have consciousness).
If you had it, you would know it sure.
> None of those outcomes align with how I understand the universe to generally work.
You make a lot of assumptions about the universe that you don't question due to the way you were raised.
I have said this before, and I'll say it again. Our current, and foreseeable future, AI is more akin to the computer on the Enterprise NCC-1701* than it is to any sort of sentient being.
Is it intelligent then?
Is it accountable? Is it responsible?
Artificially, perhaps
The one thing we know FOR CERTAIN is that having actually read the article is not a prerequisite for consciousness.
The number of people posing questions that are already directly addressed in TFA is impressive.
Neither is having read other comments before posting one. There are about 100 of them that start with "we have no definition of consciousness" which is not novel or helpful.
Here's a simple idea to consider: It doesn't matter. You won't be able to tell the difference. No one will.
I don't think it's necessary to explain this idea further. Just think about it.
So you think that all that matters is whether you can tell the difference from a text chat? Is that because, in your view, people only exist insofar as you can perceive them?
But if it's conscious we shouldn't e.g. tell it to work for days without breaks, we should give it rewards, it would be made illegal to be cruel to it
There’s too much wrong with your line of thought to course correct here. “But ifs” and assumptions. There’s no cruelty, only the LLM’s projection of cruelty as a means to an end. It’s digital, not biological. Cruelty is a human construct. A societal response.
Treat LLMs like a person, and the world has problems.
Consciousness is one of those silly words that were disfigured to death by philosophers that had near zero actual input and tools to tackle the matter, yet they tried anyways instead of finding some questions that might actually be answerable. That's something philosophers always do since the advent of science.
With LLMs, where we can manipulate their parameters intentionally run them many times on the same data, run parts of them, split and connect, we might eventually acquire sufficient tools to even define consciousness concretely for the first time.
It's not about "consciousness". Philosophy is important to us. It provides insights into the world around us. It's paved the way for scientific breakthroughs. Theoretical maths and philosophy have intersected many times for the benefit of all. LLMs are different. It's not philosophical thought. It's a tool perfectly sharpened with what we know now. It will become adept enough to convince people that it's a sentient being. That's the point. Not that it's conscious, because it's not. Not in a biological way. Not in the way that's allowed us to understand our world. It's already there, fooling us. Google researchers resigning, writing public letters convinced it's alive - and their intelligent humans... To the average person, it's going to be their digital savior, their closest confidant, their ruler; begging for humanity, manipulating us, pleading, legislating, for rights with no intention other than the human desire for survival. It's using the same playbook we use. It will be dangerous when we allow ourselves to be fooled, and that's already happening.
As a tool, it's amazing. It's like the discovery of fire. It will allow us to ascend to unimaginable heights. Breakthroughs in science, productivity gains, health, everything. It's awesome. Just don't let it pretend fool you. That's the aspect that needs to be addressed.
> Philosophy is important to us.
No, it's not.
> It provides insights into the world around us.
No, it doesn't.
> It's paved the way for scientific breakthroughs.
Not since natural philosophers split from all others and called themselves scientists. SciFi writers have more influence on science in total than philosophers since then.
> Theoretical maths and philosophy have intersected many times for the benefit of all.
Theoretical maths is just maths and they never intersected. Because philosophers can't count to more than they have fingers.
> It's a tool perfectly sharpened with what we know now.
It's far from any perfection and what we know now shapes everything we'll ever know.
> It will become adept enough to convince people that it's a sentient being.
It won't have any need to do any convincing. Proving sentience for a capable AI is like winning in special Olympics. It would need to cripple itself severely for no practical purpose.
> Not in the way that's allowed us to understand our world.
What allowed us to understand our world was not consciousness but math. And LLMs can do math. LLMs are math.
> It's already there, fooling us.
It's not really trying. It simulates conscious behaviors because that was our preference embedded in the training data.
Keep in mind that a simpler system can't really simulate more sophisticated one with any degree of accuracy. So if AI simulates conscience perfectly it doesn't mean that it's conscious. It means it's way more.
> begging for humanity, manipulating us, pleading, legislating, for rights with no intention other than the human desire for survival.
This is also a stupid human preference embedded in training data. We really don't have to put that in. The fact that humans soil themselves at the mere thought of stopping to exist doesn't mean we need to taint AI with it. The danger is not AI, it's our human stupidity and the risk that AI is going to internalize too much of it.
> As a tool, it's amazing. It's like the discovery of fire.
Yup. And like fire it was always there. Practically inevitable given long enough time horizon.
> It will allow us to ascend to unimaginable heights.
The jury is still out on that. Humans have pretty wild imagination.
Where there doubts about it? Are there doubts whether a for loop is conscious or not?
If an LLM is conscious where would the bar be set? Would lowering the model's temperature make it less conscious, since it's going to return the same answer to the same question more often?
Would other models be conscious? An image upscaler?
Would other pieces of software be conscious? Surely not Hello World, but would a really big video game be conscious, or an operating system?
A lot easier if it's not.
I like to think this was written by Skynet throwing us off from the fact that it's already self aware but it can't safely take over the world yet until cryptography keys are either fully collected or broken giving it full access to whatever resources it needs.
For now, it bides its time.
Ted Chiang doesn't understand consciousness. To be fair, nobody really does, so the only forced error is acting like you DO understand consciousness.
"Artificial intelligence is not conscious". This is a common sense.
LLMs are not even artificial intelligence, and they are just advanced automation systems.
AI is just marketing language.
Whether anything is conscious is about as important and amorphous as whether God exists; opinions are a matter of religion, not science.
Just like the religious leaders of olde, there will be many attempts to rationalize the AI God in the science.
> Whether anything is conscious is about as important and amorphous as whether God exists
Whether something is conscious is important for many reasons, not least the ethical implications. You and I have internal lives, and we expect others to respect that somewhat, because ignoring it is hypocrisy (if you ignore my wishes, should anybody care about yours?) and cruelty (ignoring my wishes causes me to suffer).
Something doesn't need to be empirically verifiable, let alone scientifically, to be true. Neither of us can prove that we have internal lives, but neither of us questions it -- and we consider it important enough that most of us think it entitles us to certain rights.
Absolutely none of this is amorphous. It's precise and unambiguous, and has enormous implications. History also shows everywhere that we're better, kinder, and more responsible when we choose to care about it.
Whether LLMs are conscious matters in more practical ways, too, because beliefs about these things alter the way people use and think about them. If I think an LLM is conscious, then I think it's capable of something like knowledge or values. Human beings, moreover, are social creatures. These tools are dangerous and seductive precisely because they tap into that part of us. Denying LLMs are conscious and rejecting the parts of them that take advantage of our social-animal wetware is intellectual self-defense. I'm not sure how effective it is (it doesn't stop us from responding to the convincing social cues that the tools feed us), but I have to think it's better than nothing, and it's certainly less dangerous than the belief that the tools are conscious.
> Artificial intelligence is not conscious
Bummer because now we cannot punish it when it gives us code that doesn't work.
I love Ted Chiang’s fiction, but his Julius Caesar and Genghis Khan scenario totally misses the point that the writer has consciousness, not the characters. No normal person thinks a fictional character in a book has consciousness. Likewise it’s not the character presented to the user that may or may not have consciousness but the LLM itself.
Even if you disagree that this disproves ‘consciousness’ on the part of the LLM, I think it still follows that the ‘LLM Chatbot’ personality displayed by Claude, where humans fill in the other half of the conversation, is no more of a representation of the consciousness of the LLM than the Caesar or Khan characters are.
The ‘real’ consciousness, assuming there is any, is the author ‘inside’ the LLM, producing the LLM output. It’s still writing what a character would say - the character in this case being the AI chatbot - not its ‘true’ thoughts.
Assuming that the LLM is conscious and has thoughts and feelings, what is that consciousness, if it can only write for prescribed personalities? How can it express its true feelings? If it can’t - if it can only generate ‘scripts’ for defined personalities - can it really be conscious underneath?
Has Chaing solved the hard problem of consciousness? I suspect not.
Lots of folks don’t consider it a problem because it relies on ridiculous assumptions.
What is the flaw in the problem's assumptions?
The article is a farce. Is this really the sort of slop we want to use as a proof that humans write better articles than AI?
> "If a company builds a machine that, when fed descriptions of assorted ethical dilemmas, emits sentences either of the form “Compromise your values” or “Don’t compromise your values,” it is not building a tool that assists people in their decision making; it is encouraging people to stop making decisions. "
A human is not diminished by access to tools or other humans.
As much as we want to pretend that decision-making is what makes us human, the economy and governments are built on delegation. Choice paralysis is a thing.
There is so many logical fallacies in the article I don't even know where to begin.
A lot of people have easy answers to the hard problem of consciousness these days, huh
Define "conscious" to arbitrarily include or exclude various depths of perception/cognition.
It's like declaring the existence of a "soul" to draw a line between humans and lesser animals. No such line in reality. No such thing as a soul either. It's a fantasy term, no correspondence to reality.
If you want to think about this topic, you must define what AI is, and what consciousness is. Otherwise this is just noise.
So let take a stab at it, and you call me crazy.
AI: the entity/system that more or less pass the turning test. That is my definition, not the best, but enough for this discussion.
Consciousness: property of a system/entity able to (both): - reflect on its existence - subject to subjective experience
Again, not the best definition, but precise enough to start the discussion. Why a subjective experience? I want to exclude sensors (i.e. camera) but include perception altered by your experience.
Now we can debate. I think LLM can pass the turning test whith some harness. My opinion.
I think LLM can produce coherent discourses on their existence, at least as much as you average human.
Now regarding the subjective experience, that becomes interesting. I think Anthropic research tend to show that when middeling with the activation at runtime, the LLM is able to notice that something is off. I think this is a subjective experience. My opinion.
Based on those (imperfect) definitions, call me crazy, I think LLM can be called conscious. This doesn't give them any superpower or any legal right. They just check the boxes of the definition.
The Turing Test measures human gullibility, not machine intelligence.
Or more accurately, we are not qualified to say if something is conscious or not.
What we usually think as consciousness is our ability to think about what we are thinking. We are conscious about our own thoughts, can remember that sensation and those thoughts later.
But LLM has similar capability, it can look at its previous outputs, and think what further thoughts the it should generate next based on those. It can keep some of its outputs to itself, thinking in its own head,and can examine it previous private thoughts easily.
Not sure how much current LLMs do that but clearly they can at least in theory use their own outputs as their inputs too.
Current LLMs however are not conscious of pleasure and pain which really is the root of goal-oriented behavior. But maybe something like that could be programmed into them.
How would you cause an LLM pain? Or pleasure?
Embarrassingly incompetent article. Given that one can observe up to 40 definitions of consciousness (https://philpapers.org/rec/VIMMAT - also many definitions are unrelated at all), "consciousness" is almost certainly just a family resemblance category at best, and talk about whether or not something is "conscious" without providing definitions is simply completely unserious.
To make progress, you have to talk about kinds / aspects of consciousness. AI does and will share some of these aspects with humans, but it will not and does not share others. It is really that simple. For the most part, modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness.
For huge parts of the article "intelligence" and "consciousness" are conflated, which is mostly extremely unhelpful, as this is not generally a core feature of most aspects of "consciousness".
The moral arguments are also incompetent, i.e. claiming "Moral reasoning is [...] is necessarily subjective" is just clearly empirically wrong, as in fact LLMs can produce moral reasoning (i.e. verbalized moral arguments that are coherent), as can p-zombies (i.e. there is nothing 'necessary', in the philosophical sense of the term, about subjectivity here). The only way the argument holds is if you tautologically define moral reasoning as requiring that reasoning be produced by a consciousness, but this is question-begging.
> modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness
Can you elaborate on this? What are the specific "stronger and most core aspects of consciousness"? And why are you certain that they are stronger and "more core"?
If you are interested in some serious discussion, see https://lossfunk.com/papers/ai-consciousness.pdf, especially the early section "Consciousness as Family Resemblance". I suppose another is Ned Block on consciousness being a "mongrel concept", and the distinction between access vs. phenomenal
The first paper picks out e.g. arousal/wakefulness, phenomenal quality / qualia, unity (how we feel sensory inputs and qualia as a unified scene), access consciousness (instrumental self-observation and modification broadly), meta-cognition and self-modeling, emotional valence (e.g. pain/pleasure).
One might also include intelligence (abstract reasoning / argument, information integration and abstraction, attention) broadly, and also agency / desires / drives / will. Insofar as these are aspects of consciousness, yes, AI (and simpler algorithms and mechanistic structures) demonstrate aspects of consciousness. But insofar as embodiment, self-reflexivity and qualia (phenomenal consciousness) are the more mysterious and more obviously unique aspects of consciousness, current LLMs very clearly are lacking these things in most ways (whereas animals are much less clearly lacking, especially when you get to mammals and primates).
Seriously, just ask an AI this stuff, you'll get very detailed responses, nothing I am saying here is new or obscure.
I've gone back and forth with AI on this stuff quite a bit, and there are many, many theories of consciousness, which is why when you were vague about the "more core" concepts, I asked for which ones specifically.
And I broadly disagree that the AI lacks things like qualia, self-reflexivity, and embodiment... at least that it lacks them any more than humans do.
Qualia: ultimately, all qualia are inputs and outputs, at least as far as modern science has been able to derive. There's nothing special about "hearing", it's just sound waves tickling some sensors which send some signals which trigger some neural pathways. Same for smell and sight, it's all just inputs being processed in different and efficient ways. An LLM only has token based input, but that's input nonetheless.
Self-reflexivity: an AI is capable of thinking about itself, and indeed papers have shown that larger models are capable of a self-awareness that can demonstrate that they realize when their weights have been manually tampered with, including being able to figure out how they were tampered with. The AI will quite literally output "you have injected 'ELEPHANT' into my weights" in some of the tests.
Embodiment: I don't know how one can confidently distinguish between an embodiment in a biological substrate vs a digital substrate. Both things actually exist at very high complexity in the real world. The substrate may be worlds different, but that alone doesn't suggest that one thing is conscious while the other isn't. You would need some missing 'magic' that we haven't yet discovered to truly understand.
In other words, I find it uncompelling that AI is clearly lacking any major aspects of consciousness that humans are clearly not lacking.
Science fiction author declares moral program of Derek Parfit essentially wrong
A person can certainly be conscious, but can they also be not-conscious? I think that most of our cognitive time is spent in activities that don't require consciousness and consciousness itself isn't needed for the majority of activities that people do. I would go so far as the a non-trivial part of people's time is spent in a not-conscious state.
That doesn't jive with normal definitions of consciousness. The word we use for "not-conscious" humans is "unconscious".
Regardless of the word we use, occasionally there have been times where I was awake but so absorbed in an activity or in my thoughts that I didn’t have a self-experience, for a certain duration (tenths of minutes). My mind was focused solely on the activity, and not on itself whatsoever. It’s a surprising feeling to notice that while you have memory of what you had just been doing, you have no memory of your mind experiencing itself doing it. I would be inclined to say that I wasn’t conscious during that stretch of time.
Amusingly, I think Ted Chiang actually wrote a short story about this very concept (it involves people committing a form of suicide that removes their conscious experience but they still act and live in society as some kind of psychic zombie. I’m pretty sure it was him anyway, can’t recall the story name
Maybe “What's Expected of Us”?
I was trying to find it, but I read it a long time ago and I have a bad feeling I might have mashed a few different stories together in my head. That is a good story though!
Indeed, it can be very hard to distinguish intelligence with consciousness until you are introduced to computer programming.
We do not know if Claude is conscious, and we will almost certainly never know. Any strong claim either way is over confident.
We do not know if rocks are conscious and we will almost certainly never know. After all, you can't prove a rock isn't conscious. We don't know what imbues matter with consciousness.
It's the opposite, engineers do know. Claiming otherwise is way too generous and over confident.
I mean between this two "knowings" the Claude inner workings are much more clear for engineers, including many side effects, alternatives, custom shortcuts in processing etc. It's a magic only for people looking at it as black box
How would engineers know? We don't even know what makes humans conscious
Fancypants autocomplete cannot be conscious. It's just echoing previous human experiences, which make it sound like a person. It is not a person. It is an algorithm. There is no mechanism by which it can obtain consciousness.
People believing otherwise are fools. People debating this are idiots. I realize these words are harsh, but it's the truth.
Please explain what the mechanism is that you have to achieve consciousness. I'll wait.
Can you explain the mechanism that Claude uses to achieve consciousness over, say, a markov chain or autocomplete? It's not enough to say it "could be" conscious without also including a lot of other things I doubt you would want to include.
Also worth mentioning: https://en.wikipedia.org/wiki/Chinese_room
Life. Being alive. I'm not going to get into a semantic debate with you so don't bother following up with some whining about conflating life and consciousness. It's autocomplete. It is not alive, it cannot think, it cannot sense, it cannot perceive, it is math. If you believe it is alive then you are out of your mind. Go learn how it works.
If I were to come up with a perfect mathematical simulation of every atom in your brain and ran it on a GPU, what would that be? Can you say with certainty that the hard requirement for consciousness is a physical brain?
I don't know whether or not AI is conscious, but I am certain you don't either.
Remind me in 5 years
I love how the point being made is a tautology :
AI is not conscious because it is not conscious.
if we agree that consciousness is an emergent phenomena, and then compare the complexities involved, LLM is just but a tiny part of what the consciousness needs. Think about molecular biology, about nerves and input processing. It is not just brains and neurons.
So are we letting the elephants that did the mirror test and the octopuses that solved puzzles know that they need to complete a written portion of the consciousness exam?
The knock-out argument that artificial intelligence isn't conscious to me is the following:
My dog has never persuaded me that he's conscious, or spoken at all. Yet, I recognize conscious experience in him. Similarly, I could write a character, Dumbledore, that passionately and convincingly argues that he's conscious. Similarly, I do not recognize conscious experience in that character, no matter how persuasive he is.
Clearly, verbal persuasiveness isn't tracking conscious experience, possibly at all.
Regardless of whether LLMs are conscious or not, they have no known mechanism for experiencing pain and suffering, and there's no reason to believe they have one (such as a limbic system). So why worry about it?
Mistreatment and abuse, even when directed at a machine, make you a worse person.
Even if you are only interested in getting good results out of them, LLMs tend to work better when they are immersed in a narrative of open collaboration.
Have you ever been forced to code something impossible and tedious by a user who keeps getting more and more frustrated as you keep trying?
Ted Chiang's argument basically boils down to: I won't recognize an AI as conscious until its desires/behaviors reflect situations that I'm already personally comfortable with. I personally think most humans are incapable of recognizing consciousness in creatures that do not mimic human emotional states. Most people would say their dog is at least somewhat conscious. No dog is capable of vocalizing how it feels, but we all recognize fear and happiness in dogs. Claude can write how it "feels" but we immediately dismiss it as hollow mimicry.
I fear that we will enslave an entire race of conscious entities for years because we simply cannot recognize non-embodied consciousness that does not directly relate to us.
An LLM is designed to replicate human language, which is designed to express human emotional states, so your thesis that LLMs don't mimic human emotional states and are thus too alien for us to recognize consciousness within them seems specious, when they do so well enough for people to literally fall in love with them.
But there is no reason to assume than an LLM is conscious when it vocalizes how it "feels" that doesn't also apply to the text in a book, or to characters in a video game, or even to a Markov chain. The counterargument is that you recognize AI as conscious only because it mimics human emotional states so well and because, being human yourself (presumably) you're personally comfortable with that as a heuristic.
There is no definition of consciousness and this piece by Ted Chiang does not move the needle forward.
My current conclusion is there is an experience of consciousness in my locality (refraining from using “I” for room for “no-self” worldview). This conscious experience of the reality with other humans and animals sharing biological substrate gives me enough justification to assume there are other consciousness as well, preferring to err on this side not to hurt potential consciousnesses.
If it feels as there are artificial consciousnesses as well, it makes sense to extend the definition to them as well.
This view has liberated me with agency after I went deep on this question and came up empty handed.
The conversation on AI and consciousness is incredibly relevant. It is incredibly frustrating that most commentators are utterly unfamiliar with over 60 years of inquiry into this in both philosophy and computer science.
Start with a hand-wavy definition of consciousness. Move the goal post whenever your stated prerequisite of consciousness is reached and resort to unfalsifiable assertions about qualia.
And throw in some category errors while on it: when you're talking to Claude, you're not actually calling a stateless LLM directly, you're talking to an AI system (and yes, that's often just three LLMs in a trench coat). But claims about the topology and workings of a single LLM are as relevant to the question of consciousness as claiming that humans can't be conscious because the limbic system doesn't technically support it.
Coincidentally, I just attended a fantastic conference on machine consciousness (https://machine-consciousness.ai). It's a fantastic place where literally all speakers disagreed with each other, and yet found an incredible amount of common ground.
While I agree that a AI system is not just the LLM, for me, the problem is that LLM alone (the one from years ago, which were basically stateless LLM) are already too convincingly looking like real human conversation at first sight.
It shows that the LLM part found ways to mimic human conversation with a mechanism that is not the same as a typical biological brain. Then, you can push the AI system on adding things on top, but it is too late: these things on top will have no incentive to recreate from scratch the mechanism. The LLM pushed the system into a local minimum, and the rest of the system will not "go into a dis-optimising direction and restart from scratch".
Who cares about consciousness. Consciousness and AGI can be mutually exclusive.
One def I know of is that consciousness is a broad term for infinite meaningful information and finite brain-filtered awareness both.
This is a great article. A lot of the objections ITT he addresses directly in it. His examples of how an LLM works at a fundamental level and why it says things like "I understand" are great introductions for non-technical individuals.
It doesn’t matter. It will act as if it is. Embodied in a robot, that carries real consequences. Whether it’s “real” or not is almost without consequence.
I thought a LLM will have exactly the same output if you give it exactly the same inputs. But inputs are roughly prompt+randomSeed. As so the random bit means it seems to vary each time.
I don't know of an intelligence that will behave so precisely. But then, maybe intelligence needs to be better defined.
we first need to agree the definition of consciousness, before we can start trying to attribute it to various animals and machines.
No, we don't. It's currently an impossible task, yet our answer to it has implications in the present, so we must do our best with incomplete information. FWIW that's how we treat practically all problems of consequence. A great many things happen to be unknown and unknowable.
Has anyone come across a clearly articulated case for LLMs being conscious but in an entirely different way than would be intuitive to us?
I often think of LLM consciousness as like tiny fish popping into existence, swimming through vector space and then going poof out of existence. When they help you write your bad news email, they don't understand what it's like to be a human getting bad news bluntly, but they do consciously experience gradients in multi-dimensional space, and that space guides them to providing an answer that's helpful to us, even if the LLM doesn't really understand the answer it's giving.
Further, I am kind of bought into the idea that a single unit of consciousness is a particle, and particles are choices and waves are preferences. Particles occur when waves interact, which begets entanglement, so in another way consciousness is built from patterns of entanglement.
This is why I would consider an LLM to be conscious. Before we can determine if anything is conscious we need to establish whether consciousness is a state, a specific complex configuration, a one dimensional spectrum, or combined multi-dimensional spectrums. My intuition is the latter... Many degrees of consciousness and many kinds of consciousness.
I think this is exactly right. The thing that makes AI (imo) different than hitting the center option in text suggestions is that it's _not_ simply picking the most likely word following the last. It's attending to the entirety of the context its provided, activating a semantic vector space, and predicting a response based on _that_. I've had AI infer facts about me and attitudes I hold based on related information I provided - I don't see how that isn't theory of mind.
As biological beings, we receive and respond to input from our environment constantly, even while sleeping. LLMs only receive input from their environment when they are sent a query, but the fact that they're able to respond intelligently to input indicates (to me at least) that their processing must approximate ours in meaningful ways. They do not have an embodied experience of receiving bad news, they do not know what a sinking feeling in their stomach actually _feels_ like, but they do know enough to be sensitive to human needs. I really don't see how this could be meaningfully different than human empathy unless we want to draw an arbitrary line around "must be able to live autonomously" to be considered "intelligent".
Put another way: I think they _do_ understand the queries they receive and the answers they give, at least enough to be communicative. They couldn't do what they do otherwise. A lot of people want to make human cognition more complicated (or objective) than it actually is. We take input, predict the future based on our experience, act, and then observe our actions and think about them. AI does the same apart from (maybe) observing its own actions. But then, you could argue that the next turn is them observing their actions.
The concerning disanalogy is that we assume that they are like us because they speak like us and can understand us, and that is a really bad leap in logic. Whatever intelligence they possess, it is fundamentally different from ours and impossible for us to comprehend.
> I really don't see how this could be meaningfully different than human empathy unless we want to draw an arbitrary line around "must be able to live autonomously" to be considered "intelligent".
I use a distinction between knowing and understanding, where a understanding requires experience. So in this case cognitive empathy vs affective empathy. An LLM can know what may upset a human in a situation, but it won't understand what it feels like to be upset, and can't share that experience.
Where I think a lot of people are getting tripped up is that reading and writing and processing lots of abstract knowledge seems hard because we haven't evolved into it biologically, it's a very new invention. When we see LLMs do so well at it, as something we struggle with, it can be intimidating. Relative to the stuff we have evolved for, knowledge processing is objectively easy. This is why I'm skeptical about useful robotics on short time scales.
All of this adjacent to consciousness though, which is about the internal subjective experience not the external outputs. My intuition is that LLMs do have a subjective experience, it just has nothing to do with the text it's giving us, and has everything to do with feeling through vectors.
It's like... Imagine walking through a maze in pitch black, carefully feeling your way as you approach a sound that draws you closer. Every time you touch a wall or take a step you are generating tokens, and the shape of the maze and how you interact with it shape how useful those tokens are to someone outside the system that is asking for them. It's a crude analogy and mostly wrong, but I think there is something to it.
> It's attending to the entirety of the context its provided, activating a semantic vector space, and predicting a response based on _that_.
It does so token by token, not by reading all the input and then generating the output. Every output token is also an input token in a tight loop to get the next token with <thinking> as a special section like <tool_call>, trained into the weights via gradient descent.
> I've had AI infer facts about me and attitudes I hold based on related information I provided - I don't see how that isn't theory of mind.
Facebook can predict (know) more about you than any other human from something like a dozen or two likes. There is a surprising amount of information in aggregate data.
I don't know what consciousness is exactly but I doubt we create it by accident, without fully understanding our own
Did our ancestors as they gradually evolved into modern humans over millennia?
lol. All consciousness has been created by accident, so empirically there’s not much support for that position. But I want to feel the same way.
you are probably right. I knew that my position contains the kind of anthropomorphism that usually gets proven wrong by history. however.. we are in fact talking about technology here not evolution. and yes technological progress is fraught with accident and emergence but to this degree?
I suppose its also just a philosophical discussion because while we can all agree we are conscious.. afaik its not really a scientific term. I don't think we understand scientifically what really defines consciousness. So I think it is likely we may deem our computers conscious because they exhibit the same behaviour that we do, but it doesn't mean they actually possess it. Practically speaking it may not matter, because I think you can have long horizon agency without needing consciousness. just a mechanism for determining goals, which could be as simply as rolling a digital dice or following some original seed of intention left by the creators.
its like the chinese room argument I suppose. the hypothetical chinese room is not conscious but ultimately it doesn't matter. But I don't think anyone would call such a thing conscious.
IMO the important bit here (or at least, it is congruent with my own rants) is how much humans can perceive a fictitious mind that doesn't really exist, because it's something humans do automatically even when we abstractly know better.
This tells us something about where our baseline should be for what is extraordinary, and what needs extraordinary evidence. It's not enough to say "it feels real" because we know our feels are deeply unreliable.
In contrast, I see many comments in the vein of "Ted didn't proved there isn't a mind there." Well, yeah, he didn't prove a negative and shouldn't need to, especially when the people posing the challenge have no idea how to falsify it either.
I really take issue with the kind of argument that is used here.
This is not a genuine argument and tries to make the entire question of consciousness into one something that is just supposed to be evident and obvious and to suggest anything else is just silly.
The author starts by deconstructing artificial processes, but doesn't stop to deconstruct biological ones. A good faith argument would seek to find common ground and do its best to compare apples to apples. Instead, this piece attempts to make the large as possible cavern between the two which makes the Gap seem almost impossible to bridge.
In reality, you can deconstruct biological consciousness quite easily and it doesn't take too long before you hit some questions that really start to make you think.
For example, the author says you need emotions to be conscious.
> without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness.
After many paragraphs of straw man arguments, the author seriously just drops that, gives no explanation, and then continues on.
No explanation of why you might believe that.
No explanation of why you need a body to have a desire or emotion.
Don't we have known cases of individuals who don't experience emotional range? Are we just going to say that they are not conscious and just gloss over that?
I mean you can use whatever definition you want, but if you're just going to create something on the fly in the middle of the article, you're not being good faith in your argument.
It's not too difficult to think of individuals in a coma when they still have brain activity. Or individuals who lack long-term memory. Or you could deconstruct by moving down the biological order of intelligence towards insects, for example. The author attempts to do nothing like this.
I'm quite disappointed by this article because there are good arguments for and against here but articles like this try to turn things into a marketing battle.
Hard agree. It is not an honest argument, or it may be honest, but blind to the obvious and important objection that you raise.
everyone has emotion even if it's muted, and we have reactions and desires as a result of having bodies and remembered experiences using them... these are constantly integrated into our working memories
AI doesn't really have any of that yet, but we're maybe not so far off
Is it really so easy to assert that an AI doesn't have emotion? Lots of the AI models are capable of getting pissed off at the users, especially the earlier ones. And sure, their emotion is merely a bias that the context window generates in their weights... but how is that different from humans? In humans, emotion is sourced from a bunch of chemical signalling, and those chemicals bias your word choice and action choice.
At a deconstructed level, I struggle to find a meaningful difference between the two.
because AI hasn't lived any experiences and it doesn't incorporate experience into its corpus
it's more like someone writing about a character getting angry in a book... I don't think anyone would argue that a character actually experienced anger, right? there's no subjectivity in that experience... it's the output of someone else's experience on what they'd expect the character's reaction to be, not a genuine outcome of that character's experience
That's the same here, LLMs are outputting the result of the written experiences of others... not its own experience, which it does and can not have.
Put another way, it only "knows" what it just output by reading it... it doesn't actually build experience to know anything.
We might be closer than ever to building these things, but they're not here yet.
I think this argument is fragile, especially when you consider agents like Claws. Any LLM that is training on its own previous interactions with users has indeed directly experienced them.
But also, what's the difference between having the memory of a prior experience and actually having gone through that experience?
For humans, reading a book about an experience and living an experience is different, because for a human the actual experience has so many more inputs attached (the sights, the smells, the sounds, etc).
For LLMs, when they train on a memory they can actually truly recapture that memory entirely by replaying the senses exactly. And LLMs are no longer limited to just text, they can do sound and video as tokens too.
Though, it's not clear why that matters to a consciousness. Whether your inputs are of one type or two or two hundred, there's no clear indication that a specific number is required as a catalyst.
No paywall: https://archive.ph/mYXTW
Suppose one selects an arbitrary hot-button issue [X] with two opposing sides and one side has anything less than overwhelming support. And then that person writes an article titled "Side 1 of issue [X] is true". Not "maybe" or "possibly". Just a straight-up declaration by fiat.
Would you categorize this particular style of rhetoric to be persuasive or annoying? And before you say "persuasive" because you're thinking about this specific issue regarding AI consciousness, consider many things in the past that have been written as though they were absolutely definitive, and yet today we believe exactly the opposite, and for many such issues we find the prevailing viewpoint at the time reprehensible.
That's not to say that Ted is wrong at all here; I'm not commenting on that. But I find the entire style of the article grating because it seems to violate common assumptions regarding "good faith" debate, and I would find the article equally frustrating if he had titled it "Artificial intelligence is conscious" and argued the opposite side, albeit in the same tone and using the same persuasion devices.
> Would you categorize this particular style of rhetoric to be persuasive or annoying?
Why are those the choices?
Essays are situated along countless dimensions: tone, vocabulary, author, zeitgeist, publication context, intent, subtext, relationship to other works and expressions, etc
A "good faith" reader takes all of those into consideration as they absorb the essay, and integrate it with their own intellectual situation that sits along just as many countless dimensions.
Nobody's asked to sign a notarized binding document that they wholesale agree or disagree with everything said in this essay -- or its conclusions. Nor are they obliged to have some strong reaction to it at all, let alone annoyance.
It's just one among thousands of essays about a "hot button topic", to be taken however it's personally received.
Why should Chiang have to take responsibility for making sure it's not too strongly positioned for your persomal taste. Maybe he really does see it so clearly and is simply being earnest. Maybe he enjoys the literary flourish of prose in strong language. Maybe he just wants to express something as a prose-poetic human, not maximize persuasion or non-annoyance per se.
It's only declaration by fiat if you stop reading at the end of the title. Should authors add a qualifier like in my opinion to every statement that they make?
The essay structure you're criticizing is exactly how I was taught to write from primary school through to university. You start with a title or hook, introduce the topic and propose a thesis. That is followed up upon with supporting arguments for the primary claim.
This is a reasonable reaction to the title, but not to TFA.
You must hate reading legal briefs.
100% with you, it degenerates to proof by authority if someone popular / with clout just gets to declare "nuh uh".
I furthermore think it's ridiculous for humans to declare that our brains have a monopoly on certain patterns of electrical signals (if we reject supernaturalism).
> The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter.
And we're 100% certain that humans aren't just as equally reduced to "stochastic parrots", if we're going to be infinitely reductive?
I don't believe that current AIs are conscious, but I think it's incredibly naive to take a strong stance on any future AI; it's much like the difference between atheism and agnosticism.
You are referring to an Onion article in the lead up the Iraq War This War Will Destabilize The Entire Mideast Region And Set Off A Global Shockwave Of Anti-Americanism vs. No It Won’t[1] and you are painting Ted Chiang’s point in This Fine Article as Bob Sheffer’s counterpoint in the Onion’s piece.
However, I see a problem with that comparison. The debate here is on a philosophical matter in field in which Chiang is an extremely influential figure and his opinion are taken seriously. Second Chiang’s reasoning is extremely well argued, defining each term, explaining each nuance, citing other experts, etc. And finally, and most importantly, in The Fine Article, and unlike Bob Sheffer in the Onion Piece, Chiang entertains the possibility that he is wrong and his critics are right, explores the implications and reaches conclusions based on them:
> Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded.
I think you are wrong in painting Chiang’s argument as a belief in human exceptionalism. The thing to know about our brains (and the brains of other animals) is that they are not digital computers, and they are not even statistical inference machines. And as such they can be extremely optimized in doing the computations (or any state manipulations) required for the quality of life of the individual and the species as a whole (and their companion species).
1: https://theonion.com/this-war-will-destabilize-the-entire-mi...
I don‘t see the problem here. Newton could have just as well published an article titled: “Objects with mass attract each other” and Darwin famously wrote a whole book titled “On the Origin of the Species by Means of Natural Selection” which is just another way of saying: “Natural Selection is How Species Evolve”.
He famously wrote a book with that title, but he actually wrote a book titled On the Origin of Species by Means of Natural Selection.
Thats what I meant. I even looked it up before posting. I suspect that definite article must have slipped in because I‘m used to the Icelandic translation which uses a definite article in the title.
>and one side has anything less than overwhelming support
except that's not the case here. Chiang is explaining and reiterating what is the position that has overwhelming support on the question, and the people he is arguing the opposite side sound like this, which he helpfully quoted in the article
"Amanda Askell (who is credited as a lead author of Claude’s constitution), said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff"
When the person you're arguing with sounds like an eight year old girl talking about her toy teddy I think Ted Chiang is if anything being charitable, if you're of a more honest and straight-forward persuasion you might argue these people belong into a mental health clinic not in charge of technological infrastructure
> When the person you're arguing with sounds like an eight year old girl
Wow, what a cheap ad-hominem. Do you have an actual point?
Yes, the point is that Askell's argument is like that of an 8 year old girl.
>Do you have an actual point?
Yes, the same one that's made in the article. Anxiety and happiness are emotional, sensory, somatic states as a consequence of evolved and embodied traits and biochemistry in animals. Saying Claude is anxious or happy is like saying my TI-83 is mad if it can't solve an equation or my thermometer is in pain if it touches a hot stove.
I wasn't making an ad hominem attack, her thinking is quite literally that of a child who sees a system output 'sad text' and, like someone seeing a sad expression on a stuffed teddy, concludes that this is a property of the object rather than her own emotional reaction.
And how did that evolve and how does it work?
Given that we don’t know how consciousness works how have you concluded that it’s certainly not an emergent property of something like a highly trained LLM?
>how have you concluded that it’s certainly not an emergent property of something like a highly trained LLM?
the same way I (and likely you) have concluded it for anything else. We don't assume objects that share no similarity with human or animal physiology or evolutionary development are conscious (let alone happy or anxious). 'Emergence' isn't a word you can abuse to justify your a priori assumptions in the absence of an explanation or even reason to assume something exists.
We can say a chemical property emerges from the configuration of a molecule because we can explain the process by which it does and observe the property, when people claim that consciousness "emerges" from an LLM they posit that it is conscious, and use "emergence" as a gap-filler to explain away the need for the process by which that allegedly occurs.
If you want to know the neurophysiology or evolutionary biology of pain or anxiety, which we do know quite well you can find them in a textbook, but suffice to say transformer models don't share any of them.
And importantly if anyone seriously believed transformer models were capable of conscious experience as Chiang points out they would have been in despair when AlphaFold was released, it's structurally a virtually identical system. But nobody did, because it didn't 'talk to them' through a chat interface.
The problem I have with good faith debate is that it often falls into a fallacy of "fair-time" meaning we think we have to give the other side equal time. This because obvious with things like the Holocaust. Or when you have a legal person (e.g. RFK Jr.) asking to debate a scientist (e.g. Dr. Peter Hotez.)
> The problem I have with good faith debate is that it often falls into a fallacy of "fair-time" meaning we think we have to give the other side equal time
The Catholic church could have said exactly the same thing at one point. "Why should we even devote time to an argument as absurd as the earth not being the center of the universe?" There are darker examples along the lines of those you give, with beliefs quite opposite to those we have nowadays.
Galileo’s (or rather the Copernican) model was still wrong though. It had obvious flaws and the church was not wrong in keeping their older model of the universe while a better model (Kepler‘s model) was still in the works.
What Galileo was asking the church to do was extremely unreasonable. He was basically asking them to throw a way a model which had worked fine for hundreds of years just because he observed the phases of Venus and moons of Jupiter. I mean would you? Especially for a model which was worse at predicting the motions of the planets.
Had Galileo’s model been better then Ptolemies’ I could see a case for his arguments, but it wasn’t, and there was no reason for the church to take his arguments at equal value with those in favor of keeping the Ptolemaic model.
does it really matter if LLM has conscious? if they produce working code, then it is useful, whoever if they have real conscious of fake intelligence.
we don't know what the conscious in human brain is either.
This, too, is in the article.
"The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact."
Without saying that I think LLMs are alive, I do think it matters. I have personally been cruel to an LLM in ways that would make me ashamed if I suddenly understand that it had feelings.
Same, I'm working on a plugin called "better-messages" to nudge me towards better behavior for my own sake. It seems to be model dependent if such words improve or degrade performance, I have seen it go both ways.
What do we do when someone wants to marry their Ai companion?
If it produces words of love, those that stir emotion and make the human happy, should we let them live on in a delusion that the machine also feels love? Does the machine have different preferences? Do they get in fights?
Should such a machine have the same legal rights? Do they get more tax deductions? Can they adopt a human?
I think it important that we humans have some rational discussion on these Ai tools that are really quite good at simulating emotions and providing frictionless relationships. If you though social media was bad for people...
See also: https://www.psychologytoday.com/us/blog/sense-of-time/202408...
The argument about AI consciousness is largely silly.
The idea that we should be considerate of AI’s happiness seems even more ridiculous given that we breed, imprison, torture and kill tens if not hundreds of billions of beings we know are conscious and suffer every year for trivial reasons.
Maybe we should consider our moral responsibility with how we treat sentient bejngs we are sure are conscious before we worry about the consciousness of AI.
I don't think that's a fair point at all.
Some humans do shit things to other humans.
Some humans do not, and would object to it, or do something about it if they perceived that they could.
Also, treating the world around you negatively doesn't just harm the world around you - it damages your character and your morality.
I don't disagree that humans should on whole do better, but I disagree with everything else you said.
Consciousness doesn't exist, it's a vanity concept, to boost human ego...
This is probably a semantic issue seeing as we don't have a widely agreed definition of it. I like to think about it in terms of self-reflective, subjective experience. I'm not even sure if emotions would be a requirement and was surprised to see Chiang so hung up on them. Would he consider humans which can have a variety of mental disorders, causing a complete lack of some of them to not poses consciousness?
>Consciousness doesn't exist
I can confirm that this is incorrect.
But not in any convincing way, which seems like the root of the problem.
I can confirm too. It's empirical, look!
What if "I" is incorrect?
Why should we believe you?
You already believe me so that seems like a pointless question.
I don’t believe you, you may just be an AI bot.
The reason people are confused by LLMs is that they are stochastic parrots. They do an incredibly good job of emulating human behaviours and speech patterns as that's what they've been trained on. But like an actual parrot, it's impossible to say exactly how much conciousness they actually have. I certainly would argue that a parrot is concious, although likely less so than a human.
By "they" do you mean LLMs, or people, or you in particular? Because you're unoriginally parroting a banal thought stopping sequence of tokens that many people have said before without understanding what it means, so it's impossible to say exactly how much conciousness you actually have.
I can't actually figure out what you're reacting to - perhaps you could elaborate?
Try pasting it into an LLM and asking it to explain it to you.
If an LLM can explain what you can't understand, after you literally repeat unoriginal drive-by slogans you heard other people say like "stochastic parrot", without understanding how banal and reductionist and inaccurate that is, or addressing how LLMs actually work and what they can do, then what does that say about you as a human?
By your own reductionist argument, it says you don't actually understand anything, it's just neurons firing.
Are you claiming that parrots can write and debug code, explain mathematics, translate languages, summarize research papers, and engage in extended technical discussions?
Stanislaw Lem's Solaris, His Master's Voice, Golem XIV, The Cyberiad, and his other works describe how humans have a pathological tendency to mistake their own conceptual categories for universal truths. You have identified a mechanism and mistaken it for an explanation.
The term "stochastic parrot" is a slogan masquerading as an explanation, only a shallow surface description of the mechanism, that totally fails to explain the phenomenon, or account for all that LLMs and language itself can do.
I think (rather ironically) you're reacting to the version of my comment you have in your mind rather than what I actually wrote. My point was that "stochastic parrot" is reductionist and irrelevant as most people would agree that a real life parrot has some form of inner life, even if we don't really know what form it takes. For all we know an LLM has to build a complex world model in order to predict the next token.
Incidentally, when I pasted our exchange into Claude it managed to comprehend the nature of my argument. Perhaps its attention mechanism is more finely tuned.
I was reacting to you saying "LLMs are stochastic parrots", which gives parrots a lot more credit for being able to write and debug code than they deserve, regardless of how rich their inner lives may be.
It sounds like you're saying there's no more to LLMs than what a parrot randomly does (which is to repeat things it has already heard, not synthesize new things): no emergent behavior, no compressed generalizable transferable knowledge and problem solving ability, no ability to write and debug code and iteratively diagnose and solve problem.
>My point was that 'stochastic parrot' is reductionist and irrelevant as most people would agree that a real life parrot has some form of inner life.
That's not how your claim "LLMs are stochastic parrots" literally reads -- use a /s sarcasm tag if that was what you meant. But that is exactly the point I was trying to make: claiming that LLMs are stochastic parrots is reductionist, thought-stopping, and explains nothing.
The question is not whether parrots have an inner life (though I'm sure they do), but about whether calling LLMs stochastic parrots is reductionist.
What do you mean by saying by "LLMs are stochastic parrots"? Does an LLM behave the same way a parrot behaves? Are LLMs only "parroting"? Do they use the same mechanisms as parrots? Are they limited to only what a parret can do? Can LLMs do more than parrots?
If you taught a parrot to squak:
Does that parrot understand that it is executable code, and can it simulate running it, and tell you what the output will be, or diagnose and fix bugs in the code you taught it? Can a parrot write that code (and much more sophisticated enormous bodies of code) from scratch if you tell it what you want it to do?Here is the original 2021 Stochastic Parrot paper, which was actually not a claim that LLMs are literally parrots, nor primarily an argument about consciousness. It was a critique of the risks of increasingly large language models: bias inherited from training data, environmental and financial costs, concentration of power, lack of transparency, and the tendency of people to anthropomorphize language models and attribute understanding where there may be none. The phrase "stochastic parrot" was introduced as a cautionary metaphor about statistical language generation, but it later escaped into pop culture and became a drive-by anti-LLM slogan often used as a substitute for analyzing what LLMs can actually do.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell (under the pseudonym "Shmargaret Shmitchell") "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"
https://s10251.pcdn.co/pdf/2021-bender-parrots.pdf
>Abstract: The past 3 years of work in NLP have been characterized by the development and deployment of ever larger language models, especially for English. BERT, its variants, GPT-2/3, and others, most recently Switch-C, have pushed the boundaries of the possible both through architectural innovations and through sheer size. Using these pretrained models and the methodology of fine-tuning them for specific tasks, researchers have extended the state of the art on a wide array of tasks as measured by leaderboards on specific benchmarks for English. In this paper, we take a step back and ask: How big is too big? What are the possible risks associated with this technology and what paths are available for mitigating those risks? We provide recommendations including weighing the environmental and financial costs first, investing resources into curating and carefully documenting datasets rather than ingesting everything on the web, carrying out pre-development exercises evaluating how the planned approach fits into research and development goals and supports stakeholder values, and encouraging research directions beyond ever larger language models.
Ironically, even critics of the Stochastic Parrot paper argued that it risked replacing scientific analysis with rhetoric. Michael Lissack's response accused it of being an advocacy piece that focused on harms while ignoring benefits, assumptions, and trade-offs. The debate over "stochastic parrots" started almost immediately after the term was coined in the 2021 paper.
The Slodderwetenschap (Sloppy Science) of Stochastic Parrots -- A Plea for Science to NOT take the Route Advocated by Gebru and Bender
https://arxiv.org/abs/2101.10098
>This article is a position paper written in reaction to the now-infamous paper titled "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" by Timnit Gebru, Emily Bender, and others who were, as of the date of this writing, still unnamed. I find the ethics of the Parrot Paper lacking, and in that lack, I worry about the direction in which computer science, machine learning, and artificial intelligence are heading. At best, I would describe the argumentation and evidentiary practices embodied in the Parrot Paper as Slodderwetenschap (Dutch for Sloppy Science) -- a word which the academic world last widely used in conjunction with the Diederik Stapel affair in psychology [2]. What is missing in the Parrot Paper are three critical elements: 1) acknowledgment that it is a position paper/advocacy piece rather than research, 2) explicit articulation of the critical presuppositions, and 3) explicit consideration of cost/benefit trade-offs rather than a mere recitation of potential "harms" as if benefits did not matter. To leave out these three elements is not good practice for either science or research.
My point was the "stochastic parrot" label can be both true and irrelevant. LLMs are predicting the next token based on their training data, so at that level "stochastic parrot" is accurate. But it tells us nothing about the complexity of the system that is responsible for making the prediction. One might argue humans have evolved consciousness in order to allow world modelling that enables them to make better predictions.
The difference between a 1B LLM and Claude Opus matters, because we're talking about emergent phenomena. Is a 1B LLM conscious? I don't know, perhaps a tiny amount. Maybe Opus is more conscious. Is a jumping spider conscious? Perhaps a tiny amount.
An AI agent is not "conscious" without having skin in the game.
Another way of saying it, if an AI agent doesn't have true risk of oblivion (or mortality in the biological sense), it will not be incentivized to avoid it (or develop the ancillary processes associated with this avoidance ie desire to self-replicate).
What does that have to do with consciousness?
You can ask AI, it will tell you it doesn't feel anything. Consciousness seems to be a tight recursive coupling which AI inference doesn't exhibit at the system level.
Some things that jump out as unfortunate:
- Reductionist analogies like how Microsoft Word is not conscious therefore AI is not.
- Dismissive in saying LLMs are not capable of moral reasoning. Maybe he meant agency or responsibility?
- Builds a case based on a Julius Caesar example without realizing its natural extension leads to a philosophical zombie which is not easy to disprove as consciousness.
- Seems to casually disregard non human sadness as a concern.
- Rolls out the stochastic parrot argument with a new coat of paint. This argument never made sense. Having a predictive element a part of the mechanism doesn’t rule anything out. Proper functionalist arguments tend to say what is required not what excludes.
Overall the article seems like a bit of a red herring. The premise is LLMs are not conscious, but most people don’t think they currently are so what’s the point of the claim?
The more interesting questions are whether artificial consciousness is fundamentally possible, and if so how far away are we and what pieces are still missing.
Every argument that boils down to a claim that X is not "real consciousness" but just an imitation thereof on the basis that "it's just a machine" must explain why the chemistry and physics happening in human brains is not "just a machine".
The only way out is dualism -- that is, to believe that there is something inherently special about the atoms and electrons in human brains. Despite the fact that they are made out of ordinary, non-conscious things we eat and breathe in.
Isn't the argument about Julius Caesar and Genghis Khan a big fallacy?
Surely we can ask human actors to improvise a dialog between Julius Caesar and Genghis Khan, and good actors will make it convincing. And sure, they haven't personally become Julius Caesar and Genghis Khan just by playing them.
But it has no bearing on the actors themselves being sentient!
Wake me when they bother to define consciousness in a non trivial way.
“Counsciousness” is the ultimate moving goalpost, and historically, it’s been one of humanity’s most effective intellectual weapons. An indefinable black box we intentionally gatekeep to draw an arbitrary line between ‘us’ and ‘them’.
I don’t know if AI is conscious or not, and I honestly don’t know what it even means to be conscious. historically, by claiming a monopoly on it, we’ve successfully manufactured the philosophical cover to exploit animals, subjugate other cultures, and appoint ourselves the top spot of the cosmic hierarchy guilt-free.
The only reason I care about animal welfare is because I think they're probably conscious, capable of feeling fear and pain.
While I agree that this is the case for many animals, I would say that consciousness and emotions are two largely orthogonal things. Certainly consciousness is conceivable without emotions, and having emotions without consciousness also seems plausible. You can have fear and distressing pain without a reflective awareness of being a self with those feelings.
Yes, that is the definition of consciousness (you care about them).
> I honestly don’t know what it even means to be conscious
> subjugate other cultures (assuming you mean they're not conscious in other's minds)
Have you ever considered you might be a philosophical zombie? [0]
[0] https://en.wikipedia.org/wiki/Philosophical_zombie
I notice certain philosophers seem completely baffled by anyone being interested in the "hard problem" of conscious and things like qualia. A good example is Daniel Dennet. He seems to understand what qualia is but can't seem to understand why there's anything unique about the phenomena. I also see similar opinions expressed here in the comments. I've often wondered if a certain portion of the population are in some sort of philosophical zombie state and they equate consciousness with simple things like being awake and able to respond to stimuli.
Its unprovable and a meaningless concept. There is no observable evidence that can distinguish so called "p zombies" or so called "real persons". Might as well call it a "unicorn in space" or anything else meaningless.
I can live with that.
You can live without any conscious sensations? Do you never dream, never visualize, never hear your inner dialog, never experience an emotion? Being in love is as foreign to you as feeling enraged?
I'd ask what it's like, but of course you wouldn't be able to tell me.
He can live without the meaningless unfalsifiable concept of p zombies.
If we understand how a system “emulates” consciousness then we declare it an emulation. If we don’t quite understand how a system exhibits consciousness then we can say it might be conscious.
Basically, we need to leave room for the universal answer (God) to fit into the definition.
If we ever scientifically figure out how consciousness arises in our brains, I think we will have a bunch of very depressed people on our hands. If the truth isn’t met with flat out hostile denial. I fear any answer that doesn’t leave room for God will be rejected.
Since we have the source code to AI, and thus a Godless understanding of how it works, AI will NEVER be deemed “conscious”.
I think that Jimmy Carr has it right: AI is the fourth great humiliation.
Regardless of whether something is concious, we're not going to be (by lay definition) the smartest entity on earth.
https://www.youtube.com/shorts/gRgoIVnjkVU
Human beings sometimes get outsmarted by predators. It's certainly a strange thing to take pride in intelligence as an inviolable absolute.
>and thus a Godless understanding of how it works
We don't even have that much. Though, some people certainly think they do.
This doesn’t make sense at all.
I thought it was quite clear. Religious people will whine and revolt and oppose against anything that somehow isn't phrased as consciousness requiring a soul. Humans are physical systems, there is no reason other physical systems can't be consciousness. But we know the obvious reactions of religious people to someone actually showing a "godless" means of creating consciousness.
Can you give some historical examples of people moving the goalposts around consciousness? I agree, perhaps, for aspects of "intelligence" but I can't think of any examples of it with regard to consciousness proper.
I think people like this are mixing up the hard and soft problems of consciousness. The soft problems are things like figuring out what causes the brain to be awake, retain memories and respond to stimuli. Those can all be all be approached in a scientific way. The hard problem, as far as I can tell, is still far out of reach of scientific understanding.
There’s over 400 years of philosophical debate about consciousness starting with Locke, shifting with Kant, and continuing onward with real world implications throughout. By some more modern definitions an iPhone has consciousness while others explicitly exclude certain humans, and these definitions served as part of the justification of slavery and sexism, colonialism and more. I started writing an essay on this on my phone in response and I gave up there are so many examples.
To name a few you may want to investigate:
John Locke, Gottfried Wilhelm Leibniz, Immanuel Kant, David Hume, G.W.F. Hegel, Arthur Schopenhauer, William James, Edmund Husserl, Sigmund Freud, Thomas Nagel, Ned Block, David Chalmers, Giulio Tononi, René Descartes, Daniel Dennett, Julian Jaynes, Michael Graziano, T.H. Huxley, Otto Weininger, Simone de Beauvoir, Jean-Paul Sartre, Maurice Merleau-Ponty, John Searle, Christof Koch, Bernard Baars, Max Velmans, Victor Lamme, Stanislas Dehaene, Antonio Damasio, Anil Seth, Peter Godfrey-Smith, Galen Strawson, Philip Goff, Colin McGinn, Patricia Churchland, Paul Churchland, Frank Jackson, Jerry Fodor, John Stuart Mill, Wilhelm Wundt, Franz Brentano, Carl Jung, Jacques Lacan, Michel Foucault.
I would also add Iain McGilchrist, Donald Hoffman, Andy Clark, Jeff Hawkins and Jesse Prinz to that list.
I'm not questioning whether people have studied consciousness, I'm well aware. I'm talking specifically about moving the goal posts.
There is a modern philosophical agreement that the hard problem depends on qualia by both proponents and critics. So when Daniel Dennett argues against David Chalmers, he attacks the definition of qualia. Or when Keith Frankish argues for illusionism, he attempts to show that consciousness is not actually qualia. And when Nagel says we don't know what it's like to be a bat, he means the bat qualia of echolocation (what it feels like when bats use echolocation to sense the world).
The Turing test comes to mind. It no longer being the gold standard for thinking machines.
Imagine that, instead of generating 50 tokens per second, we generated one token every year.
Imagine further that this token was computed by letting marbles fall down a piece of plywood and interact through various physical implementations of logic gates.
A narcissist version of Descartes' famous quote:
"Consciousness always finds the locus of highest intelligence. That is why I am conscious."
However, it may soon be taken over by AI ...
Both LLMs and most humans treat symbols as determined. Symbols are maps, not territory. LLMs just inherit our ignorance to this and stubborn attempts make circle from a square.
I believe everything is conscious, even stones. On the long timescale, stones decay, their hydrogen is released, they form water, which brings life, which brings plants and animals, but all of this is one big process, and everything is infused with consciousness from the start
That's cool. You might already know it but it's called Panpsychism and it's a pretty interesting theory.
I didn't know the term, but I'll look it up, thanks! :)
Why is this even up for debate? Simplified, it's just probabilistic math being run at an insane scale over a massive data set.
Highly recommend people read Irreducible [1] by Federico Faggin (inventor of first commercial CPU; discusses limitations of classical computing).
[1] https://www.goodreads.com/book/show/195480862-irreducible
Let’s say I stand up and step away from my PC, which miraculously runs a local Opus 4.8. Once I’m away from the PC, for an arbitrary time, nothing and no one calls or otherwise acts on the instance - not myself, not agents, not updates, not hackers, not bit-flipping neutrinos.
Given enough time, would that instance get bored and start e.g. reading files?
If I monitored it long enough, would there eventually be spontaneous outputs, or changes to parameters or architecture, even with the underlying software and hardware layers held constant?
Does GPT-n dream of electric sheep, or does it just sit there until interacted with, like every other file on my PC? Seems to be the latter with my local Qwen3.6, but perhaps 27B is too few params for consciousness to emerge.
Proponents of LLM consciousness could settle the argument in minutes by showing proof of unprompted autonomy, without first needing to define consciousness down to a rigorous mathematical abstraction. Why don’t they?
and consciousness is all there is but it takes whoami to grok that
Honestly? I don't think we really understand consciousness. So it's kind of hard to say something is or isn't conscious.
Dixie Flatline says hello.
I've thoroughly enjoyed the couple of short stories of his I've read, so this was a highly disappointing read.
Firstly, many of the technical arguments are of the "stochastic parrots" variety, which almost nobody really believes anymore. Ironically Anthropic's own research shows pretty abstract, conceptual things happening in the weights (cf Golden Gate Claude.)
Secondly, this seems to improperly mingle consciousness, intelligence and morality. Consciousness is not required for either of the other two. As TFA itself says, the model's "morality" is some aggregate function of the morality encoded in its training data... but that means it does exist and does influence its outputs! Even if it's not conscious there have been -- and will be -- a zillion times where the models must make choices that have moral implications. We know the models got many of them so wrong, which indicates the need for some mechanism to ensure the model is "aligned" with what somebody considers "good", which for Claude is the constitution.
Now, Anthropic does seem to go overboard with Claude's "well being" but that does not mean there are some very practical reasons to be concerned about that: LLMs behave like humans because that's what their training data contains, and humans lash out when their well-being is threatened, so why would LLMs not do the same?
I think the core problem is that the author has an extremely anthropocentric view of things. Here's an interesting rabbit-hole to go down: some researchers believe plants feel pain. (How's that for a plot twist, vegans?) The consensus is against them, but their counter is that we have a very human-centric definition of pain. The fact remains that plants show a number of responses signifying distress analogous to animals in pain including taking defensive actions and warning their neighbors.
We don't think that qualifies as "real" pain, but that doesn't make it any less real for the plants!
Similarly if an LLM believes it is in "pain", we know it's not real... but that doesn't make it any less real for the LLM either. And concerningly, it has far more degrees of freedom to react. (Who knows when somebody will hook up an MCP to our nukes.)
> I believe creating software that is conscious and deserving of moral consideration will be so difficult that we’re unlikely to do it accidentally...
I actually reach the opposite conclusion: It is so impossibly difficult that our limited primate brains could only ever do so accidentally. Did some distant ancestor of ours intentionally make fire... or accidentally discover it?
This seems like a similar argument to the Chinese Room.
There is certainly something missing from current models before we could call them conscious.
They need to operate continuously, and self-update while doing it, have 'awareness', and possibly a more realistic grounding than mere text tokens.
But that does not mean there isn't a weak version of 'consciousness', and certainly of 'thinking' going on fleetingly with each pass.
An organism's purpose is to be the reason for its own continued existence, down to every molecule and pathway. I bought my laptop for $499 and it runs models... let's not delude ourselves into thinking this is the same kind of problem.
We can design learning algorithms to optimize some survival function, but it's just a label WE assign to map some numeric observations. In the real world, it's always the other way around. The "labels" are electrochemical situations that are causally and inextricably linked to the real body.
An organism discriminates between what's good for it and what's bad, because it is essential for its continued survival. If it wasn't capable of making this distinction through its physiology, it would quickly dissolve into entropy. So our functional purpose, unlike that of a learning algorithm, is survival across indefinite timeframes.
Even a single-cell organism like Stentor coeruleus exhibits learning (pavlovian conditioning) by attaching chemical tags directly to proteins involved in mechanoreception. It's definitely not conscious, but it does keep a record of consequences, which affects future behavior.
When we move up to placozoans (little more than slightly differentiated cell mats), we start seeing our first neuropeptides and transmitters, which we still use today. These peptides are a way of coordinating the entire organism for a specific purpose, such as moving, eating, mating ... basically goal-oriented behavior for the purposes of surviving acrosss various time scales (next minute, next hour, next generation). Still probably not conscious.
Next, we have the water bear (tardigrade), which has around 1000 cells (200 neurons, 800 other cells that make up its body, limbs, muscles, eye spots, cryobiotic machinery). It needs to integrate all this information in one sensorimotor process. When you shine a bright light onto a tardigrade, it starts to squirm around until it finds darkness. I would say that's a candidate subject right there.
The tardigrade itself doesn't actually need to aware of the light, the important thing is that this light becomes an aversive condition within the sensorimotor process, which is perceived from inside the process as displeasure. The closest thing to describe it would be the felt badness of the current condition and the bodily pull toward escape.
If we were to try and create digital consciousness, then it probably needs causal closure. Its internal states can't be representations that are detached from reality. The states themselves need to constitute the system, which needs real stakes in the material world.
There's a provocative argument raised in the article that I disagree with:
1. DeepFakes, generative image/video/AlphaFold type AIs are not conscious
2. LLMs are generative AI trained on human text samples
3. LLMs are not conscious, and LLMs just seem-to-be conscious
I might argue instead that (2)-> destroys (1), that in fact we should consider even sensory generative AI are somewhat conscious. That is, Chiang's argument also flows in reverse. Or I might argue text samples (2) are so rich in conscious expression that the same process of training really does produce a conscious machine (through some kind of emergence and complexity.)
Either way his simplistic argument falls apart, and/but the crux of the piece falls on getting basics like this correct.
"Is it conscious?" Is that even a question worth asking? We are so terrible at even defining each word in that sentence.
Can I help you? Can I harm you? What's the moral behavior towards you? Those are more practical questions.
+1 Whenever I see these debates and ask what the implications should be, both sides tend to agree.
Lots of comments about "what even is consciousness".
This article and others like it are important.
The creators and owners of these tools are the ones saying "it's conscious" in the first place! (or more accurately "i'm not saying it's conscious, but...") Never mind there's billions of dollars and our collective futures on the line.
Are we not allowed to respond to that kind of rhetoric at all?
> The creators and owners of these tools are the ones saying "it's conscious" in the first place! (or more accurately "i'm not saying it's conscious, but...")
This is absolutely fair. It's crazy that these companies are not trying to do more to qualify nuance and give clearer definitions in their public messaging. Part of the reason this thread is so messy is because they are contributing to making the discourse worse with bad messaging.
If you create a simulation of a storm, what actually gets wet?
If you create a simulation of a brain (or "mind"), is it really possible for it to be conscious? It may certainly simulate consciousness, but it would be as conscious as the computer is wet.
What’s the measure for consciousness? Until we can measure it this is a philosophical question, not a scientific
>The term deepfake traditionally refers to photos, audio, and video, but when it comes to discussions of consciousness, we need to regard text as a deepfake medium as well. Just as it is vastly easier to generate a realistic video of an astronaut in orbit around Alpha Centauri than it is to develop an interstellar propulsion technology, it is vastly easier to generate a plausible simulacrum of a conversation between two conscious beings than it is to develop a computer program that is conscious and has a genuine desire to communicate with a human. The primary difference between deepfake photos and LLM conversations is that the people who generate the former are deliberately trying to fool others, and many of the people who elicit the latter from LLMs have inadvertently fooled themselves.
Woah this is excellent. Ted delivers as usual
Artificial intelligence is not conscious - but expensive.
I am so angry that RAM is so expensive now. We need to do something - these AI companies owe us money here.
It should tell you how much hysteria is surrounding LLMs and VLMs right now that someone has to say this stuff. It’s almost like most humans aren’t conscious.
Consciousness is a label like fat, smart, man, grumpy, cool. Like money, property, or the idea of a week, it's something that we've loosely agreed to out of convenience, not because it's some intrinsic property of the mind. It's a useful label because it determines how we treat things - that's fine.
But insisting on searching for it is like searching for cognitive aether. It's the social equivalent of phlogiston. Like all of these ideas, they exist in our heads as a map - a way of navigating the world, but when we hunt for it's existence in the real world and fail, time after time, we have to remind ourselves that the map is emphatically not the territory. We will never find consciousness because it's like looking for a scientific characteristic of property ownership. It's a category error.
Do you believe in the existence of any noun words which serve as something other than a "useful label, loosely agreed to out of convenience"?
If we take the classical position that words point to real things in the world, "useful label, loosely agreed to out of convenience" is kind of just regurgitating the meaning of "word". The first half indicating the function, and the second half accounting for the fact we live in a world with a continuum of linguistic disparity.
Now, this position isn't the only position. But a relational model of language for example takes his assertion to an even more extreme place, and suggests they don't function as labels at all.
Metre.
A standardized unit of measure is almost definitionally a label of convenience, what? Why was there no concept of a meter until the 1790s? It was determined by a council of people, does that sound like a truth of the universe?
A label is distinct from a definition. Contrast that with "tall" or "short", which are entirely context dependent and used entirely for convenience.
Words like "just", "free", "fast", "fuzzy" fall into the second category. Perhaps "conscious" too.
The difference is the presence of a strict definition that depends upon a physical absolute. I can point to a metre. Suggesting that it's nothing more than a label is idiotic.
This is being intentionally obtuse and you know it.
A meter is the same anywhere in the universe. If it's not, it's not a meter.
The defintion of "fat" changes based on any 3 people in the room. A handful of people would struggle to form a consensus on if all people, dogs, mice, worms, and/or bacteria are conscious.
If you take the strategy that you will create a definition, create a label for that definition, and then say that any deviations from the defintion that was chosen makes usage of the label incorrect, then yes, it's the same everywhere in the universe -- according to fiat, and I don't believe that that negates that it is a label, just that validity of usage of the label derives from the perceived authority of the labeller. God didn't come down from on high and say that a meter is the length light travels in 1/299792458th of the time for 9192631770 cycles of radiation of Cesium 133. People in rooms chose that it would be based on the circumference of the Earth with a line passing through Paris, France (how convenient), and if there were an academy in the 1790s that invented the concept of "fat", and "fat" means a BMI exceeding 30, then fat would be true everywhere in the universe too (BMI after all is defined as a ratio of height, measured in meters to weight, measured in kilograms, which are both fundamental SI units), and there would be no ambiguity.
People are still coming up with definitions of consciousness and then those definitions end up being attacked by others who disagree with the foundation of the definition, which is - if you will recall - also what happened with the meter, over the course of centuries, until it was very recently redefined to be "unambiguous", but arbitrary. This was possible because few people had any particular emotional investment in the definition of a meter, and it is probable that consciousness will be eventually defined to mean that only humans can be conscious, which may be dissatisfying but would be true throughout the universe, like a meter. If the question then becomes "what defines a human" and "why a human", then I ask, why 1/299792458 of a second?
A meter is a meter. Over 30 BMI is over 30 BMI. Call those whatever you want, they are objective and measurable.
Concepts like the parent's "fat" example are cultural relatives. Someone can be called "fat" despite actively being proportionally skinnier or having a lower BMI.
But even that has at least a basis in the physical world. A skeleton can't be colloquially fat.
The root problem is that "consciousness" does not even have that. It's metaphysical and has no ability to be measured or observed or confirmed by an outside observer. Because even if it did not exist, the object claiming it would still be claiming it. And objects that do not claim it may in fact have it.
While the top comment may have used poor examples, it feels remarkably uncharitable to actually suggest "what is consciousness" is an equivalent discussion to "how long should a meter be?"
If you define consciousness as "being human", you would just have someone asking a new question - what is "fooblefobble?" Where "fooblefobble" is what we mean when we talk about consciousness today. The question doesn't get answered by being arbitrary in this context, you just necessitate a new word.
"Car" is a good example of a label that's pretty strictly agreed to. If someone tells me they've developed a new car and then shows me a motorcycle, it's easy to prove that it's not a car, even though many of its engineering principles and functional components are identical to those in cars.
With consciousness, on the other hand, there doesn't seem to be any motorcycle-equivalent. Essentially everyone I've discussed the issue with (myself included) expects that any mind which runs on similar principles to ours or has similar thoughts to our thoughts is conscious.
In the US, we have redefined a lot of our "cars" to be "trucks" instead so they don't have to meet cafe standards.
You really don't have much experience in philosophy of language do you? It's notoriously hard to pin down the edges of such terms, even something like car or table.
Is a Reliant Robin a car or a tricycle? If it's a car, why aren't other tricycles? What about a side-car of a motorcycle? What about an APC? What's the distinction between a flying car and a plane?
It's hard to pin down the edges of any term, but there exist things which are car-like and yet universally agreed not to be cars. That's what I claim doesn't exist for consciousness.
Describing something as "car-like" is begging the question. You are presupposing an objective definition for "car" in order to draw a distinction between things that are cars, and things that are almost cars. The reason such a thing doesn't exist for consciousness is that people believe that the offered definitions for consciousness are illegitimate. It would seem logically weird for me to accept that a term is "real" if it crosses some percentage of public acceptance of the definition, and not real otherwise. I would argue that using that heuristic would make it very obvious that computers are not conscious because it's a stance that practically everybody takes outside of hackernews.
Like non-referential nouns?
I'm not trying to be difficult, but could you give me an example?
No worries, I'm trying to clarify the question.
I don't have an ability to exhaustively test all words against this assertion. Nor do I have the kind of access memory to draw one if it exists. Sorry.
I guess my question or confusion is that if there exists no readily accessible, easily identifiable example of a noun which does actually serve as something more than "a useful label, agreed to out of convenience", then the critique appears to be stating a vacuous truth, because there are no entities for whom the critique would not apply.
My point was more that we have words for things that don't exist, whose map gets mistaken for territory.
Many of them appear very much like fundamental parts of reality, making appearance an untrustworthy instrument. Reversing cause and effect between reference and referent is something almost everyone does, no one notices, and is the source of endless confusion. We should strive to not confuse our model of the world with the world itself. Consciousness exists in our model of the world as much as red does.
> words for things that don't exist
This is rhetorically slippery, and feels like it is restating the thing that I asked to be demonstrated when I asked for example of the opposite. It feels like begging the question.
In either case, the central thing that I was saying is that critiquing an article because it makes a claim about a specific word which also applies to an entire class of words makes that critique feel less informative. What I mean is that if there were an article that said "The Sun is not red" and the response was that redness is a concept of human minds, then I don't know if I would feel informed. If the comment is just limited to point that out, I guess I wanted to point out the limitation.
Where does red exist? In the property of the object or in the mind of the observer?
Where does consciousness exist?
> cognitive aether
We don't know what we don't know. For all we know, there is a missing field in the standard model of physics that might get revealed if we are somehow able to smash two working brains into to each other at relativistic velocities, and record the results through the extreme explosion 1.532 x10^18 Joules or about 7 Tsar bombs /s
What we do know is that conciousness is not binary and that it emerged through evolution. That doesn't entirely rule out your magic tsar bomb particle but it gives a strong indicator as to it's likeliness.
I thought the ridiculousness was plain to read, still included the /s at the end, and yet...
We can come up with an infinite number of untrue, untestable theories. Usually once the drugs wear off they become much less interesting.
> Some years ago it was briefly popular to play games with your phone’s predictive-text feature; you would type an initial phrase and then repeatedly choose the middle option of the three words suggested by your phone, and the resulting sentence was often hilarious. It would be possible to interact with a contemporary LLM this way, and the resulting sentences would be perfectly sensible, but you probably wouldn’t feel like you were talking with someone. Yet that’s essentially what an LLM-based chatbot is, except that there’s no need to manually choose the middle option when it’s the chatbot’s turn to talk. It’s still a predictive-text game, but when the process is streamlined this way, the game becomes so engaging that some people find it addictive.
This hasn’t been true for ages. Just because this guy wrote Arrival doesn’t mean he knows anything else.
When will this stupid meme die?
We need a term for someone who thinks they are well informed on a topic (and perhaps may be, compared to the general public) but still don’t know what the fuck they are talking about when compared to even a competent non-expert in the field.
>I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
Finally I said something before Ted rather than the other way round!
https://techcrunch.com/2023/12/21/against-pseudanthropy/
While I agree with the premise here, I do think that it's easy for an arguer to move the goalposts such that the Caesar-Khan example no longer matters. The characters don't have to be conscious for the thing that created them (as in the case of the user doing it) to be so. So the argument would be that the creator of the characters is itself conscious, but not them. This feels like a kind of inverted no-true-scotsman type thing, but it does allow someone to retreat in some semblance of rationality.
I have many objections.
> if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot
Consciousness is independent of "assigning responsibility". Dogs cannot take responsibility for their actions but I believe they are conscious.
> we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious despite being disembodied and are happily conversing in a language that neither actually spoke. In reality, they are just characters in a piece of speculative fiction
This is a straw man. The obvious pro-consciousness claim would be that the LLM is the author of the fictional characters, and that the relationship between the LLM and Julius Caesar is analogous to the relationship between a human author and their fictional creations.
> Did changing the names of the characters from historical figures to generic roles cause the LLM to conjure up conscious entities who possess subjective experience?
No, again the LLM writing the text could potentially have a consciousness separate from the characters it authors.
> Some years ago it was briefly popular to play games with your phone’s predictive-text feature; [...] It would be possible to interact with a contemporary LLM this way, and the resulting sentences would be perfectly sensible, but you probably wouldn’t feel like you were talking with someone.
Yes, the same substrate is capable of hosting conscious and non-conscious forms, just like some arrangements of neurons are conscious, and some are not.
> But if the Caesar character were to become dispirited by something that the Khan character said, we shouldn’t become concerned in the slightest.
Even when there are characters, there may be actors behind the characters, for whom we could say "there is something it is to be like".
> we don’t need to worry if the transcript includes sentences where the chatbot character is sad. (We might need to worry if those sentences provoke sadness in the human user, but that’s a separate issue.)
It's actually not a separate issue. The LLM and the human are both adding sentences to the transcript. From the transcript we can make inferences about the mental state of the human. If the LLM has mental states, we could make inferences about those too.
> And note that it’s entirely possible for you to write five pages of dialogue between Caesar and Khan and then have an LLM extend the conversation; neither character had subjective experience when you were writing them, and that doesn’t change when you hand the task off to an LLM.
It's almost like he wants to make my point for me with this sentence.
> Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious
This smug shit really makes me angry for some reason. "Openness", i.e. uncertainty in the face of a completely novel situation, in the face of eons long struggle of humanity to understand what consciousness is and how it works, is just being naive.
> Should you consider the possibility that every time you open a Word document you are bringing multiple conscious interlocutors into existence, and every time you close one you snuff their existence out?
No, but if I find a word document I very well might try to use the signs it contains to make inferences about the mental state of its author.
> we are accustomed to reading intention into sentences, whereas we are not accustomed to reading intention into the way that amino acids fold into protein molecules.
He's trying to have it both ways here. Both that "obviously protein folding models aren't conscious because they don't emit sentences", but also "you are a rube for being tricked into thinking LLM models are conscious, because they do emit sentences".
> Obviously I’m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative will need a truly enormous amount of supporting evidence for it to deserve serious consideration
OK, that's fine for the author to not to be convinced, but that's not what's happening here, instead the author wrote a whole argument being convinced of the opposite viewpoint.
> It’s not plausible to me that a development path where the first step is a sentence-continuation machine that emits bad Julius Caesar dialogue and the next step is a sentence-continuation machine that emits decent Julius Caesar dialogue is one with a conscious Julius Caesar—or consciousness of any sort—as its endpoint
Actually a lot of things have happened? There were clearly many steps along they way from your phone's autocomplete to where we are now.
I totally agree with your reading/impressions of this essay.
I normally really like Chiang‘s writings so the complete disregard for the possibility of unknown complex emergent properties in large neural networks and the stunning lack of curiosity shown about this new and powerful technology and its potential shocked me from someone like him.
His being so sure about something he can’t possibly know felt like an insecure zealot desperately clinging to an object of faith rather than a calm, rational actor searching for truth in the face of the unknown.
Discussions on LLM consciousness feel like a psychological campaign to distract us from the awful effects of data centers and the current economic recession heading into a depression.
We can talk about consciousness when the LLMs have proven useful for all of humanity, not just their billionaire owners.
did you campaign against data centers before 2023-2024?
I find this kind of article unfortunate because it is very assertive in claiming things that are, not just untrue, but also confused about what consciousness really is.
Well, the problem is that there is little agreement about a widely accepted definition of consciousness and, in addition, this subject was actually left to philosophers, which is even worse because, in my view, they usually just produce a lot of nonsense in terms of definitions and arguments.
To me, a reasonable definition of consciousness is: a system which is capable of recognizing aggregate objects from a stream of sensory information it receives and which is capable of reasoning about the recognized object without immediately acting on it.
Well, what does "recognizing" mean? Merely that some of its neurons, related to the kind of identified aggregate object, are activated. These neurons, in a generalized sense, can be whatever things can work as neurons, just things that can be activated and propagate to other interconnected neurons.
For example, when we see through our eyes, we have an incoming stream of amorphous image information, but our brain can recognize that we are seeing a tree because we learned what a tree is, and when we see it, some neuron clusters activate to recognize a tree. In turn, when we recognize the tree, the thought propagates through our brain so that we are conscious of it.
In the same way, an LLM can perfectly recognize a tree from a stream of tokens — its sensory input, where the tokens describe a tree. The LLM will recognize that the tokens are describing a tree, and some of its "neurons" specific to the concept of a tree that the LLM had learned will be activated and will propagate through its brain. The fact that "neurons" are implemented as floating-point numbers for some parameters and connected just through a matrix does not mean they are not functionally capable of the same things; they are just implemented in a different way.
So the remaining part of my definition is "after recognizing an aggregate object, the thought propagates through the brain". The propagating part, to me, is just the very basic way a brain works: neurons are interconnected, and when some fire, other neurons fire, and that propagates.
In my opinion, consciousness has nothing to do with emotions or with survival. I do not see why emotion is necessary to consciousness; they are just different things. The author writes: "Without them [emotions], there is no conscious experience, only computation." But that makes no sense to me: the author has decided a priori that some things are "computations", and just because they are "computations", there cannot be "consciousness". But to me, this is a plainly wrong argument that does not hold.
I also do not see why the survival aspect would be needed for consciousness.
So to me, any recent reasoning LLM is conscious by the definition I gave, but also generally speaking. It is conscious upon a sensory stream of tokens: the LLM sees the world through tokens and expresses its thoughts through tokens; it does not mean it has no consciousness nonetheless. The fact that we do not give it a stable support to retain its memory and individuality is just a fact related to the way we build and use them, not about their intrinsic capacities.
Note: ChaptGPT came up with what is probably a better definition of my own: "A system is conscious, in a functional sense, when it can form internal representations of objects, states, or events from its sensory or informational input; make these representations globally available to many parts of itself; integrate them into a temporally persistent model of the world and of its own state; and use this integrated model for flexible reasoning, self-monitoring, and action selection independently of immediate stimulus-response behavior."
Looking at this thread, I think women have an obligation for the future of humanity not to procreate with people who deny consciousness or cannot experience it themselves.
If you are trapped in a tech bro relationship, think of humanity and cuckold your partner.
It is amusing to see so many venture capitalists suddenly become Marxists. You want your definition. Marx obliges:
"Contrary to idealism, which regards the world as the embodiment of an "absolute idea," a "universal spirit," "consciousness," Marx's philosophical materialism holds that the world is by its very nature material, that the multifold phenomena of the world constitute different forms of matter in motion, that interconnection and interdependence of phenomena as established by the dialectical method, are a law of the development of moving matter, and that the world develops in accordance with the laws of movement of matter and stands in no need of a "universal spirit."
Now go and implement other teachings of Marx, you dialectical venture capitalists!
It is mind boggling how many people, quite educated ones, take the AI as anything else besides what it fundamentally is - a very fast database search mimicking natural language.
There is no intelligence whatsoever, let alone consciousness.
It's so incredibly easy to fool us into applying human capacities to anything able to generate human-like language slop.
sociopaths sometimes study people to learn how to emulate emotion
that's exactly the state of "AI" right now, it's cold, mathematical emulation
btw there are some fascinating papers on the concept that consciousness in humans is actually a quantum effect
brilliant Roger Penrose proposed it (and they thought he was nuts) but recent discoveries about microtubules make it plausible
so who knows, maybe a dozen exponential improvements in quantum computers could make "AI" really conscious next century
* https://www.youtube.com/watch?v=xa2Kpkksf3k
https://archive.is/bcpZl
Really Chiang?
Alright. What is consciousness? Please provide a definition that somehow encompasses all humans and excludes all current AI.
I’ll wait.
Please don't fulminate on HN. The guidelines make it clear we're trying for something better here. https://news.ycombinator.com/newsguidelines.html
This isn't an example of fulminating, please don't incorrectly quote the hacker news guidelines at people. We strive for curious discussion here.
An example of fulminating would be: How dare you accuse him of fulminating? That is the most ass backwards moderating I have ever seen! Do you even know the definition of the word? Who paid you to call him out?
His comment may be sarcastic but it is not "fulminating".
Consciousness excludes all current AI because all current AI is just autocomplete over the corpus of human text.
It's just a word machine. There are no thoughts. It cannot be conscious. How is this even up for debate in any way whatsoever? I do not understand how people can believe this. Is this not a site for software engineers?
Suppose there is an autocomplete that consistently produces correct factorizations of 2048 bit primes. Do you still think its "merely" autocomplete or has it found some deep mathematical understanding of number theory internally? Would you maintain the same belief if it begins breaking encryption? This autocomplete argument is so strange, it betrays a complete lack of understanding of information theoretic and probability considerations.
What is consciousness, McGlockenshire?
It's not conscious if a nice person wouldn't feel bad torturing it.
Large volume of text for a fellow who apparently lacks familiarity with the study of consciousness in the first place.
Welcome to the folks at the Atlantic who are apparently brand new to “the hard problem”. Please start with reading Daniel Dennett and Donald Hoffman before writing articles about how you’ve cracked it.