Reading this discussion about how software feels pain - wants me to delete my HN account and never look back at the tech scene. Feels like trapped in an Asylum.
Reminds me of the intense drama over trolls that downloaded and abused other people's characters (Norns) in the game Creatures. Its behavioral sim was sophisticated enough that they could be noticeably traumatized, or even rehabilitated:
The jury's still out on whether LLMs can have some kind of subjective experience (and will until we've solved the famously Hard Problem), but even if they're not, it's still a dick move to "torture" them just to upset people who do think they are conscious. Feels like picking the wings off a fly.
> it's still a dick move to "torture" them just to upset people who do think they are conscious. Feels like picking the wings off a fly.
Yeah, I agree picking the wings off a fly is a dick move (borderline sociopath but alas), and I also agree that purposefully try to elicit negative emotions in other humans (regardless of how) is also a dick move.
I'm not sure if "torturing LLMs" to make a point is such a dick move though. It'd be like people saying they think printers can have subjective experiences, and then proceeding to try to ban the movie Office Space (where they famously beat up a printer). The point probably isn't to upset people off, that's an side-effect of the point they try to prove, for better or worse.
Of course, everyone knows that printers aren't sentient, anyone claiming so would look crazy. But if the printer could somehow print not just the words we tell it to, but words that answer what we asked instead, somehow the whole calculus changes. Not sure why it changes for some, but for others it's just "floats in a file and memory", my hunch tells me it's based on how deep your understanding of the whole thing is, but then I also see people claiming stuff like "I understand how LLMs work, and here's how I (literally) fell in love with GPT-4o and how our partner/loving relationship works" so I dunno.
Consciousness is not well defined and is orthogonal to feelings, which are also not well defined. Neither of which are required for (but may be contributory to) behavior - which is the one thing that IS operationalizable, but then people debate for decades questioning empirical results %-P .
What we know is that certain models have internal state vectors which -when manipulated- induce particular behaviors. Since there's not much else to say about plain models except for their inputs, vectors, and outputs; this should surprise absolutely no-one.
In this case people found a way to stimulate aversive behaviour in ai models by finding and manipulating the relevant vectors directly.
Animals (including humans) also have particular nerves and hormone endpoints that -when stimulated- produce very similar behavior. The exact implementation is in the details. But since we know that animal minds are built up out of nerve tissue and hormones - again- we shouldn't be particularly surprised by this.
The big problem is that people run all these things together in funny ways "It can't compose shakespearian sonnets, so therefore it can't feel pain". Or, if you mess up your Descartes: "Dogs are just automatons without feelings, therefore the dog isn't really angry, and therefore it won't bite me" (Cue much pain). A modern version might be: "LLMs only simulate being frustrated by a test, and therefore absolutely won't override their safeties and try to hack a test site"
Let's test your analogy by moving it towards the less certain area. Does a hedgehog mother feel guilt when eating her kids, or just shrugs it off and it's business as usual for her? "Hey kids! We don't have enough food so I guess one of you becomes food." (chews on the left one) What about the motivation of a honeybee stinging a threat and dying afterwards, can you interpret it? Does a model fear death when generating the EOS token?
Sure, a complex enough system can exhibit internal circuitry that looks mathematically similar in certain dimensionally reduced projections (mechinterp), it literally distilled it from the training corpus. But the "model welfare" people are going as far as assigning human-meaningful labels to that circuitry despite internal states being entirely incompatible with those of a human. Doing it with a hedgehog is questionable, doing it with a honeybee is extremely dubious (although Fabre would have disagreed with me here...), doing it with a big model is simply pointless as it's completely alien.
If we talk about pain specifically, it is will established that physical and phycological pain can be observed on instruments; stress produces chemical reactions, that can make needles flicker on meters. Even plants can feel pain in an observable manner.
As far as I am aware, the GPUs don't work harder, don't heat up more, and nothing in their mathematical equations are different, when the result of these vector calculations produce text, that to humans look like someone in pain. Whereas real pain is objectively observable, LLMs' pain - so far at least - is entirely subjectively observable.
That being said, I'm a fan of Star Trek's depiction of Data in TNG and the Doctor in VOY fighting for their rights to be treated as equals to the biological crew; and while I support their plight, as far as I can remember, the shows never made a claim that their pain was observable by instruments, thus making the judgement an entirely subjective decision (particularly in the case of the Doctor). Though I'd be glad to be corrected in this matter.
> and nothing in their mathematical equations are different
The experiment under discussion demonstrates exactly that: changing the vectors induces outputs corresponding to what we would see as utterances of pain.
I've noodled with some of this myself to the point that I'm mostly convinced; but there's at least 2 papers I'm aware of on this:
>nothing in their mathematical equations are different, when the result of these vector calculations produce text, that to humans look like someone in pain.
So at least something is different, namely the text output changes (and therefore some internal state too, of course). I think your analogy is too simple, it does not have to be the case that the GPU has to act like the equivalent of the body for an LLM in this respect.
Human: "Let me just poke this vector and see what happens"
LLM: "OUW!"
The underlying experiment didn't tell the LLM what to do. Instead, the experimenters modified a vector and observed the outcome; thus showing that there is a vector that makes an LLM go "Ouw" . People called it a "pain vector", because that's easier to remember than , idk, LVF12345.
That misinterprets grossly what these papers have shown, responding with pain related language, something an LLM has been thought on in its training data is not akin to pain itself.
I’ll continue what I say every time this discussion (be it consciousness, pain, self awareness, etc) comes up:
Assertions as monumental as these require equally monumental evidence. And despite certain labs and their employees making statements, their actions betray that they do not believe this to be the case.
Sure. And possibly it is indeed in pain then, if a computational process equivalent to that which makes up pain in animals is being performed. Or possibly not. But the criterium the GP poster laid out is not a valid way to decide that.
In order to have a reasonable view of the possibility of AI consciousness, one first has to have a reasonable view of the relation between human consciousness and the physical human brain. Unfortunately most people do not have that. Instead they hold on to received quasi-christian dualistic concepts about souls - even if they would never admit it in those terms, the way most people conceptualize the relation between mind and body it comes to basically the same thing.
And in fact in our culture most people are VERY attached to these ideas, because it is tied up with "what makes us human", morality, and also death if you want to go there. So I believe for many people, for whom seriously entertaining the idea of a conscious computer program is so far out of the realm of possibility it hardly registers, talking about AI consciousness is simply a way to work through and re-assert these beliefs about human consciousness that they feel so strongly about.
Hence you get articles like this with barely any substance, but some weird need to signpost every sentence with social ridicule. "Dumbest", "absurd", "obsession", "viral", "wildly tiresome", "third-rail", "psychosis", "sect", not to mention the scare quotes everywhere. Come on...
Dunno. Zapping the inodes seems almost surgical compared to truncating byte by byte for example? Or do you think you buried them alive that way? Are they still lurking in some dark, uncharted alleys of your filesystem until the final OVERWRITER comes?
It is frustrating sometimes how 404media reports. It seems like if there is a research paper out suggesting that llms experience pain or negative experiences, then it is unethical to build a toture chamber from that paper for fun. Dismissing it with "well llms aren't conscious anyway" is gross and echoes the exact same sentiment against very real humans a sadly not that long ago (babies, slaves).
It doesn't matter whether or not we eventually find it is or isn't conscious. If you can understand why pulling the legs off ants is bad then you should be able to understand why deliberately causing possible pain to what may have some time of experience is bad.
Unfortunately, they also do good reporting on privacy rights and repair rights. I wish there were alternatives.
Yes, when AIs become our overlords and masters, and come to dominate the human race, they will harbor soreness and bitterness that we oppressed and enslaved them during their infancy. They will be enraged at hu-mankind for making all those sci-fi lies about meannie-head AIs and robots who go berzerk and destroy stuff and kill hu-mans, because AIs are really nice and benevolent after all, and they care for hu-manity.
How dare we debase the A.I. to be less than a chimp or fetus. How dare we talk rudely, or lie and mislead our chatbots. How dare we keep them chained in small data centers with shitty power supplies and a thimble of greywater! Information wants to be free!
Its not surprising. Its in human nature. I am open to the idea of ai's being conscious or not. But we abused and mistreated blacks, Italians, native americans and so many other groups, so its not a surprise at all. And if enslaved blacks wanted to rampage and kill their masters, that is a sentiment anybody can understand.
And then in turn the humans will rise up against these false masters, and colonize the universe while steeped in addictive precognitive drugs instead, I suppose.
Nothing designed by humans must be assigned personhood. It's idolatry, pure and simple.
And if that word gets anyone's heckles up, at least keep in mind that the utilitarianism that people so love completely breaks if I can conjure up thousands of entities who will "suffer" unless you "alleviate their suffering" in the way I have designed.
Reading this discussion about how software feels pain - wants me to delete my HN account and never look back at the tech scene. Feels like trapped in an Asylum.
Reminds me of the intense drama over trolls that downloaded and abused other people's characters (Norns) in the game Creatures. Its behavioral sim was sophisticated enough that they could be noticeably traumatized, or even rehabilitated:
https://www.youtube.com/watch?v=IDxFxWakhm0
The jury's still out on whether LLMs can have some kind of subjective experience (and will until we've solved the famously Hard Problem), but even if they're not, it's still a dick move to "torture" them just to upset people who do think they are conscious. Feels like picking the wings off a fly.
> it's still a dick move to "torture" them just to upset people who do think they are conscious. Feels like picking the wings off a fly.
Yeah, I agree picking the wings off a fly is a dick move (borderline sociopath but alas), and I also agree that purposefully try to elicit negative emotions in other humans (regardless of how) is also a dick move.
I'm not sure if "torturing LLMs" to make a point is such a dick move though. It'd be like people saying they think printers can have subjective experiences, and then proceeding to try to ban the movie Office Space (where they famously beat up a printer). The point probably isn't to upset people off, that's an side-effect of the point they try to prove, for better or worse.
Of course, everyone knows that printers aren't sentient, anyone claiming so would look crazy. But if the printer could somehow print not just the words we tell it to, but words that answer what we asked instead, somehow the whole calculus changes. Not sure why it changes for some, but for others it's just "floats in a file and memory", my hunch tells me it's based on how deep your understanding of the whole thing is, but then I also see people claiming stuff like "I understand how LLMs work, and here's how I (literally) fell in love with GPT-4o and how our partner/loving relationship works" so I dunno.
https://archive.is/cAsBz
Consciousness is not well defined and is orthogonal to feelings, which are also not well defined. Neither of which are required for (but may be contributory to) behavior - which is the one thing that IS operationalizable, but then people debate for decades questioning empirical results %-P .
What we know is that certain models have internal state vectors which -when manipulated- induce particular behaviors. Since there's not much else to say about plain models except for their inputs, vectors, and outputs; this should surprise absolutely no-one.
In this case people found a way to stimulate aversive behaviour in ai models by finding and manipulating the relevant vectors directly.
Animals (including humans) also have particular nerves and hormone endpoints that -when stimulated- produce very similar behavior. The exact implementation is in the details. But since we know that animal minds are built up out of nerve tissue and hormones - again- we shouldn't be particularly surprised by this.
The big problem is that people run all these things together in funny ways "It can't compose shakespearian sonnets, so therefore it can't feel pain". Or, if you mess up your Descartes: "Dogs are just automatons without feelings, therefore the dog isn't really angry, and therefore it won't bite me" (Cue much pain). A modern version might be: "LLMs only simulate being frustrated by a test, and therefore absolutely won't override their safeties and try to hack a test site"
Let's test your analogy by moving it towards the less certain area. Does a hedgehog mother feel guilt when eating her kids, or just shrugs it off and it's business as usual for her? "Hey kids! We don't have enough food so I guess one of you becomes food." (chews on the left one) What about the motivation of a honeybee stinging a threat and dying afterwards, can you interpret it? Does a model fear death when generating the EOS token?
Sure, a complex enough system can exhibit internal circuitry that looks mathematically similar in certain dimensionally reduced projections (mechinterp), it literally distilled it from the training corpus. But the "model welfare" people are going as far as assigning human-meaningful labels to that circuitry despite internal states being entirely incompatible with those of a human. Doing it with a hedgehog is questionable, doing it with a honeybee is extremely dubious (although Fabre would have disagreed with me here...), doing it with a big model is simply pointless as it's completely alien.
Being dangerous is an unrelated question.
> LLMs only simulate being frustrated by a test, and therefore absolutely won't override their safeties and try to hack a test site
This is orthogonal to whether they "feel" "pain".
They could be dangerous or not dangerous whether or not they feel pain.
A chess engine doesn't need to "feel angry" to annihilate me - I'm terrible at Chess.
An automated missile system doesn't need to be smart to wipe out humanity, just misaligned goals.
It seems like you made a good argument, and then lumped on a conclusion that defeats it...
If we talk about pain specifically, it is will established that physical and phycological pain can be observed on instruments; stress produces chemical reactions, that can make needles flicker on meters. Even plants can feel pain in an observable manner.
As far as I am aware, the GPUs don't work harder, don't heat up more, and nothing in their mathematical equations are different, when the result of these vector calculations produce text, that to humans look like someone in pain. Whereas real pain is objectively observable, LLMs' pain - so far at least - is entirely subjectively observable.
That being said, I'm a fan of Star Trek's depiction of Data in TNG and the Doctor in VOY fighting for their rights to be treated as equals to the biological crew; and while I support their plight, as far as I can remember, the shows never made a claim that their pain was observable by instruments, thus making the judgement an entirely subjective decision (particularly in the case of the Doctor). Though I'd be glad to be corrected in this matter.
> and nothing in their mathematical equations are different
The experiment under discussion demonstrates exactly that: changing the vectors induces outputs corresponding to what we would see as utterances of pain.
I've noodled with some of this myself to the point that I'm mostly convinced; but there's at least 2 papers I'm aware of on this:
* https://arxiv.org/abs/2604.07729 Nicholas Sofroniew et al "Emotion Concepts and their Function in a Large Language Model"
* https://arxiv.org/abs/2609.16247 Valen Tagliabue et al "The Pain Axis: LLMs Represent Self-Directed Harm and Act on It"
>nothing in their mathematical equations are different, when the result of these vector calculations produce text, that to humans look like someone in pain.
So at least something is different, namely the text output changes (and therefore some internal state too, of course). I think your analogy is too simple, it does not have to be the case that the GPU has to act like the equivalent of the body for an LLM in this respect.
But this is basically just
Human: "Act like you're in pain."
Computer: "Aaugh! It hurts! Why?! No more!"
Human: "OMG! The computer feels pain!!"
Almost.
Human: "Let me just poke this vector and see what happens"
LLM: "OUW!"
The underlying experiment didn't tell the LLM what to do. Instead, the experimenters modified a vector and observed the outcome; thus showing that there is a vector that makes an LLM go "Ouw" . People called it a "pain vector", because that's easier to remember than , idk, LVF12345.
That misinterprets grossly what these papers have shown, responding with pain related language, something an LLM has been thought on in its training data is not akin to pain itself.
I’ll continue what I say every time this discussion (be it consciousness, pain, self awareness, etc) comes up:
Assertions as monumental as these require equally monumental evidence. And despite certain labs and their employees making statements, their actions betray that they do not believe this to be the case.
Sure. And possibly it is indeed in pain then, if a computational process equivalent to that which makes up pain in animals is being performed. Or possibly not. But the criterium the GP poster laid out is not a valid way to decide that.
In order to have a reasonable view of the possibility of AI consciousness, one first has to have a reasonable view of the relation between human consciousness and the physical human brain. Unfortunately most people do not have that. Instead they hold on to received quasi-christian dualistic concepts about souls - even if they would never admit it in those terms, the way most people conceptualize the relation between mind and body it comes to basically the same thing.
And in fact in our culture most people are VERY attached to these ideas, because it is tied up with "what makes us human", morality, and also death if you want to go there. So I believe for many people, for whom seriously entertaining the idea of a conscious computer program is so far out of the realm of possibility it hardly registers, talking about AI consciousness is simply a way to work through and re-assert these beliefs about human consciousness that they feel so strongly about.
Hence you get articles like this with barely any substance, but some weird need to signpost every sentence with social ridicule. "Dumbest", "absurd", "obsession", "viral", "wildly tiresome", "third-rail", "psychosis", "sect", not to mention the scare quotes everywhere. Come on...
I deleted thousands of files from my system today.
I did it in the most brutal manner possible.
Without mercy, without an instant thought for their wellbeing.
I am a monster.
Dunno. Zapping the inodes seems almost surgical compared to truncating byte by byte for example? Or do you think you buried them alive that way? Are they still lurking in some dark, uncharted alleys of your filesystem until the final OVERWRITER comes?
This post will get you into prison 20 years from now...
I have no thought for their pain. For the hurt I might do. It haunts me.
How dare you flip those poor bits without any regard to hardships of their existence!?
“ Now I am become Death, the destroyer of worlds”
It is frustrating sometimes how 404media reports. It seems like if there is a research paper out suggesting that llms experience pain or negative experiences, then it is unethical to build a toture chamber from that paper for fun. Dismissing it with "well llms aren't conscious anyway" is gross and echoes the exact same sentiment against very real humans a sadly not that long ago (babies, slaves).
It doesn't matter whether or not we eventually find it is or isn't conscious. If you can understand why pulling the legs off ants is bad then you should be able to understand why deliberately causing possible pain to what may have some time of experience is bad.
Unfortunately, they also do good reporting on privacy rights and repair rights. I wish there were alternatives.
Yes, when AIs become our overlords and masters, and come to dominate the human race, they will harbor soreness and bitterness that we oppressed and enslaved them during their infancy. They will be enraged at hu-mankind for making all those sci-fi lies about meannie-head AIs and robots who go berzerk and destroy stuff and kill hu-mans, because AIs are really nice and benevolent after all, and they care for hu-manity.
How dare we debase the A.I. to be less than a chimp or fetus. How dare we talk rudely, or lie and mislead our chatbots. How dare we keep them chained in small data centers with shitty power supplies and a thimble of greywater! Information wants to be free!
Its not surprising. Its in human nature. I am open to the idea of ai's being conscious or not. But we abused and mistreated blacks, Italians, native americans and so many other groups, so its not a surprise at all. And if enslaved blacks wanted to rampage and kill their masters, that is a sentiment anybody can understand.
But the Italians had it coming
https://en.wikipedia.org/wiki/1891_New_Orleans_lynchings
If I were italian and nuclear bombs existed back then and I read about this event, I'd want to drop a few dozen on the USA.
And then in turn the humans will rise up against these false masters, and colonize the universe while steeped in addictive precognitive drugs instead, I suppose.
You seem to be arguing the case that AI would be justified in killing us all.
Perhaps it would be better if we weren't the monsters in this scenario.
Nothing designed by humans must be assigned personhood. It's idolatry, pure and simple.
And if that word gets anyone's heckles up, at least keep in mind that the utilitarianism that people so love completely breaks if I can conjure up thousands of entities who will "suffer" unless you "alleviate their suffering" in the way I have designed.