I'm not so sure that the argument that some humans will survive would placate anyone who is worried about a catastrophic outcome from AI development. You could make the same argument for climate collapse or global nuclear war, but I also don't think that's gonna make anyone feel better about those scenarios
My most likely doom scenario is that a frontier model convinces the CIC to press the nuclear retaliation button. Maybe this will only kill 90% of humans, but as far as I am concerned, that’s good enough.
I think a realistic scenario is mass murders triggered by AI psychosis. Some weirdos start an AI cult and end up killing a bunch of randos because the AI said doing so would break the matrix or something similar. I am actually surprised this hasn't happened yet.
"Bro just don't synthesize the virus that wipes out humanity if the ai tries to make one"
Because humans are famously excellent at never running files that someone posing as a trusted colleague with a solution to an urgent problem gives them. /s
I'm not so sure that the argument that some humans will survive would placate anyone who is worried about a catastrophic outcome from AI development. You could make the same argument for climate collapse or global nuclear war, but I also don't think that's gonna make anyone feel better about those scenarios
My most likely doom scenario is that a frontier model convinces the CIC to press the nuclear retaliation button. Maybe this will only kill 90% of humans, but as far as I am concerned, that’s good enough.
not that good
I think a realistic scenario is mass murders triggered by AI psychosis. Some weirdos start an AI cult and end up killing a bunch of randos because the AI said doing so would break the matrix or something similar. I am actually surprised this hasn't happened yet.
Great job, now you just told AI how to get away with it.
i thought virus engineering was a scenario already confirmed
Yeah, but the counter is:
"Bro just don't synthesize the virus that wipes out humanity if the ai tries to make one"
Because humans are famously excellent at never running files that someone posing as a trusted colleague with a solution to an urgent problem gives them. /s